An adaptive optoelectronic hybrid computing system for end-to-end autonomous driving

Abstract

The application provides an adaptive optoelectronic hybrid computing system for end-to-end automatic driving, and relates to the technical field of automatic driving, and comprises: a heterogeneous computing abstraction layer, which is used for shielding the differences of optoelectronic hybrid hardware based on a pre-constructed unified device model for an automatic driving application software stack; an optoelectronic operator library, which is used for providing a unified description framework for each operator based on a pre-constructed optoelectronic operator model; an optoelectronic collaborative graph compiler, which is used for calling the heterogeneous computing abstraction layer and the optoelectronic operator library based on an automatic driving algorithm flowchart, formalizing a hardware mapping problem of the automatic driving algorithm flowchart into a constraint multi-objective optimization problem, and outputting a static scheduling strategy based on a solving result; and a system scheduler, which is used for constructing a state vector based on a running state and inputting the state vector into a trained DQN network, and executing a scheduling strategy corresponding to a target action output by the DQN network. The system can optimize optoelectronic hybrid computing resource allocation and realize an optimal global scheduling strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to an adaptive optoelectronic hybrid computing system for end-to-end automatic driving. BACKGROUND

[0002] At present, the pure digital processing unit-based computing architecture has increasingly prominent inherent von Neumann bottleneck, memory wall problem and high power consumption problem when coping with the growing algorithm complexity and massive data flow. These bottlenecks make it difficult for the existing architecture to continuously provide computing capability meeting the safety requirements of future high-level automatic driving under the strict constraints of the vehicle-mounted environment.

[0003] The light computing technology brings new technical dawn for breaking through the automatic driving computing bottleneck, with its inherent high parallelism, low power consumption and low delay. However, how to systematically and efficiently integrate the light computing unit into the complex automatic driving system and realize seamless and efficient collaboration with the digital computing unit is a key bottleneck faced by current technology development. SUMMARY

[0004] In view of this, the present application provides an adaptive optoelectronic hybrid computing system for end-to-end automatic driving.

[0005] In the first aspect, the present application provides an adaptive optoelectronic hybrid computing system for end-to-end automatic driving, comprising a heterogeneous computing abstraction layer, an optoelectronic operator library, an optoelectronic collaborative graph compiler and a system scheduler.

[0006] The heterogeneous computing abstraction layer is used to shield the differences of the optoelectronic hybrid hardware based on a pre-constructed unified device model, and to manage and schedule the underlying hardware resources for the automatic driving application software stack.

[0007] The optoelectronic operator library provides a unified description framework for each operator based on a pre-constructed optoelectronic operator model, and defines the key attributes of each operator.

[0008] The optoelectronic collaborative graph compiler is used to receive an automatic driving algorithm flowchart, invoke the heterogeneous computing abstraction layer and the optoelectronic operator library based on the automatic driving algorithm flowchart, formalize the hardware mapping problem of the automatic driving algorithm flowchart into a constrained multi-objective optimization problem, solve the constrained multi-objective optimization problem to obtain an optimal allocation scheme of hardware resources, and output a static scheduling strategy based on the optimal allocation scheme.

[0009] The system scheduler is used to monitor the running state of the automatic driving system in real time, construct a state vector based on the running state and input it into a trained DQN network, obtain a target action with the highest Q value, and control the automatic driving system to execute the scheduling strategy corresponding to the target action.

[0010] In an embodiment, the unified device model comprises a light computing unit capability characterization sub-model, which comprises an operation support module, a format adaptation module and an operation module;

[0011] The operation support module is configured to determine specific operations that the light computing unit is good at performing;

[0012] The format adaptation module is configured to evaluate the degree of adaptation of the light computing unit to specific input data formats and the necessary preprocessing requirements;

[0013] The operation module is configured to model the output of the light computing unit as a probability distribution centered on an ideal digital computing result, wherein the distribution parameters of the probability distribution are obtained based on the state parameters of the light computing unit and a pre-trained mapping model.

[0014] In an embodiment, the automatic driving algorithm flowchart is a directed acyclic graph, the nodes of the directed acyclic graph represent computing operators that perform specific operations, and the edges of the directed acyclic graph represent data dependency relationships.

[0015] In an embodiment, the optoelectronic collaborative compiler comprises an identification module and a decision module;

[0016] The identification module is configured to identify a bottleneck module in the automatic driving algorithm that is computationally intensive and suitable for acceleration by optical computing, replace the node corresponding to the bottleneck module in the automatic driving algorithm flowchart with an optical operator, and fuse a sequence of operators in the automatic driving algorithm flowchart that are continuous and can be efficiently processed by the same optical computing unit;

[0017] The decision module is configured to invoke the heterogeneous computing abstraction layer and the optoelectronic operator library based on the automatic driving algorithm flowchart, generate a multi-dimensional cost matrix for possible mapping combinations in the automatic driving algorithm flowchart, formalize the hardware mapping problem of the automatic driving algorithm flowchart into a constrained multi-objective optimization problem based on the structure of the automatic driving algorithm flowchart, node meta-information and each multi-dimensional cost matrix, solve the constrained multi-objective optimization problem using a heuristic algorithm, obtain an optimal allocation scheme of hardware resources, output a static scheduling strategy based on the optimal allocation scheme, and wherein the mapping combination is a combination of associatively mapping an optical acceleration candidate node with an optical computing unit.

[0018] In an embodiment, the system scheduler comprises a monitoring module and a scheduling module;

[0019] The monitoring module is configured to monitor a current hardware state and a task state of the autonomous driving system, and acquire sensor data processed by the optical computing unit and the digital processing unit, and determine a current environment state based on the sensor data;

[0020] The scheduling module is configured to model a runtime scheduling decision problem of the autonomous driving system as a Markov decision process, input a state vector into a trained DQN network, solve the Markov decision process by using the DQN network, obtain a target action with a highest Q value, and control the autonomous driving system to execute a scheduling strategy corresponding to the target action.

[0021] In an embodiment, the monitoring module is further configured to determine performance degradation prediction information of the optical processing unit and load peak prediction information of the digital processing unit by using a pre-trained prediction model based on historical running data and a change trend of the current state;

[0022] The scheduling module is further configured to calibrate an output result of the optical processing unit and adjust the scheduling strategy corresponding to the target action according to the performance degradation prediction information and the load peak prediction information.

[0023] In an embodiment, the scheduling module is further configured to preset a periodic calibration time window based on hardware characteristics of the optical processing unit, and calibrate the output result of the optical processing unit according to the performance degradation prediction information when the periodic calibration time window is reached.

[0024] In an embodiment, the scheduling module is further configured to determine a performance degradation parameter of the optical processing unit by multi-dimensional detection, determine a fault level of the optical processing unit according to the performance degradation parameter, determine affected information of an autonomous driving function according to the fault level of the optical processing unit, and execute a preset fault response strategy according to the affected information.

[0025] In an embodiment, the scheduling module is further configured to:

[0026] In a case where the affected autonomous driving function is a core function and the faulty optical processing unit has a backup digital processing unit, the working task of the faulty optical processing unit is transferred to the backup digital processing unit;

[0027] In a case where the affected autonomous driving function is a core function and the faulty optical processing unit does not have a backup digital processing unit, the autonomous driving system is controlled to execute a control strategy corresponding to a minimum safety risk;

[0028] In a case where the affected autonomous driving function is a non-core function, the scheduling strategy currently executed by the autonomous driving system is switched to a scheduling strategy with higher safety.

[0029] In a second aspect, the application further provides an automatic driving method, which is executed based on the adaptive optoelectronic hybrid computing system for end-to-end automatic driving according to the first aspect, and the automatic driving method comprises:

[0030] obtaining a current running state of the automatic driving system;

[0031] constructing a state vector based on the running state and inputting the state vector into the trained DQN network to obtain a target action with the highest Q value;

[0032] controlling the automatic driving system to execute a scheduling strategy corresponding to the target action, wherein different actions of the DQN network correspond to static scheduling strategies output by the optoelectronic collaborative graph compiler in different scenarios, and the running state comprises a hardware state, a task state and an environment state.

[0033] The adaptive optoelectronic hybrid computing system for end-to-end automatic driving according to the application has the following beneficial effects relative to the related art:

[0034] 1. The adaptive optoelectronic hybrid computing system according to the application, the heterogeneous computing abstraction layer shields the differences of the optoelectronic hybrid hardware based on the pre-constructed unified device model for the automatic driving application software stack, and can realize unified management and scheduling of heterogeneous resources. The optoelectronic operator library provides a unified description framework for each operator based on the pre-constructed optoelectronic operator model, and defines the key attributes of each operator. On this basis, the optoelectronic collaborative graph compiler can call the heterogeneous computing abstraction layer and the optoelectronic operator library based on the automatic driving algorithm flowchart, formalize the hardware mapping problem of the automatic driving algorithm flowchart into a constrained multi-objective optimization problem, and solve the constrained multi-objective optimization problem to obtain an optimal allocation scheme of hardware resources, output a static scheduling strategy based on the optimal allocation scheme, thereby optimizing the allocation of optoelectronic hybrid computing resources, giving full play to the advantages of optical computing in specific operations, while ensuring the flexibility and versatility of digital computing, realizing the collaboration of optical computing units and digital processing units, and breaking through the performance and power consumption bottleneck of traditional digital chips in automatic driving applications through intelligent fusion of optical computing and electrical computing.

[0035] 2. The system scheduler can construct a state vector based on the running state of the automatic driving system, input the state vector into the trained DQN network, obtain a target action with the highest Q value, and control the automatic driving system to execute a scheduling strategy corresponding to the target action, thereby realizing real-time perception of driving scenarios and risks, and autonomously selecting the optimal global scheduling strategy according to the complex and dynamically changing system state and driving environment, and systematically improving the overall safety, intelligent level and real feasibility of high-level automatic driving systems. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0037] Figure 1 A structural schematic diagram of an adaptive optoelectronic hybrid computing system for end-to-end autonomous driving in an embodiment of the present application;

[0038] Figure 2 A structural schematic diagram of an adaptive optoelectronic hybrid computing system for end-to-end autonomous driving in another embodiment of the present application;

[0039] Figure 3 A structural schematic diagram of an adaptive optoelectronic hybrid computing system for end-to-end autonomous driving in still another embodiment of the present application;

[0040] Figure 4 A structural schematic diagram of an adaptive optoelectronic hybrid computing system for end-to-end autonomous driving in yet another embodiment of the present application;

[0041] Figure 5 A flowchart of an autonomous driving method in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the embodiments of the present application. Obviously, the described embodiments only constitute some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0043] In some embodiments, as shown in FIG. 1, the present application provides an adaptive optoelectronic hybrid computing system 10 for end-to-end autonomous driving, which comprises a heterogeneous computing abstraction layer 11, an optoelectronic operator library 12, an optoelectronic co-graph compiler 13 and a system scheduler 14. Figure 1

[0044] The heterogeneous computing abstraction layer 11 is used to shield the differences of the optoelectronic hybrid hardware based on a pre-constructed unified device model, and to manage and schedule the underlying hardware resources, and the unified device model defines a set of standard interface specifications and core attribute sets for both the optical computing unit (OCU) and the digital processing unit (DPU).

[0045] ​The heterogeneous computing abstraction layer 11 is the core enabler for building efficient opto-electric hybrid computing systems. Based on the pre-constructed unified device model, the heterogeneous computing abstraction layer 11 can effectively shield the upper-layer autonomous driving application software stack from the inherent physical differences, complex interface specifications, and tedious operation details of the underlying opto-electric hybrid hardware. More importantly, the heterogeneous computing abstraction layer 11 provides a hardware capability accurate characterization oriented to the specific sensor data processing needs of autonomous driving, i.e., the unified device model defines a set of standard interface specifications and core attribute sets for both optical computing units and digital processing units, and implements unified management and scheduling of heterogeneous resources in a standardized manner, thereby providing solid support for perception, decision-making, and planning of the upper-layer application.

[0046] Another important function of the heterogeneous computing abstraction layer 11 is to serve as a context manager. The heterogeneous computing abstraction layer 11 plays the role of a core coordinator of the heterogeneous device runtime environment. It is responsible for comprehensively managing the runtime state of all heterogeneous devices in the system, implementing efficient resource allocation strategies, and controlling the initialization and safe release processes of the devices. For optical computing units (OCUs), the heterogeneous computing abstraction layer 11 also undertakes additional and more detailed management responsibilities, including accurate management of OCU-specific optical configuration parameters such as light intensity modulation depth, critical bias voltage settings, and temperature compensation coefficients for compensating for environmental changes. At the same time, the heterogeneous computing abstraction layer 11 can also track and manage the state of the OCU to ensure that it maintains optimal computing performance and stability under various working conditions.

[0047] The opto-electric operator library 12 provides a unified description framework for each operator based on the pre-constructed opto-electric operator model, defining the key attributes of each operator. The key attributes of an operator can include: detailed description of input / output tensors, such as data type, dimension, and domain (optical domain or electrical domain) where the data resides; specific parameters required for executing the operator; the type of unified device model (PeDevice) that the operator depends on during runtime; and performance estimation indicators for the operator, covering latency, power consumption, and potential accuracy loss. The opto-electric operator model provides a theoretical basis for the standardization, scheduling, and optimization of operators.

[0048] The basic optical operators constitute the core of optical computing. These operators include but are not limited to optical matrix vector multipliers (OMVMs), optical convolution engines (OCEs), optical Fourier transform units (OFTUs), and optical correlators. Each optical operator encapsulates an abstraction of its physical implementation within. For example, an OCE can be implemented based on free-space optical architecture or integrated optical waveguide technology, and its model will accordingly reflect the performance differences (such as throughput, energy efficiency, crosstalk, etc.) brought about by different physical implementations, allowing the upper-layer caller to be free from concerns about the underlying hardware details.

[0049] A key feature of optical operators is the embedded photoelectric conversion model. For optical operators that need to interact with electrical signals, they internally encapsulate sophisticated encoding and decoding models. The photoelectric conversion model not only considers the basic parameters of analog-to-digital converters (ADCs) and digital-to-analog converters (DACs) (such as bit number, sampling rate), but more importantly, it incorporates the characteristics of autonomous driving sensor data, such as the dynamic range of LiDAR intensity values, the distribution of image pixel values, and the statistical properties of data (such as the sparsity of point clouds). In this way, the photoelectric conversion model aims to optimize the fidelity of signal conversion between the optical and electrical domains, effectively suppress quantization noise, and improve overall conversion efficiency. Taking an optical operator that processes sparse radar point clouds as an example, its encoding model may employ advanced techniques such as compressed sensing to adapt to data sparsity and optimize the representation of optical signals.

[0050] It should be noted that the photoelectric operator library 12 also includes an autonomous driving specific high-level API (AD_OptiAPI). This API aims to significantly lower the threshold for autonomous driving algorithm developers to use complex opto-electric hybrid computing architectures. The autonomous driving specific high-level API provides a series of task-oriented interfaces, such as: optical processing LiDAR point cloud API, optical accelerating CNN perception backbone API, and optical fusing multi-modal API.

[0051] Optical processing LiDAR point cloud API (process_lidar_point_cloud_optically): contains point cloud data, voxel size, detection parameters (i.e. point_cloud_data, voxel_size, detection_params). This API may internally encapsulate a complete opto-electric hybrid processing flow. It may first call the DPU for point cloud preprocessing (such as filtering, ground point removal), then map the point cloud data (or its voxelized representation) to optical signals through an efficient electro-optical (E-O) encoder, then drive the OCU to perform core operations such as optical 3D convolution or FFT-based feature extraction, and then convert the processing results back to electrical signals via an opto-electric (O-E) decoder, and finally complete post-processing steps such as clustering and candidate box generation by the DPU.

[0052] Optical accelerating CNN perception backbone:

[0053] Using the API (accelerate_cnn_perception_backbone_optically), it contains picture batch processing, CNN model optical weights, target layer range, i.e.:

[0054] image_batch, cnn_model_weights_optical, target_layer_range

[0055] This API allows batching of image data into the OCU to accelerate the computation-intensive convolutional layers and some fully-connected layers in a convolutional neural network (CNN) perception backbone network. The relevant weight parameters can be pre-loaded into the OCU or dynamically configured on demand.

[0056] Optical fusion multi-modal feature API (fuse_multimodal_features_optically):

[0057] Feature vector camera, feature vector lidar, attention weights optical (feature_vector_camera, feature_vector_lidar, attention_weights_optical). This API leverages the efficiency of the OCU in performing large-scale matrix multiplication to enable deep fusion of multi-modal features (e.g., camera feature vector and lidar feature vector) based on an attention mechanism.

[0058] The internal logic of these high-level APIs encapsulates complex sequences of optoelectronic operator calls, data flow management between the OCU and the DPU, and necessary digital domain preprocessing and postprocessing steps. They directly map to specific perception or decision-making sub-tasks in an autonomous driving system, enabling algorithm developers to leverage the potential of optoelectronic computing with an abstraction closer to the application level, without needing to delve into the details of the underlying hardware and optoelectronic conversion.

[0059] The optoelectronic collaborative graph compiler 13 is configured to receive an autonomous driving algorithm flow graph, invoke the heterogeneous computing abstraction layer 11 and the optoelectronic operator library 12 based on the autonomous driving algorithm flow graph, formalize a hardware mapping problem of the autonomous driving algorithm flow graph into a constrained multi-objective optimization problem, solve the constrained multi-objective optimization problem, obtain an optimal allocation scheme of hardware resources, and output a static scheduling strategy based on the optimal allocation scheme.

[0060] The photoelectric collaborative graph compiler 13 is a key software component to release the potential of photoelectric hybrid computing. The core design idea is to efficiently and intelligently map the end-to-end algorithm flow defined by the user or the upper automatic driving framework, usually represented as a computational graph from sensor input to control instruction output, i.e., an automatic driving algorithm flow graph, to heterogeneous photoelectric hybrid hardware for execution. The photoelectric collaborative graph compiler 13 calls the heterogeneous computing abstraction layer 11 and the photoelectric operator library 12 based on the automatic driving algorithm flow graph, formalizes the hardware mapping problem of the automatic driving algorithm flow graph into a constrained multi-objective optimization problem, solves the constrained multi-objective optimization problem, obtains an optimal allocation scheme of hardware resources, and outputs a static scheduling strategy based on the optimal allocation scheme. Using the static scheduling strategy can optimize the key safety path in the automatic driving system and improve the overall system energy efficiency, fully exert the advantages of optical computing in specific operations, and at the same time guarantee the flexibility and versatility of digital computing.

[0061] The system scheduler 14 is used to monitor the running state of the automatic driving system in real time, constructs a state vector based on the running state and inputs it into the trained DQN network, obtains the target action with the highest Q value, and controls the automatic driving system to execute the scheduling strategy corresponding to the target action, wherein different actions of the DQN network correspond to the static scheduling strategies output by the photoelectric collaborative graph compiler 13 in different modes, and the running state includes hardware state, task state and environment state. Exemplarily, the working modes can include high-performance priority mode, low-power consumption mode, balanced mode and emergency safety mode.

[0062] The system scheduler 14 is the "brain" and "nerve center" of the photoelectric hybrid computing architecture in actual operation. The system scheduler 14 monitors and dynamically manages the photoelectric hybrid computing resources in real time during system operation. It can autonomously select the optimal global scheduling strategy according to complex and dynamically changing system states and driving environments, and systematically improves the overall safety, intelligence level and real feasibility of high-level automatic driving systems.

[0063] The aforementioned adaptive optoelectronic hybrid computing system includes a heterogeneous computing abstraction layer 11, an optoelectronic operator library 12, an optoelectronic co-graph compiler 13, and a system scheduler 14. The heterogeneous computing abstraction layer 11, based on a pre-built unified device model, shields the autonomous driving application software stack from the differences in optoelectronic hybrid hardware. Furthermore, the unified device model defines a standard set of interface specifications and core attribute sets for both optical computing units and digital processing units, employing a standardized approach to achieve unified management and scheduling of heterogeneous resources. The optoelectronic operator library 12, based on a pre-built optoelectronic operator model, provides a unified description framework for each operator and defines the key attributes of each operator. Based on this, the optoelectronic co-graph compiler 13 can call the heterogeneous computing abstraction layer 11 and the optoelectronic operator library 12 based on the autonomous driving algorithm flowchart, formalize the hardware mapping problem of the autonomous driving algorithm flowchart into a constrained multi-objective optimization problem, solve the constrained multi-objective optimization problem, obtain the optimal allocation scheme of hardware resources, and output a static scheduling strategy based on the optimal allocation scheme, thereby optimizing the allocation of optoelectronic hybrid computing resources, giving full play to the advantages of optical computing in specific operations, while ensuring the flexibility and versatility of digital computing, realizing the collaboration between optical computing units and digital processing units, and breaking through the performance and power consumption bottlenecks of traditional digital chips in autonomous driving applications by intelligently integrating optical computing and electrical computing. The system scheduler 14 can construct a state vector based on the running state of the autonomous driving system and input it into the pre-trained DQN network to obtain the target action with the highest Q value, and control the autonomous driving system to execute the scheduling strategy corresponding to the target action, thereby perceiving the driving scenario and risks in real time, and being able to autonomously select the optimal global scheduling strategy online according to the complex and dynamically changing system state and driving environment, systematically improving the overall safety, intelligence level and practical feasibility of the high-level autonomous driving system.

[0064] In some embodiments, such as Figure 2 As shown, the unified device model includes a sub-model representing the capabilities of optical computing units, which contains a computation support module 111, a format adaptation module 112, and a computation module 113.

[0065] The computation support module 111 is used to determine the specific operations that the optical computing unit is good at performing. The format adaptation module 112 is used to evaluate the optical computing unit's adaptability to specific input data formats and the necessary preprocessing requirements. The computation module 113 is used to model the output of the optical computing unit as a probability distribution centered on the ideal digital computation result, wherein the distribution parameters of the probability distribution are obtained based on the state parameters of the optical computing unit and a pre-trained mapping model. It should be noted that the unified device model may also include a sub-model representing the capabilities of the digital processing unit.

[0066] Among them, for the uniqueness of the optical computing unit (OCU), the OCU capability characterization sub-model particularly contains detailed OCU capability characterization. The OCU capability characterization not only covers the general optical parameters of the OCU, such as the working wavelength, the available bandwidth, the supported modulation mode, and the basic optical characteristics such as the configurability, but also focuses on evaluating and describing the ability to process typical complex data types in the automatic driving scene. These typical data include high-dimensional sparse lidar point cloud, multi-channel high dynamic range image sequence, and radar signal cube, etc.

[0067] Specifically, the optical computing unit capability characterization sub-model includes an operation support module 111, a format adaptation module 112, and an operation module 113.

[0068] The operation support module 111 explicitly specifies the specific operations that the OCU is good at performing, such as efficient Fourier transform, specific types of convolution operations (such as depth separable convolution, sparse convolution to adapt to different data structures), and matrix multiplication (and details of the supported matrix dimension range).

[0069] The format adaptation module 112 evaluates the degree of adaptation of the OCU to specific input data formats and the necessary preprocessing requirements. This includes compatibility and optimization potential for different lidar scanning modes (such as traditional rotating, MEMS solid-state lidar, Flash solid-state lidar) or various image sensor output formats (such as RAW, YUV).

[0070] The operation module 113 can realize modeling the output of the optical computing unit as a probability distribution centered on the ideal digital calculation result based on the pre-constructed probabilistic precision-fidelity model (Probabilistic Precision-Fidelity Model, P²FM). Modeling the output of the optical computing unit as a probability distribution centered on the ideal digital calculation result can more scientifically and finely quantify the inherent analog computing characteristics of the OCU, rather than just mapping to a single equivalent digital bit width. The probabilistic precision-fidelity model no longer regards the output of the OCU as a deterministic value with a certain fixed error, but models it as a probability distribution centered on the ideal digital calculation result. Specifically, for a given optical operation, its output y_optical is described as y_optical~P(y|y_ideal, θ_env, φ_op), where y_ideal is the error-free ideal output, θ_env is a parameter vector containing environmental factors such as temperature and voltage fluctuations, and φ_op is a configuration parameter related to the specific operation (such as optical intensity modulation depth). The form of the probability distribution P (such as mean, variance, skewness) is determined by the physical characteristics of the OCU and obtained through offline calibration and online fine-tuning.

[0071] For an operator (e.g., matrix multiplication) executed by an OCU, the relationship between its output y_optical and the ideal output y_ideal can be expressed as:

[0072] y_optical=y_ideal+ε( output y_optical)

[0073] where y_ideal is the error-free ideal output, and ε is an error term, which is modeled by P²FM as a random variable following a probability distribution D: ε ~ D(μ_ε,σ²_ε,...|θ_env,φ_op).

[0074] In one example, D can be approximated as a Gaussian distribution N(μ_ε,σ²_ε), which is reasonable in many physical systems. But in other cases, more complex distributions such as skew distributions or mixture Gaussian models can also be adopted according to the nonlinear characteristics of the OCU.

[0075] The distribution parameters (e.g., mean μ_ε and variance σ²_ε) of the probability distribution D are not fixed but functions of the current system state. θ_env is the environmental parameter vector, containing physical quantities that can be monitored in real time, such as the OCU core temperature T, the supply voltage V of key components, etc. φ_op is the operator operation configuration parameter, such as the light modulation depth M used to encode the input data, the operating wavelength λ, etc.

[0076] The mean and variance are related to the state parameters through mapping functions g_μ and g_σ:

[0077] μ_ε=g_μ(θ_env,φ_op)

[0078] σ²_ε=g_σ(θ_env,φ_op)

[0079] These two mapping functions (which can be polynomial regression models, lookup tables (LUTs), or small neural networks) are the core of the P²FM model and are learned through an offline calibration process. This can include the following steps: data collection, error sample acquisition, distribution fitting, and mapping function learning.

[0080] Data collection: In a controlled experimental environment, systematically scan the working parameter space of an OCU prototype. That is, under different combinations of temperature T, voltage V, and modulation depth M, input a large number of test data with known ideal output y_ideal (e.g., random matrices, real image blocks from autonomous driving data sets, etc.).

[0081] Error sample acquisition: For each working point (θ_env,φ_op), collect a large number of outputs y_optical and calculate the error sample set {ε_i=y_optical_i-y_ideal_i}.

[0082] Distribution fitting: Perform statistical analysis on each error sample set, fit the most suitable probability distribution D, and estimate its parameters (such as μ_ε and σ²_ε).

[0083] Mapping function learning: Take all the collected working points (θ_env, φ_op) as input, and their corresponding distribution parameters (μ_ε, σ²_ε) as output, use machine learning methods (such as multiple regression, gradient boosting tree or neural network) to train the mapping function g_μ and g_σ. The trained model is solidified in the HCAL.

[0084] In some embodiments, the automatic driving algorithm flowchart is a directed acyclic graph, the nodes of the directed acyclic graph represent the calculation operators that perform specific operations, and the edges of the directed acyclic graph represent the data dependency relationship.

[0085] Among them, each node in the directed acyclic graph can be assigned rich meta-information, such as its criticality in the entire automatic driving process (such as whether it belongs to the perception path of automatic emergency braking AEB or the control path of lane keeping assistance LKA), strict real-time requirement and specific demand for calculation accuracy. These meta-information are the basis for subsequent optimization decisions.

[0086] It can be understood that the operator is the basic functional unit of the algorithm, which may include, for example, noise reduction processing of image data, clustering analysis of laser radar point cloud, convolutional feature extraction of obstacles, cost function calculation of path planning, etc. Each node encapsulates a specific input and output format and operation logic. The edges in the graph are directional and are used to represent the data dependency between nodes: if there is a directed edge from node A to node B, it means that the operation of node B (i.e. the calculation operator corresponding to B) must rely on the output data of node A as input, for example, the output (feature vector) of the "obstacle feature extraction" node will be used as the input of the "obstacle risk assessment" node, and there will be a directed edge from the feature extraction node to the risk assessment node, which clearly indicates the data flow direction and dependency order.

[0087] At the same time, there is no closed directed path in the directed acyclic graph (i.e. starting from a node, advancing along the directed edges, and unable to return to the node). This feature ensures that the automatic driving algorithm flowchart does not have logical circular dependencies. This acyclicity allows the algorithm to execute each calculation operator in topological order (i.e. following the order of dependency relationship), ensuring that each operator can start after all its dependent pre-operators have completed their operations and provided data, thereby avoiding process blocking or logical contradictions, and providing a structural basis for stable and orderly execution of the algorithm.

[0088] In some embodiments, as Figure 3As shown, the photoelectricity collaborative graph compiler 13 comprises an identification module 131 and a decision module 132.

[0089] The identification module 131 is configured to identify a bottleneck module in an automatic driving algorithm which is computationally intensive and suitable for acceleration by optical computing, replace a node corresponding to the bottleneck module in an automatic driving algorithm flowchart with an optical operator, and fuse a sequence of operators in the automatic driving algorithm flowchart which are continuous and can be efficiently processed by the same optical computing unit.

[0090] Exemplarily, typical bottleneck modules can include convolutional layers in a backbone network of a large-scale convolutional neural network, fully connected layers, 3D sparse convolution after voxelization in a laser radar point cloud processing flow, and a feature extraction module based on a PointNet / PointPillar architecture, large-scale matrix multiplication involved in self-attention (Self-Attention) and cross-attention (Cross-Attention) computation in a Transformer structure in a multi-modal data fusion scenario, and a signal processing task based on fast Fourier transform (FFT). In addition, the identification module 131 fuses a sequence of operators in the automatic driving algorithm flowchart which are continuous and can be efficiently processed by the same optical computing unit. This operator fusion strategy can minimize unnecessary photoelectric conversion times and data transfer overheads between an OCU and a DPU, thereby improving overall execution efficiency.

[0091] The decision module 132 is configured to call the heterogeneous computing abstraction layer 11 and the photoelectric operator library 12 based on the automatic driving algorithm flowchart, generate a multi-dimensional cost matrix for possible mapping combinations in the automatic driving algorithm flowchart, formalize a hardware mapping problem of the automatic driving algorithm flowchart into a constrained multi-objective optimization problem based on a structure of the automatic driving algorithm flowchart, node meta-information, and each multi-dimensional cost matrix, solve the constrained multi-objective optimization problem by using a heuristic algorithm, obtain an optimal allocation scheme of hardware resources, and output a static scheduling strategy based on the optimal allocation scheme, wherein the mapping combination is a combination of associated mapping of an optical acceleration candidate node and an optical computing unit.

[0092] The decision module 132 can use a safety-aware multi-objective graph partitioning (SAM-GP) model to formalize the hardware mapping problem of the automatic driving algorithm flowchart into a constrained multi-objective optimization problem, and solve to find an optimal allocation scheme of an operator to a hardware unit (OCU / DPU). The scheme can mathematically preferentially guarantee the safety and real-time performance of an automatic driving system, while taking into account performance and power consumption.

[0093] Specifically, the input of the SAM-GP model is an autonomous driving algorithm flow graph G = (V, E), where V is a set of operator nodes and E is a set of data dependency edges. The device set in the SAM-GP model is represented as D = D_OCU∪D_DPU; the node metadata of the SAM-GP model is , which contains the task it belongs to TaskID(v_i) and the importance weight w_i; the path metadata of the SAM-GP model is a set of critical paths , and the deadline of each path Deadline(p); the cost function of the SAM-GP model (provided by HCAL, including the output of P²FM) includes execution delay, average power consumption, and fidelity loss, where Lat(v_i, d_j) represents the execution delay of node v_i on device d_j; Pwr(v_i, d_j) represents the average power consumption of node v_i on device d_j; AccLoss(v_i, d_j) represents the fidelity loss of node v_i on device d_j (if d_j is DPU, this value is 0; if it is OCU, it is provided by P²FM).

[0094] The decision variable of the SAM-GP model is x_ij∈{0,1}, where if node v_i is assigned to device d_j, it is 1, otherwise it is 0.

[0095] The objective function of the SAM-GP model MinimizeF(X) is:

[0096] F(X) = λ_crit×T_crit(X) + λ_total×T_total(X) + λ_pow×E_total(X) + λ_acc×A_loss_total(X)

[0097] Where the critical path delay T_crit(X) is:

[0098] max_{p∈P_crit}{Σ_{v_i∈p}Σ_{d_j∈D}Lat(v_i,d_j)×x_ij}. The total execution time T_total(X): the makespan of the autonomous driving algorithm flow graph (the total time required for all computing operators in the autonomous driving algorithm flow graph to execute, that is, the total time length of the entire process from the start of the first operator in the graph to the completion of the operation of the last operator). The total energy consumption E_total(X):

[0099] Σ_{v_i∈V}Σ_{d_j∈D}Pwr(v_i,d_j) x Lat(v_i,d_j) x x_ij; Weighted total fidelity loss A_loss_total(X): Σ_{v_i∈V}w_i x Σ_{d_j∈D}AccLoss(v_i,d_j) x x_ij. λ_crit, λ_total, λ_pow and λ_acc are configurable weight coefficients to balance different optimization objectives.

[0100] Subject to of SAM-GP model includes: allocation uniqueness, safety real-time hard constraint, task-level accuracy hard constraint and system resource constraint. Allocation uniqueness: ∀v_i∈V, Σ_{d_j∈D}x_ij=1; Safety real-time hard constraint:

[0101] ∀p∈P_crit, Σ_{v_i∈p}Σ_{d_j∈D}Lat(v_i,d_j) x x_ij≤Deadline(p)

[0102] Task-level accuracy hard constraint:

[0103] ∀k∈Tasks, Σ_{v_i|TaskID(v_i)=k}Σ_{d_j∈D_OCU}AccLoss(v_i,d_j) x x_ij≤Threshold_acc(k)

[0104] Wherein, the task to which the operator node belongs TaskID(v_i), Threshold_acc(k) represents the maximum fidelity loss threshold that the task k can tolerate.

[0105] System resource constraint: Σ_{v_i∈V}Σ_{d_j∈D}Pwr(v_i,d_j) x x_ij≤P_budget, wherein P_budget is the system total power consumption budget threshold.

[0106] Exemplarily, the simulated annealing algorithm can be used to solve the constraint multi-objective optimization problem.

[0107] The specific steps are as follows: first step, initialization, generate an initial allocation scheme X_0 that meets all the hard constraints (for example, all first assign to the highest precision DPU, and then try to move to OCU step by step). Calculate its objective function value F(X_0). Set the initial temperature T_start. Second step, iteration, each iteration includes the following a-d steps: a. Generate a new neighbor, generate a new candidate solution X_new from the current solution X_current through a "move" operation (for example, randomly select a node v_i, move it from device d_j to another device d_k); b. Check constraints, if X_new violates any hard constraints, directly reject. c. Evaluation, calculate ΔF=F(X_new)-F(X_current); d. Acceptance criterion, if ΔF<0 (new solution is better), accept the new solution X_current=X_new; if ΔF≥0, accept the worse solution with probability p=exp(-ΔF / T). Third step, cooling: according to a cooling schedule (such as T_new=α×T_old, α<1) to reduce the temperature T. Fourth step, termination: when the temperature is lower than a certain threshold or the number of iterations reaches the upper limit, the algorithm terminates and returns the best solution found. According to the final partitioning and scheduling results, the decision module 132 generates the corresponding executable instruction sequence for each OCU and DPU, and inserts optical-electric conversion, synchronization and data transfer instructions at data dependencies. Finally, an end-to-end, statically optimized static scheduling strategy is output. These instruction streams not only contain the computing tasks themselves, but also accurately insert necessary data synchronization instructions, optical-electric conversion control signals, and efficient buffer management instructions to ensure smooth cooperation between optical-electric computing units.

[0108] In one example, a typical autonomous driving task flow is taken as an example, which includes laser radar point cloud processing, visual image analysis, multi-modal feature fusion and local path planning. This example elaborates how the photoelectric cooperative graph compiler 13 applies the P²FM model to perform cost evaluation, and solves the SAM-GP model to complete intelligent partitioning and optimization of hardware perception. Specifically, it can be divided into the following steps 1 to 4.

[0109] Step 1: Compute graph reception and semantic enhancement. The compiler frontend receives a compute graph (DAG) that describes the entire task flow. The nodes in the graph not only contain operators (such as convolution, matrix multiplication), but are also assigned with meta-information related to autonomous driving, for example: Node_Voxelization (LiDAR point cloud voxelization), Node_SparseConv3D (3D sparse convolution, optical acceleration candidate: yes, safety criticality: high), Node_CNN_Backbone (visual backbone network, optical acceleration candidate: yes, delay budget: 10ms), Node_AttentionFusion (attention fusion, optical acceleration candidate: yes, precision requirement: high), Node_PathPlanner (path planning, DPU affinity: high).

[0110] Step 2: Hardware cost accurate evaluation based on P2FM model. The compiler enters the hardware-aware cost evaluation phase. For each node marked as an "optical acceleration candidate" (such as Node_SparseConv3D), the compiler iterates through all available optical computing units (such as OCU_A, OCU_B). For each <node, OCU> combination, the compiler performs the following operations: query HCAL, call P2FM model, obtain fidelity loss, and obtain other costs.

[0111] Query HCAL: Obtain the current environment parameter vector θ_env (e.g., estimated operating temperature, voltage) from the runtime system.

[0112] Call P2FM model: Input θ_env and the operator configuration φ_op of the node into the P2FM model solidified in HCAL.

[0113] Obtain fidelity loss: The P2FM model outputs the probability distribution D(μ_ε, σ²_ε) of the expected error of the operation. The compiler then calculates the KL divergence of this distribution from the ideal error-free case, obtaining a quantitative fidelity loss value AccLoss(Node_SparseConv3D, OCU_A). This value will be a key input to the SAM-GP model.

[0114] Obtain other costs: At the same time, the compiler obtains the estimated delay Lat() and power consumption Pwr() of the combination from HCAL.

[0115] Through this step, the compiler generates an accurate, multi-dimensional cost matrix for all possible node-device mappings in the graph, including dynamic precision loss.

[0116] Step 3: Construct and solve the constrained multi-objective optimization problem, i.e., the SAM-GP optimization problem. Next, the compiler formalizes the partitioning problem into a SAM-GP model for solution. Step 3 can include two sub-steps: problem construction and model solving.

[0117] Problem Construction: Based on the structure of the computation graph, node metadata (critical path, deadline), and the cost matrix generated in the previous step, the compiler automatically constructs the objective function and constraints of SAM-GP. Objective Function: MinimizeF(X) = λ_crit × T_crit(X) + ... + λ_acc × A_loss_total(X), where λ_crit is assigned the highest weight to reflect the highest priority for safety. Hard Constraints: Strict constraints are set, for example, the total latency of Path_AEB ≤ 25ms; (Path_AEB: the critical computation path for Automatic Emergency Braking (AEB) function); the total cumulative accuracy loss of Task_ObjectDetection ≤ 2% mAPdrop; (Task_ObjectDetection: object detection task); and the total system power consumption ≤ 150W.

[0118] Model Solving: The compiler uses a built-in heuristic solver based on simulated annealing to solve this constrained multi-objective optimization problem. The solver iteratively searches for the optimal allocation scheme while satisfying all hard constraints. (i.e., the final value of the variable x_ij), in order to minimize the objective function F(X).

[0119] Step 4: Instruction Stream Generation and Scheduling Plan Output. The solver returns the optimal allocation scheme. This clarifies which hardware unit each computation node should execute on. For example, the result might be: Node_SparseConv3D->OCU_A (low latency, excellent energy efficiency, acceptable accuracy loss), Node_CNN_Backbone->OCU_B (high parallelism, suitable for dense convolutions), Node_AttentionFusion->DPU_GPU (although OCU can execute it, this task has extremely high accuracy requirements, and the SAM-GP model considers DPU to be superior after trade-offs), Node_PathPlanner->DPU_CPU.

[0120] Based on this scheme, the compiler generates corresponding local instructions for each OCU and DPU, and inserts photoelectric conversion, synchronization, and data transfer instructions at data dependencies, ultimately outputting an end-to-end, statically optimized photoelectric co-execution plan.

[0121] In some embodiments, such as Figure 4 As shown, the system scheduler 14 includes a monitoring module 141 and a scheduling module 142.

[0122] The monitoring module 141 is configured to monitor the current hardware state and task state of the autonomous driving system, and to obtain sensor data processed by the optical computing unit and the digital processing unit, and determine the current environment state based on the sensor data.

[0123] The scheduling module 142 is configured to model the runtime scheduling decision problem of the autonomous driving system as a Markov decision process, input a state vector into a trained DQN network, solve the Markov decision process using the DQN network, obtain a target action with the highest Q value, and control the autonomous driving system to execute a scheduling strategy corresponding to the target action.

[0124] The monitoring module 141 can comprehensively perceive the current running state of the system, and realize hardware state monitoring. The hardware state monitoring is real-time collection of the temperature, optical power, signal-to-noise ratio, calibration drift of the OCU, and the core utilization rate, memory bandwidth, and temperature of the DPU. In an application, the monitoring module 141 can also realize application performance index monitoring, which is indirect monitoring of the output quality of key autonomous driving functions, such as the average precision mean (mAP) of the target detection task, the multi-target tracking precision (MOTA) and the multi-target tracking positioning precision (MOTP) of the multi-target tracking task, the positioning precision, the smoothness of the planned path, and the like, as indirect feedback of the health status of the OCU or the entire optoelectronic hybrid link.

[0125] In order to maximize the parallel processing capability of the optoelectronic hybrid computing, the runtime autonomous driving system adopts a multi-level parallel scheduling model, which specifically supports sensor data stream parallel processing and model / level pipeline optimization. The sensor data stream parallel processing refers to data streams from multiple sensors (such as a front camera group, a surround camera, a laser radar, and a millimeter wave radar array) that can be scheduled to different OCU channels or heterogeneous processing units formed by the combination of the OCU and the DPU for parallel processing, so as to realize 360° real-time perception and understanding of the environment around the vehicle. The model / level pipeline optimization refers to the construction of a fine-grained pipeline across the optoelectronic computing domain from the perception result (such as a detected obstacle list and a drivable area segmentation map) to the decision module 132 (such as behavior prediction and path planning) to the generation of the final control instruction. The system scheduler 14 is responsible for accurately managing the complex dependency relationship between the stages of the pipeline and efficiently synchronizing data in this process.

[0126] The system scheduler 14 models the runtime scheduling decision problem as a Markov Decision Process (MDP) based on a pre-built Reinforcement Learning Adaptive Scheduling Policy (RL-ASP) model, and trains an offline a deep reinforcement learning agent (i.e., a DQN network) to autonomously select the optimal global scheduling policy online according to the complex and dynamic system state and driving environment, to maximize the long-term reward directly related to the safety and efficiency of autonomous driving.

[0127] The state space (State, S) in the core element (MDP formalization) of the DQN network is a feature vector s_t that describes the current "world", expressed as s_t=[s_hw, s_task, s_env]. Among them, the hardware state s_hw is [OCU_temp_avg, OCU_snr_min, DPU_util_%, Mem_bw_%, Power_total] (all normalized values), OCU_temp_avg refers to the average temperature of the optical computing unit (OCU), OCU_snr_min refers to the minimum signal-to-noise ratio of the OCU, DPU_util_% refers to the utilization rate of the DPU, Mem_bw_% refers to the memory bandwidth utilization rate, and Power_total refers to the total power consumption of the system; the task state s_task is [Crit_path_latency_margin, Task_queue_depth] (critical path latency margin and deadline), Crit_path_latency_margin refers to the margin of critical path latency and deadline, and Task_queue_depth refers to the task queue depth.

[0128] The environment state s_env is [Scene_ID_onehot, TTC_min, Weather_ID, V2X_alert_level], which corresponds to one-hot encoding of scene classification, minimum time to collision, weather condition encoding, and V2X warning level, respectively.

[0129] The action space (Action, A) is the discrete action a_t that the runtime scheduler can perform, which is not to schedule a single operator, but to switch to one of a set of global scheduling policies generated by the compiler in advance, i.e.

[0130] a_t∈A={Policy_MaxSafety,Policy_Balanced,Policy_Eco,Trigger_Calibration}

[0131] Policy_MaxSafety is a scheduling policy compiled by SAM-GP when the weight of is extremely high, which guarantees the minimum latency and maximum accuracy at the cost of power consumption; Policy_Balanced is a balanced policy between latency, power consumption and accuracy; Policy_Eco is a policy compiled by SAM-GP when the weight of is extremely high, which maximizes the use of OCU to reduce power consumption; Trigger_Calibration is a special action, which is used to trigger the online calibration process actively when the system load is low.

[0132] The reward function (Reward, R) is a scalar signal R_t designed to guide the learning direction, and its expression is:

[0133] R_t=w_safe×f_safe(TTC_min)+w_perf×f_perf(latency)-w_cost×f_cost(power)-w_stab×f_stab(a_t,a_{t-1})

[0134] Involving the minimum collision time TTC_min, the end-to-end latency of the critical task latency, and the total power consumption power of the system. f_safe is the safety reward, which is a huge negative value (penalty) when TTC_min is lower than the safety threshold, and a small positive value otherwise, for example ; f_perf is the performance reward, which is inversely proportional to the end-to-end latency of the critical task, i.e. f_perf=1 / EndToEndLatency; f_cost is the cost penalty, which is proportional to the total power consumption of the system; f_stab is the stability penalty, which introduces a small negative value if the current action is different from the last time, i.e. , to avoid frequent policy jitter; w_safe, w_perf, w_cost and w_stab are the weights of the reward components.

[0135] The model training and deployment process is divided into an offline training phase and an online deployment phase. In the offline training phase, a simulation environment is first built using a high-fidelity autonomous driving simulation platform (such as CARLA, LGSVL), and an execution backend (including P2FM) that simulates the architecture of the present application is integrated. This environment can simulate various traffic scenarios, sensor data, and hardware behavior. Then, an agent is constructed using a deep Q network (DQN) as the reinforcement learning agent. The network input is a state vector s_t, and the output is the Q value Q(s, a) for each possible action a. Then, the training loop is entered, and the agent performs millions of time steps of "simulated driving" in the simulation environment. The ε-greedy strategy is used to explore the action space, and the experience tuples (s_t, a_t, R_t, s_{t+1}) are stored in the experience replay pool. Small batches of data are periodically sampled from the pool, and the Bellman equation is used to update the weights of the DQN network to minimize the temporal difference error. Techniques such as the target network (Target Network) are used to stabilize the training process.

[0136] In the online deployment phase, the trained DQN network weight file is first deployed to the vehicle-mounted computing platform for policy solidification. Then, real-time decision-making is performed, and the runtime scheduler executes a loop at a fixed frequency (e.g., 10 Hz) to collect data from various parts of the system, construct the current state vector s_t, input s_t to the DQN network to obtain the Q values of all actions, select the action with the highest Q value , and issue instructions to the underlying resource manager (e.g., the heterogeneous computing abstraction layer 11) to switch to the corresponding global scheduling policy.

[0137] In one example, the runtime scheduler continuously collects information from various parts of the system at a frequency of 10 Hz and constructs it into the state vector s_t required by the RL-ASP model. At a specific time t, assume that the vehicle is driving on an urban road at 40 km / h, there is a pedestrian 15 meters ahead, and the OCU_A temperature is slightly elevated. In this case, the hardware state s_hw is:

[0138] [OCU_temp_avg = 0.6, OCU_snr_min = 0.85, DPU_util = 0.5,...]

[0139] The task state s_task is: [Crit_path_latency_margin = 0.4,...] (there is a 40% margin for critical path latency); and the environment state s_env is:

[0140] [Scene_ID_onehot=[0,1,0,...],TTC_min=3.5s,Weather_ID=0,...](The scene is an urban road, and the minimum time-to-collision is 3.5 seconds). This vector s_t completely describes the entire context required for the current decision.

[0141] Note that OCU_temp_avg represents the average temperature of the optical computing unit (OCU); OCU_snr_min represents the minimum signal-to-noise ratio of the OCU; DPU_util_% represents the utilization rate of the DPU; Crit_path_latency_margin represents the margin between the critical path latency and the deadline; Scene_ID_onehot represents the one-hot encoding of the scene classification; TTC_min represents the minimum time-to-collision; and Weather_ID represents the weather condition code.

[0142] Instead of using fixed if-then rules, the scheduler inputs the state vector s_t into a deployed, trained DQN network. In the application, the autonomous driving process can include forward inference, action selection, policy execution, and adaptive response under scenario mutation, etc.

[0143] Forward inference: The DQN network performs a fast forward calculation, outputting the Q values for all possible actions under the current state s_t: Q(s_t, Policy_MaxSafety) = -5.2; Q(s_t, Policy_Balanced) = +10.5; Q(s_t, Policy_Eco) = +2.1; Q(s_t, Trigger_Calibration) = -15.0.

[0144] Policy_MaxSafety represents a scheduling scheme compiled by SAM-GP when the weight of λ_crit is extremely high, which guarantees the lowest latency and the highest accuracy at any cost. Policy_Balanced represents a scheme that balances between latency, power consumption, and accuracy. Policy_Eco represents a scheme compiled when the weight of λ_pow is extremely high, which maximizes the use of OCU to reduce power consumption. Trigger_Calibration represents a special action that actively triggers the online calibration process when the system load is low.

[0145] Action selection: The scheduler selects the action with the highest Q value, i.e.:

[0146] .

[0147] Policy Execution: The system immediately loads and switches to the "Balanced Mode" scheduling policy pre-generated by the compiler. This decision is the optimal choice learned by the agent based on maximizing long-term return (considering safety, efficiency, and power consumption), which believes that in the current scenario, there is no need to switch to the extreme safety mode, and the balanced mode is the best choice.

[0148] Adaptive Response to Scenario Mutation: Suppose that in the next second, a pedestrian suddenly crosses the road in front, causing the TTC to drop to 1.5 seconds. The new state s_{t+1}:TTC_min becomes 1.5s, and this key change is reflected in the new state vector s_{t+1}. New Decision: Input s_{t+1} into the DQN network. Since the network has learned in training that low TTC is strongly correlated with huge negative return R_safety, the output Q value will change dramatically: Q(s_{t+1},Policy_MaxSafety)=+50.8 (selecting this policy can maximize the avoidance of huge penalties); Q(s_{t+1},Policy_Balanced)=-80.3; Q(s_{t+1},Policy_Eco)=-150.7. Emergency Switch Execution: The scheduler immediately selects . This policy will force all AEB (Automatic Emergency Braking) related computing tasks (regardless of where they were originally assigned) to be preemptively scheduled on the hardware path with the lowest latency and highest accuracy (possibly the backup path of the high-performance DPU), regardless of power consumption, to ensure the fastest safety response.

[0149] In some embodiments, the monitoring module 141 is also configured to determine performance degradation prediction information of the optical processing unit and load peak prediction information of the digital processing unit based on historical running data and the change trend of the current state using a pre-trained prediction model.

[0150] The scheduling module 142 is also configured to calibrate the output result of the optical processing unit and adjust the scheduling policy corresponding to the target action according to the performance degradation prediction information and the load peak prediction information.

[0151] The monitoring module 141 acts as a "forward-looking sensor" of the system running state, not only tracking the current hardware and task state in real time, but also deeply mining the historical running data (covering the long-term performance record of the optical processing unit OCU at different temperatures and light powers, the load fluctuation curve of the digital processing unit DPU, the task execution time consumption statistics, etc.) and the dynamic change trend of the current state (for example, the recent temperature of the OCU continues to rise at a rate, the slow attenuation slope of the signal-to-noise ratio, the surge amplitude of the DPU load in a short period of time, etc.), and with the help of pre-trained prediction models (such as LSTM, Transformer, etc. Deep learning models, or hybrid models that integrate hardware physical characteristics), accurately capture the evolution law of performance and load, and generate two types of key prediction information: one is the performance degradation prediction information of the optical processing unit, including the calculation accuracy reduction amplitude (such as the percentage of target detection accuracy reduction) that the OCU may appear due to device aging, environmental interference (such as high temperature, increasing light signal noise) in a future specific period of time, the length of processing delay (such as the number of milliseconds added to the current delay) and the like; the second is the load peak prediction information of the digital processing unit, that is, the maximum load rate (such as the core utilization rate peak) that the DPU may reach at a future time, the duration of the load peak, and the potential task type (such as intensive perception computing task) that causes the peak.

[0152] The scheduling module 142, based on these forward-looking prediction results, actively takes double optimization measures: on the one hand, for the performance degradation prediction information of the OCU, a dynamic calibration mechanism is started. For example, when the OCU output error increases due to temperature rise, the scheduling module 142 will call the pre-stored calibration algorithm, adjust the modulation parameters of the light signal, introduce redundant light computing paths, or combine the auxiliary correction calculation of the DPU, to real-time correct the output results of the OCU, and ensure that its processing accuracy always meets the hard constraints of the autonomous driving function. On the other hand, combined with the load peak prediction information of the DPU, the scheduling strategy corresponding to the target action is adaptively adjusted. For example, if the DPU is predicted to have a load peak in the short term, the scheduling module 142 will migrate part of the non-critical tasks (such as low-priority environmental map updating tasks) to the OCU for processing to offload the load, or adjust the execution order of the tasks (preferentially schedule critical path tasks), or extend the scheduling period of non-urgent tasks, so as to avoid the surge of processing delay caused by the overload of the DPU, and ultimately ensure that the entire autonomous driving system always maintains an efficient and stable running state under the dynamic changing hardware and load conditions.

[0153] In some embodiments, the scheduling module 142 is also configured to preset a periodic calibration time window based on the hardware characteristics of the optical processing unit; when the periodic calibration time window is reached, the output results of the optical processing unit are calibrated according to the performance degradation prediction information.

[0154] The scheduling module 142 plays an active maintenance role in ensuring the stable operation of the optical processing unit (OCU). It first analyzes the hardware characteristics of the OCU, such as the temperature sensitivity of optoelectronic devices (high temperature can cause light signal attenuation), the aging rate of devices after long-term operation (such as the efficiency decline of optical modulators), and the cumulative impact of environmental light interference (such as the slow decrease of signal-to-noise ratio over time), etc. Based on these characteristics, a reasonable periodic calibration time window is preset. For example, for an OCU in high-temperature working conditions, the time window may be set to once every 30 minutes due to the faster performance degradation rate; while in a stable environment at room temperature, the time window can be extended to once every 2 hours, ensuring that the calibration period matches the pace of OCU actual performance degradation, neither overusing computing resources nor causing precision deterioration due to too long intervals.

[0155] When the system timer or state monitoring module 141 detects that the current time reaches the preset periodic calibration time window, the scheduling module 142 will call the performance degradation prediction information generated by the monitoring module 141. This information has estimated the possible performance offset of the current OCU (such as a 2% decrease in target recognition accuracy, a 5ms increase in feature extraction delay, etc.), and accordingly initiates a targeted calibration process. For example, if the prediction shows that the output error has increased due to light signal drift, the calibration will compensate for the signal offset by adjusting the optical phase adjuster inside the OCU and optimizing the photoelectric conversion gain parameters; if the prediction points to the calculation accuracy degradation caused by device aging, a pre-trained error compensation model will be loaded to correct the feature vector or calculation result output by the OCU, so that the output result after calibration meets the accuracy and delay hard constraints of the autonomous driving algorithm again, while avoiding interference with the ongoing key tasks (such as real-time obstacle detection) during the calibration process, achieving a balance between maintenance operations and system running efficiency.

[0156] In some embodiments, the scheduling module 142 is also used to detect the performance degradation parameters of the optical processing unit in multiple dimensions, determine the fault level of the optical processing unit according to the performance degradation parameters, determine the affected information of the autonomous driving function according to the fault level of the optical processing unit, and execute the preset fault response strategy according to the affected information.

[0157] The scheduling module 142 has comprehensive fault diagnosis and response capabilities in ensuring the reliable operation of the optical processing unit (OCU). It first accurately captures the performance degradation parameters of the OCU through a multi-dimensional detection mechanism. For example, through the built-in self-test (BIST) sequence of the OCU, it performs function-level consistency verification (such as comparing the deviation of the optical acceleration module output from the millimeter wave radar or visual perception results), or compares with the low-fidelity digital twin version running on the DPU when resources permit. The multi-dimensional detection mechanism can include multiple detection methods and multiple detection parameters. By comprehensively analyzing the results obtained through the multi-dimensional detection mechanism, a set of quantitative indicators reflecting the degree of OCU performance degradation can be formed.

[0158] On this basis, the scheduling module 142 determines the fault level according to the preset fault level division standard (which is formulated in combination with the design parameters of the OCU and the safety requirements of the autonomous driving function). For example, when only individual non-core parameters slightly deviate from the normal range (such as a signal-to-noise ratio decrease of less than 5% and a delay increase of less than 10ms), and do not affect the core function output, it is determined as "minor fault"; when multiple key parameters are significantly abnormal (such as a signal-to-noise ratio falling below 80% of the critical value and an accuracy error rate exceeding 3%), causing performance degradation of some non-safety-related functions (such as environment rendering), it is determined as "moderate fault"; and when the core parameters are seriously out of standard (such as a sudden drop in optical power causing calculation interruption and an accuracy error rate exceeding 10%), directly threatening safety functions such as obstacle detection and emergency braking, it is determined as "serious fault".

[0159] Subsequently, the scheduling module 142 further analyzes the affected information of the autonomous driving function according to the fault level, including the affected function modules (such as minor faults that may only affect the smoothness of lane line recognition, and serious faults that may cause laser radar point cloud processing failure), the degree of function affected (partial performance degradation / complete failure), potential safety risks (such as misjudging obstacle position / cannot generate brake instruction in time), etc. For example, under moderate fault, the affected information may be "20% decrease in long-distance target detection accuracy, but near-distance detection still meets safety requirements, with the risk of misjudgment in low-speed scenarios".

[0160] Finally, the scheduling module 142 executes the preset fault response strategy based on the affected information: for minor faults, start the dynamic compensation mechanism, such as calling the auxiliary computing resources of the DPU to correct the output results of the OCU, and record the degradation parameters for subsequent calibration. For moderate faults, execute the function degradation strategy, such as temporarily closing the affected non-core function (such as pedestrian attribute recognition), concentrating the OCU resources on the core safety function (such as obstacle distance calculation), and sending a warning message. For serious faults, immediately trigger the safety redundancy switching, such as quickly migrating all tasks undertaken by the OCU to the standby DPU cluster, and issuing a takeover prompt (such as "system failure, please take over the vehicle immediately") to the driver through the vehicle interactive system, to ensure that the automatic driving system can still minimize safety risks through reasonable response strategies in the case of OCU failure.

[0161] In some embodiments, the scheduling module 142 is also used to: in the case of a fault optical processing unit and a backup digital processing unit, the work task of the fault optical processing unit is transferred to the backup digital processing unit; in the case of a fault optical processing unit and no backup digital processing unit, the control strategy corresponding to the minimum safety risk is executed by the automatic driving system; in the case of an affected automatic driving function being a non-core function, the scheduling strategy currently executed by the automatic driving system is switched to a higher safety scheduling strategy.

[0162] When processing the fault of the optical processing unit (OCU), the scheduling module 142 will take different response measures according to the type of the affected automatic driving function and the hardware backup condition: when the affected is a core function (such as obstacle real-time detection, emergency braking instruction generation, vehicle lateral control, etc. The function directly related to driving safety), and the fault OCU has a backup digital processing unit (DPU), the scheduling module 142 will immediately start the task migration mechanism. Through the preset task mapping table, all core tasks currently undertaken by the fault OCU (such as three-dimensional reconstruction of laser radar point cloud, clutter suppression of millimeter wave radar signal, etc.) are quickly located, and the context data of the task (including intermediate calculation results, input parameter cache, task dependency relationship, etc.) is synchronized to the backup DPU through the hardware interface protocol, and the system resource allocation table is adjusted, and the original OCU The power quota is transferred to the backup DPU, to ensure that the task execution of the core function is not interrupted, for example, the obstacle distance calculation task originally processed by the OCU will be completely migrated to the backup DPU within 50ms, to ensure the real-time performance of safety decision.

[0163] When the affected is the core function, but the fault OCU does not exist the backup DPU, the scheduling module 142 will preferentially trigger the minimum safety risk control strategy, which is the core target of reducing the accident probability, including controlling the vehicle to gradually slow down to a safe speed (such as reducing to below 30km / h on urban roads and reducing to below 60km / h on highways), turning on the double flash warning light to prompt the surrounding vehicles, activating the lane keeping assistance to maintain the current lane driving, and at the same time, issuing a clear takeover request to the driver through the vehicle voice and display screen (such as "core system failure, please take over the vehicle immediately, the remaining takeover time is 10 seconds"), if the driver does not respond in time, further execute the emergency parking operation (such as slowly parking to the right emergency lane of the road), which can minimize the safety hazards caused by the core function failure under the condition of no backup.

[0164] When the affected is the non-core function (such as the in-vehicle ambient light linkage control, pedestrian age and gender recognition, entertainment information push, etc. which do not directly affect the driving safety), the scheduling module 142 will switch the currently executed scheduling strategy to a higher safety version. For example, the "energy efficiency priority strategy" originally adopted will be switched to the "safety redundancy strategy", which increases the computing resource quota of the core function (such as lane line recognition, following distance control) (such as increasing the DPU utilization rate upper limit from 70% to 90%), shortens the inspection period of the core task (such as shortening from 100ms once to 50ms once), and closes the hardware resources occupied by the non-core function (such as releasing the optical channel in the OCU used for the non-core function), so as to ensure that the stability and reliability of the core safety function are not affected when the non-core function is affected, and the overall safety baseline of the automatic driving system is maintained.

[0165] In one example, assuming that the monitoring system detects that the OCU_A responsible for laser radar (LiDAR) processing has its output signal-to-noise ratio continuously decreasing due to hardware aging and falling below the safety threshold, the fault information will be encoded into a specific dimension of the state vector s_t. When the agent receives the state containing this fault information, its trained strategy will tend to choose the action that can avoid the fault: if there is a redundant backup for this OCU, the Q value output by the deep Q network (DQN) will be highest for the action of switching to the backup path (this logic may be built into Policy_MaxSafety); if there is no direct backup but the system can still operate in degraded mode, the agent may choose a less energy-efficient but more robust strategy, while triggering the human-machine interface (HMI) to notify the driver of "limited perception ability, please drive carefully"; if the fault is serious enough to guarantee safety, the optimal strategy learned by the agent will be to trigger the minimum risk state (MRS), because any continue driving action will result in an unacceptable large negative reward.

[0166] Based on the same inventive concept, as Figure 5As shown, the present application also provides an automatic driving method, which is executed based on the adaptive optoelectronic hybrid computing system for end-to-end automatic driving according to any of the above solutions, and the automatic driving method comprises the following steps S501-S503.

[0167] S501: Obtain the current running state of the automatic driving system.

[0168] S502: Based on the running state, construct a state vector input into the trained DQN network, and obtain the target action with the highest Q value.

[0169] S503: Control the automatic driving system to execute the scheduling strategy corresponding to the target action, wherein different actions of the DQN network correspond to static scheduling strategies output by the optoelectronic collaborative graph compiler in different scenarios, and the running state includes hardware state, task state and environment state.

[0170] It should be noted that the automatic driving method provided by the embodiments of the present application is based on the same application concept as the adaptive optoelectronic hybrid computing system for end-to-end automatic driving provided by the embodiments of the present application, and therefore the specific implementation of this embodiment can be referred to the implementation of the aforementioned adaptive optoelectronic hybrid computing system for end-to-end automatic driving, and the repeated parts will not be described again.

[0171] In some embodiments, the electronic device provided by the embodiments of the present application comprises a processor and a memory; the memory has a computer program stored therein, wherein the computer program, when executed by the processor, implements the automatic driving method described above.

[0172] The present application also provides a computer readable medium having a computer program stored thereon, which, when executed by a processor, implements the automatic driving method described above. The computer readable medium can be included in the device / apparatus / system described in the above embodiments; or it can exist separately and not be assembled into the device / apparatus / system. The computer readable medium carries one or more programs, which, when executed, implement the method of the embodiments of the present application.

[0173] Those skilled in the art can understand that the features described in the various embodiments of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not specifically described in the present application. In particular, the features described in the various embodiments of the present application can be combined and / or combined in various ways without departing from the spirit and teachings of the present application. All these combinations and / or combinations fall within the scope of the present application. Therefore, the scope of the present application should not be limited to the above embodiments. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An adaptive optoelectronic hybrid computing system for end-to-end autonomous driving, characterized in that, include: Heterogeneous computing abstraction layer, optoelectronic operator library, optoelectronic co-graph compiler and system scheduler; The heterogeneous computing abstraction layer is used to shield the differences in optoelectronic hybrid hardware of the autonomous driving application software stack based on a pre-built unified device model, and to manage and schedule the underlying hardware resources. The photoelectric operator library provides a unified description framework for each operator based on a pre-built photoelectric operator model, and defines the key attributes of each operator; The optoelectronic co-graph compiler is used to receive the autonomous driving algorithm flowchart, call the heterogeneous computing abstraction layer and the optoelectronic operator library based on the autonomous driving algorithm flowchart, formalize the hardware mapping problem of the autonomous driving algorithm flowchart into a constrained multi-objective optimization problem, solve the constrained multi-objective optimization problem, obtain the optimal allocation scheme of hardware resources, and output a static scheduling strategy based on the optimal allocation scheme. The system scheduler is used to monitor the operating status of the autonomous driving system in real time, construct a state vector based on the operating status and input it into the pre-trained DQN network to obtain the target action with the highest Q value, and control the autonomous driving system to execute the scheduling strategy corresponding to the target action.

2. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 1, characterized in that, The unified device model includes a sub-model representing the capabilities of an optical computing unit, which comprises a computation support module, a format adaptation module, and a computation module. The computation support module is used to determine the specific computations that the optical computing unit is good at performing. The format adaptation module is used to evaluate the degree of adaptation of the optical computing unit to a specific input data format and the necessary preprocessing requirements; The computation module is used to model the output of the optical computing unit as a probability distribution centered on the ideal digital computation result, wherein the distribution parameters of the probability distribution are obtained based on the state parameters of the optical computing unit and a pre-trained mapping model.

3. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 1, characterized in that, The flowchart of the autonomous driving algorithm is a directed acyclic graph (DAG). The nodes of the DAG represent computational operators that perform specific operations, and the edges of the DAG represent data dependencies.

4. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 3, characterized in that, The photoelectric co-graph compiler includes: a recognition module and a decision module; The identification module is used to identify computationally intensive bottleneck modules in the autonomous driving algorithm that are suitable for acceleration by optical computing, replace the nodes corresponding to the bottleneck modules in the autonomous driving algorithm flowchart with optical operators, and fuse the continuous operator sequences in the autonomous driving algorithm flowchart that can be efficiently processed by the same optical computing unit. The decision module is used to call the heterogeneous computing abstraction layer and the photoelectric operator library based on the autonomous driving algorithm flowchart, generate a multi-dimensional cost matrix for possible mapping combinations in the autonomous driving algorithm flowchart, and formalize the hardware mapping problem of the autonomous driving algorithm flowchart into a constrained multi-objective optimization problem based on the structure of the autonomous driving algorithm flowchart, node metadata and each of the multi-dimensional cost matrices, and solve the constrained multi-objective optimization problem using a heuristic algorithm to obtain the optimal allocation scheme of hardware resources, and output a static scheduling strategy based on the optimal allocation scheme, wherein the mapping combination is a combination of optical acceleration candidate nodes and optical computing units that are associated and mapped.

5. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 1, characterized in that, The system scheduler includes: a monitoring module and a scheduling module; The monitoring module is used to monitor the current hardware status and task status of the autonomous driving system, and to acquire sensor data processed by the optical computing unit and digital processing unit, and to determine the current environmental status based on the sensor data. The scheduling module is used to model the scheduling decision problem of the autonomous driving system as a Markov decision process. The state vector is input into a pre-trained DQN network, and the DQN network is used to solve the Markov decision process to obtain the target action with the highest Q value. The module then controls the autonomous driving system to execute the scheduling strategy corresponding to the target action.

6. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 5, characterized in that, The monitoring module is also used to determine the performance degradation prediction information of the optical processing unit and the load peak prediction information of the digital processing unit based on historical operating data and the changing trend of the current state using a pre-trained prediction model. The scheduling module is also used to calibrate the output of the optical processing unit based on the performance degradation prediction information and the load peak prediction information, and to adjust the scheduling strategy corresponding to the target action.

7. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 5, characterized in that, The scheduling module is also used to preset a periodic calibration time window based on the hardware characteristics of the optical processing unit; when the periodic calibration time window is reached, the output result of the optical processing unit is calibrated according to the performance degradation prediction information.

8. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 6, characterized in that, The scheduling module is also used to detect and determine the performance degradation parameters of the light processing unit in multiple dimensions, determine the fault level of the light processing unit based on the performance degradation parameters, determine the affected information of the autonomous driving function based on the fault level of the light processing unit, and execute a preset fault response strategy based on the affected information.

9. The adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in claim 8, characterized in that, The scheduling module is also used for: If the affected autonomous driving function is the core function, and there is a backup digital processing unit for the faulty light processing unit, the work tasks of the faulty light processing unit will be transferred to the backup digital processing unit. When the affected autonomous driving function is the core function and there is no backup digital processing unit for the faulty light processing unit, the autonomous driving system is controlled to execute the control strategy with the lowest safety risk. If the affected autonomous driving function is a non-core function, the scheduling strategy currently executed by the autonomous driving system will be switched to a safer scheduling strategy.

10. An autonomous driving method, characterized in that, The autonomous driving method is executed based on an adaptive optoelectronic hybrid computing system for end-to-end autonomous driving as described in any one of claims 1 to 9, the autonomous driving method comprising: Obtain the current operating status of the autonomous driving system; Based on the aforementioned operating state, a state vector is constructed and input into the pre-trained DQN network to obtain the target action with the highest Q value. The autonomous driving system is controlled to execute the scheduling strategy corresponding to the target action. The different actions of the DQN network correspond to the static scheduling strategies output by the photoelectric cooperative graph compiler in different scenarios. The running state includes hardware state, task state and environment state.

Images