Message scheduling method, device and program product for multi-view heterogeneous sensor data

Abstract

This application provides a message scheduling method, device, and program product for multi-view heterogeneous sensor data. It relates to the field of communication control technology. The method includes: receiving multi-view heterogeneous sensor data from an autonomous driving device and determining the source category of the sensor data; constructing a factor graph model of the sensor data based on the source category; representing each target state, each sensor observation, and the relationship between each target state and each sensor observation as nodes and edges in the factor graph based on the factor graph model, thereby constructing a Bayesian inference framework; and processing the received multi-view heterogeneous sensor data based on a sequential processing strategy. In this solution, a sequential / parallel switching scheduling strategy based on dependencies, combined with the factor graph model and the Bayesian inference framework, solves the non-convex optimization and convergence dilemmas of existing technologies.

Description

Technical Field

[0001] This invention belongs to the field of communication control technology, specifically relating to a message scheduling method, device, and program product for multi-view heterogeneous sensor data. Background Technology

[0002] As the popularity of new energy vehicles gradually increases, autonomous driving has become the mainstream trend of future automobiles, and various autonomous vehicles are constantly emerging. The core of all autonomous vehicles is the central computing platform, which is responsible for the vehicle's fusion perception, localization, planning and other computing tasks. Therefore, the real-time requirements of the autonomous driving system are extremely high, thus requiring a high-real-time environment.

[0003] In existing fusion sensing systems using multi-view heterogeneous sensors (such as cameras, millimeter-wave radar, and lidar) for fusion tracking, the message passing mechanism based on probabilistic factor graph models is the core for achieving multi-target tracking. However, when data from multiple sensors are fused through factor graph models, a complex network structure containing internal and external loops is formed. This leads to the issue of nonconvex optimization: variational inference based on multi-sensor factor graphs has been proven to be a nonconvex optimization problem. This means that if the message scheduling strategy is inappropriate, traditional iterative message passing algorithms (such as loop sum-product algorithms) may not converge to the global optimum, or even get stuck in local optima, ultimately leading to inaccurate target state estimation.

[0004] Therefore, it is necessary to improve the message scheduling strategy for multi-sensor data fusion to overcome the non-convex optimization problem. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a message scheduling method, device and program product for multi-view heterogeneous sensor data, which can solve the non-convex optimization problem caused by the fusion perception of multiple sensors in the prior art of autonomous driving when the data of multiple sensors are fused through a factor graph model.

[0006] To achieve the above technical objectives, the technical solution adopted in this application is as follows:

[0007] Firstly, this application provides a message scheduling method for multi-view heterogeneous sensor data, applied to the autonomous driving control of unmanned vehicles, the method comprising:

[0008] Receive multi-view heterogeneous sensor data from autonomous driving equipment and determine the source category of the sensor data;

[0009] Construct a factor graph model of the sensor based on the source category of the sensor data;

[0010] Based on the aforementioned factor graph model, each target state, each sensor observation, and the relationship between each target state and each sensor observation are represented as nodes and edges in the factor graph, respectively, thereby constructing a Bayesian inference framework.

[0011] In response to the received multi-view heterogeneous sensor data, data processing is performed based on a sequential processing strategy, wherein the sequential processing strategy is configured to assign priority to each sensor according to the characteristics of the sensor (including the sensor's accuracy, reliability, and data update rate), and high-priority sensor data is processed first.

[0012] Based on the Bayesian inference framework, the dependency relationship between the next sensor data and the current sensor data is determined. If the dependency relationship is characterized as independent, the processing method for the next sensor data is switched from sequential processing to parallel processing. The multi-view heterogeneous sensors include, but are not limited to, cameras or vision sensors, magnetic sensors, lidar, millimeter-wave radar, ultrasonic radar, inertial measurement units (including accelerometers, gyroscopes, magnetometers, and pressure sensors), odometers, and altimeters. These sensors are installed in different locations on the autonomous driving equipment. For example, for autonomous vehicles: multiple cameras and radars are installed at the front, rear, and four corners to cover the 360-degree environment around the vehicle, eliminating blind spots; lidar on the roof rotates 360 degrees for scanning; cameras behind the windshield observe road conditions and traffic signs ahead; cameras at the four corners observe surrounding blind spots; and front and rear radars detect nearby obstacles.

[0013] In conjunction with the first aspect, in some alternative implementations, the method further includes:

[0014] When constructing the Bayesian inference framework, a GNN graph neural network is used to optimize the correlation between each target state and each sensor observation.

[0015] The relationships are quantified and prioritized according to the quantified values.

[0016] In conjunction with the first aspect, in some optional implementations, the step of receiving multi-view heterogeneous sensor data from the autonomous driving device and determining the source category of the sensor data includes:

[0017] The first receiving component receives multi-view heterogeneous sensor data from the autonomous driving device.

[0018] In response to the operation of the first receiving component, the transmission path occupied by the multi-view heterogeneous sensor data is monitored;

[0019] The communication ID of the sensor is determined based on the transmission path, and the source category of the sensor data is determined based on the communication ID.

[0020] In conjunction with the first aspect, in some optional implementations, the step of constructing a factor graph model of the sensor based on the source category of the sensor data includes:

[0021] Based on the source category of the sensor data, a corresponding factor graph framework is matched, and the factor graph framework is expanded through variable nodes and factor nodes;

[0022] Based on a preset cost function, the variable nodes and factor nodes of different sensors are used to complete the recursion and update of the state, and the factor graph model is constructed.

[0023] In conjunction with the first aspect, in some alternative implementations, the method further includes:

[0024] Within the target time period, acquire the sensor observation values ​​of the previous moment and the current moment, and compare the changes in the sensor observation values ​​between the two moments;

[0025] If the change value of a sensor exceeds a preset change threshold, the priority of the corresponding sensor will be increased, and the level of priority increase will be executed according to a preset control strategy.

[0026] In conjunction with the first aspect, in some alternative implementations, the method further includes:

[0027] If new sensor data is available, new factor nodes and variable nodes are added incrementally to the factor graph framework to build a new factor graph model.

[0028] In conjunction with the first aspect, in some optional implementations, the multi-view heterogeneous sensor includes at least a visual sensor, a lidar, a millimeter-wave radar, and an inertial navigation measurement unit; the Bayesian inference framework is centered on the visual sensor, with the other sensors assisting navigation; the new factor graph model establishment step includes:

[0029] Initialization of the factor graph model involves setting the variable nodes and prior factor nodes defined in the factor graph at the initial moment.

[0030] The introduction of variable nodes and factor nodes in a factor graph; when a new variable node is introduced into the factor graph. At that time, factor nodes The following cost function is used for evaluation:

[0031] ;

[0032] in, Let k represent the navigation state variables of the autonomous vehicle at time k, which indicate its position, velocity, and attitude. This represents the calibration parameters of the vision sensor at time k, used to correct the data output by the vision sensor. This represents the data output by the vision sensor at time k. The function matrix for system state transitions, factor nodes Connecting different moments and Variable nodes of navigation state and , Used to evaluate the error of vision sensors This is an estimate of the navigation state variables, obtained by using new visual sensor data. Obtain the predicted values ​​of the navigation state variables.

[0033] Secondly, this application also provides a computer device, the computer device including a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the computer device to perform the method described thereon.

[0034] Thirdly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method described.

[0035] The invention employing the above technical solution has the following advantages:

[0036] In the technical solution provided in this application, by constructing a factor graph model and a Bayesian inference framework, and combining a message scheduling strategy that switches between sequential and parallel processing, the problem that traditional iterative message passing algorithms are prone to getting trapped in local optima and failing to converge when fusing multi-sensor data in the prior art is effectively solved, thus ensuring the accuracy of target state estimation.

[0037] Priority is assigned based on sensor characteristics, with high-priority data being processed first. At the same time, sequential / parallel processing methods are dynamically switched according to data dependencies to reduce data processing latency while ensuring processing accuracy and meeting the high real-time requirements of autonomous driving.

[0038] By optimizing the correlation between target state and sensor observations through graph neural networks, and combining the complementary advantages of multi-view heterogeneous sensors, the accuracy and stability of target state estimation are improved, and errors caused by environmental interference are reduced.

[0039] It supports incremental updates of factor graph models, which can flexibly adapt to new sensor data. At the same time, by dynamically adjusting sensor priorities, it can adapt to the complex and ever-changing driving scenarios of autonomous driving, thereby improving the versatility and robustness of the technical solution. Attached Figure Description

[0040] This application can be further illustrated by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate some embodiments of this application and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any inventive effort.

[0041] Figure 1 A schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0042] Figure 2 A flowchart illustrating the message scheduling method for multi-view heterogeneous sensor data provided in this application embodiment. Detailed Implementation

[0043] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts are referred to by the same reference numerals in the drawings or description. Implementations not shown or described in the drawings are forms known to those skilled in the art. In the description of this application, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0044] Please refer to Figure 1 This application provides a computer device, including a processor and a memory coupled to the processor. The memory stores a computer program that, when executed by the processor, performs corresponding steps in the following message scheduling method for multi-view heterogeneous sensor data.

[0045] In this embodiment, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor. For example, it can be a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0046] The memory can be, but is not limited to, random access memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, etc. In this embodiment, the memory can be used to store factor graphs, Bayesian inference, graph neural networks, etc. Of course, the memory can also be used to store computer programs, which the processor executes after receiving execution instructions.

[0047] Understandable, Figure 1 The computer device structure shown is only a schematic diagram and may include more advanced components. Figure 1 More components are shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0048] Optional, refer to Figure 2 This application also provides a message scheduling method for multi-view heterogeneous sensor data, which can be applied to the processor in the aforementioned computer device, and the processor executes or implements the steps of the method. This multi-view heterogeneous sensor data message scheduling method is applied to the autonomous driving control of unmanned vehicles, including but not limited to new energy vehicles, unmanned vehicles, drones, and flying cars. This embodiment takes an unmanned vehicle as an example. The device is equipped with multi-view heterogeneous sensors, including at least two of the following: a camera, millimeter-wave radar, a vision sensor, a lidar, and an inertial conduction unit. These sensors are used to collect environmental data and motion state data of the unmanned vehicle's surroundings, achieving fusion perception, positioning, and navigation control. For example, the millimeter-wave radar is used to collect target distance and speed information, with an update rate of 10Hz-20Hz and an accuracy error ≤0.5m; the vision sensor is used to collect target appearance features and contour information, with an update rate of 5Hz-10Hz and a resolution of 1080P; the lidar is used to collect three-dimensional environmental point cloud data, with an update rate of 10Hz and a ranging range of 0.5m-150m; and the camera is used to assist in collecting visual images and supplement target detail information, with an update rate of 8Hz-12Hz. The method includes steps 210 to 250; specifically,

[0049] Step 210: Receive multi-view heterogeneous sensor data from the autonomous driving device and determine the source category of the sensor data;

[0050] Step 220: Construct a factor graph model of the sensor based on the source category of the sensor data;

[0051] Step 230: Based on the factor graph model, each target state, each sensor observation, and the relationship between each target state and each sensor observation are represented as nodes and edges in the factor graph, respectively, thereby constructing a Bayesian inference framework;

[0052] Step 240: In response to the received multi-view heterogeneous sensor data, perform data processing based on a sequential processing strategy, wherein the sequential processing strategy is configured to assign priority to each sensor according to the characteristics of the sensor, and process the data of high-priority sensors first.

[0053] Step 250: Based on the Bayesian inference framework, determine the dependency relationship between the next sensor data and the current sensor data. If the dependency relationship is characterized as non-dependency, switch the processing method of the next sensor data from sequential processing to parallel processing.

[0054] In the above implementation, by constructing a factor graph model and a Bayesian inference framework, combined with a message scheduling strategy that switches between sequential and parallel processing, the problem that traditional iterative message passing algorithms are prone to getting trapped in local optima and failing to converge when fusing multi-sensor data is effectively solved, ensuring the accuracy of target state estimation. Priority is assigned based on sensor characteristics, with high-priority data being processed first. At the same time, the sequential / parallel processing mode is dynamically switched according to data dependencies, reducing data processing latency while ensuring processing accuracy, and meeting the high real-time requirements of autonomous driving.

[0055] The following section will elaborate on each step of the message scheduling method for multi-view heterogeneous sensor data:

[0056] In step 210, the step of receiving multi-view heterogeneous sensor data from the autonomous driving device and determining the source category of the sensor data includes:

[0057] The first receiving component receives multi-view heterogeneous sensor data from the autonomous driving device.

[0058] In response to the operation of the first receiving component, the transmission path occupied by the multi-view heterogeneous sensor data is monitored;

[0059] The communication ID of the sensor is determined based on the transmission path, and the source category of the sensor data is determined based on the communication ID.

[0060] The first receiving component is a data receiving module, used to receive all sensor data sent by the multi-view heterogeneous sensor module. It supports multi-channel parallel reception, and the data transmission protocol adopts CAN bus or Ethernet to ensure the stability and real-time performance of data transmission, with a reception delay of ≤10ms. A channel monitoring module can be set up and integrated into the data receiving module to monitor the transmission path of each sensor data, identify the communication ID of the transmission path, match the corresponding sensor type through the communication ID, and thus determine the source category of the sensor data.

[0061] For example, the CAN bus receives data from all sensors in real time, including distance and speed data from millimeter-wave radar, appearance feature data from vision sensors, point cloud data from lidar, and image data from cameras. Simultaneously, a pre-configured channel monitoring module responds to the data reception, monitoring the transmission paths occupied by each sensor's data in real time. Based on the communication protocol and identification information of each transmission path, it determines the corresponding communication ID for each path; for example, the communication ID for millimeter-wave radar is 001, for vision sensors it is 002, for lidar it is 003, and for cameras it is 004. According to the pre-defined correspondence between communication IDs and sensor types, the source category of each sensor's data is determined: communication ID 001 corresponds to millimeter-wave radar data, ID 002 to vision sensor data, ID 003 to lidar data, and ID 004 to camera data.

[0062] If the channel monitoring module detects that the communication delay of a certain transmission path exceeds 50ms or the packet loss rate exceeds 5%, it will issue an early warning signal. The scheduling control module will suspend the sensor data processing of that path and prioritize the processing of sensor data with good communication quality. Once the communication quality is restored, the data processing of that path will be resumed. The channel monitoring module mentioned here is existing technology and is not specifically limited here.

[0063] Step 220, the step of constructing a factor graph model of the sensor based on the source category of the sensor data, includes:

[0064] Based on the source category of the sensor data, a corresponding factor graph framework is matched, and the factor graph framework is expanded through variable nodes and factor nodes. For example, for a combination of millimeter-wave radar, visual sensors, and lidar, a preset multi-sensor fusion factor graph framework is matched, which includes basic variable nodes (target position, velocity, attitude) and factor nodes (sensor observation factors). Subsequently, the factor graph framework is expanded through variable nodes and factor nodes: the variable nodes include the autonomous vehicle's position (x, y, z) and velocity (v). x ,v y ,v z The variables include the attitude, and the observation variables of each sensor (distance r and velocity v of the millimeter-wave radar, feature vector f of the visual sensor, and point cloud coordinates P of the lidar). The factor nodes include millimeter-wave radar observation factor f1, visual sensor observation factor f2, and lidar observation factor f3, which are used to associate the variable nodes with the observation variables of each sensor.

[0065] Based on a preset cost function, the variable nodes and factor nodes of different sensors are used to complete the recursion and update of the state, and the factor graph model is constructed.

[0066] In this embodiment, the reference cost function is:

[0067] (1);

[0068] in, Let k be the navigation state variables (position, velocity, attitude). Let k be the sensor observation value at time k. The function matrix represents the system state transition. Calibrate the sensor parameters (used to evaluate the error of the vision sensor). λ is the estimated value of the navigation state variable, and λ is the regularization coefficient (with a value of 0.1-0.3) used to balance observation error and estimation error.

[0069] In this embodiment, the cost function (1) is specifically designed for the aforementioned visual sensor. Then, when new variable nodes are introduced into the factor graph... At that time, factor nodes The following cost function is used for evaluation:

[0070] (2);

[0071] in, Let k represent the navigation state variables of the autonomous vehicle at time k, which indicate its position, velocity, and attitude. This represents the calibration parameters of the vision sensor at time k, used to correct the data output by the vision sensor. This represents the data output by the vision sensor at time k. The function matrix for system state transitions, factor nodes Connecting different moments and Variable nodes of navigation state and , Used to evaluate the error of vision sensors This is an estimate of the navigation state variables, obtained by using new visual sensor data. Obtain the predicted values ​​of the navigation state variables.

[0072] In step S230, a Bayesian inference framework is constructed based on the factor graph model; specifically, it can be implemented as follows:

[0073] Each target state (i.e., variable node) ), each sensor observation (i.e., the observed variable) The relationship between each target state and each sensor observation is represented as nodes and edges in the factor graph, where the variable node is the target state node, the observed variable is the observed node, and the relationship is the edge connecting the target state node and the observed node.

[0074] In some scenarios of this embodiment, when constructing the Bayesian inference framework, a GNN graph neural network is used to optimize the correlation between each target state and each sensor observation value.

[0075] The relationships are quantified and prioritized according to the quantified values.

[0076] For example, a graph convolutional neural network (GNN) is used as the graph neural network model. The node features (target state parameters, observation parameters) and edge features (association strength) of the input factor graph are used. The features are extracted through two layers of graph convolutional layers, and the optimized association weights are output. The association weights are quantized (quantization range is 0-1), and the priority is sorted according to the quantized values ​​from large to small. The higher the association weight, the greater the impact of the corresponding sensor observation on the target state estimation, and the higher the priority.

[0077] In step S240, multi-view heterogeneous sensor data is processed based on a sequential processing strategy. The sequential processing strategy is configured to assign priority to each sensor according to its characteristics (including sensor accuracy, reliability, and data update rate), with high-priority sensor data being processed first. In this embodiment, the sensor priorities from high to low are as follows: millimeter-wave radar (high accuracy, high update rate, priority 0.9) > lidar (high accuracy, strong reliability, priority 0.8) > visual sensor (rich appearance features, priority 0.7) > camera (auxiliary observation, priority 0.6).

[0078] The sequential processing strategy is implemented as follows: first, the precise distance and velocity information provided by the millimeter-wave radar is processed to initially lock the target's position and motion state; then, the appearance features provided by the visual sensor are fused to refine the target state; next, the point cloud data from the lidar is fused to supplement the target's three-dimensional position information; finally, the image data from the camera is fused to further refine the target details. By utilizing the complementary advantages of different sensors in a sequential manner, the accuracy of target state estimation is improved.

[0079] In step S250, based on the Bayesian inference framework, the dependency relationship between the next sensor data and the current sensor data is determined. The dependency relationship is calculated by using Bayesian posterior probability to determine the correlation degree between the two data points. The correlation degree ranges from 0 to 1; the higher the correlation degree, the stronger the dependency. For example, a first threshold of 0.3 can be set to characterize the correlation degree. If the correlation degree is less than 0.3, it is considered as no dependency.

[0080] If the calculated dependency is lower than the first threshold (<0.3), it indicates that the correlation between the next sensor data and the current sensor data is low and the mutual influence is small. The scheduling control module will switch the processing method of the next sensor data from sequential processing to parallel processing. That is, while the current sensor data is being processed, another processing thread is started to process the next sensor data to reduce data processing delay. If the dependency is higher than or equal to the first threshold (≥0.3), the sequential processing strategy will continue to be used to ensure the continuity and accuracy of data processing.

[0081] In this embodiment, the method further includes:

[0082] Within the target time period, acquire the sensor observation values ​​of the previous moment and the current moment, and compare the changes in the sensor observation values ​​between the two moments;

[0083] If the change value of a sensor exceeds a preset change threshold, the priority of the corresponding sensor will be increased, and the level of priority increase will be executed according to a preset control strategy.

[0084] In some optional implementations of this embodiment, the target time period can be set to 100ms, and the preset change threshold can be set to 5%; of course, it can be adjusted according to the sensor type, such as a change threshold of 3% for millimeter-wave radar, a change threshold of 8% for visual sensors, and so on.

[0085] Obtain the sensor observation values ​​from the previous and current times, and calculate the change in the sensor observation values ​​between the two times (change = |current observation value - previous observation value| / previous observation value × 100%).

[0086] If the change value of a certain sensor exceeds a preset change threshold, it indicates that the target state observed by that sensor has changed significantly, and the priority of the corresponding sensor will be increased by one level (e.g., the priority of a vision sensor is increased from 0.7 to 0.8). The priority increase level is executed according to a preset control strategy: the priority after the increase will not exceed the highest priority (0.9), the increase duration is 500ms, and after 500ms, the original priority will be restored to avoid frequent priority fluctuations;

[0087] In this embodiment, the method further includes:

[0088] If new sensor data is available, new factor nodes and variable nodes are added incrementally to the factor graph framework to build a new factor graph model.

[0089] The factor graph model in this embodiment adds new factor nodes and variable nodes incrementally to the original factor graph framework to establish a new factor graph model without having to rebuild the entire factor graph model, thus reducing the amount of computation and improving real-time performance.

[0090] In this embodiment, the multi-view heterogeneous sensor includes a visual sensor, millimeter-wave radar, lidar, and a camera. The Bayesian inference framework uses the visual sensor as its core, with the other sensors (millimeter-wave radar, lidar, and camera) assisting in navigation. The new factor graph model establishment step includes:

[0091] The factor graph model is initialized at the initial time (t=0) by defining the variable nodes and prior factor nodes in the factor graph; variable nodes... The initial values ​​are the initial position (such as GPS coordinates), initial velocity (0 m / s), and initial attitude (0°) of the autonomous driving device (i.e., the car); the initial weight of the prior factor node is 0.5, which is used to represent the confidence level of the initial state.

[0092] The introduction of variable nodes and factor nodes in a factor graph; when a new variable node is introduced into the factor graph. At that time, factor nodes The cost function described above, i.e., equation (2), is used for evaluation.

[0093] The multi-view heterogeneous sensor data message scheduling method in this embodiment can effectively reduce data processing latency, improve target state estimation accuracy and algorithm convergence rate, solve the non-convex optimization and convergence dilemma of existing technologies, and meet the real-time and accuracy requirements of autonomous driving.

[0094] In one embodiment, such as Figure 1 As shown, a computer device includes a memory and a processor. The memory stores a computer program that, when executed by the processor, enables the processor to perform the message scheduling method for multi-view heterogeneous sensor data.

[0095] In one embodiment, this embodiment provides a computer program product, including a computer program; the computer program is stored in a USB flash drive (computer-readable storage medium), and when the computer program is executed by a processor (such as a PLC processor or an industrial computer processor), it implements the above-mentioned control method for the dust removal system of the cigarette production equipment.

[0096] In fact, in some scenarios of this embodiment, the computer program includes the following functional modules:

[0097] Multi-view heterogeneous sensor module: including millimeter-wave radar, vision sensor, lidar, and camera. The millimeter-wave radar is used to collect target distance and velocity information, with an update rate of 10Hz-20Hz and an accuracy error ≤0.5m; the vision sensor is used to collect target appearance features and contour information, with an update rate of 5Hz-10Hz and a resolution of 1080P; the lidar is used to collect 3D environmental point cloud data, with an update rate of 10Hz and a ranging range of 0.5m-150m; the camera is used to assist in acquiring visual images and supplement target detail information, with an update rate of 8Hz-12Hz.

[0098] Data receiving module (i.e., the first receiving component): used to receive all sensor data sent by the multi-view heterogeneous sensor module, supports multi-channel parallel reception, and uses CAN bus or Ethernet as the data transmission protocol to ensure the stability and real-time performance of data transmission, with a receiving delay of ≤10ms.

[0099] Channel monitoring module: Integrated into the data receiving module, it is used to monitor the transmission path of each sensor data, identify the communication ID of the transmission path, match the corresponding sensor type through the communication ID, and thus determine the source category of the sensor data; at the same time, it monitors the communication quality of the transmission path, and issues an early warning signal when the communication delay exceeds 50ms or the packet loss rate exceeds 5%.

[0100] Factor graph modeling module: It is used to match the corresponding factor graph framework according to the source category of sensor data, expand the factor graph framework through variable nodes and factor nodes, complete state recursion and update based on a preset cost function, and build a factor graph model; it supports incremental updates, when new sensor data is input, add corresponding factor nodes and variable nodes, and update the factor graph model.

[0101] Bayesian Inference Module: Based on the factor graph model, the target state, sensor observations and their relationship are represented as nodes and edges of the factor graph, and a Bayesian inference framework is constructed. The relationship is optimized using a graph neural network, the relationship is quantified and sorted, and the dependency of sensor data is determined based on the inference results.

[0102] Scheduling and control module: Processes sensor data based on a sequential processing strategy, assigns priorities according to sensor characteristics (including accuracy, reliability, and update rate), and processes high-priority data first; at the same time, it determines whether to switch processing modes (sequential / parallel) based on the dependency relationship output by the Bayesian inference module; dynamically adjusts sensor priorities, and increases the priority of the corresponding sensor when the change in sensor observation exceeds a preset threshold.

[0103] The computer program product can be a storage medium such as a USB flash drive, optical disc, or portable hard drive, or it can be a program package downloaded from the network, which facilitates deployment and application on different autonomous driving devices.

[0104] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the electronic device described above can be referred to the corresponding steps in the aforementioned method, and will not be elaborated further here.

[0105] Based on the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause an electronic device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various implementation scenarios of this application.

[0106] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0107] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A message scheduling method for multi-view heterogeneous sensor data, characterized in that, The method for autonomous driving control of unmanned vehicles includes: Receive multi-view heterogeneous sensor data from autonomous driving equipment and determine the source category of the sensor data; Construct a factor graph model of the sensor based on the source category of the sensor data; Based on the aforementioned factor graph model, each target state, each sensor observation, and the relationship between each target state and each sensor observation are represented as nodes and edges in the factor graph, respectively, thereby constructing a Bayesian inference framework. In response to the received multi-view heterogeneous sensor data, data processing is performed based on a sequential processing strategy, wherein the sequential processing strategy is configured to assign priority to each sensor according to the characteristics of the sensor, and high-priority sensor data is processed first. Based on the Bayesian inference framework, the dependency relationship between the next sensor data and the current sensor data is determined. If the dependency relationship is characterized as non-dependency, the processing method of the next sensor data is switched from sequential processing to parallel processing.

2. The message scheduling method for multi-view heterogeneous sensor data according to claim 1, characterized in that, The method further includes: When constructing the Bayesian inference framework, a GNN graph neural network is used to optimize the correlation between each target state and each sensor observation. The relationships are quantified and prioritized according to the quantified values.

3. The message scheduling method for multi-view heterogeneous sensor data according to claim 1, characterized in that, The step of receiving multi-view heterogeneous sensor data from an autonomous driving device and determining the source category of the sensor data includes: The first receiving component receives multi-view heterogeneous sensor data from the autonomous driving device. In response to the operation of the first receiving component, the transmission path occupied by the multi-view heterogeneous sensor data is monitored; The communication ID of the sensor is determined based on the transmission path, and the source category of the sensor data is determined based on the communication ID.

4. The message scheduling method for multi-view heterogeneous sensor data according to claim 1, characterized in that, The step of constructing a factor graph model of the sensor based on the source category of the sensor data includes: Based on the source category of the sensor data, a corresponding factor graph framework is matched, and the factor graph framework is expanded through variable nodes and factor nodes; Based on a preset cost function, the variable nodes and factor nodes of different sensors are used to complete the recursion and update of the state, and the factor graph model is constructed.

5. The message scheduling method for multi-view heterogeneous sensor data according to claim 1, characterized in that, The method further includes: Within the target time period, acquire the sensor observation values ​​of the previous moment and the current moment, and compare the changes in the sensor observation values ​​between the two moments; If the change value of a sensor exceeds a preset change threshold, the priority of the corresponding sensor will be increased, and the level of priority increase will be executed according to a preset control strategy.

6. The message scheduling method for multi-view heterogeneous sensor data according to claim 1, characterized in that, The method further includes: If new sensor data is available, new factor nodes and variable nodes are added incrementally to the factor graph framework to build a new factor graph model.

7. The message scheduling method for multi-view heterogeneous sensor data according to claim 6, characterized in that, The multi-view heterogeneous sensor includes at least a visual sensor, a lidar, a millimeter-wave radar, and an inertial navigation measurement unit. The Bayesian inference framework is centered on the visual sensor, with the other sensors assisting in navigation. The steps for building the new factor graph model include: Initialization of the factor graph model involves setting the variable nodes and prior factor nodes defined in the factor graph at the initial moment. The introduction of variable nodes and factor nodes in a factor graph; when a new variable node is introduced into the factor graph. At that time, factor nodes The following cost function is used for evaluation: ； in, Let k represent the navigation state variables of the autonomous vehicle at time k, which indicate its position, velocity, and attitude. This represents the calibration parameters of the vision sensor at time k, used to correct the data output by the vision sensor. This represents the data output by the vision sensor at time k. The function matrix for system state transitions, factor nodes Connecting different moments and Variable nodes of navigation state and , Used to evaluate the error of vision sensors This is an estimate of the navigation state variables, obtained by using new visual sensor data. Obtain the predicted values ​​of the navigation state variables.

8. A computer device, characterized in that, The computer device includes a processor and a memory; the processor is configured to execute instructions stored in the memory to cause the computer device to perform the method as described in any one of claims 1 to 7.

9. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images