An adaptive multi-source sensor fusion recognition method based on directed graph convolution

Abstract

The application relates to a kind of adaptive multi-source sensor fusion identification methods based on directed graph convolution, comprising the following steps: obtaining the information collected by different types of sensors of vehicle;Using each type of sensor corresponding single sensor identification model respectively to identify the information collected by different types of sensors, to obtain single sensor identification result;Based on single sensor identification result, dynamic directed graph is constructed;With dynamic directed graph as the input of directed graph fusion model, multi-source sensor information fusion and identification are carried out using directed graph fusion model, and the identification result is output.Compared with the prior art, the application deeply fuses multi-source sensor information by considering the characteristics and differentiated attention features of different types of sensors, has the advantages of improving the identification accuracy and being able to adapt to extreme driving environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving vehicles, and in particular to an adaptive multi-source sensor fusion and recognition method based on directed graph convolution. Background Technology

[0002] Intelligent vehicles have garnered attention for their ability to reduce the frequency of accidents. However, because intelligent vehicles are required to operate in open and complex driving environments, ensuring their safety in these challenging and ever-changing conditions has become a challenge. Improving the accuracy of perception and recognition in various extreme environments is one of the primary means of addressing this challenge.

[0003] Existing perception algorithms are mainly divided into two categories: single-sensor recognition and multi-source sensor recognition methods. Due to the differentiated characteristics of different sensor types, multi-source sensor recognition methods are significantly more adaptable than single-sensor recognition methods and have become the mainstream recognition method for intelligent vehicles. For example, cameras have extremely high object recognition accuracy in good weather conditions, but their recognition ability significantly degrades in low-visibility environments such as heavy fog or heavy snow. In contrast, millimeter-wave radar maintains a robust, generally acceptable level of performance across various weather conditions, but its ability to identify obstacle attributes is limited.

[0004] Multi-source sensor recognition methods can be further categorized into front-end fusion and back-end fusion based on the characteristics of the fused information. Front-end fusion uses a unified network to process raw information from different sensors (images from cameras, point clouds from LiDAR). While this method provides a means to deeply fuse information from various sensors, the different sensor characteristics are not finely processed, leading to conflicts in the information required for perception and reducing recognition robustness. Back-end fusion methods select the final recognition results from different sensors. Although they effectively consider the different sensor characteristics, the lack of direct connections between different sensor information means that recognition accuracy still depends on the recognition capability of a single sensor, limiting scalability. Summary of the Invention

[0005] The purpose of this invention is to provide an adaptive multi-source sensor fusion recognition method based on directed graph convolution, which takes into account the characteristics of multi-source sensors in detail and deeply fuses multi-source sensor information to improve recognition accuracy.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] An adaptive multi-source sensor fusion recognition method based on directed graph convolution includes the following steps:

[0008] Acquire information collected by different types of sensors on the vehicle;

[0009] By using the single sensor recognition model corresponding to each type of sensor, the information collected by different types of sensors is identified, and the single sensor recognition result is obtained.

[0010] Constructing a dynamic directed graph based on the recognition results of a single sensor;

[0011] Using a dynamic directed graph as input to a directed graph fusion model, the system performs multi-source sensor information fusion and recognition, and outputs the recognition results.

[0012] The sensors include lidar, cameras, and millimeter-wave radar.

[0013] The single-sensor recognition model is built on a deep neural network and includes an input layer, a hidden layer, an output layer, and a softmax layer connected in sequence. The number of neurons in the input layer is determined according to the number of sensors of the current type, the number of neurons in the output layer is determined according to the type of object to be recognized, the number of output layers is determined according to the sensor category and resolution, and the softmax layer outputs the type of object recognized and its maximum probability.

[0014] The number of neurons in the last layer of the hidden layer of the single sensor recognition model corresponding to different sensor types is the same.

[0015] The construction of the dynamic directed graph follows the dynamic principle and the directed principle. The dynamic principle is to calculate the maximum probability of the single sensor recognition result corresponding to different sensor types in real time according to the changes in the vehicle driving environment and sort them from largest to smallest. The directed principle is to expand in a directed manner according to the maximum probability sort of the single sensor recognition result updated in real time.

[0016] The dynamic directed graph consists of nodes U and edges E connecting the nodes, i.e., G = (U, E). The neurons of the last hidden layer of the single-sensor recognition model are used as nodes in the dynamic directed graph, i.e., each node is 1×N. G Vector, N G The number of neurons in the last hidden layer of the single sensor recognition model; the edge matrix E of the dynamic directed graph is an N×N square matrix that expresses the information connection relationship between pairs of sensors, where N is the number of vehicle sensors. Each element e in the edge matrix takes the value 0 or 1, and is determined according to the following principle: for edges between sensor nodes of the same type, e is represented by 1; for sensor nodes of different types, according to the expansion order determined in the direction principle, edges e that follow the expansion order are represented by 1, and edges e that do not follow the expansion order are represented by 0.

[0017] The directed graph fusion model includes an information fusion module based on graph convolutional neural networks and a recognition module based on convolutional neural networks.

[0018] The information fusion module performs the following operations: calculates the adjacency matrix using A = E + I, where I is the identity matrix and E is the edge matrix of the dynamic directed graph; and calculates the adjacency matrix using d mm =∑A mn Calculate the metric matrix element d mm , forming a metric matrix D, where D is a diagonal matrix;

[0019] Iteration is performed using the following formula:

[0020]

[0021] In the formula, W is the network constant to be learned, l G Z represents the number of graph convolution iterations, and Z is the iteration element, i.e., Zi. 0 =U.

[0022] The recognition module is a convolutional neural network with a softmax layer. Its input is the output of the graph convolutional neural network of the information fusion module corresponding to the maximum probability sensor determined by dynamic principles, i.e., z. max The recognition module performs recognition on the input using the following convolution formula and outputs the recognition result through a softmax layer:

[0023] H = Conv(z) max )

[0024] In the formula, Conv(·) is the convolution calculation formula, and H is the output of the convolutional neural network.

[0025] During the iterative process of the directed graph fusion model, each graph convolution iteration expands one type of sensor information. The adaptive iterative judgment is made by using the maximum probability in the output result of the recognition module during the iteration. That is, if the maximum probability output by the recognition module in the next iteration is less than the maximum probability output in the current iteration, the result of the current iteration is used as the final output, and the graph convolution iteration is terminated.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) This invention determines the direction of information fusion by using the recognition results output by a single sensor recognition model, thereby constructing a dynamic directed graph. It also utilizes the fusion of multi-source information during the convolution stage of the directed graph, taking into account the characteristics of different types of sensors and the differentiated features of interest, which can significantly improve the adaptability of the recognition algorithm in extreme driving environments.

[0028] (2) The present invention uses the output results during the graph convolution iteration process to make adaptive iteration judgment, thereby providing a standard for stopping the fusion of information, avoiding the inclusion of erroneous information, and further improving the recognition accuracy. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention;

[0030] Figure 2 This is a network architecture diagram of a single sensor recognition model in one embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram illustrating the directionality of information expansion in one embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a dynamic directed graph in one embodiment of the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0034] Perception and recognition accuracy significantly impacts the safety of intelligent vehicles, especially in extreme driving environments. A fusion algorithm that deeply integrates multi-source sensor information and meticulously considers the characteristics of these sensors is urgently needed to ensure that intelligent vehicles can detect obstacles in real time. By considering the characteristics of different sensor types and their differentiated features, the adaptability of the recognition algorithm in extreme driving environments can be significantly improved. Therefore, this embodiment provides an adaptive multi-source sensor fusion recognition method based on directed graph convolution, comprising three parts: constructing a dynamic directed graph based on sensor characteristics, constructing a directed graph fusion model, and an adaptive decision model. The first part determines the direction of information fusion from multiple sensors; the second part uses a directed graph convolution stage to fuse multi-source information; and the third part provides a criterion for stopping information fusion to avoid incorporating erroneous information.

[0035] Specifically, such as Figure 1 As shown, it includes the following steps:

[0036] S1 acquires information collected by different types of sensors on the vehicle.

[0037] Commonly used sensor types in intelligent vehicles include cameras, lidar, and millimeter-wave radar. This embodiment will use these three types of sensors as examples for illustration.

[0038] S2, using the single sensor recognition model corresponding to each type of sensor to identify the information collected by different types of sensors, and obtain the single sensor recognition result.

[0039] The single-sensor recognition model is built on a deep neural network and includes an input layer, a hidden layer, an output layer, and a softmax layer connected in sequence. The number of neurons in the input layer is determined according to the number of sensors of the current type, the number of neurons in the output layer is determined according to the type of object to be recognized, the number of output layers is determined according to the sensor category and resolution, and the softmax layer outputs the type of object recognized and its maximum probability.

[0040] In one embodiment, such as Figure 2 As shown, a single-sensor recognition model is constructed using a unified 4-layer deep neural network with a softmax layer, consisting of an input layer, hidden layers, an output layer, and a softmax layer connected sequentially. Furthermore, the types of objects to be recognized are simplified to unify the network, including: people, bicycles, electric vehicles, cars, and trucks, meaning the output layer has 5 neurons. In this network, the number of output layers varies depending on the sensor type and resolution. The softmax layer normalizes probabilities, using the result with the highest probability as the network's recognition outcome. It is important to note the design of the hidden layers, which has strict requirements on the number of neurons in the last hidden layer; the number of neurons in the last hidden layer of the model must be consistent across different sensors, defined here as N. G That is, lh m-1 ≡N G Where m is the total number of hidden layers. In this embodiment, as... Figure 2 As shown, the total number of hidden layers is 2, meaning the number of neurons in the second hidden layer is required to be uniform, which can be initially set to 5. This is related to the computing power of the controller being used. Generally speaking, more network layers and more neurons per hidden layer increase the computing power requirement, while also improving accuracy.

[0041] S3 constructs a dynamic directed graph based on the recognition results of a single sensor.

[0042] Different sensors, due to their varying characteristics, are best suited to different environments. In good weather, cameras offer higher recognition accuracy than electromagnetic wave-based sensors, but are susceptible to visibility limitations. In contrast, lidar and millimeter-wave radar offer higher penetration and robustness, but their accuracy is questionable because they cannot recognize effective information such as color. Furthermore, it's important to note that while both lidar and millimeter-wave radar output point clouds, they differ in the electromagnetic wave frequency bands they employ. Compared to lidar, millimeter-wave radar exhibits better penetration under adverse weather conditions (such as fog, rain, and snow). Therefore, different sensors capture different types of obstacle information and are suitable for different environments. To address this, this invention constructs a dynamic directed graph as the basis for information fusion. This graph primarily consists of two core components: the dynamic principle and the directed principle.

[0043] 1. Dynamic principle

[0044] Since intelligent vehicles experience changing environments, relying on fixed primary and secondary sensors is unreasonable. To adapt to the environment and improve the efficiency of using effective information, this invention designs a unique dynamic principle: based on changes in the vehicle's driving environment, it calculates in real time the maximum probability of the single sensor recognition results corresponding to different sensor types and sorts them from highest to lowest.

[0045] In autonomous vehicles, multiple sensors of the same type are typically used. Generally, there are four cameras (front, rear, left, and right), three LiDARs (one main LiDAR and two blind-spot LiDARs), and six millimeter-wave radars (three on each of the front and rear bumpers). It's important to note that the sensor configurations provided here are common configurations; the number of sensors can be varied based on cost. Then, based on the maximum probability output of the softmax layer of each individual sensor's recognition model, the maximum probability among various types of sensors is statistically analyzed to evaluate environmental adaptability. In this embodiment, the maximum probability ranking of the recognition results is assumed to be: P... max-L >P max-C >P max-R , where P max-L P max-C and P max-R These represent the maximum output probabilities in the recognition models of LiDAR, camera, and millimeter-wave radar, respectively. It's important to note that this ranking is dynamic and changes based on the vehicle's driving environment; that is, the ranking is updated in real-time as the vehicle is in motion.

[0046] 2. Direction Principle

[0047] To ensure efficient use of effective information while avoiding interference from erroneous information, the information used should be expanded in a directional manner. Based on the above dynamic principles, the ranking of the highest probability results from a single sensor is used as the ranking of the effective sensor information, and directional expansion is performed according to this ranking. It is important to note that each expansion incorporates only information from one sensor.

[0048] Based on the ranking results above, the expansion order in this embodiment is lidar, camera, and millimeter-wave radar, as follows: Figure 3 As shown.

[0049] The dynamic directed graph consists of nodes U and edges E connecting the nodes, i.e., G = (U, E). The neurons of the last hidden layer of the single-sensor recognition model are used as nodes in the dynamic directed graph, meaning each node is 1 × N. G Vector, N G This represents the number of neurons in the last hidden layer of a single-sensor recognition model. This embodiment uses three sensors—LiDAR, camera, and millimeter-wave radar—as examples. To effectively represent dynamic directed graphs, a unified description method is used to represent different types of sensor nodes, i.e., U. Ci U represents a node composed of hidden layer neurons of a single-sensor recognition model from the i-th camera. Lj U represents a node composed of hidden layer neurons in a single-sensor recognition model of the j-th lidar. Rk This represents a node composed of hidden layer neurons from a single sensor recognition model of the k-th millimeter-wave radar.

[0050] The edge matrix E of the dynamic directed graph is an N×N square matrix representing the information connection relationships between pairs of sensors, where N is the number of vehicle sensors. Each element e in the edge matrix takes either 0 or 1, determined according to the following principle:

[0051] 1. For sensors of the same type, they exhibit similar performance under the same environment, and their information should be equivalent. Therefore, nodes of the same type are represented by class 1, i.e., e. LL =e RR =e CC =1 Where, the subscript L represents LiDAR, R represents millimeter-wave radar, and C represents camera.

[0052] 2. For different types of sensor nodes, according to the expansion order determined in the directed principle, edges e that follow the expansion order are represented by 1, and edges e that do not follow the expansion order are represented by 0. In this embodiment, the expansion order determined according to the aforementioned steps is LiDAR, camera, and millimeter-wave radar. Therefore, information transmission proceeds from LiDAR to camera and finally to millimeter-wave radar, i.e., e LC =e CR=1, the corresponding reverse information cannot be passed, i.e., e RC =e RL =e CL =0. The directed graph constructed based on this is as follows: Figure 4 As shown, Figure 4 The system contains 13 sensors: 4 cameras, 3 LiDARs, and 6 millimeter-wave radars.

[0053] S4 uses a dynamic directed graph as input to a directed graph fusion model, and uses the directed graph fusion model to fuse and identify information from multiple sources of sensors, outputting the identification results.

[0054] The directed graph fusion model includes an information fusion module based on graph convolutional neural networks and a recognition module based on convolutional neural networks.

[0055] The information fusion module is used to fuse information from multiple sensor sources. It performs the following operations: calculates the adjacency matrix using A = E + I, where I is the identity matrix and E is the edge matrix of the dynamic directed graph; and then... mm =∑A mn Calculate the metric matrix element d mm , forming a metric matrix D, where D is a diagonal matrix;

[0056] Iteration is performed using the following formula:

[0057]

[0058] In the formula, W is the network constant to be learned, l G Z represents the number of graph convolution iterations, and Z is the iteration element, i.e., Zi. 0 =U.

[0059] The recognition module is a convolutional neural network with a softmax layer. Its input is the output of the graph convolutional neural network of the information fusion module corresponding to the maximum probability sensor determined by dynamic principles, i.e., z. max It is a node in the iterative output of a graph convolutional neural network. The recognition module recognizes the input using the following convolution formula and outputs the recognition result through a softmax layer:

[0060] H = Conv(z) max )

[0061] In the formula, Conv(·) is the convolution calculation formula, and H is the output of the convolutional neural network. In this embodiment, the structure of the recognition module is consistent with the structure of the single-sensor recognition model.

[0062] In the iterative process of the directed graph fusion model, each graph convolution iteration expands one type of sensor information. Taking the expansion direction determined in this embodiment as an example, the first graph convolution fuses LiDAR and camera information, and the second graph convolution fuses LiDAR, camera, and millimeter-wave radar information. It should be noted that this description is based on the results of the above embodiment only. When the expansion order changes according to the environment, the fusion process in this part will also change accordingly. To avoid the inclusion of erroneous information that leads to a decrease in recognition accuracy, this embodiment uses the maximum probability from the output of the graph convolutional neural network combined with the recognition module in the iterative process for adaptive judgment. If the maximum probability output by the recognition module in the next iteration is less than the maximum probability output in the current iteration, the result of the current iteration is used as the final output, and the graph convolution iteration is terminated, i.e., the lth iteration... G The maximum probability output by the recognition module in the next iteration The l G The maximum probability output by the recognition module in +1 iterations At that time, the lth G The result of fusing the information from the graph convolution is used as the final output, and the graph convolution iteration is terminated.

[0063] The deployment of this invention in a real vehicle involves five steps, as detailed below:

[0064] 1. Determine the types and number of sensors required for vehicle deployment in order to deploy a single-sensor recognition model.

[0065] 2. Prepare a dataset for extreme environments. This dataset is used to train the model and therefore contains accurate results with human annotations.

[0066] 3. Train the single-sensor recognition model until it converges.

[0067] 4. Based on the output of the single-sensor recognition model, train a directed graph fusion model based on directed graph convolution and converge it. Steps 1 to 4 above are for offline model training.

[0068] 5. Download the converged single-sensor recognition model and the directed graph fusion model to the intelligent vehicle recognition and control system simultaneously to complete the model deployment.

[0069] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An adaptive multi-source sensor fusion recognition method based on directed graph convolution, characterized in that, Includes the following steps: Acquire information collected by different types of sensors in the vehicle, including lidar, cameras, and millimeter-wave radar; By using the single sensor recognition model corresponding to each type of sensor, the information collected by different types of sensors is identified, and the single sensor recognition result is obtained. The single-sensor recognition model outputs the type of object identified and its maximum probability. Constructing a dynamic directed graph based on the recognition results of a single sensor; The construction of the dynamic directed graph follows the dynamic principle and the directed principle. The dynamic principle is: based on the changes in the vehicle driving environment, the maximum probability of the single sensor recognition result corresponding to different sensor types is calculated in real time and sorted from largest to smallest. The directed principle is: the maximum probability of the single sensor recognition result is sorted in a directed manner based on the real-time updated single sensor recognition result. The dynamic directed graph consists of nodes and the edges connecting nodes Composition, that is The neurons of the last hidden layer of the single-sensor recognition model are treated as nodes in a dynamic directed graph, meaning each node is... vector, The number of neurons in the last hidden layer of a single-sensor recognition model; the edge matrix of a dynamic directed graph. For one The square matrix represents the information connection relationship between each pair of sensors. For the number of vehicle sensors, each element in the side matrix The value is either 0 or 1, determined according to the following principle: for edges between sensor nodes of the same type... Represented by 1; for different types of sensor nodes, according to the expansion order determined in the directed principle, for edges following the expansion order... An edge that does not follow the expansion order is represented by 1. Represented by 0; Using a dynamic directed graph as input to a directed graph fusion model, the system performs multi-source sensor information fusion and recognition, and outputs the recognition results.

2. The adaptive multi-source sensor fusion recognition method based on directed graph convolution according to claim 1, characterized in that, The single-sensor recognition model is built on a deep neural network and includes an input layer, a hidden layer, an output layer, and a softmax layer connected in sequence. The number of neurons in the input layer is determined according to the number of sensors of the current type, the number of neurons in the output layer is determined according to the type of object to be recognized, and the number of output layers is determined according to the sensor category and resolution.

3. The adaptive multi-source sensor fusion recognition method based on directed graph convolution according to claim 2, characterized in that, The number of neurons in the last layer of the hidden layer of the single sensor recognition model corresponding to different sensor types is the same.

4. The adaptive multi-source sensor fusion recognition method based on directed graph convolution according to claim 1, characterized in that, The directed graph fusion model includes an information fusion module based on graph convolutional neural networks and a recognition module based on convolutional neural networks.

5. The adaptive multi-source sensor fusion recognition method based on directed graph convolution according to claim 4, characterized in that, The information fusion module performs the following operations: through Calculate the adjacency matrix, where, It is the identity matrix. The edge matrix of a dynamic directed graph; through Calculate the elements of the metric matrix , forming a metric matrix , D It is a diagonal matrix; Iteration is performed using the following formula: In the formula The network constants to be learned This indicates the number of graph convolution iterations. For the iterable element, i.e. .

6. The adaptive multi-source sensor fusion recognition method based on directed graph convolution according to claim 4, characterized in that, The recognition module is a convolutional neural network with a softmax layer. Its input is the output of the graph convolutional neural network of the information fusion module corresponding to the maximum probability sensor determined by dynamic principles. The recognition module performs recognition on the input using the following convolution formula and outputs the recognition result through a softmax layer: In the formula Here is the formula for calculating convolution. This is the output of the convolutional neural network.

7. The adaptive multi-source sensor fusion recognition method based on directed graph convolution according to claim 4, characterized in that, During the iterative process of the directed graph fusion model, each graph convolution iteration expands one type of sensor information. The adaptive iterative judgment is made by using the maximum probability in the output result of the recognition module during the iteration. That is, if the maximum probability output by the recognition module in the next iteration is less than the maximum probability output in the current iteration, the result of the current iteration is used as the final output, and the graph convolution iteration is terminated.

Images