Multi-modal data fusion processing method, vehicle and storage medium

Abstract

The invention discloses a multi-modal data fusion processing method, a vehicle and a storage medium, and the method comprises the steps: carrying out the feature extraction of the data of various modalities, so as to obtain first feature vectors corresponding to the data of various modalities; performing feature processing on each first feature vector to obtain a second feature vector corresponding to each first feature vector under a bird's-eye view; and performing fusion processing on each second feature vector to generate a fused feature vector. Information loss or distortion can be avoided, the data scale of each mode is the same, and the interpretability of the system is improved; the generated fusion feature vector may include details and information captured by different modalities, i.e., these details and information may be combined, thereby improving the perceptual performance of the vehicle.

Description

Technical Field

[0001] The present application belongs to the field of intelligent driving technology, and in particular, relates to a multimodal data fusion processing method, a vehicle, and a computer-readable storage medium. Background Art

[0002] At present, environmental perception components such as radar sensors and visual sensors generally installed on vehicles can be used to perceive the vehicle's surrounding environment, such as detecting vehicles, pedestrians, roads, etc. in the surrounding environment.

[0003] However, at present, the main method used is to fuse the detection results of various perception components to obtain the final environmental perception result. That is, each sensor outputs its own result after algorithm processing, and the results are fused according to certain rules. This method lacks effective interpretability, and the characteristic scales between the sensors cannot be unified, and the accuracy of the fusion results cannot be guaranteed, resulting in poor vehicle perception performance. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a multi-modal data fusion processing method, a vehicle and a computer-readable storage medium. The data of multiple modalities can be fused and the accuracy of the fusion result can be guaranteed, and the perception performance of the vehicle is better.

[0005] The multimodal data fusion processing method provided in the present application includes: performing feature extraction on data of various modalities respectively to obtain first feature vectors corresponding to the data of various modalities respectively; performing feature processing on each of the first feature vectors respectively to obtain second feature vectors corresponding to each of the first feature vectors from a bird's-eye view; and performing fusion processing on each of the second feature vectors to generate a fused feature vector.

[0006] In some embodiments, the multimodal data includes image data and point cloud data, and the feature extraction of the data of each modality is performed to obtain the first feature vector corresponding to each modality of the data, including: feature extraction of the image data to obtain the first image feature vector corresponding to the image data; feature extraction of the point cloud data to obtain the first point cloud feature vector corresponding to the point cloud data.

[0007] In some embodiments, the feature extraction of the image data to obtain a first image feature vector corresponding to the image data includes: based on a preset residual network, feature extraction of the image data to generate image feature vectors corresponding to multiple different image resolutions; based on a preset feature graph pyramid network, processing the image feature vectors of multiple different image resolutions to generate the first image feature vectors of multiple different scales.

[0008] In some embodiments, the feature extraction of the point cloud data to obtain a point cloud feature vector corresponding to the point cloud data includes: segmenting the point cloud data to obtain a plurality of point cloud units; encoding the point clouds in each of the point cloud units to obtain tensors corresponding to each of the point cloud data; decoding each of the tensors to obtain point cloud coordinates; and generating the first point cloud feature vector based on the point cloud coordinates.

[0009] In some embodiments, the feature processing is performed on each of the first feature vectors to obtain a second feature vector from a bird's-eye view, including: feature processing the first image feature vector to obtain a second image feature vector corresponding to the first image feature vector from a bird's-eye view; feature processing the first point cloud feature vector to obtain a second point cloud feature vector corresponding to the first point cloud feature vector from a bird's-eye view.

[0010] In some embodiments, the feature processing of the first image feature vector to obtain the second image feature vector corresponding to the first image feature vector from a bird's-eye view includes: projecting the first image feature vector corresponding to each camera to multiple preset heights based on the intrinsic parameters and extrinsic parameters of each camera to obtain the image features at each preset height; sampling the image features at each preset height corresponding to each camera to obtain the sampled image features corresponding to each camera; fusing the sampled image features corresponding to each camera to generate the third image feature vector; generating a fourth image feature vector based on the first point cloud feature vector of the current frame and the second image feature vector of the previous frame, the second image feature vector of the first frame being equal to the third image feature vector; generating the second image feature vector of the current frame based on the fourth image feature vector and the first image feature vector of the current frame.

[0011] In some embodiments, generating the second image feature vector of the current frame based on the fourth image feature vector and the first image feature vector of the current frame includes: based on an attention mechanism, fusing the fourth image feature vector and the first image feature vector of the current frame to generate the second image feature vector of the current frame.

[0012] In some embodiments, the attention mechanism is used to calculate the degree of association between the point cloud data and the image data.

[0013] In some embodiments, the feature processing of the first point cloud feature vector to obtain a second point cloud feature vector corresponding to the first point cloud feature vector from a bird's-eye view includes: based on a preset transformation relationship between a world coordinate system and a point cloud coordinate system, converting the first point cloud feature vector into the second point cloud feature vector.

[0014] In some embodiments, the method further includes: processing the fused feature vector based on a multi-layer perceptron to output multiple processing results, wherein the processing results include at least one of a target detection result, a panoramic segmentation result, a lane line detection result, and an environment detection result.

[0015] The present application provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the multimodal data fusion processing method described above is implemented.

[0016] The present application provides a vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multimodal data fusion processing method when executing the program.

[0017] The multimodal data fusion processing method, vehicle and computer-readable storage medium provided in the embodiments of the present application respectively extract features from data of various modalities to obtain first feature vectors corresponding to the data of various modalities, and then perform feature processing on each first feature vector to convert each first feature vector to a bird's-eye view, thereby obtaining a second feature vector corresponding to each first feature vector, so that the feature scale of each modality is unified, and the second feature vector corresponding to each modality has a clear meaning and interpretability, thereby improving the interpretability of the system; finally, each second feature vector is fused to generate a fused feature vector, which may include details and information captured by different modalities, thereby improving the accuracy of environmental perception based on the fused feature vector, thereby improving the perception performance of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0019] Figure 1 It is an application scenario diagram of the multimodal data fusion processing method provided in the embodiment of the present application;

[0020] Figure 2It is a scene schematic diagram of the fusion processing method of multimodal data provided in an embodiment of the present application;

[0021] Figure 3 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0022] Figure 4 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0023] Figure 5 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0024] Figure 6 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0025] Figure 7 It is a scene schematic diagram of the fusion processing method of multimodal data provided in an embodiment of the present application;

[0026] Figure 8 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0027] Fig. 9 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0028] Fig.10 It is a scene schematic diagram of the fusion processing method of multimodal data provided in an embodiment of the present application;

[0029] Fig.11 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0030] Fig.12 It is a scene schematic diagram of the fusion processing method of multimodal data provided in an embodiment of the present application;

[0031] Fig.13 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0032] Fig.14 It is a flowchart of a method for fusion processing of multimodal data provided in an embodiment of the present application;

[0033] Fig.15 It is a scene schematic diagram of the fusion processing method of multimodal data provided in an embodiment of the present application;

[0034] Fig.16 It is a structural schematic diagram of a multimodal data fusion processing device provided in an embodiment of the present application;

[0035] Fig.17 It is a schematic diagram of the connection status of the non-volatile computer-readable storage medium and the processor provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0037] The following is an introduction to some of the technical terms involved in this application:

[0038] Bird's Eye View (BEV): In the field of autonomous driving, BEV refers to the scene view from above the vehicle. BEV images can provide a complete view of the vehicle's surroundings, including the front, rear, sides and top of the vehicle.

[0039] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0040] Deep Learning: It is a new research direction in the field of machine learning. Its goal is to enable machines to have analytical learning capabilities like humans, so as to solve complex pattern recognition problems. Deep learning aims to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds.

[0041] The core of deep learning lies in neural networks, which are composed of a large number of artificial neurons. Each neuron processes input data by connecting weights and activation functions. The hierarchical structure of a neural network includes input layer, hidden layer and output layer, where there can be multiple hidden layers. In deep learning, data is transmitted through the forward propagation process of the neural network to obtain the output result, and then backpropagation is used to update the weights and biases of the neural network so that it can better adapt to the training data.

[0042] Deep learning model: It is a discriminative model and a machine learning method based on deep neural networks. It can automatically extract features from raw data and abstract them layer by layer to achieve regression, classification or sorting. Deep learning models are usually composed of multiple hidden layers, each of which can perform nonlinear transformations on the input data, thereby increasing the expressive power and complexity of the model. Deep learning models can be regarded as a special discriminative model, especially in classification tasks, which constructs decision boundaries by learning the mapping relationship from input to output.

[0043] After introducing the technical concepts related to this application, the application scenarios of this application are introduced below:

[0044] Figure 1 An application scenario diagram of a multimodal data fusion processing method provided in an embodiment of the present application, such as Figure 1 As shown, the application scenario provided in the present application includes a vehicle 100 , and the multimodal data fusion processing method provided in the present application can be executed by the vehicle 100 .

[0045] The vehicle 100 may be any vehicle 100 equipped with a sensor of any modality, such as a car, a truck, etc. In the implementation manner of the present application, the vehicle 100 is taken as an example to be described.

[0046] The vehicle 100 includes a vehicle body 10 , a processor 20 , and a memory 30 , and the processor 20 is disposed inside the vehicle body 10 .

[0047] Optionally, the memory 30 can be used to store a preset residual network, a feature map pyramid network, etc.

[0048] In one embodiment, the vehicle 100 further includes a visual sensor, which is used to collect scene images around the vehicle 100 .

[0049] Optionally, the visual sensor includes one or more. The visual sensor can be arranged on the side wall of the vehicle body 10, for example, see Figure 1 , the visual sensors include 6, namely, visual sensor 41, visual sensor 42, visual sensor 43, visual sensor 44, visual sensor 45 and visual sensor 46. Among them, the visual sensor 41 is arranged at the left front of the vehicle body 10, the visual sensor 42 is arranged at the left B-pillar of the vehicle body 10, the visual sensor 43 is arranged at the left rear of the vehicle body 10, the visual sensor 44 is arranged at the right front of the vehicle body 10, the visual sensor 45 is arranged at the right B-pillar of the vehicle body 10, and the visual sensor 46 is arranged at the right rear of the vehicle body 10.

[0050] In one embodiment, the vehicle 100 further includes a radar sensor 50 , which is used to collect point cloud information of objects around the vehicle 100 .

[0051] Optionally, the radar sensor may be a laser direct structuring (LDS) radar, such as a TOF radar based on the time of flight (TOF) principle, a structured light radar based on the structured light principle, and the like.

[0052] Optionally, the radar sensor may be a millimeter-wave radar or the like.

[0053] Optionally, the radar sensor is disposed on the vehicle body 10, and the radar sensor includes one or more radar sensors, and the multiple radar sensors may be radar sensors of different types. Figure 1 The radar sensors include two, namely a laser radar 51 and a millimeter wave radar 52.

[0054] Intelligent driving has attracted more and more attention due to its potential to reduce the burden on drivers and improve driving safety. The perception performance of vehicles plays a vital role in intelligent driving. Vehicles can provide necessary data and feedback for intelligent driving by perceiving the surrounding environment in real time, providing a basis for intelligent driving planning and decision-making.

[0055] At present, environmental perception components such as radar sensors and visual sensors generally installed on vehicles can be used to perceive the vehicle's surrounding environment, such as detecting vehicles, pedestrians, roads, etc. in the surrounding environment.

[0056] However, at present, the main method used is to fuse the detection results of various perception components to obtain the final environmental perception result. That is, each sensor outputs its own result after algorithm processing, and the results are fused according to certain rules. This method lacks effective interpretability, and the characteristic scales between the sensors cannot be unified, and the accuracy of the fusion results cannot be guaranteed, resulting in poor vehicle perception performance.

[0057] Based on the above basic concepts and related scenarios, the present application embodiment provides a fusion processing method for multimodal data. The fusion processing method for multimodal data is described in detail below:

[0058] See also Figure 1 , Figure 2 and Figure 3 , Figure 1 is a schematic diagram of a scenario in which the multimodal data fusion processing method provided in an embodiment of the present application is applied to a vehicle. Figure 2is a schematic diagram of a scenario of a fusion processing method for multimodal data provided in an embodiment of the present application, Figure 3 1 is a flow chart of a multimodal data fusion processing method provided in an embodiment of the present application. A multimodal data fusion processing method provided in an embodiment of the present application comprises the following steps:

[0059] Step 011: extract features from data of various modes respectively, so as to obtain first feature vectors corresponding to the data of various modes respectively.

[0060] The data of various modalities may be data generated after various types of sensors collect the environment of the scene in which the vehicle is located. For example, the data of various modalities may include data of visual sensors (such as images) and data of laser radar sensors (such as point clouds). Feature extraction can capture key features in data of various modalities and represent these key features through feature vectors.

[0061] Specifically, the vehicle may include two types of sensors, a visual sensor and a radar sensor (the embodiment of the present application takes the radar sensor as a laser radar sensor as an example), and feature extraction is performed on the data of the visual sensor and the laser radar sensor respectively to obtain the first feature vectors corresponding to the data of the two modes. For example, the vehicle captures the scene information around the vehicle through a visual sensor to obtain an image, and then extracts the color features, texture features, and shape features in the image through a feature extraction algorithm such as a Histogram of Oriented Gradient (HOG), a Local Binary Pattern (LBP) feature extraction algorithm, or a Haar (Haar-like features, Haar) feature extraction algorithm, and generates a corresponding first feature vector 1; the vehicle obtains point cloud data through a laser radar sensor, and then extracts the point cloud data through a method such as PointNet, and generates a corresponding first feature vector 2.

[0062] Among them, PointNet is a deep learning architecture for processing point cloud data. It can perform tasks such as classification, segmentation, and detection on point cloud data through a combination of fully connected neural networks and convolutional neural networks.

[0063] Step 012: Perform feature processing on each first eigenvector to obtain a second eigenvector corresponding to each first eigenvector from a bird's-eye view.

[0064] The feature processing may include converting the first feature vectors corresponding to each mode into a bird's-eye view to perform a scale-unified conversion process.

[0065] Specifically, the features of data measured or captured by sensors of different modalities differ in magnitude or range. Different sensors may measure similar physical quantities or phenomena, such as light intensity, at different scales or units. That is, sensors of different modalities have different feature scales. When performing data fusion, directly fusing data of different scales may result in information loss or distortion.

[0066] Therefore, the first eigenvectors corresponding to the data of each modality can be processed separately, such as performing view transformation on the first eigenvector corresponding to the data of the visual sensor modality to project the first eigenvector corresponding to the image of the visual sensor modality into the space corresponding to the bird's-eye view, and performing coordinate transformation on the first eigenvector corresponding to the point cloud of the lidar sensor modality to project it into the space corresponding to the bird's-eye view, thereby avoiding information loss or distortion. The data scales of each modality are the same, and the second eigenvectors corresponding to each modality have clear meanings and interpretability, thereby improving the interpretability of the system.

[0067] Step 013: Perform fusion processing on each second feature vector to generate a fused feature vector.

[0068] Specifically, after converting each second eigenvector to a bird's-eye view, each second eigenvector is fused through methods such as vector splicing, weighted sum, etc. to generate a fused eigenvector. The second eigenvectors corresponding to different modalities represent the details and information captured by different modalities. By fusing each second eigenvector, these details and information can be combined to improve the vehicle's perception performance.

[0069] In this way, feature extraction is performed on data of various modalities respectively to obtain the first feature vectors corresponding to the data of various modalities respectively, and then feature processing is performed on each first feature vector respectively to convert each first feature vector to a bird's-eye view, so as to obtain the second feature vector corresponding to each first feature vector, so that the feature scale of each modality is unified, and the second feature vector corresponding to each modality has a clear meaning and interpretability, thereby improving the interpretability of the system; finally, the second feature vectors are fused to generate a fused feature vector, which can include details and information captured by different modalities, thereby improving the accuracy of environmental perception based on the fused feature vector, thereby improving the perception performance of the vehicle.

[0070] See also Figure 4 In some embodiments, the multimodal data includes image data and point cloud data. Step 011: extracting features from the data of each modality to obtain first feature vectors corresponding to the data of each modality, including:

[0071] Step 0111: extracting features from the image data to obtain a first image feature vector corresponding to the image data;

[0072] Step 0112: Perform feature extraction on the point cloud data to obtain a first point cloud feature vector corresponding to the point cloud data.

[0073] The image data may be an image collected by a visual sensor, and the point cloud data may be a collection of points collected by a radar sensor. The point cloud data may include laser point cloud data, millimeter wave point cloud data, or both laser point cloud data and millimeter wave point cloud data.

[0074] Specifically, by performing feature extraction on the image data to obtain the first image feature vector corresponding to the image data, and performing feature extraction on the point cloud data to obtain the first point cloud feature vector corresponding to the point cloud data, when the point cloud data includes laser point cloud data, after performing feature extraction on the point cloud data, the first point cloud feature vector corresponding to the laser point cloud data is obtained; when the point cloud data includes millimeter wave point cloud data, after performing feature extraction on the point cloud data, the first point cloud feature vector corresponding to the millimeter wave point cloud data is obtained; when the point cloud data includes laser point cloud data and millimeter wave point cloud data, after performing feature extraction on the point cloud data, the first point cloud feature vector obtained includes the laser point cloud feature vector corresponding to the laser point cloud data and the millimeter wave point cloud feature vector corresponding to the millimeter wave point cloud data. The key information contained in the image data and the point cloud data is respectively refined for subsequent processing.

[0075] See also Figure 5 In some implementations, step 0111: extracting features from the image data to obtain a first image feature vector corresponding to the image data includes:

[0076] Step 01111: Based on a preset residual network, feature extraction is performed on the image data to generate image feature vectors corresponding to a plurality of different image resolutions;

[0077] Step 01112: Based on a preset feature graph pyramid network, multiple image feature vectors of different image resolutions are processed to generate multiple first image feature vectors of different scales.

[0078] Among them, the preset residual network can be such as Resnet-50, which is a deep learning model composed of multiple residual blocks and includes some additional layers such as pooling layers and fully connected layers. The structure of the entire network is very deep and has strong feature extraction capabilities. The residual learning of information can be achieved by adding the input to the output of the main path. In addition, batch normalization and activation functions (such as ReLU functions) are used in each residual block to further improve the performance of the model. These convolutional layers extract features from images at different sizes and depths, allowing the model to capture features at different levels.

[0079] Batch Normalization is a technique used to improve the neural network training process. Its core is to normalize the output of each layer of the neural network. The purpose of batch normalization is to solve the problem of internal covariate shift caused by parameter updates during the training of the neural network, that is, the input distribution of each layer will change during the training process. Through batch normalization, the input of each layer can be forced to return to a standard normal distribution with a mean of 0 and a variance of 1, which can make the activation value of the network more stable, help speed up convergence, and improve training efficiency.

[0080] The ReLU function, or linear rectification function, is an activation function commonly used in deep learning and neural networks. It can simplify the calculation process and avoid the problems of gradient explosion and gradient disappearance.

[0081] Among them, the preset feature graph pyramid network can be (FeaturePyramidNetworks, FPN), etc. FPN is a network structure used for target detection and semantic segmentation. It effectively improves the semantic expression ability of low-level features through cross-layer connections and top-level feature pyramids, and generates a series of feature pyramids of different scales. In FPN, cross-layer connections combine high-resolution low-level features with high-level features to obtain feature representations with more semantic information. Low-level features can provide more detailed information, while high-level features can provide higher-level semantic information.

[0082] Specifically, one preprocessing module and four convolution modules can be set for Resnet-50. The preprocessing module includes a convolution layer and a pooling layer, and the convolution layer and the pooling layer can reduce the size of the image by two times respectively. The four convolution modules are named res2, res3, res4 and res5 respectively. The image can be first passed through the preprocessing module to reduce the image size by 4 times, and then the preprocessed image can be sent to the four convolution modules respectively to extract features from the image, thereby generating image feature vectors with four different image resolutions.

[0083] Based on the preset feature map pyramid network, multiple image feature vectors with different image resolutions are used as the input of the preset feature map pyramid network. First, the corresponding image feature vector generated after res5 processing is subjected to channel reduction processing (such as reducing the number of channels to 256 through 1*1 convolution) and convolution processing (such as through 3*3 convolution) to obtain a first image feature vector P5; then the first image feature vector P5 is upsampled to obtain an upsampling result with a channel number of 256.

[0084] Then, the corresponding image feature vector generated after the processing of res4 is subjected to channel reduction processing (such as reducing the number of channels to 256 through 1*1 convolution) to obtain the convolution result corresponding to the output of res4, and then the values ​​of the upsampled first image feature vector P5 and the convolution result corresponding to the output of res4 are added and averaged at the positions corresponding to each pixel to generate the added image feature vector, and then the added image feature vector is subjected to 3*3 convolution to obtain the corresponding first image feature vector P4. Similarly, P4 is processed as described above to obtain the first image feature vector P3, and P3 is processed as described above to obtain the first image feature vector P2. Finally, the first image feature vector P5 is subjected to maximum pooling processing to obtain the first image feature vector P6. The first feature vector includes the first image feature vector P2, the first image feature vector P3, the first image feature vector P4, the first image feature vector P5, and the first image feature vector P6.

[0085] For example, taking an image size of 3*800*1216 as an example, based on the preset Resnet-50, the 3*800*1216 image is first sent to the preprocessing module to reduce the image size by 4 times. In order to ensure the effect of feature extraction and avoid missing image details, the image processed by the preprocessing module will be reduced to 64*200*304 in size; then the 64*200*304 image is sent to 4 convolution modules in turn. After being processed by res2, res3, res4 and res5, the corresponding image feature vectors of 4 different image resolutions are generated, among which res2 generates an image feature vector of 256*200*304, res3 generates an image feature vector of 512*100*152, res4 generates an image feature vector of 1024*50*76, and res5 generates an image feature vector of 2048*25*38.

[0086] Based on the preset FPN, the image feature vectors generated by res2, res3, res4 and res5 are used as the input of FPN. First, the 2048*25*38 image feature vector generated by res5 is convolved by 1*1 to reduce the number of channels to 256, and then a 3*3 convolution is performed to obtain the first image feature vector P5, where the scale of P5 is 256*25*38.

[0087] Then, the first image feature vector P5 is upsampled to obtain an upsampling result with a channel number of 256 (size is 256*50*76), and the image feature vector of 1024*50*76 generated by res4 is 1*1 convolved to reduce the number of channels to 256, and then the 1*1 convolution result of the image feature vector of 1024*50*76 generated by res4 and the upsampling result of the first image feature vector P5 are added and averaged at each corresponding pixel position to generate an added image feature vector (size is 256*50*76), and then the added image feature vector (size is 256*50*76) is convolved 3*3 to obtain the first image feature vector P4, where the scale of P4 is 256*50*76.

[0088] Next, the first image feature vector P4 is upsampled to obtain an upsampling result with a channel number of 256 (size is 256*100*152), and the image feature vector of 512*100*152 generated by res3 is 1*1 convolved to reduce the number of channels to 256, and then the 1*1 convolution result of the image feature vector of 512*100*152 generated by res3 and the upsampling result of the first image feature vector P4 are added and averaged at each corresponding pixel position to generate an added image feature vector (size is 256*100*152), and then the added image feature vector (size is 256*100*152) is convolved 3*3 to obtain the first image feature vector P3, where the scale of P3 is 256*100*152.

[0089] After that, the first image feature vector P3 is upsampled to obtain an upsampling result with a channel number of 256 (size is 256*200*304), and the image feature vector of 256*200*304 generated by res2 is convolved 1*1 to reduce the number of channels to 256, and then the 1*1 convolution result of the image feature vector of 256*200*304 generated by res2 and the upsampling result of the first image feature vector P3 are added and averaged at each corresponding pixel position to generate an added image feature vector (size is 256*200*304), and then the added image feature vector (size is 256*200*304) is convolved 3*3 to obtain the first image feature vector P2, where the scale of P2 is 256*200*304.

[0090] Finally, P5 is subjected to maximum pooling (MaxPooling) to obtain a first image feature vector P6, wherein the scale of the first image feature vector P6 is 256*13*19.

[0091] That is to say, the image feature vectors of different resolutions generated corresponding to res2, res3, res4 and res5 are used as the input of FPN to output: the first image feature vector P2 (scale is 256*200*304), the first image feature vector P3 (scale is 256*100*152), the first image feature vector P4 (scale is 256*50*76), the first image feature vector P5 (scale is 256*25*38) and the first image feature vector P6 (scale is 256*13*19).

[0092] At present, in order to achieve better feature fusion, when performing image fusion, the feature maps of the last layer processed by the neural network are generally used directly. Although the semantics of the feature maps of the last layer are stronger, the position and resolution of the feature maps of the last layer are low, and it is easy to fail to detect smaller objects. Through FPN, feature maps from the bottom layer to the last layer can be fused, so as to make full use of the features extracted at each stage to provide more detailed detection.

[0093] See also Figure 6 and Figure 7 In some embodiments, step 0112: extracting features from the point cloud data to obtain a point cloud feature vector corresponding to the point cloud data includes:

[0094] Step 01121: segmenting the point cloud data to obtain a plurality of point cloud units;

[0095] Step 01122: Encode the point cloud in each point cloud unit to obtain the tensor corresponding to each point cloud data;

[0096] Step 01123: Decode each tensor to obtain point cloud coordinates;

[0097] Step 01124: Generate a first point cloud feature vector based on the point cloud coordinates.

[0098] The point cloud unit may be a columnar structure pillars containing point cloud data.

[0099] Specifically, a Cartesian coordinate system can be established, and the point cloud can be segmented according to a preset step size on the XY plane of the Cartesian coordinate system to obtain multiple point cloud units pillar containing point cloud data. The point cloud in each pillar is then encoded to obtain a 9-dimensional vector D (x, y, z, r, xc, yc, zc, xp, yp), where x, y, z are the three coordinates of the center point of the pillar in the coordinate system, r is the reflection intensity of the point cloud, xc, yc, zc correspond to the length, width, and height of the pillar column structure, and xp, yp are the offset values ​​from the point cloud to the x and y centers of the pillars of the column structure.

[0100] Due to the sparsity of point cloud data, there may be pillars that do not contain point clouds or contain a relatively small number of point clouds. In order to reduce the complexity of calculation, the number of pillars can be limited, that is, in each frame of point cloud data, at most P non-empty pillars are processed, and each pillar contains at most N point cloud feature vectors. When the number of point clouds contained in the pillar is greater than N, N point clouds can be selected from it by random sampling to obtain N point cloud feature vectors; when the number of point clouds contained in the pillar is less than N, the number of point clouds can be filled to N by zero filling (such as changing r). In this way, one frame of point cloud data can be encoded to obtain a dense tensor with a dimension of (D, P, N).

[0101] Then, through the PointNet network, a linear layer, Batch-Norm and ReLU activation functions are used to process each point containing the features of the vector D to generate a tensor of dimension (C, P, N), and then a maximum pooling operation is performed on each pillar unit to obtain a tensor of dimension (C, P).

[0102] Then, the scatter operator is used to decode the tensors of each dimension (C, P) to obtain the point cloud coordinates. That is, the tensor of dimension (C, P) is converted back to the original pillar coordinates (that is, the Cartesian coordinate system) through the pillar index value of each point to obtain the point cloud coordinates. Based on the point cloud coordinates, the first point cloud feature vector is created. The first point cloud feature vector is a pseudo image of size (C, H, W). Among them, regarding the size of H and W: when the point cloud is segmented to obtain multiple point cloud units, the range of the point cloud coordinates on the XY plane and the size of each pillar are generally set. Assuming that the range of the X-axis is [0, 0.69], the range of the Y-axis is [-39.68, 39.68], and the size of each pillar is 0.16*0.16, then, if the X-axis represents the width and the Y-axis represents the height, and one pillar represents the pixel of a pseudo image, then the width W of the pseudo image is (69.12-0) / 0.16=432, and the height H is (39.68-(-39.68)) / 0.16=496.

[0103] See also Figure 8 In some embodiments, step 012: performing feature processing on each first feature vector to obtain a second feature vector from a bird's eye view includes:

[0104] Step 0121: performing feature processing on the first image feature vector to obtain a second image feature vector corresponding to the first image feature vector from a bird's-eye view;

[0105] Step 0122: Perform feature processing on the first point cloud feature vector to obtain a second point cloud feature vector corresponding to the first point cloud feature vector from a bird's-eye view.

[0106] Specifically, the first image feature vector transmits semantic information, and the first point cloud feature vector transmits geometric information. Figure 1 In the example of the vehicle including 6 visual sensors (cameras), the first image feature vectors obtained after feature extraction from the images acquired by the 6 cameras are stitched together, and then the stitched multiple first image feature vectors are feature processed to project the first image feature vector to a bird's-eye view, thereby obtaining a corresponding second image feature vector. The first point cloud feature vector is then feature processed by, for example, coordinate transformation, to obtain a second point cloud feature vector corresponding to the first point cloud feature vector under a bird's-eye view.

[0107] It should be pointed out that, depending on the point cloud data, the generated first point cloud feature vector is different, and based on different first point cloud feature vectors, the second point cloud feature vectors generated after feature extraction are also different.

[0108] In the case where the point cloud data includes laser point cloud data, the first point cloud feature vector corresponding to the laser point cloud data is obtained, and the second point cloud feature vector corresponding to the laser point cloud data is obtained by performing feature extraction on the first point cloud feature vector corresponding to the laser point cloud data; in the case where the point cloud data includes millimeter wave point cloud data, the first point cloud feature vector corresponding to the millimeter wave point cloud data is obtained, and the second point cloud feature vector corresponding to the millimeter wave point cloud data is obtained by performing feature extraction on the first point cloud feature vector corresponding to the millimeter wave point cloud data; in the case where the point cloud data includes laser point cloud data and millimeter wave point cloud data, the first point cloud feature vector obtained includes the laser point cloud feature vector corresponding to the laser point cloud data and the millimeter wave point cloud feature vector corresponding to the millimeter wave point cloud data. The second point cloud feature vector is obtained by performing feature extraction on the laser point cloud feature vector and the millimeter wave point cloud feature vector respectively, and the second point cloud feature vector includes the second point cloud feature vector corresponding to the laser point cloud data and the second point cloud feature vector corresponding to the millimeter wave point cloud data.

[0109] See also Fig. 9 and Fig.10 In some implementations, step 0121: performing feature processing on the first image feature vector to obtain a second image feature vector corresponding to the first image feature vector from a bird's-eye view, includes:

[0110] Step 01211: Based on the intrinsic parameters and extrinsic parameters of each camera, project the first image feature vector corresponding to each camera to multiple preset heights to obtain image features at each preset height;

[0111] Step 01212: sampling the image features at each preset height corresponding to each camera to obtain the sampled image features corresponding to each camera;

[0112] Step 01213: Fusing the sampled image features corresponding to each camera to generate a third image feature vector;

[0113] Step 01214: generating a fourth image feature vector based on the first point cloud feature vector of the current frame and the second image feature vector of the previous frame, wherein the second image feature vector of the first frame is equal to the third image feature vector;

[0114] Step 01215: Generate a second image feature vector of the current frame based on the fourth image feature vector and the first image feature vector of the current frame.

[0115] The preset height may be determined based on the vertical height of objects around the vehicle in the scene in which the vehicle is located.

[0116] Specifically, the vehicle still includes 6 cameras, and any one of the 6 cameras is taken as an example. It represents the second image feature vector under the initial bird's-eye view of any camera, and i can be a time series, that is, the i-th frame.

[0117] The images captured by each camera are subjected to feature extraction to obtain a first image feature vector; then, according to the intrinsic parameters, extrinsic parameters and H preset heights (including (h1, h2, ..., hH)) of each camera, for example, refer to Fig.10 , where H = 4, the preset heights include height1, height2, height3 and height4. For simplicity, h1 represents height1, h2 represents height2, h3 represents height3, and h4 represents height4. The first image feature vector corresponding to each camera (such as in formula (1) ) is projected to four preset heights to obtain the image features at each preset height, i.e., ((i, j, h1), (i, j, h2), …, (i, j, h4)) in formula (1). As shown in formula (2), the maximum image feature of the camera at the four preset heights is sampled to obtain the sampled image features corresponding to each camera. As shown in formula (3), the sampled image features corresponding to each camera are Fusion (such as calculating the effective image features M in the 6 camera perspectives) hit The average value of the third image feature vector B i ,Right now:

[0118]

[0119] Among them, M is the number of cameras, that is, the number of viewing angles. In the embodiment of the present application, M=6, n is the number of preset heights, n=1, 2, ..., 4, m represents the mth viewing angle, m=1, 2, ..., 6.

[0120] In the first frame, the second image feature vector That is the third image feature vector B i , as shown in formula (4), the first point cloud feature vector B of the current frame is LiDAR and the second image feature vector B of the previous frame i (Right now ) to generate the fourth image feature vector The fourth image feature vector The first image feature vector of the current frame is fused to generate the second image feature vector B of the current frame. i , that is, for B i Perform iterative update. For example, after the above processing, the B of the second frame is 2 Perform iterative updates to obtain the B of the third frame 2 .

[0121]

[0122] in, is the BEV feature of the camera in the i-th layer, and f is the connection operation.

[0123] Please refer to formula (5). When fusion is performed, starting from the second frame, the first point cloud feature vector B of the current frame is LiDAR and the second image feature vector B of the previous frame i (Right now ) for fusion. For the first frame, the initial second image feature vector obtained by calculation based on formula (3) is used

[0124] See also Fig.10 and Fig.11 In some embodiments, step 01215: generating a second image feature vector of the current frame based on the fourth image feature vector and the first image feature vector of the current frame includes:

[0125] Step 01216: Based on the attention mechanism, the fourth image feature vector and the first image feature vector of the current frame are fused to generate the second image feature vector of the current frame.

[0126] Optionally, the attention mechanism may be a self-attention mechanism or a deformable attention mechanism. The following takes the deformable attention mechanism as an example for explanation.

[0127] Specifically, as shown in formula (6), based on the attention mechanism, at the i-th frame, the fourth image feature vector And the first image feature vector of the current frame is merged to calculate the fourth image feature vector The correlation between each pixel position in and each pixel position in the first image feature vector of the current frame is calculated, and based on these correlations, a second image feature vector B of the current frame is generated. i .

[0128]

[0129] In order to better perform fusion, the local area of ​​the image can be selected according to the information in the point cloud sequence, that is, the image generation under the BEV perspective can be controlled according to the point cloud data. Fig.12 Piont Embedding in) and image (i.e. Fig.12 The degree of association between different regions of the Image) in the current frame, that is, calculating the fourth image feature vector including the point cloud data The similarity scores of all positions of the first image feature vector V of the current frame are normalized and weighted summed to obtain the final image representation with conditional control (the second image feature vector B i ).

[0130] Therefore, please refer to Fig.12 , in the attention mechanism:

[0131] Attention(Q, K, V)

[0132] Among them, Q is the first image feature vector of the current frame, K and V are the fourth image feature vectors of the current frame In order to reduce the load of calculating similarity and matching, the vectors of the two modes should be equal in the last dimension, that is, assuming W Q The dimension is [c, d], W K and W V The dimensions of are all [d, d].

[0133] For each position of Q, it is necessary to calculate its degree of association with all positions in K, that is, Fig.12 QK T, to obtain the similarity matrix, and then normalize the similarity matrix to obtain the weight matrix to represent the degree of association between the image and each position of the point cloud feature vector K, and then multiply it with the point cloud feature vector V for weighted summation to obtain the second image feature vector processed based on the attention mechanism.

[0134] See also Fig.13 In some embodiments, step 0122: performing feature processing on the first point cloud feature vector to obtain a second point cloud feature vector corresponding to the first point cloud feature vector from a bird's eye view, includes:

[0135] Step 01221: Based on a preset conversion relationship between the world coordinate system and the point cloud coordinate system, convert the first point cloud feature vector into a second point cloud feature vector.

[0136] Among them, the preset conversion relationship is the mapping relationship between the point cloud coordinate system of the laser radar and the world coordinate system.

[0137] Specifically, according to a preset conversion relationship between the world coordinate system and the point cloud coordinate system, the first point cloud feature vector is converted into a second point cloud feature vector in the world coordinate system.

[0138] See also Fig.14 and Fig.15 In some embodiments, the multimodal data fusion processing method further includes:

[0139] Step 014: Based on the multi-layer perceptron, the fused feature vector is processed to output a plurality of processing results, wherein the processing results include at least one of a target detection result, a panoramic segmentation result, a lane line detection result and an environment detection result.

[0140] Specifically, by adding one or more hidden layers to the network, this architecture is usually called a multilayer perceptron (MLP). MLP can overcome the limitations of linear models and handle more general types of functional relationships. It stacks many fully connected layers together, and each layer outputs to the layer above until the final output is generated. Based on the multilayer perceptron, different fusion processes are performed on the fused feature vector to output multiple processing results corresponding to different needs, and the processing results include at least one of target detection results, panoramic segmentation results, lane detection results, and environmental detection results. Through the MLP layer, the results of tasks such as target detection tasks, panoramic segmentation tasks, lane detection tasks, and environmental detection tasks can be output simultaneously, and feature sharing can be achieved in the same network to save resources.

[0141] For example, see Fig.15, the fused feature vectors x[1], x[2] and x[3] are used as the input of MLP. After different fusion processes, two processing results including h[1] and h[2] are obtained. According to the two processing results h[1] and h[2], the vehicle is helped to perform target detection tasks, panoramic segmentation tasks, lane line detection tasks and environmental detection tasks, thereby saving resources and improving efficiency.

[0142] According to the method described in the above embodiment, the present application embodiment also provides a multimodal data fusion processing device, which is used to execute the steps in the multimodal data fusion processing method. Fig.16 , Fig.16 300 is a schematic diagram of the structure of a multimodal data fusion processing device 300 provided in an embodiment of the present application. The multimodal data fusion processing device 300 includes an extraction module 301, a processing module 302 and a fusion module 303, wherein:

[0143] The extraction module 301 is used to perform feature extraction on data of various modes respectively, so as to obtain first feature vectors corresponding to the data of various modes respectively;

[0144] The processing module 302 is used to perform feature processing on each first feature vector to obtain a second feature vector corresponding to each first feature vector from a bird's eye view;

[0145] The fusion module 303 is used to perform fusion processing on each second feature vector to generate a fused feature vector.

[0146] It should be noted that the specific details of each module unit in the above-mentioned multimodal data fusion processing device 300 have been described in detail in the embodiment of the above-mentioned multimodal data fusion processing method, and will not be repeated here.

[0147] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0148] In some embodiments, the multimodal data fusion processing device 300 in the embodiments of the present application can be implemented in hardware, such as an electronic device, or a component in an electronic device, such as an integrated circuit or a chip; the multimodal data fusion processing device can also be implemented in software, such as as an application installed in an electronic device.

[0149] The electronic device may be a terminal device or other device other than a terminal device. For example, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It may also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiments of the present application are not specifically limited.

[0150] In some embodiments, see Figure 1 , Figure 1 1 is a schematic diagram of the structure of the vehicle 100 provided in the embodiment of the present application. The vehicle 100 includes a processor 20, a memory 30, and a computer program stored in the memory 30 and executable on the processor 20. When the program is executed by the processor 20, each process of the embodiment of the fusion processing method of the multimodal data described above is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0151] See also Fig.17 The embodiment of the present application also provides a non-transitory computer-readable storage medium 500, on which a computer program 510 is stored. When the computer program 510 is executed by the processor 520, the various processes of the embodiment of the above-mentioned multimodal data fusion processing method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0152] The processor may be a processor in the electronic device in the above embodiment. The computer readable storage medium may be a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0153] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A fusion processing method for multimodal data, characterized in that: include: Performing feature extraction on data of various modes respectively to obtain first feature vectors corresponding to the data of various modes respectively; Performing feature processing on each of the first feature vectors respectively to obtain a second feature vector corresponding to each of the first feature vectors from a bird's-eye view; A fusion process is performed on each of the second feature vectors to generate a fused feature vector.

2. The multimodal data fusion processing method according to claim 1, characterized in that: The multimodal data includes image data and point cloud data, and the feature extraction is performed on the data of each modality to obtain the first feature vectors corresponding to the data of each modality, including: Performing feature extraction on the image data to obtain a first image feature vector corresponding to the image data; Feature extraction is performed on the point cloud data to obtain a first point cloud feature vector corresponding to the point cloud data.

3. The multimodal data fusion processing method according to claim 2, characterized in that: The extracting features from the image data to obtain a first image feature vector corresponding to the image data includes: Based on a preset residual network, feature extraction is performed on the image data to generate image feature vectors corresponding to a plurality of different image resolutions; Based on a preset feature graph pyramid network, a plurality of image feature vectors of different image resolutions are processed to generate a plurality of first image feature vectors of different scales.

4. The method for fusion processing of multimodal data according to claim 2, characterized in that: The extracting features of the point cloud data to obtain a point cloud feature vector corresponding to the point cloud data includes: Segmenting the point cloud data to obtain a plurality of point cloud units; Encoding the point clouds in each of the point cloud units to obtain tensors corresponding to each of the point cloud data; Decoding each of the tensors to obtain point cloud coordinates; Based on the point cloud coordinates, the first point cloud feature vector is generated.

5. The multimodal data fusion processing method according to claim 2, characterized in that: The performing feature processing on each of the first feature vectors to obtain a second feature vector from a bird's-eye view includes: Performing feature processing on the first image feature vector to obtain a second image feature vector corresponding to the first image feature vector from a bird's-eye view; Feature processing is performed on the first point cloud feature vector to obtain a second point cloud feature vector corresponding to the first point cloud feature vector from a bird's-eye view.

6. The method for fusion processing of multimodal data according to claim 5, characterized in that: The performing feature processing on the first image feature vector to obtain a second image feature vector corresponding to the first image feature vector from a bird's-eye view includes: Based on the intrinsic parameters and extrinsic parameters of each camera, projecting the first image feature vector corresponding to each camera to multiple preset heights to obtain image features at each preset height; Sampling the image features at each preset height corresponding to each camera to obtain the sampled image features corresponding to each camera; Fusing the sampled image features corresponding to each of the cameras to generate a third image feature vector; Generate a fourth image feature vector based on the first point cloud feature vector of the current frame and the second image feature vector of the previous frame, wherein the second image feature vector of the first frame is equal to the third image feature vector; The second image feature vector of the current frame is generated based on the fourth image feature vector and the first image feature vector of the current frame.

7. The method for fusion processing of multimodal data according to claim 6, characterized in that: The step of generating the second image feature vector of the current frame based on the fourth image feature vector and the first image feature vector of the current frame comprises: Based on the attention mechanism, the fourth image feature vector and the first image feature vector of the current frame are fused to generate the second image feature vector of the current frame.

8. The method for fusion processing of multimodal data according to claim 7, characterized in that: The attention mechanism is used to calculate the degree of association between the point cloud data and the image data.

9. The method for fusion processing of multimodal data according to claim 5, characterized in that: The performing feature processing on the first point cloud feature vector to obtain a second point cloud feature vector corresponding to the first point cloud feature vector from a bird's-eye view includes: Based on a preset conversion relationship between a world coordinate system and a point cloud coordinate system, the first point cloud feature vector is converted into the second point cloud feature vector.

10. The multimodal data fusion processing method according to claim 1, characterized in that: The method further comprises: Based on a multi-layer perceptron, the fused feature vector is processed to output a plurality of processing results, wherein the processing results include at least one of a target detection result, a panoramic segmentation result, a lane line detection result, and an environment detection result.

11. A vehicle comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multimodal data fusion processing method as described in any one of claims 1 to 10 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal data fusion processing method as described in any one of claims 1 to 10 is implemented.

Images