Road target detection method and system based on multi-source data fusion technology

Abstract

The invention discloses a road target detection method and system based on a multi-source data fusion technology, and belongs to the technical field of intelligent traffic. The credibility of different data sources in a complex environment is evaluated through a self-adaptive method, the weights of the different data sources are balanced, meanwhile, the features of the different data sources are further aligned through a feature matching method, high-quality fusion features are generated, and therefore the performance of a road target detection algorithm is improved; the detection accuracy and robustness are effectively improved by using a multi-modal data source adaptive fusion method.

Description

Background Art

[0002] With the acceleration of urbanization and the diversification of transportation modes, the field of smart transportation has received more and more attention. Road target detection algorithm is one of the key technologies to realize autonomous driving in smart transportation systems. Traditional road target detection algorithms mainly detect targets based on single sensor data, such as images or lidar sensors. However, single sensor data has its own limitations, such as image data is sensitive to lighting conditions, and lidar data has low resolution for targets. Therefore, how to use multi-source data fusion methods to improve the performance of road target detection algorithms is a problem worth studying.

[0003] Current work based on image and millimeter wave multi-source data fusion has made relatively strict assumptions about the data and fusion process:

[0004] (1) Data of the same modality have the same credibility in different environments.

[0005] (2) Different modes are strictly aligned after being mapped to the same coordinate.

[0006] However, the credibility of information provided by a specific modality in different environments is not the same. The credibility of information provided by a specific modality at different locations in the same environment will also vary due to the characteristics of the modal sensor. The data will be distorted in the coordinate system conversion, and the time difference between the sampling of sensors of different modalities will also cause positional offsets between modal features, which leads to alignment deviations after different modalities are mapped to the same coordinate system. The failure of the above two assumptions leads to poor multimodal fusion effects of these models in some extreme environments. For example: in rainy and snowy weather or when there is insufficient light at night, the credibility of image data will be reduced. If you rely too much on image data, the accuracy of target detection will decrease; in the blind area and occluded area of ​​the image, the image data cannot provide effective information at all. At this time, if target detection is performed based on image data, it will lead to missed detection or false detection. When the vehicle where the sensor is located moves at high speed, the data misalignment problem caused by the time difference between the sampling of different modal data will be further aggravated, which will cause ambiguity or even contradiction between the data of different modalities, thereby affecting the performance of the fusion model. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a road target detection method and system based on multi-source data fusion technology to address the deficiencies in the above-mentioned prior art, so as to solve the technical problem of decreased accuracy and robustness of road target detection caused by environmental changes and sensor differences.

[0008] The present invention adopts the following technical solutions:

[0009] A road target detection method based on multi-source data fusion technology comprises the following steps:

[0010] The deep learning method is used to extract features from image data and millimeter-wave radar point cloud data to obtain a bird's-eye view image feature map and a millimeter-wave point cloud feature map;

[0011] Conduct a trustworthy evaluation on the bird's-eye view image feature map and the millimeter-wave point cloud feature map, and adaptively adjust the weights of different data sources according to the evaluation results to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map;

[0012] Input the balanced bird's-eye view image feature map, millimeter-wave point cloud feature map and sensor metadata into the feature matching network, output two feature offset maps of the same size as the feature map, use the feature offset map as the deformation offset of the deformable convolutional neural network, use the deformable convolutional neural network to align the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map, input the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map into the fusion network, and output a fused feature map of the same size as the feature map;

[0013] Use the anchor-free target detection method to perform target detection on the fused feature map and generate the target detection result.

[0014] Preferably, the image data and millimeter wave radar point cloud data are specifically:

[0015] Calibrate the millimeter-wave radar and camera to obtain the internal and external parameters of the sensors: the camera's internal parameter matrix, distortion coefficient, external parameter matrix relative to the reference sensor, the millimeter-wave radar's internal parameter matrix, and the external parameter matrix relative to the reference sensor;

[0016] Use multiple cameras and millimeter-wave radars to simultaneously collect image data and millimeter-wave radar data;

[0017] Performing distortion correction on image data according to the distortion coefficient of the camera;

[0018] Use constant false alarm rate algorithm to denoise millimeter wave radar data and extract point cloud;

[0019] Annotate 3D target data, including the target's position, size, rotation angle, and speed information.

[0020] Preferably, the bird's-eye view image feature map and the millimeter wave point cloud feature map are obtained specifically as follows:

[0021] The perspective view features of image data are extracted using a deep learning-based method, where the input is image data collected by multiple cameras and the output is the features of the image data under the perspective view;

[0022] Using a deep learning-based monocular depth estimation method, a pixel depth distribution probability map is generated according to the extracted image data features;

[0023] The image data is converted into a frustum space using the image pixel depth distribution probability map;

[0024] Using the GPU acceleration method, project downward from the bird's-eye view, accumulate the spatial features to the corresponding pixel position of the bird's-eye view feature map corresponding to the downward projection, and convert the perspective image data into the bird's-eye view image data;

[0025] A deep learning-based method is used to extract the features of millimeter-wave point cloud data, and finally the point cloud features within the grid are averaged to obtain the millimeter-wave point cloud features at the corresponding position of the bird's-eye view feature map corresponding to the grid;

[0026] The bird's-eye view feature map is input into a multi-level two-dimensional convolutional network to obtain feature maps of different scales. The feature maps of all scales are upsampled to the maximum scale and then spliced ​​to obtain the final millimeter-wave point cloud feature map.

[0027] Preferably, the image feature vector of a position corresponding to a pixel is multiplied with each component of the pixel depth distribution probability vector of the corresponding position to obtain the spatial feature of each position.

[0028] Preferably, the trustworthy assessment is specifically:

[0029] Generate the position code from the bird's-eye view, which consists of four channels: sine and cosine of the angle with the center, distance to the center, and square of the distance to the center;

[0030] Generate a millimeter wave point cloud distribution mask based on the millimeter wave point cloud data, and output a mask map of the same size as the data feature map. The mask value is between 0 and 1, indicating the distribution density of the millimeter wave point cloud data at that location. When the number of point clouds in the cylindrical space corresponding to the location reaches the threshold, the mask reaches the maximum value of 1. When the number of point clouds in the cylindrical space corresponding to the location is 0, the mask is 0.

[0031] Accumulate the image pixel distribution probability map along the direction of the pixel ray to obtain an approximate image pixel occlusion probability distribution map;

[0032] The position code, millimeter wave point cloud distribution mask, and millimeter wave point cloud feature map are input into the millimeter wave modal evaluation network, and a millimeter wave modal evaluation vector map of the same size as the feature map is output. Specifically, a two-dimensional convolutional network is used to process it;

[0033] Input the image pixel occlusion probability distribution map and the bird's-eye view image feature map into the image modality assessment network, and output an image modality assessment vector map of the same size as the feature map;

[0034] Input the millimeter wave point cloud feature map and the bird's-eye view image feature map into the fusion network, and output an environmental semantic evaluation vector map of the same size as the feature map;

[0035] The evaluation vector map is input into the cross-modal evaluation network, and a final evaluation vector map with the same size as the feature map and 2 channels is output;

[0036] Use the SoftMax function to normalize the final evaluation vector graph to obtain the final weight vector;

[0037] The feature credibility of the bird's-eye view image feature map and the millimeter-wave point cloud feature map is balanced according to the weight vector, and the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map are output.

[0038] Preferably, a self-attention mechanism is used to process it, and the evaluation vector graph is used as the three vectors of Query, Key, and Value in the self-attention mechanism.

[0039] Preferably, the feature alignment is specifically:

[0040] The generated position codes, millimeter wave point cloud distribution masks, and image pixel occlusion probability distribution maps are stitched together to obtain sensor metadata;

[0041] The balanced bird's-eye view image feature map, millimeter wave point cloud feature map and sensor metadata are input into the feature matching network. By comparing the features of different data sources, the offset of the features of different data sources is estimated, and two feature offset maps of the same size as the feature map are output.

[0042] A deformable convolutional neural network is used to align the balanced bird's-eye view image feature map, where the deformation offset used by the deformable convolutional neural network is the feature offset map, and finally the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are output;

[0043] The aligned bird's-eye view image feature map and millimeter wave point cloud feature map are input into the fusion network, and a high-quality fused feature map of the same size as the feature map is output. Specifically, a two-dimensional convolutional network is used to process it.

[0044] Preferably, the generated balanced bird's-eye view image feature map, millimeter wave point cloud feature map and sensor metadata are input into the feature matching network, and a common convolutional neural network is used to fuse and reduce the dimension of the spliced ​​data; then the fused data is input into the convolutional neural network of two branches, and two feature offset maps of the same size as the feature map are output respectively.

[0045] Preferably, target detection is specifically:

[0046] The generated fused feature map is input into the shared dimension reduction convolutional neural network, and a feature map of the same size as the feature map is output;

[0047] The generated feature maps are input into target detection branches of different categories. The input includes the target center position heat map from a bird's-eye view, the target three-dimensional size prediction result, the target rotation angle prediction result, the target speed prediction result, and the target position offset prediction result.

[0048] In a second aspect, an embodiment of the present invention provides a road target detection system based on multi-source data fusion technology, comprising:

[0049] The extraction module uses deep learning methods to extract features from image data and millimeter-wave radar point cloud data to obtain a bird's-eye view image feature map and a millimeter-wave point cloud feature map;

[0050] An evaluation module performs a credible evaluation on the bird's-eye view image feature map and the millimeter-wave point cloud feature map, and adaptively adjusts the weights of different data sources according to the evaluation results to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map;

[0051] An alignment module inputs the balanced bird's-eye view image feature map, the millimeter-wave point cloud feature map, and the sensor metadata into a feature matching network, outputs two feature offset maps of the same size as the feature maps, uses the feature offset maps as deformation offsets of a deformable convolutional neural network, uses the deformable convolutional neural network to perform feature alignment on the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map, inputs the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map into a fusion network, and outputs a fused feature map of the same size as the feature map;

[0052] The detection module uses an anchor-free target detection method to perform target detection on the fused feature map and generate target detection results.

[0053] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned road target detection method based on multi-source data fusion technology when executing the computer program.

[0054] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned road target detection method based on multi-source data fusion technology.

[0055] Compared with the prior art, the present invention has at least the following beneficial effects:

[0056] A road target detection method based on multi-source data fusion technology uses multiple data sensors, utilizes a deep learning method to extract features, evaluates the credibility of different data sources through an attention mechanism, aligns features of different data sources through feature matching, and finally performs target detection on the fused data through a target detection network. The present invention effectively improves the accuracy and robustness of road target detection and has good application prospects.

[0057] Furthermore, the present invention uses millimeter wave radar and camera sensors, which are relatively cheap and provide complementary environmental information. The two sensor data complement each other and ultimately achieve multimodal fusion target detection. The sensor used in this article can improve the accuracy and robustness of road target detection without increasing the cost (purchasing a high-precision lidar sensor), and has good practicality.

[0058] Furthermore, the present invention uses a feature extraction network designed for millimeter wave radar data and image data to extract the image features from a bird's eye view and the features of the millimeter wave radar point cloud data from a bird's eye view. The former is obtained by predicting the pixel depth distribution probability map and using it as a basis for coordinate transformation of the image features, while the latter is obtained by extracting features from the millimeter wave point cloud after spatial region sampling and mapping it to a bird's eye view. The bird's eye view perspective feature can effectively reduce data representation ambiguity and occlusion problems, and improve the accuracy and robustness of road target detection.

[0059] Furthermore, the present invention uses a multimodal trust evaluation and feature adaptive balancing method based on the attention mechanism to evaluate the credibility of different data sources and adaptively adjust the weights of different data sources according to the evaluation results. Different from the traditional method of using training techniques, the present invention uses a special network structure to evaluate the credibility of different data sources, so that the model can dynamically adjust its dependence on different modalities when the environment changes, thereby reducing the interference of weak modal data on fused data and improving the accuracy and robustness of road target detection. Trust evaluation and feature balance of multiple data sources have the advantage of strong adaptability to complex environments.

[0060] Furthermore, the present invention uses a multimodal feature alignment and fusion method based on feature matching to align and fuse features from different data sources to generate high-quality fusion features. Different from the traditional direct fusion method, the present invention uses a feature matching method to estimate the offset of features from different data sources, aligns features from different data sources according to the offset, eliminates data feature alignment errors caused by data coordinate system conversion and data sampling time difference, and improves the accuracy and robustness of road target detection; multimodal feature alignment and fusion has the advantage of being less affected by multimodal sensor alignment errors.

[0061] Furthermore, the present invention uses an anchor-free target detection method and uses the fused data to detect road targets. Different from the traditional anchor-based target detection method, the present invention uses an anchor-free target detection method that does not require pre-setting anchor points and can better adapt to targets of different scales and shapes, thereby improving the accuracy and robustness of road target detection. The anchor-free target detection module has better adaptability and generalization ability, and can effectively generalize the model to targets of various sizes and shapes.

[0062] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0063] In summary, the technical solution of the present invention can effectively adapt to complex road environments and adaptively eliminate the alignment error problem between sensors of different models.

[0064] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Design diagram for the method of the present invention;

[0066] Figure 2 Design diagram for multimodal trustworthy assessment and feature adaptive balancing module;

[0067] Figure 3 Design diagram for the transmembrane spatial attention module;

[0068] Figure 4 Visualize the result graph of multimodal adaptive weights;

[0069] Figure 5 Design diagram for multimodal feature alignment and fusion module;

[0070] Figure 6 Schematic diagram of multimodal feature alignment offset;

[0071] Figure 7 This is the modal feature map alignment algorithm diagram;

[0072] Figure 8 This is the network structure diagram of anchor-free target detection from a bird's-eye view;

[0073] Fig. 9 A schematic diagram of a computer device provided by an embodiment of the present invention;

[0074] Fig.10 The present invention is a block diagram of an electronic device provided according to an embodiment.

[0075] Fig.11This is a performance comparison chart of the present invention and other models.

[0076] Fig.12 It is a confusion matrix diagram of the prediction results of the present invention on different categories.

[0077] Fig.13 This is a rendering of the present invention for detecting road targets. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0080] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0081] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0082] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0083] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0084] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0085] The present invention provides a road target detection method based on multi-source data fusion technology, and a multimodal trust evaluation and feature adaptive alignment algorithm based on an attention mechanism. The algorithm evaluates the credibility of different modalities from three aspects: modal features, modal data characteristics, and inter-modal relationships. It uses trust evaluation weights to balance the credibility of features of different modalities, so that the model can dynamically adjust its dependence on different modalities when the environment changes, thereby reducing the interference of weak modal data on fused data; a multimodal data alignment fusion algorithm based on feature matching. The algorithm generates feature alignment offsets by matching features of different modalities, and then performs alignment correction on features of different modalities, eliminating data feature alignment errors caused by data coordinate system conversion and data sampling time difference.

[0086] The present invention provides a road target detection method based on multi-source data fusion technology, comprising the following steps:

[0087] S1, obtaining image data and millimeter wave radar point cloud data;

[0088] Preprocess the collected image data and millimeter-wave radar point cloud data, and annotate the original data.

[0089] The image data includes internal and external parameters of the measurement sensor, and the internal and external parameters include the camera's internal parameter matrix, distortion coefficient, the camera's external parameter matrix, the millimeter-wave radar's internal parameter matrix, and the millimeter-wave radar's external parameter matrix.

[0090] Preprocessing includes distortion correction of image data, denoising of millimeter-wave radar data, point cloud extraction, and point cloud format conversion.

[0091] S101, calibrate the millimeter wave radar and the camera to obtain the internal and external parameters of the sensors: the internal parameter matrix of the camera, the distortion coefficient, the external parameter matrix relative to the reference sensor, the internal parameter matrix of the millimeter wave radar, and the external parameter matrix relative to the reference sensor;

[0092] S102, using multiple cameras and millimeter-wave radars to simultaneously collect image data and millimeter-wave radar data;

[0093] S103, performing distortion correction on the image data according to the distortion coefficient of the camera;

[0094] S104, using a constant false alarm rate algorithm to denoise the millimeter wave radar data and extract a point cloud;

[0095] S105 , annotating 3D target data, including information such as the position, size, rotation angle, and speed of the target.

[0096] S2. Using a deep learning method to perform feature extraction on the image data and millimeter-wave radar point cloud data obtained in step S1, to obtain an image feature map and a millimeter-wave point cloud feature map from a bird's-eye view perspective;

[0097] The deep learning method is used to extract features from image data and millimeter-wave radar point cloud data, generate a pixel depth distribution probability map of the image data, and convert the perspective image data into a bird's-eye view image data.

[0098] S201, extracting perspective view features of image data using a deep learning-based method, where the input is image data collected by multiple cameras, and the output is features of the image data under perspective view;

[0099] S202, using a monocular depth estimation method based on deep learning to generate a pixel depth distribution probability map according to the image data features extracted in step S201;

[0100] S203, using the image pixel depth distribution probability map in step S202 to convert the image data in step S101 into a frustum space, specifically by multiplying the image feature vector at a position corresponding to a pixel with each component of the pixel depth distribution probability vector at the corresponding position to obtain the spatial features of each position;

[0101] S204, using a GPU acceleration method to project downward from the bird's-eye view to convert the image data from the perspective view into image data from the bird's-eye view, wherein the specific method used is to accumulate the spatial features in step S203 to the corresponding pixel position of the bird's-eye view feature map corresponding to the downward projection;

[0102] S205, dividing the space into grids from a bird's-eye view perspective, and independently extracting features from the millimeter-wave point cloud in each grid, specifically using a deep learning-based method to extract features of the millimeter-wave point cloud data, and finally averaging the point cloud features in the grid to obtain the millimeter-wave point cloud features at the corresponding position of the bird's-eye view feature map corresponding to the grid;

[0103] S206. Use a two-dimensional convolutional network to further process the millimeter-wave point cloud features from a bird's-eye view perspective. Specifically, input the bird's-eye view perspective feature map in step S205 into a multi-level two-dimensional convolutional network to obtain feature maps of different scales, upsample the feature maps of all scales to the maximum scale, and then splice them to obtain the final millimeter-wave point cloud feature map.

[0104] S3, performing a credibility assessment on the image features and millimeter wave radar features under the bird's-eye view obtained in step S2, and adaptively adjusting the weights of different data sources according to the assessment results;

[0105] Specifically, a preliminary evaluation vector is generated based on data features, environmental data, and sensor metadata information, the preliminary evaluation vector is fused using a cross-modal evaluation network to generate a final evaluation vector, and the weights of different data sources are adaptively adjusted based on the final evaluation vector.

[0106] S301, generating a position code from a bird's-eye view perspective, specifically four channels, namely: sine and cosine values ​​of the angle with the center, distance to the center, and square of the distance to the center;

[0107] S302, generate a millimeter wave point cloud distribution mask based on the millimeter wave point cloud data, and output a mask map of the same size as the data feature map, with a mask value between 0 and 1, indicating the distribution density of the millimeter wave point cloud data at that location. When the number of point clouds in the cylindrical space corresponding to the location reaches a threshold, the mask reaches a maximum value of 1, and when the number of point clouds in the cylindrical space corresponding to the location is 0, the mask is 0;

[0108] S303, accumulating the image pixel distribution probability map along the direction of the pixel ray to obtain an approximate image pixel occlusion probability distribution map;

[0109] S304: Input the position code generated in step S301, the millimeter wave point cloud distribution mask generated in step S302, and the millimeter wave point cloud feature map generated in step S206 into the millimeter wave modal evaluation network, and output a millimeter wave modal evaluation vector map of the same size as the feature map. Specifically, a two-dimensional convolutional network is used to process it;

[0110] S305, input the image pixel occlusion probability distribution map generated in step S303 and the bird's-eye view image feature map generated in step S204 into the image modality assessment network, and output an image modality assessment vector map of the same size as the feature map. Specifically, a two-dimensional convolutional network is used to process it;

[0111] S306, inputting the millimeter wave point cloud feature map generated in step S206 and the bird's-eye view image feature map generated in step S204 into the fusion network, and outputting an environmental semantic evaluation vector map of the same size as the feature map; specifically, using a two-dimensional convolutional network to process it;

[0112] S307, input the evaluation vector graph generated in step S304, step S305, and step S306 into the cross-modal evaluation network, and output a final evaluation vector graph of the same size as the feature graph and with a channel number of 2. Specifically, the self-attention mechanism is used to process it, and the evaluation vector graphs generated in S304, S305, and S306 are used as the three vectors of Query, Key, and Value in the self-attention mechanism.

[0113] S308, using the SoftMax function to normalize the final evaluation vector graph generated in step S307 to obtain a final weight vector;

[0114] S309, performing feature credibility balance on the bird's-eye view image feature map generated in step S204 and the millimeter-wave point cloud feature map generated in step S206 according to the weight vector generated in step S308, and outputting the balanced bird's-eye view image feature map and millimeter-wave point cloud feature map.

[0115] S4, aligning and fusing the features of data from different modalities based on the weights obtained in step S3;

[0116] The feature matching method is used to estimate the offset of features from different data sources, and the features of different data sources are aligned according to the offset. The aligned features are fused using a fusion network to generate high-quality fused features.

[0117] S401, stitching together the position codes, millimeter wave point cloud distribution masks, and image pixel occlusion probability distribution maps generated in steps S301, S302, and S303 to obtain sensor metadata;

[0118] S402, inputting the balanced bird's-eye view image feature map generated in step S309, the millimeter-wave point cloud feature map, and the sensor metadata generated in step S401 into a feature matching network, estimating the offset of the features of different data sources by comparing the features of different data sources, and outputting two feature offset maps of the same size as the feature map;

[0119] Specifically:

[0120] First, a common convolutional neural network is used to fuse and reduce the dimension of the spliced ​​data;

[0121] The fused data is then input into the two-branch convolutional neural network, which outputs two feature offset maps of the same size as the feature map.

[0122] S403, using a deformable convolutional neural network to perform feature alignment on the balanced bird's-eye view image feature map generated in step S309, wherein the deformation offset used by the deformable convolutional neural network is the feature offset map generated in step S402, and finally outputting the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map;

[0123] S404: Input the aligned bird's-eye view image feature map and the millimeter wave point cloud feature map generated in step S403 into a fusion network, and output a high-quality fused feature map of the same size as the feature map. Specifically, a two-dimensional convolutional network is used to process it.

[0124] S5, using the data fused in step S4 to perform road target detection;

[0125] Use the anchor-free target detection method to perform target detection on the data fused in step S4 to generate a target detection result.

[0126] S501, input the fused feature map generated in step S404 into a shared dimensionality reduction convolutional neural network, and output a feature map of the same size as the feature map;

[0127] S502, input the feature map generated in step S501 into target detection branches of different categories, the input includes the target center position heat map from the bird's-eye view, the target three-dimensional size prediction result, the target rotation angle prediction result, the target speed prediction result, and the target position offset prediction result; specifically, use a two-dimensional convolutional network to process it.

[0128] S6. Generate target detection neural network training labels based on the 3D target data marked in step S1.

[0129] S601, generating a target center position heat map training label according to the labeled data;

[0130] The specific method is to generate a Gaussian distribution heat map from a bird's-eye view with the 3D target annotation position as the center. The size of each 3D target heat map is proportional to the size of the target. The format is a two-dimensional map from a bird's-eye view. The meaning of each pixel data in the map is: the probability that there is a target center in the 3D space cylinder corresponding to the pixel.

[0131] S602, generate training targets of target regression data according to the labeled data, including target three-dimensional size, target rotation angle, target speed, and target position offset information. The format is the same as S601, which is a two-dimensional image from a bird's-eye view. The meaning of each pixel data in the image is: if there is a target center in the 3D space cylinder, the pixel corresponds to the specific information of the target;

[0132] S7. Based on the images collected in S1, millimeter wave point cloud data and the neural network training labels generated in S6, train the feature extraction network in S2, the feature map trust evaluation network in S3, the feature alignment fusion network in S4, and the target detection network in S5.

[0133] The original data collected in S1 is input into S2 to extract features, the feature map output by S2 is used as the input of the S3 trusted evaluation network, the balanced feature map output by S3 is used as the input of the S4 feature alignment fusion network, and the fused feature map output by S4 is used as the input of the S5 target detection network. GaussianFocalLoss and SmoothL1Loss are used to calculate the loss gradients of the two prediction data respectively and back-propagate to update the network parameters.

[0134] S701, using the GaussianFocalLoss loss function, the loss is calculated according to the target center position heat map training label generated in step S601 and the target center position heat map prediction result generated in step S502;

[0135] S702, using the SmoothL1Loss loss function, calculating the loss according to the target information regression training label generated in step S602 and the target regression data prediction result generated in step S502;

[0136] S703, using GradNorm to perform gradient clipping on the losses generated in step S701 and step S702, and balancing the loss gradients corresponding to the target position detection task and the target information prediction task;

[0137] S704: Use the AdamW optimizer to optimize the loss generated in step S703 and update the network parameters.

[0138] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0139] In another embodiment of the present invention, a road target detection system based on multi-source data fusion technology is provided, which can be used to implement the above-mentioned road target detection method based on multi-source data fusion technology. Specifically, the road target detection system based on multi-source data fusion technology includes an extraction module, an evaluation module, an alignment module and a detection module.

[0140] Among them, the extraction module uses a deep learning method to extract features from image data and millimeter-wave radar point cloud data to obtain a bird's-eye view image feature map and a millimeter-wave point cloud feature map;

[0141] An evaluation module performs a credible evaluation on the bird's-eye view image feature map and the millimeter-wave point cloud feature map, and adaptively adjusts the weights of different data sources according to the evaluation results to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map;

[0142] An alignment module inputs the balanced bird's-eye view image feature map, the millimeter-wave point cloud feature map, and the sensor metadata into a feature matching network, outputs two feature offset maps of the same size as the feature maps, uses the feature offset maps as deformation offsets of a deformable convolutional neural network, uses the deformable convolutional neural network to perform feature alignment on the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map, inputs the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map into a fusion network, and outputs a fused feature map of the same size as the feature map;

[0143] The detection module uses an anchor-free target detection method to perform target detection on the fused feature map and generate target detection results.

[0144] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the road target detection method based on multi-source data fusion technology, including:

[0145] A deep learning method is used to extract features from image data and millimeter-wave radar point cloud data to obtain a bird's-eye view image feature map and a millimeter-wave point cloud feature map; a credible assessment is performed on the bird's-eye view image feature map and the millimeter-wave point cloud feature map, and the weights of different data sources are adaptively adjusted according to the assessment results to obtain a balanced bird's-eye view image feature map and a millimeter-wave point cloud feature map; the balanced bird's-eye view image feature map, the millimeter-wave point cloud feature map and the sensor metadata are input into a feature matching network, and two feature offset maps of the same size as the feature maps are output. The feature offset maps are used as deformation offsets of a deformable convolutional neural network, and the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map are feature aligned using a deformable convolutional neural network. The aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are input into a fusion network, and a fused feature map of the same size as the feature map is output; an anchor-free target detection method is used to perform target detection on the fused feature map to generate a target detection result.

[0146] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0147] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0148] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0149] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the road target detection method based on the multi-source data fusion technology in the above embodiment; the processor may load and execute the following steps:

[0150] A deep learning method is used to extract features from image data and millimeter-wave radar point cloud data to obtain a bird's-eye view image feature map and a millimeter-wave point cloud feature map; a credible assessment is performed on the bird's-eye view image feature map and the millimeter-wave point cloud feature map, and the weights of different data sources are adaptively adjusted according to the assessment results to obtain a balanced bird's-eye view image feature map and a millimeter-wave point cloud feature map; the balanced bird's-eye view image feature map, the millimeter-wave point cloud feature map and the sensor metadata are input into a feature matching network, and two feature offset maps of the same size as the feature maps are output. The feature offset maps are used as deformation offsets of a deformable convolutional neural network, and the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map are feature aligned using a deformable convolutional neural network. The aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are input into a fusion network, and a fused feature map of the same size as the feature map is output; an anchor-free target detection method is used to perform target detection on the fused feature map to generate a target detection result.

[0151] See also Fig. 9 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the method for calculating the composition of fluid in the reservoir transformation wellbore in the embodiment is realized. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the calculation system for the composition of fluid in the reservoir transformation wellbore in the embodiment are realized. To avoid repetition, it is not described one by one here.

[0152] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Fig. 9 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0153] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, central processing units, graphics processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, data processing logic based on quantum computing, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0154] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0155] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0156] Any reference to memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0157] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0158] See also Fig.10 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0159] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0160] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0161] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0162] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0163] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0164] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0165] Factors that affect the credibility of sensor data and how to deal with them:

[0166] Other targets or obstacles block the target in the image, making it invisible in the image but visible in the millimeter-wave radar data. In this case, the credibility of the image data is low, while the credibility of the millimeter-wave radar data is high. Accumulate the predicted pixel depth distribution probability map along the direction of the pixel ray to obtain an approximate image pixel occlusion probability distribution map, which is used as the input of the image modality evaluation network to evaluate the credibility of the image data;

[0167] The credibility of millimeter-wave radar data is uneven due to the distribution of pixel point clouds. The distribution mask of millimeter-wave radar data and the feature map of millimeter-wave radar data are used as the input of the millimeter-wave modal evaluation network to evaluate the credibility of millimeter-wave radar data;

[0168] The angular resolution of millimeter-wave radar at different angles is different, resulting in different credibility of millimeter-wave radar data at different angles. The position code is used as the input of the millimeter-wave modal evaluation network to evaluate the credibility of the millimeter-wave radar data;

[0169] The changes in ambient lighting lead to instability in the credibility of image data. The image features are used as the input of the image modality evaluation network to evaluate the credibility of the image data.

[0170] Factors that may cause misalignment of multi-source data features:

[0171] The data features are not aligned due to inaccurate internal and external parameters of the sensor.

[0172] The data features are not aligned due to the asynchronous sampling of sensors.

[0173] The data accuracy is reduced due to the aggregation of sensor data frames (such as aggregating the data of multiple frames of millimeter-wave radar into one frame to increase the number of millimeter-wave point clouds in a single frame).

[0174] Solution for multi-source data feature alignment:

[0175] Generate position codes, concatenate the position codes, millimeter-wave point cloud distribution masks, and image pixel occlusion probability distribution maps to obtain sensor metadata, input the sensor metadata and the balanced bird's-eye view image feature map and millimeter-wave point cloud feature map into the feature matching network, compare the features of different data sources, estimate the offsets of the features of different data sources, and output two feature offset maps of the same size as the feature map;

[0176] A deformable convolutional neural network is used to align the balanced bird's-eye view image feature map, where the deformation offset used by the deformable convolutional neural network is the feature offset map, and the aligned bird's-eye view image feature map and millimeter-wave point cloud feature map are output.

[0177] In summary, the present invention provides a road target detection method and system based on multi-source data fusion technology, which effectively adapts to complex road environments, dynamically adjusts the weights of different data sources to adapt to environmental changes, and adaptively eliminates the alignment error problem between sensors of different models. It can effectively reduce the problem of decreased accuracy and robustness of road target detection caused by environmental changes and sensor differences.

[0178] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0179] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0180] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0181] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A road target detection method based on multi-source data fusion technology, characterized in that: The following steps are involved: The deep learning method is used to extract features from image data and millimeter-wave radar point cloud data to obtain a bird's-eye view image feature map and a millimeter-wave point cloud feature map; Conduct a trustworthy evaluation on the bird's-eye view image feature map and the millimeter-wave point cloud feature map, and adaptively adjust the weights of different data sources according to the evaluation results to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map; Input the balanced bird's-eye view image feature map, millimeter-wave point cloud feature map and sensor metadata into the feature matching network, output two feature offset maps of the same size as the feature map, use the feature offset map as the deformation offset of the deformable convolutional neural network, use the deformable convolutional neural network to align the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map, input the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map into the fusion network, and output a fused feature map of the same size as the feature map; Use the anchor-free target detection method to perform target detection on the fused feature map and generate the target detection result.

2. The road target detection method based on multi-source data fusion technology according to claim 1 is characterized in that: The image data and millimeter wave radar point cloud data are as follows: Calibrate the millimeter-wave radar and camera to obtain the internal and external parameters of the sensors: the camera's internal parameter matrix, distortion coefficient, external parameter matrix relative to the reference sensor, the millimeter-wave radar's internal parameter matrix, and the external parameter matrix relative to the reference sensor; Use multiple cameras and millimeter-wave radars to simultaneously collect image data and millimeter-wave radar data; Performing distortion correction on image data according to the distortion coefficient of the camera; Use constant false alarm rate algorithm to denoise millimeter wave radar data and extract point cloud; Annotate 3D target data, including the target's position, size, rotation angle, and speed information.

3. The road target detection method based on multi-source data fusion technology according to claim 1 is characterized in that: The feature map of the bird's-eye view image and the feature map of the millimeter-wave point cloud are obtained as follows: The perspective view features of image data are extracted using a deep learning-based method, where the input is image data collected by multiple cameras and the output is the features of the image data under the perspective view; Using a deep learning-based monocular depth estimation method, a pixel depth distribution probability map is generated according to the extracted image data features; The image data is converted into a frustum space using the image pixel depth distribution probability map; Using the GPU acceleration method, project downward from the bird's-eye view, accumulate the spatial features to the corresponding pixel position of the bird's-eye view feature map corresponding to the downward projection, and convert the perspective image data into the bird's-eye view image data; A deep learning-based method is used to extract the features of millimeter-wave point cloud data, and finally the point cloud features within the grid are averaged to obtain the millimeter-wave point cloud features at the corresponding position of the bird's-eye view feature map corresponding to the grid; The bird's-eye view feature map is input into a multi-level two-dimensional convolutional network to obtain feature maps of different scales. The feature maps of all scales are upsampled to the maximum scale and then spliced ​​to obtain the final millimeter-wave point cloud feature map.

4. The road target detection method based on multi-source data fusion technology according to claim 1 is characterized in that: The image data is converted into the frustum space using the image pixel depth distribution probability map as follows: The image feature vector of the position corresponding to a pixel and each component of the pixel depth distribution probability vector of the corresponding position are multiplied to obtain the spatial features of each position.

5. The road target detection method based on multi-source data fusion technology according to claim 1 is characterized in that: Credibility assessments include: Generate the position code from the bird's-eye view, which consists of four channels: sine and cosine of the angle with the center, distance to the center, and square of the distance to the center; Generate a millimeter wave point cloud distribution mask based on the millimeter wave point cloud data, and output a mask map of the same size as the data feature map. The mask value is between 0 and 1, indicating the distribution density of the millimeter wave point cloud data at that location. When the number of point clouds in the cylindrical space corresponding to the location reaches the threshold, the mask reaches the maximum value of 1. When the number of point clouds in the cylindrical space corresponding to the location is 0, the mask is 0. Accumulate the image pixel distribution probability map along the direction of the pixel ray to obtain an approximate image pixel occlusion probability distribution map; The position code, the millimeter wave point cloud distribution mask, and the millimeter wave point cloud feature map are input into the millimeter wave modal evaluation network, and a millimeter wave modal evaluation vector map of the same size as the feature map is output; specifically, a two-dimensional convolutional network is used to process it; Input the image pixel occlusion probability distribution map and the bird's-eye view image feature map into the image modality assessment network, and output an image modality assessment vector map of the same size as the feature map; Input the millimeter wave point cloud feature map and the bird's-eye view image feature map into the fusion network, and output an environmental semantic evaluation vector map of the same size as the feature map; Input the evaluation vector map into the cross-modal evaluation network and output a final evaluation vector map with the same size as the feature map and 2 channels; Use the SoftMax function to normalize the final evaluation vector graph to obtain the final weight vector; The feature credibility of the bird's-eye view image feature map and the millimeter-wave point cloud feature map is balanced according to the weight vector, and the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map are output.

6. The road target detection method based on multi-source data fusion technology according to claim 5 is characterized in that: Use the self-attention mechanism to process it, and use the evaluation vector graph as the three vectors of Query, Key, and Value in the self-attention mechanism.

7. The road target detection method based on multi-source data fusion technology according to claim 1 is characterized in that: Feature alignment is as follows: The generated position codes, millimeter wave point cloud distribution masks, and image pixel occlusion probability distribution maps are stitched together to obtain sensor metadata; The balanced bird's-eye view image feature map, millimeter wave point cloud feature map and sensor metadata are input into the feature matching network. By comparing the features of different data sources, the offset of the features of different data sources is estimated, and two feature offset maps of the same size as the feature map are output. A deformable convolutional neural network is used to align the balanced bird's-eye view image feature map, where the deformation offset used by the deformable convolutional neural network is the feature offset map, and finally the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are output; The aligned bird's-eye view image feature map and millimeter-wave point cloud feature map are input into the fusion network, and a high-quality fused feature map of the same size as the feature map is output; specifically, a two-dimensional convolutional network is used to process it.

8. The road target detection method based on multi-source data fusion technology according to claim 7 is characterized in that: The generated balanced bird's-eye view image feature map, millimeter wave point cloud feature map and sensor metadata are input into the feature matching network, and a common convolutional neural network is used to fuse and reduce the dimension of the spliced ​​data; the fused data is then input into the two-branch convolutional neural network, which output two feature offset maps of the same size as the feature map.

9. The road target detection method based on multi-source data fusion technology according to claim 1 is characterized in that: Target detection is specifically: The generated fused feature map is input into the shared dimension reduction convolutional neural network, and a feature map of the same size as the feature map is output; The generated feature maps are input into target detection branches of different categories. The input includes the target center position heat map from a bird's-eye view, the target three-dimensional size prediction result, the target rotation angle prediction result, the target speed prediction result, and the target position offset prediction result.

10. A road target detection system based on multi-source data fusion technology, characterized in that: include: The extraction module uses deep learning methods to extract features from image data and millimeter-wave radar point cloud data to obtain a bird's-eye view image feature map and a millimeter-wave point cloud feature map; An evaluation module performs a credible evaluation on the bird's-eye view image feature map and the millimeter-wave point cloud feature map, and adaptively adjusts the weights of different data sources according to the evaluation results to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map; An alignment module inputs the balanced bird's-eye view image feature map, the millimeter-wave point cloud feature map, and the sensor metadata into a feature matching network, outputs two feature offset maps of the same size as the feature maps, uses the feature offset maps as deformation offsets of a deformable convolutional neural network, uses the deformable convolutional neural network to perform feature alignment on the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map, inputs the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map into a fusion network, and outputs a fused feature map of the same size as the feature map; The detection module uses an anchor-free target detection method to perform target detection on the fused feature map and generate target detection results.

Images