A Road Target Detection Method and System Based on Multi-Source Data Fusion Technology

Abstract

This invention discloses a road target detection method and system based on multi-source data fusion technology, belonging to the field of intelligent transportation technology. It uses deep learning methods to fuse images with millimeter-wave radar point cloud data, and uses an adaptive method to evaluate the credibility of different data sources in complex environments and balance the weights of different data sources. At the same time, it further uses feature matching methods to align the features of different data sources to generate high-quality fused features, thereby improving the performance of the road target detection algorithm. The adaptive fusion method of multi-modal data sources effectively improves the accuracy and robustness of detection.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, specifically relating to a road target detection method and system based on multi-source data fusion technology. Background Technology

[0002] With the acceleration of urbanization and the diversification of transportation modes, intelligent transportation has received increasing attention. Road target detection algorithms are one of the key technologies for realizing autonomous driving in intelligent transportation systems. Traditional road target detection algorithms mainly rely on single sensor data for target detection, such as images or LiDAR sensors. However, single sensor data has its own limitations; for example, image data is sensitive to lighting conditions, while LiDAR data has low target resolution. Therefore, how to improve the performance of road target detection algorithms by utilizing multi-source data fusion methods is a problem worthy of research.

[0003] Current work on the fusion of image and millimeter-wave multi-source data makes relatively strict assumptions about the data and the fusion process:

[0004] (1) Data of the same modality has the same reliability in different environments.

[0005] (2) Different modes are strictly aligned after being mapped to the same coordinate system.

[0006] However, the reliability of information provided by specific modalities varies under different environments, and even within the same environment, the reliability of information provided by specific modalities at different locations will differ due to the characteristics of the modal sensors. Positional distortion occurs during coordinate system transformation, and the time difference in sampling between different modal sensors also leads to positional shifts between modal features. This results in alignment deviations after different modalities are mapped to the same coordinate system. The failure of the above two assumptions causes these models to perform poorly in certain extreme environments. For example, in rainy or snowy weather or at night with insufficient lighting, the reliability of image data decreases. Over-reliance on image data can lead to a decline in the accuracy of target detection. In blind spots and occluded areas of the image, image data provides no effective information, and relying on image data for target detection can result in missed or false detections. When the vehicle carrying the sensors moves at high speed, the data misalignment caused by the time difference in sampling between different modal data is further exacerbated. This can lead to ambiguity or even contradictions between data from different modalities, thus affecting the performance of the fusion model. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a road target detection method and system based on multi-source data fusion technology to address the shortcomings of the prior art, thereby solving the technical problem of decreased accuracy and robustness of road target detection caused by environmental changes and sensor differences.

[0008] The present invention adopts the following technical solution:

[0009] A road target detection method based on multi-source data fusion technology includes the following steps:

[0010] Deep learning methods were used to extract features from image data and millimeter-wave radar point cloud data to obtain bird's-eye view image feature maps and millimeter-wave point cloud feature maps.

[0011] A reliability assessment is performed on the feature maps of bird's-eye view images and millimeter-wave point cloud features. Based on the assessment results, the weights of different data sources are adaptively adjusted to obtain balanced feature maps of bird's-eye view images and millimeter-wave point cloud features.

[0012] The balanced bird's-eye view image feature map, millimeter-wave point cloud feature map, and sensor metadata are input into the feature matching network, which outputs two feature offset maps of the same size as the feature map. The feature offset maps are used as the deformation offset of the deformable convolutional neural network. The deformable convolutional neural network is used to align the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map. The aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are input into the fusion network, which outputs a fused feature map of the same size as the feature map.

[0013] An anchorless object detection method is used to perform object detection on the fused feature map and generate object detection results.

[0014] Preferably, the image data and millimeter-wave radar point cloud data are specifically:

[0015] The millimeter-wave radar and camera are calibrated to obtain the intrinsic and extrinsic parameters of the sensors: the intrinsic parameter matrix of the camera, the distortion coefficient, the extrinsic parameter matrix relative to the reference sensor, the intrinsic parameter matrix of the millimeter-wave radar, and the extrinsic parameter matrix relative to the reference sensor.

[0016] Simultaneously acquire image data and millimeter-wave radar data using multiple cameras and millimeter-wave radar;

[0017] The image data is distorted based on the camera's distortion coefficients;

[0018] The constant false alarm rate algorithm is used to denoise millimeter-wave radar data and extract point clouds;

[0019] Label 3D target data, including the target's position, size, rotation angle, and velocity information.

[0020] Preferably, the bird's-eye view image feature map and millimeter-wave point cloud feature map are obtained as follows:

[0021] This method uses deep learning to extract perspective features from image data. The input is image data acquired by multiple cameras, and the output is the features of the image data under the perspective view.

[0022] A pixel depth distribution probability map is generated based on the extracted image data features using a deep learning-based monocular depth estimation method.

[0023] Image data is transformed into a truncated cone space using the image pixel depth distribution probability map;

[0024] Using GPU acceleration, the image data is projected downwards from the bird's-eye view, and the spatial features are accumulated onto the corresponding pixel positions of the bird's-eye view feature map corresponding to the downward projection, thus converting the perspective view image data into the bird's-eye view image data.

[0025] The features of millimeter-wave point cloud data are extracted using a deep learning-based method. Finally, the point cloud features within the grid are averaged to obtain the millimeter-wave point cloud features at the corresponding location of the bird's-eye view feature map for that grid.

[0026] The bird's-eye view feature map is input into a multi-level two-dimensional convolutional network to obtain feature maps of different scales. After upsampling all feature maps of all scales to the maximum scale, they are stitched together to obtain the final millimeter-wave point cloud feature map.

[0027] Preferably, the spatial features of each location are obtained by multiplying the image feature vector of the location corresponding to a pixel with each component of the pixel depth distribution probability vector of the corresponding location.

[0028] Preferably, the reliability assessment specifically includes:

[0029] The location code is generated from the bird's-eye view, specifically in four channels: the sine and cosine of the angle with the center, the distance to the center, and the square of the distance to the center.

[0030] A millimeter-wave point cloud distribution mask is generated based on the millimeter-wave point cloud data. A mask map of the same size as the data feature map is output. The mask value is between 0 and 1, which represents the distribution density of the millimeter-wave point cloud data at that location. When the number of point clouds in the cylindrical space corresponding to the location reaches the threshold, the mask reaches the maximum value of 1. When the number of point clouds in the cylindrical space corresponding to the location is 0, the mask is 0.

[0031] By summing the probability distribution maps of image pixels along the direction of pixel rays, an approximate probability distribution map of image pixel occlusion is obtained.

[0032] The location encoding, millimeter-wave point cloud distribution mask, and millimeter-wave point cloud feature map are input into the millimeter-wave mode evaluation network, which outputs a millimeter-wave mode evaluation vector map of the same size as the feature map. Specifically, a two-dimensional convolutional network is used to process it.

[0033] The image pixel occlusion probability distribution map and the bird's-eye view image feature map are input into the image modality evaluation network, and an image modality evaluation vector map of the same size as the feature map is output.

[0034] The millimeter-wave point cloud feature map and the bird's-eye view image feature map are input into the fusion network, and an environmental semantic evaluation vector map of the same size as the feature map is output.

[0035] The evaluation vector map is input into the cross-modal evaluation network, and a final evaluation vector map with the same size as the feature map and 2 channels is output.

[0036] The final evaluation vector graph is normalized using the SoftMax function to obtain the final weight vector.

[0037] Based on the weight vector, the feature credibility of the bird's-eye view image feature map and the millimeter-wave point cloud feature map is balanced, and the balanced bird's-eye view image feature map and millimeter-wave point cloud feature map are output.

[0038] Preferably, a self-attention mechanism is used to process it, and the evaluation vector graph is used as the three vectors Query, Key, and Value in the self-attention mechanism.

[0039] Preferably, feature alignment specifically involves:

[0040] The generated location code, millimeter-wave point cloud distribution mask, and image pixel occlusion probability distribution map are stitched together to obtain sensor metadata.

[0041] The balanced bird's-eye view feature map, millimeter-wave point cloud feature map, and sensor metadata are input into the feature matching network. By comparing the features from different data sources, the offset of the features from different data sources is estimated, and two feature offset maps of the same size as the feature map are output.

[0042] A deformable convolutional neural network is used to align the features of the balanced bird's-eye view image. The deformable offset used by the deformable convolutional neural network is the feature offset map. Finally, the aligned bird's-eye view image feature map and millimeter-wave point cloud feature map are output.

[0043] The aligned bird's-eye view feature map and millimeter-wave point cloud feature map are input into the fusion network, which outputs a high-quality fused feature map of the same size as the original feature map. Specifically, a two-dimensional convolutional network is used to process it.

[0044] Preferably, the generated balanced bird's-eye view feature map, millimeter-wave point cloud feature map, and sensor metadata are input into a feature matching network, and a shared convolutional neural network is used to fuse and reduce the dimensions of the stitched data; then the fused data is input into two branches of convolutional neural networks, which output two feature offset maps of the same size as the feature maps.

[0045] Preferably, target detection specifically includes:

[0046] The generated fused feature map is input into a shared dimensionality-reducing convolutional neural network, which outputs a feature map of the same size as the original feature map.

[0047] The generated feature maps are input into different categories of target detection branches. The inputs include the target center position heatmap from a bird's-eye view, the target's 3D size prediction, the target's rotation angle prediction, the target's velocity prediction, and the target's position offset prediction.

[0048] Secondly, embodiments of the present invention provide a road target detection system based on multi-source data fusion technology, comprising:

[0049] The extraction module uses deep learning methods to extract features from image data and millimeter-wave radar point cloud data, resulting in bird's-eye view image feature map and millimeter-wave point cloud feature map.

[0050] The evaluation module performs a reliability evaluation on the feature maps of the bird's-eye view image and the millimeter-wave point cloud feature map. Based on the evaluation results, it adaptively adjusts the weights of different data sources to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map.

[0051] The alignment module inputs the balanced bird's-eye view image feature map, millimeter-wave point cloud feature map, and sensor metadata into the feature matching network, and outputs two feature offset maps of the same size as the feature map. The feature offset maps are used as the deformation offset of the deformable convolutional neural network. The deformable convolutional neural network is used to align the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map. The aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are input into the fusion network, and output a fused feature map of the same size as the feature map.

[0052] The detection module uses an anchorless target detection method to perform target detection on the fused feature map and generate target detection results.

[0053] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the road target detection method based on multi-source data fusion technology described above.

[0054] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the road target detection method based on multi-source data fusion technology described above.

[0055] Compared with the prior art, the present invention has at least the following beneficial effects:

[0056] A road target detection method based on multi-source data fusion technology is proposed. This method uses multiple data sensors, extracts features using deep learning, evaluates the credibility of different data sources through an attention mechanism, aligns features from different data sources through feature matching, and finally performs target detection on the fused data through a target detection network. This invention effectively improves the accuracy and robustness of road target detection and has good application prospects.

[0057] Furthermore, this invention utilizes millimeter-wave radar and camera sensors, both of which are relatively inexpensive and provide complementary environmental information. The data from these two sensors complement each other, ultimately achieving multimodal fusion target detection. The sensors used in this paper can improve the accuracy and robustness of road target detection without increasing costs (by purchasing high-precision lidar sensors), demonstrating good practicality.

[0058] Furthermore, this invention employs a feature extraction network specifically designed for millimeter-wave radar data and image data to extract image features from a bird's-eye view and features from millimeter-wave radar point cloud data from a bird's-eye view. The former is obtained by predicting the pixel depth distribution probability map and performing coordinate transformation on the image features based on this map, while the latter is obtained by extracting features from the millimeter-wave point cloud after spatial region sampling and mapping them to the bird's-eye view. Bird's-eye view features can effectively reduce data representation ambiguity and occlusion problems, improving the accuracy and robustness of road target detection.

[0059] Furthermore, this invention employs a multimodal reliability assessment and feature adaptive balancing method based on an attention mechanism to evaluate the reliability of different data sources and adaptively adjust the weights of different data sources based on the evaluation results. Unlike traditional methods that utilize training techniques, this invention uses a specialized network structure to evaluate the reliability of different data sources, enabling the model to dynamically adjust its dependence on different modalities when the environment changes. This reduces the interference of weak modal data on the fused data, improving the accuracy and robustness of road target detection. The multi-data source reliability assessment and feature balancing method also offers the advantage of strong adaptability to complex environments.

[0060] Furthermore, this invention employs a multimodal feature alignment and fusion method based on feature matching to align and fuse features from different data sources, generating high-quality fused features. Unlike traditional direct fusion methods, this invention uses feature matching to estimate the offset of features from different data sources and aligns the features based on the offset, eliminating data feature alignment errors caused by data coordinate system transformation and data sampling time differences, thus improving the accuracy and robustness of road target detection. Multimodal feature alignment and fusion also has the advantage of being less affected by alignment errors from multimodal sensors.

[0061] Furthermore, this invention employs an anchor-free target detection method, utilizing fused data for road target detection. Unlike traditional anchor-based target detection methods, this invention eliminates the need for pre-defined anchor points, allowing for better adaptation to targets of varying scales and shapes, thus improving the accuracy and robustness of road target detection. The anchor-free target detection module exhibits superior adaptability and generalization capabilities, effectively generalizing the model to targets of diverse sizes and shapes.

[0062] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0063] In summary, the technical solution of the present invention can effectively adapt to complex road environments and adaptively eliminate alignment error problems between different types of sensors.

[0064] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0065] Figure 1 This is a design diagram of the method of the present invention;

[0066] Figure 2 Design diagram for multimodal reliable evaluation and feature adaptive balancing module;

[0067] Figure 3 Design diagram of the attention module across membrane space;

[0068] Figure 4 A visualization of the multimodal adaptive weights;

[0069] Figure 5 Design diagram for the multimodal feature alignment and fusion module;

[0070] Figure 6 This is a schematic diagram of multimodal feature alignment offset.

[0071] Figure 7 Diagram of the modal feature map alignment algorithm;

[0072] Figure 8 A diagram of the network structure for anchorless target detection from a bird's-eye view.

[0073] Figure 9 A schematic diagram of a computer device provided in an embodiment of the present invention;

[0074] Figure 10 This is a block diagram of an electronic device according to an embodiment of the present invention.

[0075] Figure 11 This is a performance comparison chart of the present invention with other models.

[0076] Figure 12 This is a confusion matrix diagram showing the prediction results of this invention across different categories.

[0077] Figure 13 This is a diagram illustrating the effect of the present invention in detecting road targets. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0080] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0081] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0082] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0083] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0084] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0085] This invention provides a road target detection method based on multi-source data fusion technology, and a multimodal reliability assessment and adaptive feature alignment algorithm based on an attention mechanism. This algorithm evaluates the reliability of different modalities from three aspects: modal features, the characteristics of the modal data itself, and the relationships between modalities. It uses reliability assessment weights to balance the reliability of features from different modalities, enabling the model to dynamically adjust its dependence on different modalities when the environment changes, thereby reducing the interference of weak modal data on the fused data. The invention also includes a multimodal data alignment and fusion algorithm based on feature matching. This algorithm generates feature alignment offsets by matching features from different modalities, and then corrects the alignment of features from different modalities, eliminating data feature alignment errors caused by data coordinate system transformation and data sampling time differences.

[0086] This invention discloses a road target detection method based on multi-source data fusion technology, comprising the following steps:

[0087] S1. Acquire image data and millimeter-wave radar point cloud data;

[0088] The acquired image data and millimeter-wave radar point cloud data are preprocessed, and the original data is labeled.

[0089] Image data includes the intrinsic and extrinsic parameters of the measurement sensors, including the camera's intrinsic parameter matrix, distortion coefficients, camera's extrinsic parameter matrix, millimeter-wave radar's intrinsic parameter matrix, and millimeter-wave radar's extrinsic parameter matrix.

[0090] Preprocessing includes distortion correction of image data, denoising of millimeter-wave radar data, point cloud extraction, and point cloud format conversion.

[0091] S101. Calibrate the millimeter-wave radar and camera to obtain the intrinsic and extrinsic parameters of the sensors: the intrinsic parameter matrix of the camera, the distortion coefficient, the extrinsic parameter matrix relative to the reference sensor, the intrinsic parameter matrix of the millimeter-wave radar, and the extrinsic parameter matrix relative to the reference sensor.

[0092] S102. Simultaneously acquire image data and millimeter-wave radar data using multiple cameras and millimeter-wave radar.

[0093] S103. Perform distortion correction on the image data based on the camera's distortion coefficient;

[0094] S104. Use the constant false alarm rate algorithm to denoise millimeter-wave radar data and extract point cloud.

[0095] S105. Label 3D target data, including information such as the target's position, size, rotation angle, and speed.

[0096] S2. Use deep learning methods to extract features from the image data and millimeter-wave radar point cloud data obtained in step S1 to obtain image feature maps and millimeter-wave point cloud feature maps from a bird's-eye view.

[0097] Deep learning methods are used to extract features from image data and millimeter-wave radar point cloud data, generating a pixel depth distribution probability map of the image data, thus converting the perspective image data into a bird's-eye view image data.

[0098] S201. Use a deep learning-based method to extract perspective features from image data. The input is image data collected by multiple cameras, and the output is the features of the image data under the perspective view.

[0099] S202. Using a deep learning-based monocular depth estimation method, a pixel depth distribution probability map is generated based on the image data features extracted in step S201.

[0100] S203. Using the image pixel depth distribution probability map from step S202, the image data from step S101 is converted into a truncated cone space. Specifically, the image feature vector at the location corresponding to a pixel is multiplied by each component of the pixel depth distribution probability vector at the corresponding location to obtain the spatial features at each location.

[0101] S204. Using GPU acceleration, project the perspective view downwards from the bird's-eye view to convert the perspective view image data into the bird's-eye view image data. Specifically, the spatial features in step S203 are accumulated to the corresponding pixel positions of the bird's-eye view feature map corresponding to the downward projection.

[0102] S205. Divide the space into grids from the bird's-eye view perspective, and extract features of the millimeter-wave point cloud in each grid independently. Specifically, use a deep learning-based method to extract features of the millimeter-wave point cloud data. Finally, average the point cloud features in the grid to obtain the millimeter-wave point cloud features at the corresponding position of the bird's-eye view feature map of the grid.

[0103] S206. Further process the millimeter-wave point cloud features from the bird's-eye view using a two-dimensional convolutional network. Specifically, input the bird's-eye view feature map from step S205 into a multi-level two-dimensional convolutional network to obtain feature maps of different scales. After upsampling all feature maps to the maximum scale, stitch them together to obtain the final millimeter-wave point cloud feature map.

[0104] S3. Perform a credibility assessment on the image features and millimeter-wave radar features obtained from the bird's-eye view in step S2, and adaptively adjust the weights of different data sources based on the assessment results.

[0105] Specifically, the process involves generating a preliminary evaluation vector based on data features, environmental data, and sensor metadata; fusing the preliminary evaluation vector using a cross-modal evaluation network to generate a final evaluation vector; and adaptively adjusting the weights of different data sources based on the final evaluation vector.

[0106] S301. Generate the position code from the bird's-eye view perspective, specifically four channels: the sine and cosine of the angle with the center, the distance to the center, and the square of the distance to the center.

[0107] S302. Generate a millimeter-wave point cloud distribution mask based on the millimeter-wave point cloud data, and output a mask image of the same size as the data feature map. The mask value is a value between 0 and 1, representing the distribution density of the millimeter-wave point cloud data at that location. When the number of point clouds in the cylindrical space corresponding to the location reaches a threshold, the mask reaches its maximum value of 1; when the number of point clouds in the cylindrical space corresponding to the location is 0, the mask is 0.

[0108] S303. Accumulate the image pixel distribution probability map along the direction of the pixel ray to obtain an approximate image pixel occlusion probability distribution map;

[0109] S304. Input the position encoding generated in step S301, the millimeter-wave point cloud distribution mask generated in step S302, and the millimeter-wave point cloud feature map generated in step S206 into the millimeter-wave mode evaluation network, and output a millimeter-wave mode evaluation vector map of the same size as the feature map. Specifically, a two-dimensional convolutional network is used to process it;

[0110] S305. Input the image pixel occlusion probability distribution map generated in step S303 and the bird's-eye view feature map generated in step S204 into the image modality evaluation network, and output an image modality evaluation vector map of the same size as the feature map. Specifically, a two-dimensional convolutional network is used to process it;

[0111] S306. Input the millimeter-wave point cloud feature map generated in step S206 and the bird's-eye view image feature map generated in step S204 into the fusion network, and output an environmental semantic evaluation vector map of the same size as the feature map; specifically, a two-dimensional convolutional network is used to process it;

[0112] S307. Input the evaluation vector map generated in steps S304, S305, and S306 into the cross-modal evaluation network, and output a final evaluation vector map of the same size as the feature map with 2 channels. Specifically, a self-attention mechanism is used to process it, and the evaluation vector map generated in S304, S305, and S306 is used as the Query, Key, and Value vectors in the self-attention mechanism.

[0113] S308. Use the SoftMax function to normalize the final evaluation vector map generated in step S307 to obtain the final weight vector.

[0114] S309. Based on the weight vector generated in step S308, balance the feature credibility of the bird's-eye view image feature map generated in step S204 and the millimeter-wave point cloud feature map generated in step S206, and output the balanced bird's-eye view image feature map and millimeter-wave point cloud feature map.

[0115] S4. Align and fuse the features of different modal data based on the weights obtained in step S3;

[0116] The feature matching method is used to estimate the offset of features from different data sources. The features from different data sources are aligned according to the offset. The aligned features are then fused using a fusion network to generate high-quality fused features.

[0117] S401. The location code, millimeter-wave point cloud distribution mask, and image pixel occlusion probability distribution map generated in steps S301, S302, and S303 are stitched together to obtain sensor metadata.

[0118] S402. Input the balanced bird's-eye view feature map, millimeter-wave point cloud feature map generated in step S309 and the sensor metadata generated in step S401 into the feature matching network. By comparing the features of different data sources, estimate the offset of the features of different data sources and output two feature offset maps of the same size as the feature map.

[0119] Specifically:

[0120] First, a shared convolutional neural network is used to fuse and reduce the dimensionality of the concatenated data;

[0121] The fused data is then input into two branches of a convolutional neural network, which output two feature offset maps of the same size as the feature maps.

[0122] S403. Use a deformable convolutional neural network to perform feature alignment on the balanced bird's-eye view image feature map generated in step S309. The deformable offset used by the deformable convolutional neural network is the feature offset map generated in step S402. Finally, output the aligned bird's-eye view image feature map and millimeter-wave point cloud feature map.

[0123] S404. Input the aligned bird's-eye view feature map and millimeter-wave point cloud feature map generated in step S403 into the fusion network, and output a high-quality fused feature map of the same size as the feature maps. Specifically, a two-dimensional convolutional network is used to process it.

[0124] S5. Use the data fused in step S4 to perform road target detection;

[0125] Anchor-free target detection method is used to perform target detection on the data fused in step S4, generating target detection results.

[0126] S501. Input the fused feature map generated in step S404 into the shared dimensionality reduction convolutional neural network and output a feature map of the same size as the feature map.

[0127] S502. Input the feature map generated in step S501 into different categories of target detection branches. The input includes the target center position heatmap from the bird's-eye view, the target 3D size prediction result, the target rotation angle prediction result, the target velocity prediction result, and the target position offset prediction result; specifically, a two-dimensional convolutional network is used to process it.

[0128] S6. Generate training labels for the target detection neural network based on the 3D target data annotated in step S1.

[0129] S601. Generate a heat map training label for the target center location based on the labeled data;

[0130] The specific method involves generating a Gaussian distribution heatmap from a bird's-eye view, centered on the 3D target's labeled location. The size of each 3D target heatmap is proportional to the size of the target. The format is a two-dimensional image from a bird's-eye view, where each pixel represents the probability that a target center exists within the corresponding 3D spatial cylinder.

[0131] S602. Generate training targets for target regression data based on the labeled data, including the target's 3D size, rotation angle, velocity, and position offset. The format is the same as S601: a 2D image from a bird's-eye view. The meaning of each pixel in the image is: if a target exists at the center of a 3D cylinder, the pixel represents the specific information of that target.

[0132] S7. Based on the images and millimeter-wave point cloud data acquired in S1 and the neural network training labels generated in S6, train the feature extraction network in S2, the feature map confidence evaluation network in S3, the feature alignment and fusion network in S4, and the target detection network in S5.

[0133] The raw data collected in S1 is input into S2 to extract features. The feature map output from S2 is used as the input to the confidence evaluation network in S3. The balanced feature map output from S3 is used as the input to the feature alignment and fusion network in S4. The fused feature map output from S4 is used as the input to the object detection network in S5. Gaussian FocalLoss and SmoothL1Loss are used to calculate the loss gradients for the two types of prediction data, respectively, and the network parameters are updated by backpropagation.

[0134] S701. Using the Gaussian Focal Loss function, calculate the loss based on the target center location heatmap generated in step S601 (training labels) and the target center location heatmap prediction results generated in step S502.

[0135] S702. Using the SmoothL1Loss loss function, calculate the loss based on the target information generated in step S602 to regress the training labels and the target regression data prediction results generated in step S502.

[0136] S703. Use GradNorm to perform gradient clipping on the loss generated in steps S701 and S702 to balance the loss gradients corresponding to the target location detection task and the target information prediction task.

[0137] S704. Optimize the loss generated in step S703 using the AdamW optimizer and update the network parameters.

[0138] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0139] In another embodiment of the present invention, a road target detection system based on multi-source data fusion technology is provided. This system can be used to implement the above-mentioned road target detection method based on multi-source data fusion technology. Specifically, the road target detection system based on multi-source data fusion technology includes an extraction module, an evaluation module, an alignment module, and a detection module.

[0140] The extraction module uses deep learning methods to extract features from image data and millimeter-wave radar point cloud data, resulting in bird's-eye view image feature map and millimeter-wave point cloud feature map.

[0141] The evaluation module performs a reliability evaluation on the feature maps of the bird's-eye view image and the millimeter-wave point cloud feature map. Based on the evaluation results, it adaptively adjusts the weights of different data sources to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map.

[0142] The alignment module inputs the balanced bird's-eye view image feature map, millimeter-wave point cloud feature map, and sensor metadata into the feature matching network, and outputs two feature offset maps of the same size as the feature map. The feature offset maps are used as the deformation offset of the deformable convolutional neural network. The deformable convolutional neural network is used to align the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map. The aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are input into the fusion network, and output a fused feature map of the same size as the feature map.

[0143] The detection module uses an anchorless target detection method to perform target detection on the fused feature map and generate target detection results.

[0144] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, the computer program including program instructions, and the processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a road target detection method based on multi-source data fusion technology, including:

[0145] Feature extraction from image data and millimeter-wave radar point cloud data using deep learning methods yields bird's-eye view image feature maps and millimeter-wave point cloud feature maps. A reliability assessment is performed on these feature maps, and the weights of different data sources are adaptively adjusted based on the assessment results to obtain balanced bird's-eye view image feature maps and millimeter-wave point cloud feature maps. These balanced feature maps, along with sensor metadata, are input into a feature matching network, which outputs two feature offset maps of the same size as the feature maps. These feature offset maps are used as deformation offsets in a deformable convolutional neural network (CNN), which aligns the balanced bird's-eye view image feature maps and millimeter-wave point cloud feature maps. The aligned bird's-eye view image feature maps and millimeter-wave point cloud feature maps are then input into a fusion network, which outputs a fused feature map of the same size as the feature maps. Finally, an anchorless target detection method is used to perform target detection on the fused feature map, generating target detection results.

[0146] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that more specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0148] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0149] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the road target detection method based on multi-source data fusion technology in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0150] Feature extraction from image data and millimeter-wave radar point cloud data using deep learning methods yields bird's-eye view image feature maps and millimeter-wave point cloud feature maps. A reliability assessment is performed on these feature maps, and the weights of different data sources are adaptively adjusted based on the assessment results to obtain balanced bird's-eye view image feature maps and millimeter-wave point cloud feature maps. These balanced feature maps, along with sensor metadata, are input into a feature matching network, which outputs two feature offset maps of the same size as the feature maps. These feature offset maps are used as deformation offsets in a deformable convolutional neural network (CNN), which aligns the balanced bird's-eye view image feature maps and millimeter-wave point cloud feature maps. The aligned bird's-eye view image feature maps and millimeter-wave point cloud feature maps are then input into a fusion network, which outputs a fused feature map of the same size as the feature maps. Finally, an anchorless target detection method is used to perform target detection on the fused feature map, generating target detection results.

[0151] Please see Figure 9 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the fluid composition calculation method in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the fluid composition calculation system in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here.

[0152] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 9 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0153] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, CPUs, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, quantum computing-based data processing logic units, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0154] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0155] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0156] Any references to memory, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0157] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0158] Please see Figure 10 The terminal device 600 is an electronic device, which takes the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0159] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0160] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0161] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0162] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0163] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0164] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0165] Factors affecting the reliability of sensor data and their handling methods:

[0166] Other targets or obstacles may occlude the target in the image, making it invisible in the image but visible in the millimeter-wave radar data. In this case, the reliability of the image data is lower, while the reliability of the millimeter-wave radar data is higher. The predicted pixel depth distribution probability map is accumulated along the direction of the pixel ray to obtain an approximate image pixel occlusion probability distribution map, which is used as the input to the image modality evaluation network to evaluate the reliability of the image data.

[0167] The uneven distribution of pixel point clouds leads to inconsistent reliability of millimeter-wave radar data. The distribution mask and feature map of millimeter-wave radar data are used as inputs to a millimeter-wave mode evaluation network to assess the reliability of the millimeter-wave radar data.

[0168] The angular resolution of millimeter-wave radar varies at different angles, leading to different data reliability. Position encoding is used as input to a millimeter-wave mode evaluation network to assess the reliability of the millimeter-wave radar data.

[0169] Variations in ambient lighting lead to instability in the reliability of image data. Image features are used as input to an image modality evaluation network to assess the reliability of the image data.

[0170] Factors that may lead to feature misalignment in multi-source data:

[0171] Inaccurate intrinsic and extrinsic parameters of the sensor lead to misalignment of data features.

[0172] Data feature misalignment caused by asynchronous sensor sampling.

[0173] The aggregation of sensor data frames leads to a decrease in data accuracy (such as aggregating data from multiple frames of millimeter-wave radar into one frame to increase the number of millimeter-wave point clouds in a single frame).

[0174] Solution for multi-source data feature alignment:

[0175] A location code is generated. The location code, millimeter-wave point cloud distribution mask, and image pixel occlusion probability distribution map are stitched together to obtain sensor metadata. The sensor metadata, the balanced bird's-eye view image feature map, and the millimeter-wave point cloud feature map are input into the feature matching network. By comparing the features of different data sources, the offset of the features of different data sources is estimated, and two feature offset maps of the same size as the feature map are output.

[0176] A deformable convolutional neural network is used to align the features of the balanced bird's-eye view image. The deformable offset used by the deformable convolutional neural network is the feature offset map. The output is the aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map.

[0177] In summary, the road target detection method and system based on multi-source data fusion technology of this invention effectively adapts to complex road environments, dynamically adjusts the weights of different data sources to adapt to environmental changes, and adaptively eliminates alignment errors between different sensor models. It can effectively reduce the decline in the accuracy and robustness of road target detection caused by environmental changes and sensor differences.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0181] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0184] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0185] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A road target detection method based on multi-source data fusion technology, characterized in that, Includes the following steps: Deep learning methods were used to extract features from image data and millimeter-wave radar point cloud data to obtain bird's-eye view image feature maps and millimeter-wave point cloud feature maps. A credibility assessment is performed on the feature maps of the bird's-eye view image and the millimeter-wave point cloud image. Based on the assessment results, the weights of different data sources are adaptively adjusted to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map. The credibility assessment is specifically as follows: The location code is generated from the bird's-eye view, specifically in four channels: the sine and cosine of the angle with the center, the distance to the center, and the square of the distance to the center. A millimeter-wave point cloud distribution mask is generated based on the millimeter-wave point cloud data. A mask map of the same size as the data feature map is output. The mask value is between 0 and 1, which represents the distribution density of the millimeter-wave point cloud data at that location. When the number of point clouds in the cylindrical space corresponding to the location reaches the threshold, the mask reaches the maximum value of 1. When the number of point clouds in the cylindrical space corresponding to the location is 0, the mask is 0. By summing the probability distribution maps of image pixels along the direction of pixel rays, an approximate probability distribution map of image pixel occlusion is obtained. The location encoding, millimeter-wave point cloud distribution mask, and millimeter-wave point cloud feature map are input into the millimeter-wave mode evaluation network, and a millimeter-wave mode evaluation vector map of the same size as the feature map is output; specifically, a two-dimensional convolutional network is used to process it. The image pixel occlusion probability distribution map and the bird's-eye view image feature map are input into the image modality evaluation network, and an image modality evaluation vector map of the same size as the feature map is output. The millimeter-wave point cloud feature map and the bird's-eye view image feature map are input into the fusion network, and an environmental semantic evaluation vector map of the same size as the feature map is output. The evaluation vector map is input into the cross-modal evaluation network, and a final evaluation vector map with the same size as the feature map and 2 channels is output. The final evaluation vector graph is normalized using the SoftMax function to obtain the final weight vector. Based on the weight vector, the feature credibility of the bird's-eye view image feature map and the millimeter-wave point cloud feature map is balanced, and the balanced bird's-eye view image feature map and millimeter-wave point cloud feature map are output. The balanced bird's-eye view image feature map, millimeter-wave point cloud feature map, and sensor metadata are input into the feature matching network, which outputs two feature offset maps of the same size as the feature map. The feature offset maps are used as the deformation offset of the deformable convolutional neural network. The deformable convolutional neural network is used to align the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map. The aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are input into the fusion network, which outputs a fused feature map of the same size as the feature map. An anchorless object detection method is used to perform object detection on the fused feature map and generate object detection results.

2. The road target detection method based on multi-source data fusion technology according to claim 1, characterized in that, The image data and millimeter-wave radar point cloud data are specifically as follows: The millimeter-wave radar and camera are calibrated to obtain the intrinsic and extrinsic parameters of the sensors: the intrinsic parameter matrix of the camera, the distortion coefficient, the extrinsic parameter matrix relative to the reference sensor, the intrinsic parameter matrix of the millimeter-wave radar, and the extrinsic parameter matrix relative to the reference sensor. Simultaneously acquire image data and millimeter-wave radar data using multiple cameras and millimeter-wave radar; The image data is distorted based on the camera's distortion coefficients; The constant false alarm rate algorithm is used to denoise millimeter-wave radar data and extract point clouds; Label 3D target data, including the target's position, size, rotation angle, and velocity information.

3. The road target detection method based on multi-source data fusion technology according to claim 1, characterized in that, Specifically, the feature maps of the bird's-eye view and the millimeter-wave point cloud are obtained as follows: This method uses deep learning to extract perspective features from image data. The input is image data acquired by multiple cameras, and the output is the features of the image data under the perspective view. A pixel depth distribution probability map is generated based on the extracted image data features using a deep learning-based monocular depth estimation method. Image data is transformed into a truncated cone space using the image pixel depth distribution probability map; Using GPU acceleration, the image data is projected downwards from the bird's-eye view, and the spatial features are accumulated onto the corresponding pixel positions of the bird's-eye view feature map corresponding to the downward projection, thus converting the perspective view image data into the bird's-eye view image data. The features of millimeter-wave point cloud data are extracted using a deep learning-based method. Finally, the point cloud features within the grid are averaged to obtain the millimeter-wave point cloud features at the corresponding location of the bird's-eye view feature map for that grid. The bird's-eye view feature map is input into a multi-level two-dimensional convolutional network to obtain feature maps of different scales. After upsampling all feature maps of all scales to the maximum scale, they are stitched together to obtain the final millimeter-wave point cloud feature map.

4. The road target detection method based on multi-source data fusion technology according to claim 3, characterized in that, The specific steps for transforming image data into a truncated cone space using the image pixel depth distribution probability map are as follows: Multiply the image feature vector corresponding to a pixel location by each component of the pixel depth distribution probability vector at that location to obtain the spatial features of each location.

5. The road target detection method based on multi-source data fusion technology according to claim 1, characterized in that, The self-attention mechanism is used to process it, and the evaluation vector graph is used as the three vectors Query, Key and Value in the self-attention mechanism.

6. The road target detection method based on multi-source data fusion technology according to claim 1, characterized in that, Feature alignment specifically refers to: The generated location code, millimeter-wave point cloud distribution mask, and image pixel occlusion probability distribution map are stitched together to obtain sensor metadata. The balanced bird's-eye view feature map, millimeter-wave point cloud feature map, and sensor metadata are input into the feature matching network. By comparing the features from different data sources, the offset of the features from different data sources is estimated, and two feature offset maps of the same size as the feature map are output. A deformable convolutional neural network is used to align the features of the balanced bird's-eye view image. The deformable offset used by the deformable convolutional neural network is the feature offset map. Finally, the aligned bird's-eye view image feature map and millimeter-wave point cloud feature map are output. The aligned bird's-eye view feature map and millimeter-wave point cloud feature map are input into the fusion network, and a high-quality fusion feature map of the same size as the feature map is output; specifically, a two-dimensional convolutional network is used to process it.

7. The road target detection method based on multi-source data fusion technology according to claim 6, characterized in that, The generated balanced bird's-eye view feature map, millimeter-wave point cloud feature map, and sensor metadata are input into a feature matching network. A shared convolutional neural network is used to fuse and reduce the dimensionality of the stitched data. Then, the fused data is input into two branches of convolutional neural networks, which output two feature offset maps of the same size as the feature maps.

8. The road target detection method based on multi-source data fusion technology according to claim 1, characterized in that, Target detection specifically includes: The generated fused feature map is input into a shared dimensionality-reducing convolutional neural network, which outputs a feature map of the same size as the original feature map. The generated feature maps are input into different categories of target detection branches. The inputs include the target center position heatmap from a bird's-eye view, the target's 3D size prediction, the target's rotation angle prediction, the target's velocity prediction, and the target's position offset prediction.

9. A road target detection system based on multi-source data fusion technology, characterized in that, include: The extraction module uses deep learning methods to extract features from image data and millimeter-wave radar point cloud data, resulting in bird's-eye view image feature map and millimeter-wave point cloud feature map. The evaluation module performs a credibility assessment on the bird's-eye view image feature map and the millimeter-wave point cloud feature map. Based on the evaluation results, it adaptively adjusts the weights of different data sources to obtain a balanced bird's-eye view image feature map and millimeter-wave point cloud feature map. The credibility assessment is specifically as follows: The location code is generated from the bird's-eye view, specifically in four channels: the sine and cosine of the angle with the center, the distance to the center, and the square of the distance to the center. A millimeter-wave point cloud distribution mask is generated based on the millimeter-wave point cloud data. A mask map of the same size as the data feature map is output. The mask value is between 0 and 1, which represents the distribution density of the millimeter-wave point cloud data at that location. When the number of point clouds in the cylindrical space corresponding to the location reaches the threshold, the mask reaches the maximum value of 1. When the number of point clouds in the cylindrical space corresponding to the location is 0, the mask is 0. By summing the probability distribution maps of image pixels along the direction of pixel rays, an approximate probability distribution map of image pixel occlusion is obtained. The location encoding, millimeter-wave point cloud distribution mask, and millimeter-wave point cloud feature map are input into the millimeter-wave mode evaluation network, and a millimeter-wave mode evaluation vector map of the same size as the feature map is output; specifically, a two-dimensional convolutional network is used to process it. The image pixel occlusion probability distribution map and the bird's-eye view image feature map are input into the image modality evaluation network, and an image modality evaluation vector map of the same size as the feature map is output. The millimeter-wave point cloud feature map and the bird's-eye view image feature map are input into the fusion network, and an environmental semantic evaluation vector map of the same size as the feature map is output. The evaluation vector map is input into the cross-modal evaluation network, and a final evaluation vector map with the same size as the feature map and 2 channels is output. The final evaluation vector graph is normalized using the SoftMax function to obtain the final weight vector. Based on the weight vector, the feature credibility of the bird's-eye view image feature map and the millimeter-wave point cloud feature map is balanced, and the balanced bird's-eye view image feature map and millimeter-wave point cloud feature map are output. The alignment module inputs the balanced bird's-eye view image feature map, millimeter-wave point cloud feature map, and sensor metadata into the feature matching network, and outputs two feature offset maps of the same size as the feature map. The feature offset maps are used as the deformation offset of the deformable convolutional neural network. The deformable convolutional neural network is used to align the balanced bird's-eye view image feature map and the millimeter-wave point cloud feature map. The aligned bird's-eye view image feature map and the millimeter-wave point cloud feature map are input into the fusion network, and output a fused feature map of the same size as the feature map. The detection module uses an anchorless target detection method to perform target detection on the fused feature map and generate target detection results.

Images