Feature fusion-based detection method and device, electronic equipment and storage medium

Abstract

The application provides a feature fusion-based detection method and device, electronic equipment and storage medium. Point cloud data of a vehicle radar and image data of a camera are obtained, the point cloud data is converted to obtain point cloud projection, image coding and first features of the image data are obtained based on an image encoder, point cloud coding and second features of the point cloud projection are obtained based on a point cloud encoder, the image coding and the second features are decoded to obtain image features, the point cloud coding and the first features are decoded to obtain point cloud features, the image features and the point cloud features are bidirectionally fused to obtain a fused feature map, and the fused feature map is input into a preset target detection head to obtain a detection result. This method realizes feature fusion of radar and camera data for detection, makes up for the deficiency of single data detection, and improves detection performance and accuracy.

Description

Technical Field

[0001] This application relates to the field of target detection technology, specifically to a detection method, apparatus, electronic device, and storage medium based on feature fusion. Background Technology

[0002] In Level 2 autonomous driving, the 1v1r solution (one camera + one radar) has shown excellent applicability in driving environments with speeds below 60km / h due to its high efficiency and economy. The typical configuration of the 1v1r solution includes a forward-facing camera and a forward-facing millimeter-wave radar to achieve lateral and longitudinal directional control of the vehicle.

[0003] Forward-facing cameras can capture road sign information, including lane lines, traffic signs, and pedestrians, providing vehicles with detailed environmental information to support vehicle decision-making. Forward-facing millimeter-wave radar has a narrow antenna beam, which can accurately locate targets and measure the speed and distance of objects, providing vehicles with information such as obstacles. When vehicles perform target detection, relying solely on image information may result in low target detection accuracy due to various adverse weather conditions. Relying solely on radar information has the disadvantages of weak anti-interference ability and insufficient classification ability. Summary of the Invention

[0004] In view of the shortcomings of the above-mentioned related technologies, this application provides a detection method, apparatus, electronic device and storage medium based on feature fusion to solve the technical problem of low detection performance and accuracy in existing detection methods.

[0005] This application provides a detection method based on feature fusion. The detection method includes: acquiring point cloud data from a vehicle radar and image data from a camera; converting the point cloud data into a point cloud projection; obtaining an image code and a first feature from the image data based on an image encoder, and obtaining a point cloud code and a second feature from the point cloud projection based on a point cloud encoder; decoding the image code and the second feature to obtain image features, and decoding the point cloud code and the first feature to obtain point cloud features; performing bidirectional feature fusion on the image features and the point cloud features to obtain a fused feature map; and inputting the fused feature map into a preset target detection head to obtain a detection result.

[0006] In one embodiment of this application, converting the point cloud data into a point cloud projection includes: projecting each point in the point cloud data onto a top-view plane to obtain two-dimensional plane coordinates corresponding to each point in the point cloud data; if multiple points in the point cloud data correspond to the same two-dimensional plane coordinates, calculating region filling data corresponding to the two-dimensional plane coordinates based on the multiple points; creating a to-be-filled image based on a preset resolution, wherein the pixels in the to-be-filled image have a mapping relationship with the two-dimensional plane coordinates; filling the region filling data of the two-dimensional plane coordinates into the pixels corresponding to the to-be-filled image; and using the filled image as a point cloud projection.

[0007] In one embodiment of this application, calculating the region filling data corresponding to the two-dimensional plane coordinates based on the plurality of points includes: calculating the region density based on the number of all points mapped to the same two-dimensional plane coordinates; selecting the maximum reflection intensity value among all points mapped to the same two-dimensional plane coordinates as the region reflection intensity; calculating the region projection length and region projection width according to the distribution of all points mapped to the same two-dimensional plane coordinates; and using the region density, the region reflection intensity, the region projection length, and the region projection width as the region filling data.

[0008] In one embodiment of this application, the bidirectional feature fusion of the image features and the point cloud features to obtain a fused feature map includes: concatenating the point cloud features and the image features through channels to obtain concatenated features; obtaining a first fused feature based on a first preset mapping function, a preset activation function, and the concatenated features; obtaining a second fused feature based on a second preset mapping function, a preset activation function, and the concatenated features; subtracting the first fused feature from the point cloud features and then multiplying it by the image features to obtain fused image features; subtracting the second fused feature from the image features and then multiplying it by the point cloud features to obtain fused point cloud features; and multiplying the fused image features and the fused point cloud features to obtain the fused feature map.

[0009] In one embodiment of this application, obtaining a first fused feature based on a first preset mapping function, a preset activation function, and concatenated features, and obtaining a second fused feature based on a second preset mapping function, a preset activation function, and concatenated features includes: setting the first preset mapping function and the second preset mapping function based on a convolutional layer; performing a first transformation on the concatenated features based on the first preset mapping function, applying the preset activation function to the concatenated features after the first transformation, and obtaining the first fused feature; performing a second transformation on the concatenated features based on the second preset mapping function, applying the preset activation function to the concatenated features after the second transformation, and obtaining the second fused feature.

[0010] In one embodiment of this application, after inputting the fused feature map into a preset target detection head and obtaining the detection result, the method further includes: calculating a feature loss based on the detection result and a loss function, wherein the loss function includes:

[0011]

[0012] in, For feature loss, I F For the detection result, I A For the projection of the point cloud, I B For the image data, For L2 norm, W A and W B The weight estimate is calculated based on the point cloud projection and the image data, and W A +W B =1.

[0013] In one embodiment of this application, the fused feature map is input into a preset target detection head to obtain the detection result, which includes: collecting historical feature maps, training a preset neural network model based on the historical feature maps to obtain a preset target detection head, wherein the input of the preset target detection head includes a feature image, and the output result of the preset target detection head includes at least a detection box, target category, confidence score, center point coordinates and angle;

[0014] The fused feature map is input into the preset target detection head to obtain the output result of the preset target detection head. The output result is then optimized. The optimization process includes: selecting multiple detection boxes as target boxes based on the confidence level; determining whether the overlap between the target box and other detection boxes is greater than a preset overlap threshold; deleting the detection boxes with an overlap greater than the preset overlap threshold; and so on, until the overlap between any detection box and other detection boxes is less than or equal to the overlap threshold.

[0015] The optimized output is used as the detection result.

[0016] Embodiments of this application also provide a detection device based on feature fusion, comprising: an acquisition module for acquiring point cloud data from a vehicle radar and image data from a camera, and converting the point cloud data into a point cloud projection; an encoding module for obtaining an image code and a first feature from the image data based on an image encoder, and obtaining a point cloud code and a second feature from the point cloud projection based on a point cloud encoder; a decoding module for decoding the image code and the second feature to obtain an image feature, and decoding the point cloud code and the first feature to obtain a point cloud feature; a fusion module for bidirectional feature fusion of the image feature and the point cloud feature to obtain a fused feature map; and a detection module for inputting the fused feature map into a preset target detection head to obtain a detection result.

[0017] Embodiments of this application also provide an electronic device, the electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the detection method based on feature fusion described in any of the above embodiments.

[0018] Embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform a feature fusion-based detection method as described in any of the above embodiments.

[0019] The beneficial effects of this application are as follows: Embodiments of this application provide a detection method, apparatus, electronic device, and storage medium based on feature fusion. The method acquires point cloud data from vehicle radar and image data from a camera. The point cloud data is converted into a point cloud projection. An image encoder is used to obtain the image code and a first feature of the image data. The point cloud encoder is used to obtain the point cloud code and a second feature of the point cloud projection. The image code and the second feature are decoded to obtain image features. The point cloud code and the first feature are decoded to obtain point cloud features. Simultaneous modeling of the point cloud projection and image data is introduced during the encoding stage to extract more complementary features. This allows for a better understanding of the relationship between the two types of data during subsequent target detection, thereby obtaining richer feature information and improving the target detection effect. The image features and point cloud features are bidirectionally fused to obtain a fused feature map. The fused feature map is input into a preset target detection head to obtain the detection result. This method achieves detection by fusing radar and camera data features, overcoming the shortcomings of detection using single data and improving detection performance and accuracy.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the implementation environment of a feature fusion-based detection method, as shown in an exemplary embodiment of this application.

[0022] Figure 2 This is a flowchart illustrating a feature fusion-based detection method as an exemplary embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating the structure of a feature fusion-based detection model, as shown in an exemplary embodiment of this application.

[0024] Figure 4 This is a schematic diagram of a bidirectional conditional feature fusion module shown in an exemplary embodiment of this application;

[0025] Figure 5 This is a block diagram illustrating a feature fusion-based detection device as an exemplary embodiment of this application;

[0026] Figure 6 This is a schematic diagram illustrating the structure of an electronic device as shown in an exemplary embodiment of this application. Detailed Implementation

[0027] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0029] It should be noted that in this application, terms such as "first" and "second" are merely for distinguishing similar objects, and do not limit the order or sequence of similar objects. The variations of "including" and "having" indicate that the scope covered by the subject of the word is not exclusive, except for the examples shown by the word.

[0030] It is understood that the various numerical designations, step numbers, and other identifiers recorded in this application are for descriptive convenience and are not intended to limit the scope of this application. The size of the identifiers in this application does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0031] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0032] The embodiments of this application respectively propose a detection method based on feature fusion, a detection device based on feature fusion, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail below.

[0033] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation environment of a feature fusion-based detection method, as shown in an exemplary embodiment of this application.

[0034] like Figure 1 As shown, the implementation environment may include a camera 101, a radar 102, and a vehicle 103, wherein the camera 101 and radar 102 are connected to the data acquisition module of the vehicle 103.

[0035] In one embodiment of this application, camera 101 may be a forward-facing camera capable of capturing rich road marking information. Whether it's lane lines, traffic signs, or pedestrians, camera 101 can capture them in real-time with high-definition image quality, providing the vehicle with detailed environmental information. This information is crucial for intelligent driving systems, as it not only helps the vehicle understand current road conditions but also provides strong support for decision-making.

[0036] In one embodiment of this application, radar 102 can be a forward-facing millimeter-wave radar, providing powerful perception capabilities for the intelligent driving system. The narrow antenna beam of millimeter-wave radar enables precise target localization; its high resolution ensures the accuracy of measurement data; its wide bandwidth allows it to operate stably in various environments; and its strong anti-interference capability effectively addresses various interference factors in complex road environments. These characteristics enable radar 102 to measure the speed and distance of objects with high accuracy, providing vehicles with real-time and reliable obstacle information. In terms of weather and environment, radar 102 has significant advantages over other sensors. Whether in rain, snow, or fog, radar 102 maintains stable performance, unaffected by adverse weather conditions. This stability allows the 1v1r solution to maintain high perception robustness under various weather conditions, providing continuous and reliable perception data for the intelligent driving system.

[0037] In one embodiment of this application, a 1v1r solution combining camera 101 and radar 102 achieves complementary and enhanced perception capabilities. Camera 101 provides rich road marking information, while radar 102 provides accurate obstacle information. The combination of the two not only enhances the system's prediction accuracy for target attributes such as depth and speed, but also effectively compensates for the shortcomings of millimeter-wave radar in terms of multipath interference, height loss, and classification capabilities.

[0038] Please see Figure 2 , Figure 2 This is a flowchart illustrating a feature fusion-based detection method as an exemplary embodiment of this application. This method can be applied to... Figure 1 The implementation environment shown is intended to illustrate a method that can also be applied to other exemplary implementation environments and executed by devices in those environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0039] like Figure 2 As shown, in an exemplary embodiment, the detection method based on feature fusion includes at least steps S210 to S250, which are described in detail below:

[0040] Step S210: Acquire point cloud data from the vehicle radar and image data from the camera, and convert the point cloud data and image data into a point cloud projection.

[0041] In one embodiment of this application, a vehicle-mounted LiDAR is used to generate point cloud data. The LiDAR is installed on the vehicle, and the radar is connected to the vehicle's data acquisition system. The raw data is read from the radar device, and the raw data is stored based on the point cloud data format. The acquired point cloud data is preprocessed, including noise reduction and filtering, to improve data quality for subsequent applications.

[0042] In one embodiment of this application, a suitable camera is selected according to application requirements, the camera is installed on the vehicle, the camera is connected to the vehicle's data acquisition system, image data is read from the camera, the image data is stored in the form of RGB three-channel color images, and the image data is preprocessed, including noise reduction, enhancement and other operations, to improve data quality for subsequent applications.

[0043] In one embodiment of this application, point cloud data acquired by radar and image data acquired by camera are synchronized and calibrated to ensure consistency of the two types of data in time and space, including matching timestamps and unifying coordinate systems.

[0044] In one embodiment of this application, a projection plane and resolution are defined, it is determined that the point cloud data will be projected onto a bird's-eye view (BEV) plane, and a preset resolution (H and W) is set. According to the projection type, the 3D coordinates of each point in the point cloud data are converted into corresponding 2D plane coordinates. For all points mapped to the same 2D plane coordinates, the statistical feature point cloud density, maximum reflection intensity, projection length, and projection width are calculated. These features are used as region filling data, and the region filling data is filled into the image pixels corresponding to the 2D plane coordinates to generate the final 2D projected image. This 2D projected image is then used as the point cloud projection.

[0045] In one embodiment of this application, converting point cloud data into a point cloud projection includes: projecting each point in the point cloud data onto a top-view plane to obtain the two-dimensional plane coordinates corresponding to each point in the point cloud data; if multiple points in the point cloud data correspond to the same two-dimensional plane coordinates, calculating the region filling data corresponding to the two-dimensional plane coordinates based on the multiple points; creating an image to be filled based on a preset resolution, wherein the pixels in the image to be filled have a mapping relationship with the two-dimensional plane coordinates; filling the region filling data of the two-dimensional plane coordinates into the pixels corresponding to the image to be filled; and using the filled image to be filled as the point cloud projection.

[0046] In one embodiment of this application, calculating the region filling data corresponding to two-dimensional plane coordinates based on multiple points includes: calculating the region density based on the number of all points mapped to the same two-dimensional plane coordinates; selecting the maximum reflection intensity value among all points mapped to the same two-dimensional plane coordinates as the region reflection intensity; calculating the region projection length and region projection width according to the distribution of all points mapped to the same two-dimensional plane coordinates; and using the region density, region reflection intensity, region projection length, and region projection width as region filling data.

[0047] Step S220: Obtain the image code and first feature of the image data based on the image encoder, and obtain the point cloud code and second feature of the point cloud projection based on the point cloud encoder.

[0048] In one embodiment of this application, image data is input into an image encoder based on an image U-Net model. The image data is encoded based on the image encoder. Through layer-by-layer convolution and pooling operations, visual features of the image data are extracted step by step, and the visual features are used as the first feature, so as to obtain the image code and the first feature during the encoding process of the image data.

[0049] In one embodiment of this application, point cloud data is input into a point cloud encoder based on a point cloud U-Net model. The point cloud data is encoded based on the point cloud encoder. Through layer-by-layer convolution and pooling operations, spatial structural features describing the point cloud data are extracted step by step, and these spatial structural features are used as second features to obtain point cloud encoding and second features during the encoding process of point cloud data.

[0050] Step S230: Decode the image encoding and the second feature to obtain image features, and decode the point cloud encoding and the first feature to obtain point cloud features.

[0051] In one embodiment of this application, before the decoding stage of the image U-Net model begins, the second feature output by the point cloud encoder in the point cloud U-Net model is obtained, and the second feature is concatenated with the image code. The concatenated image code is then decoded to obtain the image feature, thereby achieving cross-modal feature fusion.

[0052] In one embodiment of this application, before the decoding stage of the point cloud U-Net model begins, the first feature output by the image encoder in the image U-Net model is obtained, and the first feature is concatenated with the point cloud code. The concatenated point cloud code is then decoded to obtain the point cloud feature, thereby realizing cross-modal feature fusion.

[0053] In one embodiment of this application, after feature fusion, the point cloud U-Net model and the image U-Net model continue their decoding process. At this time, the decoder processes not only information from its own modality but also fused features from the other modality. In this way, the two networks can share and utilize each other's information to improve the ability to recognize target features.

[0054] Step S240: Perform bidirectional feature fusion of image features and point cloud features to obtain a fused feature map.

[0055] In one embodiment of this application, the bidirectional feature fusion of image features and point cloud features to obtain a fused feature map includes: concatenating point cloud features and image features through channels to obtain concatenated features; obtaining a first fused feature based on a first preset mapping function, a preset activation function, and the concatenated features; obtaining a second fused feature based on a second preset mapping function, a preset activation function, and the concatenated features; subtracting the first fused feature from the point cloud features and then multiplying it by the image features to obtain fused image features; subtracting the second fused feature from the image features and then multiplying it by the point cloud features to obtain fused point cloud features; and multiplying the fused image features and the fused point cloud features to obtain a fused feature map.

[0056] In one embodiment of this application, obtaining a first fused feature based on a first preset mapping function, a preset activation function, and concatenated features, and obtaining a second fused feature based on a second preset mapping function, a preset activation function, and concatenated features includes: setting a first preset mapping function and a second preset mapping function according to a 3×3 convolutional layer; performing a first transformation on the concatenated features based on the first preset mapping function, applying a preset activation function to the concatenated features after the first transformation, and obtaining the first fused feature; performing a second transformation on the concatenated features based on the second preset mapping function, applying a preset activation function to the concatenated features after the second transformation, and obtaining the second fused feature.

[0057] In one embodiment of this application, features are fused based on conditional probabilities. Conditional probabilities can help deep learning models quantify the importance of information under different conditions, thereby achieving weighted adjustment of information. For example, in natural language processing, conditional probabilities can be used to calculate the importance of words in a given context, thereby achieving a more accurate language model.

[0058] In one embodiment of this application, the target detection model includes a two-way conditional feature fusion module. This module can not only exchange information between millimeter-wave radar data and camera data, but also dynamically combine them according to the actual situation to obtain richer and more accurate feature representations. Through this two-way conditional feature fusion, the model can better capture the texture, shape, and spatial information of the target, thereby further improving the performance and accuracy of target detection.

[0059] In one embodiment of this application, bidirectional feature fusion is performed in a bidirectional conditional feature fusion module, wherein the bidirectional conditional feature fusion module is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of a bidirectional conditional feature fusion module shown in an exemplary embodiment of this application.

[0060] like Figure 4 As shown, the bidirectional conditional feature fusion module is used to further fuse the decoded image features and point cloud features. This module mainly handles message passing between these two types of information. Figure 4 In the diagram, red arrows indicate inputs, and green arrows indicate outputs. φ represents channel-dimensional concatenation, h and f represent mapping functions, both of which are convolution operations consisting of 3×3 kernels, and φ represents the activation function. ⊕ indicates Hadamard product, ⊕ indicates addition, and ◎ indicates subtraction.

[0061] In one embodiment of this application, specifically, the point cloud features are represented as F p The image features are represented as F i The resulting point cloud features include:

[0062] F pi =φ(h(Concat(F) p ,F i Formula (1)

[0063] F p_ =F pi -F i Equation (2)

[0064]

[0065] In equations (1), (2), and (3), F p For point cloud features, F ii For image features, F pi The second fusion feature is φ, which is the Sigmoid-based activation function, and h is the second preset mapping function. p_ The second fusion feature is subtracted from the image feature, O p The fused point cloud features are represented by 'a', which is a trainable parameter initialized to 0. This indicates the calculation of conditional probability. Indicates that when F is known p_ F is generated under the condition i The conditional probability, where Concat(F) p ,F i The image features and point cloud features are stitched together by channel stitching.

[0066] In one embodiment of this application, specifically, the point cloud features are represented as F p The image features are represented as F i The features obtained after fusion include:

[0067] F ip =φ(f(Concat(F) i ,F p Formula (4)

[0068] F i_ =F ip -F p Equation (5)

[0069]

[0070] In equations (4), (5), and (6), F p For point cloud features, F i For image features, F ip Let φ be the first fusion feature, φ be the Sigmad activation function, f be the first preset mapping function, and F be the first fusion feature. i_ The second fused feature is subtracted from the point cloud feature, Oi Let b represent the features of the fused image, and b be a trainable parameter initialized to 0. This indicates the calculation of conditional probability. Indicates that F is known i_ F is generated under the condition p The conditional probability, where Concat(F) p ,F i The image features and point cloud features are stitched together by channel stitching.

[0071] Step S250: Input the fused feature map into the preset target detection head to obtain the detection result.

[0072] In one embodiment of this application, inputting the fused feature map into a preset target detection head to obtain the detection result includes: inputting the fused feature map into the output of the preset target to the detection head, the target detection head including the target category, confidence level, center point coordinates and angle; and using the output of the target detection head as the detection result.

[0073] In one embodiment of this application, the input of the fused feature map into a preset target detection head to obtain the detection result further includes: collecting historical feature maps, training a preset neural network model based on the historical feature maps to obtain a preset target detection head, wherein the input of the preset target detection head includes feature images, and the output result of the preset target detection head includes at least a detection box, target category, confidence score, center point coordinates and angle;

[0074] The fused feature map is input into a preset target detection head to obtain the output result of the preset target detection head. The output result is then optimized. The optimization process includes: selecting multiple detection boxes as target boxes based on confidence, determining whether the overlap between the target box and other detection boxes is greater than a preset overlap threshold, deleting detection boxes with an overlap greater than the preset overlap threshold, until the overlap between any detection box and other detection boxes is less than or equal to the overlap threshold.

[0075] The optimized output is used as the detection result.

[0076] In one embodiment of this application, all detection boxes in the target detection head output are sorted according to their confidence levels. The detection box with the highest confidence level is selected as the target box from the sorted list. The overlap between the target box and all remaining detection boxes in the list is calculated. If the overlap between a detection box and the target box is greater than a preset overlap threshold (e.g., 0.5), the detection box is considered redundant and is removed from the detection box list. The above process is repeated, selecting the detection box with the highest confidence level from the remaining detection box list as the new target box, and determining whether the overlap between the target box and other detection boxes is greater than the preset overlap threshold. Detection boxes with an overlap greater than the preset overlap threshold are deleted until the overlap between any detection box and other detection boxes is less than or equal to the overlap threshold.

[0077] In one embodiment of this application, acquiring historical feature maps and training a preset neural network model based on the historical feature maps to obtain a preset target detection head includes: acquiring a large amount of historical point cloud data and historical image data; obtaining historical feature maps based on the historical point cloud data and historical image data; selecting a neural network model as a preset model; and training the preset model with the historical feature maps as input to obtain the preset target detection head.

[0078] In one embodiment of this application, the preset target detection head can be a commonly used 3D target detection head, responsible for outputting detection results based on the fused feature map. This preset target detection head has pre-set parameters used to extract relevant target information from the feature map. In this embodiment, the target category indicates which category the detected target belongs to, such as pedestrian, vehicle, etc.; the confidence level indicates the degree of confidence in the detection result, typically a probability value between 0 and 1; the center coordinates represent the target's position in the image, usually a two-dimensional coordinate; and the angle is the output rotation angle for some rotating targets to more accurately locate the target.

[0079] In one embodiment of this application, after the fused feature map is input into a preset target detection head and the detection result is obtained, the method further includes unsupervised training of the target detection model using the detection result. During the training process, a loss function needs to be designed to ensure that the coupling of the two U-Nets not only enhances the model's performance but also maintains a stable training process. However, the L2 norm has certain limitations in representing structures, which often leads to problems such as structural information degradation in the fused data. To ensure that the fused result contains more relevant information from the source data, the following loss function is designed to calculate the feature loss:

[0080]

[0081] In equation (7), For feature loss, I F For the test results, IA For point cloud projection, I B For image data, For L2 norm, W A and W B The weight estimate is calculated based on point cloud projection and image data, and W A +W B =1. Where W A The larger, I F Includes I A The more information you have, the less information you have, and vice versa.

[0082] In one embodiment of this application, a weight estimate is calculated based on point cloud projection and image data, and a method combining saliency and contrast is used to construct {W}. A W B To preserve more detail in the input data, all source data is first contrast-enhanced, then image processing methods are used to generate salient data of the enhanced results, which are then combined with the source data to construct {W}. A W B}, as shown below:

[0083]

[0084]

[0085] In equations (8) and (9), SM A For I A Significance data after contrast enhancement, SM B For I B The significance data after contrast enhancement, where α and β represent preset non-negative constants used to measure the contribution of each sub-item. In this embodiment, the contrast enhancement of the point cloud projection is obtained by calculation on the 2D features obtained from the projection.

[0086] In one embodiment of this application, considering that staged training may lead to significant differences in information across different modalities, a method is adopted... The model is trained using feature loss to optimize the coupling effect between the two networks. Training the model in an unsupervised manner can not only improve the model's performance but also reduce training costs, making the model more practical and efficient.

[0087] Figure 3 This is a schematic diagram illustrating the structure of a feature fusion-based detection model, as shown in an exemplary embodiment of this application. Figure 3The feature fusion-based detection model includes a decoding-encoding module and a bidirectional conditional feature fusion module. The red arrows indicate the inputs. Radar point cloud data and camera image data are used as inputs to the model. Preliminary feature fusion is performed in the encoding-decoding module: point cloud data is encoded, and image data is encoded. After point cloud encoding, the first feature of the image encoding is obtained and fused into the point cloud decoding to obtain point cloud features. After image encoding, the second feature of the point cloud encoding is obtained and fused into the image decoding to obtain image features. The image features and point cloud features are then input into the bidirectional conditional feature fusion module for secondary feature fusion. Figure 3 The light yellow blocks represent the bidirectional conditional feature fusion module. After fusion, the bidirectional conditional feature fusion module outputs the fused feature image. The fused feature image is then input into the 3D detection head to obtain the output result, which is the detection result. After obtaining the detection result, the entire model network is trained unsupervised based on the detection result and the loss function.

[0088] Please see Figure 5 , Figure 5 This is a block diagram illustrating a feature fusion-based detection apparatus as an exemplary embodiment of this application. This apparatus can be applied to... Figure 1 The implementation environment shown can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0089] like Figure 5 As shown, the exemplary feature fusion-based detection device includes an acquisition module 501, an encoding module 502, a decoding module 503, a fusion module 504, and a detection module 505.

[0090] The acquisition module 501 is used to acquire point cloud data from the vehicle radar and image data from the camera, and convert the point cloud data into a point cloud projection.

[0091] The encoding module 502 is used to obtain image encoding and first features of image data based on the image encoder, and to obtain point cloud encoding and second features of point cloud projection based on the point cloud encoder.

[0092] The decoding module 503 is used to decode the image encoding and the second feature to obtain image features, and to decode the point cloud encoding and the first feature to obtain point cloud features;

[0093] The fusion module 504 is used to perform bidirectional feature fusion of image features and point cloud features to obtain a fused feature map.

[0094] The detection module 505 is used to input the fused feature map into the preset target detection head to obtain the detection result.

[0095] Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0096] like Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from Storage Unit 608 into Random Access Memory (RAM) 603, such as performing the methods described in the above embodiments. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0097] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0098] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0099] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0102] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the feature fusion-based detection method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0103] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the feature fusion-based detection method provided in the various embodiments described above.

[0104] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A feature fusion-based detection method, characterized in that, The feature fusion-based detection method comprises: Obtaining point cloud data of a vehicle radar and image data of a camera, and converting the point cloud data to obtain a point cloud projection; Obtaining image encoding and first features of the image data based on an image encoder, and obtaining point cloud encoding and second features of the point cloud projection based on a point cloud encoder; Decoding the image encoding and the second features to obtain image features, and decoding the point cloud encoding and the first features to obtain point cloud features; decoding the image encoding and the second features to obtain image features comprises splicing the second features and the image encoding, and decoding the spliced image encoding to obtain the image features; decoding the point cloud encoding and the first features to obtain point cloud features comprises splicing the first features and the point cloud encoding, and decoding the spliced point cloud encoding to obtain the point cloud features; Performing bidirectional feature fusion on the image features and the point cloud features to obtain a fused feature map; performing bidirectional feature fusion on the image features and the point cloud features to obtain a fused feature map comprises splicing the point cloud features and the image features to obtain spliced features; obtaining first fused features based on a first preset mapping function, a preset activation function and the spliced features, and obtaining second fused features based on a second preset mapping function, a preset activation function and the spliced features; multiplying the first fused features by the image features after subtracting the point cloud features, and multiplying the second fused features by the point cloud features after subtracting the image features to obtain fused image features and fused point cloud features; and multiplying the fused image features and the fused point cloud features to obtain the fused feature map; Inputting the fused feature map into a preset target detection head to obtain a detection result.

2. The feature fusion-based detection method of claim 1, wherein, Converting the point cloud data to obtain a point cloud projection comprises: Projecting each point in the point cloud data onto a bird's-eye plane to obtain a two-dimensional plane coordinate corresponding to each point in the point cloud data; If multiple points in the point cloud data correspond to the same two-dimensional plane coordinate, calculating region filling data corresponding to the two-dimensional plane coordinate based on the multiple points; Creating a to-be-filled image based on a preset resolution, wherein a pixel in the to-be-filled image is in a mapping relationship with the two-dimensional plane coordinate; Filling the region filling data of the two-dimensional plane coordinate into the pixel corresponding to the to-be-filled image; Taking the filled to-be-filled image as the point cloud projection.

3. The feature fusion based detection method of claim 2, wherein, Calculating the region filling data corresponding to the two-dimensional plane coordinate based on the multiple points comprises: Calculating a region density degree based on the number of all points mapped to the same two-dimensional plane coordinate; Selecting a maximum reflection intensity value in all points mapped to the same two-dimensional plane coordinate as a region reflection intensity; Calculating a region projection length and a region projection width according to the distribution of all points mapped to the same two-dimensional plane coordinate; Taking the region density degree, the region reflection intensity, the region projection length and the region projection width as the region filling data.

4. The feature fusion based detection method of claim 1, wherein, The first fusion feature is obtained based on the first preset mapping function, the preset activation function and the spliced feature, and the second fusion feature is obtained based on the second preset mapping function, the preset activation function and the spliced feature, including: The first preset mapping function and the second preset mapping function are set based on a convolution layer; The first change is performed on the spliced feature based on the first preset mapping function, and the preset activation function is applied to the spliced feature after the first change to obtain the first fusion feature; The second change is performed on the spliced feature based on the second preset mapping function, and the preset activation function is applied to the spliced feature after the second change to obtain the second fusion feature.

5. The feature fusion based detection method of claim 1, wherein, After the fused feature map is input into the preset target detection head to obtain a detection result, the method further includes: The feature loss is calculated based on the detection result and a loss function, and the loss function includes: wherein, is a characteristic loss, is the detection result, is the point cloud projection, is the image data, is a norm, and is a weight estimate computed based on the point cloud projection and the image data, and + = 1.

6. The feature fusion based detection method according to any one of claims 1 or 2, characterized in that, The fused feature map is input into the preset target detection head to obtain a detection result, including: The preset neural network model is trained based on the historical feature map to obtain the preset target detection head, an input of the preset target detection head includes a feature image, and an output result of the preset target detection head at least includes a detection frame, a target category, a confidence, a center point coordinate and an angle; The fused feature map is input into the preset target detection head to obtain the output result of the preset target detection head, and the output result is optimized, the optimization includes: selecting multiple detection frames as target frames based on the confidence, judging whether the overlap degree of the target frame and other detection frames is greater than a preset overlap degree threshold, deleting the detection frame with the overlap degree greater than the preset overlap degree threshold until the overlap degree of any detection frame and other detection frames is less than or equal to the overlap degree threshold; The output result after the optimization is taken as the detection result.

7. A feature fusion-based detection apparatus, characterized by, The detection device based on feature fusion includes: An acquisition module is configured to acquire point cloud data of a vehicle radar and image data of a camera, and convert the point cloud data to obtain point cloud projection; An encoding module is configured to obtain image encoding and first features of the image data based on an image encoder, and obtain point cloud encoding and second features of the point cloud projection based on a point cloud encoder; A decoding module is configured to decode the image encoding and the second features to obtain image features, and decode the point cloud encoding and the first features to obtain point cloud features; decoding the image encoding and the second features to obtain image features includes splicing the second features and the image encoding, and decoding the spliced image encoding to obtain the image features; decoding the point cloud encoding and the first features to obtain point cloud features includes splicing the first features and the point cloud encoding, and decoding the spliced point cloud encoding to obtain the point cloud features; The fusion module is configured to perform bidirectional feature fusion on the image feature and the point cloud feature to obtain a fused feature map. The bidirectional feature fusion on the image feature and the point cloud feature to obtain the fused feature map includes: performing channel splicing on the point cloud feature and the image feature to obtain spliced features; obtaining first fused features based on a first preset mapping function, a preset activation function and the spliced features, and obtaining second fused features based on a second preset mapping function, the preset activation function and the spliced features; multiplying the first fused features by the image feature after the first fused features are subtracted from the point cloud feature to obtain a fused image feature, and multiplying the second fused features by the point cloud feature after the second fused features are subtracted from the image feature to obtain a fused point cloud feature; and multiplying the fused image feature and the fused point cloud feature to obtain the fused feature map. The detection module is configured to input the fused feature map into a preset target detection head to obtain a detection result.

8. An electronic device, comprising: The electronic device includes: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the feature fusion-based detection method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, a computer program stored thereon, which, when executed by a processor of a computer, causes the computer to perform the feature fusion-based detection method of any one of claims 1-6.

Images