A target detection method and system based on context clustering and multi-modal fusion

Abstract

The application discloses a target detection method and system based on context clustering and multi-modal fusion, relates to the technical field of computer vision and radar signal processing, and comprises the following steps: acquiring image data and point cloud data and performing data enhancement processing; extracting features from the image data after the data enhancement processing, calculating corresponding heat maps and attribute parameters based on visual branch features, and obtaining a first detection result of a target; performing rectangular expansion on the point cloud data after the data enhancement, and obtaining an extended point cloud rectangular surface; performing regional association on the first detection result, obtaining associated data and extracting features, and obtaining radar branch features; fusing the visual branch features and the radar branch features, calculating a second detection result, inputting the first detection result and the second detection result into a decoder together, and obtaining a final detection result of the target. The multi-modal feature fusion network structure based on context clustering designed in the application realizes the fusion and complementation of multi-modal feature data, and improves the precision and efficiency of target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and radar signal processing, and more particularly, to a target detection method and system based on context clustering and multi-modal fusion. BACKGROUND

[0002] In the field of autonomous driving and assisted driving, the target detection effect of intelligent perception system has become one of the key factors to determine the degree of environmental perception. In recent years, there have been relatively extensive research on target detection algorithms based on single modal sensors, such as vision and radar, especially the target detection algorithm based on deep learning vision sensor has been studied in depth. However, in real application scenarios, the target detection algorithm based on single modal is extremely susceptible to environmental and detection efficiency factors, resulting in unstable performance of target detection algorithm, poor robustness and other problems. For example, the target detection algorithm based on vision sensor is extremely susceptible to complex environment such as rain, fog or low illumination, which leads to the decline of target detection performance, and even leads to the failure of detection algorithm. These problems will have a great impact on the operation of autonomous driving and assisted driving system.

[0003] In recent years, the target detection algorithm based on multi-modal sensor information fusion has attracted widespread attention because it can improve the detection accuracy of the surrounding objects of the perception system. The existing multi-modal fusion method is mainly a multi-modal fusion network based on convolutional neural network structure or self-attention mechanism network structure. Although these networks can fuse multi-modal data to achieve better target detection effect, they usually require a large number of network parameters to train the model, so they have high requirements for the computing performance of the deployment device. At present, the information fusion technology based on vision sensor and radar sensor has become a very promising research direction, because the two modal sensors can complement each other well and are relatively inexpensive. The vision sensor can accurately detect the target class features in the visual space under standard environment, but lacks the perception ability of object depth and speed, and in complex environment, such as occlusion, it will further deteriorate the target perception ability. The data provided by the radar sensor is relatively sparse in spatial dimension, but is not affected by rain, fog or low illumination, and can measure the distance and speed of the reflected objects in the environment at the same time. Therefore, the fusion of data from two modalities of vision sensor and radar sensor can complement their deficiencies and provide more accurate and robust target detection effect.

[0004] The prior art discloses an object detection method based on semantic segmentation enhancement, including preparing an image with a label, and dividing the collected image; designing a deep convolutional neural network structure based on semantic segmentation enhancement, including a main network, a segmentation subnetwork and a detection subnetwork; the main network is used to extract general features of the image, the segmentation subnetwork is used to extract semantic segmentation features and predict a segmentation heat map of each type of object, and the detection subnetwork adopts a specific type of detector to extract and predict the features of the type; the deep convolutional neural network is trained by using a training data set, and the detection result of the image is calculated by using the trained network. The comparison file is mainly based on a convolutional neural network structure, although the convolutional neural network structure can fuse multi-modal data to a certain extent, a large number of network parameters are needed to train the model, the computing performance of a deployment device has a higher requirement, and the target detection efficiency is low. SUMMARY

[0005] In order to overcome the defects that the above-mentioned single-mode target detection algorithm is extremely susceptible to environmental and detection efficiency factors, leading to unstable performance and poor robustness of the target detection algorithm, the application provides a target detection method and system based on context clustering and multi-modal fusion, which uses multi-modal feature data for fusion and complementation to improve the accuracy and efficiency of target detection.

[0006] To solve the above technical problems, the technical scheme of the application is as follows:

[0007] The application provides a target detection method based on context clustering and multi-modal fusion, comprising:

[0008] S1: acquiring paired image data and point cloud data, and performing data enhancement processing on the image data and the point cloud data to obtain data-enhanced image data and point cloud data;

[0009] S2: performing feature extraction on the data-enhanced image data to obtain visual branch features;

[0010] S3: based on the visual branch features, calculating corresponding heat maps and attribute parameters to obtain a first detection result of the target;

[0011] S4: mapping the data-enhanced point cloud data to an image coordinate system and performing rectangular expansion to obtain an expanded point cloud rectangular surface;

[0012] S5: performing region association on the first detection result of the target and the expanded point cloud rectangular surface to obtain associated data;

[0013] S6: performing feature extraction on the associated data to obtain radar branch features;

[0014] S7: performing feature fusion on the visual branch feature and the radar branch feature to obtain a fusion feature;

[0015] S8: calculating a second detection result of the target based on the fusion feature;

[0016] S9: inputting the first detection result of the target and the second detection result of the target into a preset decoder to obtain a final detection result of the target.

[0017] Preferably, in the step S1, the image data is acquired by using a visual sensor, and the point cloud data is acquired by using a radar sensor; the data enhancement processing performed on the image data and the point cloud data includes data random horizontal flipping and data random moving.

[0018] Preferably, in the step S2, the image data after data enhancement is input into an existing deep aggregation network model to perform feature extraction, thereby obtaining the visual branch feature.

[0019] Preferably, in the step S3, the visual branch feature is input into a trained first regression network to calculate a heat map and attribute parameters represented by the visual branch feature, thereby obtaining the first detection result of the target; the attribute parameters include target size, offset, three-dimensional size, depth and direction.

[0020] Preferably, a specific method for obtaining the trained first regression network is as follows:

[0021] The visual branch feature of the image training data is input into the constructed first regression network to calculate a corresponding heat map, target size, offset, three-dimensional size, depth and direction.

[0022] For the heat map, a focal loss function is set for supervised training, and specifically:

[0023]

[0024] In the formula, L focal represents a focal loss value, N represents a target quantity of the image training data, Y k represents a real heat map of the target, and represents a predicted heat map of the target. xyc represents a predicted heat map of the target, and represents a predicted heat map of the target.

[0025] For the target size, offset, three-dimensional size, depth and direction, a mean absolute error is set as a loss function for optimization, and specifically:

[0026]

[0027] In the formula, L represents a mean absolute error, and represents a real value of the target size, offset, three-dimensional size, depth and direction. MAE ​denotes the average absolute error loss value, N denotes the target number of image training data, f(x i ) denotes the first prediction result of the i-th target, y i denotes the true label of the i-th target;

[0028] When the average absolute error loss value and the focal loss value reach the minimum value, the trained first regression network is obtained.

[0029] In view of the problem that the point cloud data of the radar sensor is not accurate in height information, the enhanced point cloud data is mapped to an image coordinate system and expanded in a rectangular shape to obtain an expanded point cloud rectangular surface, and the size and position information of the expanded point cloud rectangular surface in the image coordinate system plane are obtained.

[0030] Preferably, the first detection result of the target and the expanded point cloud rectangular surface are regionally associated by using a conical association method to obtain associated data.

[0031] The conical association method is to set a threshold τ d in the [-τ d ,τ d ] region range to associate all the expanded radar detection targets with the first detection result; the threshold τ d is calculated by taking the maximum and minimum values of the three-dimensional bounding box in depth or z-axis:

[0032]

[0033] In the formula, denotes the maximum value of the z-axis of the three-dimensional bounding box, denotes the minimum value of the z-axis of the three-dimensional bounding box;

[0034] If there are multiple radar detection targets within the threshold range, the point cloud rectangular surface with the smallest z-axis coordinate is selected; in addition, since the first detection result output by the trained first regression network is inaccurate due to the uncertainty of the depth estimation of the visual sensor used to obtain the image data, a relaxation factor δ is introduced to expand the association region and associate more points with the object, and the relaxation factor δ proportionally increases the threshold τ d .

[0035] Preferably, in the step S6, feature extraction is performed on the associated data to obtain a specific method for obtaining a radar branch feature:

[0036] For each point cloud rectangular surface associated with the first detection result of the target, a corresponding radar branch feature F radar is generated at the position, and the calculation formula is:

[0037]

[0038] where i = 1, 2, 3 represent three feature channels of radar branch features respectively; represents the radar branch feature of the jth target in the ith feature channel, M i represents the normalization factor in the ith feature channel, f i represents the depth or radial velocity feature value in the ith feature channel, x and y represent the components of target velocity in the horizontal direction and the vertical direction respectively, and represents the center point of the jth target, w j and h j represents the width and height of the two-dimensional bounding box of the jth target; γ is a hyperparameter for controlling the size of the target two-dimensional bounding box.

[0039] Preferably, in the step S7, the visual branch features and the radar branch features are input into a constructed context clustering fusion network for feature fusion to obtain fusion features; the formula for feature fusion is:

[0040]

[0041] where F fused represents the fusion features, Φ CCN represents the context clustering fusion network, F radar represents the radar branch features, F camera represents the visual branch features, represents feature splicing.

[0042] The context clustering fusion network can adaptively adjust the clustering center, thereby effectively capturing the feature correlation information between different modalities to obtain the final fusion features. Target detection using the fusion features makes the results more accurate and more robust. In addition, the context clustering fusion network adopts a structure without convolution and self-attention mechanism, which greatly reduces the network parameters and improves the training speed and detection efficiency of the network. The context clustering fusion network mainly consists of two parts, feature aggregation and feature assignment, which cluster the input features into multiple clusters. The feature points of each cluster will be aggregated and then assigned back. Context clustering is to give a set of feature points P is linearly projected to P s for calculating the similarity between features; then c cluster centers are set in the feature space, the features of the cluster centers are calculated by averaging the nearest k points, and P s and the cosine similarity between the center point set Since the distance between features and the feature similarity are highlighted implicitly when calculating the similarity, each feature point is then assigned to the most similar center, and finally c clusters are generated, thereby achieving the effect of feature aggregation, and a fully connected layer is used to re-assign the aggregated features back to the original dimension.

[0043] Preferably, in the step S8, the fusion features are input into the trained secondary regression network to recalculate the depth, speed, direction and category of the target.

[0044] For the depth, speed and direction of the target, the mean absolute error is still used as the loss function for supervised training.

[0045] For the category, a cross-entropy loss function is set for optimization, specifically:

[0046]

[0047]

[0048] In the formula, L BCE represents the cross-entropy loss value, and sigma (x i ) represents the second prediction result of the i-th target.

[0049] When the mean absolute error loss value and the cross-entropy loss value reach the minimum, the trained secondary regression network is obtained, and the second prediction result of the corresponding i-th target is taken as the second detection result of the target.

[0050] Preferably, in the step S9, the preset decoder is a bounding box decoder.

[0051] The application also provides a target detection system based on context clustering and multi-modal fusion, which is used to implement the target detection method described above, and comprises:

[0052] A data acquisition processing module is configured to acquire paired image data and point cloud data, and perform data enhancement processing on the image data and the point cloud data to obtain data-enhanced image data and point cloud data.

[0053] A visual feature extraction module is configured to perform feature extraction on the data-enhanced image data to obtain visual branch features.

[0054] A first detection module is configured to calculate corresponding heat maps and attribute parameters based on the visual branch features to obtain a first detection result of the target.

[0055] A point cloud extension module is configured to map the data-enhanced point cloud data to an image coordinate system and perform rectangular extension to obtain an extended point cloud rectangular surface.

[0056] A multi-modal data association module is configured to associate the first detection result of the target with the extended point cloud rectangular surface to obtain association data;

[0057] A radar feature extraction module is configured to extract features from the association data to obtain radar branch features;

[0058] A feature fusion module is configured to fuse the visual branch features and the radar branch features to obtain fused features;

[0059] A second detection module is configured to calculate a second detection result of the target based on the fused features;

[0060] A target detection module is configured to input the first detection result of the target and the second detection result of the target into a preset decoder to obtain a final detection result of the target.

[0061] Compared with the prior art, the technical scheme of the present application has the following advantages:

[0062] The present application first acquires paired image data and point cloud data and performs data enhancement processing, extracts features from the image data after data enhancement processing to obtain visual branch features, calculates corresponding heat maps and attribute parameters based on the visual branch features to obtain a first detection result of the target, then maps the point cloud data after data enhancement to an image coordinate system and performs rectangular extension to obtain an extended point cloud rectangular surface, and associates the extended point cloud rectangular surface with the first detection result of the target to obtain association data, extracts features from the association data to obtain radar branch features, fuses the features of the two modalities of the visual branch features and the radar branch features, calculates a second detection result of the target based on the fused features, and finally inputs the first detection result of the target and the second detection result of the target into a preset decoder to obtain a final detection result of the target. The present application utilizes multi-modal feature data for fusion and complementation, thereby improving the accuracy and efficiency of target detection. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 A method flowchart of a target detection method based on context clustering and multi-modal fusion according to Embodiment 1.

[0064] Figure 2 A framework schematic diagram of a target detection method based on context clustering and multi-modal fusion according to Embodiment 2.

[0065] Figure 3 A schematic diagram of an extended point cloud rectangular surface according to Embodiment 2.

[0066] Figure 4 A principle schematic diagram of a conical association method according to Embodiment 2.

[0067] Figure 5 The final detection result map of the two-dimensional target in scenario one described in embodiment 2.

[0068] Figure 6 The final detection result map of the two-dimensional target in scenario two described in embodiment 2.

[0069] Figure 7 The final detection result map of the two-dimensional target in scenario three described in embodiment 2.

[0070] Figure 8 The final detection result map of the three-dimensional target in scenario one described in embodiment 2.

[0071] Figure 9 The final detection result map of the three-dimensional target in scenario two described in embodiment 2.

[0072] Figure 10 The final detection result map of the three-dimensional target in scenario three described in embodiment 2.

[0073] Figure 11 The structure diagram of a target detection system based on context clustering and multi-modal fusion described in embodiment 3. DETAILED DESCRIPTION

[0074] The accompanying drawings are only used for illustrative purposes and should not be construed as limiting the patent;

[0075] In order to better illustrate the embodiments, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0076] For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0077] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0078] Embodiment 1

[0079] The present embodiment provides a target detection method based on context clustering and multi-modal fusion, as shown in Figure 1 , comprising:

[0080] S1: Obtain paired image data and point cloud data, and perform data enhancement processing on the image data and point cloud data to obtain data-enhanced image data and point cloud data;

[0081] S2: Extract features from the data-enhanced image data to obtain visual branch features;

[0082] S3: based on the visual branch feature, calculate the corresponding heat map and attribute parameter, obtain the first detection result of the target;

[0083] S4: map the data enhanced point cloud data to the image coordinate system and perform rectangular expansion, obtain the expanded point cloud rectangular surface;

[0084] S5: regionally associate the first detection result of the target and the expanded point cloud rectangular surface, obtain the associated data;

[0085] S6: feature extraction is performed on the associated data to obtain radar branch features;

[0086] S7: the visual branch feature and the radar branch feature are fused to obtain the fusion feature;

[0087] S8: based on the fusion feature, calculate the second detection result of the target;

[0088] S9: input the first detection result of the target and the second detection result of the target into a preset decoder to obtain the final detection result of the target.

[0089] In the specific implementation process, the embodiment first acquires paired image data and point cloud data and performs data enhancement processing, extracts features from the data enhanced image data, and obtains visual branch features; based on the visual branch features, calculate the corresponding heat map and attribute parameter, obtain the first detection result of the target; then map the data enhanced point cloud data to the image coordinate system and perform rectangular expansion, obtain the expanded point cloud rectangular surface, and regionally associate the first detection result of the target, obtain the associated data; feature extraction is performed on the associated data to obtain radar branch features; the features of the two modalities of the visual branch features and the radar branch features are fused, and the second detection result of the target is calculated based on the fusion feature; finally, the first detection result and the second detection result of the target are input into a preset decoder to obtain the final detection result of the target. The embodiment utilizes multi-modal feature data for fusion and complementation, improving the accuracy and efficiency of target detection.

[0090] Embodiment 2

[0091] The embodiment provides a target detection method based on context clustering and multi-modal fusion, as shown in Figure 2 , comprising:

[0092] S1: acquire paired image data and point cloud data, and perform data enhancement processing on the image data and point cloud data to obtain data enhanced image data and point cloud data;

[0093] Specifically, the image data is acquired by using a visual sensor, and the point cloud data is acquired by using a radar sensor; the data enhancement processing on the image data and the point cloud data includes data random horizontal flipping and data random movement; in the embodiment, the probability of data random horizontal flipping is set to 50%, and the range of data random movement is set to 0% to 20%;

[0094] S2: input the image data after data enhancement into an existing deep aggregation network model to perform feature extraction, and obtain visual branch features;

[0095] S3: input the visual branch features into the trained first regression network to calculate a heat map and attribute parameters represented by the visual branch features, and obtain a first detection result of the target; the attribute parameters include target size, offset, three-dimensional size, depth and direction;

[0096] The specific method for obtaining the trained first regression network is as follows:

[0097] The visual branch features of the image training data are acquired and input into the constructed first regression network to calculate corresponding heat maps, target size, offset, three-dimensional size, depth and direction;

[0098] For the heat map, a focal loss function is set for supervised training, and specifically:

[0099]

[0100] In the formula, L k represents the focal loss value, N represents the number of targets of the image training data, Y xyc represents the real heat map of the target, represents the predicted heat map of the target, and alpha and beta represent the first and second hyperparameters of the focal loss, respectively;

[0101] For the target size, offset, three-dimensional size, depth and direction, the mean absolute error is set as a loss function for optimization, and specifically:

[0102]

[0103] In the formula, L MAE represents the mean absolute error loss value, N represents the number of targets of the image training data, f(x i ) represents the first prediction result of the i-th target, y i represents the real label of the i-th target;

[0104] When the mean absolute error loss value and the focal loss value reach the minimum value, the trained first regression network is obtained.

[0105] S4: mapping the data-enhanced point cloud data into an image coordinate system and performing rectangular expansion to obtain an expanded point cloud rectangular surface; as shown in Figure 3 , in view of the problem that the point cloud data of the radar sensor is inaccurate in height information, the data-enhanced point cloud data is mapped into an image coordinate system and rectangular expansion is performed to obtain an expanded point cloud rectangular surface, and the size and position information of the expanded point cloud rectangular surface in the image coordinate system plane are obtained;

[0106] S5: using a cone correlation method to perform regional correlation between the first detection result of the target and the expanded point cloud rectangular surface to obtain correlation data; as shown in Figure 4 , the cone correlation method is to set a threshold τ d in the center of the three-dimensional bounding box of the target, and all expanded radar detection targets are associated with the first detection result within the range of [-τ d ,τ d ]; the threshold τ d is calculated by taking the maximum and minimum values of the three-dimensional bounding box in depth or z-axis:

[0107]

[0108] In the formula, represents the maximum value of the three-dimensional bounding box in the z-axis, represents the minimum value of the three-dimensional bounding box in the z-axis;

[0109] If there are multiple radar-detected targets within the threshold range, the point cloud rectangular surface with the smallest z-axis coordinate is selected; in addition, due to the uncertainty of the depth estimation of the visual sensor used to obtain image data, the first detection result output by the trained first regression network is inaccurate, so a relaxation factor δ is introduced to expand the correlation area and associate more points with the object, and the relaxation factor δ proportionally increases the threshold τ d ;

[0110] S6: performing feature extraction on the correlation data to obtain radar branch features; specifically:

[0111] For each point cloud rectangular surface associated with the first detection result of the target, a corresponding radar branch feature F radar is generated at the position, and the calculation formula is:

[0112]

[0113] In the formula, i=1, 2, 3 represent three feature channels of the radar branch feature; represents the radar branch feature of the jth target in the ith feature channel, M i represents the normalization factor in the ith feature channel, and fi denotes the depth or radial velocity feature value on the i-th feature channel, x and y represent the components of the target velocity in the horizontal and vertical directions, respectively, and denotes the center point of the j-th target, w j and h j denotes the width and height of the two-dimensional bounding box of the j-th target; γ is a hyperparameter for controlling the size of the target two-dimensional bounding box;

[0114] S7: input the visual branch feature and the radar branch feature into the constructed context clustering fusion network for feature fusion to obtain a fused feature; the formula for feature fusion is:

[0115]

[0116] In the formula, F fused denotes the fused feature, Φ CCN denotes the context clustering fusion network, F radar denotes the radar branch feature, F camera denotes the visual branch feature, denotes feature splicing;

[0117] The context clustering fusion network can adaptively adjust the clustering center, thereby effectively capturing the feature correlation information between different modalities to obtain the final fused feature. Target detection using the fused feature makes the result more accurate and more robust. In addition, the context clustering fusion network adopts a structure without convolution and self-attention mechanism, and the network parameter is greatly reduced, thereby improving the training speed and detection efficiency of the network. The context clustering fusion network mainly consists of two parts, namely feature aggregation and feature assignment, which cluster the input features into multiple clusters. The feature points of each cluster will be aggregated and then assigned back. The context clustering is to give a set of feature points P is linearly projected to P s for calculating the similarity between features; then, c cluster centers are set in the feature space, the features of the cluster centers are calculated by averaging the nearest k points, and P s and the cosine similarity between the center point set Since the distance between features and the feature similarity are highlighted implicitly when calculating the similarity, then each feature point is assigned to the most similar center, and finally c clusters are generated, thereby achieving the effect of feature aggregation. A fully connected layer is used to reassign the aggregated features to the original dimension.

[0118] Compared with other fusion networks such as convolutional neural networks and self-attention networks, the context clustering fusion network has a significantly reduced network parameter based on the context clustering fusion network structure, which makes the model have a faster target detection speed on a device with low computing power.

[0119] The context clustering fusion network can adaptively adjust the clustering center, thereby effectively capturing the feature correlation information between different modalities to obtain the final fusion feature, and using the fusion feature for target detection makes the result more accurate and more robust.

[0120] In addition, the context clustering fusion network has more explainability in the result of feature fusion due to the use of a simplified clustering algorithm.

[0121] S8: The fusion feature is input into the trained secondary regression network to recalculate the depth, speed, direction and category of the target;

[0122] For the depth, speed and direction of the target, the mean absolute error is still used as the loss function for supervised training;

[0123] For the category, a cross-entropy loss function is set for optimization, specifically:

[0124]

[0125]

[0126] In the formula, L BCE represents the cross-entropy loss value, σ(x i ) represents the second prediction result of the i-th target;

[0127] When the mean absolute error loss value and the cross-entropy loss value reach the minimum, the trained secondary regression network is obtained, and the corresponding second prediction result of the i-th target is taken as the second detection result of the target;

[0128] S9: The first detection result of the target and the second detection result of the target are input into a preset bounding box decoder to obtain the final detection result of the target.

[0129] In the specific implementation process, the method provided in the embodiment is implemented and verified on the disclosed Nuscenes dataset. As shown in Figures 5-7 , it is the final detection result graph of the two-dimensional target in the selected three scenes, as shown in Figures 8-10 , it is the final detection result graph of the three-dimensional target in the selected three scenes. From Figures 5-10 , it can be seen that the method provided in the embodiment can accurately detect two-dimensional targets and three-dimensional targets in different scenes.

[0130] Embodiment 3

[0131] The embodiment also provides a target detection system based on context clustering and multi-modal fusion, which is used to implement the target detection method in the embodiment 1 or 2, as shown in Figure 11 The target detection system comprises:

[0132] a data acquisition processing module, configured to acquire paired image data and point cloud data, and perform data enhancement processing on the image data and the point cloud data to obtain data-enhanced image data and point cloud data;

[0133] a visual feature extraction module, configured to perform feature extraction on the data-enhanced image data to obtain visual branch features;

[0134] a first detection module, configured to calculate corresponding heat maps and attribute parameters based on the visual branch features to obtain a first detection result of a target;

[0135] a point cloud extension module, configured to map the data-enhanced point cloud data to an image coordinate system and perform rectangular extension to obtain an extended point cloud rectangular surface;

[0136] a multi-modal data association module, configured to perform regional association on the first detection result of the target and the extended point cloud rectangular surface to obtain associated data;

[0137] a radar feature extraction module, configured to perform feature extraction on the associated data to obtain radar branch features;

[0138] a feature fusion module, configured to perform feature fusion on the visual branch features and the radar branch features to obtain fused features;

[0139] a second detection module, configured to calculate a second detection result of the target based on the fused features;

[0140] a target detection module, configured to input the first detection result of the target and the second detection result of the target into a preset decoder to obtain a final detection result of the target.

[0141] The same or similar reference numerals correspond to the same or similar components;

[0142] The terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the patent;

[0143] Obviously, the above embodiments of the present application are merely exemplary but not intended to limit the embodiments of the present application. Based on the above description, any other variations or changes can be made by those skilled in the art without departing from the spirit and principles of the present application. It is not necessary to list all the embodiments here. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall fall within the scope of the claims of the present application.

Claims

1. A target detection method based on context clustering and multi-modal fusion, characterized in that, The method comprises the following steps: S1: obtaining paired image data and point cloud data, and performing data enhancement processing on the image data and the point cloud data to obtain data-enhanced image data and point cloud data; S2: performing feature extraction on the data-enhanced image data to obtain visual branch features; S3: based on the visual branch features, calculating corresponding heat maps and attribute parameters to obtain a first detection result of the target; S4: mapping the data-enhanced point cloud data to an image coordinate system and performing rectangular expansion to obtain an expanded point cloud rectangular surface, and obtaining size and position information of the expanded point cloud rectangular surface in the image coordinate system plane; S5: performing regional association on the first detection result of the target and the expanded point cloud rectangular surface to obtain associated data; S6: performing feature extraction on the associated data to obtain radar branch features, and the specific method is as follows: For each point cloud rectangular surface associated with the first detection result of the target, a corresponding radar branch feature is generated at the position , the calculation formula is: wherein, respectively represent three feature channels of radar branch features; respectively represent radar branch features of the th target in the th feature channel, respectively represent normalization factors in the th feature channel, respectively represent depth or radial velocity feature values in the th feature channel, respectively represent components of target velocity in horizontal and vertical directions, respectively represent components of target velocity in horizontal and vertical directions, respectively represent components of target velocity in horizontal and vertical directions, respectively represent components of target velocity in horizontal and vertical directions, respectively represent center points of the th target, respectively represent center points of the th target, is a hyper-parameter for controlling the size of the two-dimensional bounding box of the target; S7: performing feature fusion on the visual branch features and the radar branch features to obtain fused features; S8: based on the fused features, calculating a second detection result of the target; S9: inputting the first detection result of the target and the second detection result of the target into a preset decoder to obtain a final detection result of the target. 2.The target detection method based on context clustering and multi-modal fusion according to claim 1, characterized in that, In the step S1, the data enhancement processing performed on the image data and the point cloud data comprises data random horizontal flipping and data random movement. 3.The target detection method based on context clustering and multi-modal fusion according to claim 1, characterized in that, In the step S2, the image data after data enhancement is input into an existing deep aggregation network model to extract features and obtain visual branch features .

4. The target detection method based on context clustering and multi-modal fusion according to claim 1 or 3, characterized in that, In the step S3, the visual branch features are input into a trained first regression network to calculate heat maps and attribute parameters represented by the visual branch features, thereby obtaining the first detection result of the target; the attribute parameters comprise target size, offset, three-dimensional size, depth and direction. 5.The target detection method based on context clustering and multi-modal fusion according to claim 4, characterized in that, The specific method for obtaining the trained first regression network is as follows: The visual branch features of image training data are input into the constructed first regression network to calculate corresponding heat maps, target size, offset, three-dimensional size, depth and direction; For the heat maps, a focal loss function is set for supervised training, and the specific method is as follows: wherein, denotes a focal loss value, denotes a target number of image training data, denotes a real heat map of a target, denotes a predicted heat map of a target, and denote first and second hyperparameters of the focal loss, respectively; For the target size, offset, three-dimensional size, depth and direction, the mean absolute error is set as a loss function for supervised training, and the specific method is as follows: In the formula, represents the average absolute error loss value, represents the target number of image training data, represents the first prediction result of the th target, represents the true label of the th target; When the mean absolute error loss value and the focal loss value reach the minimum value, the trained first regression network is obtained. 6.The target detection method based on context clustering and multi-modal fusion according to claim 4, characterized in that, The first detection result of the target and the expanded point cloud rectangular surface are associated in regions by using a conical association method to obtain associated data. 7.The target detection method based on context clustering and multi-modal fusion according to claim 1, characterized in that, In the step S7, the visual branch features and the radar branch features are input into a constructed context clustering fusion network for feature fusion to obtain fused features; The formula for feature fusion is as follows: In the step S8, the fused features are input into a trained second regression network to recalculate the depth, speed, direction and category of the target; In the formula, denotes fusion features, denotes a context clustering fusion network, denotes radar branch features, denotes visual branch features, denotes feature concatenation. 8.The target detection method based on context clustering and multi-modal fusion according to claim 7, characterized in that, For the depth, speed and direction of the target, the mean absolute error is still used as a loss function for supervised training; For the category, a cross-entropy loss function is set for optimization, and the specific method is as follows: The method comprises the following steps: In the formula, represents a cross-entropy loss value, represents the second prediction result of the i th target; When the average absolute error loss value and the cross-entropy loss value reach the minimum, a trained quadratic regression network is obtained, and the corresponding second prediction result of the first target is taken as the second detection result of the target. the first target.

9. A target detection system based on context clustering and multi-modal fusion, for implementing the target detection method of any one of claims 1-8, characterized in that, A data acquisition and processing module is configured to obtain paired image data and point cloud data, and perform data enhancement processing on the image data and the point cloud data to obtain data-enhanced image data and point cloud data; ​ a visual feature extraction module, configured to perform feature extraction on the data-enhanced image data to obtain visual branch features; a first detection module, configured to calculate a corresponding heat map and attribute parameter based on the visual branch features to obtain a first detection result of the target; a point cloud extension module, configured to map the data-enhanced point cloud data to an image coordinate system and perform rectangular extension to obtain an extended point cloud rectangular surface; a multi-modal data association module, configured to perform regional association on the first detection result of the target and the extended point cloud rectangular surface to obtain associated data; a radar feature extraction module, configured to perform feature extraction on the associated data to obtain radar branch features; a feature fusion module, configured to perform feature fusion on the visual branch features and the radar branch features to obtain fused features; a second detection module, configured to calculate a second detection result of the target based on the fused features; a target detection module, configured to input the first detection result of the target and the second detection result of the target into a preset decoder to obtain a final detection result of the target.

Images