A target detection method, device and medium based on multi-modal feature fusion

Abstract

The application discloses a target detection method and device based on multi-modal feature fusion and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: 1. acquiring multi-modal images of a target through a binocular camera, obtaining an RGB image and a depth image of the target object, performing data enhancement and target position labeling, and thus constructing a multi-modal data set; 2. constructing a target detection network based on multi-modal feature fusion, comprising a multi-modal double-branch feature extraction network, a multi-modal feature fusion module, a multi-scale feature aggregation module, a feature pyramid network, a target detection head, and performing processing on the RGB image and the depth image to obtain a prediction result; and 3. constructing a total loss function based on the prediction result and a real label to train the target detection network based on multi-modal feature fusion, obtaining a trained multi-modal target detection model to realize target detection. The application can improve the accuracy and reliability of target detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a target detection method, device and medium based on multi-modal feature fusion. BACKGROUND

[0002] At present, the traditional image target detection method mainly depends on single modal image information, and the common one is visible light image. It is convenient to obtain and simple to process, and can realize basic target detection in a simple scene. However, this single modal detection method has obvious limitations. The visible light image is greatly affected by the light condition, and under complex light conditions such as strong light, weak light and backlight, the target feature is easy to change, which leads to blurred image edge and reduced contrast, and the algorithm is difficult to accurately extract the feature, resulting in false detection or missed detection. At the same time, the single modal visible light image is difficult to fully reflect the complex features of the target. The color, texture and shape of the target in different states change, and it is difficult to accurately capture them only by the visible light image. For the targets with similar appearance, it is also difficult to distinguish them only according to the visible light image.

[0003] These limitations make the target detection method based on single modal visible light image have insufficient accuracy and reliability of the detection result in complex practical application scenes, which limits the wider application and development of the target detection technology. SUMMARY

[0004] The application is proposed to solve the above-mentioned problems of the prior art, and provides a target detection method, device and medium based on multi-modal feature fusion, so as to break through the limitation that the target feature extraction of single modal image is not comprehensive and easy to be disturbed in a complex environment, comprehensively and accurately capture the multi-dimensional features of the target in different states and different scenes by comprehensively utilizing various modal image information, significantly improve the accuracy and robustness of target detection in complex and changeable application scenes, reduce the false detection rate and the missed detection rate, provide more reliable and efficient technical support for the intelligent application of various industries based on target detection, and promote the development and application upgrading of related fields.

[0005] In order to achieve the above-mentioned application purposes, the application adopts the following technical solutions.

[0006] The target detection method based on multi-modal feature fusion has the characteristics that the method comprises the following steps.

[0007] Step 1, acquiring an RGB image and a depth image and real label information thereof;

[0008] Step 2: Construct a target object detection network, including: a multimodal bi-branch feature extraction network, a multimodal feature fusion module, a multi-scale feature aggregation module, a feature pyramid network, and a target detection head. The multimodal spatial attention module in the multimodal feature fusion module is obtained using equation (1). Visible light spatial features at the x-th scale and Depth spatial features at the x-th scale and finally obtained and The prediction results;

[0009] (1)

[0010] In Equation (1), Conv represents the convolution operation, Maxpooling and Averagepooling represent the global average pooling operation and the global maximum pooling operation, respectively, Concat represents the concatenation operation, and Sigmoid represents the activation function. express The visible light feature at the x-th scale, express The depth image features at the x-th scale;

[0011] Step 3, based on and Based on the real-name annotation information and prediction results, construct the total loss function of the target object detection network. It is used to train a target object detection network to obtain a multimodal target detection model, thereby realizing the detection of multimodal target data.

[0012] The target detection method based on multimodal feature fusion described in this invention is also characterized in that step 1 includes:

[0013] Step 1.1: Obtain an RGB image of the target object and its corresponding depth image, and perform data augmentation to obtain an augmented RGB image. and an enhanced depth image ;and The resolution is H×W×3. The resolution is H×W×1, where, The height of the image. 3 represents the width of the image; 3 represents the number of channels in the RGB image of the target object; 1 represents the number of channels in the depth image of the target object.

[0014] Steps 1 and 2: respectively and The target location is marked to obtain the actual annotation information, including: the first one bounding box wherein, the center point coordinates of , , the width and height of , respectively.

[0015] Further, the step 2 includes:

[0016] Step 2.1, the multi-modal double-branch feature extraction network is composed of an RGB modal backbone feature extraction network and a depth modal feature extraction network, and respectively performs feature extraction on and , and correspondingly obtains X-scale visible light features and X-scale depth image features , wherein, represents the xth scale visible light feature, represents the xth scale depth image feature, and X represents the total number of scales;

[0017] Step 2.2, the multi-modal feature fusion module fuses and to obtain the xth scale fusion feature ;

[0018] Step 2.3, the multi-scale feature aggregation module performs multi-scale aggregation on to obtain the corresponding multi-scale aggregated feature ; wherein, represents the xth scale aggregated feature;

[0019] Step 2.4, the feature pyramid network processes to obtain the multi-scale pyramid feature , wherein, represents the xth scale pyramid feature;

[0020] Step 2.5, the target detection head processes to output the prediction results of and , including: the jth prediction box , wherein, represents the center point coordinates of , and respectively represent the width and height of ; represents the confidence of .

[0021] Further, step 2.1 includes:

[0022] Step 2.1.1, the RGB modal backbone feature extraction network includes: X CSP feature extraction modules;

[0023] When x = 1, the xth CSP feature extraction module processes the visible light image to obtain the visible light feature of the xth scale ;

[0024] When x = 2, 3, …, X, the visible light feature of the x-1th scale is input into the xth CSP feature extraction module for processing to obtain the visible light feature of the xth scale , so that the visible light feature of the Xth scale is output from the Xth CSP feature extraction module ;

[0025] Step 2.1.2, the depth modal feature extraction network includes: X CSP feature extraction modules, and the process of step 2.1.1 is performed on to obtain X scale depth image features .

[0026] Further, the multi-modal feature fusion module in step 2.2 further includes a feature interaction module, which obtains the xth scale fusion feature using formula (2):

[0027] (2)

[0028] In formula (2), represents element multiplication, and represents element addition.

[0029] Further, the multi-scale feature aggregation module in step 2.3 is obtained using formula (3): :

[0030] (3)

[0031] In formula (3), Conv represents convolution operation, Concat represents concatenation operation; SSU represents scale unification operation; represents element addition, and Attention represents attention mechanism module.

[0032] Further, step 3 includes:

[0033] Step 3.1, the xth loss function is constructed using formula (4): :

[0034] (4)

[0035] In formula (4), represents the first position loss, and is obtained by formula (5), represents the first confidence loss, and is obtained by formula (6);

[0036] (5)

[0037] In formula (5), represents the 2-order Wassersterin distance, represents a set constant;

[0038] (6)

[0039] In formula (6), represents the true label of whether the jth prediction frame contains the target object, if , it represents that the jth prediction frame contains the target object, if , it represents that the jth prediction frame does not contain the target object, and the BCE represents the binary cross entropy loss;

[0040] Step 3.2, constructing a total loss function by using formula (7) :

[0041] (7)

[0042] In formula (7), J represents the number of detection frames.

[0043] The electronic device comprises a memory and a processor, wherein the memory is used for storing a program supporting the processor to execute the target detection method, and the processor is configured to execute the program stored in the memory.

[0044] The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to execute the steps of the target detection method.

[0045] Compared with the prior art, the beneficial effects of the present application are as follows:

[0046] ​1、The present application focuses on the field of target detection, and successfully overcomes the inherent limitations of single modal image in target detection by fusing multi-modal image data. Traditional methods excessively rely on single modal image, and in complex scenes such as light change and target occlusion, they are easily disturbed, resulting in detection precision difficult to meet actual needs. The present application acquires multi-modal images of the target with the aid of binocular cameras, and carefully constructs a multi-modal data set after registration, data enhancement, and labeling and other processing procedures. This innovative way of multi-modal feature fusion can capture the appearance, spatial position and depth information of the target from all directions and angles. Especially in the face of dense target occlusion, the present application shows significant advantages, effectively solves the occlusion problem, and greatly improves the precision and accuracy of the detection model under complex conditions such as occlusion and weak light, providing a more scientific and accurate basis for decision-making based on target detection in fields such as agricultural production and industrial detection.

[0047] 2、The present application constructs a target detection network based on multi-modal feature fusion, which includes a multi-modal double-branch feature extraction network, a multi-modal feature fusion module, a multi-scale feature aggregation module, a feature pyramid network and a target detection head and other key components, realizing efficient and accurate detection of the target. Compared with traditional methods, the present application realizes high intelligence and automation in the feature extraction and fusion link. It can automatically extract the most representative features from multi-modal images, and further strengthen the detection ability of the model for targets of different scales through multi-scale feature aggregation and feature pyramid network. This innovative network structure design not only greatly improves the detection efficiency, but also significantly improves the accuracy and robustness of the detection, providing strong support for accurate target detection and promoting the high-quality development of related industries. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the method of the present application;

[0049] Figure 2 is a structure diagram of the network of the present application;

[0050] Figure 3 is a detection result diagram of the present application. DETAILED DESCRIPTION

[0051] In this embodiment, a target detection method based on multi-modal feature fusion, as shown in Figure 1 , includes the following steps:

[0052] Step 1, construct a multi-modal data set:

[0053] Step 1.1, acquire an RGB image of wheat in a field and its corresponding wheat depth image and perform data enhancement to obtain an enhanced wheat RGB image and the enhanced one of the wheat depth image ; and the resolution of the RGB image is 640x640x3, the resolution of the depth image is 640x640x1;

[0054] Step 1.2, labeling the target position of the wheat in and to obtain the real label information, including: the first labeling box , wherein, represents the center point coordinates of , , are the width and height of , respectively;

[0055] Since the image has been aligned when it is obtained, only the RGB image needs to be labeled, that is, the labeling of the multi-modal image can be realized. The construction of the multi-modal data set can provide more comprehensive information for the model. The RGB image contains color, texture and other visual information, and the depth image provides spatial distance information, which helps the model better understand the wheat scene and improve the detection accuracy.

[0056] Step 2, construct a wheat detection network based on multi-modal feature fusion, such as Figure 2 shows the corresponding network structure diagram, including: a multi-modal double-branch feature extraction network, a multi-modal feature fusion module, a multi-scale feature aggregation module, a feature pyramid network, and a target detection head:

[0057] Step 2.1, the multi-modal double-branch feature extraction network is composed of an RGB modality backbone feature extraction network and a depth modality feature extraction network, and performs feature extraction operations on and respectively:

[0058] Step 2.1.1, the RGB modality backbone feature extraction network includes 5 CSP feature extraction modules;

[0059] When x=1, the first CSP feature extraction module processes to obtain the visible light feature of the first scale ;

[0060] When x=2, 3, …, 5, the visible light feature of the (x-1)th scale is input into the xth CSP feature extraction module for processing to obtain the visible light feature of the xth scale , so that the visible light feature of the 5th scale is output from the 5th CSP feature extraction module;

[0061] When the pixel size of the input multi-modal image selection is 640x640, 5 layers of feature extraction modules are selected, and multi-scale feature images under different modalities can be effectively extracted.

[0062] Step 2.1.2, the depth modal feature extraction network processes the depth image according to the process of step 2.1.1 using 5 CSP feature extraction modules, and obtains 5 scales of depth image features , wherein , x represents the xth scale of the depth image feature.

[0063] Similarly, the corresponding depth modal feature extraction network is constructed, which can extract the features of the wheat depth image synchronously to obtain the corresponding depth image features.

[0064] The dual-branch feature extraction network can extract effective features for RGB images and depth images respectively, the CSP module can reduce the amount of calculation, improve the feature extraction efficiency, and at the same time retain more feature information.

[0065] Step 2.2, the multi-modal feature fusion module respectively fuses and , and , and to obtain the corresponding 3rd, 4th and 5th scale fusion features , , .

[0066] In this embodiment, the multi-modal feature fusion module comprises a multi-modal spatial attention module and a feature interaction module.

[0067] The modal spatial attention module obtains the xth scale of the visible light spatial feature and the xth scale of the depth spatial feature using formula (1).

[0068] (1)

[0069] In formula (1), Conv represents convolution operation, Maxpooling and Averagepooling represent global average pooling operation and global maximum value pooling operation respectively, Concat represents splicing operation, and Sigmoid represents activation function.

[0070] Formula (1) represents the operation process of the spatial attention mode, and via the spatial attention mode, the corresponding visible light spatial feature and depth spatial feature can be obtained. ​

[0071] The feature interaction module obtains the fused feature of the xth scale by formula (2) .

[0072] (2)

[0073] In formula (2), represents element multiplication, represents element addition.

[0074] Formula (2) represents the operation process of the feature interaction module. Via the feature interaction module, the visible light space feature and the depth space can be effectively fused. Meanwhile, the residual operation is introduced, which can increase the fusion effect and prevent overfitting.

[0075] In this embodiment, the multi-modal features of the high-layer 3rd, 4th and 5th scales are selected for multi-modal feature fusion, that is, the fine texture features after feature extraction can be effectively obtained, and the multi-scale multi-modal features can be effectively fused, thereby improving the fusion effect of the multi-modal feature fusion module. The multi-modal feature fusion module can fully utilize the feature information of the RGB image and the depth image, pay attention to important feature regions through the spatial attention mechanism, and the feature interaction module can effectively fuse different modal features and improve the detection precision.

[0076] Step 2.3, the multi-scale feature aggregation module aggregates the features of the 3rd, 4th and 5th scales to obtain corresponding multi-scale aggregated features . ; wherein, represents the aggregated feature of the xth scale;

[0077] The multi-scale feature aggregation module obtains the aggregated feature of the xth scale by formula (3) :

[0078] (3)

[0079] In formula (3), Conv represents a convolution operation, Concat represents a concatenation operation, and SSU represents a scale unification operation. represents element addition, and CA represents a coordinate attention mechanism module.

[0080] In this embodiment, the features of the 3rd, 4th and 5th scales are selected for feature fusion to obtain the corresponding aggregated features of the 3rd, 4th and 5th scales .

[0081] Meanwhile, the operation formula of the specific SSU operation can be simplified as formula (3a) below, After the SSU operation, the corresponding aligned features can be obtained .

[0082] (3a)

[0083] In formula (3a), 、 、 respectively represent the input and output of the three scales, and Concat represents feature concatenation. Upsample represents a feature up-sampling operation; Maxpooling represents a maximum value pooling operation.

[0084] The multi-scale aggregation operation is performed on the multi-modal fusion features of the third, fourth and fifth scales of the high layer, which can effectively fuse the multi-modal fusion features at different scales and further enhance the feature exchange between different scale features. The multi-scale feature aggregation module can integrate feature information at different scales, improve the detection capability of the model for different size targets, and enhance the robustness of the model.

[0085] Step 2.4, the feature pyramid network processes to obtain multi-scale pyramid features , wherein, represents the pyramid feature of the xth scale;

[0086] The feature pyramid network can further improve the detection capability of the model for different scale targets, so that the model can better handle target detection under occlusion, weak light and other conditions.

[0087] Step 2.5, the target detection head processes to output the prediction results of and , including the jth prediction box , wherein, represents the center point coordinates of , respectively represent the width and height of ; represents the confidence of ; the target detection head converts the features extracted by the model into specific detection results, and the confidence information filters out reliable detection results.

[0088] Step 3, construct the total loss function of the multi-modal feature fusion wheat detection network ; the total loss function comprehensively considers the positioning loss and the confidence loss, which can guide the model to optimize the position and classification accuracy of the detection box during the training process, and improve the detection performance of the model.

[0089] Step 3.1, construct the first loss function using formula (4):

[0090] ​ (4)

[0091] In formula (4), represents the jth positioning loss, and is obtained by formula (5), represents the jth confidence loss, and is obtained by formula (6):

[0092] (5)

[0093] In formula (5), represents the 2nd order Wassersterin distance, represents a set constant;

[0094] The NWD loss function is selected as the positioning loss, which performs particularly well when dealing with very small targets, can significantly reduce the false detection of wheat, and improves the detection accuracy.

[0095] (6)

[0096] In formula (6), represents the jth prediction box whether it contains the target object, if , it means that the jth prediction box contains the target object, if , it means that the jth prediction box does not contain the target object, and BCE represents the binary cross entropy loss.

[0097] Step 3.2, constructing the total loss function by formula (7) :

[0098] (7)

[0099] In formula (7), J represents the number of detection boxes in the multi-modal image label.

[0100] Step 4, iteratively training the wheat detection network based on multi-modal feature fusion by the SGD optimizer, and calculating the total loss function to adjust the network parameters until the total loss function converges, thereby obtaining the trained multi-modal wheat detection network for detecting multi-modal wheat data.

[0101] The wheat detection network based on multi-modal feature fusion is iteratively trained by the SGD (Stochastic Gradient Descent) optimizer, and the total loss function is calculated to adjust the network parameters until the total loss function converges. During the training process, the learning rate is set to 0.001, the batch size is set to 16, and the training is performed for 100 rounds.​​

[0102] Step 5, after obtaining the trained multi-modal wheat detection network, the multi-modal wheat data is detected through the trained network:

[0103] A new RGB image as shown in the left image of Figure 3 and a depth image as shown in the middle image of Figure 3 are input, the network outputs the detection result of the wheat ear, including the coordinate, width and height and confidence of the prediction box and other information. And using the related information, a detection frame is drawn, and the result is as shown in the right image of Figure 3 . The occluded part and dense part of the wheat in the image do not have the problem of missing detection, and the wheat ear detection result is good.

[0104] Experimental results: on the constructed multi-modal wheat dataset, the overall precision of the model is 0.859, the recall rate is 0.889, and the average precision mean mAP is 0.931. Compared with the single modal detection method, the detection precision and accuracy of the method of the application in the case of occlusion, weak light and the like are obviously improved. In the case of an occlusion rate of 30%, the detection accuracy of the method of the application is improved by 10% compared with the single modal method, and in the case of different light, the detection performance difference of the method of the application is less than 5%.

[0105] Therefore, the method of the application can quickly and accurately detect the position and quantity of the wheat ear, and provide strong support for wheat yield estimation, disease and pest monitoring and the like.

[0106] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above target detection method, and a processor configured to execute the program stored in the memory.

[0107] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above target detection method.

Claims

1. A target detection method based on multimodal feature fusion, characterized in that, Includes the following steps: Step 1: Obtain the RGB image and depth images and its actual annotation information; Step 2: Construct the target object detection network, including: a multimodal bi-branch feature extraction network, a multimodal feature fusion module, a multi-scale feature aggregation module, a feature pyramid network, and a target detection head; Step 2.1: The multimodal dual-branch feature extraction network consists of an RGB modality backbone feature extraction network and a deep modality feature extraction network, which are used to extract features from the network. and Feature extraction is performed, resulting in visible light features at X scales. and depth image features at X scales ,in, This represents the visible light characteristics at the x-th scale. Let x represent the depth image feature at the x-th scale, where X represents the total number of scales; Step 2.2, the multimodal feature fusion module will... and Perform fusion to obtain the fusion feature at the x-th scale. ; Among them, the multimodal spatial attention module in the multimodal feature fusion module obtains the result using equation (1). Visible light spatial features at the x-th scale and Depth spatial features at the x-th scale ; (1) In Equation (1), Conv represents the convolution operation, Maxpooling and Averagepooling represent the global average pooling operation and the global maximum pooling operation, respectively, Concat represents the concatenation operation, and Sigmoid represents the activation function. express The visible light feature at the x-th scale, express The depth image features at the x-th scale; The feature interaction module in the multimodal feature fusion module obtains the fused features at the x-th scale using equation (2). : (2) In equation (2), Represents element-wise multiplication. Represents the addition of elements; Step 2.3, the multi-scale feature aggregation module will... Perform multi-scale aggregation to obtain the corresponding multi-scale aggregated features. ;in, This represents the aggregated feature at the x-th scale; Step 2.4, Feature Pyramid Network Pair The process is performed to obtain multi-scale pyramid features. ,in, Represents the pyramid features at the x-th scale; Step 2.5, Target Detection Head Pair Process and output. and The prediction results include: the j-th prediction box ,in, represent The coordinates of the center point, They represent Width and height; represent Confidence level; Step 3, based on and Based on the real-name annotation information and prediction results, construct the total loss function of the target object detection network. It is used to train a target object detection network to obtain a multimodal target detection model, thereby realizing the detection of multimodal target data.

2. The target detection method based on multimodal feature fusion according to claim 1, characterized in that, Step 1 includes: Step 1.1: Obtain an RGB image of the target object and its corresponding depth image, and perform data augmentation to obtain an augmented RGB image. and an enhanced depth image ;and The resolution is H×W×3. The resolution is H×W×1, where, The height of the image. 3 represents the width of the image; 3 represents the number of channels in the RGB image of the target object; 1 represents the number of channels in the depth image of the target object. Steps 1 and 2: respectively and The target location in the image is marked to obtain the actual annotation information, including: the first A label box ,in, express The coordinates of the center point, , They are respectively Width and height.

3. The target detection method based on multimodal feature fusion according to claim 2, characterized in that, Step 2.1 includes: Step 2.1.1: The RGB modality backbone feature extraction network includes X CSP feature extraction modules; When x=1, the x-th CSP feature extraction module... The processing yields the visible light features at the x-th scale. ; The visible light characteristics at the (x-1)th scale when x = 2, 3, ..., X The input is processed in the x-th CSP feature extraction module to obtain the visible light features at the x-th scale. Thus, the Xth CSP feature extraction module outputs the visible light features at the Xth scale. ; Step 2.1.2: The deep modal feature extraction network includes X CSP feature extraction modules, and follows the process in step 2.1.1 to extract features. The image is processed to obtain depth image features at X scales. .

4. The target detection method based on multimodal feature fusion according to claim 1, characterized in that, The multi-scale feature aggregation module in step 2.3 uses equation (3) to obtain the aggregated features at the x-th scale. : (3) In equation (3), Conv represents the convolution operation and Concat represents the concatenation operation; SSU stands for Standardized Operation; "Attention" represents the addition of elements, and "Attention" represents the attention mechanism module.

5. The target detection method based on multimodal feature fusion according to claim 1, characterized in that, Step 3 includes: Step 3.1: Construct the first equation using equation (4). loss function : (4) In equation (4), Indicates the first The positioning loss is obtained from equation (5). Indicates the first The confidence loss is calculated using equation (6). (5) In equation (5), This represents the second-order Wassersterin distance. This represents a set constant; (6) In equation (6), Represents the j-th prediction box Does it contain the actual tags of the target object? , indicating that the j-th prediction box contains the target object, if , indicating that the j-th prediction box does not contain the target object, and BCE represents binary cross-entropy loss; Step 3.2: Construct the total loss function using equation (7). : (7) In equation (7), J represents the number of detection boxes.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the target detection method according to any one of claims 1-5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the target detection method according to any one of claims 1-5.

Images