Target detection method and device based on multi-modal fusion, electronic equipment and medium

Abstract

The invention discloses a target detection method and device based on multi-modal fusion, electronic equipment and a medium, and the method comprises the steps: obtaining a first image set and a second image set, the first image set comprises a plurality of visible light images, and the second image set comprises a plurality of infrared images; performing data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set; performing feature fusion on the first feature image set and the second feature image set through a preset dual-scale image fusion algorithm to obtain a fused image set; and performing target detection on the fused image in the fused image set through a preset deep learning model to obtain the object category of the to-be-detected object and mark the bounding box of the to-be-detected object. In the embodiment of the invention, the accuracy of target detection can be improved, and the detection precision is further improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a target detection method, device, electronic device and storage medium based on multimodal fusion. Background Art

[0002] With the rapid development of drone technology, drones are playing an increasingly important role in many key areas. They have shown great potential and value in public safety monitoring, natural disaster relief, agricultural monitoring, traffic management, etc. In these diverse application scenarios, single target detection technology plays a vital role. Traditional drone target tracking systems use single-channel images such as visible light images or infrared images as input sources. The images collected by visible light technology have rich texture information and can enhance the detailed features of small targets, but they are easily affected by occlusion and insufficient light. Infrared technology can capture the thermal radiation information of objects and has strong anti-interference ability. It can be used for drone small target detection in complex backgrounds such as weak light, dark light, blur, and objects that are partially blocked. However, infrared images are single-channel images and it is difficult to provide detailed information of the target scene. In practical applications, their detection performance is often difficult to meet the needs of practical applications.

[0003] In order to overcome these limitations, multimodal technology has emerged. This technology can provide richer and more accurate target information by integrating data from different sensors, such as visible light, infrared, and lidar. Although multimodal data can provide more comprehensive perception capabilities, existing technologies find it difficult to achieve cross-modal fusion of features. At the same time, the computing resources of the embedded platforms carried by drones are limited, which further limits their ability to process complex algorithms. Summary of the invention

[0004] The embodiments of the present application provide a target detection method, device, electronic device and storage medium based on multimodal fusion, which can realize cross-modal fusion of features, improve the accuracy of target detection, and further improve the detection precision.

[0005] In a first aspect, an embodiment of the present application provides a target detection method based on multimodal fusion, comprising: Acquire a first image set and a second image set, wherein the first image set includes a plurality of visible light images, the second image set includes a plurality of infrared images, and at least a portion of the visible light images and at least a portion of the infrared images include objects to be detected; Performing a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set; Performing feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set; The fused images in the fused image set are subjected to target detection by using a preset deep learning model to obtain the object category of the object to be detected and the bounding box marking the object to be detected.

[0006] In some embodiments, performing a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set includes: Performing image denoising processing on all infrared images in the second image set; Time synchronization is performed on all the visible light images and the infrared images after image noise reduction processing by using a preset timestamp synchronization algorithm to obtain a first target image set and a second target image set corresponding to the first image set; Extracting a first feature point set of the first target image set and extracting a second feature point set of the second target image set by using a preset scale-invariant feature transformation algorithm; Performing feature point matching on the first feature point set and the second feature point set using a preset matching algorithm to obtain a matching result; A first feature image set is determined according to the first feature point set, and the infrared images in the second target image set are transformed according to the matching result to obtain a second feature image set.

[0007] In some embodiments, the step of performing feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set includes: Selecting a first feature image from the first feature image set, and selecting a second feature image from the second feature image set; Decomposing the first feature image by a mean filter to obtain a first base layer feature and a first detail layer feature, and decomposing the second feature image by a mean filter to obtain a second base layer feature and a second detail layer feature, wherein the first base layer feature is used to characterize the structural feature of the first feature image, the first detail layer feature is used to characterize the high-frequency information of the first feature image, the second base layer feature is used to characterize the structural feature of the second feature image, and the second detail layer feature is used to characterize the high-frequency information of the second feature image; The first base layer features and the second base layer features are fused by a preset dual-scale image fusion algorithm to generate a base layer image, and the first detail layer and the second detail layer are weighted averaged to obtain a detail layer image; Reconstructing the base layer image and the detail layer image to obtain a fused image; The fused images corresponding to all the feature images in the first feature image set and the second feature image set are sorted to obtain a fused image set.

[0008] In some embodiments, the deep learning model includes a fully connected layer; the performing target detection on the fused image in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and mark the bounding box of the object to be detected includes: Inputting the fused image set into a preset deep learning model, so that the deep learning model extracts features from the fused images in the fused image set and outputs feature information; The feature information is input into the fully connected layer, so that the fully connected layer predicts the category of the object to be detected in the fused image according to the feature information, and obtains the object category of the object to be detected and marks the bounding box of the object to be detected.

[0009] In some embodiments, after performing target detection on the fused image in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and marking the bounding box of the object to be detected, the method further includes: The object to be detected is continuously tracked by a preset multi-target tracking algorithm.

[0010] In some embodiments, the time synchronization of all the visible light images and the infrared images after image noise reduction is performed by a preset timestamp synchronization algorithm to obtain a first target image set and a second target image set corresponding to the first image set includes: Recording a first timestamp set of all the visible light images and a second timestamp set of all the infrared images after image noise reduction processing; Comparing first timestamps of adjacent visible light images according to the first timestamp set to obtain a first comparison difference, and comparing second timestamps of adjacent infrared images after image noise reduction processing according to the second timestamp set to obtain a second comparison difference; When the first contrast difference is greater than a preset time window, determining a first visible light image and a second visible light image corresponding to the first contrast difference, and performing interpolation processing on the first visible light image and the second visible light image by a preset interpolation method to obtain a first intermediate image, and adding the first intermediate image to the first image set to obtain a first target image set; When the second contrast difference is greater than a preset time window, the first infrared image and the second infrared image corresponding to the second contrast difference are determined, and the first infrared image and the second infrared image are interpolated by a preset interpolation method to obtain a second intermediate image, and the second intermediate image is added to the second image set to obtain a second target image set.

[0011] In some embodiments, after recording the first time stamp set of all the visible light images and the second time stamp set of all the infrared images after image noise reduction processing, the method further includes: Performing continuity detection on the first timestamp set and the second timestamp set; When there are discontinuous timestamps in the first timestamp set, determine a third visible light image and a fourth visible light image corresponding to the discontinuous timestamps, generate a first padding image according to the third visible light image and the fourth visible light image, and add the first padding image between the third visible light image and the fourth visible light image; When there are discontinuous timestamps in the second timestamp set, determine the third infrared image and the fourth infrared image corresponding to the discontinuous timestamps, generate a second complement image based on the third infrared image and the infrared image, and add the second complement image between the third infrared image and the fourth infrared image.

[0012] In some embodiments, after performing target detection on the fused image in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and marking the bounding box of the object to be detected, the method further includes: Analyzing a plurality of model weights of the deep learning model, wherein the model weights are used to characterize the importance of different branches of the deep learning model; Performing weight screening on all the model weights using a preset pruning strategy to screen out target weights that are less than a pruning threshold of the pruning strategy; The deep learning model is pruned according to the target weight to obtain a target deep model.

[0013] In some embodiments, after performing target detection on the fused image in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and marking the bounding box of the object to be detected, the method further includes: Perform a model quantization operation on the deep learning model.

[0014] In a second aspect, an embodiment of the present application provides a target detection device based on multimodal fusion, comprising: An image acquisition module, configured to acquire a first image set and a second image set, wherein the first image set includes a plurality of visible light images, the second image set includes a plurality of infrared images, and at least a portion of the visible light images and at least a portion of the infrared images include objects to be detected; A data alignment module, configured to perform a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set; A feature fusion module, used to perform feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set; The target detection module is used to perform target detection on the fused image in the fused image set through a preset deep learning model to obtain the object category of the object to be detected and to mark the boundary box of the object to be detected.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target detection method based on multimodal fusion as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the target detection method based on multimodal fusion as described in the first aspect.

[0017] The target detection method, device, electronic device and storage medium based on multimodal fusion provided by the embodiments of the present application have at least the following beneficial effects: First, a first image set and a second image set are obtained, wherein the first image set includes multiple visible light images and the second image set includes multiple infrared images, thereby avoiding the problems of low resolution, blurred target edge contours and low classification recognition accuracy caused by using infrared images alone. Since there may be differences in the sampling frequency and sampling angle of the image, the embodiments of the present application perform data alignment operations on the first image set and the second image set to achieve spatial and temporal alignment of the images, reduce noise and outliers caused by factors such as shooting angles and lighting conditions, ensure data consistency and integrity, obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set, and generate clearer images so that subsequent target detection algorithms can more accurately identify and locate targets. The first feature image set and the second feature image set are feature-fused by a preset dual-scale image fusion algorithm, which makes full use of the complementary characteristics of different image modalities, enhances the robustness and accuracy of the system in complex environments, and obtains a fused image set, which can quickly generate high-quality fused images to meet real-time requirements, so that targets can be more accurately identified and tracked under different environmental conditions, such as illumination changes or occlusions. Afterwards, the fused images in the fused image set are subjected to target detection by a preset deep learning model, and the object category of the object to be detected and the bounding box marking the object to be detected are obtained, so as to achieve accurate detection of the object to be detected and improve the accuracy and reliability of target detection. The embodiment of the present application can provide more accurate and robust target detection results under various lighting conditions by fusing multimodal data of visible light and infrared images, and can effectively retain and fuse important information in images of different modalities by introducing a dual-scale image fusion algorithm for image fusion, thereby improving the accuracy and robustness of target detection.

[0018] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a specific method flow chart of a target detection method based on multimodal fusion provided by an embodiment of the present application; Figure 2 is a flowchart of a specific method for performing data alignment operation on a first image set and a second image set provided by an embodiment of the present application; Figure 3It is a flowchart of a specific method for performing feature fusion on a first feature image set and a second feature image set by using a preset dual-scale image fusion algorithm provided in an embodiment of the present application; Figure 4 It is a flowchart of a specific method for performing target detection on a fused image in a fused image set by using a preset deep learning model provided in an embodiment of the present application; Figure 5 is a specific method flow chart of a target detection method based on multimodal fusion provided by another embodiment of the present application; Figure 6 It is a flowchart of a specific method for performing time synchronization on all visible light images and infrared images after image noise reduction processing by using a preset timestamp synchronization algorithm provided by an embodiment of the present application; Figure 7 is a specific method flow chart of a target detection method based on multimodal fusion provided by another embodiment of the present application; Figure 8 is a specific method flow chart of a target detection method based on multimodal fusion provided by another embodiment of the present application; Fig. 9 is a specific method flow chart of a target detection method based on multimodal fusion provided by another embodiment of the present application; Fig.10 is a schematic diagram of a target detection device based on multimodal fusion provided by an embodiment of the present application; Fig.11 It is a schematic diagram of the framework of a dual-scale image fusion algorithm provided in an embodiment of the present application; Fig.12 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0020] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.

[0021] In the description of the present invention, if there is a description of first and second, it is only for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0022] With the rapid development of drone technology, drones are playing an increasingly important role in many key areas. They have shown great potential and value in public safety monitoring, natural disaster relief, agricultural monitoring, traffic management, etc. In these diverse application scenarios, single target detection technology plays a vital role. Traditional drone target tracking systems use single-channel images such as visible light images or infrared images as input sources. The images collected by visible light technology have rich texture information and can enhance the detailed features of small targets, but they are easily affected by occlusion and insufficient light. Infrared technology can capture the thermal radiation information of objects and has strong anti-interference ability. It can be used for drone small target detection in complex backgrounds such as weak light, dark light, blur, and objects that are partially blocked. However, infrared images are single-channel images and it is difficult to provide detailed information of the target scene. In practical applications, their detection performance is often difficult to meet the needs of practical applications.

[0023] In order to overcome these limitations, multimodal technology has emerged. This technology can provide richer and more accurate target information by integrating data from different sensors, such as visible light, infrared, and lidar. Although multimodal data can provide more comprehensive perception capabilities, existing technologies find it difficult to achieve cross-modal fusion of features. At the same time, the computing resources of the embedded platforms carried by drones are limited, which further limits their ability to process complex algorithms.

[0024] In order to solve the above problems, the present embodiment provides a target detection method, device, electronic device and medium based on multimodal fusion, which has at least the following beneficial effects: First, a first image set and a second image set are obtained, wherein the first image set includes multiple visible light images and the second image set includes multiple infrared images, thereby avoiding the problems of low resolution, blurred target edge contours and low classification recognition accuracy caused by using infrared images alone. Since there may be differences in the sampling frequency and sampling angle of the image, the embodiment of the present application performs data alignment operations on the first image set and the second image set to achieve spatial and temporal alignment of the images, reduce noise and outliers caused by factors such as shooting angles and lighting conditions, ensure data consistency and integrity, obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set, and generate clearer images so that subsequent target detection algorithms can more accurately identify and locate targets. The first feature image set and the second feature image set are feature-fused by a preset dual-scale image fusion algorithm, which makes full use of the complementary characteristics of different image modalities, enhances the robustness and accuracy of the system in complex environments, and obtains a fused image set, which can quickly generate high-quality fused images to meet real-time requirements, so that targets can be more accurately identified and tracked under different environmental conditions, such as illumination changes or occlusions. Afterwards, the fused images in the fused image set are subjected to target detection by a preset deep learning model, and the object category of the object to be detected and the bounding box marking the object to be detected are obtained, so as to achieve accurate detection of the object to be detected and improve the accuracy and reliability of target detection. The embodiment of the present application can provide more accurate and robust target detection results under various lighting conditions by fusing multimodal data of visible light and infrared images, and can effectively retain and fuse important information in images of different modalities by introducing a dual-scale image fusion algorithm for image fusion, thereby improving the accuracy and robustness of target detection.

[0025] Reference Figure 1 , Figure 1 It is a specific method flow chart of a target detection method based on multimodal fusion provided in one embodiment of the present application. The target detection method based on multimodal fusion includes but is not limited to steps S101 to S104.

[0026] Step S101, acquiring a first image set and a second image set, wherein the first image set includes a plurality of visible light images, the second image set includes a plurality of infrared images, and at least a portion of the visible light images and at least a portion of the infrared images include objects to be detected.

[0027] In step S101 of some embodiments, the current tracking algorithm mainly relies on visible light images for monitoring. In actual application, the detection ability of visible light images in nighttime or harsh and changeable environments is limited. Images taken by visible light cameras are easily affected by strong light, reflections and other light, especially in dimly lit environments at night, where clear images are almost impossible to obtain. Although infrared thermal imaging cameras alone have the advantages of night vision and insensitivity to light, the resolution of the infrared images they take is low, resulting in blurred target edge contours and low classification and recognition accuracy. Therefore, the application of infrared detection technology has certain limitations. In order to avoid the above problems, the embodiment of the present application first obtains a first image set of multiple visible light images, and obtains a second image set of multiple infrared images, so that based on the complementary characteristics of infrared and visible light images, the texture information provided by the visible light images and the thermal radiation information provided by the infrared images can be fully utilized, thereby improving the tracking accuracy and anti-interference ability at night or in harsh environments.

[0028] It can be understood that the visible light image in the embodiment of the present application is obtained by photographing with a visible light camera, and the infrared camera image is obtained by photographing with an infrared camera.

[0029] Step S102: performing a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set.

[0030] In step S102 of some embodiments, since the visible light camera and the infrared camera collect image data under different lighting and environmental conditions, the sampling frequency and viewing angle of these data may be different. In order to achieve high-quality data fusion, the embodiment of the present application will align the visible light images in the first image set and the infrared images in the second image set to ensure the consistency of the visible light images and the infrared images in time and space, and obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set, so as to facilitate the subsequent high-quality image fusion and facilitate the subsequent target detection and tracking algorithms to more accurately identify and locate the target.

[0031] Step S103: performing feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set.

[0032] In step S103 of some embodiments, feature fusion is performed on the first feature image set and the second feature image set using a preset dual-scale image fusion algorithm, and a fused image set is obtained by combining the advantages of visible light images and infrared images, thereby enhancing the feature information of the target and improving the accuracy and robustness of target detection.

[0033] Step S104: performing target detection on the fused image in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and the bounding box marking the object to be detected.

[0034] In step S104 of some embodiments, target detection is performed on the fused image in the fused image set through a preset deep learning model. The deep learning model effectively processes these multimodal data through its lightweight structure, and further integrates the features of the fused image to extract richer and more robust feature representations. Finally, the fused image is predicted to obtain the object category of the object to be detected and the bounding box to mark the object to be detected, thereby achieving accurate recognition of the object to be detected and improving recognition accuracy.

[0035] It should be noted that the deep learning model in the embodiment of the present application is the MobileNetV3-ssd model. MobileNetV3-ssd is used as the backbone network for target detection, directly processes the multimodal fusion image, and outputs the location and category of the target.

[0036] Reference Figure 2 , Figure 2 This is a flowchart of a specific method for performing data alignment operations on a first image set and a second image set provided by an embodiment of the present application. The method includes but is not limited to steps S201 to S205.

[0037] Step S201: performing image noise reduction processing on all infrared images in the second image set.

[0038] In step S201 of some embodiments, considering that infrared images are often affected by sensor noise and environmental interference, the embodiments of the present application will perform image noise reduction processing on all infrared images in the second image set. Specifically, the embodiments of the present application perform noise reduction processing on the infrared image through a Gaussian filter, and convolve the image with a Gaussian kernel to smooth the pixel values, thereby reducing the noise level, removing noise while reducing damage to image details, and improving the accuracy and reliability of target detection.

[0039] It should be noted that in the process of convolving the image with a Gaussian kernel, choosing the right kernel size and standard deviation is the key to achieving the best noise reduction effect, which can effectively remove noise while minimizing damage to image details. Since infrared images usually need to suppress noise while maintaining edge and texture features, this method makes it easier for subsequent target detection and tracking algorithms to more accurately identify and locate targets. It can generate clearer images and improve the accuracy and reliability of target detection.

[0040] Step S202 , time synchronization is performed on all visible light images and infrared images after image noise reduction processing by using a preset timestamp synchronization algorithm to obtain a first target image set and a second target image set corresponding to the first image set.

[0041] In step S202 of some embodiments, time synchronization is a key step to ensure the temporal consistency of image data. Since the sampling frequencies of visible light cameras and infrared cameras may be different, the collected images may have temporal differences. To solve this problem, the embodiments of the present application perform time synchronization on all visible light images and infrared images after image noise reduction through a preset timestamp synchronization algorithm to obtain a first target image set and a second target image set corresponding to the first image set, thereby ensuring the temporal consistency of visible light images and infrared images and avoiding temporal differences between visible light images and infrared images.

[0042] Step S203: extracting a first feature point set of the first target image set and extracting a second feature point set of the second target image set by using a preset scale-invariant feature transformation algorithm.

[0043] In step S203 of some embodiments, since the visible light camera and the infrared camera collect image data under different lighting and environmental conditions, the sampling frequency and viewing angle of these data may be different. The embodiment of the present application detects and describes the key points of the images in the first target image set through a preset scale-invariant feature transform algorithm (Scale-Invariant Feature Transform, SIFT), and detects and describes the key points of the images in the second target image set through a scale-invariant feature transform algorithm, thereby extracting the first feature point set of the first target image set and the second feature point set of the second target image set, extracting the feature points in the first target image set and the second target image set, and describing the feature points of the visible light image and the infrared image, so that the first target image set and the second target image set can remain stable under image transformation.

[0044] Specifically, the scale-invariant feature transformation algorithm of the embodiment of the present application is expressed as follows: = * ; in, It represents the scale, and its selection determines the blur degree of the image in scale space. Specifically, The smaller the value, the less smoothing, and the more image details are retained; The larger it is, the greater the smoothness is and the less detail information is in the image. represents the spatial coordinates of pixels within the image, Indicates that the image is at position The pixel intensity value at represents the Gaussian function.

[0045] Step S204: performing feature point matching on the first feature point set and the second feature point set using a preset matching algorithm to obtain a matching result.

[0046] In step S204 of some embodiments, after extracting the first feature point set and the second feature point set, the embodiments of the present application will also perform feature point matching on the first feature point set and the second feature point set through a preset matching algorithm, such as brute force matching, FLANN (Fast Library for Approximate Nearest Neighbors) matching, etc., to find the best correspondence between the two groups of feature points. Through feature point matching, the correspondence between the two images can be accurately found.

[0047] Step S205 , determining a first feature image set according to the first feature point set, and transforming the infrared images in the second target image set according to the matching result to obtain a second feature image set.

[0048] In step S205 of some embodiments, after obtaining the optimal correspondence between feature points through a matching algorithm, the embodiments of the present application transform the infrared image after image denoising in the second target image set according to the matched feature points, i.e., the matching results, so as to align the image with the visible light image in space, and obtain a second feature image set. At the same time, the embodiments of the present application can directly determine the first feature image set based on the first feature point set, thereby realizing feature extraction of the visible light image and the infrared image, and can maintain the spatial consistency between the visible light image and the infrared image, thereby realizing spatial alignment of the visible light image and the infrared image.

[0049] Reference Figure 3 , Figure 3 It is a flowchart of a specific method for performing feature fusion on a first feature image set and a second feature image set by using a preset dual-scale image fusion algorithm provided in an embodiment of the present application. The method includes but is not limited to steps S301 to S305.

[0050] Step S301 : selecting a first feature image from a first feature image set, and selecting a second feature image from a second feature image set.

[0051] Step S302, decomposing the first feature image through a mean filter to obtain a first base layer feature and a first detail layer feature, and decomposing the second feature image through a mean filter to obtain a second base layer feature and a second detail layer feature.

[0052] It should be noted that the first basic layer features are used to characterize the structural features of the first feature image, the first detail layer features are used to characterize the high-frequency information of the first feature image, the second basic layer features are used to characterize the structural features of the second feature image, and the second detail layer features are used to characterize the high-frequency information of the second feature image.

[0053] In steps S301 to S302 of some embodiments, in the process of feature fusion of the first feature image set and the second feature image set through a preset dual-scale image fusion algorithm, the embodiment of the present application first selects the first feature image in the first feature image set, and selects the second feature image in the second feature image set, and then decomposes the first feature image through a mean filter, decomposes the image into a base layer and a detail layer through the mean filter to obtain first base layer features and first detail layer features, and decomposes the second feature image through the mean filter to obtain second base layer features and second detail layer features, wherein the base layer represents the main structural features of the image, and the detail layer contains high-frequency information of the image, so that the key visual elements of the image can be extracted, laying the foundation for subsequent image fusion.

[0054] Step S303: The first base layer features and the second base layer features are fused by a preset dual-scale image fusion algorithm to generate a base layer image, and a weighted average operation is performed on the first detail layer and the second detail layer to obtain a detail layer image.

[0055] Step S304: reconstruct the base layer image and the detail layer image to obtain a fused image.

[0056] Step S305 , sorting the fused images corresponding to all the feature images in the first feature image set and the second feature image set to obtain a fused image set.

[0057] In steps S303 to S305 of some embodiments, based on image decomposition, the embodiments of the present application fuse the first base layer features and the second base layer features through a preset two-scale image fusion algorithm (Two-Scale Image Fusion, TIF), wherein a simple averaging strategy is adopted for the fusion of the base layer to combine the first base layer and the second base layer information to generate a base layer image, thereby generating a base layer image that combines the advantages of the two modal data, and performing a weighted averaging operation on the first detail layer and the second detail layer. Specifically, a weighted averaging strategy is adopted for the detail layer, and the fusion coefficient is adjusted according to the significance of the detail layer to obtain the detail layer image, thereby enhancing the detail features of the image, and then the base layer image and the detail layer image are reconstructed. Specifically, the final fused image is reconstructed by adding the fused base layer image and the detail layer image, which not only retains the main features of the image, but also enhances the detail performance of the image, so that the fused image is clearer and richer in edge contour and feature information. Afterwards, the fused images corresponding to all the feature images in the first feature image set and the second feature image set are sorted to obtain a fused image set, thereby quickly generating a high-quality fused image set to meet real-time requirements.

[0058] It can be understood that the fusion coefficient in the embodiment of the present application refers to the weight used for the contribution of two source images when the dual-scale image fusion algorithm combines visible light and infrared light into a single image.

[0059] Specifically, the mathematical expression of the TIF algorithm of the present application can be summarized by the following formula: I fusion = (x,y)+ ; Among them, I fusion represents the final fused image, (x,y) is the fused base layer, It is the detail layer after fusion.

[0060] Reference Figure 4 , Figure 4 This is a flowchart of a specific method for performing target detection on a fused image in a fused image set through a preset deep learning model provided in an embodiment of the present application. The method includes but is not limited to steps S401 to S402.

[0061] It should be noted that the deep learning model includes a fully connected layer.

[0062] Step S401: input the fused image set into a preset deep learning model, so that the deep learning model extracts features from the fused images in the fused image set and outputs feature information.

[0063] Step S402: input the feature information into the fully connected layer, so that the fully connected layer predicts the category of the object to be detected in the fused image according to the feature information, and obtains the object category of the object to be detected and a bounding box marking the object to be detected.

[0064] In steps S401 to S402 of some embodiments, in the process of performing target detection on the fused image in the fused image set through the preset deep learning model, the embodiment of the present application first inputs the fused image set into the preset deep learning model, so that the deep learning model performs feature extraction on the fused image in the fused image set. Specifically, the deep learning model extracts features through its multi-layer convolution and pooling operations, outputs feature information, and realizes the extraction of fusion features of the fused image, extracts richer and more robust feature representation, and then inputs the feature information into the fully connected layer, so that the fully connected layer predicts the category of the object to be detected in the fused image according to the feature information, predicts the problem category and bounding box of the object to be detected, obtains the object category of the object to be detected and the bounding box corresponding to the object to be detected, thereby obtaining the position and category of the object to be detected, and then marks the bounding box of the object to be detected, so as to realize accurate recognition of the object to be detected and improve the recognition accuracy.

[0065] Reference Figure 5 , Figure 5 It is a specific method flow chart of a target detection method based on multimodal fusion provided by another embodiment of the present application, and the method includes but is not limited to step S501.

[0066] It should be noted that step S501 occurs after target detection is performed on the fused image in the fused image set through a preset deep learning model to obtain the object category of the object to be detected and mark the bounding box of the object to be detected.

[0067] Step S501: continuously track the object to be detected using a preset multi-target tracking algorithm.

[0068] In step S501 of some embodiments, after obtaining the object category of the object to be detected and marking the bounding box of the object to be detected, the embodiments of the present application will also continuously track the object to be detected through a preset multi-target tracking algorithm, wherein the multi-target tracking algorithm in the embodiments of the present application is the DeepSORT (Deep Simple Online and Realtime Tracking) algorithm. DeepSORT combines the Kalman filter and the Hungarian algorithm, and can effectively handle the problems of target occlusion, overlap and re-appearance, thereby realizing continuous tracking and efficient monitoring of the object to be detected, and can ensure the real-time and efficiency of the algorithm while improving the tracking stability.

[0069] It should be noted that the Kalman filter is used to predict the state of the target in the next frame, while the Hungarian algorithm is used to solve the data association problem between the detection results and the predicted trajectory.

[0070] It can be understood that in the process of target detection and tracking of the object to be detected, the target tracking adopts a multimodal fusion target detection and tracking method that combines the lightweight MobileNetV3-ssd and DeepSORT algorithms, aiming to improve the detection accuracy and tracking stability while ensuring the real-time and efficiency of the algorithm.

[0071] Reference Figure 6 , Figure 6 This is a flowchart of a specific method for performing time synchronization on all visible light images and infrared images after image noise reduction processing by using a preset timestamp synchronization algorithm provided by an embodiment of the present application. The method includes but is not limited to steps S601 to S604.

[0072] Step S601 : recording a first time stamp set of all visible light images and a second time stamp set of all infrared images after image noise reduction processing.

[0073] Step S602 , comparing first timestamps of adjacent visible light images according to the first timestamp set to obtain a first comparison difference, and comparing second timestamps of adjacent infrared images after image noise reduction according to the second timestamp set to obtain a second comparison difference.

[0074] In steps S601 to S602 of some embodiments, in the process of time synchronization of all visible light images and infrared images after image denoising processing by a preset timestamp synchronization algorithm, since all visible light images and infrared images are recorded with timestamps during the acquisition process, the embodiments of the present application first record a first timestamp set of all visible light images and a second timestamp set of all infrared images after image denoising processing, so as to facilitate subsequent analysis of the temporal consistency of visible light images and infrared images, and then compare the first timestamps of adjacent visible light images according to the first timestamp set to compare the timestamp differences of adjacent visible light images to obtain a first comparison difference, and compare the second timestamps of adjacent infrared images after image denoising processing according to the second timestamp set to compare the timestamp differences of adjacent infrared images to obtain a second comparison difference, so as to facilitate subsequent analysis of whether there is a timestamp mismatch or missing.

[0075] Step S603: When the first contrast difference is greater than a preset time window, determine a first visible light image and a second visible light image corresponding to the first contrast difference, and interpolate the first visible light image and the second visible light image using a preset interpolation method to obtain a first intermediate image, and add the first intermediate image to the first image set to obtain a first target image set.

[0076] In step S603 of some embodiments, when the first contrast difference is greater than a preset time window, it indicates that the timestamp of the current visible light image does not match, and it is necessary to determine the first visible light image and the second visible light image corresponding to the first contrast difference, thereby determining the two visible light images with timestamp mismatches, and interpolating the first visible light image and the second visible light image through a preset interpolation method. Specifically, first obtain a first timestamp corresponding to the first visible light image and a second timestamp corresponding to the second visible light image, determine an interpolation time point that needs to be interpolated according to the first timestamp and the second timestamp, and then generate a first intermediate image through a linear interpolation formula, add the first intermediate image to the first image set, and obtain a first target image set, thereby ensuring the temporal continuity of the visible light images in the first target image set and ensuring the temporal consistency of the image data.

[0077] It should be noted that the preset time window in the embodiment of the present application refers to the expected time range within which the timestamps of two adjacent images should fall during continuous image acquisition, wherein the preset time window can be set according to the frequency of image acquisition. For example, if the image acquisition frame rate is 25 frames per second, then the ideal time interval between each frame is 1 / 25 second, i.e. 40 milliseconds, etc. The embodiment of the present application does not impose any specific restrictions.

[0078] Step S604, when the second contrast difference is greater than the preset time window, determine the first infrared image and the second infrared image corresponding to the second contrast difference, and interpolate the first infrared image and the second infrared image by a preset interpolation method to obtain a second intermediate image, add the second intermediate image to the second image set, and obtain a second target image set.

[0079] In step S604 of some embodiments, when the second contrast difference is greater than the preset time window, it indicates that the timestamp of the current infrared image does not match, and it is necessary to determine the first infrared image and the second infrared image corresponding to the second contrast difference to determine the two infrared images where the timestamps do not match, and interpolate the first infrared image and the second infrared image using a preset interpolation method. Specifically, first obtain the third timestamp corresponding to the first infrared image and the fourth timestamp corresponding to the second infrared image, determine the interpolation time point that needs to be interpolated based on the third timestamp and the fourth timestamp, and then generate a second intermediate image using a linear interpolation formula, add the second intermediate image to the second image set, and obtain a second target image set, thereby ensuring the temporal continuity of the infrared images in the second target image set and ensuring the temporal consistency of the image data.

[0080] Reference Figure 7 , Figure 7 It is a specific method flow chart of a target detection method based on multimodal fusion provided in another embodiment of the present application, and the method includes but is not limited to steps S701 to S703.

[0081] It should be noted that step S701 to step S703 occur after recording the first time stamp set of all visible light images and the second time stamp set of all infrared images after image noise reduction processing.

[0082] Step S701: Perform continuity detection on the first timestamp set and the second timestamp set.

[0083] In step S701 of some embodiments, after recording the first timestamp set and the second timestamp set, the embodiments of the present application will also perform continuity detection on the first timestamp set and the second timestamp set. Specifically, if the timestamps of the images are continuous, then the timestamps can be considered complete. If it is found that there are gaps between the timestamps, such as a sudden jump in a series of continuous timestamps, this may indicate that the timestamps of some images are missing. It is possible to determine whether the timestamps of the images are missing to ensure the integrity of the data.

[0084] Step S702: When there are discontinuous timestamps in the first timestamp set, determine the third visible light image and the fourth visible light image corresponding to the discontinuous timestamps, generate a first complementary image based on the third visible light image and the fourth visible light image, and add the first complementary image between the third visible light image and the fourth visible light image.

[0085] In step S702 of some embodiments, when there are discontinuous timestamps in the first timestamp set, it means that there are visible light images with missing timestamps in the first image set. At this time, the third visible light image and the fourth visible light image corresponding to the discontinuous timestamps are determined, and a first complementary image is generated based on the third visible light image and the fourth visible light image. Specifically, the embodiment of the present application can generate the first complementary image through a linear interpolation method, and add the first complementary image between the third visible light image and the fourth visible light image, so as to ensure the integrity of the data and avoid the occurrence of missing timestamps.

[0086] Step S703, when there are discontinuous timestamps in the second timestamp set, determine the third infrared image and the fourth infrared image corresponding to the discontinuous timestamps, generate a second complementary image according to the third infrared image and the infrared image, and add the second complementary image between the third infrared image and the fourth infrared image.

[0087] In step S703 of some embodiments, when there are discontinuous timestamps in the second timestamp set, it means that there are infrared images with missing timestamps in the second image set, and the third infrared image and the fourth infrared image corresponding to the discontinuous timestamps are determined, and a second complementary image is generated based on the third infrared image and the infrared image. Specifically, the embodiment of the present application can generate the second complementary image through a linear interpolation method, and add the second complementary image between the third infrared image and the fourth infrared image, so as to ensure the integrity of the data and avoid the situation of missing timestamps.

[0088] Reference Figure 8 , Figure 8 It is a specific method flow chart of a target detection method based on multimodal fusion provided in another embodiment of the present application, and the method includes but is not limited to steps S801 to S803.

[0089] It should be noted that steps S801 to S803 occur after target detection is performed on the fused image in the fused image set through a preset deep learning model to obtain the object category of the object to be detected and mark the bounding box of the object to be detected.

[0090] Step S801, analyzing multiple model weights of the deep learning model, wherein the model weights are used to characterize the importance of different branches of the deep learning model.

[0091] Step S802: weight screening is performed on all model weights using a preset pruning strategy to screen out target weights that are less than a pruning threshold of the pruning strategy.

[0092] Step S803: Perform a pruning operation on the deep learning model according to the target weight to obtain a target deep model.

[0093] In steps S801 to S803 of some embodiments, after obtaining the object category of the object to be detected and marking the bounding box of the object to be detected, the embodiments of the present application will also optimize the deep learning model. Specifically, multiple model weights of the deep learning model are analyzed, and then all model weights are weight-screened through a preset pruning strategy to screen out target weights that are less than the pruning threshold of the pruning strategy, thereby screening out the weights of connections and nodes that have less impact on the output, and then performing a pruning operation on the deep learning model according to the target weight, that is, pruning the connection corresponding to the target weight in the deep learning model to obtain a target deep model, thereby reducing the complexity and computational burden of the model.

[0094] It is understandable that in the embodiment of the present application, connections corresponding to weights greater than or equal to the pruning threshold will be retained.

[0095] Reference Fig. 9 , Fig. 9 It is a specific method flow chart of a target detection method based on multimodal fusion provided by another embodiment of the present application, and the method includes but is not limited to step S901.

[0096] It should be noted that step S901 occurs after target detection is performed on the fused image in the fused image set through a preset deep learning model to obtain the object category of the object to be detected and mark the bounding box of the object to be detected.

[0097] Step S901, performing a model quantization operation on the deep learning model.

[0098] In step S901 of some embodiments, the embodiments of the present application also perform a model quantization operation on the deep learning model, specifically, by converting the model parameters from 32-bit floating point numbers to 8-bit integers to reduce the amount of calculation and memory usage. The quantization process involves mapping the floating point parameters to the integer space, which not only reduces the size of the model, but also reduces the complexity of the calculation.

[0099] It is worth noting that in the quantization strategy, the embodiments of the present application adopt a symmetric quantization or asymmetric quantization method. Symmetric quantization evenly maps the value range of the parameter to the integer range, while asymmetric quantization performs non-uniform mapping according to the actual distribution of the parameter. This method can more effectively use a limited number of bits to represent the parameters, thereby reducing the storage requirements of the model while maintaining the accuracy of the model.

[0100] See also Fig.10 , Fig.10 It is a schematic diagram of a target detection device based on multimodal fusion provided by an embodiment of the present application.

[0101] In some embodiments, the present application also provides a target detection device based on multimodal fusion, the device comprising: An image acquisition module 901 is used to acquire a first image set and a second image set, wherein the first image set includes a plurality of visible light images, the second image set includes a plurality of infrared images, and at least a portion of the visible light images and at least a portion of the infrared images include objects to be detected; A data alignment module 902 is used to perform a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set; A feature fusion module 903 is used to perform feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set; The target detection module 904 is used to perform target detection on the fused image in the fused image set through a preset deep learning model to obtain the object category of the object to be detected and the bounding box marking the object to be detected.

[0102] In some embodiments, the target detection device based on multimodal fusion further includes a continuous tracking module, a continuity detection module and a model optimization module, wherein the continuous tracking module is used to continuously track the object to be detected through a preset multi-target tracking algorithm. The continuity detection module is used to perform continuity detection on the first timestamp set and the second timestamp set; when there are discontinuous timestamps in the first timestamp set, determine the third visible light image and the fourth visible light image corresponding to the discontinuous timestamps, and generate a first complementary image based on the third visible light image and the fourth visible light image, and add the first complementary image between the third visible light image and the fourth visible light image; and when there are discontinuous timestamps in the second timestamp set, determine the infrared image and the fourth infrared image corresponding to the discontinuous timestamps, and generate a second complementary image based on the third infrared image and the infrared image, and add the second complementary image between the third infrared image and the fourth infrared image. The model optimization module is used to analyze multiple model weights of the deep learning model, where the model weights are used to characterize the importance of different branches of the deep learning model; to filter all model weights through a preset pruning strategy to filter out target weights that are less than the pruning threshold of the pruning strategy; and to prune the deep learning model according to the target weights to obtain the target deep model. The model optimization module is also used to perform model quantization operations on the deep learning model.

[0103] The specific implementation of the target detection device based on multimodal fusion is basically the same as the specific implementation of the target detection method based on multimodal fusion mentioned above, and will not be repeated here.

[0104] In order to more clearly illustrate the above-mentioned target detection method based on multimodal fusion, a specific example is given below for illustration.

[0105] Example 1: Example 1 takes the application scenario of the target detection method based on multimodal fusion as an example of a multimodal target tracking system for unmanned aerial vehicles to illustrate.

[0106] First of all, the current drone target tracking algorithm mainly relies on visible light images for monitoring. In actual application, the detection ability of visible light images in nighttime or harsh and changeable environments is limited. The images taken by visible light cameras are easily affected by strong light, reflections and other light, especially in dimly lit environments at night, where clear images are almost impossible to obtain. Although the infrared thermal imaging camera alone has the advantages of night vision and insensitivity to light, the infrared image it captures has low resolution, resulting in blurred target edge contours and low classification and recognition accuracy. Therefore, the application of infrared detection technology has certain limitations. Therefore, the embodiment of the present application fuses visible light images and infrared images, and based on the complementary characteristics of infrared and visible light images, fully utilizes the texture information provided by visible light images and the thermal radiation information provided by infrared images, thereby improving tracking accuracy and anti-interference capabilities at night or in harsh environments.

[0107] Secondly, one of the key challenges in the target tracking algorithm of multimodal images is to extract features from multiple modalities that are helpful for target tracking. Although the existing image fusion methods from the perspective of drones have made some progress in multimodal image fusion detection, there are still difficult technical problems in further improving the false detection rate and detection accuracy in target detection in extreme environments.

[0108] Finally, the complex network leads to reduced real-time performance of the algorithm, while the embedded platform used by the drone has limited computing resources, which not only limits the real-time performance of the system, but also increases the demand for computing resources, thus affecting the drone's endurance and deployment flexibility.

[0109] Taking the above problem as an example, the target detection method based on multimodal fusion in an embodiment of the present application is described below.

[0110] In the multimodal data fusion system of drones, data preprocessing and synchronization are the basis for ensuring the effective operation of the system. The visible light camera and infrared camera carried by the drone collect image data under different lighting and environmental conditions, and the sampling frequency and viewing angle of these data may be different. Therefore, in order to achieve high-quality data fusion, the data must first be preprocessed and synchronized to ensure their consistency in time and space. This step is crucial for subsequent tasks such as feature extraction, target detection and tracking.

[0111] In some embodiments, image alignment is one of the key links in data preprocessing. The embodiment of the present application uses a scale-invariant feature transformation algorithm to extract feature points of visible light images and infrared images. The SIFT algorithm can detect and describe key points in an image, which remain stable under image transformations (such as rotation and scaling). By matching these feature points, the embodiment of the present application can achieve spatial alignment of images.

[0112] The specific implementation steps include: first, apply the SIFT algorithm to the visible light image and the infrared image respectively to extract the feature point set; then use the matching algorithm (such as brute force matching, FLANN matching, etc.) to find the best correspondence between the two sets of feature points; finally, transform the infrared image according to the matched feature points to align it with the visible light image in space. The SIFT algorithm formula is as follows: = * ; in, It represents the scale, and its selection determines the blur degree of the image in scale space. Specifically, The smaller the value, the less smoothing, and the more image details are retained; The larger it is, the greater the smoothness is and the less detail information is in the image. represents the spatial coordinates of pixels within the image, Indicates that the image is at position The pixel intensity value at represents the Gaussian function.

[0113] In addition, time synchronization is a key step to ensure the temporal consistency of image data. Since the sampling frequencies of visible light cameras and infrared cameras may be different, the captured images may differ in time. In order to solve this problem, the present invention adopts a timestamp synchronization algorithm to interpolate or fill in the image according to the acquisition timestamp of the image. The specific implementation steps include: recording the acquisition timestamp of each image; for images with mismatched timestamps, generating intermediate images by interpolation methods to ensure temporal continuity; for images with missing timestamps, generating missing images by filling methods to ensure data integrity.

[0114] In the multimodal data fusion system of UAV, denoising infrared images is a key step to improve image quality. Considering that infrared images are often affected by sensor noise and environmental interference, the use of Gaussian filters is an effective denoising method. With Gaussian filters, random noise can be reduced while retaining the main features of the image. This method reduces the noise level by convolving the image with a Gaussian kernel to smooth the pixel values. Choosing the right kernel size and standard deviation is the key to achieving the best denoising effect, which can effectively remove noise while minimizing damage to image details. This method is particularly suitable for infrared images because they usually need to suppress noise while maintaining edge and texture features so that subsequent target detection and tracking algorithms can more accurately identify and locate targets. In this way, the UAV system is able to generate clearer images and improve the accuracy and reliability of target detection.

[0115] In some embodiments, after preprocessing and synchronizing the data, the embodiments of the present application will perform feature fusion on the visible light image and the infrared image, and this process is described in detail below.

[0116] Reference Fig.11 , Fig.11 It is a schematic diagram of the framework of a dual-scale image fusion algorithm provided in an embodiment of the present application.

[0117] In the multimodal data fusion system of UAV, feature fusion is a crucial step. It enhances the feature information of the target by combining the advantages of visible light and infrared images, thereby improving the accuracy and robustness of target detection. Here, a dual-scale image fusion algorithm is used to achieve this goal.

[0118] The core of the TIF algorithm is to optimize the visual effect of the image through three steps: image decomposition, image fusion, and image reconstruction. For two images input at the same time, the mean filter is first used to decompose the image into a base layer and a detail layer. The base layer represents the main structural features of the image, while the detail layer contains the high-frequency information of the image. In this way, the algorithm can extract the key visual elements of the image and lay the foundation for subsequent fusion operations. Based on the image decomposition, the TIF algorithm further fuses the base layer and the detail layer. The fusion of the base layer adopts a simple averaging strategy to combine the base layer information of the visible and infrared images to generate a base layer image that combines the advantages of the two modal data. For the detail layer, the algorithm adopts a weighted averaging strategy to adjust the fusion coefficient according to the significance of the detail layer, thereby enhancing the detail features of the image. Finally, the final fused image is reconstructed by adding the fused base layer and detail layer. This method not only retains the main features of the image, but also enhances the detail performance of the image, making the fused image clearer and richer in edge contours and feature information. The advantages of the TIF algorithm lie in its ease of operation and high efficiency. In practical applications, especially in scenarios where drones detect urban road targets, this algorithm can quickly generate high-quality fused images to meet real-time requirements.

[0119] The mathematical expression of the TIF algorithm can be summarized by the following formula: I fusion = (x,y)+ ; Among them, I fusion represents the final fused image, (x,y) is the fused base layer, It is the detail layer after fusion.

[0120] In some embodiments, in the multimodal data fusion system of drones, target detection and tracking are key links to achieve efficient monitoring. Target tracking uses a multimodal fusion target detection and tracking method that combines lightweight MobileNetV3-ssd and DeepSORT algorithms, aiming to improve detection accuracy and tracking stability while ensuring the real-time and efficiency of the algorithm.

[0121] MobileNetV3 is an efficient deep learning model optimized for mobile and edge devices. It reduces the amount of computation and parameters through deep separable convolution and lightweight bottleneck blocks, thereby achieving fast inference while maintaining high accuracy. MobileNetV3-ssd replaces the VGG network in the traditional ssd algorithm with MobileNetV3. In this study, MobileNetV3-ssd is used as the backbone network for target detection, directly processing multimodal fusion images and outputting the location and category of the target. The process of target detection involves inputting the fused image into the MobileNetV3-ssd network. The network extracts features through its multi-layer convolution and pooling operations, and predicts the category and bounding box of the target through the final fully connected layer.

[0122] Based on target detection, the DeepSORT algorithm is used to achieve continuous tracking of targets. DeepSORT combines the Kalman filter and the Hungarian algorithm to effectively handle the problems of target occlusion, overlap, and reappearance. The Kalman filter is used to predict the state of the target in the next frame, while the Hungarian algorithm is used to solve the data association problem between the detection results and the predicted trajectory.

[0123] In addition, in UAV multimodal data fusion systems, model optimization is key to ensure that the algorithm runs efficiently on resource-constrained devices. This paper proposes an optimization strategy that combines model pruning and quantization to reduce the computational requirements and memory usage of the model while maintaining detection accuracy.

[0124] In this study, the present invention adopts a pruning strategy based on weight size or sparsity. This method analyzes the importance of weights and selectively removes those connections and nodes that have little impact on the final output, thereby reducing the complexity and computational burden of the model.

[0125] When implementing pruning, the present invention first determines the pruning threshold, and then screens the model weights according to the threshold. Connections with weights less than the threshold will be pruned, while connections with larger weights will be retained. Model quantization is another optimization technique that reduces the amount of computation and memory usage by converting model parameters from 32-bit floating point numbers to 8-bit integers. The quantization process involves mapping floating point parameters to integer space, which not only reduces the size of the model but also reduces the complexity of the calculation. In the quantization strategy, the present invention adopts symmetric quantization or asymmetric quantization methods. Symmetric quantization uniformly maps the value range of the parameter to the integer range, while asymmetric quantization performs non-uniform mapping based on the actual distribution of the parameter. This method can more effectively use a limited number of bits to represent parameters, thereby reducing the storage requirements of the model while maintaining the accuracy of the model.

[0126] Through model pruning and quantization, the size and computational requirements of the model can be significantly reduced. The pruned model reduces the number of parameters and computational complexity while maintaining high detection accuracy. Quantization further reduces the model's memory usage and computational requirements, making the model more suitable for deployment on resource-constrained drone platforms.

[0127] The multimodal target tracking system of the drone of the embodiment of the present application uses innovative multimodal data fusion technology, combined with visible light and infrared images, to significantly improve the accuracy and robustness of target detection. The system uses the TIF algorithm, lightweight MobileNetV3-ssd and DeepSORT algorithms, which not only achieves efficient and high-precision target detection and continuous tracking, but also optimizes resource usage through model pruning and quantization technology, ensuring the real-time and efficiency of the algorithm on resource-constrained drone platforms. This comprehensive technical solution enables drones to operate stably under various complex environmental conditions, and is widely used in urban traffic monitoring, public safety, disaster monitoring and other fields.

[0128] Reference Fig.12 , taking the control processor 1001 and memory 1002 in the electronic device 1000 as an example that can be connected via a bus. The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk memory, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the control processor 1001, and these remote memories may be connected to the electronic device 1000 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0129] Those skilled in the art will understand that Fig.12 The device structure shown in the figure does not constitute a limitation on the electronic device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by one or more control processors, for example, Fig.12 One of the control processors 1001 executes, which can enable the above one or more control processors to execute the target detection method based on multimodal fusion in the above method embodiment.

[0131] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A target detection method based on multimodal fusion, characterized in that: include: Acquire a first image set and a second image set, wherein the first image set includes a plurality of visible light images, the second image set includes a plurality of infrared images, and at least a portion of the visible light images and at least a portion of the infrared images include objects to be detected; Performing a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set; Performing feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set; The fused images in the fused image set are subjected to target detection by using a preset deep learning model to obtain the object category of the object to be detected and the bounding box marking the object to be detected.

2. The target detection method based on multimodal fusion according to claim 1, characterized in that: The performing of a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set includes: Performing image denoising processing on all infrared images in the second image set; Time synchronization is performed on all the visible light images and the infrared images after image noise reduction processing by using a preset timestamp synchronization algorithm to obtain a first target image set and a second target image set corresponding to the first image set; Extracting a first feature point set of the first target image set and extracting a second feature point set of the second target image set by using a preset scale-invariant feature transformation algorithm; Performing feature point matching on the first feature point set and the second feature point set using a preset matching algorithm to obtain a matching result; A first feature image set is determined according to the first feature point set, and the infrared images in the second target image set are transformed according to the matching result to obtain a second feature image set.

3. The target detection method based on multimodal fusion according to claim 1, characterized in that: The step of performing feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set includes: Selecting a first feature image from the first feature image set, and selecting a second feature image from the second feature image set; Decomposing the first feature image by a mean filter to obtain a first base layer feature and a first detail layer feature, and decomposing the second feature image by a mean filter to obtain a second base layer feature and a second detail layer feature, wherein the first base layer feature is used to characterize the structural feature of the first feature image, the first detail layer feature is used to characterize the high-frequency information of the first feature image, the second base layer feature is used to characterize the structural feature of the second feature image, and the second detail layer feature is used to characterize the high-frequency information of the second feature image; The first base layer features and the second base layer features are fused by a preset dual-scale image fusion algorithm to generate a base layer image, and the first detail layer and the second detail layer are weighted averaged to obtain a detail layer image; Reconstructing the base layer image and the detail layer image to obtain a fused image; The fused images corresponding to all the feature images in the first feature image set and the second feature image set are sorted to obtain a fused image set.

4. The target detection method based on multimodal fusion according to claim 1, characterized in that: The deep learning model includes a fully connected layer; the target detection is performed on the fused image in the fused image set by using the preset deep learning model to obtain the object category of the object to be detected and the bounding box marking the object to be detected, including: Inputting the fused image set into a preset deep learning model, so that the deep learning model extracts features from the fused images in the fused image set and outputs feature information; The feature information is input into the fully connected layer, so that the fully connected layer predicts the category of the object to be detected in the fused image according to the feature information, and obtains the object category of the object to be detected and marks the bounding box of the object to be detected.

5. The target detection method based on multimodal fusion according to claim 1, characterized in that: After performing target detection on the fused images in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and marking the bounding box of the object to be detected, the method further includes: The object to be detected is continuously tracked by a preset multi-target tracking algorithm.

6. The target detection method based on multimodal fusion according to claim 2, characterized in that: The method of performing time synchronization on all the visible light images and the infrared images after image noise reduction processing by using a preset timestamp synchronization algorithm to obtain a first target image set and a second target image set corresponding to the first image set includes: Recording a first timestamp set of all the visible light images and a second timestamp set of all the infrared images after image noise reduction processing; Comparing first timestamps of adjacent visible light images according to the first timestamp set to obtain a first comparison difference, and comparing second timestamps of adjacent infrared images after image noise reduction processing according to the second timestamp set to obtain a second comparison difference; When the first contrast difference is greater than a preset time window, determining a first visible light image and a second visible light image corresponding to the first contrast difference, and performing interpolation processing on the first visible light image and the second visible light image by a preset interpolation method to obtain a first intermediate image, and adding the first intermediate image to the first image set to obtain a first target image set; When the second contrast difference is greater than a preset time window, the first infrared image and the second infrared image corresponding to the second contrast difference are determined, and the first infrared image and the second infrared image are interpolated by a preset interpolation method to obtain a second intermediate image, and the second intermediate image is added to the second image set to obtain a second target image set.

7. The target detection method based on multimodal fusion according to claim 6, characterized in that: After recording the first time stamp set of all the visible light images and the second time stamp set of all the infrared images after image noise reduction processing, the method further includes: Performing continuity detection on the first timestamp set and the second timestamp set; When there are discontinuous timestamps in the first timestamp set, determine a third visible light image and a fourth visible light image corresponding to the discontinuous timestamps, generate a first padding image according to the third visible light image and the fourth visible light image, and add the first padding image between the third visible light image and the fourth visible light image; When there are discontinuous timestamps in the second timestamp set, determine the third infrared image and the fourth infrared image corresponding to the discontinuous timestamps, generate a second complement image based on the third infrared image and the infrared image, and add the second complement image between the third infrared image and the fourth infrared image.

8. The target detection method based on multimodal fusion according to claim 1, characterized in that: After performing target detection on the fused images in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and marking the bounding box of the object to be detected, the method further includes: Analyzing a plurality of model weights of the deep learning model, wherein the model weights are used to characterize the importance of different branches of the deep learning model; Performing weight screening on all the model weights using a preset pruning strategy to screen out target weights that are less than a pruning threshold of the pruning strategy; The deep learning model is pruned according to the target weight to obtain a target deep model.

9. The target detection method based on multimodal fusion according to claim 1, characterized in that: After performing target detection on the fused images in the fused image set by using a preset deep learning model to obtain the object category of the object to be detected and marking the bounding box of the object to be detected, the method further includes: Perform a model quantization operation on the deep learning model.

10. A target detection device based on multimodal fusion, characterized in that: include: An image acquisition module, configured to acquire a first image set and a second image set, wherein the first image set includes a plurality of visible light images, the second image set includes a plurality of infrared images, and at least a portion of the visible light images and at least a portion of the infrared images include objects to be detected; a data alignment module, configured to perform a data alignment operation on the first image set and the second image set to obtain a first feature image set corresponding to the first image set and a second feature image set corresponding to the second image set; A feature fusion module, used to perform feature fusion on the first feature image set and the second feature image set by using a preset dual-scale image fusion algorithm to obtain a fused image set; The target detection module is used to perform target detection on the fused image in the fused image set through a preset deep learning model to obtain the object category of the object to be detected and to mark the boundary box of the object to be detected.

11. An electronic device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target detection method based on multimodal fusion as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the target detection method based on multimodal fusion as described in any one of claims 1 to 9.

Images