Multispectral target detection method and device based on feature fusion and electronic equipment

Abstract

The invention relates to a multi-spectral target detection method and device based on feature fusion and electronic equipment. The method comprises the following steps: respectively extracting visible light features of an RGB image and thermal radiation features of a thermal imaging image; carrying out bimodal feature fusion on the visible light features and the thermal radiation features by adopting a double-cross attention mechanism to obtain a fusion feature map; and inputting the fused feature map into a detection network, and carrying out target classification and positioning to obtain a detection result. By comprehensively utilizing the advantages of each image type, the limitation of another mode is compensated by the strong point of one mode, so that the fuzzy degree of the color image under a severe weather condition or a changing illumination environment is effectively solved, abundant extra information is further provided, and the robustness and accuracy of target detection are enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of target detection technology, and in particular to a multi-spectral target detection method, device and electronic equipment based on feature fusion. Background Art

[0002] The technical background of multispectral target detection methods based on feature fusion is driven by the needs of many fields, especially in remote sensing, security monitoring and natural scene understanding. Multispectral imaging technology, especially images including thermal spectral range, provides a unique way to perceive natural scenes and is regarded as an important supplement to visible spectrum images in computer vision. These images can provide additional information in harsh weather conditions or changing lighting environments, thereby enhancing the robustness and accuracy of target detection systems.

[0003] In terms of technical implementation, the feature representation and fusion of multispectral images are particularly critical. This process involves how to effectively integrate information from different spectral bands to support various visual tasks, including target detection, semantic segmentation, and target tracking. Among them, target detection, as one of the basic visual tasks, has always been a research hotspot in academia and industry, especially in the fields of small target detection, face and pedestrian detection, and directional target detection. Significant progress has been made, which is largely due to the rapid development of convolutional neural network technology.

[0004] However, although existing methods have made progress in many aspects, the performance of the detection system still has accuracy issues when faced with challenges brought by environmental factors. In particular, it is unable to solve the blur of color images in bad weather or changing lighting environments, making it difficult to capture additional information, thus affecting the robustness and accuracy of target detection. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a multispectral target detection method, device and electronic device based on feature fusion, which independently extract features from input images of RGB and thermal imaging modalities, and combine the advantages of both, so that while learning RGB color images, the characteristics of thermal images can be effectively taken into account to solve the above problems.

[0006] In a first aspect, an embodiment of the present application provides a multispectral target detection method based on feature fusion, comprising the following steps:

[0007] Extract visible light features of RGB images and thermal radiation features of thermal imaging images respectively;

[0008] The visible light feature and the thermal radiation feature are subjected to bimodal feature fusion by adopting a double cross attention mechanism to obtain a fused feature map;

[0009] The fused feature map is input into the detection network to perform target classification and positioning to obtain the detection result.

[0010] In some embodiments, the visible light feature and the thermal radiation feature are subjected to bimodal feature fusion using a double cross attention mechanism to obtain a fused feature map, including:

[0011] Performing attention weighting on the visible light feature and the thermal radiation feature based on the double cross attention mechanism to obtain attention-weighted features;

[0012] The attention-weighted features are fused using a preset feature merging method to obtain the fused feature map.

[0013] In some embodiments, the extracting of the visible light features of the RGB image and the thermal radiation features of the thermal imaging image respectively includes:

[0014] Receive RGB images and thermal imaging images respectively;

[0015] Using a pre-trained convolutional neural network to perform multi-layer feature extraction on the RGB image and the thermal imaging image, and correspondingly obtain the visible light feature and the thermal radiation feature;

[0016] The extraction of the visible light feature can be expressed as: Fi R =Ψ backbone (I R ,θ R );

[0017] The extraction of the thermal radiation feature can be expressed as: Fi T =Ψ backbone (I T ,θ T );

[0018] Where F i R is the feature map extracted from the RGB image, i represents the level of the feature map, Ψ backbone represents the feature extraction function, I R represents the received RGB image, θ R represents the network parameters corresponding to the RGB mode, F i T is the feature map extracted from the thermal infrared image, I T The received thermal infrared image, θ T Represents the network parameters corresponding to the thermal infrared modality.

[0019] In some embodiments, the fused feature map is input into a detection network to perform target classification and positioning to obtain a detection result, including:

[0020] Using the detection network to capture the target in the fusion feature map;

[0021] The classification head is used to predict the category of each of the targets, and the regression head is used to predict the bounding box of each of the targets to obtain the detection result.

[0022] In some embodiments, after the visible light feature and the thermal radiation feature are subjected to bimodal feature fusion using a double cross attention mechanism to obtain a fused feature map, the method further includes:

[0023] The fused feature map is iteratively fused according to a preset iterative fusion strategy to obtain an iterative fused feature map which is input into the detection network.

[0024] In some embodiments, the process of iteratively fusing the fused feature maps according to a preset iterative fusion strategy includes:

[0025] The double cross attention mechanism is repeatedly executed on the fused feature map according to the iterative fusion strategy to capture complementary information of different features in the fused feature map.

[0026] In some of the embodiments, when the fused feature maps are iteratively fused according to a preset iterative fusion strategy, it is implemented based on the same shared parameters.

[0027] In a second aspect, an embodiment of the present application provides a multispectral target detection device based on feature fusion, comprising:

[0028] A single-mode feature extraction module is used to extract the visible light features of the RGB image and the thermal radiation features of the thermal imaging image respectively;

[0029] A bimodal feature fusion module, used to fuse the visible light feature and the thermal radiation feature using a double cross attention mechanism to obtain a fused feature map;

[0030] The detection module is used to input the fused feature map into the detection network, perform target classification and positioning, and obtain the detection result.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a storage medium, and a computer program, wherein the computer program is stored in the storage medium, and when the computer program is executed by the processor, it implements any of the multispectral target detection methods based on feature fusion described above.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multispectral target detection method based on feature fusion as described above is implemented.

[0033] Technical effect of the present invention: The present invention integrates the advantages of RGB color images and thermal imaging images, and can effectively take into account the characteristics of thermal images while learning RGB color images. Specifically, the visible light characteristics of the RGB image and the thermal radiation characteristics of the thermal imaging image are extracted respectively; the visible light characteristics and the thermal radiation characteristics are bimodally fused using a double cross-attention mechanism to obtain a fused feature map; the fused feature map is input into the detection network to perform target classification and positioning to obtain the detection result. In other words, it has both the characteristics of RGB digital pixels and the key features of images in the thermal spectrum range, and comprehensively utilizes the advantages of each image type, using the strengths of one modality to compensate for the limitations of another modality, effectively solving the blurring of color images under severe weather conditions or changing lighting environments, and then providing rich additional information, thereby enhancing the robustness and accuracy of target detection.

[0034] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0036] Figure 1 The flowchart of the multi-spectral target detection method based on feature fusion according to the first embodiment of the present application is shown;

[0037] Figure 2 The figure shows a flow chart of a multi-spectral target detection method based on feature fusion according to the second embodiment of the present application;

[0038] Figure 3 The structure block diagram of the multi-spectral target detection device based on feature fusion according to the third embodiment of the present application is shown;

[0039] Figure 4 Shown is a structural block diagram of an electronic device according to a fourth embodiment of the present application. DETAILED DESCRIPTION

[0040] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0041] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0042] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0043] In view of the imperfections of the prior art, the present invention provides a multi-spectral target detection method based on feature fusion, which integrates the color advantages of multiple different spectra, and has both the characteristics of RGB digital pixels and the key features of images in the thermal spectrum range. That is, by combining the advantages of both, the blurring of color images under severe weather conditions or changing lighting environments is solved, and rich additional information is provided, thereby enhancing the robustness and accuracy of the target detection system.

[0044] Embodiment 1

[0045] The present invention provides a multi-spectral target detection method based on feature fusion. Figure 1 is a flowchart of the steps of the multispectral target detection method based on feature fusion according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0046] S100, extracting visible light features of the RGB image and thermal radiation features of the thermal imaging image respectively;

[0047] In this step, features are extracted independently from the input images of RGB and thermal imaging modalities, that is, visible light features are extracted from RGB images, and thermal radiation features are extracted from thermal imaging images. This is the basis of the multispectral target detection process. The purpose is to obtain as much information as possible from each modality so that the subsequent fusion stage can effectively integrate data from different sources.

[0048] S200, using a double cross attention mechanism to perform bimodal feature fusion on the visible light feature and the thermal radiation feature to obtain a fused feature map;

[0049] In this step, the visible light features from the RGB image and the thermal radiation features from the thermal imaging image are processed and fused based on the double cross attention mechanism. It should be noted that by effectively combining the features extracted independently from different modalities, namely RGB images and thermal infrared images, two different types of sensor information are effectively utilized to enhance the accuracy and robustness of target detection.

[0050] S300: Input the fused feature map into the detection network to perform target classification and positioning to obtain the detection result.

[0051] In this step, the objects in the image are identified and located based on the fused features. The iteratively fused feature map is input into a detection network to classify and locate the objects, thereby outputting the final detection result. The detection network can more accurately detect and outline the objects in the scene using the fused feature map.

[0052] In some embodiments, the visible light feature and the thermal radiation feature are fused using a double cross attention mechanism to obtain a fused feature map, including:

[0053] Based on the double cross attention mechanism, the visible light features and thermal radiation features are weighted to obtain the attention-weighted features;

[0054] The attention-weighted features are fused using a preset feature merging method to obtain a fused feature map.

[0055] This embodiment aims to effectively combine the features extracted independently from different modalities. It mainly includes two steps. First, the Cross-modal Attention Module (CAM) uses the attention mechanism in the transformer architecture to allow the model to dynamically select relevant features between the two modalities. The model calculates the correlation between different modalities and optimizes the combination of features to improve the accuracy and robustness of target detection. Secondly, the attention-weighted features are merged through a specific fusion strategy to form a comprehensive feature representation, thereby obtaining a fused feature map.

[0056] Specifically, the process of bimodal feature fusion includes:

[0057] First, input features: This stage receives feature maps corresponding to RGB and thermal infrared modalities respectively;

[0058] Second, cross-attention mechanism: Use the cross-attention mechanism to dynamically adjust and weight the feature contributions of different modalities, so that the model can focus on the more critical feature areas in the target detection task and ignore irrelevant information;

[0059] Finally, feature fusion: The feature maps after cross-attention adjustment are combined together through a specific feature merging method to generate a comprehensive feature representation for subsequent object detection tasks. Among them, the feature merging method is preferably weighted summation and splicing.

[0060] Let F i R and F i T The features extracted from RGB images and thermal images are respectively, and the fusion process can be represented by the following steps:

[0061] (1) Feature preparation: Preprocess the RGB and thermal infrared feature maps in preparation for fusion.

[0062] Specifically including resizing and normalization;

[0063] (2) Cross-attention module: The cross-attention module is used to enhance the complementarity between features. Specifically, it is expressed as follows:

[0064]

[0065] Among them, Q, K, and V are query, key, and value, respectively, usually from the feature graph F i R and F i T Derived from, in bimodal fusion, the features of one modality can be used as Q, while the features of the other modality can be used to construct K and V; here, it should be noted that the goal of feature fusion is to effectively combine the features of RGB and infrared images to enhance the final image representation. Specifically, when fusing RGB and thermal infrared images, the features of RGB images are preferred as query Q, because RGB images usually contain rich color and detail information, which has a significant effect on queries in the attention mechanism. The features of thermal infrared images are preferably used to construct the key K and value V, because thermal infrared images provide different thermal information, which can complement the visual features of RGB images;

[0066] That is, the RGB features are used as Q through the cross attention module, and the thermal infrared features are used to construct K and V. The attention scores are normalized using the softmax function to highlight the feature areas that contribute to the fusion;

[0067] (3) Feature merging: combining the feature maps processed by cross attention, the fusion strategy includes simple weighting or more complex operations, where complex operations are preferably feature concatenation or nonlinear combination;

[0068]

[0069] Among them, φ represents a feature fusion function that may contain nonlinear activation, F fused It is the final fused feature representation.

[0070] It should be noted that, in this embodiment, through the cross-attention mechanism, the model can not only take advantage of the advantages of each modality, but also dynamically readjust the contribution of the features to make the fused features more powerful and robust. The above fusion strategy is particularly suitable for processing multispectral data because the information captured by different sensor types has different emphases, and the intelligent fusion in this embodiment can maximize the use of this information to improve the performance of target detection.

[0071] In some embodiments, the visible light features of the RGB image and the thermal radiation features of the thermal imaging image are extracted respectively, including:

[0072] Receive RGB images and thermal imaging images respectively;

[0073] A pre-trained convolutional neural network is used to perform multi-layer feature extraction on RGB images and thermal imaging images, and corresponding visible light features and thermal radiation features are obtained;

[0074] Among them, the extraction of visible light features can be expressed as: Fi R =Ψ backbone (I R ,θ R );

[0075] The extraction of thermal radiation features can be expressed as: Fi T =Ψ backbone (I T ,θ T );

[0076] Where F i R is the feature map extracted from the RGB image, i represents the level of the feature map, Ψ backbone represents the feature extraction function, I R represents the received RGB image, θ R represents the network parameters corresponding to the RGB mode, F i T is the feature map extracted from the thermal infrared image, I T The received thermal infrared image, θ T Represents the network parameters corresponding to the thermal infrared modality.

[0077] It should be noted that, in this embodiment, features are extracted independently from RGB images and thermal infrared images, wherein a pre-trained convolutional neural network is used to implement single-modal feature extraction, and each modality image is converted into a series of feature maps, and the feature maps include spatial and channel information of the image. It should also be noted that, as the basis of the multispectral target detection process, as much information as possible can be obtained from each modality so that the subsequent fusion stage can effectively integrate data from different sources.

[0078] Specifically, the feature extraction process of this embodiment includes:

[0079] Firstly, two types of images are received as input, namely, RGB images and thermal infrared images, and the visible light features of the target are captured by the RGB images, and the thermal radiation features of the target are captured by the thermal infrared images;

[0080] Subsequently, the images of each modality are processed using a pre-trained convolutional neural network (CNN), preferably VGG16 or ResNet, wherein CNN extracts features from low-level to high-level through its multi-layer structure;

[0081] Finally, the corresponding feature map is output. Each CNN converts the image into a series of feature maps, where the feature maps include spatial and abstract features about the image content, and the feature maps usually have multiple channels, each channel representing a different feature response in the image.

[0082] Moreover, the feature extraction process for RGB images and thermal infrared images can be expressed by the following equations:

[0083] Let I R and I T Represent the input RGB image and thermal infrared image respectively;

[0084] Ψ backbone Represents the feature extraction function, which is implemented by a specific CNN architecture, preferably VGG16 or ResNet;

[0085] θ R and θ T denote the network parameters corresponding to RGB and thermal infrared modalities, respectively;

[0086] For each modality, feature extraction can be expressed as:

[0087] Fi R =Ψ backbone (I R ,θ R ), Fi T =Ψ backbone (I T ,θ T )

[0088] Among them, F i R and F i T are feature maps extracted from RGB images and thermal infrared images, respectively. i represents the level of the feature map. Usually, in a deep neural network, different layers capture features at different levels. It should be noted here that the features in this embodiment include edges, textures, and shapes.

[0089] The above feature maps are used in the subsequent bimodal feature fusion stage to integrate information from different modalities through further processing to enhance the performance of object detection. By processing each modality independently, the model is able to retain key information unique to each image type, providing strong support for subsequent fusion and recognition.

[0090] In some of the embodiments, the fused feature map is input into the detection network to perform target classification and positioning to obtain the detection results, including:

[0091] Use the detection network to capture the target in the fusion feature map;

[0092] The classification head is used to predict the category of each target, and the regression head is used to predict the bounding box of each target to obtain the detection result.

[0093] It should be noted that, in this embodiment, a fused feature map obtained by the above-mentioned feature fusion is received and input into a detection network, preferably YOLO or SSD, to classify and locate the target. In this process, the target's border regression and category judgment are involved, and the final detection result is output.

[0094] Specifically, the process of using the detection network to detect the target includes the following steps:

[0095] First, receive the fused feature map from the previous stage, that is, the fused feature map after bimodal feature fusion;

[0096] Secondly, the feature pyramid network FPN or other network structures that can achieve the purpose of integrating multi-scale features are used to integrate multi-scale features and improve the detection accuracy. This step helps the model capture targets of various sizes from small to large.

[0097] Finally, the classification head is used to predict the category of each object, and the regression head is used to predict the bounding box, i.e. the location, of the object. It should be noted that the "head" in this step is usually a series of convolutional layers or fully connected layers, which are specially designed to output category probabilities and bounding box coordinates.

[0098] Let F fused It is the fusion feature map passed in from the bimodal feature fusion stage. The task of the detection network is to convert these features into specific detection results, which are expressed by the following equation:

[0099]

[0100] in, Represents the predicted category probability, usually a vector in which each element represents the probability of a category. Represents the predicted bounding box coordinates, usually including the center point coordinates, width and height of the bounding box. DetectNet is the detection network responsible for converting the fused feature map into the detection result. D are network parameters.

[0101] It should be noted that, in this embodiment, the detection head is implemented based on the classification head and the regression head.

[0102] Among them, the classification head part of the network usually includes several convolutional layers or fully connected layers, and finally outputs the probability of each category through a softmax layer; while the regression head part of the network is responsible for predicting the specific location of the target and outputting four regression values, indicating the center point coordinates and size of the bounding box.

[0103] In practical applications, when using target detection frameworks such as YOLO, SSD or Faster R-CNN, the specific architecture of the detection network will be different accordingly. Each framework will implement classification and regression tasks according to a specific design to optimize the accuracy and efficiency of detection. Through a precisely designed detection network, high-level feature representations can be effectively converted into practical detection results, making the multispectral target detection system applicable to various application scenarios.

[0104] It can be seen that the present application realizes the above-mentioned multi-spectral target detection properties with feature fusion, integrates the color advantages of multiple different spectra, and enhances the feature fusion between different modalities, especially the integration of RGB and thermal imaging data. Among them, the system uses independent convolutional neural networks to extract features from RGB and thermal imaging respectively, ensuring that the unique effects of each feature are effectively captured. And the core of the entire fusion process is the dual cross-attention mechanism, which allows the system to dynamically weigh and integrate features from the two modalities, so that the combination of RGB and thermal imaging provides the most information for detecting the area of ​​the object. Not only that, after the feature fusion, the fused features are sent to the detection network, which is responsible for the final object classification and positioning. Based on the above fused features, objects in the scene can be more accurately detected and outlined.

[0105] Embodiment 2

[0106] like Figure 2 As shown, the second embodiment provides a multi-spectral target detection method based on feature fusion, and the second embodiment is carried out on the basis of the first embodiment.

[0107] After the visible light feature and the thermal radiation feature are subjected to bimodal feature fusion using a double cross attention mechanism to obtain a fused feature map, the second embodiment further includes: iteratively fusing the fused feature map according to a preset iterative fusion strategy to obtain an iterative fused feature map and inputting it into a detection network.

[0108] It should be particularly emphasized that in this embodiment, after the bimodal features are fused to obtain the fused feature map, an iterative learning strategy is used to continuously refine and optimize the feature expression, and the iterative fused feature map obtained by iterative fusion is input into the detection network for target detection. That is, through the feature fusion strategy, the RGB features and thermal infrared features that have been processed by attention are weighted summed or nonlinearly combined to obtain the fused feature representation. That is, the method that combines cross-attention and iterative fusion strategies makes full use of the complementarity of RGB and thermal infrared images, effectively captures complementary information from multispectral images, and improves the expression ability of fused features, thereby improving the target detection performance under various environmental conditions. In addition, the model is also particularly suitable for dealing with image quality problems caused by factors such as large changes in illumination or obstructed vision.

[0109] In some embodiments, the process of iteratively fusing the fused feature maps according to a preset iterative fusion strategy includes:

[0110] According to the iterative fusion strategy, the double cross attention mechanism is repeatedly performed on the fused feature map to capture the complementary information of different features in the fused feature map.

[0111] In this embodiment, the model repeatedly performs the cross-attention fusion process, and each iteration further refines the features based on the previous one, which is similar to the process when humans review knowledge, which helps to strengthen the model's memory and recognition ability of features. That is to say, in each iteration, the model enhances the interaction between features from different modalities through a specific mechanism, namely cross-attention, so as to better capture and utilize the complementary information in these features. Not only that, in this embodiment, through the process of gradual optimization, the model can gradually improve the feature representation, and each iteration further improves the quality and relevance of the features based on the previous one to achieve more accurate detection results.

[0112] In some of the embodiments, when the fused feature maps are iteratively fused according to a preset iterative fusion strategy, it is implemented based on the same shared parameters.

[0113] It should be noted that in order to reduce the computational complexity and the number of model parameters, the iterative process in this embodiment shares the same parameters, which helps to improve the efficiency and generalization ability of the model. R +F T

[0114] F(n)=Φ(F(n- 1) , F R , F T ;θ)

[0115] To reduce model complexity and reduce the risk of overfitting, parameters are shared during iteration, which means that each iteration uses the same model weights for feature processing.

[0116] The iterative fusion process is described in detail below.

[0117] Let F R and F T are the features extracted from RGB images and thermal images respectively. Iterative fusion can represent a recursive process, where Φ represents the feature fusion function, including attention mechanism or other feature integration techniques. Let n represent the number of iterations, and F(n) represents the fused features after the nth iteration;

[0118] Among them, F(0) is the initial fusion feature, which is a simple feature sum, and Φ is the feature fusion function, which is responsible for further refining the feature representation in each iteration. This function can be a cross-attention based model or any complex function suitable for feature integration. θ represents the shared parameters in the fusion model, which remain unchanged in all iterations.

[0119] The iterative fusion strategy allows the model to have the opportunity to re-evaluate and optimize the combination of features in each iteration, so that the model can better adapt to complex feature distribution and changing environmental conditions. It is especially suitable for applications that require accurate detection in unstable or changing environments, such as autonomous driving, night monitoring and other scenarios. By iteratively refining and optimizing feature fusion, the model can gradually improve its ability to express and identify target features, ultimately achieving higher detection accuracy and robustness.

[0120] Furthermore, the iterative fusion feature map after iterative fusion is input into the detection network, and the detection network is responsible for the final object positioning and classification, wherein the detection network accurately detects and outlines the objects in the scene based on the above iterative fusion features.

[0121] It can be seen that the method of this embodiment adopts an iterative strategy to refine feature fusion. By repeatedly processing the combined features, the system can gradually improve the quality of the feature map used for detection, thereby enhancing accuracy and robustness. The advantages of each image type are fully utilized, that is, the strengths of one modality are used to compensate for the limitations of another modality, and the detection performance is continuously improved through iterative refinement.

[0122] In summary, the multispectral target detection method based on feature fusion of the present application can effectively utilize the complementary information of RGB and thermal imaging data through the double cross attention mechanism and iterative fusion strategy, improve the accuracy of target detection, and perform well in complex environments, so as to achieve the purpose of improving detection accuracy, enhancing system robustness, reducing computational complexity and improving real-time performance. Among them, the iterative fusion strategy optimizes the feature representation through multiple iterations, enhances the adaptability of the detection system to environmental changes, and ensures that the system maintains high performance under different conditions. In addition, through the iterative process of parameter sharing and optimization, the model controls the consumption of computing resources while improving performance, making the system suitable for deployment on resource-constrained devices. Finally, the optimized design of the detection network makes it possible to process the fused feature map quickly, realizes real-time target detection, and is particularly important for application scenarios that require immediate response. These effects jointly improve the overall performance and application value of the system, making it have broad application prospects in multiple fields.

[0123] Embodiment 3

[0124] like Figure 3 As shown, the embodiment of the present application provides a multispectral target detection device based on feature fusion, comprising:

[0125] The single-mode feature extraction module 100 is used to extract the visible light features of the RGB image and the thermal radiation features of the thermal imaging image respectively;

[0126] The bimodal feature fusion module 200 is used to fuse the visible light feature and the thermal radiation feature using a double cross attention mechanism to obtain a fused feature map;

[0127] The detection module 300 is used to input the fused feature map into the detection network, perform target classification and positioning, and obtain the detection result.

[0128] In some embodiments, it also includes:

[0129] The iterative fusion module is used to iteratively fuse the fused feature map according to a preset iterative fusion strategy to obtain an iterative fused feature map and input it into the detection network.

[0130] Embodiment 4

[0131] like Figure 4 As shown, an embodiment of the present application provides an electronic device, which includes a processor, a storage medium and a computer program. The computer program is stored in the storage medium. When the computer program is executed by the processor, any of the above-mentioned multi-spectral target detection methods based on feature fusion is implemented. Specific examples can refer to the capabilities described in the above-mentioned embodiments and optional implementation methods, and this embodiment will not be repeated here.

[0132] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0133] In addition, in combination with the multispectral target detection method based on feature fusion in the above embodiments, the fourth embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by the processor, any multispectral target detection method based on feature fusion in the above embodiments is implemented, and the method includes:

[0134] Extract visible light features of RGB images and thermal radiation features of thermal imaging images respectively;

[0135] The visible light features and thermal radiation features are fused using a double cross attention mechanism to obtain a fused feature map.

[0136] The fused feature map is input into the detection network to perform target classification and positioning to obtain the detection result.

[0137] Taking a processor as an example, the processor, memory, input device and output device in the electronic device can be connected via a bus or other methods, taking connection via a bus as an example.

[0138] As a computer-readable storage medium, the memory may include a high-speed random access memory, a non-volatile memory, etc., and may be used to store operating systems, software programs, computer executable programs, and databases, such as program instructions / modules of the multispectral target detection method based on feature fusion in Example 1 of the present invention, and may also include a memory, which may be used to provide an operating environment for operating systems and computer programs. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device via a network.

[0139] The processor is used to provide computing and control capabilities, and may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application. The processor executes various functional applications and data processing of the electronic device by running computer executable programs, software programs, instructions, and modules stored in the memory, that is, implementing the multispectral target detection method based on feature fusion of embodiment 1.

[0140] The output device of the electronic device can be a liquid crystal display or an electronic ink display, and the input device of the electronic device can be a touch layer covering the display, or a case, trackball or touchpad provided on the housing of a computer device, or an external keyboard, touchpad or mouse.

[0141] The electronic device may also include a network interface / communication interface, and the network interface of the electronic device is used for external terminals to communicate via a network connection. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a partial structural block diagram related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DD RSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DR AM (SDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0144] It is worth noting that in the embodiment of the multispectral target detection method based on feature fusion, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0145] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "one", "the" and the like involved in this application do not indicate a quantitative limitation and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more, and "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0146] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but the limitation of the scope of the invention patent cannot be understood as a result. It should be pointed out that, for a person of ordinary skill in the art, several modifications or improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multispectral target detection method based on feature fusion, characterized in that: The steps include: Extract visible light features of RGB images and thermal radiation features of thermal imaging images respectively; The visible light feature and the thermal radiation feature are subjected to bimodal feature fusion by adopting a double cross attention mechanism to obtain a fused feature map; The fused feature map is input into the detection network to perform target classification and positioning to obtain the detection result.

2. The multispectral target detection method based on feature fusion according to claim 1, characterized in that: The method of fusing the visible light feature and the thermal radiation feature by using a double cross attention mechanism to obtain a fused feature map includes: Performing attention weighting on the visible light feature and the thermal radiation feature based on the double cross attention mechanism to obtain attention-weighted features; The attention-weighted features are fused using a preset feature merging method to obtain the fused feature map.

3. The multispectral target detection method based on feature fusion according to claim 1, characterized in that: The step of respectively extracting the visible light features of the RGB image and the thermal radiation features of the thermal imaging image comprises: Receive RGB images and thermal imaging images respectively; Using a pre-trained convolutional neural network to perform multi-layer feature extraction on the RGB image and the thermal imaging image, and correspondingly obtain the visible light feature and the thermal radiation feature; The extraction of the visible light feature can be expressed as: Fi R =Ψ backbone (I R ,θ R ); The extraction of the thermal radiation feature can be expressed as: Fi T =Ψ backbone (I T ,θ T ); Where F i R is the feature map extracted from the RGB image, i represents the level of the feature map, Ψ backbone represents the feature extraction function, I R represents the received RGB image, θ R represents the network parameters corresponding to the RGB mode, F i T is the feature map extracted from the thermal infrared image, I T The received thermal infrared image, θ T Represents the network parameters corresponding to the thermal infrared modality.

4. The multispectral target detection method based on feature fusion according to claim 1, characterized in that: The step of inputting the fused feature map into a detection network, performing target classification and positioning, and obtaining a detection result includes: Using the detection network to capture the target in the fusion feature map; The classification head is used to predict the category of each of the targets, and the regression head is used to predict the bounding box of each of the targets to obtain the detection result.

5. The multispectral target detection method based on feature fusion according to any one of claims 1 to 4, characterized in that: After the visible light feature and the thermal radiation feature are fused by a double cross attention mechanism to obtain a fused feature map, the method further includes: The fused feature map is iteratively fused according to a preset iterative fusion strategy to obtain an iterative fused feature map which is input into the detection network.

6. The multispectral target detection method based on feature fusion according to claim 5 is characterized in that: The process of iteratively fusing the fused feature graph according to the preset iterative fusion strategy includes: The double cross attention mechanism is repeatedly executed on the fused feature map according to the iterative fusion strategy to capture complementary information of different features in the fused feature map.

7. The multispectral target detection method based on feature fusion according to claim 6 is characterized in that: When the fused feature maps are iteratively fused according to a preset iterative fusion strategy, it is achieved based on the same shared parameters.

8. A multispectral target detection device based on feature fusion, characterized in that: include: A single-mode feature extraction module is used to extract the visible light features of the RGB image and the thermal radiation features of the thermal imaging image respectively; A bimodal feature fusion module, used to fuse the visible light feature and the thermal radiation feature using a double cross attention mechanism to obtain a fused feature map; The detection module is used to input the fused feature map into the detection network, perform target classification and positioning, and obtain the detection result.

9. An electronic device comprising a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, wherein: When the computer program is executed by a processor, the multispectral target detection method based on feature fusion described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multispectral target detection method based on feature fusion described in any one of claims 1 to 7 is implemented.

Images