Infrared and visible light target detection method, electronic equipment, storage medium and product

Abstract

The present invention provides an infrared and visible light target detection method, electronic device, storage medium, and product. The infrared and visible light target detection method comprises: extracting features from infrared and visible light images to obtain feature maps at multiple scales; enhancing the representation and discrimination capabilities of each scale feature to obtain multiple enhanced feature maps; cross-modally fusing the multiple enhanced feature maps to obtain fused multi-scale features; and detecting targets based on the fused multi-scale features and a composite loss function that combines the inner intersection-over-union (IoU) and the minimum point distance IoU. This improves representational power and boundary location accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of image detection technology, and in particular relates to an infrared and visible light target detection method, electronic equipment, storage medium and product. Background Art

[0002] With the development of artificial intelligence and computer vision technologies, unmanned platforms have been widely used in fields such as autonomous driving, intelligent patrol, security monitoring, and disaster relief. As the core module of the perception system, target detection and recognition technology directly determines the accuracy and responsiveness of unmanned systems to identify key targets in the environment.

[0003] Traditional object detection methods rely heavily on single-modality image input, particularly visible light images (RGB images). However, in scenarios such as bright light, at night, in fog and haze, under occlusion, or with complex backgrounds, visible light images are susceptible to lighting variations and visual interference, resulting in a significant decrease in object detection accuracy and insufficient robustness.

[0004] To improve the stability of detection systems in complex environments, researchers have recently proposed multimodal perception methods, particularly the combined perception of infrared and visible light images. Infrared images, based on the thermal radiation characteristics of objects, enable stable imaging in low-light or even no-light environments. Visible light images, on the other hand, contain rich texture and color information, facilitating detailed analysis. The complementary nature of these two imaging mechanisms offers a new path for robust object detection.

[0005] At present, for bimodal target detection, the mainstream research methods can be divided into three categories: image-level fusion, decision-level fusion and feature-level fusion. Image-level fusion generates a fused image through pixel-level merging, but it is prone to redundancy or information loss; the decision-level fusion method merges the detection results at the back-end of two independent networks, which has high computational cost and lacks inter-modal feature collaboration; the feature-level fusion method performs cross-modal feature extraction and fusion in the middle layer of the network.

[0006] Although previous studies have attempted to use dual-branch networks, attention mechanisms, or Transformer structures for modal fusion, there are still deficiencies in information selectivity, key area perception, feature alignment, and bounding box regression. In particular, the fusion effect is unstable and the target positioning is inaccurate under severe occlusion, complex background, or extreme lighting conditions, which limits its application in actual scenarios. Summary of the Invention

[0007] In response to the problems existing in the prior art, the present invention provides an infrared and visible light target detection method, electronic device, storage medium and product, which at least partially solve the problems of insufficient feature expression capability and inaccurate bounding box positioning existing in the prior art.

[0008] In a first aspect, an embodiment of the present disclosure provides an infrared and visible light target detection method, comprising:

[0009] Feature extraction is performed on infrared and visible light images respectively to obtain feature maps at multiple scales;

[0010] Improve the representation and discrimination capabilities of each scale feature and obtain multiple enhanced feature maps;

[0011] Perform cross-modal fusion on multiple enhanced feature maps to obtain fused multi-scale features;

[0012] Target detection is achieved based on the fused multi-scale features and a composite loss function, where the composite loss function combines the inner intersection-over-union (IoU) and the minimum point distance IoU.

[0013] Optionally, the feature extraction is performed on the infrared image and the visible light image respectively to obtain feature maps of multiple scales, including:

[0014] Based on the constructed dual-stream feature extraction and enhancement module, feature extraction is performed on infrared and visible light images respectively;

[0015] The dual-stream feature extraction and enhancement module includes two independent backbone networks, which are used to extract modal features of infrared images and visible light images respectively;

[0016] The Backbone network includes basic convolutional layers, C2f structures, and spatial pyramid pooling modules.

[0017] Optionally, the step of improving the representation and discrimination capabilities of each scale feature to obtain multiple enhanced feature maps includes:

[0018] Each scale feature includes infrared features and visible light features;

[0019] The infrared features and visible light features are spliced ​​along the channel dimension to obtain the joint features;

[0020] Perform global average pooling and maximum pooling on the joint features to extract the channel description vector;

[0021] The channel description vector is convolved to construct a compressed excitation structure to obtain a channel attention weight map;

[0022] Enhance the features based on the attention weight map to obtain enhanced features;

[0023] Based on the multi-scale direction-aware convolutional structure, the enhanced features are processed to obtain spatial feature responses in multiple directions;

[0024] The spatial feature responses in all directions are added to the enhanced features and then convolved to obtain the spatial attention features;

[0025] The spatial attention features are fused with the joint features based on residual connections to obtain enhanced modality fusion features.

[0026] Optionally, the enhanced modal fusion feature is the product of the spatial attention feature and the enhanced feature plus the joint feature.

[0027] Optionally, cross-modal fusion of the multiple enhanced feature maps to obtain fused multi-scale features includes:

[0028] Calculate the differential mode information between the enhanced features from infrared and the enhanced features from visible light to obtain the differential mode features;

[0029] Then, each differential mode feature is subjected to maximum pooling and average pooling operations in the channel direction, and then the spatial feature representation is extracted by convolution, and the spatial attention weight map is output using the Sigmoid function;

[0030] Apply the spatial attention weight map to the corresponding differential mode features to obtain enhanced differential mode features;

[0031] Based on the enhanced differential mode features, the spatial attention weight map and the differential mode features, the enhanced feature maps corresponding to infrared and the enhanced feature maps corresponding to visible light are obtained respectively;

[0032] The enhanced feature map corresponding to infrared and the enhanced feature map corresponding to visible light are spliced ​​in the channel dimension to obtain the fused multimodal features.

[0033] Optionally, the composite loss function combining the inner intersection-and-union ratio and the minimum point distance intersection-and-union ratio includes:

[0034] The inner intersection and union ratio and the minimum point distance intersection and union ratio are based on Perform weighted combination, A hyperparameter that controls the contribution weights of the inner IoU loss and the minimum point distance IoU loss.

[0035] Optionally, the internal intersection and comparison includes:

[0036] By setting the scaling ratio, auxiliary frames corresponding to the predicted frame and the real frame are constructed respectively;

[0037] Calculate the inner intersection and union ratio based on the auxiliary box.

[0038] In a second aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0039] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the infrared and visible light target detection methods described in the first aspect.

[0040] In a third aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any infrared and visible light target detection method described in the first aspect.

[0041] In a fourth aspect, an embodiment of the present disclosure further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements any of the infrared and visible light target detection methods described in the first aspect.

[0042] The present invention provides an infrared and visible light target detection method, electronic device, storage medium and product, wherein the infrared and visible light target detection method can enhance the spatial structure and semantic detail perception capabilities of infrared and visible light images in their respective modalities by enhancing the extracted feature maps at multiple scales, effectively improving the integrity and discriminability of feature expression, performing cross-modal fusion, improving the complementary and collaborative capabilities between different modalities, achieving accurate perception and information interaction in key areas, and performing target detection with a composite loss function, which can more accurately measure the degree of match between the predicted frame and the real frame, effectively improving the accuracy of target positioning, the stability of training and the robustness of detection in complex scenes, thereby achieving the purpose of improving expression ability and boundary positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0044] Figure 1 A schematic diagram of a method for detecting infrared and visible light targets according to an embodiment of the present disclosure;

[0045] Figure 2 A block diagram of the principle of the enhancement module in the dual-stream feature extraction and enhancement module provided in an embodiment of the present disclosure;

[0046] Figure 3 A block diagram of the cross-modal fusion module provided in an embodiment of the present disclosure;

[0047] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0049] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0050] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0051] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0052] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0053] SPPF refers to spatial pyramid pooling;

[0054] CPCA (Channel and Position-aware Convolutional Attention) refers to the channel and space joint perception module;

[0055] CFFM (Cross-modality Feature Fusion Module) refers to the cross-modality feature fusion module;

[0056] Inner-IoU refers to inner intersection-over-union;

[0057] MPDIoU refers to the minimum point distance intersection-union ratio;

[0058] The detection method of this embodiment can be specifically applied to image recognition and detection in harsh environments such as low illumination, complex background, and target occlusion, and is widely applicable to unmanned system perception tasks such as autonomous driving, border inspection, security monitoring, and disaster emergency response.

[0059] This embodiment discloses a method for detecting infrared and visible light targets, including:

[0060] Feature extraction is performed on infrared and visible light images respectively to obtain feature maps at multiple scales;

[0061] Improve the representation and discrimination capabilities of each scale feature and obtain multiple enhanced feature maps;

[0062] Perform cross-modal fusion on multiple enhanced feature maps to obtain fused multi-scale features;

[0063] Target detection is achieved based on the fused multi-scale features and a composite loss function, where the composite loss function combines the inner intersection-over-union (IoU) and the minimum point distance IoU.

[0064] like Figure 1 As shown in the figure, feature extraction is performed on infrared and visible light images respectively to obtain feature maps of multiple scales, including:

[0065] Based on the constructed dual-stream feature extraction and enhancement module, feature extraction is performed on infrared and visible light images respectively;

[0066] The dual-stream feature extraction and enhancement module includes two independent backbone networks, which are used to extract modal features of infrared images and visible light images respectively;

[0067] The Backbone network includes basic convolutional layers, C2f structures, and spatial pyramid pooling modules.

[0068] Optionally, the step of improving the representation and discrimination capabilities of each scale feature to obtain multiple enhanced feature maps includes:

[0069] Each scale feature includes infrared features and visible light features;

[0070] The infrared features and visible light features are spliced ​​along the channel dimension to obtain the joint features;

[0071] Perform global average pooling and maximum pooling on the joint features to extract the channel description vector;

[0072] The channel description vector is convolved to construct a compressed excitation structure to obtain a channel attention weight map;

[0073] Enhance the features based on the attention weight map to obtain enhanced features;

[0074] Based on the multi-scale direction-aware convolutional structure, the enhanced features are processed to obtain spatial feature responses in multiple directions;

[0075] The spatial feature responses in all directions are added to the enhanced features and then convolved to obtain the spatial attention features;

[0076] The spatial attention features are fused with the joint features based on residual connections to obtain enhanced modality fusion features.

[0077] like Figure 2 As shown in the figure, the dual-stream feature extraction and enhancement module integrates the channel attention mechanism and the spatial direction perception structure, which can simultaneously model the significant areas of the feature map in the channel dimension and the spatial dimension. Assume that the output features of the infrared and visible light branches at a certain layer are infrared features and visible light characteristics , first concatenate the two along the channel dimension to obtain the joint feature In order to improve the responsiveness of key semantic channels, a channel attention mechanism is introduced. Perform global average pooling and maximum pooling to extract channel description vectors and The above two items construct a compressed excitation structure through two shared layers of 1×1 convolution to obtain the channel attention weight map : is convolution, is the activation function, is the Sigmoid activation function, is the feature after global average pooling, It is the feature after global maximum pooling.

[0078] ;

[0079] ;

[0080] ;

[0081] The channel attention map is applied to the fusion feature to enhance the important channels and obtain the feature Subsequently, a multi-scale direction-aware convolution structure was introduced, including 5×5, 1×7, 7×1 and other direction-aware convolution kernels, and After all the directional convolution results are added to the channel enhancement results, a 1×1 convolution is performed to obtain the spatial attention features:

[0082] ;

[0083] in, Represents the depth-wise separable convolution operation in different directions. Finally, the spatial enhancement result is fused with the original feature through the residual connection to obtain the final enhanced modal fusion feature. This feature preserves the original information within the modality while strengthening the discriminative salient areas. The enhanced feature maps are fed into the three-layer cross-modal fusion module for further information complementation and fusion operations. Ultimately, the fused multi-scale features are uniformly input into the Neck structure of YOLOv8, completing bottom-up feature integration and top-down information transfer, thereby providing a higher-quality feature foundation for the detection head.

[0084] The dual-stream feature extraction and enhancement module not only achieves deep feature complementarity between infrared and visible light modalities through a processing flow of first enhancement and then fusion, but also effectively alleviates the perception difficulties caused by background interference, detail loss and low-light scenes in complex scenes.

[0085] Optionally, the enhanced modal fusion feature is the product of the spatial attention feature and the enhanced feature plus the joint feature.

[0086] That is, the enhanced modal fusion feature .

[0087] Optionally, cross-modal fusion of the multiple enhanced feature maps to obtain fused multi-scale features includes:

[0088] Calculate the differential mode information between the enhanced features from infrared and the enhanced features from visible light to obtain the differential mode features;

[0089] Then, each differential mode feature is subjected to maximum pooling and average pooling operations in the channel direction, and then the spatial feature representation is extracted by convolution, and the spatial attention weight map is output using the Sigmoid function;

[0090] Apply the spatial attention weight map to the corresponding differential mode features to obtain enhanced differential mode features;

[0091] Based on the enhanced differential mode features, the spatial attention weight map and the differential mode features, the enhanced feature maps corresponding to infrared and the enhanced feature maps corresponding to visible light are obtained respectively;

[0092] The enhanced feature map corresponding to infrared and the enhanced feature map corresponding to visible light are spliced ​​in the channel dimension to obtain the fused multimodal features.

[0093] like Figure 3 As shown, the cross-modal fusion module in this embodiment is different from the traditional modal splicing or weighted fusion method. The CFFM module adopts the strategy of "aligning first, then improving the difference". First, the enhanced features from the infrared and visible light branches are used to calculate the difference mode information between the two. , are the enhanced features of visible light and infrared images respectively, It represents the difference from visible light to infrared (visible light minus infrared), and vice versa, it represents the difference from infrared to visible light (infrared minus visible light).

[0094] ;

[0095] ;

[0096] Then, each differential mode feature is subjected to maximum pooling and average pooling operations in the channel direction. After splicing, a 7×7 convolution is used to extract the spatial feature representation, and the Sigmoid function is used to output the spatial attention weight map. , and then the weight map is applied to the corresponding differential mode features to achieve the screening and enhancement of the significant areas. The enhanced differential mode features are then added to the original modal features to complete the cross-enhancement process, which is mathematically expressed as:

[0097] ;

[0098] ;

[0099] Finally, the two enhanced feature maps are spliced ​​in the channel dimension to obtain the fused multimodal features. The fused features are then fed into the Neck portion of the detection network for multi-scale fusion and target recognition. The advantage of the CFFM module over the traditional attention mechanism is that it not only retains the important features of each modality, but also accurately extracts the complementary regional information between the two modalities through differential mode modeling. This module is particularly suitable for scenes with occlusion, low light, or significant background interference, and can significantly improve the model's responsiveness to the target area and detection accuracy. This module has a compact structure and low computational overhead, making it easy to integrate into lightweight detection frameworks. It improves fusion effectiveness while ensuring detection speed, and is an important component in improving multimodal target detection performance.

[0100] Optionally, the composite loss function combining the inner intersection and union ratio and the minimum point distance intersection and union ratio includes: combining the inner intersection and union ratio and the minimum point distance intersection and union ratio based on Perform weighted combination, A hyperparameter that controls the contribution weights of the inner IoU loss and the minimum point distance IoU loss.

[0101] Optionally, the inner intersection and union ratio includes: constructing auxiliary frames corresponding to the predicted frame and the real frame respectively by setting a scaling ratio; and calculating the inner intersection and union ratio based on the auxiliary frames.

[0102] The traditional IoU loss is prone to gradient vanishing problems when there is a large deviation between the predicted box and the real box or when there is no overlap at all. MPDIoU can effectively capture the geometric differences between the boxes by calculating the minimum distance between the key points of the predicted box and the real box. and real frame , its MPDIoU loss is calculated as follows:

[0103] ;

[0104] in, Indicates the minimum point distance between the predicted box and the real box, Represents the maximum point distance between the predicted box and the true box. MPDIoU comprehensively considers the overlap area, center point distance, and width and height difference information, thereby better distinguishing the positioning errors in different prediction situations.

[0105] Inner-IoU solves the problem of traditional IoU being insensitive to gradients in highly overlapping areas by introducing dynamic auxiliary frames. Specifically, for the predicted frame and real frame (defined as the upper left and lower right corner coordinates), by setting the scaling factor (generally ), construct auxiliary frame and And calculate the inner IOU:

[0106] ;

[0107] The design idea of ​​the composite loss function is to organically combine the dynamic adjustment mechanism of the inner intersection-over-union (IoU) with the full-factor geometric constraint of MPDIoU. This allows for stronger gradient feedback (provided by Inner-IoU) in the early stages of training, while fully capturing the subtle geometric differences between the predicted and true boxes (characterized by MPDIoU) during the fine-grained localization phase. Specifically, the two losses can be weighted and combined proportionally:

[0108] ;

[0109] in, The hyperparameter controlling the contribution weights of the inner-intersection-of-union (IoU) and MPDIoU losses is set to 0.5 based on experimental validation. This combined strategy offers the following advantages: First, through the auxiliary box mechanism of Inner-IoU, more detailed gradient feedback can be provided for high IoU samples, accelerating convergence. Second, MPDIoU further considers the keypoint distance of the box, distinguishing between cases where the predicted box is fully embedded and moderately beyond the ground-truth box, thereby prompting the model to make more refined adjustments to the structure and position during the overall regression process. Finally, the combined loss formed by the two provides a continuous and reasonable gradient signal under different error distributions, improving the model's detection accuracy in multi-scale and complex background scenes. Experimental results demonstrate that this combined method exhibits significant advantages in scenes with low illumination and severe occlusion, providing more accurate and robust bounding box regression support for multimodal object detection systems.

[0110] This embodiment has the following beneficial effects:

[0111] 1. This embodiment introduces a channel and spatial joint perception module (CPCA) based on a dual-stream feature extraction network. This module can enhance the perception of spatial structure and semantic details of infrared and visible light images within their respective modalities, effectively improving the integrity and discriminability of feature expression.

[0112] 2. The proposed Cross-Modal Feature Fusion Module (CFFM) improves the complementary and collaborative capabilities between different modalities by modeling modal difference information and combining it with a spatial attention mechanism, enabling precise perception and information interaction in key areas and addressing the information redundancy or mismatch issues of traditional fusion methods.

[0113] 3. Introducing a composite bounding box regression loss function consisting of the minimum point distance IoU and InnerIoU in the detection head can more accurately measure the degree of match between the predicted box and the true box, effectively improving the accuracy of target positioning, training stability, and detection robustness in complex scenarios.

[0114] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc.

[0115] The processor can be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the infrared and visible light target detection methods described in the various embodiments of the present disclosure.

[0116] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0117] like Figure 4 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device according to an embodiment of the present invention. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0118] like Figure 4 As shown, an electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0119] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes and hard disks; and communication devices. The communication device allows the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 4 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0120] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the infrared and visible light target detection method of the embodiment of the present disclosure are performed.

[0121] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0122] The computer-readable storage medium disclosed in this embodiment stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the infrared and visible light target detection method of each embodiment of the present disclosure are executed.

[0123] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0124] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0125] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0126] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0127] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0128] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0129] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0130] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0131] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for detecting infrared and visible light targets, characterized in that: include: Feature extraction is performed on infrared and visible light images respectively to obtain feature maps at multiple scales; Improve the representation and discrimination capabilities of each scale feature and obtain multiple enhanced feature maps; Perform cross-modal fusion on multiple enhanced feature maps to obtain fused multi-scale features; Target detection is achieved based on the fused multi-scale features and a composite loss function that combines the inner intersection-over-union (IoU) and the minimum point distance IoU. The above method improves the representation and discrimination capabilities of each scale feature to obtain multiple enhanced feature maps, including: Each scale feature includes infrared features and visible light features; The infrared features and visible light features are spliced ​​along the channel dimension to obtain the joint features; Perform global average pooling and maximum pooling on the joint features to extract the channel description vector; The channel description vector is convolved to construct a compressed excitation structure to obtain a channel attention weight map; Enhance the features based on the attention weight map to obtain enhanced features; Based on the multi-scale direction-aware convolutional structure, the enhanced features are processed to obtain spatial feature responses in multiple directions; The spatial feature responses in all directions are added to the enhanced features and then convolved to obtain the spatial attention features; The spatial attention feature is fused with the joint feature based on the residual connection to obtain the enhanced modality fusion feature; The cross-modal fusion of multiple enhanced feature maps to obtain fused multi-scale features includes: Calculate the differential mode information between the enhanced features from infrared and the enhanced features from visible light to obtain the differential mode features; Then, each differential mode feature is subjected to maximum pooling and average pooling operations in the channel direction, and then the spatial feature representation is extracted by convolution, and the spatial attention weight map is output using the Sigmoid function; Apply the spatial attention weight map to the corresponding differential mode features to obtain enhanced differential mode features; Based on the enhanced differential mode features, the spatial attention weight map and the differential mode features, the enhanced feature maps corresponding to infrared and the enhanced feature maps corresponding to visible light are obtained respectively; The enhanced feature map corresponding to infrared and the enhanced feature map corresponding to visible light are spliced ​​in the channel dimension to obtain the fused multimodal features.

2. The infrared and visible light target detection method according to claim 1, characterized in that: The feature extraction is performed on the infrared and visible light images respectively to obtain feature maps of multiple scales, including: Based on the constructed dual-stream feature extraction and enhancement module, feature extraction is performed on infrared and visible light images respectively; The dual-stream feature extraction and enhancement module includes two independent backbone networks, which are used to extract modal features of infrared images and visible light images respectively; The Backbone network includes basic convolutional layers, C2f structures, and spatial pyramid pooling modules.

3. The infrared and visible light target detection method according to claim 1, characterized in that: The enhanced modal fusion feature is the sum of the product of the spatial attention feature and the enhanced feature and the joint feature.

4. The infrared and visible light target detection method according to claim 1, characterized in that: The composite loss function combining the inner intersection-union ratio and the minimum point distance intersection-union ratio includes: The inner intersection and union ratio and the minimum point distance intersection and union ratio are based on Perform weighted combination, A hyperparameter that controls the contribution weights of the inner IoU loss and the minimum point distance IoU loss.

5. The infrared and visible light target detection method according to claim 4, characterized in that: The internal cross comparison includes: By setting the scaling ratio, auxiliary frames corresponding to the predicted frame and the real frame are constructed respectively; Calculate the inner intersection and union ratio based on the auxiliary box.

6. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the infrared and visible light target detection method described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the infrared and visible light target detection method according to any one of claims 1 to 5.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the infrared and visible light target detection method according to any one of claims 1 to 5 is implemented.

Images