Infrared-visible light fusion target detection method and device based on conditional perception

Abstract

The application relates to the technical field of target detection, in particular to an infrared-visible light fusion target detection method and device based on conditional perception, wherein the method comprises the following steps: acquiring first and second image data; extracting first and second multi-scale features; obtaining an environment token through an environment encoder and a learnable embedding; extracting third and fourth multi-scale features by using a feature adapter; fusing the third and fourth multi-scale features through the environment token to obtain enhanced features; generating a feature sequence based on the enhanced features, and determining an initialized target query; and inputting the initialized target query and the feature sequence into a multispectral decoder to output a target detection result. Thus, the problem that the feature alignment accuracy and the robustness in a complex environment of an IR- RGB fusion method in related technologies are still insufficient is solved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a condition-aware infrared-visible light fusion target detection method and apparatus. Background Technology

[0002] In fields such as autonomous driving and security monitoring, object detection is a critical task for ensuring system safety and reliability. Visible light (RGB) cameras provide rich color, texture, and shape information, but their performance degrades under extreme conditions such as low light, strong fog, and occlusion. Infrared (IR) sensors can detect the thermal radiation of objects in dark environments and perform well in low visibility conditions such as nighttime, smoke, rain, and snow, but lack detailed texture. LiDAR sensors provide accurate depth information and are widely used for 3D object detection, but they are susceptible to interference under strong light and rain / snow conditions. Therefore, fusing these multimodal information can make object detection more robust in complex environments. However, how to effectively fuse features from different modalities remains a significant research challenge.

[0003] Early research on multimodal target detection primarily focused on the fusion of RGB and LiDAR sensors to compensate for the limitations of single sensors. For example, VoxelNet voxels LiDAR point clouds and extracts voxel features using 3D convolution, combining these features with RGB visual features for detection. PointPillars proposed the Pillar structure, projecting point clouds into 2D space for efficient processing while simultaneously combining RGB information to improve detection accuracy. In recent years, IR-RGB sensor fusion has become a research hotspot, especially for target detection tasks in extreme environments such as low light, fog, and rain / snow. For instance, FusionNet uses a CNN structure to extract features from IR and RGB and fuses them at multiple levels to improve detection performance; CrossModalNet uses a cross-modal attention mechanism to dynamically exchange key information between IR and RGB to enhance detection results; and MBNet proposes a modal balancing mechanism to reduce the impact of single-modal failure on the final detection results.

[0004] However, IR-RGB fusion methods in related technologies typically process sensor modes uniformly under all conditions, resulting in poor target detection performance in complex environments. Furthermore, the differences in resolution and noise characteristics between IR and RGB images complicate feature alignment and fusion, leaving shortcomings in feature alignment accuracy and robustness in complex environments, which urgently need improvement. Summary of the Invention

[0005] This application provides a condition-aware infrared-visible light fusion target detection method and apparatus to address the problems in related technologies, such as the differences in resolution and noise characteristics between IR and RGB images, and the inadequacy in feature alignment accuracy and robustness in complex environments caused by the uniform processing of sensor modes under all conditions.

[0006] The first aspect of this application provides a condition-aware infrared-visible light fusion target detection method, comprising the following steps: acquiring first image data collected by an infrared sensor and second image data collected by a visible light sensor; extracting first multi-scale features of the first image data and second multi-scale features of the second image data based on a shared pre-trained backbone network, the first image data, and the second image data; obtaining an environment token by using an environment encoder that meets preset lightweight conditions and a learnable embedding; extracting third multi-scale features and fourth multi-scale features from the first multi-scale features and the second multi-scale features respectively using a modality-specific feature adapter; fusing the third multi-scale features and the fourth multi-scale features in the latent space guided by the environment token to obtain enhanced features; generating a feature sequence based on the enhanced features and determining an initialized target query, inputting the initialized target query and the feature sequence into a multispectral decoder, and outputting a target detection result.

[0007] Through the above technical solution, the embodiments of this application can obtain an environment token by using an environment encoder that meets preset lightweight conditions and a learnable embedding. The environment token, which encodes the current environment information, is used as input to a multilayer perceptron to output weight parameters corresponding to IR and RGB images, respectively. Based on these weight parameters, the fusion strategy for IR and RGB features is dynamically adjusted to ensure optimal contribution of each modality under different environmental conditions. Simultaneously, IR and RGB features are projected into a shared latent space through a modality-specific feature adapter, effectively eliminating the differences in resolution and noise characteristics between IR and RGB images, achieving alignment of the two modal features, and ensuring efficient feature fusion in the shared backbone network.

[0008] Optionally, in one embodiment of this application, generating a feature sequence based on the enhanced features includes: generating a multi-scale feature map based on the enhanced features; flattening the multi-scale feature map and stitching together the feature maps of all scales to generate the feature sequence.

[0009] Through the above technical solutions, the embodiments of this application can flatten and stitch multi-scale feature maps to generate feature sequences, thereby integrating semantic and spatial information at different levels and enhancing the model's ability to capture complex visual patterns.

[0010] Optionally, in one embodiment of this application, the flattening formula for the multi-scale feature map is:

[0011]

[0012] in, This indicates a splicing operation. Indicates the flattening operation. This represents the multi-scale features extracted from the IR image. This represents the multi-scale features extracted from the RGB image. Indicates the space to which the image belongs. , These refer to the number of channels, width, and height of the image, respectively.

[0013] Through the above technical solution, the embodiments of this application can convert all scale features into a unified feature sequence by flattening and stitching the multi-scale features of two modal images, and input it into the self-attention mechanism of the encoder, thereby providing strong data support for target detection.

[0014] Optionally, in one embodiment of this application, the step of guiding the fusion of the third multi-scale features and the fourth multi-scale features in the latent space through the environment token to obtain enhanced features includes: obtaining a first fusion weight of the first image data and a second fusion weight of the second image data based on the environment token; and using the first fusion weight and the second fusion weight to guide the fusion of the third multi-scale features and the fourth multi-scale features in the latent space.

[0015] Through the above technical solution, the embodiments of this application can use environment tokens to guide the fusion of third-scale features and fourth-scale features in the hidden space, and can dynamically adjust the fusion weights of IR and RGB features to ensure that the contribution of each mode is optimal under different environmental conditions.

[0016] A second aspect of this application provides a condition-aware infrared-visible light fusion target detection device, comprising: an acquisition module for acquiring first image data acquired by an infrared sensor and second image data acquired by a visible light sensor; a first extraction module for extracting first multi-scale features of the first image data and second multi-scale features of the second image data based on a shared pre-trained backbone network, the first image data, and the second image data; a generation module for obtaining an environment token by using an environment encoder that meets preset lightweight conditions and a learnable embedding; a second extraction module for extracting third multi-scale features and fourth multi-scale features from the first multi-scale features and the second multi-scale features, respectively, using a modality-specific feature adapter; a fusion module for fusing the third multi-scale features and the fourth multi-scale features in a latent space guided by the environment token to obtain enhanced features; and a detection module for generating a feature sequence based on the enhanced features and determining an initialized target query, inputting the initialized target query and the feature sequence into a multispectral decoder, and outputting a target detection result.

[0017] Through the above technical solution, the embodiments of this application can dynamically adjust the fusion strategy of IR and RGB features by using an environment encoder that meets preset lightweight conditions and an environment token obtained through learnable embedding, ensuring that the contribution of each modality is optimal under different environmental conditions. Simultaneously, by using a modality-specific feature adapter, IR and RGB features are projected onto a shared latent space, ensuring efficient feature fusion in a shared backbone network.

[0018] Optionally, in one embodiment of this application, the detection module includes: a first generation unit, configured to generate a multi-scale feature map based on the enhanced features; and a second generation unit, configured to flatten the multi-scale feature map and stitch together the feature maps of all scales to generate the feature sequence.

[0019] Through the above technical solutions, the embodiments of this application can flatten and stitch multi-scale feature maps to generate feature sequences, thereby integrating semantic and spatial information at different levels and enhancing the model's ability to capture complex visual patterns.

[0020] Optionally, in one embodiment of this application, the flattening formula for the multi-scale feature map is:

[0021]

[0022] in, This indicates a splicing operation. Indicates the flattening operation. This represents the multi-scale features extracted from the IR image. This represents the multi-scale features extracted from the RGB image. Indicates the space to which the image belongs. , These refer to the number of channels, width, and height of the image, respectively.

[0023] Through the above technical solution, the embodiments of this application can convert all scale features into a unified feature sequence by flattening and stitching the multi-scale features of two modal images, and input it into the self-attention mechanism of the encoder, thereby providing strong data support for target detection.

[0024] Optionally, in one embodiment of this application, the fusion module includes: a third generation unit, configured to obtain a first fusion weight of the first image data and a second fusion weight of the second image data based on the environment token; and a guidance unit, configured to guide the third multi-scale feature and the fourth multi-scale feature to be fused in the latent space using the first fusion weight and the second fusion weight.

[0025] Through the above technical solution, the embodiments of this application can use environment tokens to guide the fusion of third-scale features and fourth-scale features in the hidden space, and can dynamically adjust the fusion weights of IR and RGB features to ensure that the contribution of each mode is optimal under different environmental conditions.

[0026] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the condition-aware infrared-visible light fusion target detection method as described in the above embodiments.

[0027] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the condition-aware infrared-visible fusion target detection method described above.

[0028] A fifth aspect of this application provides a computer program product that stores a computer program that, when executed by a processor, implements the condition-aware infrared-visible fusion target detection method described above.

[0029] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0030] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0031] Figure 1 This is a flowchart of a condition-aware infrared-visible light fusion target detection method according to an embodiment of this application;

[0032] Figure 2 This is a schematic diagram illustrating the process of generating and optimizing an environment token according to a specific embodiment of this application;

[0033] Figure 3 This is a schematic flowchart of a condition-aware infrared-visible light fusion target detection method according to a specific embodiment of this application;

[0034] Figure 4 This is a schematic diagram of a condition-aware infrared-visible light fusion target detection device according to an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0037] The following description, with reference to the accompanying drawings, illustrates a condition-aware infrared-visible light fusion target detection method and apparatus according to embodiments of this application. Addressing the issues raised in the background section regarding the differences in resolution and noise characteristics between IR and RGB images in related IR-RGB fusion methods, and the inadequacy in feature alignment accuracy and robustness in complex environments due to the uniform processing of sensor modalities under all conditions, this application provides a condition-aware infrared-visible light fusion target detection method. In this method, the fusion strategy for IR and RGB features is dynamically adjusted by using an environment encoder that meets preset lightweight conditions and an environment token obtained through learnable embedding, ensuring optimal contribution of each modality under different environmental conditions. Simultaneously, IR and RGB features are projected into a shared latent space through a modality-specific feature adapter, ensuring efficient feature fusion within a shared backbone network. This resolves the problems in related IR-RGB fusion methods, such as the differences in resolution and noise characteristics between IR and RGB images, and the inadequacy in feature alignment accuracy and robustness in complex environments due to the uniform processing of sensor modalities under all conditions.

[0038] Specifically, Figure 1This is a schematic flowchart of a condition-aware infrared-visible light fusion target detection method provided in an embodiment of this application.

[0039] like Figure 1 As shown, this condition-aware infrared-visible fusion target detection method includes the following steps:

[0040] In step S101, the first image data acquired by the infrared sensor and the second image data acquired by the visible light sensor are obtained.

[0041] Infrared sensors include, but are not limited to, thermal imagers and infrared cameras; visible light sensors include, but are not limited to, ambient light sensors, color light sensors, and image sensors; the specific models of the above two types of sensors can be set by those skilled in the art in combination with the actual scenario.

[0042] For example, in the embodiments of this application, a first image data can be acquired by a thermal imager, and a second image data can be acquired by an ambient light sensor. The two types of image data are obtained by combining hardware connection, driver configuration, data reading and software processing.

[0043] The embodiments of this application can obtain complex environmental information by acquiring first image data collected by an infrared sensor and second image data collected by a visible light sensor, thereby improving the comprehensiveness and accuracy of analysis in complex scenarios.

[0044] In step S102, based on the shared pre-trained backbone network, the first image data, and the second image data, the first multi-scale features of the first image data and the second multi-scale features of the second image data are extracted.

[0045] Pre-trained backbone networks include, but are not limited to, convolutional neural network series (such as AlexNet, VGG, ResNet), Transformer series, etc. In this embodiment, ResNet can be selected as the pre-trained backbone network.

[0046] As one possible approach, embodiments of this application can input image data acquired from the two types of sensors described above into a ResNet backbone network for multi-scale feature extraction; the ResNet backbone network generates a set of multi-scale feature maps. , where each feature map Represents features at different scales. This indicates the category to which the feature map belongs.

[0047] Specifically, embodiments of this application can extract multi-scale features of IR images using a shared pre-trained backbone network (ResNet). Multiscale features of RGB images . , representing feature maps at different scales. The multi-scale feature maps extracted above contain rich spatial and contextual information, providing a reliable data foundation for subsequent object detection.

[0048] In step S103, an environment token is obtained by an environment encoder that meets preset lightweight conditions and a learnable embedding.

[0049] Understandably, "lightweight conditions" are a type of condition constraint or control mechanism with low computational overhead, typically used to dynamically adjust the behavior of a model in resource-constrained environments (such as mobile devices and embedded systems) without significantly increasing model complexity.

[0050] An environmental encoder is a model component that encodes environmental perception data (such as sensor inputs and scene information) into low-dimensional feature vectors. Essentially, it is a feature extractor that transforms high-dimensional, multimodal raw environmental data (such as images, point clouds, radar signals, etc.) into compact, semantically rich vector representations.

[0051] "Learnable embeddings" refer to low-dimensional vector representations that are automatically learned through a data-driven approach. They can map discrete, high-dimensional inputs (such as categories, words, graph nodes, etc.) to a continuous embedding space while preserving their semantic or structural relationships.

[0052] In a specific embodiment, such as Figure 2 As shown, environment tokens can be obtained in the following ways:

[0053] First, the multi-scale feature map extracted in step S102 is... After flattening using the following formula, feature maps at all scales are concatenated into a long sequence. :

[0054] (1)

[0055] The reshape operation transforms the feature map from... Flattened, it is a sequence with dimensions of The sequence length is The Concat operation concatenates 2*L (here L=4) feature sequences into a long sequence. After the above processing, the feature maps of all scales will become a unified feature sequence, which can be input into the self-attention mechanism of the encoder.

[0056] Secondly, an MLP (Multilayer Perceptron) is used to extract features from both modalities. :

[0057] (2)

[0058] Finally, the Context Token (CT) is obtained through a learnable embedding W:

[0059] (3)

[0060] After obtaining the environment token, the matrix W can be optimized through visual-linguistic contrastive loss learning, enabling the environment token to more accurately describe environmental information. The specific optimization process can be implemented as follows:

[0061] First, with the corresponding text environment description Comparative learning:

[0062] (4)

[0063] in, It is a text encoder obtained after pre-training (the CLIP text encoder can be used).

[0064] Then, contrastive loss is used to compare the environment token (CT) with the correct... Higher similarity:

[0065] (5)

[0066] in, (A, B) represents the similarity between A and B (cosine similarity is used here). Indicates a negative sample (incorrect environmental description);

[0067] By minimizing ,optimization This allows the Environment Token (CT) to learn to align with real-world environmental information.

[0068] Through the above technical solution, the embodiments of this application can dynamically adjust the fusion strategy of IR and RGB features by obtaining and optimizing the environment token, so that the environment token learns meaningful environmental information and ensures that the contribution of each mode is optimal under different environmental conditions.

[0069] In step S104, a modality-specific feature adapter is used to extract the third and fourth multi-scale features from the first and second multi-scale features, respectively.

[0070] The modality-specific feature adapters mentioned above can be lightweight convolutional neural networks with 1*1 convolutional layers.

[0071] In this embodiment, a lightweight convolutional neural network with 1*1 convolutional layers can be used to project the first and second multi-scale features of IR and RGB onto a shared latent space, thereby obtaining the third and fourth multi-scale features.

[0072] The differences in resolution and noise characteristics between IR and RGB images make feature alignment and fusion complex. By setting modality-specific feature adapters to project IR and RGB features into a shared latent space, feature alignment in a shared backbone network can be ensured, facilitating efficient feature fusion in subsequent processes.

[0073] In step S105, the third and fourth multi-scale features are fused in the latent space using an environment token to obtain the enhanced features.

[0074] Specifically, since the reliability of IR and RGB varies under different environments (such as daytime, nighttime, foggy weather, etc.), the fusion ratio of the two modalities can be adaptively adjusted using an environment token (CT). For example, embodiments of this application can employ adaptive weighting fusion, specifically calculated as follows:

[0075] (6)

[0076] In this model, CT encodes the current environmental information, and MLP (Multilayer Perceptron) is a small neural network that takes CT as input and outputs two weight parameters. The Softmax function makes... The value range of is (0, 1), while ensuring that the sum of the two is 1. For IR image fusion weights, The fusion weights for RGB images.

[0077] Yes Then, the fusion features can be calculated:

[0078] (7)

[0079] in, and These are the features projected onto the shared hidden space through a feature adapter (1*1 convolutional layer), namely the third and fourth multi-scale features. These are the final fused features, which are then input into the object detection network for further processing.

[0080] Through the above technical solution, the embodiments of this application can adaptively adjust the fusion ratio of the two modes based on the environment token, and then perform feature fusion based on the fusion ratio of the IR and RGB modes combined with the features projected onto the shared hidden space through the feature adapter (1*1 convolutional layer) in S104, thereby enhancing the features. This can ensure the feature alignment accuracy and improve the robustness of feature fusion in complex environments.

[0081] In step S106, a feature sequence is generated based on the enhanced features, and the initialized target query is determined. The initialized target query and feature sequence are then input into the multispectral decoder to output the target detection result.

[0082] In actual implementation, the enhanced features can be flattened again into feature sequence Z. The top K most significant queries are selected as the initial features of the target queries in the subsequent decoding process. The initialized target queries and the fused feature sequences are input into the multispectral decoder. The decoder uses a multi-head attention mechanism to decode the bounding boxes and class labels as the prediction results.

[0083] The aforementioned decoder can be based on a feedforward neural network and a multi-head self-attention mechanism, and can output the target box in the last decoder layer through a layer-by-layer attention refinement method. and class tags As a prediction result.

[0084] Specifically, the explanation regarding query initialization is as follows:

[0085] In embodiments of this application, query initialization can be performed using fusion features optimized with Context Tokens (CT).

[0086] (8)

[0087] This represents the K features selected as the query. Top-K() represents the selection of the top K most significant query features. Linear represents the linear transformation layer, which performs a linear transformation on the fused features.

[0088] The explanation regarding target decoding is as follows:

[0089] The purpose of the target decoding module is to decode the initialized target query to generate the target's bounding box and category label. The target query is then processed by the query refinement module. Input is fed into the multispectral decoder. Target query is performed in the decoder. A multi-head self-attention mechanism interacts with encoded features Z to extract target information corresponding to each query. The multi-head self-attention mechanism extracts the spatial location and category information of targets by focusing on different regions in the image using different attention heads. Each attention head calculates the bounding box and category score corresponding to the query. The final decoder output includes the bounding box location for each target. and category labels .in, Indicates the coordinates of the center point of the bounding box. and These represent the width and height of the bounding box, respectively.

[0090] The initial target query and the fused feature sequence are input into the multispectral decoder. In the decoder, the target query can interact with the encoded features through a multi-head self-attention mechanism to extract the target information corresponding to each query. The decoder can also calculate the bounding box and class score corresponding to the query to achieve target detection.

[0091] like Figure 3 As shown, the specific execution process of the target detection method described above can be referred to by those skilled in the art. Figure 3 The given process can be further understood.

[0092] Optionally, in one embodiment of this application, generating a feature sequence based on the enhanced features includes: generating a multi-scale feature map based on the enhanced features; flattening the multi-scale feature map; and stitching together the feature maps of all scales to generate a feature sequence.

[0093] Those skilled in the art should understand that, in generating the environment token as described above, after obtaining the multi-scale feature map, it can be flattened and concatenated into a long sequence. Then, using MLP, features under the long sequence can be extracted and the environment token can be obtained through a learnable embedding. The environment token can guide the fusion of IR and RGB features to obtain the enhanced features. After obtaining the enhanced features, a multi-scale feature map can also be generated based on the enhanced features. The multi-scale feature map can be flattened and concatenated to generate a feature sequence, which can be input into the multispectral decoder along with the initialized target query to obtain the detection result.

[0094] The core function of flattening and stitching multi-scale feature maps to generate feature sequences is to integrate semantic and spatial information at different levels, thereby enhancing the model's ability to capture complex visual patterns.

[0095] Optionally, in one embodiment of this application, the flattening formula for the multi-scale feature map is:

[0096] (1)

[0097] in, This indicates a splicing operation. Indicates the flattening operation. This represents the multi-scale features extracted from the IR image. This represents the multi-scale features extracted from the RGB image. Indicates the space to which the image belongs. , These refer to the number of channels, width, and height of the image, respectively.

[0098] This application embodiment can convert all scale features into a unified feature sequence by flattening and stitching the multi-scale features of two modal images, and input it into the self-attention mechanism of the encoder, thus providing strong data support for target detection.

[0099] Optionally, in one embodiment of this application, the third multi-scale feature and the fourth multi-scale feature are fused in the latent space using an environment token to obtain the enhanced feature, including: obtaining a first fusion weight of the first image data and a second fusion weight of the second image data based on the environment token; and using the first fusion weight and the second fusion weight to guide the fusion of the third multi-scale feature and the fourth multi-scale feature in the latent space.

[0100] In actual implementation, the embodiments of this application can adaptively adjust the fusion ratio of the two images after obtaining the environment token, thereby obtaining the fusion weight of the IR image. fusion weights with RGB images Okay, I have and Then, the features projected onto the shared hidden space by the feature adapter (1*1 convolutional layer) are assigned corresponding weights to obtain the final fused features.

[0101] The above technical solution utilizes environment tokens to guide the fusion of third-scale and fourth-scale features in the latent space, dynamically adjusting the fusion weights of IR and RGB features to ensure optimal contribution of each modality under different environmental conditions.

[0102] The condition-aware infrared-visible light fusion target detection method proposed in this application dynamically adjusts the fusion strategy of IR and RGB features by using an environmental encoder that meets preset lightweight conditions and an environmental token obtained through learnable embedding, ensuring optimal contribution of each modality under different environmental conditions. Simultaneously, IR and RGB features are projected into a shared latent space through a modality-specific feature adapter, ensuring efficient feature fusion in a shared backbone network. This solves the problems in related IR-RGB fusion methods, such as differences in resolution and noise characteristics between IR and RGB images, and the inadequacy in feature alignment accuracy and robustness in complex environments due to uniform processing of sensor modalities under all conditions.

[0103] Next, refer to the appendix. Figure 4 This application describes a condition-aware infrared-visible light fusion target detection device based on embodiments thereof.

[0104] Figure 4 This is a block diagram of a condition-aware infrared-visible light fusion target detection device according to an embodiment of this application.

[0105] like Figure 4 As shown, the condition-aware infrared-visible light fusion target detection device 10 includes: an acquisition module 100, a first extraction module 200, a generation module 300, a second extraction module 400, a fusion module 500, and a detection module 600.

[0106] The acquisition module 100 is used to acquire the first image data collected by the infrared sensor and the second image data collected by the visible light sensor.

[0107] The first extraction module 200 is used to extract first multi-scale features of the first image data and second multi-scale features of the second image data based on a shared pre-trained backbone network, first image data and second image data.

[0108] The generation module 300 is used to obtain an environment token by an environment encoder that meets preset lightweight conditions and a learnable embedding.

[0109] The second extraction module 400 is used to extract the third and fourth multi-scale features from the first and second multi-scale features, respectively, using a modality-specific feature adapter.

[0110] The fusion module 500 is used to guide the fusion of third-scale and fourth-scale features in the latent space through an environment token to obtain enhanced features.

[0111] The detection module 600 is used to generate a feature sequence based on the enhanced features and determine the initialized target query, so as to input the initialized target query and feature sequence into the multispectral decoder and output the target detection result.

[0112] Optionally, in one embodiment of this application, the detection module 600 includes: a first generation unit and a second generation unit; wherein, the first generation unit is used to generate a multi-scale feature map based on the enhanced features; and the second generation unit is used to flatten the multi-scale feature map and stitch together the feature maps of all scales to generate a feature sequence.

[0113] Optionally, in one embodiment of this application, the flattening formula for the multi-scale feature map is:

[0114]

[0115] in, This indicates a splicing operation. Indicates the flattening operation. This represents the multi-scale features extracted from the IR image. This represents the multi-scale features extracted from the RGB image. Indicates the space to which the image belongs. , These refer to the number of channels, width, and height of the image, respectively.

[0116] Optionally, in one embodiment of this application, the fusion module 500 includes: a third generation unit and a guidance unit; wherein, the third generation unit is used to obtain a first fusion weight of the first image data and a second fusion weight of the second image data based on an environment token; the guidance unit is used to guide the fusion of the third multi-scale features and the fourth multi-scale features in the latent space using the first fusion weight and the second fusion weight.

[0117] It should be noted that the foregoing explanation of the embodiment of the condition-aware infrared-visible light fusion target detection method also applies to the condition-aware infrared-visible light fusion target detection device of this embodiment, and will not be repeated here.

[0118] The condition-aware infrared-visible light fusion target detection device proposed in this application can dynamically adjust the fusion strategy of IR and RGB features by using an environmental encoder that meets preset lightweight conditions and an environmental token obtained through learnable embedding, ensuring optimal contribution of each modality under different environmental conditions. Simultaneously, IR and RGB features are projected into a shared latent space through a modality-specific feature adapter, ensuring efficient feature fusion in a shared backbone network. This solves the problems in related IR-RGB fusion methods, such as differences in resolution and noise characteristics between IR and RGB images, and the inadequacy in feature alignment accuracy and robustness under complex environments due to uniform processing of sensor modalities under all conditions.

[0119] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0120] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0121] When the processor 502 executes the program, it implements the condition-aware infrared-visible light fusion target detection method provided in the above embodiments.

[0122] Furthermore, electronic devices also include:

[0123] Communication interface 503 is used for communication between memory 501 and processor 502.

[0124] The memory 501 is used to store computer programs that can run on the processor 502.

[0125] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0126] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0127] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0128] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0129] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described condition-aware infrared-visible light fusion target detection method.

[0130] This application also provides a computer program product on which a computer program is stored, which, when executed by a processor, implements the above-described condition-aware infrared-visible light fusion target detection method.

[0131] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0132] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0133] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0134] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0135] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0136] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0138] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A condition-aware infrared-visible light fusion target detection method, characterized in that, Includes the following steps: Acquire the first image data collected by the infrared sensor and the second image data collected by the visible light sensor; Based on the shared pre-trained backbone network, the first image data, and the second image data, the first multi-scale features of the first image data and the second multi-scale features of the second image data are extracted. An environment token is obtained by using an environment encoder that meets preset lightweight conditions and a learnable embedding. The third and fourth multi-scale features are extracted from the first and second multi-scale features, respectively, using a modality-specific feature adapter; The environment token guides the fusion of the third and fourth multi-scale features in the hidden space to obtain the enhanced features. Based on the enhanced features, a feature sequence is generated, and an initialized target query is determined. The initialized target query and the feature sequence are then input into a multispectral decoder to output the target detection result. The process of obtaining an environment token through an environment encoder that meets preset lightweight conditions and a learnable embedding includes: The first multi-scale feature and the second multi-scale feature are flattened and concatenated into a feature sequence. ; The environmental features of the feature sequence are extracted using a multilayer perceptron. ; The environmental features are combined with a learnable embedding matrix Multiply to obtain the environment token CT: 。 2. The method according to claim 1, characterized in that, The generation of feature sequences based on the enhanced features includes: A multi-scale feature map is generated based on the enhanced features; The multi-scale feature map is flattened, and the feature maps of all scales are stitched together to generate the feature sequence.

3. The method according to claim 2, characterized in that, The flattening formula for the multi-scale feature map is: in, This indicates a splicing operation. Indicates the flattening operation. This represents the multi-scale features extracted from the IR image. This represents the multi-scale features extracted from the RGB image. Indicates the space to which the image belongs. , These refer to the number of channels, width, and height of the image, respectively.

4. The method according to claim 1, characterized in that, The process of fusing the third and fourth multi-scale features in the latent space using the environment token to obtain enhanced features includes: Based on the environment token, the first fusion weight of the first image data and the second fusion weight of the second image data are obtained respectively; The first fusion weight and the second fusion weight are used to guide the fusion of the third multi-scale feature and the fourth multi-scale feature in the latent space.

5. A condition-aware infrared-visible light fusion target detection device, characterized in that, include: The acquisition module is used to acquire the first image data collected by the infrared sensor and the second image data collected by the visible light sensor; The first extraction module is used to extract a first multi-scale feature of the first image data and a second multi-scale feature of the second image data based on a shared pre-trained backbone network, the first image data, and the second image data. The generation module is used to obtain an environment token by using an environment encoder that meets preset lightweight conditions and a learnable embedding. The second extraction module is used to extract the third multi-scale feature and the fourth multi-scale feature from the first multi-scale feature and the second multi-scale feature, respectively, using a modality-specific feature adapter. The fusion module is used to guide the fusion of the third multi-scale feature and the fourth multi-scale feature in the latent space through the environment token to obtain the enhanced feature; The detection module is used to generate a feature sequence based on the enhanced features and determine the initialized target query, so as to input the initialized target query and the feature sequence into the multispectral decoder and output the target detection result; The process of obtaining an environment token through an environment encoder that meets preset lightweight conditions and a learnable embedding includes: The first multi-scale feature and the second multi-scale feature are flattened and concatenated into a feature sequence. ; The environmental features of the feature sequence are extracted using a multilayer perceptron. ; The environmental features are combined with a learnable embedding matrix Multiply to obtain the environment token CT: 。 6. The apparatus according to claim 5, characterized in that, The detection module includes: The first generation unit is used to generate a multi-scale feature map based on the enhanced features; The second generation unit is used to flatten the multi-scale feature map and stitch together the feature maps of all scales to generate the feature sequence.

7. The apparatus according to claim 6, characterized in that, The flattening formula for the multi-scale feature map is: in, This indicates a splicing operation. Indicates the flattening operation. This represents the multi-scale features extracted from the IR image. This represents the multi-scale features extracted from the RGB image. Indicates the space to which the image belongs. , These refer to the number of channels, width, and height of the image, respectively.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the condition-aware infrared-visible fusion target detection method as described in any one of claims 1-4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the condition-aware infrared-visible fusion target detection method as described in any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the condition-aware infrared-visible fusion target detection method as described in any one of claims 1-4.

Images