A target detection method and apparatus

Abstract

Embodiments of the present application disclose a target detection method and device, which can obtain a visible light image and its corresponding infrared image under the same scene, and respectively perform feature extraction on the visible light image and its corresponding infrared image; perform saliency detection on the visible light image, and generate three groups of spatial attention weights based on a saliency map; input an initial visible light feature map and an initial infrared feature map into three parallel feature extraction branches respectively, to obtain a corresponding visible light multi-scale feature group and an infrared multi-scale feature group; based on the three groups of spatial attention weights, respectively perform element-by-element weighted fusion on feature maps output by each branch in the visible light multi-scale feature group and the infrared multi-scale feature group, to obtain enhanced visible light features and enhanced infrared features; perform feature cross fusion processing on the enhanced visible light features and the enhanced infrared features, to obtain a cross fusion map; and perform target detection according to the cross fusion map. Thus, embodiments of the present application improve target detection accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a target detection method and apparatus. Background Technology

[0002] Object detection is a fundamental task in computer vision, aiming to locate and identify one or more objects of interest from a given image. Specifically, object detection not only determines the presence of objects of a specific category (such as people, vehicles, animals, etc.) in an image, but also accurately labels the location of each object within the image, typically represented by bounding boxes, and assigns a category label and confidence score to each bounding box. Object detection is widely used in fields such as intelligent surveillance, autonomous driving, drone inspection, remote sensing analysis, and robot perception, and is one of the key technologies for achieving environmental perception and intelligent decision-making.

[0003] However, existing methods still have significant limitations in multimodal object detection: on the one hand, using separate detection models for visible light and infrared images significantly increases the resource consumption for model storage and runtime; on the other hand, if the two modalities are input into the same model and fused through simple channel concatenation or element-wise addition, it is impossible to distinguish the degree of contribution of different regions in the image to the detection task, resulting in the failure to effectively integrate useful information between modalities, and redundant or conflicting features are treated equally. In addition, existing single-model architectures usually aggregate features in a fixed way, lacking the ability to dynamically adjust the multi-branch feature fusion strategy according to the image content, making it difficult to fully utilize the complementary characteristics of features at different scales in local detail and structural representation, thus limiting the detection accuracy. Summary of the Invention

[0004] This application provides a target detection method and apparatus that can improve the accuracy of target detection.

[0005] This application provides a target detection method, the method comprising:

[0006] Acquire visible light images and their corresponding infrared images in the same scene, and extract features from the visible light images and their corresponding infrared images respectively to obtain initial visible light feature maps and initial infrared feature maps;

[0007] Saliency detection is performed on the visible light image to generate the corresponding saliency map. Based on the saliency map, three sets of spatial attention weights are generated. The three sets of spatial attention weights are non-negative at each pixel position and their sum is 1.

[0008] The initial visible light feature map and the initial infrared feature map are respectively input into three parallel feature extraction branches to obtain the corresponding visible light multi-scale feature group and infrared multi-scale feature group. The three feature extraction branches include a depth-separable convolution branch, a dilated convolution branch and a standard convolution branch, and the three feature extraction branches correspond one-to-one with three sets of spatial attention weights.

[0009] Based on three sets of spatial attention weights, the feature maps output by each branch in the visible light multi-scale feature group are fused element-wise to obtain enhanced visible light features, and the feature maps output by each branch in the infrared multi-scale feature group are fused element-wise to obtain enhanced infrared features.

[0010] Enhanced visible light features and enhanced infrared features are subjected to feature cross-fusion processing to obtain a cross-fused image;

[0011] Target detection is performed based on the cross-fusion graph to obtain the target detection results.

[0012] This application embodiment also provides a target detection device, the device comprising:

[0013] The image processing unit is used to acquire the visible light image and its corresponding infrared image in the same scene, and to extract features from the visible light image and its corresponding infrared image respectively to obtain the initial visible light feature map and the initial infrared feature map.

[0014] The weight generation unit is used to perform saliency detection on the visible light image, generate the corresponding saliency map, and generate three sets of spatial attention weights based on the saliency map. The three sets of spatial attention weights are non-negative at each pixel position and their sum is 1.

[0015] The feature extraction unit is used to input the initial visible light feature map and the initial infrared feature map into three parallel feature extraction branches respectively to obtain the corresponding visible light multi-scale feature group and infrared multi-scale feature group. The three feature extraction branches include a depth-separable convolution branch, a dilated convolution branch and a standard convolution branch, and the three feature extraction branches correspond one-to-one with three sets of spatial attention weights.

[0016] The weighted fusion unit is used to perform element-wise weighted fusion of the feature maps output by each branch in the visible light multi-scale feature group based on three sets of spatial attention weights to obtain enhanced visible light features, and to perform element-wise weighted fusion of the feature maps output by each branch in the infrared multi-scale feature group to obtain enhanced infrared features.

[0017] The cross-fusion unit is used to perform feature cross-fusion processing on enhanced visible light features and enhanced infrared features to obtain a cross-fusion map;

[0018] The target detection unit is used to perform target detection based on the cross-fusion graph and obtain the target detection result.

[0019] This application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any of the target detection methods provided in this application.

[0020] This application also provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute steps in any of the target detection methods provided in this application.

[0021] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in any of the target detection methods provided in this application.

[0022] In this application, after acquiring a visible light image and its corresponding infrared image of the same scene, feature extraction can be performed on the visible light image and its corresponding infrared image respectively to obtain an initial visible light feature map and an initial infrared feature map. Furthermore, a corresponding saliency map can be generated based on the visible light image, and then three sets of spatial attention weights can be generated based on the saliency map. The initial visible light feature map and the initial infrared feature map are then input into three parallel feature extraction branches. These three feature extraction branches are a depthwise separable convolution branch, a dilated convolution branch, and a standard convolution branch. Through these branches, feature extraction can be specifically performed from different scales and perspectives. Visible light and infrared multi-scale feature sets are generated. Then, based on three sets of spatial attention weights, the visible light and infrared multi-scale feature maps output from each branch are weighted and fused element-wise. Since these three sets of spatial attention weights are non-negative at each pixel position and sum to 1, and correspond one-to-one with the three feature extraction branches, this precise weighted fusion mechanism can effectively emphasize or suppress different feature information, generating more representative and discriminative enhanced visible light and infrared features. Finally, the enhanced visible light and infrared features are cross-fused to generate a comprehensive cross-fusion map. This process not only integrates information from visible light and infrared images but also fully utilizes the detail enhancement effect brought by the spatial attention mechanism, thereby improving the richness and accuracy of the overall feature representation. Thus, when performing target detection based on the cross-fusion map, it can not only fully utilize the advantages of both modalities but also more accurately distinguish between the target and the background, improving the accuracy of target detection. The entire process, through integrating multi-source data and optimizing feature representation, achieves effective target recognition in complex scenes, thereby improving target detection accuracy. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1a This is a schematic flowchart of a target detection method provided in an embodiment of this application;

[0025] Figure 1b This is a schematic diagram of the feature processing flow of the depth-separable dilated convolution module provided in the embodiments of this application;

[0026] Figure 1c This is a schematic diagram of the feature cross-fusion processing provided in the embodiments of this application;

[0027] Figure 1d This is a schematic diagram of the long-range dependency modeling process provided in the embodiments of this application;

[0028] Figure 1e This is a schematic diagram of the feature processing flow of the context-aware upsampling module provided in the embodiments of this application;

[0029] Figure 1f This is an overall network framework diagram of an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] This application provides a target detection method and apparatus.

[0033] One type of target detection device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0034] In some embodiments, the target detection device may also be integrated into multiple electronic devices. For example, the target detection device may be integrated into multiple servers, and the target detection method of this application may be implemented by multiple servers.

[0035] In some embodiments, the server may also be implemented as a terminal.

[0036] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0037] In this embodiment, a target detection method is provided, such as Figure 1a The specific process of this target detection method can be shown as follows:

[0038] 101. Obtain the visible light image and its corresponding infrared image in the same scene, and extract features from the visible light image and its corresponding infrared image respectively to obtain the initial visible light feature map and the initial infrared feature map.

[0039] In this context, the acquisition of visible light and infrared images from the same scene ensures that they are from the same point in time and the same geographical location, thus guaranteeing a direct correlation between them and facilitating subsequent cross-modal fusion analysis.

[0040] Visible light images are photographs taken by ordinary camera devices under normal lighting conditions, containing rich color and brightness information.

[0041] Infrared images are captured by infrared cameras and primarily reflect the temperature distribution on the surface of an object.

[0042] The initial visible light feature map is the result of preliminary processing of the visible light image using a shared backbone network, aiming to highlight salient features in the image.

[0043] The initial infrared feature map is obtained by using the same shared backbone network to perform preliminary processing on the infrared image, thereby obtaining its unique feature representation and preparing it for subsequent steps.

[0044] 102. Perform saliency detection on the visible light image, generate the corresponding saliency map, and generate three sets of spatial attention weights based on the saliency map. The three sets of spatial attention weights are non-negative at each pixel position and their sum is 1.

[0045] Saliency detection is an image processing technique that simulates the human visual attention mechanism to identify the regions in an image that are most likely to attract visual attention.

[0046] The saliency map is the output of saliency detection. Its size is the same as the input visible light image, and it is usually a single-channel grayscale image. The value of each pixel represents the saliency of that location in the entire image. Areas with higher values ​​usually correspond to edges, areas with strong color contrast, or areas with unique textures, that is, parts that are more easily noticed by the human eye.

[0047] The three sets of spatial attention weights are used to guide the attention of the three subsequent parallel feature extraction branches to regions with different perceptual granularities: for example, high saliency regions focus on detail enhancement, medium saliency regions retain structural information, and low saliency regions suppress redundant responses.

[0048] In some embodiments, generating a saliency map by performing saliency detection on visible light images can effectively identify and highlight regions in the image that are highly relevant to the target (such as pedestrians, vehicles, obstacles, etc.), while suppressing interference from irrelevant background or redundant information. This saliency map serves as the basis for subsequent spatial attention weight generation, enabling the feature enhancement module to adaptively focus on semantically important regions, allocating stronger feature extraction capabilities to highly saliency regions and reducing computational overhead in low-saliency regions, thereby achieving efficient allocation of computational resources. Especially in complex scenes (such as low illumination, complex and cluttered backgrounds, or the coexistence of multiple scale targets), this mechanism significantly improves the model's perception and localization accuracy for small and weak targets, providing high-quality attention priors for subsequent cross-modal fusion and target detection, and overall enhancing the system's robustness and detection performance.

[0049] Saliency detection is performed on visible light images to generate corresponding saliency maps, including:

[0050] Image analysis and processing are performed on visible light images to determine the salience of each pixel position in the visible light image;

[0051] A saliency map is generated based on the degree of saliency. The pixel value of the saliency map represents the probability that the corresponding pixel location belongs to the target-related region.

[0052] Saliency refers to the likelihood that a pixel location in an image is visually prioritized relative to its surrounding area. This metric comprehensively reflects the difference or uniqueness of that location from its neighborhood in terms of color, brightness, texture, edges, or semantics. A higher saliency indicates that the location is more likely to belong to a foreground object or semantically important region in the image, rather than redundant background.

[0053] In some embodiments, by generating three sets of spatial attention weights based on the saliency map, this application achieves refined perception and differentiated processing of different saliency regions of an image: high saliency regions are enhanced with detail representation through the first spatial attention weight, medium saliency regions retain structural integrity through the second spatial attention weight, and low saliency regions capture overall contour and contextual information through the third spatial attention weight. The three weights work synergistically to effectively suppress background interference, enhance key target features, and improve the model's perception ability for small targets and complex scenes without increasing computational burden, thereby significantly improving the accuracy and robustness of subsequent cross-modal fusion and target detection.

[0054] Three sets of spatial attention weights are generated based on the saliency map, including:

[0055] Normalize the saliency map to obtain a normalized saliency map;

[0056] The normalized saliency map is processed using a first preset function to obtain a first spatial attention weight, which is used to enhance the detail representation of highly saliency regions in the image.

[0057] The normalized saliency map is processed using a second preset function to obtain a second spatial attention weight, which is used to extract structural information of equally saliency regions in the image.

[0058] The normalized saliency map is processed using a third preset function to obtain the third spatial attention weight, which is used to capture the overall contour information of low saliency regions.

[0059] In this context, a normalized saliency map refers to an image obtained by mapping the pixel values ​​of the saliency map to a uniform numerical range. Normalization ensures that the saliency responses of different input images are comparable and facilitates subsequent function processing.

[0060] The first presupposition function refers to a nonlinear mapping function used to amplify the response in highly significant regions and suppress less significant regions. Typical implementations include the Softmax function or exponential enhancement functions (such as f(x)=x). p (p>1). Its output emphasizes the most significant part.

[0061] The first spatial attention weight is a spatial mask generated by the first preset function. It has a larger value in the high saliency region and is used to guide subsequent branches (such as dilated convolution) to perform fine modeling of target details (such as edges and textures).

[0062] Highly saliency regions refer to areas in the normalized saliency map where the pixel value is close to 1. They usually correspond to foreground objects or semantically key regions in the image (such as pedestrian heads or vehicle fronts) and have strong visual uniqueness.

[0063] The second predefined function refers to a smoothing or linear transformation function used to preserve the structural continuity of moderately salient regions. Typical implementations include identity mapping or biased linear scaling (such as f(x) = ax + b, where a ≈ 1, b ≈ 0).

[0064] The second spatial attention weight, generated by the second preset function, assigns a moderate response to moderately salient areas (such as the target torso and building body) to extract the overall shape and component connection relationships.

[0065] Medium saliency regions refer to areas in the normalized saliency map where pixel values ​​are in the middle range (e.g., 0.3, 0.7). These regions typically constitute the main structure of the target. Although they are not as prominent as local details, they are crucial to the integrity of the target.

[0066] The third preset function refers to an inverse mapping or low-pass filtering function used to enhance global context awareness in low-saliency regions. Typical implementations include complementary functions (such as f(x) = 1 − x) or low-frequency filtering with threshold truncation.

[0067] The third spatial attention weights are weights generated by a third preset function. They have higher values ​​in low saliency regions and are used to guide lightweight branches (such as depthwise separable convolution) to capture the overall contour and contextual information of the background or the outer periphery of the target.

[0068] Low saliency regions refer to areas in the normalized saliency map where the pixel value is close to 0. These are usually blurred areas at the edges of the background or target (such as the sky or distant roads). Although they do not contain fine details, they can help with scene understanding and boundary localization.

[0069] Understandably, the process of generating three sets of spatial attention weights based on the saliency map can be implemented using a saliency-guided depthwise separable dilated convolutional module (SM-DW-DSC). This module aims to dynamically adjust the multi-scale feature fusion strategy using the saliency map to enhance the network's ability to perceive key regions. Its specific structure is as follows... Figure 1b As shown, the main steps include:

[0070] 1) Input visible light image First, it is processed through two consecutive convolutional layers:

[0071] The first convolutional layer, Conv1, is a standard convolution with a stride of 1 and a kernel size of 3×3, followed by an activation function (ReLU).

[0072] The second convolutional layer, Conv2, is also a 3×3 convolution with a stride of 1, and its output is a saliency map. , representing the spatial saliency weighted plot, is calculated using the following formula:

[0073] ;

[0074] in, , , This represents a convolution operation with a stride of 1 and a kernel size of 3x3. This is the Sigmoid function.

[0075] 2) Using the aforementioned saliency map S, three sets of spatial attention weights are generated through three parallel feature extraction branches. , , Each set of weights is obtained by nonlinear transformation based on the saliency map. For example, different preset functions (such as normalized exponential function (softmax function), logistic function (such as sigmoid function), linear transformation, etc.) or channel-level / spatial-level adjustment strategies are used to enable different branches to focus on the details of high saliency regions, the structure of medium saliency regions, and the overall outline of low saliency regions.

[0076] 3) The three sets of spatial attention weights satisfy the following condition: at each pixel location, α1+α2+α3≥0 and This ensures the stability and interpretability of the weighted fusion process and avoids feature distortion.

[0077] 103. Input the initial visible light feature map and the initial infrared feature map into three parallel feature extraction branches respectively to obtain the corresponding visible light multi-scale feature group and infrared multi-scale feature group. The three feature extraction branches include a depth-separable convolution branch, a dilated convolution branch and a standard convolution branch, and the three feature extraction branches correspond one-to-one with three sets of spatial attention weights.

[0078] In deep learning, especially in computer vision tasks, the feature extraction branch refers to a specific part of the network architecture whose main function is to extract useful feature information from input data (such as images). These branches typically consist of a series of convolutional layers and other operations designed to capture different aspects or levels of features from the input data. Three different feature extraction branches are involved here: depthwise separable convolutional branches, dilated convolutional branches, and standard convolutional branches. Each branch extracts features from the input image using different convolutional techniques to suit specific task requirements.

[0079] The depthwise separable convolution branch employs depthwise separable convolution, which first applies a filter independently to each input channel using depthwise convolution, and then uses pointwise convolution (i.e., 1x1 convolution) to combine these filtered features. This approach significantly reduces computational cost and the number of model parameters while maintaining good feature extraction capabilities.

[0080] The dilated convolution branch utilizes dilated convolution (also known as dilated convolution) to expand the receptive field without increasing computation, helping to capture a wider range of spatial information. It is particularly useful for detecting small targets or fine-grained features.

[0081] The standard convolution branch is a traditional convolution method that uses a sliding window to apply filters to the input to extract features, effectively capturing local spatial information.

[0082] Visible light multi-scale feature sets refer to the set of multi-scale features extracted from visible light images. Through the three different types of feature extraction branches mentioned above, feature representations with different scales and hierarchical structures can be obtained, including but not limited to detailed information such as edges, textures, and colors, as well as higher-level object shapes and scene layout information.

[0083] Infrared multi-scale feature sets refer to the set of multi-scale features extracted from infrared images. Since infrared images carry thermal radiation information, their features primarily reflect the temperature distribution characteristics of objects. By processing through three different feature extraction branches, a series of feature representations from low to high levels can be obtained, which helps in identifying targets and scenes based on heat distribution.

[0084] It is understandable that by inputting the initial visible light feature map and the initial infrared feature map into these three parallel feature extraction branches respectively, and corresponding them one-to-one with the three sets of spatial attention weights, the model's ability to capture and distinguish different types of features can be effectively improved, thereby enhancing the perception and decision-making capabilities of the entire system.

[0085] 104. Based on three sets of spatial attention weights, the feature maps output by each branch in the visible light multi-scale feature group are fused element-wise to obtain enhanced visible light features, and the feature maps output by each branch in the infrared multi-scale feature group are fused element-wise to obtain enhanced infrared features.

[0086] Enhanced visible light features refer to the result obtained by element-wise weighted fusion of the feature maps output from each branch of the visible light multi-scale feature set, applying corresponding spatial attention weights. This process utilizes features at different levels and scales extracted by different convolutional branches (depthseparable convolution, dilated convolution, and standard convolution), assigning different weights based on their saliency, thereby enhancing the representation of important regions in the image while suppressing irrelevant background information. The resulting enhanced visible light features not only contain rich detail information (such as edges and textures) but also enhance the expressive power of the target structure, helping to improve the accuracy of subsequent tasks (such as object detection and classification).

[0087] Enhanced infrared features are obtained by applying appropriate spatial attention weights to the feature maps output from each branch of the infrared multi-scale feature set and then performing element-wise weighted fusion. Since infrared images primarily reflect the thermal radiation information of objects, this enhancement process particularly emphasizes the identification and enhancement of temperature-sensitive areas, while also optimizing the understanding of the overall scene. The enhanced infrared features can more accurately capture target contours and structural information based on heat distribution, which is of great significance for nighttime surveillance or target detection under low-light conditions.

[0088] Specifically: for each feature map from the depthwise separable convolution branch, the dilated convolution branch, and the standard convolution branch Use the corresponding attention weights Perform element-wise multiplication to sum all the weighted feature maps element by element, forming the final enhanced feature map. .

[0089] Enhance visible light characteristics ,in, It is the spatial attention weight corresponding to the i-th feature extraction branch. It is the feature map output by the i-th feature extraction branch after the initial visible light feature map.

[0090] Enhanced infrared signature ,in, It is the spatial attention weight corresponding to the i-th feature extraction branch. It is the feature map output by the i-th feature extraction branch after the initial infrared feature map.

[0091] The formula for element-wise weighted fusion based on three sets of spatial attention weights is:

[0092] ,in, It can be or , The feature map represents the output of the depthwise separable convolution branch. The feature map representing the output of the dilated convolution branch. This represents the feature map output by the standard convolution branch (1x1 convolution). These are the spatial attention weights corresponding to the depthwise separable convolution branches. These are the spatial attention weights corresponding to the dilated convolution branches. These are the spatial attention weights corresponding to the standard convolutional branches.

[0093] It is understandable that, such as Figure 1bAs shown, this weighted fusion mechanism adaptively selects features at different scales by element-wise multiplying spatial attention weights with the feature maps output by each branch. Specifically, in regions with high saliency in the image, the corresponding attention weights are assigned a higher fusion ratio to branches with larger receptive fields or stronger context modeling capabilities (such as dilated convolution branches), thereby enhancing local context awareness. In regions with low saliency, branches with higher computational efficiency (such as depthwise separable convolution or standard convolution branches) are preferred to reduce the computational complexity of the model. Therefore, the entire fusion process achieves "dynamic adjustment of feature extraction strategies based on image content," balancing detection accuracy and inference efficiency.

[0094] 105. Perform feature cross-fusion processing on the enhanced visible light features and enhanced infrared features to obtain a cross-fused image.

[0095] Among them, cross-fusion graphs refer to a new feature representation generated by combining enhanced visible light features with enhanced infrared features through a specific fusion strategy. This fusion not only integrates information from two different modalities (i.e., visible light and infrared), but also makes full use of the advantages of each to compensate for the shortcomings of a single modality under certain conditions or scenarios.

[0096] Specifically, cross-fusion processing can include the following aspects:

[0097] Multi-scale feature fusion: Considering that visible light images and infrared images may provide useful information at different scales, the cross-fusion process may involve feature integration at multiple scales to capture more comprehensive target information;

[0098] Channel-dimensional fusion: This method directly aligns and combines feature maps from the visible and infrared channels along the channel dimension. This helps to highlight common features of targets in different bands and improves the accuracy of tasks such as target detection and recognition.

[0099] Feature concatenation or weighted summation: A simple fusion method is to directly concatenate the two feature maps according to their pixel positions, or to perform a weighted summation based on a certain weight ratio. This method is simple and direct, but may require further optimization of the weight parameters to achieve the best results;

[0100] Deep learning methods utilize deep neural network models (such as convolutional neural networks, CNNs) to automatically learn how to optimally fuse two types of features. For example, a fusion layer can be designed with inputs of enhanced visible light features and enhanced infrared features, and output a cross-fused image. By continuously adjusting the network parameters using a training dataset, the model can effectively extract complementary information from the two modalities.

[0101] Attention mechanism: Introducing an attention mechanism to dynamically adjust the importance of different regions or features, thereby allowing the model to focus more on those parts that are crucial to task solving.

[0102] In some embodiments, by introducing a feature projection mechanism based on channel dimension alignment (note: "channel dimension" in this application specifically refers to the number of channels), and fusing the projected low-level features with high-level enhanced features, the semantic gap between shallow detail information and deep semantic representation is effectively bridged. This mechanism not only preserves high-resolution spatial details (such as edges and textures) in the initial visible light / infrared feature map, but also precisely aligns them with the high-level features enhanced by multi-branch attention in the channel dimension, thereby generating optimized enhanced visible light features and optimized enhanced infrared features. Furthermore, using these optimized features in the cross-fusion stage significantly improves the accuracy and consistency of cross-modal fusion, avoiding feature confusion caused by channel mismatch. Especially in small target detection scenarios, this design can effectively recover the fine structure of the target, enhance boundary localization capabilities, and ultimately enable the generated cross-fusion map to have both high spatial resolution and strong semantic discriminative power, providing a high-quality input representation for subsequent target detection tasks. The method also includes:

[0103] Based on the channel dimension of enhanced visible light features, feature projection processing is performed on the initial visible light feature map to obtain visible light projection features; and based on the channel dimension of enhanced infrared features, feature projection processing is performed on the initial infrared feature map to obtain infrared projection features.

[0104] The visible light projection feature and the enhanced visible light feature are fused to obtain the optimized enhanced visible light feature, and the infrared projection feature and the enhanced infrared feature are fused to obtain the optimized enhanced infrared feature.

[0105] Enhanced visible light features and enhanced infrared features are subjected to feature cross-fusion processing to obtain a cross-fused map, including:

[0106] The optimized enhanced visible light features and optimized enhanced infrared features are subjected to feature cross-fusion processing to obtain a cross-fusion map.

[0107] Here, the channel dimension of the enhanced visible light feature refers to the number of channels included in the enhanced visible light feature, denoted as .

[0108] Visible light projection features refer to the feature map obtained by applying a lightweight projection operation (such as a 1×1 convolution) to the initial visible light feature map, adjusting its channel number to match the number of channels used to enhance the visible light features. This operation ensures that the two are aligned in the channel dimension to support subsequent element-wise fusion.

[0109] The channel dimension of enhanced infrared features refers to the number of channels for enhancing infrared features, denoted as... Normally, = This is to ensure compatibility of dual-modal features.

[0110] Infrared projection features refer to the feature representation obtained by adjusting the number of channels of the initial infrared feature map (e.g., through 1×1 convolution) to align its number of channels with the number of channels of the enhanced infrared features. This feature is used to supplement the low-level detailed information of the infrared modes.

[0111] The optimized enhanced visible light feature refers to the feature map obtained by fusing the visible light projection feature and the enhanced visible light feature element-wise (or convolving them after concatenation) in the same channel dimension. It retains the details of the original feature and integrates the semantic information of the multi-branch enhancement.

[0112] The optimized enhanced infrared feature refers to the feature map obtained by fusing the infrared projection feature and the enhanced infrared feature after aligning the channels, so as to achieve the collaborative enhancement of "shallow details + deep semantics" within the infrared mode.

[0113] Understandably, by introducing skip connection branches and using a 1×1 convolutional layer to perform channel dimensionality reduction on the input initial visible light feature map or initial infrared feature map x, the projected visible light features are made to match the corresponding enhanced visible light features in the channel dimension. To maintain consistency, and to ensure that the projected infrared features are identical to the corresponding enhanced infrared features in the channel dimension. To maintain consistency and thus satisfy the conditions for subsequent element-by-element addition, the mathematical expression for this operation is as follows:

[0114] ,in, This represents the fused output feature map, where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively. yes or .

[0115] This feature projection mechanism not only preserves the low-level details of the original input but also integrates enhanced features from multiple branches, forming an optimized representation rich in multi-scale local context. This optimized feature map... It is fed as input into modules based on selective state-space models (such as Mamba networks) to model long-distance dependencies, thereby improving the accuracy of global modeling while effectively reducing computational complexity and enhancing the expressive power and inference efficiency of the overall network.

[0116] In some embodiments, optimized enhanced visible light features and optimized enhanced infrared features are cross-fused to obtain a cross-fused map with high information complementarity and enhanced semantic representation capabilities. This method not only effectively combines feature information from two different modalities, but also enhances the model's understanding of complex scenes through unique serialization and recombination strategies, especially in detailed and multi-scale object detection tasks.

[0117] The optimized enhanced visible light features and optimized enhanced infrared features are subjected to feature cross-fusion processing to obtain a cross-fusion map, including:

[0118] The optimized enhanced visible light features are expanded in row priority order from top to bottom and from left to right to obtain the first feature sequence;

[0119] The optimized enhanced infrared features are expanded in column priority order from left to right and top to bottom to obtain the second feature sequence;

[0120] The first feature sequence and the second feature sequence are concatenated position by position along the feature dimension to form a combined feature sequence;

[0121] The combined feature sequence is reshaped into a two-dimensional tensor to obtain the cross-fusion graph.

[0122] The first feature sequence is a one-dimensional array obtained by expanding the optimized enhanced visible light features in row-major order from top to bottom and from left to right.

[0123] For example, suppose there is a 3x3 enhanced visible light feature with the following value: After being expanded in row priority order, the first feature sequence is: [V11,V12,V13,V21,V22,V23,V31,V32,V33].

[0124] The second feature sequence is a one-dimensional array obtained by expanding the optimized enhanced infrared features in column-priority order from left to right and from top to bottom.

[0125] For example, suppose there is a 3x3 enhanced infrared feature with the following value: After expanding according to column priority, the second feature sequence is: [I11,I21,I31,I12,I22,I32,I13,I23,I33].

[0126] A combined feature sequence is a new sequence formed by concatenating the first feature sequence and the second feature sequence position by position along the feature dimension.

[0127] For example, the combined feature sequence could be:

[0128] [V11,I11,V12,I21,V13,I31,V21,I12,V22,I22,V23,I32,V31,I13,V32,I23,V33,I33].

[0129] In some embodiments, by further processing the combined feature sequences and utilizing selective state-space models and deep convolution operations, long-range dependency information and local spatial context information can be effectively extracted. This dual processing method not only enhances the richness and diversity of feature representations but also significantly improves the model's ability to understand complex scenes, particularly in tasks such as object detection and classification. The combined feature sequences are then reshaped into two-dimensional tensors to obtain a cross-fusion graph, including:

[0130] Long-range dependency modeling is performed on the combined feature sequence input selective state-space model to obtain the selection features;

[0131] By performing a depthwise convolution operation on the combined feature sequence, local spatial context information is extracted to obtain local features;

[0132] The selected features and local features are added together and reshaped into a two-dimensional tensor to obtain the cross-fusion graph.

[0133] Among them, the selective state-space model is a high-level model for processing sequential data, capable of effectively capturing long-range dependencies in the sequence. It selectively models important information by focusing on specific parts of the sequence to construct state transition equations. In this application, this model is used to process combined feature sequences to extract long-range dependency information.

[0134] Long-range dependency modeling refers to identifying and utilizing correlations between distant elements in a sequence during sequence data processing. This is crucial for understanding complex patterns and structures, especially in time series analysis or image processing tasks. Selective state-space models can effectively capture these long-range dependencies, enhancing the model's ability to understand global information.

[0135] Feature selection is a feature representation obtained by processing a combination of feature sequences using a selective state-space model. This feature contains important long-range dependency information from the sequence, helping to enhance the model's understanding of the overall structure and patterns, and performing particularly well when dealing with large-scale spatial associations.

[0136] Local features are obtained by performing depthwise convolution operations on combined feature sequences. Depthwise convolution operations can extract local spatial context information at multiple levels, thereby generating feature representations containing rich details. These features help enhance the model's understanding of local structures and improve the accuracy of object detection and classification.

[0137] The cross-fusion graph is the result of adding selected features and local features and reshaping them into a two-dimensional tensor. This process combines global long-range dependency information and local spatial context information to generate a comprehensive feature representation. This graph retains the key information of the original visible light and infrared features, and enhances the diversity and robustness of the features through a fusion strategy, providing a high-quality input representation for subsequent tasks such as object detection and classification.

[0138] Specifically, when performing feature cross-fusion processing on the optimized enhanced visible light features and the optimized enhanced infrared features, this application employs a lightweight cross-modal interaction mechanism based on a selective state-space model to avoid problems such as modal confusion, sequence redundancy, and decreased small target perception capability caused by directly concatting bimodal features. Figure 1c As shown, this mechanism mainly includes the following steps:

[0139] 1) Channel alignment and input preparation: Enhanced visible light features and enhanced infrared features, which have been enhanced and optimized by the SM-DW-DSC module, are used as the current inputs. To adapt to subsequent sequence modeling, the channel dimension can be adjusted by 1×1 convolution (if needed) to ensure that the number of channels is consistent.

[0140] 2) Cross-sequence construction: The optimized enhanced visible light features are expanded in row-priority order from top to bottom and left to right to obtain the first feature sequence. The optimized enhanced infrared features are expanded in column-priority order from left to right and top to bottom to obtain the second feature sequence. The first feature sequence and the second feature sequence are concatenated position by position along the feature dimension to form a combined feature sequence. This allows the information of the two modalities to be interleaved in the sequence, providing a structured cross-modal context for subsequent modeling.

[0141] 3) Joint modeling of combined feature sequences: For combined feature sequences, the following two parallel operations are performed simultaneously:

[0142] a) Long-range dependency modeling: Input the combined feature sequence into a selective state-space model (such as a Mamba network), and adaptively model the cross-modal global dependency relationship through dynamically generated state-space parameters to obtain the selected features;

[0143] The Mamba network is the core computational unit built on the Selective State Space Model (SSM), and its basic structure is as follows: Figure 1d As shown, the main steps are:

[0144] a-1. Constructed combined feature sequences After passing through a linear projection layer, it is divided into two branches to generate the feature transformation X and dynamic parameters. The linear projection layer uses a learnable weight matrix to perform a linear transformation on the input sequence and splits it into two feature representations, expressed as follows:

[0145]

[0146] Where d represents the input feature. Dimensional information, The projected weight matrix is ​​a learnable matrix. .

[0147] a-2. Dynamic system parameter generation and state-space modeling: Utilizing a lightweight parametric projection network based on the dynamic feature representation. The four system parameters required for adaptively generating the selective state-space model are determined. The parameter projection network typically consists of two cascaded linear transformation layers and intermediate activation functions, whose outputs are the discretization step size Δ, the state transition matrix A, the input mapping matrix B, and the output mapping matrix C. This process can be represented as:

[0148] ;

[0149] in, These represent the discretization time step, state evolution matrix, input action matrix, and output observation matrix, respectively. Subsequently, the dynamically generated system parameters and the transformed feature X are fed into the discretization module of the state-space model, and the global state update of the sequence is completed through an efficient parallel scanning algorithm, thereby achieving adaptive modeling of cross-modal long-range dependencies.

[0150] b) Local Context Extraction: A deep convolution operation (e.g., using depthwise separable convolution) is performed on the combined feature sequence to capture contextual information within the local spatial neighborhood, obtaining local features. In some embodiments, this deep convolution operation can be embedded in a residual connection structure, weighting and fusing the convolution output with the original input. This is then followed by batch normalization and a non-linear activation function (such as ReLU) to achieve feature normalization and non-linear enhancement, ultimately outputting the fused features. .

[0151] Subsequently, the selected features and local features are added element-wise, and the result is reshaped into a two-dimensional tensor to obtain the final cross-fusion graph.

[0152] The above process achieves the synergistic fusion of global long-range dependencies and local detailed information, which not only retains Mamba's advantages in modeling efficiency, but also supplements its ability to perceive local structures through deep convolution, thereby effectively improving the detection performance of small targets in complex scenes.

[0153] 106. Perform target detection based on the cross-fusion graph to obtain the target detection results.

[0154] The object detection result refers to the set of information about one or more objects of interest in the input image (in this case, the scene corresponding to the cross-fusion map) output by the system after object detection processing. This result typically includes the following:

[0155] Target category (Class Label): such as "pedestrian", "vehicle", "bicycle", "traffic sign", "animal", etc.;

[0156] Bounding Box: Represented by a rectangle, usually described by coordinates or center point plus width and height;

[0157] Confidence Score: Represents the confidence level of the model in the detection result, and its value range is generally [0,1].

[0158] In this application, the target detection result is generated based on a cross-fusion map (i.e., an enhanced feature map that combines the advantages of visible light and infrared modes), thus exhibiting stronger robustness and being particularly suitable for detection tasks in complex environments (such as nighttime, foggy weather, strong light changes, etc.).

[0159] In some embodiments, by separating visible and infrared sub-features from the cross-fusion map and generating a mode-aware gated map based on global average pooling, dynamic, pixel-level weighted fusion of dual-modal information is achieved. This mechanism can adaptively adjust the fusion weights of visible and infrared features at each location—relying more on visible light details in textured, well-lit areas, and enhancing infrared response in low-light or thermally contrasting areas. The resulting comprehensive feature map combines the high-resolution semantic information of visible light with the robust thermal sensing capability of infrared, effectively overcoming the limitations of single-modality detection in complex scenes (such as nighttime, fog, strong reflections, etc.), and significantly improving the accuracy, robustness, and environmental adaptability of target detection. Target detection is performed based on the cross-fusion map to obtain target detection results, including:

[0160] Visible photon features associated with the visible light mode and infrared sub-features associated with the infrared mode were separated from the cross-fusion graph;

[0161] Global average pooling is calculated for visible light sub-features and infrared sub-features respectively to obtain global feature vectors for visible light and infrared light.

[0162] The visible light global feature vector and the infrared global feature vector are added element by element and normalized to generate a gating map. The value of each pixel in the gating map is used to represent the fusion weight of the visible light modal information relative to the infrared modal information at the corresponding pixel position.

[0163] Based on the gated graph, an element-wise weighted fusion operation is performed on visible photonic features and infrared sub-features to obtain a comprehensive feature map;

[0164] Target detection is performed based on the comprehensive feature map to obtain the target detection results.

[0165] Visible light mode refers to imaging techniques that utilize the portion of the electromagnetic spectrum perceptible to the human eye (approximately in the wavelength range of 400 to 700 nanometers). Images are generated by capturing light reflected or emitted by an object.

[0166] Visible photon features are a set of characteristics related to visible light modes separated from cross-fused images. These characteristics include information such as color, texture, and shape, and are crucial for identifying and distinguishing different objects.

[0167] Infrared mode imaging uses the infrared portion of the electromagnetic spectrum (typically with wavelengths greater than 700 nanometers) for imaging, primarily for detecting temperature differences because it can capture the thermal radiation emitted by objects.

[0168] Infrared sub-features are a set of infrared mode-related characteristics separated from cross-fused maps, mainly reflecting the target's thermal properties or other information that can be reflected by thermal radiation.

[0169] The visible light global feature vector is a one-dimensional vector obtained by applying a global average pooling operation to the visible photon features, which summarizes the main visual content of the entire visible light image.

[0170] The infrared global feature vector is obtained by performing a global average pooling operation on the infrared sub-features, and represents the overall thermal distribution of the infrared image.

[0171] A gating map is a two-dimensional matrix generated by element-wise addition of the visible light global feature vector and the infrared global feature vector, followed by normalization. The value at each pixel position represents the fusion weight of the visible light information relative to the infrared information at that position.

[0172] The integrated feature map is the result of performing an element-wise weighted fusion operation on visible photonic features and infrared sub-features based on the gated map. It integrates the advantages of the two modes and is used for subsequent target detection.

[0173] Understandably, this embodiment achieves adaptive fusion of multimodal features by explicitly decoupling the semantic components of visible light and infrared modalities within the cross-fusion graph and introducing a lightweight, globally aware dynamic gating mechanism. This method avoids the modal interference or information redundancy problems caused by traditional simple splicing or fixed-weighted fusion. Guided by the global context, the gating graph intelligently allocates the contribution ratio of visible light and infrared features at the pixel level: in well-lit and textured areas, visible light is given a higher weight to preserve details; in areas with weak light, smoke, or significant target thermal radiation, the dominance of infrared features is increased, thereby enhancing the robustness of target perception. The final generated comprehensive feature map retains the spatial fine structure of the visible light modality and incorporates the environmental invariance advantage of the infrared modality, significantly improving the model's target detection performance in complex and variable real-world scenarios (such as nighttime surveillance, rainy and foggy weather, strong glare, etc.) while maintaining low computational overhead, demonstrating good engineering practicality and generalization ability.

[0174] Specifically, such as Figure 1d As shown, the dynamic fusion gating mechanism is based on the cross-fusion graph processed by shared Mamba blocks. Adaptive gating modeling is employed to achieve pixel-level dynamic fusion of cross-modal features.

[0175] First, It is split along the channel dimension into visible photon features and infrared sub-features, i.e. , Indicates the characteristics of visible photons, Indicates infrared sub-features.

[0176] Next, and The graph is concatenated along the channel dimension, and a spatial attention map is generated by passing it through a 1×1 convolutional layer. Then, it is activated by the sigmoid function to obtain the gated graph. Its mathematical expression is:

[0177] ;

[0178] Where σ is the Sigmoid function, which ensures that each pixel value of G is in the (0,1) interval, representing the fusion weight of visible light modal information relative to infrared modal information at the corresponding position.

[0179] Finally, the visible light features were analyzed using this gated graph G. and infrared features Perform element-wise weighted fusion to obtain the comprehensive feature map. The formula is as follows:

[0180] .

[0181] This process implements an adaptive fusion strategy that dynamically adjusts the contribution of each modality based on the input content, enhancing the model's ability to collaboratively perceive multimodal information in complex scenarios.

[0182] In some embodiments, by extracting multi-scale contextual features from the cross-fusion graph and dynamically fusing it with a high-level feature map upsampled by bilinear interpolation, and then generating a target feature map through feature recombination, the model's semantic understanding and spatial localization capabilities for multi-scale targets are effectively enhanced, significantly improving the accuracy and robustness of target detection in complex scenes. Target detection is performed based on the cross-fusion graph to obtain target detection results, including:

[0183] Multi-scale contextual feature extraction is performed on the cross-fusion graph to obtain a context-aware feature map;

[0184] The context-aware feature map is fused with a high-level feature map upsampled by bilinear interpolation to generate a dynamically upsampled feature map.

[0185] The target feature map is obtained by recombining the dynamically upsampled feature map.

[0186] The target feature map is processed for target detection to obtain the target detection result.

[0187] Context-aware feature maps refer to feature maps generated during the multi-scale context feature extraction step. They not only contain basic visual information of the image (such as color and texture), but also incorporate semantic information of the surrounding environment, helping the model understand the relationship between the target and its scene, thereby improving detection performance.

[0188] High-level feature maps refer to feature maps generated in the deep layers of neural networks. These feature maps usually contain rich semantic information and abstract concepts, but may lack fine spatial location information.

[0189] Target feature maps combine rich contextual information from context-aware feature maps with high-level semantic information from high-level feature maps, and preserve or enhance spatial details through appropriate processing, ensuring accurate identification and localization of target objects.

[0190] In some embodiments, multi-scale contextual feature extraction is performed on the cross-fusion graph using dilated convolutions with different dilation rates, and the resulting features are concatenated along the channel dimension to form a context-aware feature map. This effectively expands the receptive field, fuses multi-scale spatial contextual information, and significantly enhances the model's semantic understanding and localization capabilities for targets in complex scenes. Multi-scale contextual feature extraction of the cross-fusion graph to obtain the context-aware feature map includes:

[0191] Three sets of dilated convolutional layers with different dilation rates were used to extract features from the cross-fusion map, resulting in three context feature maps at different scales.

[0192] Three context feature maps at different scales are concatenated along the channel dimension to form a context-aware feature map.

[0193] The dilatation ratio is a parameter used in convolution operations that determines the size of the intervals that the convolution kernel skips when processing input data. By increasing the dilatation ratio, the receptive field can be expanded without increasing computational cost, meaning the convolution kernel can cover a wider area and capture information at more scales.

[0194] Contextual feature maps are feature representations that extract rich scene information and object relationships by applying specific convolutional operations (such as dilated convolutions with different dilation rates mentioned above) to an input image or feature map. These feature maps help the model better understand the relationships between objects in the image and their background environment.

[0195] In some embodiments, multi-scale contextual feature extraction is performed on the cross-fusion graph to obtain a context-aware feature map, including:

[0196] Multi-scale contextual feature extraction is performed on the comprehensive feature map to obtain a context-aware feature map.

[0197] Specifically, traditional feature extraction methods, when dealing with small targets, directly utilize feature pyramids and employ interpolation algorithms to fuse high- and low-level features. This method struggles to accurately capture the correspondence between high- and low-level features, resulting in poor feature fusion performance. To address this issue, this application proposes a Context-Aware Upsampling (CAU) module. This module is implemented through dynamic upsampling. First, multi-scale contextual features are extracted, then a lightweight attention mechanism is introduced, and finally, the results are perceptually fused with the bilinear interpolation upsampling results, enabling a reasonable match between high- and low-level features. The specific structure is as follows: Figure 1e As shown, the steps are as follows:

[0198] 1) Multi-scale contextual feature extraction, such as Figure 1f As shown, specifically, three sets of dilated convolutional layers with different dilation rates are used to extract features from the input cross-fusion map, resulting in three context feature maps at different scales. These three context feature maps at different scales are then concatenated along the channel dimension to form a comprehensive context-aware feature map. The calculation formula is as follows:

[0199] ;

[0200] in, This represents the input from the previous layer. Indicates the application of different void ratios Convolution operation, =1, 2, 3 represent different hole ratio settings. This step ensures that the model can capture contextual information within different ranges, improving feature representation capabilities. The feature map size output after different hole ratios is calculated as follows:

[0201]

[0202] in, Indicates the size of the input feature map. Indicates the number of fillers. Indicates the void ratio, Indicates the kernel size. Indicates step size, This indicates rounding down to the nearest integer.

[0203] 2) Utilize context-aware feature maps again Feature map of bilinear interpolation A fusion operation is performed, introducing 1x1 convolutions to reconstruct the feature maps and perform dynamic upsampling. A lightweight attention mechanism module is then introduced, with spatial attention calculated as follows:

[0204]

[0205] in, This indicates that the feature maps are concatenated according to their dimensions. This represents a 1x1 convolution. It is input. This represents the sigmoid function.

[0206] 3) The context-aware feature map is fused with the high-level feature map upsampled by bilinear interpolation to generate a dynamically upsampled feature map, thus obtaining the final output target feature map. The calculation formula is as follows:

[0207] ;

[0208] Among them, the high-level blocks obtained by bilinear interpolation and context-sampled blocks The features are combined into a target feature map, and then 1x1 convolution is used to achieve feature recombination and complete dynamic upsampling, so that the network can learn the dependency relationship between high and low layers.

[0209] It is understandable that the overall network structure is as follows: Figure 1fAs shown, the network has two input branches. First, it passes through a shared backbone network + SM-DW-DSS for feature extraction and enhancement of intra-image features, allowing the network to focus on more salient features in the image, providing a foundation for subsequent cross-modal fusion and achieving single-modal focusing. The shared backbone network consists of two stacked 3x3 convolutions. Next, a Mamba network is used for visible and infrared image fusion, forming complementary information so that the output feature map is no longer two separate modalities but a unified cross-modal feature. Finally, a feature pyramid + CAU is used for high- and low-level multi-scale feature fusion, ensuring feature extraction for small targets. Then, the enhanced visible light and infrared features are fed into the multi-modal fusion module (i.e., the aforementioned cross-modal interaction mechanism based on the selective state-space model). Through cross-scan sequence construction and shared Mamba block processing, deep fusion of dual-modal information is achieved, generating a unified cross-modal feature representation, rather than a simple two-channel stitching result. Finally, the cross-modal feature is incorporated into the feature pyramid structure and combined with the Context-Aware Upsampling (CAU) module to perform multi-scale dynamic fusion of high- and low-level features. The CAU module extracts contextual information through multi-scale dilated convolution and adaptively weights and fuses it with the bilinear interpolation results, effectively mitigating the semantic loss problem of small targets during the upsampling process, thereby significantly improving the detection capability of small-scale targets.

[0210] In summary, this application achieves efficient complementary fusion of visible light and infrared features through saliency-guided multi-branch attention enhancement, cross-modal sequence cross-fusion, and context-aware upsampling, significantly improving the detection accuracy and system robustness of small targets in complex scenes.

[0211] For example, in this embodiment, a target detection device specifically integrated into an electronic device will be used as an example to describe the method of this application embodiment in detail.

[0212] For example, such as Figure 2 As shown, the target detection device may include an image processing unit 201, a weight generation unit 202, a feature extraction unit 203, a weighted fusion unit 204, a cross-fusion unit 205, and a target detection unit 206, as follows:

[0213] (a) Image processing unit 201.

[0214] The image processing unit 201 is used to acquire a visible light image and its corresponding infrared image in the same scene, and to extract features from the visible light image and its corresponding infrared image respectively to obtain an initial visible light feature map and an initial infrared feature map.

[0215] (ii) Weight generation unit 202.

[0216] The weight generation unit 202 is used to perform saliency detection on the visible light image, generate the corresponding saliency map, and generate three sets of spatial attention weights based on the saliency map. The three sets of spatial attention weights are non-negative at each pixel position and their sum is 1.

[0217] (III) Feature extraction unit 203.

[0218] The feature extraction unit 203 is used to input the initial visible light feature map and the initial infrared feature map into three parallel feature extraction branches to obtain the corresponding visible light multi-scale feature group and infrared multi-scale feature group. The three feature extraction branches include a depth-separable convolution branch, a dilated convolution branch and a standard convolution branch, and the three feature extraction branches correspond one-to-one with three sets of spatial attention weights.

[0219] (iv) Weighted fusion unit 204.

[0220] The weighted fusion unit 204 is used to perform element-wise weighted fusion of the feature maps output by each branch in the visible light multi-scale feature group based on three sets of spatial attention weights to obtain enhanced visible light features, and to perform element-wise weighted fusion of the feature maps output by each branch in the infrared multi-scale feature group to obtain enhanced infrared features.

[0221] (V) Cross-integration Unit 205

[0222] The cross-fusion unit 205 is used to perform feature cross-fusion processing on enhanced visible light features and enhanced infrared features to obtain a cross-fusion map.

[0223] (vi) Target Detection Unit 206

[0224] The target detection unit 206 is used to perform target detection based on the cross-fusion graph and obtain the target detection result.

[0225] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0226] Therefore, the embodiments of this application can improve the accuracy of target detection.

[0227] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0228] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps in any of the target detection methods provided in embodiments of this application.

[0229] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0230] Since the instructions stored in the storage medium can execute the steps of any of the target detection methods provided in the embodiments of this application, the beneficial effects that any of the target detection methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0231] According to one aspect of this application, a computer program product or computer program is provided, comprising a computer program / instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the method provided in the target detection aspect of the above embodiments.

[0232] The above provides a detailed description of a target detection method and apparatus provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A target detection method, characterized in that, The method includes: Acquire visible light images and their corresponding infrared images in the same scene, and extract features from the visible light images and their corresponding infrared images respectively to obtain initial visible light feature maps and initial infrared feature maps; The visible light image is subjected to saliency detection to generate a corresponding saliency map, and three sets of spatial attention weights are generated based on the saliency map. The three sets of spatial attention weights are non-negative at each pixel position and their sum is 1. The generation of three sets of spatial attention weights based on the saliency map includes: The saliency map is normalized to obtain a normalized saliency map; The normalized saliency map is processed using a first preset function to obtain a first spatial attention weight. The first spatial attention weight is used to enhance the detail representation of highly saliency regions in the image. The first preset function refers to a nonlinear mapping function. The normalized saliency map is processed using a second preset function to obtain a second spatial attention weight. The second spatial attention weight is used to extract structural information of equally saliency regions in the image. The second preset function refers to a linear transformation function. The normalized saliency map is processed using a third preset function to obtain a third spatial attention weight. The third spatial attention weight is used to capture the overall contour information of low saliency regions. The third preset function refers to a low-pass filter function. The initial visible light feature map and the initial infrared feature map are respectively input into three parallel feature extraction branches to obtain the corresponding visible light multi-scale feature group and infrared multi-scale feature group. The three feature extraction branches include a depth-separable convolution branch, a dilated convolution branch and a standard convolution branch, and the three feature extraction branches correspond one-to-one with the three sets of spatial attention weights. Based on the three sets of spatial attention weights, the feature maps output by each branch in the visible light multi-scale feature group are fused element-wise to obtain enhanced visible light features, and the feature maps output by each branch in the infrared multi-scale feature group are fused element-wise to obtain enhanced infrared features. The enhanced visible light features and the enhanced infrared features are subjected to feature cross-fusion processing to obtain a cross-fusion map; Target detection is performed based on the cross-fusion graph to obtain the target detection result.

2. The method as described in claim 1, characterized in that, Also includes: Based on the channel dimension of the enhanced visible light features, the initial visible light feature map is subjected to feature projection processing to obtain visible light projection features; and based on the channel dimension of the enhanced infrared features, the initial infrared feature map is subjected to feature projection processing to obtain infrared projection features. The visible light projection feature and the enhanced visible light feature are fused to obtain the optimized enhanced visible light feature, and the infrared projection feature and the enhanced infrared feature are fused to obtain the optimized enhanced infrared feature. The step of performing feature cross-fusion processing on the enhanced visible light features and the enhanced infrared features to obtain a cross-fusion map includes: The optimized enhanced visible light features and the optimized enhanced infrared features are subjected to feature cross-fusion processing to obtain a cross-fusion map.

3. The method as described in claim 2, characterized in that, The step of performing feature cross-fusion processing on the optimized enhanced visible light features and the optimized enhanced infrared features to obtain a cross-fusion map includes: The optimized enhanced visible light features are expanded in row priority order from top to bottom and from left to right to obtain the first feature sequence; The optimized enhanced infrared features are expanded in column priority order from left to right and top to bottom to obtain the second feature sequence; The first feature sequence and the second feature sequence are concatenated position by position along the feature dimension to form a combined feature sequence; The combined feature sequence is reshaped into a two-dimensional tensor to obtain a cross-fusion graph.

4. The method as described in claim 3, characterized in that, The step of reshaping the combined feature sequence into a two-dimensional tensor to obtain a cross-fusion graph includes: The combined feature sequence is input into a selective state-space model to perform long-range dependency modeling, thereby obtaining the selection features; Perform a depthwise convolution operation on the combined feature sequence to extract local spatial context information and obtain local features; The selected features are added to the local features and then reshaped into a two-dimensional tensor to obtain the cross-fusion graph.

5. The method as described in claim 1, characterized in that, The step of performing target detection based on the cross-fusion graph to obtain target detection results includes: Visible photon features associated with the visible light mode and infrared sub-features associated with the infrared mode are separated from the cross-fusion graph; Global average pooling is calculated for the visible light sub-features and the infrared sub-features respectively to obtain the visible light global feature vector and the infrared global feature vector; The visible light global feature vector and the infrared global feature vector are added element by element and normalized to generate a gating graph. The value of each pixel position in the gating graph is used to characterize the fusion weight of the visible light modal information relative to the infrared modal information at the corresponding pixel position. Based on the gated graph, an element-wise weighted fusion operation is performed on the visible photon features and the infrared sub-features to obtain a comprehensive feature map; Target detection is performed based on the comprehensive feature map to obtain the target detection result.

6. The method as described in claim 1, characterized in that, The step of performing saliency detection on the visible light image and generating a corresponding saliency map includes: The visible light image is subjected to image analysis processing to determine the salience of each pixel position in the visible light image; A saliency map is generated based on the saliency level, and the pixel values ​​of the saliency map represent the probability that the corresponding pixel location belongs to the target-related region.

7. The method as described in claim 1, characterized in that, The step of performing target detection based on the cross-fusion graph to obtain target detection results includes: Multi-scale contextual feature extraction is performed on the cross-fusion graph to obtain a context-aware feature map; The context-aware feature map is fused with a high-level feature map upsampled by bilinear interpolation to generate a dynamically upsampled feature map. The dynamically upsampled feature map is reconstructed to obtain the target feature map; The target feature map is subjected to target detection processing to obtain the target detection result.

8. The method as described in claim 7, characterized in that, The step of extracting multi-scale contextual features from the cross-fusion graph to obtain a context-aware feature map includes: Three sets of dilated convolutional layers with different dilation rates were used to extract features from the cross-fusion graph, resulting in three context feature maps at different scales. The three context feature maps at different scales are concatenated along the channel dimension to form a context-aware feature map.

9. A target detection device, characterized in that, The device includes: The image processing unit is used to acquire the visible light image and its corresponding infrared image in the same scene, and to extract features from the visible light image and its corresponding infrared image respectively to obtain the initial visible light feature map and the initial infrared feature map. The weight generation unit is used to perform saliency detection on the visible light image, generate a corresponding saliency map, and generate three sets of spatial attention weights based on the saliency map. The three sets of spatial attention weights are non-negative at each pixel position and their sum is 1. The generation of three sets of spatial attention weights based on the saliency map includes: The saliency map is normalized to obtain a normalized saliency map; The normalized saliency map is processed using a first preset function to obtain a first spatial attention weight. The first spatial attention weight is used to enhance the detail representation of highly saliency regions in the image. The first preset function refers to a nonlinear mapping function. The normalized saliency map is processed using a second preset function to obtain a second spatial attention weight. The second spatial attention weight is used to extract structural information of equally saliency regions in the image. The second preset function refers to a linear transformation function. The normalized saliency map is processed using a third preset function to obtain a third spatial attention weight. The third spatial attention weight is used to capture the overall contour information of low saliency regions. The third preset function refers to a low-pass filter function. The feature extraction unit is used to input the initial visible light feature map and the initial infrared feature map into three parallel feature extraction branches respectively to obtain the corresponding visible light multi-scale feature group and infrared multi-scale feature group. The three feature extraction branches include a depth-separable convolution branch, a dilated convolution branch and a standard convolution branch, and the three feature extraction branches correspond one-to-one with the three sets of spatial attention weights. The weighted fusion unit is used to perform element-wise weighted fusion of the feature maps output by each branch in the visible light multi-scale feature group based on the three sets of spatial attention weights to obtain enhanced visible light features, and to perform element-wise weighted fusion of the feature maps output by each branch in the infrared multi-scale feature group to obtain enhanced infrared features. A cross-fusion unit is used to perform feature cross-fusion processing on the enhanced visible light feature and the enhanced infrared feature to obtain a cross-fusion map; The target detection unit is used to perform target detection based on the cross-fusion graph to obtain the target detection result.

Images