A multi-band image end-to-end multi-task intelligent detection method

Abstract

The present application relates to the technical field of machine vision, and discloses a kind of multi-band image end-to-end multi-task intelligent detection method, by obtaining multi-band target image, the feature fusion is carried out to multi-band target image, three-layer feature extraction is carried out to the image after fusion, can more fully capture image information, reduce the low infrared feature influence caused by cloud layer radiation suppression or submersion target, while reducing the false alarm caused by the target-like radiation source in the edge of cloud layer and ground object background, reduce the false detection and missed detection of weak and small target identification tracking, combined with target detection head, key point detection head and situation classification detection head, each detection head independent branch decoupling task, improve the generalization ability and detection efficiency of model.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically a multi-band image end-to-end multi-task intelligent detection method. Background Technology

[0002] During the long-range search and interception phase or the initial tracking phase of the infrared seeker, an upward or downward attack posture may be formed. The main challenges in resisting complex natural / artificial background interference include the following:

[0003] At long distances, targets often exist as point targets within the sensitivity threshold of imaging systems, lacking scale information and exhibiting unstable energy levels due to airflow and target attitude. Since cloud radiation intensity may be higher than that of the target, it can suppress or even submerge the target, resulting in low infrared signatures. Simultaneously, objects with energy at cloud edges or in the background, resembling targets and possessing stable infrared gradients, are easily misidentified as targets, leading to numerous false alarms and causing false detections or missed detections in the identification and tracking of weak targets.

[0004] Against a background of strong infrared radiation, such as bright clouds in the sky, objects on the ground illuminated by sunlight, or bright bands on the sea surface, when there is a certain distance between the target and the observation point, the intensity of its own infrared radiation will be lower than that of these backgrounds. When the target itself is blocked, it will form a characteristic inversion pattern such as "dark target" in the field of view.

[0005] In different scenarios such as sky background, ground background, and sea background, and under different sky backgrounds such as cloudless, thin clouds, cloud clusters, stratus clouds, and bright clouds, the detection algorithm for small, weak targets in specific environments cannot adapt to the huge changes in the background environment, which poses a greater challenge to the stability of detection and tracking. Summary of the Invention

[0006] This invention provides a multi-band image end-to-end multi-task intelligent detection method, which solves the problem that objects with a certain energy in the cloud edge or ground background are similar to targets and have a certain infrared gradient, making them easy to be misidentified as targets, easily generating a large number of false alarms, and causing false detection or missed detection in the identification and tracking of weak targets.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A multi-band image end-to-end multi-task intelligent detection method includes:

[0009] Acquire multi-band target images;

[0010] Feature extraction is performed on multi-band target images by band, and the individual features of the multi-band target images are fused to obtain a fused feature map;

[0011] Feature extraction is performed on the fused feature map to obtain image features;

[0012] Image features are input into the intelligent detection model, and intelligent detection results are output.

[0013] The feature extraction of the fused feature map includes three layers of feature extraction, which are designed based on YOLOv8s-P2. The three layers of feature extraction are arranged from largest to smallest according to the size of the extracted feature map. The intelligent detection model includes a target detection head, a key point detection head, and a situation classification detection head.

[0014] A further improvement of the present invention is that the target detection head adopts a decoupled head structure.

[0015] A further improvement of the present invention is that the decoupled head structure includes two parallel branches, which extract category features and location features from image features respectively, and then each uses a 1×1 convolution layer to complete the classification and regression tasks.

[0016] A further improvement of this invention is that, specifically, the classification and regression tasks are completed by using focus loss to improve classification progress and intersection-union loss to improve regression accuracy.

[0017] A further improvement of the present invention is that the key point detection head is based on a convolutional layer design. The image features include multiple candidate regions. The key point detection head predicts a 15-dimensional output for each candidate region of the image features. The key point detection head first compresses the feature dimension through 3×3 convolution, and then generates key point coordinates through 1×1 convolution.

[0018] A further improvement of this invention is that the key point coordinates include the key point's x-coordinate, y-coordinate, and visibility.

[0019] A further improvement of the present invention is that the situation classification and detection head classifies the situation of image features and identifies the environmental background information in the image features through a fully connected layer. After classification and identification, it outputs the probability distribution result and uses cross-entropy loss to optimize the probability distribution result.

[0020] A further improvement of the present invention is that the target image includes the target aircraft, the key points detected by the key point detection head include the nose, tail, right wing, left wing and belly of the aircraft, and the situation classification detection head classifies the situation as lateral, rear-facing, downward-looking, head-on and upward-looking.

[0021] A further improvement of the present invention is that the first layer of feature extraction in the three-layer feature extraction integrates a dynamic head module, which is used to adaptively optimize feature extraction according to the complexity of the input image and the target distribution.

[0022] A further improvement of this invention lies in the fact that fusing the features of the multi-band target images specifically involves:

[0023] Cross-attention fusion of multi-band target images involves first extracting features from the multi-band target images, then calculating the similarity of features across each band, aligning the features of each band, and then aggregating and enhancing them. Finally, a multi-band fused feature map is obtained through a fusion function.

[0024] Compared with existing technologies, the present invention has the following beneficial effects: The present invention provides a multi-band image end-to-end multi-task intelligent detection method. By acquiring multi-band target images, performing feature fusion on the multi-band target images, and performing three-layer feature extraction on the fused images, it can capture image information more fully, reduce the influence of low infrared features caused by cloud radiation suppression or flooding of targets, and at the same time reduce false alarms caused by target-like radiation sources in cloud edges and ground backgrounds, and reduce false detections and missed detections in the identification and tracking of weak targets. By combining a target detection head, a key point detection head, and a situation classification detection head, each detection head has independent branches and decoupled tasks, which improves the generalization ability and detection efficiency of the detection model. Attached Figure Description

[0025] Figure 1 This is a flowchart of a multi-band image end-to-end multi-task intelligent detection method according to the present invention;

[0026] Figure 2 This is a diagram of the target recognition network structure according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the FCA module in an embodiment of the present invention;

[0028] Figure 4 This is the fused feature map output by an embodiment of the present invention;

[0029] Figure 5 This is a diagram of the three-layer feature extraction network architecture of an embodiment of the present invention;

[0030] Figure 6 This is a structural diagram of the multi-detector head according to an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0034] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, a multi-band image end-to-end multi-task intelligent detection method is characterized by comprising:

[0036] S1: Acquire multi-band target images;

[0037] S2: Extract features from the multi-band target image by band, and fuse the features of the multi-band target image to obtain a fused feature map;

[0038] S3: Extract features from the fused feature map to obtain image features;

[0039] S4: Input the image features into the intelligent detection model and output the intelligent detection results.

[0040] Feature extraction from the fused feature map includes three layers of feature extraction, which are designed based on YOLOv8s-P2. The three layers of feature extraction are arranged from largest to smallest according to the size of the extracted feature map. The intelligent detection model includes a target detection head, a key point detection head, and a situation classification detection head.

[0041] By acquiring multi-band target images, feature extraction is performed on the multi-band target images according to each band, and the individual features of the multi-band target images are fused to obtain a fused feature map;

[0042] like Figure 2As shown, taking a three-band infrared image as an example, the method employed uses a multi-branch backbone network specifically designed for feature extraction of infrared multi-band image pairs. This network is used for deep feature-level fusion of the three-band infrared image, with the fusion process as follows: three-band input image → feature extraction of each band → feature alignment + cross-attention fusion → unified fusion of features. This module effectively improves the contrast between the target and the background and enhances the detectability of small targets through the collaborative design of multi-scale information interaction and attention mechanisms. Specifically, Multi-scale Conv is multi-scale convolution, Upsample is convolutional upsampling, Concat is feature concatenation, and pose head is head pose estimation.

[0043] In the single-band feature extraction stage, the three-band infrared images are independently input into their respective branch networks for processing. Taking the three-band branch (mid-band feature map) as an example... Second mid-wave characteristic diagram and long-wavelength feature map Taking the i-th layer (where i is 3, 4, 5, or 6) as an example, its height, width, and number of channels are determined by the corresponding dimensions. The input infrared images are all single-channel grayscale images. Each branch uses an independent feature extraction backbone network for processing, and these backbone networks have their own independent parameters.

[0044] Next, multi-band information from different backbone networks is aggregated through cross-band feature fusion to generate the fused feature of the i-th layer. This fusion process is accomplished by a parameter-learnable feature fusion function. This invention employs a cross-attention feature fusion module (FCA module) to complete this process, as follows: Figure 3 As shown.

[0045] The FCA (Feature Channel Attention) module implements pairwise interactions and multi-directional complementarity among the three band features. First, 1×1 convolutions are applied independently to each of the three input band feature maps for channel dimensionality reduction, resulting in a uniform output channel count C' less than the original channel count, yielding three reduced-dimensional feature maps. Next, a cross-band correlation matrix is ​​constructed using dot product operations, and softmax normalization is applied to the matrix to generate attention weight maps representing the spatial similarity between different band pairs. Then, for each target band, the attention weight maps of the other two bands are used to weighted aggregate contextual information from the corresponding reduced-dimensional features, and the sum of the two contextual paths is added element-wise to the local features of that band, thus obtaining enhanced features containing complementary information from the other two bands (e.g., ...). Figure 3 Enhanced long-wavelength characteristics of the FCA module in Each of the three branches is enhanced independently in this manner; finally, the enhanced features of the three bands are input into a unified fusion function. This patent uses NiN fusion (Network in Network) to generate the final three-band fused feature map. This mechanism achieves more comprehensive multi-spectral information complementarity through a strategy of multi-band branch enhancement and unified fusion, significantly improving the robustness of target detection and the detectability of small targets in complex scenarios.

[0046] Finally, the fused feature map is fed into the neck module of the detector for further aggregation and enhancement of multi-scale features; the output fused feature map is as follows: Figure 4 As shown in the figure, the model results are plotted on the main band. The blue box represents the target recognition result (fighter, target aircraft), the white box represents the interference recognition result (point-jammer), the upper left corner represents the situation recognition result corresponding to the target (tail, wake), and the colored dots in the blue box represent the target key point detection results.

[0047] Feature extraction is performed on the fused feature map to obtain image features, specifically as follows:

[0048] By performing three-layer feature extraction on the fused feature map, image information can be captured more fully, the impact of low infrared features caused by cloud radiation suppressing or submerging the target can be reduced, and false alarms caused by target-like radiation sources in the cloud edge and ground background can be reduced. False detections and missed detections in the identification and tracking of weak targets can also be reduced. By combining the target detection head, key point detection head, and situation classification detection head, each detection head has an independent branch and decoupled tasks, which improves the model's generalization ability and detection efficiency.

[0049] Feature extraction from the fused feature map involves three layers, with a backbone architecture designed based on YOLOv8s-P2, such as... Figure 5As shown, balancing detection accuracy and inference efficiency, YOLOv8s-P2 is a network optimized based on the YOLOv8 series models, specifically designed for small object detection. Its core advantages lie in the newly added P2 detection layer and a series of structural improvements for small object detection. The backbone network input image (640×640×3) is processed by Conv... After a 3×3 (output 320x320x16) process, multi-scale features are extracted sequentially through the C2f1, C2f2, and C2f3 feature extraction modules, and then input into layers P2, P3, and P4 of the neck network, respectively. Layer P2 achieves cross-scale feature fusion through Concat (feature concatenation) and Upsample (convolutional upsampling) and inputs it into layer P3. Layer P3 achieves cross-scale feature fusion through Concat and Upsample and inputs it into layer P4, allowing feature information from different layers to complement each other and improve the adaptability of multi-scale target detection. Layers P2, P3, and P4 then output the processed features through the head network. The head network of layer P2 integrates the DyHead module, which integrates size-aware attention (assigning feature weights to targets of different sizes), spatial awareness attention (enhancing feature responses to target spatial locations), and task-aware attention (distinguishing feature requirements for detection and classification tasks).

[0050] The improvements are as follows:

[0051] The feature map size of layer P2 is 160×160, which, compared to layer P3 (80×80, detecting objects ≥ 8×8 pixels) in the standard YOLOv8 model, enables the detection of small targets as small as 4×4 pixels. This high-resolution feature map preserves more shallow details, which is crucial for the edge, texture, and spatial information of small targets. Small targets typically occupy only a few pixels in an image, and the standard model loses details due to downsampling in deep convolutions, while layer P2 significantly reduces information loss by extracting high-resolution features early on.

[0052] Small object detection relies on shallow features because deep convolutions compress spatial information, causing small object features to be obscured by background or noise. The P2 layer, located before the three-layer feature extraction, directly extracts information from the high-resolution object image, enhancing the ability to perceive small objects. The introduction of the P2 layer reduces the false negative rate for small objects, especially in dense scenes.

[0053] Secondly, YOLOv8s-p2 integrates a dynamic head (DyHead) module in layer P2, further enhancing the performance of small object detection. DyHead dynamically adjusts the feature map processing method, adaptively optimizing feature extraction based on the complexity of the input image and the target distribution. This is particularly important for small objects, as they are easily mistaken for noise or background in complex environments. DyHead improves detection accuracy by enhancing the feature representation of the target region and suppressing irrelevant background information. Furthermore, DyHead facilitates feature fusion between layer P2 and other detection layers (P3, P4), combining local features of small objects with global contextual information, further improving the model's ability to locate and classify small objects. The integration of the dynamic head module improves the overall feature pyramid network and enhances the feature fusion mechanism. It also ensures that features from layer P2 can be efficiently transferred to subsequent detection heads, ensuring that high-resolution features are effectively transmitted to the detection heads and avoiding feature degradation in deep networks.

[0054] To simultaneously perceive the target's environmental background information (sky, ground, sea), this application extends the detection network into a multi-head structure. For example... Figure 6 As shown, the target detection head, keypoint detection head, and situation classification detection head detect the extracted image features respectively. The main detection head (target detection head and keypoint detection head) is used for routine target recognition and localization, providing location information for targets, point sources, and area source interference. The auxiliary detection head (situation classification detection head) discriminates the background type of the target area, realizing environmental perception. Multiple detection heads share the backbone feature extraction network and perform appropriate information exchange between the detection heads, making target detection and environmental perception complementary and mutually optimized. This not only improves the detection accuracy of small targets in infrared images but also achieves effective perception of the surrounding environment. The layers used are: FC (Fully Connected), R (ReLU), C (Convolution), T (Transpose Convolution), U (Upsample), and B (Batch Normalization).

[0055] When the target is too small and its structure is unclear, the network cannot output key points. When the target is detected and its structure is clear, key point information can be accurately output. For experimental data, fusion has little impact on the accuracy of key points for targets with identifiable structures, but the average accuracy drops sharply. As can be seen from the target bounding box detection rates in Table 1 for each situation, the role of image fusion is mainly to help the network acquire features from three bands, improve the target recognition rate, and thus affect key point output. The accuracy of single-band image output for situational information is also low, while fused images can correctly identify targets with an accuracy of up to 99.3%. Quantitative analysis is shown in Table 1. Based on the evaluation indicators mentioned, the target bounding box detection rate, false alarm rate, key point accuracy, and situational awareness accuracy of simulation data, experimental data based on fusion, and experimental data based on single bands were statistically analyzed for each situation.

[0056] Table 1: Quantitative Analysis Table

[0057]

[0058] The target detection head, key point detection head, and situation classification detection head are specifically as follows:

[0059] This object detection head, based on a decoupled YOLOv8 design, outputs bounding boxes and categories for the target, point sources, and area sources. Focal Loss and Intersection over Union (IoU) Loss are used to optimize classification and regression accuracy. The original YOLOv8 detection head is coupled, using a single convolutional layer to perform both classification and localization simultaneously. This object detection head adopts a decoupled structure, with two parallel branches extracting category and location features respectively, each then using a 1×1 convolutional layer to complete the classification and regression tasks. The decoupled detection head accelerates convergence and improves detection accuracy.

[0060] The key point detection head regresses the coordinates (x, y, v) of five three-dimensional key points (nose, tail, right wing, left wing, and belly) of the target, where x is the horizontal coordinate, y is the vertical coordinate, and v is the visibility. In this invention, unmarked key points are in the format (0,0,0), and marked key points are in the format (x, y, 2).

[0061] The keypoint detection head is based on a convolutional layer design. Image features include multiple candidate regions, and for each candidate region, a 15-dimensional output (5 keypoints × 3-dimensional coordinates) is predicted. The keypoint detection head first compresses the feature dimensions using a 3×3 convolution, and then generates keypoint coordinates using a 1×1 convolution. The regression process employs the L2 loss function to optimize the Euclidean distance between the predicted and ground truth coordinates, ensuring high-precision localization.

[0062] The situation classification and detection head classifies five types of situations (lateral, tail, look-down, head-on, and look-up) through a fully connected layer, outputs a probability distribution, and is optimized using cross-entropy loss.

[0063] The key points include the nose, tail, right wing, left wing, and belly of the aircraft. The situational awareness includes lateral, rear-view, downward, head-on, and upward views. It is mainly used to verify the accuracy of network recognition, the accuracy of key point recognition, and situational awareness capabilities.

[0064] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art, guided by the specification, can make many other modifications without departing from the scope of the claims of the present invention, and all of these modifications are within the scope of protection of the present invention.

Claims

1. A multi-band image end-to-end multi-task intelligent detection method, characterized in that, The method comprises the following steps: obtaining a multi-band target image; performing feature extraction on the multi-band target image according to the bands respectively, fusing the respective features of the multi-band target image, and obtaining a fused feature map; performing feature extraction on the fused feature map, and obtaining an image feature; inputting the image feature into an intelligent detection model, and outputting an intelligent detection result; wherein the feature extraction on the fused feature map comprises three layers of feature extraction, the three layers of feature extraction are designed based on YOLOv8s-P2, the three layers of feature extraction are arranged in descending order according to the sizes of the extracted feature maps, and the intelligent detection model comprises a target detection head, a key point detection head and a situation classification detection head; the respective features of the multi-band target image are fused to obtain the fused feature map, specifically as follows: the respective features of the multi-band target image are fused by using an FCA module, the FCA module realizes two-by-two interaction and multi-directional complementation among the features of the three bands, first, 1×1 convolution is independently applied to the input three-band feature map to reduce the channel dimension, the output channel number is unified to be less than the original channel number C', and three reduced dimension feature maps are obtained; then, a cross-band correlation matrix is constructed through dot product operation, and softmax normalization is applied to the matrix to generate an attention weight map representing the spatial similarity between different band pairs; subsequently, for each target band, the attention weight map of the remaining two bands is used to weight and aggregate the context information from the corresponding reduced dimension feature, and the sum of the two context information is added to the local feature of the band element by element, thereby obtaining enhanced features containing the complementary information of the remaining two bands, and the three branches are independently enhanced in this way; Finally, the three enhanced band features are input into a unified fusion function, and NiN fusion is adopted to generate the final three-band fusion feature map .

2. The multi-band image end-to-end multi-task intelligent detection method according to claim 1, characterized in that, the target detection head adopts a decoupling head structure.

3. The multi-band image end-to-end multi-task intelligent detection method according to claim 2, characterized in that, The decoupling head structure comprises two parallel branches, which respectively extract class features and position features from the image feature, and then complete classification and regression tasks through a layer of 1×1 convolution.

4. The multi-band image end-to-end multi-task intelligent detection method according to claim 3, characterized in that, The classification and regression tasks are completed by using focal loss to improve the classification progress and using intersection over union loss to improve the regression accuracy.

5. The multi-band image end-to-end multi-task intelligent detection method according to claim 1, characterized in that, The key point detection head is designed based on a convolution layer, the image feature comprises a plurality of candidate regions, the key point detection head predicts a 15-dimensional output for each candidate region of the image feature, the key point detection head first compresses the feature dimension through 3×3 convolution, and then generates key point coordinates through 1×1 convolution.

6. The multi-band image end-to-end multi-task intelligent detection method according to claim 5, characterized in that, The key point coordinates include the horizontal coordinates, vertical coordinates and visibility of the key points.

7. The multi-band image end-to-end multi-task intelligent detection method according to claim 1, characterized in that, The situation classification detection head classifies the situation of the image feature and identifies the environmental background information in the image feature through a fully connected layer, outputs a probability distribution result after classification and identification, and optimizes the probability distribution result by using cross-entropy loss.

8. The multi-band image end-to-end multi-task intelligent detection method according to claim 1, characterized in that, The target image comprises a target aircraft, the key points detected by the key point detection head comprise a nose, a tail, a right wing, a left wing and a belly, and the situations classified by the situation classification detection head comprise a lateral, a tail, a downward view, a head-on and an upward view.

9. The multi-band image end-to-end multi-task intelligent detection method according to claim 1, characterized in that, A dynamic head module is integrated in the first layer of feature extraction in the three layers of feature extraction, and the dynamic head module is used for adaptively optimizing feature extraction according to the complexity and target distribution of the input image.

10. The multi-band image end-to-end multi-task intelligent detection method according to claim 1, characterized in that, The respective features of the multi-band target image are fused, specifically as follows: The cross-attention fusion is performed on the multi-band target image, features of the multi-band target image are extracted first, similarity of features of each band is calculated, features of each band are aligned, and are aggregated and enhanced, and finally multi-band fusion feature maps are obtained through a fusion function.

Images