Cross-modal and multi-scale UAV camouflage target detection method, system and medium

Abstract

The present invention discloses a cross-modal multi-scale UAV camouflage target detection method, system and medium. The present invention includes extracting visible light features and thermal infrared imaging features; encoding them as features, and then fusing them into RGB image modal features in a modal scale; encoding them as features; fusing them in a modal scale to obtain the final fusion features, extracting features layer by layer to obtain decoding results, and classifying them to obtain camouflage target detection results. The present invention aims to solve the problem of camouflage target detection in which the size of camouflage targets in airborne images is extremely small and changeable due to changes in flight altitude in practical applications of UAVs, realize accurate detection of extremely small objects, and overcome the problem of low camouflage target detection accuracy due to size changes.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a cross-modal and multi-scale unmanned aerial vehicle camouflage target detection method, system and medium. Background Art

[0002] Camouflaged target recognition refers to the detection and identification of camouflaged objects hidden in the environment in a complex background. Camouflaged target recognition is an important research direction in the field of computer vision and has broad application prospects. In natural scenes, compared with traditional machine learning methods, the accuracy of camouflaged target detection based on deep learning has been greatly improved. For example, RGB-T camouflaged target detection extracts features from visible light images and thermal infrared images, extracts semantic information of each modality, and divides the pixel-level target area. However, when identifying camouflaged targets of drones, the change in the flight altitude of the drone will cause the size of the target in the image to change. At the same time, the camouflaged target itself is small in size and hidden in a complex environment, making it difficult to detect. When the object is very small, the advantage of visible light images with rich texture details is weakened. The contrast of thermal infrared images is obvious, and the edges of objects are relatively clear. Therefore, cross-modal multi-scale camouflaged target detection becomes an important approach. In summary, how to effectively extract and fuse multi-modal information in visible light images and infrared images, accurately locate targets of variable sizes, and achieve high-precision detection of camouflaged targets in complex environments has become a key technical problem that needs to be solved urgently. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a cross-modal and multi-scale UAV camouflage target detection method, system and medium are provided. The present invention aims to solve the problem of camouflage target detection in the airborne image of the UAV due to the change of flight altitude in actual application, realize the accurate detection of objects with extremely small size, and overcome the problem of low detection accuracy of camouflage targets due to size changes.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A cross-modal and multi-scale UAV camouflage target detection method comprises the following steps:

[0006] S1, extracting S types of visible light features of sampling size from the input visible light image , thermal infrared imaging features extracted from the input thermal infrared image ;

[0007] S2, the visible light features of S sampling sizes The trained encoders are used to encode Characteristics of scale , and then the modal internal scale fusion is obtained RGB image modality features of different scales ; Thermal infrared imaging features Use the trained, independent encoder to encode Modal characteristics of thermal infrared images at different scales ;

[0008] S3, RGB image modality features of different scales and thermal infrared image modality features Perform inter-modal scale fusion to obtain the final fusion feature ;

[0009] S4, the final fusion feature Extract features layer by layer to get decoding results , the decoding result Classify and obtain the camouflaged target detection results.

[0010] Optionally, in step S1, the visible light features of S sampling sizes obtained by extracting the image pyramid from the input visible light input image are: include:

[0011] S1.1, convert the input visible light image and thermal infrared image into RGB three-channel matrices respectively to obtain the visible light image input and thermal infrared imaging input ;

[0012] S1.2, input thermal infrared imaging Reorganize the shape or size to a specified size to obtain thermal infrared imaging features ; Input the visible light image Interpolation is performed and then an image pyramid is constructed based on it to obtain visible light features of S sampling sizes. ,in Indicates the sth sampling size.

[0013] Optionally, in step S2, the visible light features of S sampling sizes are The trained encoders are used to encode Characteristics of scale , and thermal infrared imaging features Use the trained, independent encoder to encode Modal characteristics of thermal infrared images at different scales After that, it also includes Enhance the features of the previous scale: The features of the first scale are expressed using a combined module CBR consisting of convolution, normalization, and ReLU activation functions. The features of the scales are extracted using the atrous spatial convolution pooling pyramid module ASPP, which includes two layers; the first layer is composed of four convolution modules to form features of multiple different receptive fields, the convolution kernel sizes of the three convolution modules in the first layer are 3×3, the convolution kernel size of the remaining convolution module is 1×1, and the three convolution modules with the convolution kernel size of 3×3 are convolution modules with expansion factors and filling factors, and the expansion factors and filling factors of the three convolution modules are 2, 5 and 7 respectively; the second layer is a convolution module with a convolution kernel size of 1×1, which is used to splice the features of the multiple receptive fields obtained in the first layer according to the channels, and then compress the channels through the convolution module with a convolution kernel size of 1×1 to obtain the features of the scale output.

[0014] Optionally, the visible light features of the S sampling sizes in step S1 They include three kinds of visible light features with different sampling multiples; the modal internal scale fusion in step S2 is obtained RGB image modality features of different scales include:

[0015] S2.1, Visible light characteristics for S sampling sizes For each scale, the visible light feature with the largest sampling multiple is downsampled to the size of the visible light feature with a medium sampling multiple, and then the maximum pooling and average pooling are used to extract the features respectively; the visible light feature with the smallest sampling multiple is upsampled to the size of the visible light feature with a medium sampling multiple through interpolation, and then the feature expression is enhanced through a combined module CBR formed by convolution, normalization and ReLU activation function;

[0016] S2.2, the maximum pooling and average pooling extraction results of the visible light feature with medium sampling multiples, the visible light feature with the largest sampling multiples, and the enhanced feature expression results of the visible light feature with the smallest sampling multiples are concatenated to obtain the feature , thus obtaining Features corresponding to different scales The RGB image modality features .

[0017] Optionally, in step S3 RGB image modality features of different scales and thermal infrared image modality features The final fusion feature is obtained by fusion of scales The function expression is:

[0018] ,

[0019] In the above formula, is the final fusion feature of the i-th scale, is the RGB image modality feature of the i-th scale, is the modal feature of thermal infrared image at the i-th scale, It is a combined module formed by convolution, normalization and ReLU activation functions.

[0020] Optionally, in step S4, the final fusion feature Extract features layer by layer to get decoding results When including the use of The pyramid-shaped layered multi-scale unit HMU extracts features layer by layer, and the output of the last layer of the layered multi-scale unit HMU is transformed into the output size by a combined module CBR composed of convolution, normalization and Relu activation function, and a convolution with a convolution kernel size of 1×1. , and any i-th layer of hierarchical multi-scale unit HMU obtains the output Included are:

[0021] S4.1, the final fusion feature of the i-th scale The first set of feature groups is generated according to the following formula :

[0022] ,

[0023] In the above formula, Indicates average grouping by channel, Represents a combined module formed by convolution, normalization and Relu activation functions;

[0024] S4.2, grouping the first set of features Generate the weight coefficient grouping of the i-th layer according to the following formula and the enhanced extracted feature grouping of the i-th layer ;

[0025] ,

[0026] In the above formula, is the first set of feature grouping output by the i-th layer, is the first set of feature grouping output by the i-1th layer, Indicates average grouping by channel, Represents a combined module formed by convolution, normalization and Relu activation functions;

[0027] S4.3, grouping weight coefficients Transformed into weight scores through the network shown in the following formula:

[0028] ,

[0029] In the above formula, is the weight score, is the softmax activation function, is convolution, is the ReLU activation function, The network architecture is as follows: first, the mean pooling layer is used to retain more detail information, then convolution and Relu activation function are performed, and then a convolution layer and Sigmoid are used to obtain the weight score;

[0030] S4.4, calculate the i-th layer of hierarchical multi-scale unit HMU according to the following formula to get the output :

[0031] ,

[0032] In the above formula, is the activation function, is the final fusion feature of the i-th scale, is the output of the i-1th layer of hierarchical multi-scale unit HMU, For batch normalization, represents convolution, Feature grouping for enhanced extraction.

[0033] Optionally, in step S4, the decoding result Classification to obtain the camouflaged target detection result means to decode the result The camouflaged target detection result is obtained through the full connection layer and Softmax function classification.

[0034] In addition, the present invention also provides a cross-modal multi-scale UAV camouflaged target detection system, comprising an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the cross-modal multi-scale UAV camouflaged target detection method.

[0035] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cross-modal and multi-scale UAV camouflage target detection method through a processor.

[0036] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cross-modal and multi-scale drone camouflage target detection method through a processor.

[0037] Compared with the prior art, the present invention has the following advantages: the present invention comprises extracting S kinds of visible light features of sampling size from the input visible light image by using the image pyramid , thermal infrared imaging features extracted from the input thermal infrared image ; S kinds of visible light features of sampling size The trained encoders are used to encode Characteristics of scale , and then the modal internal scale fusion is obtained RGB image modality features of different scales ; Thermal infrared imaging features Use the trained, independent encoder to encode Modal characteristics of thermal infrared images at different scales ;Will RGB image modality features of different scales and thermal infrared image modality features Perform inter-modal scale fusion to obtain the final fusion feature ; The final fusion feature Extract features layer by layer to get decoding results , the decoding result The camouflaged target detection result is obtained by classification. Considering that the camouflaged object is extremely small in size, highly similar to the background, or is largely obscured, and the target size changes greatly in the airborne image due to the change in flight altitude, the detection of camouflaged objects in drone images is more complex and challenging than the traditional salient object detection. The present invention integrates the two modal information of thermal infrared image and visible light image, realizes the fusion of multiple features, can effectively discover the correlation and semantic information between the modalities, and utilizes the correlation between the two modalities of thermal infrared image and visible light image to more accurately locate the target and improve the boundary integrity of the detection result; the present invention introduces multi-scale information and provides the representation of the original image at different resolutions, so that the image can be analyzed and processed at multiple scales, and the important clues in the visible light image can be fully extracted under chaotic conditions, effectively overcoming the problem of low detection accuracy of camouflaged targets due to extremely small and variable size; the present invention is suitable for the detection of concealed targets by thermal imaging of drones in complex scenes, and can effectively overcome the size changes of drone image targets, which provides the possibility for the actual remote detection of concealed targets. The present invention can solve the problem of detecting camouflaged targets of extremely small and variable sizes in airborne images of UAVs due to changes in flight altitude in practical applications, achieve accurate detection of objects of extremely small sizes, and overcome the problem of low detection accuracy of camouflaged targets due to size changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0039] Figure 2 Schematic diagram of the network structure in an embodiment of the present invention.

[0040] Figure 3 Schematic diagram of the process of image preprocessing of thermal infrared images in an embodiment of the present invention.

[0041] Figure 4 4 is a schematic diagram of the process of image preprocessing of visible light images in an embodiment of the present invention.

[0042] Figure 5 Schematic diagram of the network structure of the atrous spatial convolutional pooling pyramid module ASPP in an embodiment of the present invention.

[0043] Figure 6 Schematic diagram of the network structure of a hierarchical multi-scale unit HMU in an embodiment of the present invention.

[0044] Figure 7 1 is the test result of the embodiment of the present invention on the UAV-5000 dataset, where (a) to (d) are visible light images on the UAV-5000 dataset, (a1) to (d1) are corresponding mask images, (a2) to (d2) are corresponding thermal infrared images, and (a3) ​​to (d3) are detection results corresponding to the method of this embodiment. DETAILED DESCRIPTION

[0045] like Figure 1 and Figure 2 As shown, the cross-modal and multi-scale UAV camouflage target detection method of this embodiment includes the following steps:

[0046] S1, extracting S types of visible light features of sampling size from the input visible light image , thermal infrared imaging features extracted from the input thermal infrared image ;

[0047] S2, the visible light features of S sampling sizes The trained encoders are used to encode Characteristics of scale , and then the modal internal scale fusion is obtained RGB image modality features of different scales ; Thermal infrared imaging features Use the trained, independent encoder to encode Modal characteristics of thermal infrared images at different scales ;

[0048] S3, RGB image modality features of different scales and thermal infrared image modality features Perform inter-modal scale fusion to obtain the final fusion feature ;

[0049] S4, the final fusion feature Extract features layer by layer to get decoding results , the decoding result Classify and obtain the camouflaged target detection results.

[0050] In step S1 of this embodiment, the visible light features of S sampling sizes are obtained by extracting the image pyramid from the input visible light input image. include:

[0051] S1.1, convert the input visible light image and thermal infrared image into RGB three-channel matrices respectively to obtain the visible light image input and thermal infrared imaging input ;

[0052] S1.2, input thermal infrared imaging Reorganize the shape or size to a specified size to obtain thermal infrared imaging features ; Input the visible light image Interpolation is performed and then an image pyramid is constructed based on it to obtain visible light features of S sampling sizes. ,in Indicates the sth sampling size. For example, in this embodiment, an image pyramid is constructed to obtain visible light features of S sampling sizes: They include: , and The three sampling sizes of visible light features have corresponding sampling multiples of 0.5, 1 and 1.5. It should be noted that interpolation is a well-known data processing method, and the required interpolation algorithm can be adopted as needed. For example, the bilinear interpolation algorithm is adopted in this embodiment. The bilinear interpolation algorithm is an interpolation method based on the weighted average of known pixel values ​​in the local area. It takes into account the changes in pixel values ​​in the horizontal and vertical directions, so as to better preserve the details and quality of the image. Specifically, bilinear interpolation will find the four pixels closest to the target pixel position in the original image, and then perform weighted averaging based on the pixel values ​​of these four pixels to obtain the value of the target pixel. First, interpolation is implemented on the X-axis, and then the value of the target pixel is obtained along the X-axis. , Find two points with the same horizontal coordinate as the insertion point and :

[0053] ,

[0054] ,

[0055] In the above formula, the coordinates of the point to be inserted are , The point coordinates are , The point coordinates are , ~ are four points near the insertion point, with coordinates corresponding to Then interpolate the Y axis and use the interpolation result from the previous step. , , linear interpolation is performed in the vertical direction.

[0056] ,

[0057] Get the value of the insertion point. Based on this, build an image pyramid and get the visible light feature , and .

[0058] It should be noted that, in the training phase, step S1.2 of this embodiment further includes: and thermal infrared imaging input For enhanced processing:

[0059] S1.2.1, input the visible light image and thermal infrared imaging input Concatenate by channel to get the input matrix ;

[0060] S1.2.2, the input matrix Flip from left to right along the horizontal direction and then rotate at random angles to obtain the enhanced input matrix , which will enhance the input matrix Divide into visible light enhancement input according to channels and thermal infrared enhanced input For example, in this embodiment, the input matrix is ​​enhanced If the size is 384×384×6, the 6 channels are divided into two visible light enhancement inputs of size 384×384×3 and thermal infrared enhanced input ;

[0061] S1.2.3, thermal infrared enhancement input Reorganize the shape or size to a specified size to obtain thermal infrared imaging features For example, in this embodiment, the thermal infrared enhancement input Reshape or size to (batch_size, 3,384, 384), which means the batch size is batch_size, the image size is 384×384, the number of channels is 3, and the thermal infrared imaging features are obtained. ; Enhance the visible light input Interpolation is performed and then an image pyramid is constructed based on it to obtain visible light features of S sampling sizes. ,in Indicates the sth sampling size.

[0062] In addition, as an optional implementation, Figure 3 As shown, for thermal infrared image input The following methods can also be used for enhancement processing: mirror symmetry transformation along the horizontal axis of the image; secondly, rotate the image by randomly selecting an angle, and finally use mean normalization to obtain the thermal infrared enhancement input .

[0063] In addition, as an optional implementation, Figure 4 As shown, for visible light image input The following method can also be used for enhancement processing: mirror symmetry transformation along the horizontal axis of the image; secondly, rotate the image by randomly selecting an angle, and finally use mean normalization to obtain visible light enhancement input ; Enhanced input for visible light , the visible light is enhanced into Interpolation is performed and then an image pyramid is constructed based on this to obtain visible light features of S sampling sizes: , and .

[0064] like Figure 2 As shown, in this embodiment, the value of M is 5, and in step S2, the visible light features of the three sampling sizes are , and The trained encoders are used to encode the features into 5 scales. , where the superscript R represents the visible light image, and the subscript i represents the i-th scale; the modal features of RGB images of 5 scales are obtained by intra-modal scale fusion ; Thermal infrared imaging features The trained, independent encoder is used to encode the thermal infrared image modality features at five scales. , where the superscript T represents the thermal infrared image, and the subscript i represents the i-th scale. As an optional implementation, Figure 2As shown, the encoder used in this embodiment is specifically selected as the ResNet-50 neural network. The structure of ResNet-50 has a deeper network layer than the traditional convolutional neural network, and introduces the idea of ​​residual connection, which solves the gradient disappearance and gradient explosion problems in deep network training. Residual connection allows the network to learn identity mapping, making network training easier. In addition, other encoders can also be used.

[0065] In step S2 of this embodiment, the visible light features of S sampling sizes are The trained encoders are used to encode Characteristics of scale , and thermal infrared imaging features Use the trained, independent encoder to encode Modal characteristics of thermal infrared images at different scales After that, it also includes Enhance the features of the previous scale: The features of the first scale are expressed using a combined module CBR consisting of convolution, normalization, and ReLU activation functions. The features of various scales are extracted using the atrous spatial convolutional pooling pyramid module ASPP.

[0066] like Figure 5 As shown, the atrous spatial convolution pooling pyramid module ASPP includes two layers; the first layer is composed of four convolution modules for forming features of multiple different receptive fields, the convolution kernel size of the three convolution modules in the first layer is 3×3, the convolution kernel size of the remaining convolution module is 1×1, and the three convolution modules with a convolution kernel size of 3×3 are convolution modules with expansion factors and filling factors, and the expansion factors and filling factors of the three convolution modules are 2, 5 and 7 respectively; the second layer is a convolution module with a convolution kernel size of 1×1, which is used to concatenate the features of multiple different receptive fields obtained in the first layer by channel, and then compress the channel through a convolution module with a convolution kernel size of 1×1 to obtain the features of the scale output.

[0067] The visible light features of the S sampling sizes in step S1 of this embodiment The three sampling sizes of visible light features are respectively included; in this embodiment, a scale merging layer module is constructed, and after filtering by the scale fusion of the modalities, the feature expression of each scale is highlighted; the scale fusion of the modalities in step S2 is obtained RGB image modality features of different scales include:

[0068] S2.1, Visible light characteristics for S sampling sizes For each scale, the visible light feature with the largest sampling multiple (in this embodiment, that is: ) downsample to the size of visible light features with a medium sampling multiple, and then use maximum pooling and average pooling to extract features; the function expression of maximum pooling is:

[0069] ,

[0070] In the above formula, Indicates The feature map is related to the rectangular area The maximum pooling output value of Represents a rectangular area Located in Selecting the maximum value as the output in each pooling window can retain the most significant features of the image and make the network robust to slight position shifts.

[0071] Among them, the function expression of average pooling is:

[0072] ,

[0073] In the above formula, Indicates The feature map is related to the rectangular area The average pooling output value of Represents a rectangular area Located in The elements at Represents a rectangular area The number of elements in . The average value of all pixel values ​​in each pooling window is calculated as the output. Average pooling is smoother and can retain more detail information.

[0074] The visible light feature with the smallest sampling multiple is upsampled to the size of the visible light feature with a medium sampling multiple through interpolation, and then the feature expression is enhanced by a combined module CBR formed by convolution, normalization and ReLU activation function. As an optional implementation, in this embodiment, the visible light feature with the smallest sampling multiple is Using bilinear interpolation, the features are upsampled to Same size;

[0075] S2.2, the maximum pooling and average pooling extraction results of the visible light feature with medium sampling multiples, the visible light feature with the largest sampling multiples, and the enhanced feature expression results of the visible light feature with the smallest sampling multiples are concatenated to obtain the feature , thus obtaining Features corresponding to different scales The RGB image modality features .

[0076] In step S3 of this embodiment, RGB image modality features of different scales and thermal infrared image modality features The final fusion feature is obtained by fusion of scales The function expression is:

[0077] ,

[0078] In the above formula, is the final fusion feature of the i-th scale, is the RGB image modality feature of the i-th scale, is the modal feature of thermal infrared image at the i-th scale, It is a combined module formed by convolution, normalization and ReLU activation functions.

[0079] In step S4 of this embodiment, the final fusion feature Extract features layer by layer to get decoding results When including the use of The pyramid-shaped HMU (Hierarchical Mixed-scale Unit) is used to extract features layer by layer, and the output of the last layer of HMU is transformed into a combined module CBR composed of convolution, normalization and Relu activation function, and a convolution with a convolution kernel size of 1×1. The final feature is converted into the output size to obtain the decoding result. .like Figure 2 As shown, the five layers of hierarchical multi-scale units HMU in this embodiment are respectively represented as HMU 5 ~HMU 1 , and from HMU 5 Output After a combined module CBR composed of convolution, normalization and Relu activation function, and a convolution with a convolution kernel size of 1×1, the final feature is converted to the output size to obtain the decoding result. .like Figure 6 As shown, any i-th layer of hierarchical multi-scale unit HMU obtains the output Included are:

[0080] S4.1, the final fusion feature of the i-th scale The first set of feature groups is generated according to the following formula :

[0081] ,

[0082] In the above formula, Indicates average grouping by channel, Represents a combined module formed by convolution, normalization and Relu activation functions;

[0083] S4.2, grouping the first set of features Generate the weight coefficient grouping of the i-th layer according to the following formula and the enhanced extracted feature grouping of the i-th layer ;

[0084] ,

[0085] In the above formula, is the first set of feature grouping output by the i-th layer, is the first set of feature grouping output by the i-1th layer, Indicates average grouping by channel, Represents a combined module formed by convolution, normalization and Relu activation functions;

[0086] S4.3, grouping weight coefficients Transformed into weight scores through the network shown in the following formula:

[0087] ,

[0088] In the above formula, is the weight score, is the softmax activation function, is convolution, is the ReLU activation function, The network architecture is as follows: first, the mean pooling layer is used to retain more detail information, then convolution and Relu activation function are performed, and then a convolution layer and Sigmoid are used to obtain the weight score;

[0089] S4.4, calculate the i-th layer of hierarchical multi-scale unit HMU according to the following formula to get the output :

[0090] ,

[0091] In the above formula, is the activation function, is the final fusion feature of the i-th scale, is the output of the i-1th layer of hierarchical multi-scale unit HMU, For batch normalization, represents convolution, To enhance the feature extraction grouping. Through the above method, iterate five HMU units, learn key clues in different channels, and obtain a powerful feature representation. Different channels contain different semantics, and valuable clues contained in different channels can be mined. Use HMU to perform information interaction and feature refinement between channels, thereby strengthening the features of the decoder from coarse-grained group-level iteration to fine-grained channel-level modulation. In step S4 of this embodiment, the decoding result Classification to obtain the camouflaged target detection result means to decode the result After the full connection layer and Softmax function classification, the camouflaged target detection result Y is obtained.

[0092] In order to test the effect of the detection model used in the cross-modal and multi-scale UAV camouflage target detection method of this embodiment, the model is trained and tested on the UAV-5000 dataset (see Wang G, Li C, Ma Y, et al. RGB-T saliency detection benchmark: Dataset, baselines, analysis and a novel approach [C] / / Image and Graphics Technologies and Applications: 13th Conference on Image and Graphics Technologies and Applications, IGTA 2018, Beijing, China, April 8–10, 2018, Revised Selected Papers 13. SpringerSingapore, 2018: 359-369.) to obtain the optimal hidden target detection model. The UAV-5000 dataset contributes an RGB-T image dataset, which includes 5000 pairs of spatially aligned RGB-T images and infrared thermal imaging images and their true value annotations for saliency detection. As a comparison of the method of this embodiment; the comparison method MROS (see W. Bo, J. Liu, X. Fan, T. Tjahjadi, Q. Ye and L. Fu, "BASNet: Burned areasegmentation network for real-time detection of damage maps in remote sensing images", IEEE Trans. Geosci. Remote Sens., vol. 60, 2022.). Moreover, the enhanced metric (E-measure) and the structural similarity metric (S-measure) are used to measure the accuracy of recognition. The higher the enhanced metric and the structural similarity metric, the better the detection effect. The enhanced metric is a metric used to evaluate the performance of a saliency detection algorithm, especially for boundary accuracy and regional similarity. It combines concepts such as structural similarity, regional similarity, and mutual information in information theory, and can more comprehensively evaluate the accuracy of saliency detection results. The structural similarity metric mainly evaluates performance by comparing the structural similarity between the detection results and the true saliency map. Table 1 shows the recognition results of the method of this embodiment and the MROS algorithm under the UAV-5000 dataset.

[0093] Table 1 Comparison of recognition results between the method in this embodiment and the WLAS algorithm on the LRS2 dataset

[0094]

[0095] It can be seen from Table 1 that the enhanced metric and structural similarity metric of hidden target detection of the method of this embodiment are higher than those of the MROS method of the prior art. The accuracy of the method of this embodiment is improved under different altitudes of the UAV and when the target is extremely small.

[0096] Figure 7 The test results of the embodiment of the method of the present invention on the UAV-5000 dataset, where (a) to (d) are visible light images on the UAV-5000 dataset, (a1) to (d1) are corresponding GTs, GT is a mask image corresponding to the image to be detected, where the salient areas are marked as white or high brightness values, and the non-salient areas are marked as black or low brightness values. This mask image provides the model with correct annotations of salient areas for training and evaluating the performance of the model, (a2) to (d2) are corresponding thermal infrared images, and (a3) ​​to (d3) are the corresponding detection results of the method of this embodiment. It can be seen that the method of this embodiment has a good detection effect when the target size is small, the light is blurred, the target is in a complex environment and hidden in the environment, fully explores the correlation between multi-modalities, extracts important semantic information in the input, can more accurately locate the target, and improve the boundary integrity of the detection result.

[0097] In summary, when the size of the hidden target is extremely small, the change in the flying altitude of the drone causes the size of the target to change, so the detection of camouflaged objects in drone images is more complex and challenging than traditional salient object detection or other object segmentation tasks. The multi-scale input of the image pyramid used in the cross-modal multi-scale drone camouflage target detection method of this embodiment takes into account the different expressions of the scene at different scales, which helps to improve the understanding and judgment of camouflaged objects. First, the scale-specific features are filtered and aggregated through the scale merging layer to enhance the feature representation. Next, in the hierarchical mixed-scale decoder, the semantics of the mixed scales are further mined through the strategies of grouping, mixing and fusion. The cross-modal multi-scale drone camouflage target detection method of this embodiment simultaneously utilizes the correlation between different modalities, extracts semantic information through multi-stage fusion, and the correlation between channels, and provides an accurate and effective solution for camouflaged object detection in complex environments.

[0098] In addition, this embodiment also provides a cross-modal multi-scale UAV camouflaged target detection system, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the cross-modal multi-scale UAV camouflaged target detection method.

[0099] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the cross-modal and multi-scale drone camouflage target detection method through a processor.

[0100] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the cross-modal and multi-scale drone camouflage target detection method through a processor.

[0101] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A cross-modal and multi-scale UAV camouflage target detection method, characterized in that: The steps include: S1, extracting S types of visible light features of sampling size from the input visible light image , thermal infrared imaging features extracted from the input thermal infrared image ; S2, the visible light features of S sampling sizes The trained encoders are used to encode Characteristics of scale , and then the modal internal scale fusion is obtained RGB image modality features of different scales ; Thermal infrared imaging features Use the trained, independent encoder to encode Modal features of thermal infrared images at different scales ; S3, RGB image modality features of different scales and thermal infrared image modality features Perform inter-modal scale fusion to obtain the final fusion feature ; S4, the final fusion feature Extract features layer by layer to get decoding results , the decoding result Classify and obtain the camouflaged target detection result; In step S4, the final fusion feature Extract features layer by layer to get decoding results When including the use of The pyramid-shaped layered multi-scale unit HMU extracts features layer by layer, and the output of the last layer of the layered multi-scale unit HMU is transformed into the output size by a combined module CBR composed of convolution, normalization and Relu activation function, and a convolution with a convolution kernel size of 1×1. , and any i-th layer of hierarchical multi-scale unit HMU obtains the output Included are: S4.1, the final fusion feature of the i-th scale The first set of feature groups is generated according to the following formula : ， In the above formula, Indicates average grouping by channel, Represents a combined module formed by convolution, normalization and Relu activation functions; S4.2, grouping the first set of features Generate the weight coefficient grouping of the i-th layer according to the following formula and the enhanced extracted feature grouping of the i-th layer ; ， In the above formula, is the first set of feature grouping output by the i-th layer, is the first set of feature grouping output by the i-1th layer, Indicates average grouping by channel, Represents a combined module formed by convolution, normalization and Relu activation functions; S4.3, grouping weight coefficients Transformed into weight scores through the network shown in the following formula: ， In the above formula, is the weight score, is the softmax activation function, is convolution, is the ReLU activation function, The network architecture is as follows: first, the mean pooling layer is used to retain more detail information, then convolution and Relu activation function are performed, and then a convolution layer and Sigmoid are used to obtain the weight score; S4.4, calculate the i-th layer of hierarchical multi-scale unit HMU according to the following formula to get the output : ， In the above formula, is the activation function, is the final fusion feature of the i-th scale, is the output of the i-1th layer of hierarchical multi-scale unit HMU, For batch normalization, represents convolution, Feature grouping for enhanced extraction.

2. The cross-modal and multi-scale UAV camouflage target detection method according to claim 1 is characterized in that: In step S1, the visible light features of S sampling sizes are obtained by extracting the image pyramid from the input visible light input image. include: S1.1, convert the input visible light image and thermal infrared image into RGB three-channel matrices respectively to obtain the visible light image input and thermal infrared imaging input ; S1.2, input thermal infrared imaging Reorganize the shape or size to a specified size to obtain thermal infrared imaging features ; Input the visible light image Interpolation is performed and then an image pyramid is constructed based on it to obtain visible light features of S sampling sizes. ,in Indicates the sth sampling size.

3. The cross-modal and multi-scale UAV camouflage target detection method according to claim 1 is characterized in that: In step S2, the visible light features of S sampling sizes are The trained encoders are used to encode Characteristics of scale , and thermal infrared imaging features Use the trained, independent encoder to encode Modal features of thermal infrared images at different scales Finally, it also includes Enhance the features of the previous scale: The features of the first scale are expressed using a combined module CBR consisting of convolution, normalization, and Relu activation functions. The features of the scales are extracted using the atrous spatial convolution pooling pyramid module ASPP, which includes two layers; the first layer is composed of four convolution modules to form features of multiple different receptive fields, the convolution kernel sizes of the three convolution modules in the first layer are 3×3, the convolution kernel size of the remaining convolution module is 1×1, and the three convolution modules with the convolution kernel size of 3×3 are convolution modules with expansion factors and filling factors, and the expansion factors and filling factors of the three convolution modules are 2, 5 and 7 respectively; the second layer is a convolution module with a convolution kernel size of 1×1, which is used to splice the features of the multiple receptive fields obtained in the first layer according to the channels, and then compress the channels through the convolution module with a convolution kernel size of 1×1 to obtain the features of the scale output.

4. The cross-modal and multi-scale UAV camouflage target detection method according to claim 1 is characterized in that: The visible light features of S sampling sizes in step S1 It includes three kinds of visible light features with different sampling multiples; In step S2, the modal internal scale fusion is obtained RGB image modality features of different scales include: S2.1, Visible light characteristics for S sampling sizes For each scale, the visible light feature with the largest sampling multiple is downsampled to the size of the visible light feature with a medium sampling multiple, and then the maximum pooling and average pooling are used to extract the features respectively; the visible light feature with the smallest sampling multiple is upsampled to the size of the visible light feature with a medium sampling multiple through interpolation, and then the feature expression is enhanced through a combined module CBR formed by convolution, normalization and ReLU activation function; S2.2, the maximum pooling and average pooling extraction results of the visible light feature with medium sampling multiples, the visible light feature with the largest sampling multiples, and the enhanced feature expression results of the visible light feature with the smallest sampling multiples are concatenated to obtain the feature , thus obtaining Features corresponding to different scales The RGB image modality features .

5. The cross-modal and multi-scale UAV camouflage target detection method according to claim 1 is characterized in that: In step S3, RGB image modality features of different scales and thermal infrared image modality features The final fusion feature is obtained by fusion of scales The function expression is: ， In the above formula, is the final fusion feature of the i-th scale, is the RGB image modality feature of the i-th scale, is the modal feature of thermal infrared image at the i-th scale, It is a combined module formed by convolution, normalization and ReLU activation functions.

6. The cross-modal and multi-scale UAV camouflage target detection method according to claim 1 is characterized in that: In step S4, the decoding result Classification to obtain the camouflaged target detection result means to decode the result The camouflaged target detection result is obtained through the full connection layer and Softmax function classification.

7. A cross-modal multi-scale drone camouflage target detection system, comprising an interconnected microprocessor and a memory, characterized in that: The microprocessor is programmed or configured to execute the cross-modal and multi-scale drone camouflage target detection method described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the cross-modal and multi-scale drone camouflage target detection method described in any one of claims 1 to 6 through a processor.

9. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the cross-modal and multi-scale drone camouflage target detection method described in any one of claims 1 to 6 through a processor.

Images