Multi-spectral camouflage target recognition method, device, electronic device and storage medium

Abstract

The present application relates to the field of image technology, and discloses a multi-spectral camouflage target recognition method, device, electronic device and storage medium; the method includes: obtaining a visible light spectrum polarization image set and an infrared spectrum polarization image set of a target scene; performing feature fusion on the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image; performing feature extraction on the first feature image based on a target feature extraction method, and outputting a second feature image; wherein the target feature extraction method includes at least one of the following: a lightweight convolution method, an improved feature pyramid model; performing context feature fusion on the second feature image, and detecting the camouflage target obscured in the second feature image based on an occlusion perception network, and outputting a target image, wherein the target image is annotated with the camouflage target. The present application can improve recognition speed, detection precision and recognition accuracy, and realize rapid recognition of camouflage targets under complex backgrounds.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a multi-spectral camouflage target recognition method, device, electronic device and storage medium. Background Art

[0002] The purpose of multispectral target recognition is to identify each unique spectral feature in a multispectral image as a target or background. The current target recognition algorithms are divided into two types: traditional algorithms and deep learning algorithms. Traditional algorithms are based on prior information and use manual feature extraction to detect targets. Considering that real scenes are much more complex, with huge changes in target size, shape, and clutter background, it is difficult to use hand-crafted features and fixed hyperparameters to handle such changes. Deep learning-based methods learn trainable features in a data-driven manner, but they are difficult to identify ground camouflaged targets in complex environments, and have low recognition accuracy and poor real-time performance. Summary of the invention

[0003] In view of the above situation, the embodiments of the present application provide a multi-spectral camouflaged target recognition method, device, electronic device and storage medium, which aim to solve the above problems or at least partially solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a multi-spectral camouflaged target recognition method, the method comprising: acquiring a visible light spectrum polarization image set and an infrared spectrum polarization image set of a target scene; performing feature fusion on the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image; performing feature extraction on the first feature image based on a target feature extraction method, and outputting a second feature image; wherein the target feature extraction method comprises at least one of the following: a lightweight convolution method, an improved feature pyramid model; performing context feature fusion on the second feature image, and detecting the occluded camouflaged target in the second feature image based on an occlusion perception network, and outputting a target image, wherein the target image is marked with the camouflaged target.

[0005] In the second aspect, an embodiment of the present application also provides a multi-spectral camouflaged target identification device, which includes: an acquisition module, used to acquire a visible light multi-spectral polarization image set and an infrared multi-spectral polarization image set of a target scene; a feature fusion module, used to perform feature fusion on the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image; a feature extraction module, used to perform feature extraction on the first feature image based on a target feature extraction method, and output a second feature image; wherein the target feature extraction method includes at least one of the following: a lightweight convolution method, an improved feature pyramid model; the feature fusion module is also used to perform context feature fusion on the second feature image, and detect the camouflaged target obscured in the second feature image based on an occlusion perception network, and output a target image, in which the camouflaged target is marked.

[0006] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the steps described in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the steps described in the second aspect above.

[0008] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: by fusing the visible light spectrum polarization image set and the infrared spectrum polarization image set of the target scene, the fused first feature image has richer and more comprehensive feature information; then, by adopting a lightweight convolution method and an improved feature pyramid model, the first feature image is feature extracted to obtain a second feature image, and the feature extraction speed is significantly improved by using a lightweight processing method, and the improved feature pyramid model is used to achieve multi-scale feature extraction and improve the feature extraction effect of small targets; finally, the second feature image is subjected to context feature fusion, and the sensitivity of small and medium features can be improved by combining context features, and more comprehensive image features can be obtained. At the same time, the occluded camouflaged target in the second feature image is detected by the occlusion perception network, and the camouflaged occluded target under complex background can be quickly identified. It can be seen that the technical solution provided in the embodiments of the present application can adapt to complex backgrounds, has good robustness to environmental changes, has good feature expression capabilities, and has good detection accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0010] Figure 1 A schematic diagram of the process of a multi-spectral camouflaged target recognition method provided in an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram of the process of the feature fusion method provided in the embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram of a feature fusion process provided by an embodiment of the present application is shown;

[0013] Figure 4 A schematic diagram of the composition of an hourglass network provided in an embodiment of the present application is shown;

[0014] Figure 5 A schematic flow chart of a method for feature extraction based on an improved feature pyramid model provided in an embodiment of the present application is shown;

[0015] Figure 6 A schematic diagram of the structure of a feature pyramid model provided in an embodiment of the present application is shown;

[0016] Figure 7 A schematic structural diagram of an improved feature pyramid model provided by an embodiment of the present application is shown;

[0017] Figure 8 A schematic diagram of a process flow of a feature extraction method provided in an embodiment of the present application is shown;

[0018] Fig. 9 A schematic diagram of the feature extraction process provided by an embodiment of the present application is shown;

[0019] Fig.10 A schematic diagram of a multi-spectral camouflage target recognition method provided by another embodiment of the present application is shown;

[0020] Fig.11 The structure diagram of the multi-spectral camouflage target recognition device provided by the embodiment of the present application is shown;

[0021] Fig.12 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such use is interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants are to be interpreted as open-ended terms meaning "including but not limited to".

[0024] The problem of multispectral target detection is essentially a binary classification problem to determine whether a pixel exists. The purpose is to identify it as a target or background based on the unique spectral characteristics of each pixel in the multispectral image. Infrared small target detection refers to the detection and location of small and weak energy objects in infrared images. It has a wide range of applications, such as reconnaissance, security monitoring and other fields. Small targets in infrared images are usually interfered by complex backgrounds, low contrast and noise, making it difficult to distinguish the target from the background.

[0025] At present, target detection algorithms are divided into two types: traditional algorithms and deep learning algorithms. Traditional multispectral target detection algorithms are mainly based on spectral information and simple related information of adjacent pixels to complete detection tasks, such as matched filters. Adaptive matched filters and adaptive coherence / cosine estimators. The orthogonal subspace projection algorithm based on the subspace model suppresses the interference of the background spectrum by projecting the spectrum of the pixel to be tested into the orthogonal subspace of the background spectrum. Most of these methods rely on strong prior information and use manual feature extraction for target detection, which has poor robustness to complex backgrounds and targets with complex shapes.

[0026] Deep learning algorithms use neural networks to train data sets, extract features from images, and use classifiers to determine whether an area is a target. Due to the good adaptability and intelligence of deep learning algorithms, a lot of progress has been made in the field of multispectral target detection in recent years. The commonly used deep learning method is the target detection algorithm based on convolutional neural network (CNN). The algorithm extracts features from images by continuously iteratively optimizing the convolution kernel, and uses classifiers to identify and locate targets. Although the performance of the network has been continuously improved recently, the feature loss of small targets in the deep layer still exists, the target is affected by the environment and is in an occluded state, the feature extraction of camouflaged targets is difficult, the recognition accuracy is low, and the real-time performance is poor. These problems ultimately lead to poor robustness to significant scene changes, and there is still room for further improvement.

[0027] Based on this, the embodiment of the present application provides a multi-spectral camouflage target recognition method, which fuses the spectral features and image features in the multi-spectral image to achieve complementary feature information, improve the distinction between the target and the background, and achieve rapid recognition of the camouflage target under a complex background. At the same time, by adopting the target feature extraction method and feature fusion method, the scheme can adapt to complex backgrounds, has good robustness to environmental changes, has good feature expression ability, and good detection accuracy and speed.

[0028] The present application is described in detail below through specific embodiments.

[0029] Figure 1 The schematic diagram of the process of the multi-spectral camouflage target recognition method provided by the embodiment of the present application is shown. Figure 1 It can be seen that the present application at least includes steps S101 to S104:

[0030] Step S101: Acquire a visible light spectral polarization image set and an infrared spectral polarization image set of a target scene.

[0031] Among them, the target scene includes camouflaged targets, which refer to targets that are concealed by adopting engineering technical measures and utilizing terrain and ground objects.

[0032] The visible light spectral polarization image set includes visible light images of multiple spectral bands, and the infrared spectral polarization image set includes infrared images of multiple spectral bands. In one embodiment, the visible light spectral polarization image set includes visible light images of 7 different spectral bands, and the infrared spectral polarization image set includes infrared images of 7 different spectral bands.

[0033] Step S102: feature fusion of the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image.

[0034] Since infrared images can detect targets even when vision is obstructed at night, they can be used all day long, and their spectral information can be used to identify camouflaged targets; visible light images have the advantage of extracting more information than general visible light images, and can also complement the target feature information detected by infrared multispectral images in extreme cases where the temperature difference is not obvious, so as to extract more target feature information. Therefore, feature fusion of visible light spectral polarization image set and infrared spectral image set can obtain richer image features.

[0035] Step S103: extracting features from the first feature image based on the target feature extraction method, and outputting a second feature image.

[0036] Among them, the target feature extraction method includes at least one of the following: a lightweight convolution method and an improved feature pyramid model.

[0037] Step S104: performing context feature fusion on the second feature image, detecting the camouflaged target that is occluded in the second feature image based on the occlusion-aware network, and outputting the target image.

[0038] Among them, the target image is marked with a camouflaged target.

[0039] In some embodiments, the occlusion-aware network consists of an occlusion sensor and a corresponding evaluation network. During the training phase, the occlusion-aware network is inserted into a backbone network that performs feature extraction and feature fusion to learn object regions and occluded regions. A robust object detector is constructed with occlusions learned in the test phase. The occlusion-aware network evaluates the predicted occluded region map to effectively detect objects with occlusion overlap.

[0040] It is worth noting that the occlusion-aware network in the embodiment of the present application does not necessarily detect the second feature image only after the context-aware feature fusion model performs feature fusion on the second feature image. The occlusion-aware network can be inserted into any step based on actual applications.

[0041] from Figure 1It can be seen from the method shown that the present application fuses the visible light spectrum polarization image set and the infrared spectrum polarization image set of the target scene so that the fused first feature image has richer and more comprehensive feature information; then, the first feature image is extracted by adopting a lightweight convolution method and an improved feature pyramid model to obtain a second feature image, and the speed of feature extraction is significantly improved by using a lightweight processing method, and the improved feature pyramid model is used to achieve multi-scale feature extraction and improve the feature extraction effect of small targets; finally, the second feature image is fused with context features, and the sensitivity of small and medium features can be improved by combining context features, and more comprehensive image features can be obtained. At the same time, the occluded camouflaged target in the second feature image is detected by the occlusion perception network, and the camouflaged occluded target is quickly identified under complex backgrounds. It can be seen that the technical solution provided by the embodiment of the present application can adapt to complex backgrounds, has good robustness to environmental changes, has good feature expression capabilities, and good detection accuracy and speed.

[0042] Figure 2 A schematic diagram of a feature fusion method according to an embodiment of the present application is shown. Figure 2 It can be seen that the feature fusion method of this embodiment includes the following steps S201 to S203:

[0043] Step S201: Fusing the spectral features and image features of the visible light spectrum image set to generate a visible light spectrum feature image.

[0044] In one implementation, the spectral features include spectral space, spectral line information, and polarization information, and the image features include spatial geometric information and continuous frame information.

[0045] Step S202: Fusing the spectral features and image features of the infrared spectral image set to generate an infrared spectral feature image.

[0046] In one implementation, the spectral features include spectral space, spectral line information, and polarization information, and the image features include spatial geometric information and continuous frame information.

[0047] In the embodiment of the present application, the spectral features and the image features are organically integrated to achieve complementarity of feature information and improve the distinction between the target and the background.

[0048] Step S203: performing feature fusion on the visible spectrum feature image and the infrared spectrum feature image based on a feature threshold mechanism to determine a first feature image.

[0049] The feature threshold mechanism is used to adjust the feature ratios of the image features of the visible spectrum feature image and the image features of the infrared spectrum feature image in the first feature image under different lighting conditions.

[0050] For example, in a daytime scene, when the light is strong, the image features of the visible spectrum feature image occupy a feature ratio of 30% in the first feature image, and the image features of the infrared spectrum feature image occupy a feature ratio of 70% in the first feature image; for another example, in a night scene, when the light is weak, the image features of the visible spectrum feature image occupy a feature ratio of 70% in the first feature image, and the image features of the infrared spectrum feature image occupy a feature ratio of 30% in the first feature image; for another example, in a daytime scene, for sunny days and cloudy days, the image features of the visible spectrum feature image and the image features of the infrared spectrum feature image in the first feature image also occupy different feature ratios.

[0051] In some embodiments, the feature ratios of the image features of the visible spectrum feature image and the image features of the infrared spectrum feature image under different lighting conditions in the first feature image can be preset and adjusted based on different lighting conditions during actual use.

[0052] Specifically, the visible spectrum feature and infrared spectrum feature images are input into a shared feature extractor to obtain their respective spectrum image features, and the infrared spectrum features and visible spectrum features are further encoded into key-value mapping pairs through a deep encoder.

[0053] like Figure 3 As shown in the figure, after obtaining the key / value mapping of the visible light spectrum feature, the similarity and difference of the pixel features are calculated in a non-local manner, and then softmax normalization is performed. Further, feature weights are assigned to the visible light spectrum features and infrared spectrum features based on the feature threshold mechanism, and the value of the visible light multi-spectral feature is retrieved by weighted summation of the generated feature weights, which is connected to the value map corresponding to the infrared multi-spectral feature. Finally, the N output results are summed to obtain the final result.

[0054] In an embodiment of the present application, for different application scenarios, a feature threshold mechanism is set to adaptively adjust the feature ratio of visible light features and infrared features in the fused first feature image to obtain feature identification that is insensitive to lighting, thereby reducing the missed detection rate of camouflaged targets in poor lighting conditions.

[0055] In some embodiments of the present application, in the above method, when the target feature extraction method is a lightweight convolution method, feature extraction is performed on the first feature image by a lightweight convolution method to generate a second feature image.

[0056] Among them, lightweight convolution methods include group convolution, channel shuffle and depth convolution.

[0057] In some embodiments, the network structure backbone used by the lightweight convolution method is an hourglass network, and its basic network unit is as follows: Figure 4As shown. First, the first feature image is input into the hourglass network, and lightweight convolution methods such as group convolution, channel shuffling, and deep convolution are used to reduce the amount of computation in the convolution process. Among them, the channel shuffling operation divides the input features of the first feature image into two parts, one of which is truly deeply separable. The calculation result is merged with the other part through channels, and finally deep convolution and group convolution operations are performed to complete the information exchange.

[0058] In the embodiment of the present application, in order to address the problems of a large amount of multi-spectral-polarization space and multi-scale feature fusion information, a large number of devices on the airborne platform, and limited GPU computing power, the embodiment of the present application integrates the multi-spectral fusion feature information through a lightweight module based on an hourglass network, improves the network speed, and realizes rapid anti-camouflage identification of camouflaged threat targets in complex ground environments.

[0059] In other embodiments of the present application, in the above method, when the target extraction method is a lightweight convolution method and an improved feature pyramid model, a third feature image is generated by performing feature extraction on the first feature image based on the lightweight convolution method, and then feature extraction is performed on the third feature image based on the improved feature pyramid model to output a second feature image.

[0060] In an embodiment of the present application, after the first feature image is subjected to feature extraction by a lightweight convolution method, in order to better obtain the features of small and medium-sized targets, an improved feature pyramid model is adopted to perform feature extraction again to achieve better detection effect for small and medium-sized targets.

[0061] Figure 5 A schematic diagram of a method for extracting features based on an improved feature pyramid model according to an embodiment of the present application is shown. Figure 5 It can be seen that the feature extraction method of this embodiment includes the following steps S501-S504:

[0062] Step S501: extracting a plurality of image feature maps with successively decreasing scales from the third feature image in a bottom-up manner to generate a first image feature pyramid.

[0063] Step S502: performing upsampling processing on the image feature map pyramid from top to bottom to generate a second image feature pyramid.

[0064] Step S503: extracting features from the second image feature pyramid in a bottom-up manner, and outputting a plurality of feature images.

[0065] The plurality of characteristic images include position information and image characteristic information of the camouflaged target;

[0066] Step S504: based on the confidences of the multiple feature images and a preset confidence, a second feature image is screened out from the multiple feature images.

[0067] In one implementation, the second feature image is a feature image having a confidence level greater than a preset confidence level among the multiple feature images.

[0068] In the embodiment of the present application, the third feature image obtained by lightweight processing is sent to the improved feature pyramid model for multi-scale feature extraction. The current feature pyramid model (FPN) structure is mainly divided into a bottom-up path and a top-down path, such as Figure 6 As shown in the figure. From bottom to top, it is the process of forward feature extraction of the deep convolutional network, and from top to bottom, it is the process of upsampling the feature map of the last convolutional layer. The horizontal connection between the two paths is the process of fusing the deep convolutional layer features with the shallow convolutional features, thus constructing a deeper feature pyramid that integrates multiple layers of feature information. However, when a network is deep, feature extraction through FPN will lose a lot of detailed feature information.

[0069] Therefore, the embodiment of the present application shortens the information path based on FPN and enhances the feature pyramid with accurate positioning information at a low level, creates a bottom-up path enhancement, and transmits semantic information from low dimensions to high dimensions again. The improved feature pyramid model structure is as follows: Figure 7 As shown in the figure, the improved feature pyramid model can simultaneously obtain the location information and feature information of small targets, which greatly improves the detection effect of small targets.

[0070] Figure 8 A schematic diagram of a feature extraction method according to another embodiment of the present application is shown. Figure 8 It can be seen that the feature extraction method of this embodiment includes the following steps S801-S802:

[0071] Step S801: Obtain image features of a second feature image at different resolutions.

[0072] In some embodiments, a larger resolution tends to focus on local detailed context information, especially for smaller objects, while a smaller resolution can capture overall information, which is beneficial for the recognition of larger objects. In one implementation, the embodiment of the present application selects three different resolutions of 4, 8, and 12 in different situations, and performs parallel pooling operations to obtain a more comprehensive feature representation to alleviate the scale change problem.

[0073] Step S802: extracting context features from image features at different resolutions, and fusing image features at different resolutions based on an adaptive mechanism.

[0074] Since image features at different resolutions contain different levels of semantic information, in order to effectively aggregate features, the embodiment of the present application also adopts an adaptive mechanism to adaptively fuse the aggregation results. Fig. 9 As shown in Figure 1, an attention branch is added to each feature consisting of two blocks, the first block contains a global average pooling, and the second block contains two consecutive fc layers. Then a softmax regularization is added to the final generated weights to balance the contribution of each feature, and finally the weighted sum of the three features is output to represent the aggregated feature.

[0075] In an embodiment of the present application, in order to address the problems that the number of samples available on the ground in a complex ground environment leads to amplified feature scale changes in the samples, weakened generalization ability of the model for new classes, and low detection accuracy, by extracting and fusing contextual features of images of different resolutions, comprehensive features of large, medium and small targets can be obtained.

[0076] In order to fully illustrate the multi-spectral camouflage target recognition method provided by the embodiment of the present application, the following is combined with Fig.10 The flowchart shown is used for explanation:

[0077] The images in the visible light multi-spectral polarization image set are feature fused to obtain a visible light spectrum image; the images in the infrared multi-spectral polarization image set are feature fused to obtain an infrared spectrum image. Then, the visible light spectrum image and the infrared spectrum image are feature fused based on the feature threshold mechanism, and the feature ratios of the visible light spectrum image and the infrared spectrum image in the fused image are adjusted under different lighting conditions, and the first feature image is obtained. The first feature image is further input into the lightweight backbone network (hourglass network) to perform lightweight processing on the first feature image, and the third feature image is output. The third feature image is further input into the improved feature pyramid model to improve the detection accuracy of small and medium-sized targets, and the second feature image is output. Finally, the recognition of camouflaged targets is achieved through context feature fusion and occlusion perception network.

[0078] In some embodiments of the present application, a multi-spectral disguised target recognition device is provided, and the multi-spectral disguised target recognition device corresponds one-to-one to the multi-spectral disguised target recognition method in the above embodiment. Fig.11 As shown, the multi-spectral camouflage target recognition device includes: an acquisition module 1101, a feature fusion module 1102 and a feature extraction module 1103. The detailed description of each functional module is as follows:

[0079] An acquisition module 1101 is used to acquire a visible light multi-spectral polarization image set and an infrared multi-spectral polarization image set of a target scene;

[0080] A feature fusion module 1102 is used to perform feature fusion on the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image;

[0081] The feature extraction module 1103 is used to extract features from the first feature image based on a target feature extraction method and output a second feature image; wherein the target feature extraction method includes at least one of the following: a lightweight convolution method and an improved feature pyramid model;

[0082] The feature fusion module 1102 is further used to perform context feature fusion on the second feature image, detect the occluded camouflaged target in the second feature image based on the occlusion-aware network, and output a target image in which the camouflaged target is marked.

[0083] In some embodiments of the present application, in the above-mentioned device, the feature fusion module 1102 is specifically used to feature fuse the spectral features and image features of the visible light spectrum image set to generate a visible spectrum feature image; feature fuse the spectral features and image features of the infrared spectrum image set to generate an infrared spectrum feature image; feature fuse the visible spectrum feature image and the infrared spectrum feature image based on a feature threshold mechanism to determine the first feature image; wherein the feature threshold mechanism is used to adjust the feature ratios of the image features of the visible spectrum feature image and the image features of the infrared spectrum feature image in the first feature image under different lighting conditions.

[0084] In some embodiments of the present application, in the above-mentioned device, the feature extraction module 1103 is specifically used to extract features of the first feature image based on the lightweight convolution method to generate a second feature image, and the lightweight convolution method includes group convolution, channel shuffling and depth convolution.

[0085] In some embodiments of the present application, in the above-mentioned device, the feature extraction module 1103 is specifically used to perform feature extraction on the first feature image based on the lightweight convolution method to generate a third feature image, and the lightweight convolution method includes group convolution, channel shuffling and depth convolution; perform feature extraction on the third feature image based on the improved feature pyramid model, and output the second feature image.

[0086] In some embodiments of the present application, in the above-mentioned device, the feature extraction module 1103 is specifically used to input the first feature image into the hourglass network model, perform group convolution processing, and obtain the first image feature; perform channel shuffling, depth convolution and group convolution processing on the first sub-image feature of the first image feature to generate a second image feature; and fuse the second sub-image feature of the first image feature with the second image feature to generate a second feature image.

[0087] In some embodiments of the present application, in the above-mentioned device, the feature extraction module 1103 is specifically used to extract multiple image feature maps with successively decreasing scales in a bottom-up manner to generate a first image feature pyramid; perform top-down upsampling processing on the image feature map pyramid to generate a second image feature pyramid; extract features from the second image feature pyramid in a bottom-up manner to output multiple feature images, wherein the multiple feature images include location information and image feature information of the camouflaged target; based on the confidence of the multiple feature images and a preset confidence, screen out the second feature image from the multiple feature images.

[0088] In some embodiments of the present application, in the above-mentioned device, the feature fusion module 1102 is specifically used to obtain image features of the second feature image at different resolutions; perform feature extraction on the image features at different resolutions based on the context-aware feature fusion model, and aggregate the image features at different resolutions based on an adaptive mechanism.

[0089] In some embodiments of the present application, in the above-mentioned device, the second convolutional neural network is formed based on an attention mechanism.

[0090] It should be noted that any of the above-mentioned multi-spectral camouflaged target recognition devices can implement the above-mentioned multi-spectral camouflaged target recognition method one-to-one, which will not be repeated here.

[0091] Fig.12 FIG. 1 is a schematic diagram showing the structure of an electronic device provided in an embodiment of the present application. Fig.12 As shown, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0092] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0093] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0094] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a multi-spectral camouflage target recognition device at the logical level. The processor executes the program stored in the memory and is specifically used to execute the above method.

[0095] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0096] The electronic device can execute the multi-spectral camouflage target recognition method provided by multiple embodiments of the present application, and realize the multi-spectral camouflage target recognition device in Fig.11 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0097] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple application programs, the electronic device can execute the multi-spectral camouflage target recognition method provided by multiple embodiments of the present application.

[0098] Those skilled in the art will appreciate that the embodiments of the present application may provide methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0103] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0104] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0105] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0107] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-spectral camouflage target recognition method, characterized in that: The method comprises: Acquire a visible light spectral polarization image set and an infrared spectral polarization image set of the target scene; Performing feature fusion on the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image; Performing feature extraction on the first feature image based on a target feature extraction method, and outputting a second feature image; wherein the target feature extraction method includes at least one of the following: a lightweight convolution method, an improved feature pyramid model; the lightweight convolution method includes group convolution, channel shuffling, and depth convolution; the improved feature pyramid model includes a bottom-up feature extraction process, a top-down feature extraction process, and a bottom-up feature extraction process; Performing context feature fusion on the second feature image, detecting the camouflaged target blocked in the second feature image based on an occlusion-aware network, and outputting a target image, wherein the camouflaged target is marked in the target image; The method of performing feature fusion on the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image includes: Fusing the spectral features and image features of the visible light spectrum polarization image set to generate a visible light spectrum feature image; Fusing the spectral features and image features of the infrared spectral polarization image set to generate an infrared spectral feature image; Performing feature fusion on the visible spectrum feature image and the infrared spectrum feature image based on a feature threshold mechanism to determine the first feature image; Wherein, the feature threshold mechanism is used to adjust the feature ratio of the image features of the visible spectrum feature image and the image features of the infrared spectrum feature image in the first feature image under different lighting conditions; When the target feature extraction method includes a lightweight convolution method and an improved feature pyramid model, extracting features from the first feature image based on the target feature extraction method and outputting a second feature image includes: Extracting features from the first feature image based on the lightweight convolution method to generate a third feature image; In a bottom-up manner, a plurality of image feature maps with successively decreasing scales are extracted from the third feature image to generate a first image feature pyramid; Performing a top-down upsampling process on the first image feature map pyramid to generate a second image feature pyramid; Extracting features from the second image feature pyramid in a bottom-up manner, and outputting a plurality of feature images, wherein the plurality of feature images include position information and image feature information of the camouflaged target; Based on the confidences of the multiple feature images and a preset confidence, the second feature image is screened out from the multiple feature images.

2. The method according to claim 1, characterized in that When the target feature extraction method includes a lightweight convolution method, extracting features from the first feature image based on the target feature extraction method and outputting a second feature image includes: Based on the lightweight convolution method, features are extracted from the first feature image to generate a second feature image. The lightweight convolution method includes group convolution, channel shuffling and depth convolution.

3. The method according to claim 2, characterized in that The step of extracting features from the first feature image based on the lightweight convolution method to generate a second feature image includes: Inputting the first feature image into the hourglass network model, performing group convolution processing, and obtaining the first image feature; Perform channel shuffling, depth convolution, and group convolution processing on the first sub-image feature of the first image feature to generate a second image feature; The second sub-image feature of the first image feature and the second image feature are fused to generate a second feature image.

4. The method according to claim 1, characterized in that: The performing context feature fusion on the second feature image includes: Acquire image features of the second feature image at different resolutions; Context features are extracted from the image features at different resolutions, and the image features at different resolutions are fused based on an adaptive mechanism.

5. A multi-spectral camouflage target recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a visible light spectrum polarization image set and an infrared spectrum polarization image set of a target scene; A feature fusion module, used for performing feature fusion on the visible light spectrum polarization image set and the infrared spectrum polarization image set to generate a first feature image; A feature extraction module, configured to extract features from the first feature image based on a target feature extraction method and output a second feature image; wherein the target feature extraction method includes at least one of the following: a lightweight convolution method and an improved feature pyramid model; the lightweight convolution method includes group convolution, channel shuffling, and depth convolution; the improved feature pyramid model includes a bottom-up feature extraction process, a top-down feature extraction process, and a bottom-up feature extraction process; The feature fusion module is further used to perform context feature fusion on the second feature image, detect the camouflaged target blocked in the second feature image based on the occlusion perception network, and output a target image, wherein the camouflaged target is marked in the target image; A feature fusion module is specifically used to perform feature fusion on the spectral features and image features of the visible light spectrum polarization image set to generate a visible spectrum feature image; Fusing the spectral features and image features of the infrared spectral polarization image set to generate an infrared spectral feature image; Performing feature fusion on the visible spectrum feature image and the infrared spectrum feature image based on a feature threshold mechanism to determine the first feature image; Wherein, the feature threshold mechanism is used to adjust the feature ratio of the image features of the visible spectrum feature image and the image features of the infrared spectrum feature image in the first feature image under different lighting conditions; When the target feature extraction method includes a lightweight convolution method and an improved feature pyramid model, the feature extraction module is specifically used to perform feature extraction on the first feature image based on the lightweight convolution method to generate a third feature image; extract multiple image feature maps with successively decreasing scales from the third feature image in a bottom-up manner to generate a first image feature pyramid; perform top-down upsampling processing on the first image feature map pyramid to generate a second image feature pyramid; extract features from the second image feature pyramid in a bottom-up manner to output multiple feature images, wherein the multiple feature images include location information and image feature information of the camouflaged target; and screen out the second feature image from the multiple feature images based on the confidence of the multiple feature images and a preset confidence.

6. An electronic device, characterized in that: include: processor; as well as A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the multi-spectral camouflaged target recognition method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, which, when executed by an electronic device including multiple application programs, enable the electronic device to perform the steps of the multi-spectral camouflaged target recognition method as described in any one of claims 1-4.

Images