Launching vehicle state identification method and system combined with remote sensing image shallow cloud and mist removal

Abstract

The invention discloses a launching vehicle state identification method and system combined with remote sensing image shallow cloud and mist removal, and the method comprises the steps: obtaining original optical remote sensing image data, carrying out the preprocessing, and inputting the data into an end-to-end integrated network; shallow cloud and mist removal is carried out on the preprocessed image through a DehazeNet cloud removal network in the network; a cascaded SC-Yolo target detection network is used to carry out target feature extraction, feature integration, launch vehicle positioning detection and state classification identification on the image after shallow cloud and mist removal; supervised training is carried out by calculating respective loss functions of the cloud removal network and the target detection network, a launching vehicle positioning detection result is fed back to the cloud removal network, and the cloud removal weight is adjusted; inputting optical remote sensing image data to be detected into the trained end-to-end integrated network, and outputting a launching vehicle positioning detection result and a state classification result; the method improves the identification precision of the state of the launch vehicle under the interference of shallow cloud and mist, and reduces the false alarm rate.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and more particularly to a method and system for identifying the state of a launch vehicle in combination with shallow cloud and fog removal from remote sensing images. Background Art

[0002] Launch vehicle status identification is crucial in military applications. Rapidly and accurately obtaining the location and status information of the enemy launch vehicle plays a key role in strategic decision-making and tactical implementation. Remote sensing technology has become an important means of obtaining launch vehicle status information due to its wide detection range, high efficiency and real-time characteristics. Using remote sensing technology to obtain the launch vehicle status can provide necessary information for strategic decision-making, effectively shorten the decision-making cycle, and has great research value.

[0003] At present, popular deep learning methods have been able to achieve good recognition of the launch vehicle status in remote sensing images. However, they are also facing increasingly severe challenges, including huge changes in the target's visual appearance due to viewpoint changes, occlusion, cloud and fog interference, etc. Especially in foggy conditions, the visual features of the launch vehicle are easily obscured, making it difficult for traditional algorithms to accurately identify the launch vehicle status. These complex background interference factors make it easy for the target to be missed or misdetected, resulting in difficulty in further improving the accuracy of launch vehicle status recognition.

[0004] Therefore, how to improve the accuracy of transmitting vehicle status identification of remote sensing images under shallow cloud and fog interference is a problem that technical personnel in this field need to solve urgently. Summary of the invention

[0005] In view of this, the present invention provides a method and system for identifying the state of a launch vehicle combined with shallow cloud and fog removal of remote sensing images to solve the technical problems mentioned in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for identifying the state of a launch vehicle combined with shallow cloud and fog removal of remote sensing images comprises the following steps:

[0008] S101, obtaining original optical remote sensing image data;

[0009] S102, preprocessing the original optical remote sensing image data, and inputting it into an end-to-end integrated network in which a cloud removal network and a target detection network are sequentially connected;

[0010] S103, removing shallow clouds and fog from the pre-processed optical remote sensing image data through the DehazeNet cloud removal network in the end-to-end integrated network;

[0011] S104, using the cascaded SC-Yolo target detection network in the end-to-end integrated network to perform target feature extraction, feature integration, launch vehicle positioning detection, and launch vehicle status classification and identification on the image data after the shallow cloud and fog are removed;

[0012] S105, performing supervised training by calculating the loss functions of the DehazeNet cloud removal network and the cascaded SC-Yolo target detection network, and feeding back the launch vehicle positioning detection result of the cascaded SC-Yolo target detection network to the DehazeNet cloud removal network to adjust the cloud removal weight during cloud removal;

[0013] S106, obtaining the optical remote sensing image data to be tested and inputting it into the trained end-to-end integrated network, and outputting the launch vehicle positioning detection result and the launch vehicle status classification and identification result.

[0014] Preferably, in step S102, the specific content of preprocessing the original optical remote sensing image data is: linear quantization and image block cropping of the obtained original remote sensing image, and marking the target true value frame and the target launch vehicle status.

[0015] Preferably, the DehazeNet cloud removal network includes a feature extraction block, a feature enhancement block and an image reconstruction block;

[0016] The feature extraction block, feature enhancement block and image reconstruction block are connected in sequence, and the feature extraction block and image reconstruction block are skipped at the same feature map scale;

[0017] The input of the feature extraction block is the original image, which is passed through two convolution layers and a downsampling layer to obtain the deep feature map of the image; the feature enhancement block uses the Swin-transformer encoder and decoder to enhance the deep feature map to restore the texture features of the transmitting vehicle; the image reconstruction block reconstructs the image after feature enhancement through two convolution layers and an upsampling layer combined with the multi-scale features obtained by the jump connection to obtain a clear image without shallow clouds and fog.

[0018] Preferably, the cascaded SC-Yolo object detection network includes a backbone network, a neck network, an object detection head, and a state classification head connected in sequence;

[0019] The backbone network is used to gradually extract feature maps of different levels from the image data after removing shallow clouds and fog. The neck network is used to perform multi-scale feature fusion. The target detection head is used to detect and locate the launch vehicle through multi-scale fusion feature maps. The state classification head is used to identify the specific state of the launch vehicle based on the output results of the target detection head.

[0020] Preferably, the target detection head takes the multi-scale fusion feature map as input and the appearance features of the transmitting vehicle as the true value. It is first processed by two 3×3 convolutional layers, and then processed by a bounding box regression layer and a 1×1 convolutional layer respectively, and then outputs the bounding box position of the transmitting vehicle in the original image.

[0021] Preferably, the state classification head performs further state identification based on the output of the detection head, taking the launch vehicle slice as input and the different state information of the launch vehicle as the true value, and outputs the state identification result of the launch vehicle after passing through a 2×2 convolution layer, a flattening layer and two fully connected layers.

[0022] Preferably, in step S105, the feedback adjustment loss function for adjusting the cloud removal weight when removing fog is specifically:

[0023]

[0024] Among them, i is the i-th pixel point of the image, X is the image to be declouded, Y is the feedback image with the target detection frame, that is, the positioning detection result of the launch vehicle, and λ is the cloud removal weight coefficient of the area around the target detection frame.

[0025] A launch vehicle state identification system combined with remote sensing image shallow cloud removal is based on the launch vehicle state identification method combined with remote sensing image shallow cloud removal, including: a remote sensing image acquisition module, a preprocessing module, an end-to-end integrated network, a supervised training module and a cloud removal weight feedback adjustment module. The end-to-end integrated network includes a sequentially connected DehazeNet cloud removal network and a cascaded SC-Yolo target detection network.

[0026] A launch vehicle status identification device combined with remote sensing image shallow cloud and fog removal, comprising an image acquisition device, a memory and a processor;

[0027] An image acquisition device, used for acquiring optical remote sensing images to be detected;

[0028] The memory stores a computer program executable on the processor;

[0029] The processor is connected to the image acquisition device and the memory data. When the processor executes the computer program, it implements the launch vehicle status identification method combined with the shallow cloud removal of remote sensing images, and automatically identifies and classifies the launch vehicle status in the optical remote sensing image to be detected collected by the image acquisition device.

[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images.

[0031] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a method and system for identifying the status of a launch vehicle combined with shallow cloud and fog removal of remote sensing images, which optimizes the problem of difficult identification of the status of the launch vehicle caused by the weakening of the appearance characteristics of the launch vehicle under cloud and fog coverage conditions, improves the accuracy of launch vehicle status identification under shallow cloud and fog interference, and at the same time reduces the false alarm rate to a certain extent, thereby obtaining a high-precision launch vehicle status identification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0033] Figure 1 A schematic flow chart of a method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images provided by the present invention;

[0034] Figure 2 A schematic diagram of an end-to-end integrated network connection provided by the present invention;

[0035] Figure 3 A schematic diagram of the DehazeNet network model provided by the present invention;

[0036] Figure 4 Schematic diagram of the SC-Yolo network model provided by the present invention;

[0037] Figure 5 A schematic diagram of the comparison between the shallow cloud and fog removal and the original image after target detection provided by the present invention;

[0038] Figure 6 Schematic diagram of the target detection head and state classification head provided by the present invention;

[0039] Figure 7 A schematic diagram of the launch vehicle status identification result provided by the present invention;

[0040] Figure 8 This is a schematic diagram of the launch vehicle status identification device provided by the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The embodiment of the present invention discloses a method for identifying the state of a launch vehicle by combining remote sensing image shallow cloud removal. Figure 1 and Figure 2 As shown, the following steps are included:

[0043] S101, obtaining original optical remote sensing image data;

[0044] In this embodiment, the original optical remote sensing image data is an original 16-bit optical remote sensing image taken by a satellite;

[0045] S102, preprocessing the original optical remote sensing image data, and inputting it into an end-to-end integrated network in which a cloud removal network and a target detection network are sequentially connected;

[0046] S103, removing shallow clouds and fog from the pre-processed optical remote sensing image data through the DehazeNet cloud removal network in the end-to-end integrated network;

[0047] S104, using the cascaded SC-Yolo target detection network in the end-to-end integrated network to perform target feature extraction, feature integration, launch vehicle positioning detection, and launch vehicle status classification and identification on the image data after the shallow cloud and fog are removed;

[0048] S105, performing supervised training by calculating the loss functions of the DehazeNet cloud removal network and the cascaded SC-Yolo target detection network, and feeding back the launch vehicle positioning detection result of the cascaded SC-Yolo target detection network to the DehazeNet cloud removal network to adjust the cloud removal weight during cloud removal;

[0049] S106, obtaining the optical remote sensing image data to be tested and inputting it into the trained end-to-end integrated network, and outputting the launch vehicle positioning detection result and the launch vehicle status classification and identification result.

[0050] In order to further implement the above technical solution, in step S102, the specific contents of preprocessing the original optical remote sensing image data are: linear quantization and image block cropping of the obtained original remote sensing image, marking the target true value frame and the target launch vehicle state;

[0051] Specifically: First, the original 16-bit remote sensing image is linearly quantized at 2% to obtain 8-bit visual image data; secondly, the obtained image data is cropped into image blocks of 512×512 size for making model training samples; then, the target truth box and the target launch vehicle status are annotated based on the processed clear image data.

[0052] In order to further implement the above technical solution, Figure 3As shown in the figure, the DehazeNet network is improved based on U-net and the internal architecture is optimized to achieve high-fidelity shallow cloud removal of small target features in remote sensing images. Specifically: the DehazeNet cloud removal network includes feature extraction block, feature enhancement block and image reconstruction block;

[0053] The feature extraction block, feature enhancement block and image reconstruction block are connected in sequence, and the feature extraction block and image reconstruction block are skipped at the same feature map scale;

[0054] The input of the feature extraction block is the original image, which is passed through two convolution layers and a 2×2 downsampling layer to obtain the deep feature map of the image; the feature enhancement block uses the Swin-transformer encoder and decoder to enhance the deep feature map to restore the texture features of the launch vehicle; the image reconstruction block reconstructs the image after feature enhancement through two convolution layers and a 2×2 upsampling layer combined with the multi-scale features obtained by the jump connection to obtain a clear image without shallow fog;

[0055] In this example, since the launch vehicle occupies fewer pixels in the remote sensing image, the features are weaker after two downsamplings in the feature extraction block, and it is difficult to retain the texture features of the launch vehicle using convolution. Therefore, the feature enhancement module uses a swin-transformer encoder instead of the convolution layer to better restore the texture features of the launch vehicle after feature enhancement; at the same time, a jump connection is made between the feature extraction block and the image reconstruction block at the same feature map scale, so that the image reconstruction block can more effectively obtain multi-scale features.

[0056] In order to further implement the above technical solution, Figure 4 ,The cascaded SC-Yolo object detection network includes a backbone network, a neck network, an object detection head, and a state classification head connected in sequence;

[0057] The backbone network is used to gradually extract feature maps of different levels from the image data after removing shallow clouds and fog. The neck network is used to perform multi-scale feature fusion. The target detection head is used to detect and locate the launch vehicle through multi-scale fusion feature maps. The state classification head is used to identify the specific state of the launch vehicle based on the output results of the target detection head.

[0058] In this embodiment, the input image size of the backbone network is an image of 512×512. Through a series of convolutional layers, pooling layers and residual blocks, feature maps of different levels are gradually extracted. In this process, three feature maps are extracted from different levels of the backbone network, namely P2, P3, and P4. Among them, P2 is a high-resolution feature map extracted from the Conv2_x layer, which mainly captures the detail information of the image, P3 is a medium-to-high-resolution feature map extracted from the Conv3_x layer, which combines the detail and semantic information, and P4 is a low-resolution feature map extracted from the Conv5_x layer, which mainly captures the deep semantic information; next, the feature maps of different levels are input into the neck network (feature pyramid network) for multi-scale feature fusion, and the multi-scale fused feature maps will be used for subsequent detection and classification tasks.

[0059] In order to further implement the above technical solution, Figure 5 As shown in the figure, the target detection head takes the multi-scale fusion feature map as input and the appearance features of the launch vehicle as the true value. It is first processed by two 3×3 convolutional layers, and then processed by the bounding box regression layer and the 1×1 convolutional layer respectively, and then outputs the bounding box position of the launch vehicle in the original image.

[0060] In order to further implement the above technical solution, the SC-Yolo network is improved on the basis of the original Yolo network. A state classification head for state identification is added on the basis of the detection head to better realize the state classification between targets with similar features. Specifically: the state classification head performs further state identification based on the output of the detection head, with the launch vehicle slice as input and the different state information of the launch vehicle as the true value. After a 2×2 convolutional layer, a flattening layer and two fully connected layers, the state identification result of the launch vehicle is output. The launch vehicle state includes parking state, dynamic state and combat state.

[0061] like Figure 6 and Figure 7 As shown, Figure 6 (a) is the image block after the original 16-bit image is quantized and cropped. Figure 6 (b) is the image after shallow cloud removal and target detection; Figure 7 (a) in the figure shows that the launch vehicle is in combat state. Figure 7 (b) in the figure shows that the state identification result of the launch vehicle is the parking state.

[0062] In this embodiment, the DehazeNet cloud removal network and the cascaded SC-Yolo target detection network calculate their respective loss functions and perform gradient backpropagation. Specifically, the DehazeNet cloud removal network uses the declouded image as the true value to calculate the loss, and the SC-Yolo target detection network uses the launch vehicle state result as the true value to calculate the loss. Among them, the loss function of the target detection head is Yolo's regression loss, and the loss function of the state classification head is Yolo's classification loss. At the same time, the loss function of DehazeNet will also adjust the weight according to the image with target position information output by the SC-Yolo network after completing target detection.

[0063] In order to further implement the above technical solution, in step S105, the feedback adjustment loss function for adjusting the cloud removal weight when removing fog is specifically:

[0064]

[0065] Among them, i is the i-th pixel point of the image, X is the image to be declouded, Y is the feedback image with the target detection frame, that is, the positioning detection result of the launch vehicle, and λ is the cloud removal weight coefficient of the area around the target detection frame. In this embodiment, with the assistance of the feedback image with the detection result, the target position information of the cloud removal network can be provided, so that the weight coefficient is used to enhance the cloud removal strength around the target, so that the target features are effectively enhanced, thereby improving the performance of state recognition.

[0066] A launch vehicle state identification system combined with shallow cloud removal of remote sensing images is based on a launch vehicle state identification method combined with shallow cloud removal of remote sensing images, including: a remote sensing image acquisition module, a preprocessing module, an end-to-end integrated network, a supervised training module and a cloud removal weight feedback adjustment module. The end-to-end integrated network includes a sequentially connected DehazeNet cloud removal network and a cascaded SC-Yolo target detection network.

[0067] A launch vehicle state identification device 303 combined with remote sensing image shallow cloud removal, such as Figure 8 As shown, it includes an image acquisition device 301, a memory 302 and a processor 303;

[0068] An image acquisition device 301 is used to acquire an optical remote sensing image to be detected;

[0069] The memory 302 stores a computer program that can be run on the processor 303;

[0070] The processor 303 is data-connected to the image acquisition device 301 and the memory 302. When the processor 303 executes the computer program, it implements a method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images, and automatically identifies and classifies the state of the launch vehicle in the optical remote sensing image to be detected collected by the image acquisition device 301.

[0071] In actual applications, the memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules; the memory 302 may include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required by at least one function, and the data storage area can store data created by the processor 302, etc.; in addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some aspects, the memory 302 may optionally include a memory remotely arranged relative to the processor 303, and these remote memories may be connected to the processor 303 via a network, and the network includes but is not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0072] The processor 303 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above chips.

[0073] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images.

[0074] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0075] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images, characterized in that: The following steps are involved: S101, obtaining original optical remote sensing image data; S102, preprocessing the original optical remote sensing image data, and inputting it into an end-to-end integrated network in which a cloud removal network and a target detection network are sequentially connected; S103, removing shallow clouds and fog from the pre-processed optical remote sensing image data through the DehazeNet cloud removal network in the end-to-end integrated network; S104, using the cascaded SC-Yolo target detection network in the end-to-end integrated network to perform target feature extraction, feature integration, launch vehicle positioning detection, and launch vehicle status classification and identification on the image data after the shallow cloud and fog are removed; S105, performing supervised training by calculating the loss functions of the DehazeNet cloud removal network and the cascaded SC-Yolo target detection network, and feeding back the launch vehicle positioning detection result of the cascaded SC-Yolo target detection network to the DehazeNet cloud removal network to adjust the cloud removal weight during cloud removal; S106, obtaining the optical remote sensing image data to be tested and inputting it into the trained end-to-end integrated network, and outputting the launch vehicle positioning detection result and the launch vehicle status classification and identification result.

2. The method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images according to claim 1 is characterized in that: In step S102, the specific content of preprocessing the original optical remote sensing image data is: linear quantization and image block cropping are performed on the obtained original remote sensing image, and the target true value frame and the target launch vehicle status are marked.

3. The method for identifying the state of a launch vehicle combined with shallow cloud and fog removal of remote sensing images according to claim 1 is characterized in that: The DehazeNet cloud removal network includes feature extraction block, feature enhancement block and image reconstruction block; The feature extraction block, feature enhancement block and image reconstruction block are connected in sequence, and the feature extraction block and image reconstruction block are skipped at the same feature map scale; The input of the feature extraction block is the original image, which is passed through two convolution layers and a downsampling layer to obtain the deep feature map of the image; the feature enhancement block uses the Swin-transformer encoder and decoder to enhance the deep feature map to restore the texture features of the transmitting vehicle; the image reconstruction block reconstructs the image after feature enhancement through two convolution layers and an upsampling layer combined with the multi-scale features obtained by the jump connection to obtain a clear image without shallow clouds and fog.

4. The method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images according to claim 1 is characterized in that: The cascaded SC-Yolo object detection network includes a backbone network, a neck network, an object detection head, and a state classification head connected in sequence; The backbone network is used to gradually extract feature maps of different levels from the image data after removing shallow clouds and fog. The neck network is used to perform multi-scale feature fusion. The target detection head is used to detect and locate the launch vehicle through multi-scale fusion feature maps. The state classification head is used to identify the specific state of the launch vehicle based on the output results of the target detection head.

5. The method for identifying the state of a launch vehicle combined with shallow cloud and fog removal of remote sensing images according to claim 4 is characterized in that: The target detection head takes the multi-scale fusion feature map as input and the appearance features of the launch vehicle as the true value. It is first processed by two 3×3 convolutional layers, and then processed by the bounding box regression layer and the 1×1 convolutional layer respectively, and then outputs the bounding box position of the launch vehicle in the original image.

6. The method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images according to claim 4 is characterized in that: The state classification head performs further state identification based on the output of the detection head, taking the launch vehicle slice as input and the different state information of the launch vehicle as the true value. After a 2×2 convolutional layer, a flattening layer and two fully connected layers, the state identification result of the launch vehicle is output.

7. The method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images according to claim 1 is characterized in that: In step S105, the feedback adjustment loss function for adjusting the cloud removal weight during cloud removal is specifically: Among them, i is the i-th pixel point of the image, X is the image to be declouded, Y is the feedback image with the target detection frame, that is, the positioning detection result of the launch vehicle, and λ is the cloud removal weight coefficient of the area around the target detection frame.

8. A launch vehicle status identification system combined with remote sensing image shallow cloud removal, characterized in that: A method for identifying the state of a launch vehicle combined with shallow cloud removal from remote sensing images based on any one of claims 1 to 7, comprising: a remote sensing image acquisition module, a preprocessing module, an end-to-end integrated network, a supervised training module, and a cloud removal weight feedback adjustment module, wherein the end-to-end integrated network comprises a sequentially connected DehazeNet cloud removal network and a cascaded SC-Yolo target detection network.

9. A launch vehicle status identification device combined with remote sensing image shallow cloud removal, characterized in that: including an image acquisition device, a memory and a processor; An image acquisition device, used for acquiring optical remote sensing images to be detected; The memory stores a computer program executable on the processor; The processor is connected to the image acquisition device and the memory data. When the processor executes the computer program, it implements a method for identifying the state of a launch vehicle combined with shallow cloud removal from remote sensing images as described in any one of claims 1 to 7, and automatically identifies and classifies the state of the launch vehicle in the optical remote sensing image to be detected collected by the image acquisition device.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a method for identifying the state of a launch vehicle combined with shallow cloud and fog removal from remote sensing images as described in any one of claims 1 to 7.

Images