Small target detection method and device based on panchromatic and hyperspectral fusion images

Abstract

The present invention discloses a method and device for detecting small targets based on panchromatic and hyperspectral fusion images, belonging to the field of image processing technology. The method comprises: obtaining panchromatic and hyperspectral remote sensing images with high matching accuracy; sequentially connecting an image fusion network and a target detection network into a hierarchical supervision integrated network; performing calculations on the image fusion network and the target detection network respectively through their own loss functions, and at the same time calculating the mutual learning loss based on the output result of the target detection network, and feeding it back to the image fusion network as a loss function for supervised training of the fusion network; then inputting the panchromatic and hyperspectral remote sensing images into the fusion network, performing panchromatic and hyperspectral image fusion, and then inputting the fused image into the target detection network, extracting the spatial and spectral joint features of the fused image, and performing small target feature enhancement, and finally outputting a high-precision target detection result; the method can improve the accuracy of small target detection in remote sensing images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for detecting small targets based on a panchromatic and hyperspectral fusion image. Background Art

[0002] Small object detection based on panchromatic and hyperspectral images is a challenging task, yet it continues to attract increasing attention due to their unique data characteristics. Panchromatic images have high spatial resolution, while hyperspectral images have high spectral resolution, each with its own advantages. Therefore, combining the two can achieve complementary and synergistic enhancement in object detection. Small object detection based on panchromatic and hyperspectral images has a wide range of applications, including food safety and quality monitoring, precision agriculture, biomedical applications, urban planning, and military surveillance, and therefore holds significant research value.

[0003] In recent years, deep learning methods have demonstrated promising performance in remote sensing imagery target detection. However, due to significant imaging differences between panchromatic and hyperspectral images, accurate small target detection based on these two image types still faces several challenges: 1) Wide-format remote sensing images captured at long distances have low spatial resolution. Even in panchromatic images, the target's shape and texture information is insufficient for effective small target detection; 2) Hyperspectral images have rich spectral features, but due to spectral mixing effects, the spectral features of small targets are inaccurate; 3) Some existing methods face challenges in accurately aligning spatial and spectral features, resulting in inaccurate small target detection.

[0004] Therefore, how to effectively utilize the information of the two images to obtain accurate target detection results is the main problem that needs to be solved urgently. Summary of the Invention

[0005] In view of this, the present invention provides a method and device for detecting small targets based on panchromatic and hyperspectral fusion images, which at least solve some of the above technical problems, so as to improve the accuracy of small target detection in remote sensing images.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, an embodiment of the present invention provides a method for detecting small targets based on a panchromatic and hyperspectral fusion image, the method comprising:

[0008] Obtain original registered panchromatic and hyperspectral images;

[0009] The image fusion network and the object detection network are sequentially connected to form a hierarchical supervision integrated network;

[0010] The network is supervised and trained, with the image fusion network and the object detection network each calculating their own loss functions. At the same time, the mutual learning loss is calculated based on the output of the object detection network and fed back to the image fusion network as a loss function for supervised training of the fusion network.

[0011] After training, the image fusion network is used to fuse the panchromatic image and the hyperspectral image; the target detection network is used to extract features from the fused image to achieve positioning detection of weak targets.

[0012] Optionally, after obtaining the original registered panchromatic image and hyperspectral image, the obtained hyperspectral image is upsampled to the same spatial resolution as the panchromatic image using bilinear interpolation.

[0013] Optionally, preprocessing is performed on the panchromatic image and the upsampled hyperspectral image, and the preprocessing includes: image cropping, labeling, and data normalization.

[0014] Optionally, when the image fusion network fuses the panchromatic image and the hyperspectral image, a coefficient matrix feature map is first generated, and then the generated coefficient matrix feature map and the original upsampled hyperspectral image are multiplied by band to obtain the final panchromatic and hyperspectral fused image.

[0015] Optionally, the image fusion network is constructed based on a Transformer network, including: three convolutional layers, the convolution kernel sizes of the three convolutional layers are: 9×9, 5×5 and 5×5 respectively; each convolutional layer is connected to a Relu activation function layer.

[0016] Optionally, the target detection network is jointly constructed based on CNN and Transformer networks, including: multiple cascaded spatial-spectral attention mapping modules, a global average pooling layer and a fully connected layer; wherein: each of the spatial-spectral attention mapping modules contains parallel calculations of: a local spatial convolution module, a global spatial-spectral calculation module and a channel spectrum calculation module.

[0017] In a second aspect, an embodiment of the present invention further provides a small target detection device based on a panchromatic and hyperspectral fusion image, comprising: an image acquisition device, a memory, and a processor, wherein:

[0018] The image acquisition device is used to acquire the registered panchromatic image and hyperspectral image;

[0019] The memory stores computer instructions;

[0020] The processor is connected to the image acquisition device and the memory data. The processor executes the computer instructions to perform the above-mentioned small target detection method based on panchromatic and hyperspectral fusion images to perform small target detection in remote sensing images.

[0021] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the above-mentioned method for detecting small targets based on panchromatic and hyperspectral fusion images is implemented.

[0022] Compared with the prior art, the present invention has at least the following beneficial effects:

[0023] In this paper, image fusion and target detection network are integrated into an integrated network model. The detection rate of small targets can be effectively enhanced through feature fusion. During the training process, the loss feedback adjustment strategy is used to feed back the target detection mask information into the fusion network and adjust its loss function, so that the network pays more attention to the target area, which can improve the accuracy of small target detection in remote sensing images.

[0024] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0025] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0028] Figure 1 A schematic diagram of the principle of a small target detection method based on panchromatic and hyperspectral fusion images provided by an embodiment of the present invention.

[0029] Figure 2 A flowchart of a small target detection method based on panchromatic and hyperspectral fusion images is provided in an embodiment of the present invention.

[0030] Figure 3 A schematic diagram of the image fusion network structure provided by an embodiment of the present invention.

[0031] Figure 4 A schematic diagram of the target detection network structure provided by an embodiment of the present invention.

[0032] Figure 5 Schematic diagram of the structure of the spatial-spectral attention mapping module provided in an embodiment of the present invention.

[0033] Figure 6 Schematic diagram of visualized image blocks after cropping the original 16-bit full-color image provided in an embodiment of the present invention.

[0034] Figure 7 Schematic diagram of the visualized image blocks after cropping the original 16-bit hyperspectral image (RGB band) provided in an embodiment of the present invention.

[0035] Figure 8 A schematic diagram of a fused image provided by an embodiment of the present invention.

[0036] Figure 9 A schematic diagram of small target detection results provided by an embodiment of the present invention.

[0037] Figure 10 A schematic structural diagram of a small target detection device based on panchromatic and hyperspectral fusion images provided in an embodiment of the present invention.

[0038] Among them, 1-image acquisition device; 2-memory; 3-processor. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0040] In describing the present invention, it should be noted that some processes described in this specification and accompanying drawings include multiple operations that appear in a specific order. However, it should be understood that these operations may be performed in a different order than the order in which they appear, or may be performed in parallel. Furthermore, the use of various sequence numbers is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0041] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0042] See also Figure 1 and Figure 2 As shown, the present invention provides a small target detection method based on panchromatic and hyperspectral fusion images, which is used to automatically detect small targets in panchromatic and hyperspectral fusion images. The method mainly includes:

[0043] Obtain original registered panchromatic and hyperspectral images;

[0044] The image fusion network (HIP) and the object detection network (SOD) are sequentially connected to form a hierarchical supervision integrated network. The image fusion network and the object detection network are then supervised by calculating their own loss functions. At the same time, the mutual learning loss is calculated based on the output of the object detection network and fed back to the image fusion network as a loss function together with other loss functions to supervise the training of the image fusion network.

[0045] After training, the image fusion network is used to fuse the panchromatic image and the hyperspectral image; the target detection network is used to extract features from the fused image to achieve positioning detection of weak targets.

[0046] The specific implementation of the method of the present invention is described in detail below:

[0047] (1) The original 16-bit registered panchromatic and hyperspectral image data taken by the satellite are obtained, and the hyperspectral image is upsampled to the same spatial resolution as the panchromatic image using bilinear interpolation to obtain the upsampled hyperspectral image.

[0048] (2) Preprocessing the obtained panchromatic and hyperspectral images to meet the machine computing requirements; specifically including:

[0049] First, the original 16-bit panchromatic and upsampled hyperspectral images are cropped into 256×256 image blocks to prepare training samples for the network model.

[0050] Then, the target ground truth box is annotated based on the panchromatic image and hyperspectral image data.

[0051] (3) Calculate the ratio transformation image of the panchromatic image and the hyperspectral image, specifically:

[0052] Calculate each band of the hyperspectral image and the panchromatic image to obtain a series of ratio transformation images (coefficient matrix feature maps); the specific calculation formula is:

[0053]

[0054] Among them, R i is the ratio transformation image for each band, P is the full color image, M i is each band of the hyperspectral image, and x and y are the pixel coordinates.

[0055] (4) Use the HIP network to perform feature fine-tuning and perform adaptive fusion processing on panchromatic and hyperspectral image blocks with complex backgrounds; complete the fusion processing of panchromatic and hyperspectral images.

[0056] In the embodiment of the present invention, the HIP network is built based on the Transformer network architecture optimization. Figure 3 As shown in Figure 1, the HIP network architecture mainly consists of three convolutional layers, with the convolution kernel sizes of each layer being 9×9, 5×5, and 5×5, respectively, followed by three ReLU activation function layers. The HIP network chooses to generate a ratio transformation image instead of directly generating a fusion image. The ratio transformation image of the panchromatic and hyperspectral images obtained in step (3) is then input into the network, the features are fine-tuned, and the output result is multiplied with the panchromatic image to obtain the fusion result, resulting in a final panchromatic and hyperspectral fusion image with higher fidelity.

[0057] (5) Use the SOD network to detect small targets in panchromatic and hyperspectral fusion images.

[0058] See also Figure 4 As shown in the figure, in an embodiment of the present invention, the SOD network is jointly constructed based on CNN and Transformer. The SOD network combines convolution and Transformer architectures, including multiple cascaded spectral attention mapping modules (Spectral-Spatial Attention Projection, SSAP) and a global average pooling layer. Finally, a fully connected layer is used to achieve effective feature fusion, which is used to solve the challenge of small target detection on hyperspectral and panchromatic images. Figure 5The internal structure of the proposed SSAP module is demonstrated. The SSAP module mainly consists of three parallel computational modules: a local spatial convolution module, a global spatial spectrum computation module, and a channel spectrum computation module; these modules are used to extract local spatial features, pixel spectral features, and global spatial-spectral joint features. Each module also consists of a multilayer perceptron (MLP), a depth multilayer perceptron (D-MLP), a channel multilayer perceptron (C-MLP), and a multi-head self-attention mechanism (MSA) module to extract image features and improve network attention. Integrating these structures can enhance the ability to capture spatial-spectral small target features and reduce information loss as the network deepens.

[0059] In the embodiment of the present invention, see Figure 1 and Figure 2 As shown in the figure, the image fusion network (HIP) and the object detection network (SOD) are combined and integrated through a feedback regulation mechanism. The HIP and SOD networks are sequentially connected, and the output fused image is directly used as the input data of the SOD network. The HIP network calculates spatial and spectral errors with the panchromatic image and hyperspectral image respectively using the mean absolute error function L1 loss, completing the gradient backpropagation. The SOD network calculates the error by comparing the target box with the ground truth using the Yolo loss function. During the training process, the loss function of the HIP network is added to the output target area of the SOD network for weight enhancement. This operation enables the HIP network to not only obtain high-fidelity fused images, but also integrates it with object detection to achieve optimal object detection results.

[0060] In this embodiment, Figures 6 to 9 An example of applying the method provided by the present invention to detect small targets based on panchromatic and hyperspectral fusion images is shown in FIG. Figure 6 and Figure 7 They are the original 16-bit panchromatic and hyperspectral images (RGB bands) after being cropped for visualization. Figure 8 is a schematic diagram of the fused image. Figure 9 This is a schematic diagram of the results of weak target detection. Figure 9 It can be seen that the method of the present invention can achieve accurate detection of weak targets in remote sensing images.

[0061] From the description of the above embodiments, those skilled in the art can understand that in order to improve the accuracy of small target detection in remote sensing images, the present invention proposes a small target detection method based on panchromatic and hyperspectral fusion images, which includes: first, obtaining panchromatic and hyperspectral remote sensing images with high matching accuracy; second, inputting paired panchromatic and hyperspectral remote sensing images into a HIP fusion network model to fuse the panchromatic and hyperspectral images to obtain a fused image with high spatial resolution and high spectral resolution with precise feature alignment; then, inputting the resulting fused image into a SOD small target detection network, extracting the spatial and spectral joint features of the fused image using three-dimensional spatial convolution, and enhancing the small target features using a feature enhancement module, ultimately outputting a high-precision target detection result; during the training process, using a loss feedback adjustment strategy, the target detection mask information is fed back to the fusion network, and its loss function is adjusted to make the network pay more attention to the target area. This method is a progressive multi-level supervised integrated network model that integrates image fusion and a small target detection network into an integrated network model. Through feature fusion, the small target detection rate can be effectively enhanced, which has high research significance and practical value.

[0062] Furthermore, the present invention also provides a small target detection system based on panchromatic and hyperspectral fusion images, which applies a small target detection method based on panchromatic and hyperspectral fusion images in the above embodiment to realize small target detection in remote sensing images. The system includes:

[0063] Image acquisition module, used to obtain original registered panchromatic image and hyperspectral image;

[0064] A network model building module, which is used to sequentially connect the image fusion network and the object detection network into a hierarchical supervision integrated network;

[0065] The network model training module is used to supervise the network training. The image fusion network and the object detection network are calculated using their own loss functions. At the same time, the mutual learning loss is calculated based on the output of the object detection network and fed back to the image fusion network as a loss function for supervised training of the fusion network.

[0066] The network model application module is used to fuse the panchromatic image and the hyperspectral image using the trained image fusion network; and to extract features from the fused image using the trained target detection network to achieve the positioning detection of small targets.

[0067] The system provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding content in the aforementioned method embodiment, which will not be repeated here.

[0068] In addition, the embodiment of the present invention also provides a small target detection device based on panchromatic and hyperspectral fusion images to achieve accurate detection of small targets. The device can be implemented by software and / or hardware. Figure 10 As shown, the apparatus includes an image acquisition device 1, a memory 2, and a processor 3. The image acquisition device 1, the memory 2, and the processor 3 may be connected via a bus or other means.

[0069] The image acquisition device 1 is used to acquire original remote sensing image data and send it to the processor 3. The processor 3 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0070] Memory 2, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as programs or instructions corresponding to a small target detection method based on panchromatic and hyperspectral fusion images.

[0071] The processor 3 executes various functional applications and data processing of the processor by running the non-transitory software program or instructions stored in the memory 2, that is, implements the dim target detection method based on panchromatic and hyperspectral fusion images in the above method embodiment.

[0072] The memory 2 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created by the processor 2 and the like.

[0073] In addition, the memory 2 may include a high-speed random access memory and may also include a non-volatile memory. More specific examples of the memory 2 (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination of the above.

[0074] In some aspects, the memory 2 optionally includes memories remotely located relative to the processor 3 , and these remote memories may be connected to the processor 3 via a network.

[0075] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, devices, or computer program products, etc. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0076] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in a claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer.

[0077] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0078] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A small target detection method based on panchromatic and hyperspectral fusion images, characterized in that: The method includes: Obtain original registered panchromatic and hyperspectral images; The image fusion network and the object detection network are sequentially connected to form a hierarchical supervision integrated network; The network is supervised and trained, with the image fusion network and the object detection network each calculating their own loss functions. At the same time, the mutual learning loss is calculated based on the output of the object detection network and fed back to the image fusion network as a loss function for supervised training of the fusion network. After training, the image fusion network is used to fuse the panchromatic image and the hyperspectral image; the target detection network is used to extract features from the fused image to achieve the positioning and detection of small targets; The target detection network is jointly constructed based on CNN and Transformer networks, including: multiple cascaded spatial-spectral attention mapping modules, a global average pooling layer and a fully connected layer; wherein: each of the spatial-spectral attention mapping modules includes parallel calculations: a local spatial convolution module, a global spatial-spectral calculation module and a channel spectrum calculation module; wherein: The local spatial convolution module includes: average pooling layer, 3D convolution layer and Sigmoid activation function layer; the global spatial spectrum calculation module includes: normalization layer, deep multi-layer perceptron and channel multi-layer perceptron; the channel spectrum calculation module includes: normalization layer, multi-head attention mechanism module and multi-layer perceptron.

2. The method for detecting small targets based on panchromatic and hyperspectral fusion images according to claim 1, characterized in that: After obtaining the original registered panchromatic and hyperspectral images, the acquired hyperspectral image is upsampled to the same spatial resolution as the panchromatic image using bilinear interpolation.

3. The method for detecting small targets based on panchromatic and hyperspectral fusion images according to claim 2, characterized in that: Preprocessing is performed on the panchromatic image and the upsampled hyperspectral image, and the preprocessing includes image cropping, labeling, and data normalization.

4. The method for detecting small targets based on panchromatic and hyperspectral fusion images according to claim 1, wherein: When the image fusion network fuses the panchromatic image and the hyperspectral image, it first generates a coefficient matrix feature map, and then multiplies the generated coefficient matrix feature map and the original upsampled hyperspectral image by band to obtain the final panchromatic and hyperspectral fused image.

5. The method for detecting small targets based on panchromatic and hyperspectral fusion images according to claim 1, characterized in that: The image fusion network is constructed based on the Transformer network, including three convolutional layers, the convolution kernel sizes of the three convolutional layers are: 9×9, 5×5 and 5×5 respectively; each convolutional layer is connected to a Relu activation function layer.

6. A small target detection device based on panchromatic and hyperspectral fusion images, characterized in that: include: An image acquisition device, a memory, and a processor, wherein: The image acquisition device is used to acquire the original registered panchromatic image and hyperspectral image; The memory stores computer instructions; The processor is connected to the image acquisition device and the memory data. The processor executes the computer instructions to perform the small target detection method based on panchromatic and hyperspectral fusion images as described in any one of claims 1 to 5, and performs small target detection in remote sensing images.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement a small target detection method based on panchromatic and hyperspectral fusion images as described in any one of claims 1 to 5.

Images