Method for detecting small target floaters on sea surface based on hyperspectrum reconstructed by RGB (red, green and blue)

Abstract

The invention relates to a sea surface small target floating object detection method based on an RGB reconstruction hyperspectrum, and the method comprises the steps: obtaining an RGB image, carrying out the preprocessing of the obtained image, and constructing an RGB image sample data set through the preprocessed image; generating a hyperspectral data set of a corresponding scene from the RGB image sample data set by using the residual dense network integrated with the frequency domain information; 31 wave bands of the hyperspectral image are grouped, and the image with the highest entropy value in each group is selected to synthesize a pseudo-color image; and inputting the pseudo-color image into the fixed detection network to obtain a target detection result. According to the invention, the detection of the small target floating object on the sea surface is realized by using the spectral dimension information of the hyperspectral image.

Description

Technical Field

[0001] The invention relates to the technical field of sea surface small target floating object detection, in particular to a sea surface small target floating object detection method based on RGB reconstruction hyperspectrum. Background Art

[0002] Small floating targets on the sea surface usually refer to some buoys floating on the sea surface. These buoys may be unmanned monitoring equipment, navigation signs, meteorological observation instruments or detectors for military purposes. Such floating objects are small in size and easily affected by marine environmental factors, such as waves, wind speed and ocean currents, making their movement trajectories more complex and difficult to predict. In complex sea conditions, timely and accurate detection and identification of these small targets are of great significance to marine monitoring, navigation safety and military operations. In addition, with the development of science and technology, these floating objects may also be equipped with various sensors for tasks such as real-time data collection, environmental monitoring and information transmission, further enhancing their application value in different fields. For this purpose, it is difficult to achieve good detection results using traditional target detection methods based on RGB images. In addition, detection in low-light environments such as at night or in foggy days will be more difficult, the application scenarios are limited and cannot work around the clock.

[0003] Hyperspectral imaging technology has many advantages. First, it can capture the reflection and emission information of objects in multiple narrow bands with extremely high spectral resolution, providing details far beyond traditional color or multispectral imaging. This gives it a strong advantage in target recognition, and it can distinguish different materials by analyzing the spectral characteristics of objects, even if these objects are very similar in appearance. Hyperspectral imaging technology can not only generate images, but also provide spectral curves of targets for in-depth analysis of physical and chemical properties. This technology is particularly applicable in the field of small target floating objects on the sea surface. Hyperspectral data has the characteristics of image and spectrum integration, so when extracting small target floating objects on the sea surface, image features and spectral features can be used simultaneously to extract target information. Since floating targets are generally plastics, metals or other materials, and the ocean background is seawater, there are obvious differences in the radiation and scattering characteristics of the background and the target, and the spectral images are also different. Therefore, the use of hyperspectral imaging technology can effectively help to quickly identify and classify these small targets in complex sea conditions. This has important practical value for ocean monitoring, navigation safety and military detection.

[0004] Although hyperspectral imaging technology has significant advantages in target recognition and classification, its high cost is an important disadvantage. Hyperspectral imaging equipment is usually expensive and requires high-performance computing resources for data processing. At the same time, its data acquisition speed is slow, which limits its widespread use in certain application scenarios. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method for detecting small target floating objects on the sea surface based on RGB reconstruction of hyperspectral, which can effectively solve the problem that hyperspectral imaging equipment is difficult to use in the field of detecting small target floating objects on the sea surface, and can reduce costs while maintaining high detection accuracy.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral, comprising the following steps:

[0007] Step S1: acquiring an RGB image, preprocessing the acquired image, and constructing RGB image sample data using the preprocessed image;

[0008] Step S2: Generate a hyperspectral dataset of the corresponding scene from the RGB image sample dataset using a residual dense network that incorporates frequency domain information;

[0009] Step S3: performing band screening for the entropy value of each band image in the hyperspectral image, and selecting three bands with the highest entropy values ​​to combine and generate a pseudo-color image;

[0010] Step S4: Input the pseudo-color image into the fixed detection network to obtain the target detection result.

[0011] In a preferred embodiment, the specific implementation method of step S1 is as follows:

[0012] Step S101: collecting a hyperspectral data set. Specifically, using a drone equipped with an RGB camera to collect images, the collected RGB images contain small floating objects on the sea surface;

[0013] Step S102: Preprocessing of hyperspectral data set:

[0014] Step S103: construct a sample data set using the preprocessed RGB image.

[0015] In a preferred embodiment, the specific implementation method of step S102 is as follows:

[0016] Step S1021: uniformly cropping the dimensions of the acquired hyperspectral image to H×W×3, where H, W, and 3 are the height, width, and number of spectral channels of the RGB image, respectively;

[0017] Step S1022: The collected RGB image is processed using a cubic interpolation method. After the spatial resolution is reduced, the size of the RGB image is h×w×3. The RGB image after the spatial resolution is reduced is used as an RGB data set.

[0018] In a preferred embodiment, step S2 includes:

[0019] Step S201: weighted window function convolution module, the weighted window function convolution module uses the Hamming window function, and multiplies the convolution kernel weight by the Hamming window function value. The Hamming window function calculation formula used is as follows:

[0020]

[0021] Where n represents the index of the discrete sampling point, and N represents the number of discrete sampling points; the convolution kernel size in the weighted window function convolution module is set to 7×7;

[0022] Let the input of the network be I RGB First, the shallow feature F is extracted through the weighted window function convolution module SF ;

[0023] F SF =H Wconv1 (I RGB )

[0024] Where H Wconv1 (·) represents the weighted window function convolution operation;

[0025] Step S202: RGB image preprocessing module; for the shallow feature F extracted in step S201 SF As the input of the image preprocessing module;

[0026] F Pre =H Pre (F SF )

[0027] Where H P re(·) represents the image preprocessing operation, F Pre Represents the output of the image preprocessing module;

[0028] Step S203: The output F of the image preprocessing module Pre As the input of DRCR module;

[0029]

[0030] Where F m-1 and F m Respectively represent the input and output of the mth DRCR block, Indicates the mth DRCR block operation, Indicates the first DRCR block operation;

[0031] DRCR module, each DRCR block contains two channel compression-excitation network modules SE-Net and U-type codec module; the specific process of SE-Net module is as follows;

[0032]

[0033] In the formula and Represent the first and second SE-Net operations respectively, and Represent the outputs of the first and second SE-Net operations respectively;

[0034] The main function of the U-type codec module is to explore the interactive characteristics of spatial information of the same scale and information of different spatial scales in a multi-scale structure to achieve high-fidelity spatial dimension information recovery; there are 6 convolution operations in the U-type codec module, and the i-th convolution operation of the m-th DRCR block is recorded as The output of its operation is F (m,i) ;

[0035]

[0036] In the formula and Represents the CRM (dual channel recalibration) module operation in the network;

[0037] Finally, the output of the last DRCR module is used as the input of the last weighted window function convolution module, and the number of channels is mapped to 31 as the band data I of the final hyperspectral output. HSI ;

[0038] I HSI =H conv (F m )

[0039] Where H conv (·) represents the convolution operation at the end of the network;

[0040] Step S204: The loss function of the residual dense network integrated with frequency domain learning consists of two parts, one is the loss function Loss1 describing the spatial-spectral dimension information, and the other is the loss function Loss2 describing the frequency domain information. The total loss function L total is the linear superposition of the two;

[0041]

[0042] L total =Loos1+εLoss2

[0043] In the formula and are the network output value and the true value of the i-th pixel, N pixels represents the number of pixels in the hyperspectral data, FFT(·) represents the fast Fourier transform, and ε represents the balance coefficient of the two loss functions.

[0044] In a preferred embodiment, the step of synthesizing the pseudo-color image in step S3 includes:

[0045] Step S301: grouping the hyperspectral images of 31 bands of 400-700nm after spectral reconstruction into 400-500nm, 510nm-600 and 610-700nm;

[0046] Step S302: Select the image with the highest entropy value in each group according to the entropy calculation formula. The entropy calculation formula is based on the definition of Shannon Entropy and is applicable to discrete probability distribution. For a grayscale image, entropy is defined as follows:

[0047]

[0048] Where H(X) is the entropy of the image, p(x i ) is the gray level x i The probability of appearing in the image; k is the number of gray levels of the image, which is usually 256 gray levels for an 8-bit grayscale image;

[0049] Step S303: synthesizing a pseudo-color image by channel for the filtered three-band images.

[0050] In a preferred embodiment, the fixed detection network adopts Yolov5 network.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention utilizes the detection of small target floating objects on the sea surface based on RGB reconstruction of hyperspectral. The present invention utilizes the spectral dimension information of hyperspectral images to realize the detection of small floating objects on the sea surface.

[0053] 2. In the present invention, since the hyperspectral image can cover the 400nm-700nm band, it is conducive to target detection in low-light environments, expanding the application scenarios of the present invention.

[0054] 3. Whether the target moves or not has no substantial effect on its scattering spectrum information, so the present invention is applicable to both moving and stationary camouflage targets.

[0055] 4. The present invention can collect and detect data sets through a drone equipped with an RGB camera, greatly expanding the detection range of small target floating objects on the sea surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flowchart of a method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral spectrum according to a preferred embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the network structure of a spectrum reconstruction algorithm incorporating frequency domain information used in a preferred embodiment of the present invention;

[0058] Figure 3 The pseudo color image synthesis process of the preferred embodiment of the present invention;

[0059] Figure 4 A pseudo-color image in an example of a preferred embodiment of the present invention;

[0060] Figure 5 A visualization detection result based on a pseudo-color image in an example of a preferred embodiment of the present invention;

[0061] Figure 6 for Figure 5 The visible light image corresponding to the visual detection result. DETAILED DESCRIPTION

[0062] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0064] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0065] like Figure 1-5 As shown, the present embodiment provides a method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral, comprising the following steps:

[0066] Step S1: acquiring an RGB image, preprocessing the acquired image, and constructing RGB image sample data using the preprocessed image;

[0067] Step S2: Generate a hyperspectral dataset of the corresponding scene from the RGB image sample dataset using a residual dense network that incorporates frequency domain information;

[0068] Step S3: performing band screening for the entropy value of each band image in the hyperspectral image, and selecting three bands with the highest entropy values ​​to combine and generate a pseudo-color image;

[0069] Step S4: Input the pseudo-color image into the fixed detection network to obtain the target detection result.

[0070] This embodiment provides the step of obtaining an RGB image in step S1, preprocessing the obtained image, and constructing an RGB image sample data set using the preprocessed image. The specific implementation method is as follows:

[0071] Step S101: collecting a hyperspectral data set. Specifically, using a drone equipped with an RGB camera to collect images, the collected RGB images contain small floating objects on the sea surface;

[0072] Step S102: Preprocessing of hyperspectral data set:

[0073] (1) The dimensions of the collected hyperspectral images are uniformly cropped to H×W×3, where H, W, and 3 are the height, width, and number of spectral channels of the RGB image, respectively;

[0074] (2) The collected RGB image is processed using the cubic interpolation method. After the spatial resolution is reduced, the size of the RGB image is h×w×3. The RGB image after the spatial resolution is reduced is used as the RGB data set.

[0075] Step S103: construct a sample data set using the preprocessed RGB image.

[0076] like Figure 2 As shown, the structure of the spectrum reconstruction algorithm incorporating frequency domain information in step S2 of this embodiment includes:

[0077] Step S201: weighted window function convolution module. The weighted window function convolution module uses the Hamming window function. The general convolution kernel weight is multiplied by the Hamming window function value. The Hamming window function calculation formula used is as follows:

[0078]

[0079] Where N represents the number of discrete sampling points. In order to further improve the spectrum leakage problem, the convolution kernel size in the weighted window function convolution module is set to 7×7.

[0080] Let the input of the network be I RGB First, the shallow feature F is extracted through the weighted window function convolution module SF .

[0081] F SF =H Wconv1 (I RGB )

[0082] Where H Wconv1 (·) represents the weighted window function convolution operation.

[0083] Step S202: RGB image preprocessing module, for the degradation problem of input RGB image, RGB image may be affected by noise, overexposure, compression, etc., which may lead to inaccurate network prediction results. A multi-scale pyramid model is constructed by downsampling to solve the RGB image degradation problem. For the shallow feature F extracted in step S201, SF As the input of the image preprocessing module.

[0084] F Pre =H Pre (F SF )

[0085] Where H Pre (·) indicates image preprocessing operations.

[0086] Step S203: The output F of the image preprocessing module Pre As the input of DRCR module.

[0087]

[0088] Where F m-1 and F m Respectively represent the input and output of the mth DRCR block, Indicates the mth DRCR block operation.

[0089] DRCR module, each DRCR block contains two channel compression-excitation network modules (Squeeze-and-Excitation Networks, SE-Net) and U-type encoding and decoding modules. The main function of the SE-Net module is to integrate the characteristic relationship between channels and improve the learning ability of the network in the spectral dimension to achieve high-fidelity spectral restoration. The specific process is as follows.

[0090]

[0091] In the formula and Represent the first and second SE-Net operations respectively, and Represent the outputs of the first and second SE-Net operations, respectively.

[0092] The main function of the U-type codec module is to explore the interactive characteristics of spatial information of the same scale and information of different spatial scales in a multi-scale structure to achieve high-fidelity spatial dimension information recovery. There are 6 convolution operations in the U-type codec module. The i-th convolution operation of the m-th DRCR block is The output of its operation is F (m,i) .

[0093]

[0094] Finally, the output of the last DRCR module is used as the input of the last weighted window function convolution module, and the number of channels is mapped to 31 as the band data I of the final hyperspectral output. HSI .

[0095] I HSI =H conv (F m )

[0096] Where H conv (·) denotes the convolution operation at the end of the network.

[0097] Step S204: The loss function of the residual dense network integrated with frequency domain learning consists of two parts, one is the loss function Loss1 describing the spatial-spectral dimension information, and the other is the loss function Loss2 describing the frequency domain information. The total loss function L total is the linear superposition of the two.

[0098]

[0099] L total =Loos1+εLoss2

[0100] In the formula and are the network output value and the true value of the i-th pixel, N pixels represents the number of pixels in the hyperspectral data, FFT(·) represents the fast Fourier transform, and ε represents the balance coefficient of the two loss functions.

[0101] like Figure 3 As shown, the step of synthesizing the pseudo color image in step S3 of this embodiment includes:

[0102] Step S301: grouping the hyperspectral images of 31 bands of 400-700nm after spectral reconstruction into 400-500nm, 510nm-600 and 610-700nm;

[0103] Step S302: Select the image with the highest entropy value in each group according to the entropy calculation formula. The entropy calculation formula is based on the definition of Shannon Entropy and is applicable to discrete probability distribution. For a grayscale image, entropy is defined as follows:

[0104]

[0105] Where H(X) is the entropy of the image, p(x i) is the probability that gray level xi appears in the image. k is the number of gray levels in the image, usually 256 gray levels (0 to 255) for an 8-bit grayscale image.

[0106] Step S303: synthesizing a pseudo-color image by channel for the filtered three-band images.

[0107] Step S4: Input the trimodal fusion image into a fixed detection network to obtain the target detection result.

[0108] Example:

[0109] Step S1: Obtain an RGB image, preprocess the acquired image, and use the preprocessed image to construct RGB image sample data:

[0110] Step S101: Use a drone equipped with an RGB camera to record RGB video:

[0111] Step S102: extract frames from the video and crop and integrate 400 RGB images with a dimension of 3200x2000x3 as a data set;

[0112] Step S103: Use cubic interpolation to reduce the spatial resolution of the RGB image to obtain 400 RGB images with a dimension of 1600x1000x3.

[0113] Step S2, spectral super-resolution of RGB data: 80 RGB images are selected and the spectral reconstruction of the input 3-band RGB image is performed using the improved DRCR method that incorporates frequency domain information to produce 80 hyperspectral images of the corresponding scene.

[0114] Step S3, pseudo-color image synthesis: the hyperspectral image generated in step S2 is screened by bands, and the three bands with the highest recognition rates are selected to form a pseudo-color image;

[0115] Step S4: Use the target detection model to detect and identify the original RGB image obtained in step S1 and the pseudo-color image obtained in step S3. Figure 5 As shown in the figure, the detected small floating objects and ships on the sea surface are in the box, and the categories corresponding to the camouflaged targets are output on the box, with confidence probabilities of 0.88, 0.85 and 0.84 respectively. Figure 6 for Figure 5 The visible light image corresponding to the detection result shows that compared with the visible light image, the method proposed in this paper can identify the target more effectively.

[0116] It can be seen that the hyperspectral image camouflage target detection method based on deep learning of the present invention has the following beneficial effects:

[0117] The present invention utilizes the detection of small target floating objects on the sea surface based on RGB reconstruction of hyperspectral. The present invention utilizes the spectral dimension information of hyperspectral images to realize the detection of small floating objects on the sea surface.

[0118] In the present invention, since the hyperspectral image can cover the 400nm-700nm band, it is conducive to target detection in low-light environments, expanding the application scenarios of the present invention.

[0119] Whether the target moves or not has no substantial effect on its scattering spectrum information, so the present invention is applicable to both moving and stationary camouflage targets.

[0120] The present invention can collect and detect data sets through a drone equipped with an RGB camera, greatly expanding the detection range of small target floating objects on the sea surface.

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral, characterized in that , including the following steps: Step S1: acquiring an RGB image, preprocessing the acquired image, and constructing RGB image sample data using the preprocessed image; Step S2: Generate a hyperspectral dataset of the corresponding scene from the RGB image sample dataset using a residual dense network that incorporates frequency domain information; Step S3: performing band screening for the entropy value of each band image in the hyperspectral image, and selecting three bands with the highest entropy values ​​to combine and generate a pseudo-color image; Step S4: Input the pseudo-color image into the fixed detection network to obtain the target detection result.

2. According to claim 1, a method for detecting small floating objects on the sea surface based on RGB reconstruction hyperspectral, characterized in that: The specific implementation method of step S1 is as follows: Step S101: collecting a hyperspectral data set. Specifically, using a drone equipped with an RGB camera to collect images, the collected RGB images contain small floating objects on the sea surface; Step S102: Preprocessing of hyperspectral data set: Step S103: construct a sample data set using the preprocessed RGB image.

3. The method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral according to claim 2 is characterized in that: The specific implementation method of step S102 is as follows: Step S1021: uniformly cropping the dimensions of the acquired hyperspectral image to H×W×3, where H, W, and 3 are the height, width, and number of spectral channels of the RGB image, respectively; Step S1022: The collected RGB image is processed using a cubic interpolation method. After the spatial resolution is reduced, the size of the RGB image is h×w×3. The RGB image after the spatial resolution is reduced is used as an RGB data set.

4. The method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral according to claim 1 is characterized in that: Step S2 comprises: Step S201: weighted window function convolution module, the weighted window function convolution module uses the Hamming window function, and multiplies the convolution kernel weight by the Hamming window function value. The Hamming window function calculation formula used is as follows: Where n represents the index of the discrete sampling point, and N represents the number of discrete sampling points; the convolution kernel size in the weighted window function convolution module is set to 7×7; Let the input of the network be I RGB First, the shallow feature F is extracted through the weighted window function convolution module SF ; F SF =H Wconv1 (I RGB ) Where H Wconv1 (·) represents the weighted window function convolution operation; Step S202: RGB image preprocessing module; for the shallow feature F extracted in step S201 SF As the input of the image preprocessing module; F Pre =H Pre (F SF ) Where H P re(·) represents the image preprocessing operation, F Pre Represents the output of the image preprocessing module; Step S203: The output F of the image preprocessing module Pre As the input of DRCR module; Where F m-1 and F m Respectively represent the input and output of the mth DRCR block, Indicates the mth DRCR block operation, Indicates the first DRCR block operation; DRCR module, each DRCR block contains two channel compression-excitation network modules SE-Net and U-type codec module; the specific process of SE-Net module is as follows; In the formula and Represent the first and second SE-Net operations respectively, and Represent the outputs of the first and second SE-Net operations respectively; The main function of the U-type codec module is to explore the interactive characteristics of spatial information of the same scale and information of different spatial scales in a multi-scale structure to achieve high-fidelity spatial dimension information recovery; there are 6 convolution operations in the U-type codec module, and the i-th convolution operation of the m-th DRCR block is recorded as The output of its operation is F (m,i ); In the formula and Represents the CRM (dual channel recalibration) module operation in the network; Finally, the output of the last DRCR module is used as the input of the last weighted window function convolution module, and the number of channels is mapped to 31 as the band data I of the final hyperspectral output. HSI ; I HSI =H conv (F m ) Where H conv (·) represents the convolution operation at the end of the network; Step S204: The loss function of the residual dense network integrated with frequency domain learning consists of two parts, one is the loss function Loss1 describing the spatial-spectral dimension information, and the other is the loss function Loss2 describing the frequency domain information. The total loss function L total is the linear superposition of the two; L total =Loos1+εLoss2 In the formula and are the network output value and the true value of the i-th pixel, N pixels represents the number of pixels in the hyperspectral data, FFT(·) represents the fast Fourier transform, and ε represents the balance coefficient of the two loss functions.

5. According to claim 1, a method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral is characterized in that ,The step of synthesizing the pseudo-color image described in step S3 includes: Step S301: grouping the hyperspectral images of 31 bands of 400-700nm after spectral reconstruction into 400-500nm, 510nm-600 and 610-700nm; Step S302: Select the image with the highest entropy value in each group according to the entropy calculation formula. The entropy calculation formula is based on the definition of Shannon Entropy and is applicable to discrete probability distribution. For a grayscale image, entropy is defined as follows: Where H(X) is the entropy of the image and p(xi) is the gray level x i The probability of appearing in the image, k is the number of gray levels of the image, which is usually 256 gray levels for an 8-bit grayscale image; Step S303: synthesizing a pseudo-color image by channel for the filtered three-band images.

6. The method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral according to claim 1 is characterized in that: The fixed detection network adopts Yolov5 network.

Images