A method for detecting small floating objects on the sea surface based on RGB reconstruction hyperspectrum

Abstract

The application relates to a method for detecting small floating targets on the sea surface based on RGB reconstruction hyperspectrum, which comprises the following steps: acquiring an RGB image, pre-processing the acquired image, and constructing an RGB image sample data set by using the pre-processed image; generating a corresponding scene hyperspectrum data set from the RGB image sample data set by using a residual dense network with frequency domain information; grouping 31 wave bands of the hyperspectrum image, selecting an image with the highest entropy value in each group to synthesize a pseudo-color image; and inputting the pseudo-color image into a fixed detection network to obtain a target detection result. The spectral dimension information of the hyperspectrum image is utilized to realize the detection of small floating targets on the sea surface.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sea surface small target floating object detection, and particularly relates to a sea surface small target floating object detection method based on RGB reconstruction hyperspectrum. BACKGROUND

[0002] The sea surface small target floating object generally refers to a buoy floating on the sea surface. The buoy can be an unmanned monitoring device, a navigation marker, a meteorological observation instrument, or a military probe, etc. The volume of the buoy is small, and it is easily affected by marine environmental factors such as sea waves, wind speed, and ocean currents, so that its movement trajectory is complex and difficult to predict. In complex sea conditions, timely and accurate detection and identification of these small targets are of great significance to marine monitoring, navigation safety, and military operations. In addition, with the development of technology, these floating objects can also carry various sensors for real-time data collection, environmental monitoring, and information transmission, etc., further enhancing their application value in different fields. For this target, it is difficult to achieve good detection effect using the traditional target detection method based on RGB image. And in the low-illumination environment such as night or fog, the detection will be more difficult, the application scene is limited, and it cannot work all day.

[0003] Hyperspectral imaging technology has many advantages. First, it can capture the reflection and emission information of objects in multiple narrow wavebands with extremely high spectral resolution, providing much more detail than traditional color or multispectral imaging. This makes it have a strong advantage in target identification, and can distinguish different materials by analyzing the spectral characteristics of the object, even if the appearance of these objects is very similar. Hyperspectral imaging technology not only generates images, but also provides spectral curves of the target object for in-depth analysis of physical and chemical properties. In the field of sea surface small target floating object detection, this technology is particularly suitable. Hyperspectral data has the characteristics of image and spectrum integration, so when extracting sea surface small target floating objects, image features and spectral features can be used to extract target information at the same time. Since the floating object target is generally made of plastic, metal, or other materials, and the marine background is seawater, the radiation and scattering characteristics of the background and the target are obviously different, and the spectral images are also different, so using hyperspectral imaging technology can effectively help to quickly identify and classify these small targets in complex sea conditions. This has important practical value for marine monitoring, navigation safety, and military detection.

[0004] Although hyperspectral imaging technology has significant advantages in target identification and classification, its high cost is an important disadvantage. Hyperspectral imaging devices are usually expensive, and require high-performance computing resources for data processing, and their data acquisition speed is slow, which limits its widespread use in some application scenarios. SUMMARY

[0005] Therefore, the present application aims to provide a method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral data, which can effectively solve the problem that hyperspectral imaging equipment is difficult to use in the field of detecting small floating objects on the sea surface, and can reduce costs while maintaining high detection accuracy.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: a method for detecting small floating objects on the sea surface based on RGB reconstruction of hyperspectral data, comprising the following steps:

[0007] Step S1: obtaining an RGB image, pre-processing the obtained image, and using the pre-processed image to construct an RGB image sample data set;

[0008] Step S2: using a residual dense network with frequency domain information to generate a hyperspectral data set corresponding to the scene from the RGB image sample data set;

[0009] Step S3: performing band selection on the entropy values of each band image in the hyperspectral image, and selecting three band combinations with the highest entropy values to generate a pseudo-color image;

[0010] Step S4: inputting the pseudo-color image into a fixed detection network to obtain a target detection result.

[0011] In a preferred embodiment, the specific implementation method of step S1 is as follows:

[0012] Step S101: collection of the hyperspectral data set, specifically, using a UAV equipped with an RGB camera to collect images, and the collected RGB images contain small floating objects on the sea surface;

[0013] Step S102: pre-processing of the hyperspectral data set:

[0014] Step S103: constructing a sample data set from the pre-processed RGB image.

[0015] In a preferred embodiment, the specific implementation method of step S102 is as follows:

[0016] Step S1021: uniformly cropping the dimensions of the collected hyperspectral image to HxWx3, where H, W, and 3 are the height, width, and number of spectral channels of the RGB image, respectively;

[0017] Step S1022: using a cubic interpolation method to process the collected RGB image, reducing the spatial resolution of the RGB image to h x w x 3, and using the RGB image with reduced spatial resolution as the RGB data set.

[0018] In a preferred embodiment, step S2 comprises:

[0019] Step S201: a weighted window function convolution module, the weighted window function convolution module uses a Hamming window function to multiply the convolution kernel weight by a Hamming window function value, and the Hamming window function calculation formula is as follows:

[0020]

[0021] In the formula, n represents the index of a discrete sampling point, and N represents the number of discrete sampling points; the convolution kernel size in the weighted window function convolution module is set to 7*7;

[0022] Let the input of the network be I RGB , and first extract a shallow feature F SF by the weighted window function convolution module;

[0023] F SF = H Wconv1 (I RGB )

[0024] In the formula, H Wconv1 (·) represents a weighted window function convolution operation;

[0025] Step S202: an RGB image preprocessing module; the shallow feature F SF extracted in step S201 is taken as the input of the image preprocessing module;

[0026] F Pre = H Pre (F SF )

[0027] In the formula, H P re(·) represents an image preprocessing operation, and F Pre represents the output of the image preprocessing module;

[0028] Step S203: taking the output F Pre of the image preprocessing module as the input of a DRCR module;

[0029]

[0030] In the formula, F m-1 and F m represent the input and output of the mth DRCR block respectively, represents the mth DRCR block operation, represents the 1st DRCR block operation;

[0031] The DRCR module, each DRCR block further contains a channel compression-excitation network module SE-Net and a U-shaped encoding-decoding module twice; the specific process of the SE-Net module is as follows:

[0032]

[0033] wherein and denote the first and second SE-Net operation, respectively, and denote the output of the first and second SE-Net operation, respectively;

[0034] The U-shaped coding module mainly functions to explore the interaction characteristics of information in the same scale space and information in different spatial scales in the multi-scale structure, so as to realize the recovery of high-fidelity spatial dimension information. There are a total of 6 convolution operations in the U-shaped coding module, and the i-th convolution operation of the m-th DRCR block is denoted as The output of the operation is F (m,i) ;

[0035]

[0036] wherein and represent the CRM (dual-channel rectification) module operation in the network;

[0037] Finally, the output of the last DRCR module is taken as the input of the last weighted window function convolution module, and the channel number is mapped to 31, as the band data I HSI of the final hyperspectral output;

[0038] I HSI = H conv (F m )

[0039] wherein H conv (·) represents the convolution operation at the end of the network;

[0040] Step S204: The loss function of the residual dense network integrated with the frequency domain learning consists of two parts, one part is the loss function Loss1 describing the spatial-spectral dimension information, and the other part is the loss function Loss2 describing the frequency domain information, and the total loss function L total is the linear superposition of the two;

[0041]

[0042] L total = Loos1+εLoss2

[0043] wherein and are the network output value and the true value of the i-th pixel value, respectively, N pixels represents the number of pixel points of the hyperspectral data, FFT(·) represents the fast Fourier transform, and ε represents the balance coefficient of the two loss functions.

[0044] In a preferred embodiment, the step of pseudo-color image synthesis in step S3 comprises:

[0045] Step S301: Group the 31 bands of 400-700nm hyperspectral image after spectral reconstruction into 400-500nm, 510nm-600nm and 610-700nm;

[0046] Step S302: Select the one with the highest entropy in each group according to the entropy calculation formula, which is based on the definition of Shannon Entropy and applicable to discrete probability distribution; for a gray image, the definition of entropy is as follows:

[0047]

[0048] In the formula, H(X) is the entropy of the image, p(x i ) is the probability of the gray level x i appearing in the image; k is the number of gray levels of the image, which is usually 256 gray levels for an 8-bit gray image;

[0049] Step S303: Synthesize pseudo-color images by channel for the three screened band images.

[0050] In a preferred embodiment, the fixed detection network adopts a Yolov5 network.

[0051] Compared with the prior art, the present application has the following beneficial effects:

[0052] 1. The present application utilizes RGB-based reconstructed hyperspectral sea surface small target floating object detection, which utilizes the spectral dimension information of the hyperspectral image to realize the detection of the sea surface small floating object.

[0053] 2. In the present application, since the hyperspectral image can cover the 400nm-700nm band, it is beneficial to target detection in low-illumination environment, which expands the use scenarios of the present application.

[0054] 3. Whether the target moves or not has no substantial influence on its scattering spectral information, so the present application is applicable to moving or stationary camouflage targets.

[0055] 4. The present application can collect data sets and detect by the unmanned aerial vehicle equipped with an RGB camera, which greatly expands the detection range of the sea surface small target floating object. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a flow chart of the preferred embodiment of the present application, a sea surface small target floating object detection method based on RGB reconstructed hyperspectral;

[0057] Figure 2 Network structure diagram of a spectrum reconstruction algorithm incorporating frequency domain information for use in a preferred embodiment of the present application;

[0058] Figure 3 Pseudo-color map synthesis process for a preferred embodiment of the present application;

[0059] Figure 4 Pseudo-color map in an example for a preferred embodiment of the present application;

[0060] Figure 5 Visualized detection result based on a pseudo-color map in an example for a preferred embodiment of the present application;

[0061] Figure 6 Visualized detection result corresponding to the visible light image of Figure 5 DETAILED DESCRIPTION

[0062] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as would be commonly understood by one of ordinary skill in the art to which the present application belongs.

[0064] It should be noted that the terms used herein are merely intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the present specification, they indicate the presence of a feature, step, operation, device, component, and / or combination thereof.

[0065] As shown in FIG. 1, the present embodiment provides a method for detecting floating objects on the sea surface based on RGB reconstructed hyperspectrum, which includes the following steps: Figures 1-5 Step S1: Obtain an RGB image, pre-process the obtained image, and use the pre-processed image to construct RGB image sample data;

[0066] Step S2: Use a residual dense network incorporating frequency domain information to generate a hyperspectrum dataset corresponding to the scene from the RGB image sample dataset;

[0067] Step S3: Perform band selection on the entropy values of each band image in the hyperspectrum image, select three bands with the highest entropy values, and combine them to generate a pseudo-color image;

[0068]

[0069] ​​Step S4: inputting the pseudo-color image into the fixed detection network to obtain a target detection result.

[0070] The embodiment provides the acquisition of the RGB image in step S1, pre-processes the acquired image, and constructs an RGB image sample data set by using the pre-processed image, and the specific implementation method is as follows:

[0071] Step S101: collection of a hyperspectral data set, specifically, an unmanned aerial vehicle carrying an RGB camera is used to collect images, and the collected RGB image contains sea surface small target floating objects;

[0072] Step S102: pre-processing of the hyperspectral data set:

[0073] (1) the dimensions of the collected hyperspectral image are uniformly cropped to HxWx3, wherein H, W and 3 are the height, width and spectral channel number of the RGB image respectively;

[0074] (2) the collected RGB image is processed by using a cubic interpolation method, the size of the RGB image after reducing the spatial resolution is h x w x 3, and the RGB image after reducing the spatial resolution is used as an RGB data set;

[0075] Step S103: constructing a sample data set by using the pre-processed RGB image.

[0076] As shown in the figure, the structure of the spectrum reconstruction algorithm incorporating frequency domain information in step S2 of the embodiment includes: Figure 2 Step S201: a weighted window function convolution module, the weighted window function convolution module uses a Hamming window function, and the weight of a general convolution kernel is multiplied by the value of the Hamming window function, and the calculation formula of the Hamming window function is as follows:

[0077]

[0078] In the formula, N represents the number of discrete sampling points. In order to further improve the spectrum leakage problem, the convolution kernel size in the weighted window function convolution module is set to 7x7.

[0079] The input of the network is denoted as I RGB , and first, the shallow feature F SF is extracted by the weighted window function convolution module.

[0080] F SF =H Wconv1 (I RGB )

[0081] In the formula, H Wconv1 (·) represents a weighted window function convolution operation.

[0082]

[0083] ​Step S202: RGB image preprocessing module, for the degradation problem of input RGB image, the RGB image may be affected by noise, overexposure, compression, etc., resulting in inaccurate network prediction results. A multi-scale pyramid model is constructed by downsampling to solve the RGB image degradation problem. For the shallow features F SF as the input of the image preprocessing module.

[0084] F Pre Pre (F SF )

[0085] H Pre (·) represents the image preprocessing operation.

[0086] Step S203: the output F Pre of the image preprocessing module as the input of the DRCR module.

[0087]

[0088] F m-1 and F m represent the input and output of the mth DRCR block, respectively, represents the mth DRCR block operation.

[0089] DRCR module, each DRCR block contains twice the channel compression-excitation network module (Squeeze-and-Excitation Networks, SE-Net) and the U-shaped coding and decoding module. The main function of the SE-Net module is to integrate the feature relationship between channels and improve the learning ability of the network in the spectral dimension to achieve high-fidelity spectral recovery. The specific process is as follows.

[0090]

[0091] and represent the first and second SE-Net operations, respectively, and represent the outputs of the first and second SE-Net operations, respectively.

[0092] The main function of the U-shaped coding and decoding module is to explore the interaction characteristics of information in the same scale space and different spatial scales in the multi-scale structure to achieve high-fidelity spatial dimension information recovery. There are 6 convolution operations in the U-shaped coding and decoding module, and the mth convolution operation in the i th DRCR block is denoted as , and its operation output is F (m,i) .

[0093] ​​

[0094] Finally, the output of the last DRCR module is taken as the input of the last weighted window function convolution module, and the number of channels is mapped to 31 as the band data I of the final hyperspectral output HSI .

[0095] I HSI = H conv (F m )

[0096] where H conv (·) represents the convolution operation at the end of the network.

[0097] Step S204: The loss function of the residual dense network integrated with frequency domain learning is composed of two parts, one part is the loss function Loss1 describing the spatial-spectral dimensional information, and the other part is the loss function Loss2 describing the frequency domain information, and the total loss function L total is the linear superposition of the two.

[0098]

[0099] L total = Loos1+εLoss2

[0100] where and are the network output value and the true value of the i-th pixel value respectively, N pixels represents the number of pixel points of the hyperspectral data, FFT(·) represents the fast Fourier transform, and ε represents the balance coefficient of the two loss functions.

[0101] As Figure 3 shown, the step of synthesizing the pseudo-color image in step S3 of the embodiment includes:

[0102] Step S301: Group the 31-band hyperspectral image after spectral reconstruction of 400-700nm according to 400-500nm, 510nm-600 and 610-700nm;

[0103] Step S302: Select the one with the highest entropy value in each group according to the entropy calculation formula, and the entropy calculation formula is based on the definition of Shannon entropy (Shannon Entropy) and is suitable for discrete probability distribution. For a gray image, the definition of entropy is as follows:

[0104]

[0105] where H(X) is the entropy of the image, p(x i) is the probability of the gray level xi appearing in the image. k is the number of gray levels of the image, usually 256 gray levels (0 to 255) for an 8-bit gray image.

[0106] Step S303: synthesizing a pseudo-color image by channel for the three-band image after screening.

[0107] Step S4: inputting the three-modal fusion image into a fixed detection network to obtain a target detection result.

[0108] Examples:

[0109] Step S1: acquiring an RGB image, pre-processing the acquired image, and constructing an RGB image sample data by using the pre-processed image:

[0110] Step S101: using a drone with an RGB camera to record an RGB video:

[0111] Step S102: frame extraction and cropping of the video to integrate 400 RGB images with a dimension of 3200x2000x3 as a data set.

[0112] Step S103: using a cubic interpolation method to reduce the spatial resolution of the RGB image, and obtaining 400 RGB images with a dimension of 1600x1000x3.

[0113] Step S2, spectral super-resolution of the RGB data: selecting 80 RGB images to use an improved DRCR method with frequency domain information to perform spectral reconstruction on the input 3-band RGB image, and producing 80 hyperspectral images corresponding to the scene.

[0114] Step S3, pseudo-color image synthesis: selecting the three bands with the highest recognition rate from the hyperspectral images generated in step S2 to form a pseudo-color image.

[0115] Step S4, using a target detection model to detect and identify the original RGB image obtained in step S1 and the pseudo-color image obtained in step S3. As shown in Figure 5 , the detected sea surface small target floating object and the ship are in the box, and the corresponding category of the camouflage target is output on the box, and the confidence probability is 0.88, 0.85 and 0.84 respectively. As a comparison, Figure 6 Figure 5 The detection result of the visible light image corresponding to

[0116] Therefore, the hyperspectral image camouflage target detection method based on deep learning has the following beneficial effects:

[0117] ​The present application utilizes the RGB-based reconstruction hyperspectral to detect the small target floating object on the sea surface, and utilizes the spectral dimension information of the hyperspectral image to realize the detection of the small floating object on the sea surface.

[0118] In the present application, since the hyperspectral image can cover the 400nm-700nm wave band, it is beneficial to the target detection in low-illumination environment, and the use scene of the present application is expanded.

[0119] The movement of the target has no substantial influence on the scattered spectral information, so the present application is suitable for both the moving and static camouflage targets.

[0120] The present application can collect data sets and detect through the unmanned aerial vehicle equipped with an RGB camera, and greatly expands the detection range of the small target floating object on the sea surface.

[0121] The above merely describes the preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting small floating objects on the sea surface based on RGB reconstruction hyperspectral, characterized in that , comprising the following steps: Step S1: acquiring an RGB image, pre-processing the acquired image, and constructing an RGB image sample data using the pre-processed image; Step S2: generating a hyperspectral dataset corresponding to the scene from the RGB image sample dataset using a residual dense network fused with frequency domain information; Step S3: performing band selection on the entropy values of each band image in the hyperspectral image, and selecting three band combinations with the highest entropy values to generate a pseudo-color image; Step S4: inputting the pseudo-color image into a fixed detection network to obtain a target detection result; Step S2 comprises: Step S201: a weighted window function convolution module, which uses a Hamming window function to multiply the convolution kernel weight by the Hamming window function value, and the Hamming window function calculation formula is as follows: In the formula, n represents the index of the discrete sampling point, and N represents the number of discrete sampling points; the convolution kernel size in the weighted window function convolution module is set to 7x7; Let the input of the network be denoted as I RGB , and first extract the shallow features F SF by a weighted window function convolution module; F SF = H Wconv1 (I RGB ) where H Wconv1 (·) denotes a weighted window function convolution operation; Step S202: RGB image preprocessing module; for the shallow features F extracted in step S201 SF as the input of the image preprocessing module; F Pre = H Pre (F SF ) where H Pre (·) denotes an image pre-processing operation, F Pre denotes the output of the image pre-processing module; Step S203: the output F of the image pre-processing module is inputted into the DRCR module as an input of the DRCR module. Pre as an input of the DRCR module. where F m-1 and F m denote the input and output of the mthDRCR block, respectively, denotes the mthDRCR block operation, denotes the 1stDRCR block operation; DRCR module, each DRCR block contains twice channel compression-excitation network module SE-Net and U-shaped encoding-decoding module; the specific process of SE-Net module is as follows; wherein and denote the first and second SE-Net operation, respectively, and denote the output of the first and second SE-Net operation, respectively; The U-type coding module mainly functions to explore the interaction characteristics of the same scale space information and different spatial scale information in the multi-scale structure to realize the recovery of high-fidelity spatial dimension information. There are 6 convolution operations in the U-type coding module, and the i-th convolution operation of the m-th DRCR block is denoted as The output of the operation is F (m,i) ; wherein and represent the dual-channel recalibration CRM module operation in the network; Finally, the output of the last DRCR module is taken as the input of the last weighted window function convolution module, and the number of channels is mapped to 31 as the band data I of the final hyperspectral output HSI ; I HSI = H conv (F m ) where H conv (·) denotes a convolution operation at the end of the network; Step S204: the loss function of the residual dense network integrated with the frequency domain learning is composed of two parts, one part is a loss function Loss1 describing the spatial-spectral dimension information, and the other part is a loss function Loss2 describing the frequency domain information, and the total loss function L total is a linear superposition of the two. L total = Loosl + εLoss2 In the formula and respectively, the network output value and the true value of the i th pixel value, N pixels represents the number of pixel points of hyperspectral data, FFT(·) represents fast Fourier transform, and ε represents the balance coefficient of the two loss functions.

2. The method according to claim 1, wherein, The specific implementation method of step S1 is as follows: Step S101: collection of the hyperspectral dataset, specifically, using a UAV equipped with an RGB camera to collect images, and the collected RGB image contains sea surface small target floating objects; Step S102: pre-processing of the hyperspectral dataset; Step S103: constructing a sample dataset using the pre-processed RGB image.

3. The method according to claim 2, wherein, The specific implementation method of step S102 is as follows: Step S1021: uniformly cropping the dimensions of the collected hyperspectral image to HxWx3, wherein H, W and 3 are the height, width and spectral channel number of the RGB image, respectively; Step S1022: using a cubic interpolation method to process the collected RGB image, and the size of the RGB image after reducing the spatial resolution is h x w x 3, and the RGB image after reducing the spatial resolution is used as the RGB dataset.

4. The method according to claim 1, wherein the method is characterized in that The step of pseudo-color image synthesis in step S3 comprises: Step S301: grouping the 31 band hyperspectral images after spectral reconstruction in 400-500nm, 510nm-600 and 610-700nm; Step S302: selecting the one with the highest entropy value in each group according to the entropy calculation formula, and the entropy calculation formula is based on the definition of Shannon Entropy and is suitable for discrete probability distribution; for a gray image, the definition of entropy is as follows: where H(X) is the entropy of the image, p(x i ) is the probability of occurrence of gray level x i in the image, and k is the number of gray levels of the image, which is 256 for an 8-bit gray scale image. Step S303: synthesizing a pseudo-color image according to the channels of the three selected band images.

5. The method according to claim 1, wherein, The fixed detection network uses a Yolov5 network.

Images