A sea surface wind field multi-modal recognition method using an array air combined thermal image

Abstract

The application discloses a sea surface wind field multi-modal recognition method using an array type air joint thermal map, and comprises the following steps: acquiring the brightness value of the pixel of seawater samples and two kinds of oil film samples under different wind field conditions by using a push-broom spectral imager; calculating the relative reflectivity by radiation calibration based on the brightness value of the pixel; acquiring the z value of the hyperspectral cube matrix, so as to obtain the hyperspectral cube matrix and the dimension information of the hyperspectral cube matrix; processing the hyperspectral cube matrix according to a single frame slice, so as to perform dimension transformation on the hyperspectral cube matrix, and obtaining the hyperspectral array imaging of the instantaneous field of view within a certain time; inputting the three-dimensional array type imaging into a deep convolution network as input data to perform migration learning; the deep convolution backbone network extracts and enhances the features of the input data; the hyperparameters of the backbone network are fine-tuned; the network is efficiently migrated by using a migration pre-training optimal weight initialization strategy; and finally, the multi-modal recognition result of the sea surface oil film is obtained.

Description

Technical Field

[0001] This invention relates to the field of sea surface wind field identification technology, and in particular to a method for multimodal identification of sea surface wind fields using arrayed spatial joint heat maps. Background Technology

[0002] With the rapid development of marine transportation engineering and maritime transport, marine production activities are becoming increasingly frequent, leading to a growing risk of marine oil spills. Sea surface wind field modes are a crucial factor influencing the intensity of reflected radiation from oil slicks on the sea surface. Under multimodal wind field conditions, the sea surface exhibits different and complex optical reflected radiation intensity characteristics. Therefore, identifying wind field modes plays a vital role in oil spill detection. To describe the significant characteristics of the external modal environment, a spatial-spectral joint heatmap reflecting reflected radiation intensity from a remote sensing hyperspectral pixel array is used as input. A deep convolutional backbone network is employed for feature extraction and enhancement of the input data. Hyperparameter fine-tuning of the backbone network is performed, and efficient transfer learning is achieved through a transfer learning pre-training optimal weight initialization strategy. The final results show that array-based spatial-spectral joint heatmap data can accurately identify the sea surface wind field modes in the affected area and extract significant features of the sea surface wind field. The wind field mode identification results are stable and accurate, meeting the needs for rapid response to sea surface wind fields in oil spill areas. This approach has good application prospects and will generate significant social and economic benefits. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention discloses a method for multimodal identification of sea surface wind fields using array-type air-space joint heatmaps, specifically employing the following steps:

[0004] Pixel brightness values ​​of seawater samples and two types of oil film samples under different wind field conditions were acquired using a pushbroom spectral imager.

[0005] The relative reflectance is calculated by radiometric calibration based on the pixel brightness value, and the z-value of the hyperspectral cube matrix is ​​obtained, thereby obtaining the hyperspectral cube matrix and its dimensional information.

[0006] The hyperspectral cube matrix is ​​sliced ​​into single frames to perform dimensional transformation, thereby obtaining a hyperspectral array imaging of the instantaneous field of view over a certain period of time.

[0007] Three-dimensional array imaging is used as input data and passed to a deep convolutional network for transfer learning. The deep convolutional backbone network extracts and enhances features from the input data and performs hyperparameter fine-tuning on the backbone network. Through the optimal weight initialization strategy of transfer pre-training, the network is efficiently transferred to learn and obtain multimodal recognition results of oil film on the sea surface.

[0008] 2. The method for multimodal identification of sea surface wind fields using array-type spatial joint heatmaps according to claim 1, characterized in that: the acquisition of the hyperspectral cube matrix and the dimensional information of the hyperspectral cube matrix is ​​specifically carried out in the following manner:

[0009] S21: Using a standard reflector white board, perform radiation calibration on the original reflected radiation intensity to eliminate differences in radiation intensity between different solar elevation angles, measurement areas, and data measurement times, and to reduce measurement errors caused by the sensor itself. Calculate the relative reflectivity after radiation calibration using the following formula:

[0010] R i rel =k*DN n +c

[0011] R represents the relative reflected radiation intensity expressed as a pixel brightness value, k represents the gain, and c is the offset that includes errors caused by atmospheric scattering, absorption, reflection, and other factors.

[0012] S22: The measured array hyperspectral imaging data is divided into {s1,s2,s3…,s…} according to the sampling time. n-1 ,s n The sequence dimension at any time ti is}

[0013] 3. The method for multimodal recognition of sea surface wind fields using array-type spatial joint heatmaps according to claim 1, characterized in that: when using a deep convolutional backbone network to extract and enhance features from the input data: based on the optimal pre-trained weights, the cost function of the transfer model is calculated using the following formula:

[0014]

[0015]

[0016]

[0017] Among them l i The loss value represents the contribution of each training sample, and L is a vector consisting of N loss values. ic Let represent the 3D hyperspectral array imaging data input to the i-th model, the same as the number of samples. Let c represent the category of the input sample, c = {1, 2, 3, ..., C}, and let y be the index of the label category set element. i W represents the predicted label of the i-th 3D HSI cube. yi This represents the weight of the i-th predicted label.

[0018] By employing the aforementioned technical solution, this invention provides a multimodal identification method for sea surface wind fields using array-based spatial joint thermal imaging. This method utilizes a transfer learning framework, using single-array spectral imaging as feature input, for multimodal identification and quantitative inversion of oil slicks covering the sea surface under varying wind conditions. This method "collectively votes" on the current wind field conditions of the floating oil sea surface through the entire imaging array, performing "thermal" imaging of the reflected radiation intensity of one array to highlight the spatial distribution characteristics of the relative reflectivity arrays of different samples. Then, a transfer learning strategy is used to rapidly train the reflected radiation intensity array data. This strategy uses the most popular deep convolutional model, VGG16, for image texture feature extraction, possessing optimal pre-trained weights. Only a few batches of fine-tuning and iterations are required, demonstrating a strong ability to distinguish modes. Even with limited training data, it can effectively identify oil-water modal environments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the indoor experimental setup in this invention;

[0021] Figure 2 This is a flowchart of the transfer learning architecture in this invention;

[0022] Figure 3 is a schematic diagram of the relative reflectivity of oil film and seawater under three experimental wind field modes in this invention. Detailed Implementation

[0023] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

[0024] like Figure 1 The method for multimodal identification of sea surface wind fields using array-based air-space joint heatmaps, as shown, specifically includes the following steps:

[0025] S1: Data Acquisition: The pixel brightness values ​​(DN values) of seawater samples and two types of oil film samples under different wind field conditions were obtained through indoor experiments using a pushbroom spectral imager. The DN value can reflect the intensity of reflected radiation.

[0026] S11: Seawater samples were collected in March 2021 from Lingshui Port on the Yellow River in China. Marine diesel and Brazilian crude oil, commonly used in ocean transport, were used.

[0027] S12: Various types of oil samples were measured using a pipette, and the sample was evenly distributed on the surface of the seawater sample. The sample was then sealed and left to stand for approximately 30 minutes. After the oil sample had spread evenly, a stable oil film with a thickness of 5 micrometers was formed on the seawater surface.

[0028] S13: To minimize interference from scattered light from the surrounding ground, the entire experiment was conducted in a darkroom environment. To further simulate a real marine environment, the experimental sample container was surrounded by a deep blue sponge. The interior of the sample container was uniformly coated with a deep black silicon coating insoluble in organic solvents. A xenon lamp parallel light source and a pushbroom spectroscopic imager were mounted on a 2.5m radius hemispherical support to emit incident light and receive reflected signals, respectively. An industrial variable-speed fan was used to control the simulated sea surface wind speed environment, with three wind speed modes: no wind, 3m / s, and 5m / s. The fan was positioned 1.0m above the experimental container at a 270° wind direction angle. A three-cup anemometer was used to record environmental factors, such as... Figure 1 As shown.

[0029] S14: The three oil film states on the sea surface under the corresponding environmental conditions are calm, rough, and rougher, respectively. A detector array orthogonal to the scanning direction (parallel to the flight direction of the sensor) senses the reflected signal of the oil film on the sea surface and converts it into an electrical signal to obtain the hyperspectral pixel brightness value (DN value) of the entire row.

[0030] S15: After measuring a set of samples, the samples are removed and the detection locations are marked.

[0031] S16: Finally, place the standard reflective white board at the corresponding mark for spectral calibration. The hyperspectral imager used in the experiment has a spectral range of 392.00–1162.67 nm, including 1936 bands, with an average spectral resolution of 0.35 nm. Each sampling grating slit consists of 1456 pixels, and the resolution of a single pixel reaches 3.5e-03 m.

[0032] S17: Measure different oil samples for 10 seconds each time at the same 75-degree solar altitude angle. Repeat each measurement group 10 times, with the exposure time set to 100 milliseconds. Perform multiple repeated measurements to reduce data errors caused by external factors and experimental equipment.

[0033] S2: Data Processing: The relative reflectance, i.e., the z-value of the hyperspectral cube (HSI cube) matrix, is calculated from the pixel brightness value (DN value) through radiometric calibration, thus obtaining the hyperspectral cube (HSI cube) matrix. The experiment measures frames per second (fps), and the final measured dimension of the hyperspectral cube (HSI cube) matrix is ​​R. n*1456*1936 , where n represents a time period t0~tn The amount of internal hyperspectral data along the time dimension t i The superposition of.

[0034] S21: Using a standard reflector white board, perform radiation calibration on the original reflected radiation intensity to eliminate differences in radiation intensity between different solar elevation angles, measurement areas, and data measurement times, while reducing measurement errors caused by the sensor itself. Calculate the relative reflectivity after radiation calibration using the following formula:

[0035] R i rel =k*DN n +c

[0036] R represents the relative reflected radiation intensity expressed as a pixel brightness value, in units of 1. k represents the gain, here the reciprocal of the pixel grayscale value (brightness value) measured by the reflector. c is the offset that includes errors caused by atmospheric scattering, absorption, reflection, and other factors, set to zero here.

[0037] S22: The measured array hyperspectral imaging data is divided into {s1,s2,s3…,s…} according to the sampling time. n-1 ,s n The sequence dimension at any time ti is} The three coordinate axes of the hyperspectral cube matrix have the following meanings: x-axis—1456 pixels along the array direction of the sensor spectral slit ( Figure 1 x-axis; y-axis—1936 bands of multi-band visible-near infrared (0.39-1.06μm) Figure 1 The x-axis represents the relative reflectance of seawater or oil film, while the y-axis represents the relative reflectance of seawater or oil film. The experiment measured frames per second (fps), and the final measured hyperspectral cube (HSI cube) matrix had dimensions R. n*1456*1936 , where n represents a time period t0~t n The amount of internal hyperspectral data along the time dimension t i The superposition of.

[0038] S3: The hyperspectral cube (HSI cube) matrix is ​​sliced ​​into single frames, allowing for dimensional transformation to obtain a hyperspectral array image of the instantaneous field of view over 0.125 seconds, with a dimension of R. (ti～ti+1) 3*1456*1936 This refers to a hyperspectral array-type color thermogram with a dimension of 1456 along the array direction (x-axis) and a dimension of 1936 along the wavelength direction (y-axis), which is a three-dimensional array imaging.

[0039] S4: The 3D array imaging is used as input data and fed into a deep convolutional network for transfer learning. The deep convolutional backbone network can extract and enhance features from the input data, and perform hyperparameter fine-tuning on the backbone network. Through a transfer learning pre-training optimal weight initialization strategy, the network is efficiently transferred to obtain multimodal discrimination results for sea surface oil films. Using the transfer learning model, we ultimately obtain three different wind field modal information for the experimental seawater or two types of oil films. The final output is the discrimination probability for three wind field conditions (no wind, 3 m / s, 5 m / s). The index corresponding to the maximum expected value is selected as t. i ~t i+1 Wind speed v during the time period i+1 The final output layer is designed as a fully connected layer, and the model outputs probability vectors corresponding to the three experimental wind field conditions.

[0040] Furthermore, this application selects the relatively mature VGG16 transfer learning model, which features a filter combination of multiple deep convolutional and pooling layers. These modules can extract grayscale features between adjacent pixels in an array-like image. Simultaneously, as the depth of the convolutional and pooling layers increases, it can learn the correlation information between texture features of pixels at greater distances. This model uses two consecutive or two 3x3 convolutional kernels, which is equivalent to obtaining a larger receptive field. On the one hand, it reduces the number of parameters in the model while increasing the network depth. On the other hand, it adds non-linear mapping, optimizes model features based on input features, and enhances the expressive power of the network model through a large number of activation functions.

[0041] Furthermore, based on the VGG16 model, we selected the optimal pre-trained weights provided by the official documentation to enable fast and efficient transfer learning. Training a model from scratch requires a relatively complex initialization strategy, and poor initialization can lead to unstable gradients in the network, causing learning to stagnate. Therefore, we chose the optimal pre-trained weights provided by the official documentation, which allows the model to perform fast and efficient transfer learning.

[0042] Furthermore, in the VGG16 model, using the officially provided optimal pre-trained weights, the cost function for transferring the model is calculated using the following formula:

[0043]

[0044]

[0045]

[0046] Among them l i The loss value represents the contribution of each training sample, and L is a vector consisting of N loss values. icy represents the 3D hyperspectral array imaging data input to the i-th model, the same as the number of samples. c represents the category of the input sample, c = {1, 2, 3, ..., C}, and is an element of the index set of the label categories, excluding any ignored label indices. i W represents the predicted label for the i-th 3D HSI cube (hyperspectral 3D cube image). yi This represents the weight of the i-th predicted label.

[0047] Furthermore, by using a migration model, three different wind field modes for the experimental seawater and the two types of oil films were obtained, and the final output was R. 3 ti～ti+1 The discrimination probabilities of the three wind field conditions are selected, and the index corresponding to the maximum expected value is chosen to correspond to t. i ~t i+1 Wind speed v during the time period i+1 The final output layer is designed as a fully connected layer, with the outputs corresponding to the probability vectors of the three wind field conditions in the experiment.

[0048] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for multimodal identification of sea surface wind fields using arrayed spatial joint heatmaps, characterized in that... include: Pixel brightness values ​​of seawater samples and two types of oil film samples under different wind field conditions were acquired using a pushbroom spectral imager. The relative reflectance is calculated by radiometric calibration based on the pixel brightness value, and the z-value of the hyperspectral cube matrix is ​​obtained, thereby obtaining the hyperspectral cube matrix and its dimensional information. The hyperspectral cube matrix is ​​sliced ​​into single frames to perform dimensional transformation, thereby obtaining a hyperspectral array imaging of the instantaneous field of view over a certain period of time. Three-dimensional array imaging is used as input data and passed to a deep convolutional network for transfer learning. The deep convolutional backbone network extracts and enhances features from the input data and performs hyperparameter fine-tuning on the backbone network. Through the optimal weight initialization strategy of transfer pre-training, the network is efficiently transferred to learn and obtain multimodal recognition results of oil film on the sea surface.

2. The method for multimodal identification of sea surface wind fields using arrayed spatial joint heatmaps according to claim 1, characterized in that: The hyperspectral cube matrix and its dimensional information are obtained in the following manner: S21: Using a standard reflector white board, perform radiation calibration on the original reflected radiation intensity to eliminate differences in radiation intensity between different solar elevation angles, measurement areas, and data measurement times, and to reduce measurement errors caused by the sensor itself. Calculate the relative reflectivity after radiation calibration using the following formula: R i rel =k*DN n +c R represents the relative reflected radiation intensity expressed as a pixel brightness value, k represents the gain, and c is the offset that includes errors caused by atmospheric scattering, absorption, reflection, and other factors. S22: The measured array hyperspectral imaging data is divided into {s1,s2,s3…,s…} according to the sampling time. n-1 ,s n The sequence dimension at any time ti is} 3. The method for multimodal identification of sea surface wind fields using arrayed spatial joint heatmaps according to claim 1, characterized in that: When using a deep convolutional backbone network to extract and enhance features from input data: Based on the optimal pre-trained weights, the cost function of the transfer model is calculated using the following formula: l(s,y)=L={l1,l2,...,l n } T , Among them l i The loss value represents the contribution of each training sample, and L is a vector consisting of N loss values. ic Let represent the 3D hyperspectral array imaging data input to the i-th model, the same as the number of samples. Let c represent the category of the input sample, c = {1, 2, 3, ..., C}, and let y be the index of the label category set element. i W represents the predicted label of the i-th 3D HSI cube. yi This represents the weight of the i-th predicted label.

Images