Multi-modal feature deep learning fused internal solitary wave detection algorithm under high sea condition

Abstract

The invention relates to the technical field of ocean internal solitary wave detection, in particular to an internal solitary wave detection algorithm fusing multi-modal feature deep learning under a high sea condition, and the algorithm comprises the steps: collecting an internal solitary wave data set under the high sea condition, and extracting the spectrum features of internal solitary wave data; constructing an internal solitary wave detection model AspectNet under the high sea condition fused with adaptive two-dimensional spectrum analysis according to the spectrum features; a multi-modal internal solitary wave detection model VIT-Multi-ISWNet under the high sea condition is constructed through an internal solitary wave detection model AspectNet under the high sea condition in combination with a multi-modal space-time attention mechanism, and the multi-modal internal solitary wave detection model VIT-Multi-ISWNet under the high sea condition is used for detecting the position of an internal solitary wave under the high sea condition. The internal solitary wave detection method fusing multi-modal feature deep learning under the high sea condition is high in universality, and target detection of more ocean phenomena can be achieved by modifying different types of data set training.

Description

Technical Field

[0001] This invention relates to the field of ocean solitary wave detection technology, and in particular to an algorithm for detecting solitary waves under high sea states that integrates deep learning of multimodal features. Background Technology

[0002] Internal solitary waves (ISWs), a special type of internal wave, are characterized by large amplitude (up to 240 m vertical amplitude), high velocity (up to 3 m / s in the deep sea), and short period (ranging from tens of minutes). ISWs can induce significant interactions between seawater and the seabed in a short period, playing a crucial role in sediment suspension and redeposition. Furthermore, during propagation, ISWs induce shear flow in the water above and below the wave surface, causing strong convergence and divergence of water, which promotes the exchange of nutrients between different water layers. However, excessive mixing at high sea states can disturb local ecosystems, affecting marine habitats and ecological balance. Simultaneously, the violent surface fluctuations at high sea states can affect navigation and operations; if the combined effects of ISWs occur, the violent surface fluctuations may be even more intense. Therefore, research on ISW ​​detection at high sea states is of great significance for marine resource protection and the safety of offshore production operations.

[0003] Traditionally, ISW detection utilizes image processing techniques such as wavelet transform and edge detection, which are widely used for feature detection and classification of ISWs due to their simple computational process. For example, wavelet transform can distinguish ISWs from background noise and estimate energy and wavelength through a time-scale plane. However, traditional methods only demonstrate the comparison of recognition results for a few typical ISWs, lacking accuracy statistics for large-scale data, and require manual parameter adjustment to achieve optimal detection results, making them difficult to apply to large-scale, real-time analysis of multi-source data.

[0004] Deep learning methods, characterized by automation and speed, have become the primary method for extracting information from remote sensing images. To address the characteristics of ISWs (Independent Sound Swings), existing technologies employ various deep learning models to detect ISWs in remote sensing images. For example, some existing technologies use the multi-stream and post-fusing multimodal deep learning network SONet, combined with Sentinel-3 SRAL and OLCI data, achieving an ISW ​​detection accuracy of 92%; others combine U-Net and KDV (Korteweg–de Vries) to solve for ISW parameters, achieving an ISW ​​detection accuracy of 92.5%; still others use DCNN models to detect ISWs at different resolutions and from different satellite sensors, achieving an ISW ​​detection accuracy of 85.75%. However, these deep learning detection methods are mostly applicable to data samples with good image quality; after training with high sea state data, the accuracy on the validation set drops to over 50%.

[0005] Existing SAR methods for detecting internal waves mostly focus on ocean conditions with low wind and wave velocities. However, at high sea states, internal waves exhibit complex nonlinear coupling with parameters such as wind field, waves, and seabed topography. These factors collectively modulate the rough scattering surface of the sea surface, resulting in unclear SAR imaging and difficulty in accurate detection of internal waves at high sea states. Furthermore, these methods often neglect internal wave detection under high sea state conditions, thus severely underestimating the number of internal waves and consequently affecting the analysis of long-term trend changes in internal wave characteristics. Summary of the Invention

[0006] To address the aforementioned shortcomings, the present invention aims to provide an algorithm for detecting internal solitary waves under high sea state conditions that integrates deep learning of multimodal features.

[0007] To achieve the above objectives, this invention provides an algorithm for detecting internal solitary waves under high sea states by integrating multimodal feature deep learning. It collects internal solitary wave datasets under high sea states, extracts the spectral features of the internal solitary wave data, and constructs an AspectNet feature extraction model for internal solitary waves under high sea states based on the spectral features, integrating adaptive two-dimensional spectral analysis. Using the AspectNet feature extraction model, a multimodal high sea state internal solitary wave detection model, VIT-Multi-ISWNet, is constructed by combining the multimodal spatiotemporal attention mechanism. The VIT-Multi-ISWNet model is used to detect the location of internal solitary waves under high sea states.

[0008] As a preferred technical solution, the following steps are included:

[0009] Step 1: Collect internal solitary wave image data and sea state reanalysis data; and preprocess the internal solitary wave image data and sea state reanalysis data to obtain an internal solitary wave image dataset suitable for target detection and sea state reanalysis data suitable for spatiotemporal matching.

[0010] Step 2: Perform latitude and longitude spatiotemporal spatiotemporal matching between the sea state reanalysis data suitable for spatiotemporal matching and the internal solitary wave image dataset suitable for target detection to obtain the high sea state internal solitary wave dataset; divide the high sea state internal solitary wave dataset into training set and test set;

[0011] Step 3: Extract spectral features from the training set of internal solitary wave data, and construct AspectNet, a high sea state internal solitary wave feature extraction model that integrates adaptive two-dimensional spectral analysis, based on the spectral features.

[0012] Step 4: Construct a multimodal internal solitary wave detection model VIT-Multi-ISWNet for high sea states based on the internal solitary wave feature extraction model ASpectNet and multimodal features;

[0013] Step 5: Train the constructed multimodal high sea state internal solitary wave detection model VIT-Multi-ISWNet using the test set of the high sea state internal solitary wave dataset to achieve internal solitary wave location detection under high sea states.

[0014] As a preferred technical solution, in step 1, the internal solitary wave image data includes the publicly available internal solitary wave SAR dataset iws_datasets_v1 and the self-collected SAR dataset;

[0015] The preprocessing of the internal solitary wave image data is as follows: the collected internal solitary wave data is locally processed using an adaptive histogram equalization algorithm to obtain contrast-enhanced internal solitary wave images; the contrast-enhanced internal solitary wave images are then subjected to median filtering to obtain low-noise internal solitary wave images; further, the low-noise internal solitary wave images are cropped into background and target samples according to the deep learning image detection size, and some background samples are removed to make the ratio of target samples to background samples 1:4, thus obtaining an internal solitary wave image dataset suitable for target detection.

[0016] As a preferred technical solution, step 2 involves performing latitude and longitude spatiotemporal matching between the sea state reanalysis data and the internal solitary wave image dataset suitable for target detection.

[0017] Step 21: Convert the sea state reanalysis data into sea state classification data.

[0018] Step 22: Based on the latitude, longitude, and time of the internal solitary wave image dataset suitable for target detection, match the sea state level data corresponding to the region and time for each image, and calculate the weighted average sea state level for the regional sea state level data of each image using the central-weighted average method. Refer to formula (1) as the sea state level to which the image belongs;

[0019] (1)

[0020] In equation (1), N represents the number of points within the region where the internal solitary wave image is located. Let i be the weight of the i-th point. Let be the sea state level of the i-th point. The calculation method is based on formula (2).

[0021] (2)

[0022] In equation (2), Let be the distance from the i-th point to the center of the image. The distance from the image center to the farthest edge;

[0023] Step 23, based on the weighted average sea state level The internal solitary wave image dataset exceeding sea state level 5 is used as the internal solitary wave dataset under high sea state.

[0024] As a preferred technical solution, in step 3, the AspectNet feature extraction model for internal solitary waves under high sea states includes adaptive two-dimensional spectrum analysis and feature extraction.

[0025] Adaptive two-dimensional spectrum analysis decomposes the data in the training set into different modal images containing internal solitary wave signals and sea clutter noise.

[0026] Feature extraction is performed by using a convolutional neural network (CNN) module to extract features from the ISW modal images obtained through adaptive two-dimensional spectral analysis, resulting in spectral features of ISW modal images at different scales.

[0027] As a preferred technical solution, adaptive two-dimensional spectrum analysis is performed as follows:

[0028] Step 31: First, the internal solitary wave image data is decomposed into K different spectral images using 2DVMD, and the corresponding modal images are obtained through inverse Fourier transform; then, the optimal mode number K value calculation method is used to automatically calculate the residual between the reconstructed signal and the original signal under different numbers of modal images, and the effect of the number of each modal image is evaluated by mean square error, referring to equation (3):

[0029] (3)

[0030] In equation (3), For the original image, For the first A modal image, × Image size;

[0031] The optimal number of modes is selected as the K value that minimizes MSE, and the 2DVMD decomposition mode number parameter is adjusted to re-decompose and obtain K optimal mode images.

[0032] The spectral entropy of the K optimal modal images is calculated using the formula shown in equation (4):

[0033] (4)

[0034] in, This represents the DC component of the signal in the frequency spectrum. The order of the frequency component. The coefficients of the nth-order cosine wave are... The coefficients are those of an nth-order sine wave. Angular frequency, representing a frequency component The period length;

[0035] Step 32: The spectral entropy of different modal images is adaptively set to a threshold of 1.5 for adaptive threshold filtering. The calculation formula is shown in equation (5):

[0036] (5)

[0037] in, and Let H represent the mean and standard deviation of the spectral entropy H, respectively. It is an adjustable parameter used to control the sensitivity of the threshold.

[0038] As a preferred technical solution, in step 4, the method for constructing the VIT-Multi-ISWNet multimodal internal solitary wave detection model under high sea states is as follows: A spatiotemporal multimodal cross-attention mechanism is used to extract environmental element features, obtaining spatiotemporal information features of multimodal environmental elements; a cross-attention mechanism is used to fuse the ISW modal image spectral features extracted by the AspectNet feature extraction model under high sea states with the spatiotemporal information features of multimodal environmental elements, obtaining multimodal fusion features of internal solitary waves under high sea states; and a visual converter network is used to construct the VIT-Multi-ISWNet multimodal internal solitary wave detection model under high sea states.

[0039] As a preferred technical solution, the VIT-Multi-ISWNet multimodal internal solitary wave detection model under high sea states includes the AspectNet feature extraction model for internal solitary waves under high sea states, an environmental feature extraction module, a feature fusion module, and a detection module.

[0040] AspectNet, an internal solitary wave feature extraction model under high sea states, is used to obtain spectral features of ISW modal images.

[0041] The environmental feature extraction module uses a multimodal cross-attention mechanism to extract environmental element features and obtain the spatiotemporal information features of multimodal environmental elements.

[0042] The feature fusion module employs a cross-attention mechanism to fuse spatiotemporal information features of multimodal environmental elements. Spectral features of ISW modal images obtained by AspectNet, a feature extraction model for internal solitary waves under high sea states. The key vector K and value vector V corresponding to the cross-attention mechanism are generated from the spectral features of the ISW modality image and used to generate attention weights for different spatiotemporal environmental features. The spatiotemporal information features of multimodal environmental elements output by the environmental feature extraction module are used to generate the corresponding query vector Q. The attention mechanism weights are generated by performing dot product and softmax processing on Q and K to obtain the multimodal fusion features. ;

[0043] The detection module incorporates a visual converter network to process the multimodal fusion features obtained by the feature fusion module. The system performs perception, combines multi-head attention with bounding box detection, and obtains the final target detection result.

[0044] As a preferred technical solution, the environmental feature extraction module includes spatially separable convolutional networks, temporal convolutional networks, and spatiotemporal cross-attention mechanisms;

[0045] Spatial variation features of the thermocline within the suspected solitary wave region are extracted, and a spatially separable convolutional network is incorporated to extract features from the input thermocline information, thus obtaining the spatial features of the thermocline. See equation (6):

[0046] (6)

[0047] In formula (6) For activation function, For convolution operations, Data for the thermocline;

[0048] Temporal convolutional networks are used to extract the temporal features of sea surface wind speed and significant wave height. The time-series focus cutoff point t-1 is improved to time t, as shown in equation (7):

[0049] (7)

[0050] In equation (7) For wind speed and wave height data, For the i-th convolutional kernel, It is the expansion factor. The width of the convolution kernel;

[0051] The spatiotemporal multimodal attention mechanism module employs a cross-attention mechanism to integrate spatial features. and time characteristics By fusing the data, the spatiotemporal information features of multimodal environmental elements are obtained. The process first involves considering the time-series characteristics. Upsampled to spatial dimensions H×W, resulting in Then, calculate the query vector Q, key vector K, and value vector V required for cross-attention. The calculation process is shown in equation (8):

[0052] (8)

[0053] In equation (8) for Learnable weights for Learnable weights for Learnable weights; then calculate cross-attention. To obtain the spatiotemporal information features of multimodal environmental elements See equation (9);

[0054] (9)

[0055] In equation (9) Here, is the normalized exponential function, and D is the dimension of the environmental field parameters.

[0056] As a preferred technical solution, in step 5, the multimodal high sea state internal solitary wave detection model VIT-Multi-ISWNet is trained using the training set of the high sea state internal solitary wave dataset. The training loss function of the multimodal high sea state internal solitary wave detection model VIT-Multi-ISWNet adopts a modified classification loss function. As shown in equation (12):

[0057] (12)

[0058] in For the classification loss function, sample balancing weights are introduced. The first coefficient of the Taylor expansion .

[0059] This invention provides an algorithm for detecting internal solitary waves under high sea states by integrating multimodal feature deep learning. Compared with other mainstream internal solitary wave target detection methods, it integrates adaptive two-dimensional spectrum analysis and spatiotemporal multimodal cross-attention mechanism to achieve the detection of internal solitary waves under high sea states by integrating multimodal feature deep learning. This is of great significance for the comprehensive detection of internal solitary waves in the ocean and the study of their interaction with climate change. The internal solitary wave detection method under high sea states by integrating multimodal feature deep learning of this invention has strong universality and can achieve the target detection of more marine phenomena by modifying different types of datasets for training. Attached Figure Description

[0060] Figure 1 This is a flowchart of the internal solitary wave detection algorithm under high sea state conditions that integrates multimodal feature deep learning according to the present invention;

[0061] Figure 2This is a diagram showing the overall structure of AspectNet, an internal solitary wave detection model for high sea states.

[0062] Figure 3 This is a schematic diagram of the structure of the VIT-Multi-ISWNet model for detecting internal solitary waves under high sea states.

[0063] Figure 4 The image shows the results of adaptive two-dimensional spectral analysis in AspectNet, an internal solitary wave feature extraction model under high sea states. The first column is the time domain image, and the second column is the spectral image. (a) is the original image, (b) and (c) are ISW mode images, and (d) and (e) are background mode images.

[0064] Figure 5 This is a comparison of the detection performance of the VIT-Multi-ISWNet multimodal internal solitary wave detection model under high sea states with other mainstream ISW models. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0066] See Figure 1 This invention provides an algorithm for detecting internal isolated waves under high sea states by integrating multimodal feature deep learning. It involves collecting a dataset of internal isolated waves under high sea states, extracting the spectral features of the data, and constructing an AspectNet feature extraction model that integrates adaptive two-dimensional spectral analysis based on these features. Then, using the AspectNet model, a multimodal detection model for internal isolated waves under high sea states, VIT-Multi-ISWNet is constructed by combining it with a multimodal spatiotemporal attention mechanism. The VIT-Multi-ISWNet model is used to detect the location of internal isolated waves under high sea states. Employing this algorithm, which integrates multimodal feature deep learning, can improve the effectiveness of deep learning in learning the features of internal isolated waves under high sea states, thereby improving the accuracy and precision of internal isolated wave detection under high sea states. Specifically, the algorithm includes the following steps:

[0067] Step 1: Collect internal solitary wave image data and sea state reanalysis data; and preprocess the internal solitary wave image data and sea state reanalysis data to obtain an internal solitary wave image dataset suitable for target detection and sea state reanalysis data suitable for spatiotemporal matching.

[0068] Internal solitary wave image data includes the publicly available internal solitary wave SAR dataset iws_datasets_v1 and the self-collected SAR dataset;

[0069] The dataset iws_datasets_v1 in this embodiment includes 390 satellite images and 775 labels collected from Sentinel-1 synthetic aperture radar (SAR) from 2014 to 2021; among them, it contains internal solitary wave samples from the Andaman Sea, the South China Sea, the Sulu Sea and the Celebes Sea.

[0070] In this embodiment, the self-collected SAR dataset under high sea state conditions selects Sentinel-1 data from the South China Sea and Andaman Sea, which are hotspots of internal solitary waves, to collect a total of 401 L2-level GRD data scenes, covering the period from January 2024 to June 2024.

[0071] The sea state reanalysis data all use the ERA5 hourly global reanalysis dataset. This dataset provides hourly wind field and wave data at 1 / 10° intervals, including wind field (wind speed, etc.) and wave data (significant wave height, etc.).

[0072] The preprocessing of sea state reanalysis data is as follows: the data is split and stored according to the latitude and longitude ranges of the Andaman Sea, the South China Sea, the Sulu Sea and the Celebes Sea, and then stored in three-level folders according to year, month and day, to obtain sea state reanalysis data suitable for spatiotemporal matching.

[0073] The preprocessing process for the internal solitary wave image data is as follows: the collected internal solitary wave data is locally processed using an adaptive histogram equalization algorithm to obtain an internal solitary wave image with enhanced contrast; the internal solitary wave image with enhanced contrast is then subjected to median filtering to obtain a low-noise internal solitary wave image.

[0074] Furthermore, the low-noise internal solitary wave image size is cropped to 640×640, a commonly used size for deep learning image detection, to create background and target samples. Excessive background samples are removed, resulting in a target sample to background sample ratio of 1:4, thus obtaining an internal solitary wave image dataset suitable for target detection.

[0075] Step 2: Perform latitude and longitude spatiotemporal spatiotemporal matching between the sea state reanalysis data suitable for spatiotemporal matching and the internal solitary wave image dataset suitable for target detection to obtain the high sea state internal solitary wave dataset; divide the high sea state internal solitary wave dataset into training set and test set.

[0076] The spatiotemporal matching process involves converting sea state reanalysis data into sea state classification data. This specifically includes the following steps:

[0077] Step 21: Convert the sea state reanalysis data applicable to spatiotemporal matching in Step 1 into sea state grade data according to the correspondence relationship, as shown in Table 1.

[0078] Table 1: Correspondence between Ocean Wind Speed, Significant Wave Height, and Sea State Class

[0079]

[0080] Step 22: Based on the latitude, longitude, and time of the internal solitary wave image dataset suitable for target detection in Step 1, match the sea state level data corresponding to the region and time for each image, and calculate the weighted average sea state level for the regional sea state level data of each image using the central-weighted average method. Refer to formula (1) as the sea state level to which the image belongs;

[0081] (1)

[0082] In equation (1), N represents the number of points within the region where the internal solitary wave image is located. Let i be the weight of the i-th point. Let be the sea state level of the i-th point. The calculation method is based on formula (2).

[0083] (2)

[0084] In equation (2), Let be the distance from the i-th point to the center of the image. This is the distance from the center of the image to the farthest edge.

[0085] Step 23, based on the weighted average sea state level The image dataset of internal solitary waves exceeding sea state 5 was used as the internal solitary wave dataset under high sea state. Referring to Table 1, the wind speed corresponding to sea state 5 is 8.224-9.766 m / s, and the significant wave height is 2.5-4 m.

[0086] Step 3: Extract spectral features from the training set of internal solitary wave data, and construct AspectNet, a high sea state internal solitary wave feature extraction model that integrates adaptive two-dimensional spectral analysis, based on the spectral features.

[0087] The overall structure of AspectNet, the feature extraction model for internal solitary waves under high sea states, is as follows: Figure 2 As shown, it includes adaptive two-dimensional spectrum analysis and feature extraction.

[0088] After adaptive two-dimensional spectral analysis, with the K value adaptively selected as 4, different modal images and corresponding spectral references were obtained. Figure 4 The adaptive two-dimensional spectrum analysis is calculated as follows:

[0089] Step 31: Through adaptive two-dimensional spectrum analysis, the data in the training set is decomposed into different modal images containing internal solitary wave signals and sea clutter noise to improve the signal-to-noise ratio of the samples. Adaptive two-dimensional spectrum analysis: First, through 2DVMD, the internal solitary wave image data is decomposed into K different spectral images, and the corresponding modal images are obtained through inverse Fourier transform; then, the optimal mode number K value calculation method is adopted to automatically calculate the residual between the reconstructed signal and the original signal under different numbers of modal images, and the effect of the number of each modal image is evaluated by mean square error (MSE), referring to equation (3).

[0090] (3)

[0091] In equation (3), For the original image, For the k-th modal image, × This refers to the image size.

[0092] The optimal number of modes is selected as the K value that minimizes the MSE. The 2DVMD decomposition mode number parameter is adjusted, and the K optimal mode images are obtained by re-decomposition.

[0093] The spectral entropy of the K optimal modal images is calculated using the formula shown in equation (4):

[0094] (4)

[0095] The instability of its image features is obtained through equation (1), where, This represents the DC component of the signal in the frequency spectrum. The order of the frequency component. The coefficients of the nth-order cosine wave are... The coefficients are those of an nth-order sine wave. Angular frequency, representing a frequency component The period length is denoted as .

[0096] Step 32: The spectral entropy of different modal images is adaptively set to a threshold of 1.5 for adaptive threshold filtering. The calculation formula is shown in equation (5):

[0097] (5)

[0098] in, and Let H represent the mean and standard deviation of the spectral entropy H, respectively. This is an adjustable parameter used to control the sensitivity of the threshold. Since the spectral characteristics of ISW modal images are relatively fixed in shape compared to sea clutter, and their spectral instability is lower, the spectrum of the first modal image below the threshold is selected as the extraction result.

[0099] Feature extraction is performed as follows: First, a convolutional neural network (CNN) module is used to extract features from the ISW modal images obtained by adaptive two-dimensional spectral analysis, resulting in spectral features of ISW modal images at different scales.

[0100] Step 4: Construct a multimodal internal solitary wave detection model VIT-Multi-ISWNet for high sea states based on the ASpectNet feature extraction model for internal solitary waves under high sea states and multimodal features such as environmental factors.

[0101] The construction method of the VIT-Multi-ISWNet multimodal internal solitary wave detection model under high sea states is as follows: A spatiotemporal multimodal cross-attention mechanism is used to extract environmental element features, obtaining spatiotemporal information features of multimodal environmental elements; a cross-attention mechanism is used to fuse the ISW modal image spectral features extracted by the AspectNet high sea state internal solitary wave feature extraction model with the spatiotemporal information features of multimodal environmental elements to obtain multimodal fusion features of internal solitary waves under high sea states; and the VIT-Multi-ISWNet multimodal internal solitary wave detection model under high sea states is constructed by combining it with a visual converter network.

[0102] The overall structure of the VIT-Multi-ISWNet multimodal internal solitary wave detection model under high sea states is as follows: Figure 3 As shown, it includes the AspectNet feature extraction model for internal solitary waves under high sea states, an environmental feature extraction module, a feature fusion module, and a detection module.

[0103] Step 41, AspectNet, an internal solitary wave feature extraction model under high sea states, refers to step 3 to obtain the spectral features of the ISW modal image;

[0104] Step 42, the environmental feature extraction module, uses a multimodal cross-attention mechanism to extract environmental element features and obtain the spatiotemporal information features of multimodal environmental elements, including spatially separable convolutional networks, temporal convolutional networks, and spatiotemporal cross-attention mechanisms.

[0105] To effectively extract the spatial variation features of the thermocline within the suspected solitary wave region, a spatially separable convolutional network is incorporated to extract features from the input thermocline information, thus obtaining the spatial features of the thermocline. See equation (6).

[0106] (6)

[0107] In formula (6) For activation function, For convolution operations, This is data from the thermocline.

[0108] Simultaneously, a temporal convolutional network is used to extract the temporal features of sea surface wind speed and significant wave height. The time-series focus cutoff point t-1 is improved to time t, as shown in equation (7), thus realizing more complete temporal features. Obtain.

[0109] (7)

[0110] In equation (7) For wind speed and wave height data, For the i-th convolutional kernel, It is the expansion factor. is the width of the convolution kernel.

[0111] Finally, the spatiotemporal multimodal attention mechanism module employs a cross-attention mechanism to integrate spatial features. and time characteristics By fusing the data, the spatiotemporal information features of multimodal environmental elements are obtained. This process first involves analyzing the temporal characteristics. Upsampled to spatial dimensions H×W, resulting in Then, calculate the query vector Q, key vector K, and value vector V required for cross attention. The calculation process is shown in Equation (8).

[0112] (8)

[0113] In equation (8) for Learnable weights for Learnable weights for Learnable weights are then used to calculate cross-attention. To obtain the spatiotemporal information features of multimodal environmental elements See equation (9).

[0114] (9)

[0115] In equation (9) Here, is the normalized exponential function, and D is the dimension of the environmental field parameters.

[0116] The feature fusion module employs a cross-attention mechanism to fuse spatiotemporal information features of multimodal environmental elements. Spectral features of ISW modal images obtained by AspectNet, a feature extraction model for internal solitary waves under high sea states. The key vector K and value vector V corresponding to the cross-attention mechanism are generated from the spectral features of the ISW modality image and used to generate attention weights for different spatiotemporal environmental features. The spatiotemporal information features of multimodal environmental elements output by the environmental feature extraction module are used to generate the corresponding query vector Q. The attention mechanism weights are generated by performing dot product and softmax processing on Q and K to obtain the multimodal fusion features. The calculation process is shown in equations (10) and (11).

[0117] (10)

[0118] (11)

[0119] The detection module incorporates a visual converter network to process the multimodal fusion features obtained by the feature fusion module. The system performs perception, combines multi-head attention with bounding box detection, and obtains the final target detection result.

[0120] Step 5: Train the constructed multimodal high sea state internal solitary wave detection model VIT-Multi-ISWNet using the high sea state internal solitary wave dataset to achieve internal solitary wave location detection under high sea states.

[0121] The training set portion of the high sea state internal solitary wave dataset was used to train the multimodal high sea state internal solitary wave detection model VIT-Multi-ISWNet. The training loss function for the VIT-Multi-ISWNet model was a modified classification loss function. As shown in equation (12):

[0122] (12)

[0123] in For the classification loss function, sample balancing weights are introduced. The first coefficient of the Taylor expansion This addresses the issues of imbalanced positive and negative samples and low contribution from difficult-to-classify samples during the training process.

[0124] The trained VIT-Multi-ISWNet model was used to detect the data on the test set, and the results are as follows: Figure 5 As shown.

[0125] This invention provides an algorithm for detecting internal solitary waves under high sea states by integrating multimodal feature deep learning. Compared with other mainstream internal solitary wave target detection methods, it integrates adaptive two-dimensional spectrum analysis and spatiotemporal multimodal cross-attention mechanism to achieve internal solitary wave detection under high sea states by integrating multimodal feature deep learning, thereby improving the detection accuracy of internal solitary waves under high sea states. This can compensate for the lack of detection of internal solitary waves in the ocean and correct the research on their interaction with climate change. The internal solitary wave detection method for high sea states by integrating multimodal feature deep learning of this invention has strong universality and can achieve target detection of more marine phenomena by modifying different types of datasets for training.

[0126] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A high sea state internal solitary wave detection algorithm fusing multi-modal feature deep learning, characterized in that, The data set of internal solitary waves under high sea conditions is collected, the spectral characteristics of the internal solitary wave data are extracted, and an internal solitary wave feature extraction model AspectNet under high sea conditions is constructed by fusing adaptive two-dimensional spectrum analysis according to the spectral characteristics; a multi-modal internal solitary wave detection model VIT-Multi-ISWNet under high sea conditions is constructed by combining a multi-modal spatio-temporal attention mechanism through the internal solitary wave feature extraction model AspectNet under high sea conditions, and the multi-modal internal solitary wave detection model VIT-Multi-ISWNet under high sea conditions is used for detecting the position of the internal solitary wave under high sea conditions.

2. The high sea state internal solitary wave detection algorithm of fusing multi-modal feature deep learning according to claim 1, characterized in that, The method comprises the following steps: Step 1, collecting internal solitary wave image data and sea condition reanalysis data; and pre-processing the internal solitary wave image data and the sea condition reanalysis data to obtain an internal solitary wave image data set suitable for target detection and sea condition reanalysis data suitable for spatio-temporal matching; Step 2, performing longitude and latitude spatio-temporal matching on the sea condition reanalysis data suitable for spatio-temporal matching and the internal solitary wave image data set suitable for target detection to obtain a high-sea internal solitary wave data set; and dividing the high-sea internal solitary wave data set into a training set and a test set; Step 3, extracting spectral characteristics of the internal solitary wave data from the training set, and constructing an internal solitary wave feature extraction model AspectNet under high sea conditions by fusing adaptive two-dimensional spectrum analysis according to the spectral characteristics; Step 4, constructing a multi-modal internal solitary wave detection model VIT-Multi-ISWNet under high sea conditions according to the internal solitary wave feature extraction model AspectNet under high sea conditions and multi-modal features; Step 5, training the constructed multi-modal internal solitary wave detection model VIT-Multi-ISWNet under high sea conditions through the test set of the high-sea internal solitary wave data set to realize internal solitary wave position detection under high sea conditions.

3. The high sea state internal solitary wave detection algorithm of claim 2, wherein, In step 1, the internal solitary wave image data comprises an internal solitary wave SAR public data set iws_datasets_v1 and a self-collected SAR data set; The process of pre-processing the internal solitary wave image data is as follows: the collected internal solitary wave data is locally processed by using an adaptive histogram equalization algorithm to obtain an internal solitary wave image with improved contrast; the internal solitary wave image with improved contrast is subjected to median filtering to obtain a low-noise internal solitary wave image; and the low-noise internal solitary wave image is further cropped to background samples and target samples according to the size of the deep learning image detection size, and part of the background samples are removed so that the ratio of the target samples to the background samples is 1:4, thereby obtaining an internal solitary wave image data set suitable for target detection.

4. The high sea state internal solitary wave detection algorithm of claim 2, wherein, In step 2, the process of performing longitude and latitude spatio-temporal matching on the sea condition reanalysis data and the internal solitary wave image data set suitable for target detection is as follows: Step 21, converting the sea condition reanalysis data into sea condition level data, Step 22, according to the longitude and latitude, time of the internal solitary wave image data set suitable for target detection, match out the sea state level data corresponding to the region and time of each image, and adopt the central key weighting average method to calculate the weighted average sea state level data of each image , as the sea state level to which the image belongs, according to formula (1) （1） In formula (1), N is the number of points in the region where the internal solitary wave image is located, is the weight of the i-th point, is the sea state level of the i-th point. The calculation method refers to formula (2), （2） In formula (2), is the distance of the i-th point to the image center, is the distance of the image center to the farthest edge; Step 23, determining the weighted average sea state rating from the sea state ratings The internal solitary wave image data sets exceeding sea state rating 5 are taken as the internal solitary wave data sets in high sea states.

5. The high sea state internal solitary wave detection algorithm of claim 2, wherein, In step 3, the internal solitary wave feature extraction model AspectNet under high sea conditions comprises adaptive two-dimensional spectrum analysis and feature extraction; The adaptive two-dimensional spectrum analysis decomposes the data in the training set into different modal images containing internal solitary wave signals and sea clutter noise through adaptive two-dimensional spectrum analysis; Feature extraction, the process is: adopting a convolutional neural network CNN module to extract features of the ISW modal image obtained by adaptive two-dimensional spectral analysis, to obtain ISW modal image spectrum features of different scales.

6. The high sea state internal solitary wave detection algorithm of claim 5, wherein, Adaptive two-dimensional spectral analysis, the process is as follows: Step 31, first, the internal solitary wave image data is decomposed into K different spectral images by 2DVMD, and the corresponding modal images are obtained by inverse Fourier transform; Then, the optimal mode number K value calculation method is used to automatically calculate the reconstruction signal and the original signal residual under different modal image quantity, and the effect of each modal image quantity is evaluated by mean square error, referring to formula (3): （3） In equation (3), For the original image, For the first A modal image, × Image size; Select the K value that makes the MSE minimum as the optimal modal number, and adjust the 2DVMD decomposition modal quantity parameter to obtain K optimal modal images; The spectral entropy of the K optimal modal images is calculated, and the calculation formula is shown in formula (4): （4） wherein a direct current component of the representative signal in the frequency spectrum, an order of the representative frequency component, is an n-th order cosine wave coefficient, is an n-th order sine wave coefficient, is an angular frequency of the representative frequency component, is a period length; Step 32, the spectral entropy of different modal images is adaptively selected as a threshold of 1.5 for adaptive threshold filtering, and the calculation formula is shown in formula (5): （5） wherein, and respectively denote the mean and standard deviation of the spectral entropy H, is an adjustable parameter that controls the sensitivity of the threshold.

7. The high sea state internal solitary wave detection algorithm of claim 2, wherein, In step 4, the construction method of the multi-modal high sea state internal solitary wave detection model VIT-Multi-ISWNet is: adopting a spatiotemporal multi-modal cross attention mechanism to extract environmental element features and obtain spatiotemporal information features of multi-modal environmental elements; Adopting a cross attention mechanism to extract multi-modal fusion features of high sea state internal solitary waves by fusing ISW modal image spectrum features extracted by the internal solitary wave feature extraction model AspectNet under high sea state and spatiotemporal information features of multi-modal environmental elements, and combining a visual transformer network to construct a multi-modal high sea state internal solitary wave detection model VIT-Multi-ISWNet.

8. The high sea state internal solitary wave detection algorithm of claim 7, wherein, The multi-modal high sea state internal solitary wave detection model VIT-Multi-ISWNet includes a high sea state internal solitary wave feature extraction model AspectNet, an environmental feature extraction module, a feature fusion module, and a detection module. The high sea state internal solitary wave feature extraction model AspectNet is used to obtain ISW modal image spectrum features. The environmental feature extraction module adopts a multi-modal cross attention mechanism to extract environmental element features and obtain spatiotemporal information features of multi-modal environmental elements. The feature fusion module adopts a cross attention mechanism to fuse the spatiotemporal information features of the multi-modal environmental elements and the ISW modal image spectrum features obtained by the internal solitary wave feature extraction model AspectNet in high sea states ; the ISW modal image spectrum features are input into the key vector K and the value vector V corresponding to the cross attention mechanism to generate attention weights of different spatiotemporal environmental features; the multi-modal environmental element spatiotemporal information features output by the environmental feature extraction module are generated into a query vector Q, and the dot product of Q and K is processed by softmax to generate an attention mechanism weight, thereby obtaining multi-modal fusion features ; The detection module fuses the multi-modal fusion features obtained by the feature fusion module into a visual transformer network The perception is performed in combination with multi-head attention and a bounding box detection to obtain a final target detection result.

9. The high sea state internal solitary wave detection algorithm of claim 8, wherein, The environmental feature extraction module includes a spatial separable convolution network, a time series convolution network, and a spatiotemporal cross attention mechanism. The spatial variation characteristics of the thermocline in the suspicious area of internal solitary waves are extracted, and a spatial separable convolution network is used to extract the thermocline information of an input to obtain thermocline spatial features See equation (6): （6） In formula (6) is an activation function, is a convolution operation, is the thermocline data; The time series convolution network is used to extract the time series features of the sea surface wind speed and the significant wave height, and the time series attention cutoff point t-1 is improved to t, as shown in formula (7): （7） In formula (7) is the wind speed, wave height data, is the i-th convolution kernel, is the dilation factor, is the convolution kernel width; The spatio-temporal multi-modal attention mechanism module adopts a cross-attention mechanism to fuse spatial features and temporal features to obtain spatio-temporal information features of multi-modal environmental elements . The process first upsamples the temporal features to the spatial size HxW to obtain , and then calculates the query vector Q, the key vector K and the value vector V required by the cross-attention, and the calculation process is shown in formula (8): （8） In formula (8) is a learnable weight, is a learnable weight, is a learnable weight, is a learnable weight, is a learnable weight, is a learnable weight; and cross-attention is calculated to obtain the multi-modal environmental element spatio-temporal information feature , see formula (9); （9） In formula (9) is a normalized exponential function, and D is the dimension of the environmental field parameter.

10. The high sea state internal solitary wave detection algorithm of claim 2, wherein, In step 5, the training set of the high sea state internal solitary wave data set is used to train the multi-modal high sea state internal solitary wave detection model VIT-Multi-ISWNet, and the loss function of the multi-modal high sea state internal solitary wave detection model VIT-Multi-ISWNet is the modified classification loss function As shown in formula (12): （12） wherein is a classification loss function, introducing sample-balancing weights and the first term coefficient of the Taylor expansion .

Images