Ocean internal wave extraction method, system and program product for complex environmental background

Abstract

The present invention belongs to the technical field of ocean internal wave extraction, and specifically discloses an ocean internal wave extraction method, system and program product for complex environmental backgrounds, by preprocessing the collected polarization SAR image data, and then using SAR data of different polarization channels to construct a false color image, increase the image information characterizing the internal wave characteristics, and then use the false color image to train and test the internal wave automatic extraction network that integrates the spatiotemporal attention mechanism and the U‑Net network and adopts a multi-scale loss function, so as to realize the automatic extraction of the ocean internal wave crest line using the ocean internal wave extraction network. The present invention can greatly improve the efficiency of ocean internal wave remote sensing detection, realize efficient and real-time ocean internal wave detection, overcome the shortcomings of insufficient recognition accuracy and stability of traditional single-channel SAR internal wave detection methods, broaden the application space of SAR technology in ocean internal wave monitoring, and is beneficial to marine disaster prevention and mitigation and ensuring the safety of submarine underwater navigation.

Description

Technical Field

[0001] The present invention belongs to the technical field of ocean internal wave extraction, and in particular relates to an ocean internal wave extraction method, system and program product facing complex environmental backgrounds. Background Art

[0002] Ocean internal waves are a type of gravity wave in the ocean, usually occurring between water layers with uneven seawater density, with a maximum amplitude of hundreds of meters. The propagation of internal waves plays an important role in seawater mixing and sediment resuspension, but it also affects the propagation direction of sound signals in the ocean, thereby reducing the effectiveness of sonar use and causing accidents such as "submarine falling deep", which has a significant impact on offshore oil drilling operations and underwater navigation safety. Detecting and extracting ocean internal waves can further deepen the understanding of the generation and propagation mechanism of internal waves, and prevent a series of catastrophic events caused by internal wave propagation, which is one of the important links in marine disaster prevention and mitigation.

[0003] Traditional ocean internal wave detection is mainly based on field observation of buoys, but this method has insufficient spatial resolution of single-point measurement devices, is expensive and inefficient, and is not suitable for internal wave detection and extraction in large sea areas. In recent years, with the rapid development of remote sensing earth observation technology, synthetic aperture radar (SAR) has been widely used in ocean internal wave observation due to its all-day and all-weather observation capabilities. Internal waves can cause changes in sea surface roughness during propagation, which is reflected in the SAR amplitude image as light and dark stripes. However, the conventional single-channel SAR image internal wave extraction method is affected by SAR image speckle noise, and the complex and diverse marine environment and atmospheric conditions, including typhoons, mesoscale vortices and other phenomena, will interfere with the accuracy of SAR image internal wave extraction. Unlike single-channel SAR systems, polarimetric SAR systems can transmit and receive electromagnetic waves in both vertical and horizontal directions to obtain SAR images of multiple polarization channels. Different types of ground objects have different scattering characteristics and mechanisms, and the polarization response characteristics of the echo signals generated under various polarization modes are also different. Therefore, polarimetric SAR data can characterize the scattering type and geometric structure of the target ground objects to a certain extent, which helps to distinguish ocean internal waves from other similar ocean phenomena and improve the accuracy of internal wave detection and extraction in complex background environments.

[0004] However, the generation and propagation of internal waves are low-probability phenomena in time and space, and the remote sensing images covering the ocean have a data volume of tens of TB every day. How to achieve real-time and efficient automatic detection and extraction of internal waves from massive data is the primary problem of remote sensing internal wave observation. In recent years, deep learning methods represented by convolutional neural networks (CNN) have achieved great success in image processing fields such as classification, semantic segmentation and target recognition. Similarly, deep learning methods have great potential in automatic interpretation of remote sensing images and detection of internal solitary waves. At present, the internal wave remote sensing detection methods combined with deep learning can be mainly divided into target detection and semantic segmentation according to their principles. Among them, the internal wave detection method based on the principle of semantic segmentation can extract the specific location information of the internal wave crest line, which has more advantages in production and practical application. However, the network structure of the current internal wave detection method based on the principle of semantic segmentation is simple, and the extraction of internal wave crest line is regarded as a simple binary classification problem, ignoring the linear relationship between the various parts of the internal wave crest line, resulting in insufficient accuracy and stability of automatic extraction of internal wave crest line, especially in remote sensing images under complex background environments. The application effect is further reduced. Therefore, there is an urgent need for an efficient and cost-effective method for automatic extraction of ocean internal waves in complex background environments. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system and program product for extracting ocean internal waves in a complex environment background, so as to solve the above-mentioned problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] First, a method for extracting ocean internal waves in a complex environment is provided, including:

[0008] Collecting a number of polarization SAR image data for the ocean, and performing image preprocessing on each polarization SAR image data to obtain corresponding preprocessed polarization SAR image data;

[0009] Perform multi-channel polarization data false color image synthesis on each pre-processed polarization SAR image data to obtain a corresponding polarization false color image;

[0010] Obtaining a manual marking instruction from a user, and marking the internal wave peak lines of each polarization false color image in response to the manual marking instruction, to obtain a corresponding marked internal wave polarization image sample;

[0011] Divide all internal wave polarization image samples into training set and test set according to the set ratio;

[0012] A preset internal wave automatic extraction network is trained using a training set until a set training condition is met, thereby obtaining a trained internal wave automatic extraction network, wherein the internal wave automatic extraction network includes a U-Net network, and a spatiotemporal attention module is connected in series after each convolution block of an encoder part of the U-Net network, wherein the spatiotemporal attention module includes a spatial attention module and a channel attention module, and a loss function of the internal wave automatic extraction network adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function, and a Matthews correlation coefficient loss function;

[0013] The trained automatic internal wave extraction network is tested using a test set until the set test conditions are met, thereby obtaining a tested automatic internal wave extraction network, and the tested automatic internal wave extraction network is used as an ocean internal wave extraction network;

[0014] The target ocean polarimetric SAR image is input into the ocean internal wave extraction network to extract the ocean internal waves, obtain the corresponding ocean internal wave extraction result, and output the ocean internal wave extraction result of the target ocean polarimetric SAR image.

[0015] In a possible design, the image preprocessing of each polarization SAR image data includes: performing data format conversion, data correction, data reprojection, data stitching and mosaicking, image filtering and image enhancement processing on each polarization SAR image data.

[0016] In a possible design, the step of labeling the internal wave peak lines of each polarization false color image in response to the manual labeling instruction includes:

[0017] According to the manual annotation instructions, the image area of ​​the internal wave crest line is annotated on the polarization false color image, and the pixel values ​​of the pixels in the image area of ​​the internal wave crest line are annotated as 1, and the pixel values ​​of the pixels outside the image area of ​​the internal wave crest line are annotated as 0.

[0018] In a possible design, before using the training set to train the preset internal wave automatic extraction network, the method further includes:

[0019] Construct a U-Net network, wherein the U-Net network includes an encoder part and a decoder part, wherein the encoder part includes four convolutional layers connected in series, and each convolutional layer includes two 3×3 convolutional blocks;

[0020] When constructing the encoder part, a spatiotemporal attention module is connected in series after each convolution block, and the spatiotemporal attention module includes a spatial attention module and a channel attention module;

[0021] The spatial attention mechanism expression of the spatial attention module is:

[0022] Wc =softmax((MaxPool(F))+MLP(AvgPool(F)))

[0023] Among them, the input feature layer is F, and the feature layer F is compressed to one dimension through the maximum pooling operation MaxPool() and the average pooling operation AvgPool() respectively, and then the two compressed features are input into the shared perceptron MLP respectively, and finally the weight W of the feature layer F in space is calculated by the normalized exponential function softmax() c ;

[0024] The channel attention mechanism expression of the channel attention module is:

[0025] W s =softmax((concat((F),AvgPool(F)))))

[0026] Among them, the input feature layer is F, and the feature layer F is compressed to one dimension through the maximum pooling operation MaxPool() and the average pooling operation AvgPool() respectively. Then the two compressed features are concat-connected and input into a 7×7 Conv() convolutional layer together. Finally, the weight W of the feature layer F on the channel is calculated by the normalized exponential function softmax() s .

[0027] In a possible design, before using the training set to train the preset internal wave automatic extraction network, the method further includes:

[0028] The loss function of the internal wave automatic extraction network is constructed. The loss function adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function and a Matthews correlation coefficient loss function. The expression of the multi-scale loss function is:

[0029] L=L BCE + SSIM + MCC

[0030] Among them, L represents the multi-scale loss function, L BCE Characterizes the binary cross entropy loss function, SSIM Characterize the structural similarity loss function, L MCC represents the Matthews correlation coefficient loss function; and

[0031]

[0032] Among them, x i Represents the predicted value of the i-th pixel, y irepresents the true value of the i-th pixel, x represents the predicted value of the pixel, y represents the true value of the pixel, i represents the pixel number, and N is the total number of pixels in the image; and

[0033] L SSiM =1-[l(x,y) α (x,y) β (x,y) γ ]

[0034]

[0035]

[0036]

[0037] Among them, μ X and μ Y are the means of the pixel prediction value set X and the pixel true value set Y, σ X and σ Y are the variances of the pixel prediction value set X and the pixel true value set Y, σ XY is the covariance of the pixel prediction value set X and the pixel true value set Y, α, β and γ are the set first exponent, second exponent and third exponent respectively, C1, C2 and C3 are the set first constant, second constant and third constant respectively; and

[0038]

[0039] Among them, TP represents the positive samples predicted as positive by the internal wave automatic extraction network, TN represents the negative samples predicted as negative by the internal wave automatic extraction network, FP represents the negative samples predicted as positive by the internal wave automatic extraction network, and FN represents the positive samples predicted as negative by the internal wave automatic extraction network.

[0040] In a possible design, the preset internal wave automatic extraction network is trained using a training set until a set training condition is met to obtain a trained internal wave automatic extraction network, including:

[0041] The trained internal wave automatic extraction network is trained using the training set until the number of training iterations reaches 120 or the batch size of the trained internal wave automatic extraction network is 4 or the learning rate of the trained internal wave automatic extraction network is 0.001, thereby obtaining the trained internal wave automatic extraction network.

[0042] In a possible design, the polarimetric SAR image data uses dual-polarized C-band Sentinel-1 data, and the multi-channel polarimetric data false color image synthesis is performed on each pre-processed polarimetric SAR image data to obtain a corresponding polarimetric false color image, including:

[0043] The VV polarization channel data and the VH polarization channel data are extracted from the preprocessed polarization SAR image data, and the ratio of the VV polarization channel data to the VH polarization channel data is used as the third polarization channel data. The VV polarization channel data, the VH polarization channel data and the third polarization channel data are used to perform multi-channel polarization data false color image synthesis to obtain the corresponding polarization false color image.

[0044] In the second aspect, an ocean internal wave extraction system for complex environment background is provided, including a data acquisition unit, an image synthesis unit, an image annotation unit, a sample division unit, a network training unit, a network testing unit and an internal wave extraction unit, wherein:

[0045] A data acquisition unit, used for acquiring a plurality of polarization SAR image data for the ocean, and performing image preprocessing on each polarization SAR image data to obtain corresponding preprocessed polarization SAR image data;

[0046] An image synthesis unit is used to perform multi-channel polarization data false color image synthesis on each pre-processed polarization SAR image data to obtain a corresponding polarization false color image;

[0047] An image annotation unit is used to obtain a manual annotation instruction from a user, and annotate the internal wave peak lines of each polarization false color image in response to the manual annotation instruction to obtain a corresponding annotated internal wave polarization image sample;

[0048] A sample division unit is used to divide all internal wave polarization image samples into a training set and a test set according to a set ratio;

[0049] A network training unit is used to train a preset internal wave automatic extraction network using a training set until a set training condition is met to obtain a trained internal wave automatic extraction network, wherein the internal wave automatic extraction network includes a U-Net network, and a spatiotemporal attention module is connected in series after each convolution block of the encoder part of the U-Net network, wherein the spatiotemporal attention module includes a spatial attention module and a channel attention module, and a loss function of the internal wave automatic extraction network adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function, and a Matthews correlation coefficient loss function;

[0050] A network testing unit is used to test the trained internal wave automatic extraction network using a test set until the set test conditions are met, thereby obtaining the tested internal wave automatic extraction network, and using the tested internal wave automatic extraction network as an ocean internal wave extraction network;

[0051] The internal wave extraction unit is used to input the target ocean polarimetric SAR image into the ocean internal wave extraction network to extract the ocean internal waves, obtain the corresponding ocean internal wave extraction result, and output the ocean internal wave extraction result of the target ocean polarimetric SAR image.

[0052] Thirdly, we provide an ocean internal wave extraction system for complex environmental backgrounds, including:

[0053] A memory for storing instructions;

[0054] A processor is used to read the instructions stored in the memory and execute any one of the methods described in the first aspect according to the instructions.

[0055] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored on the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes any one of the methods described in the first aspect. In addition, a computer program product is provided, and when the computer program product is executed on a computer, the computer executes any one of the methods described in the first aspect.

[0056] Beneficial effects: The present invention preprocesses the collected polarization SAR image data, and then uses the SAR data of different polarization channels to construct a false color image, thereby increasing the image information characterizing the internal wave characteristics, and then uses the false color image to train and test the internal wave automatic extraction network that integrates the spatiotemporal attention mechanism and the U-Net network and adopts a multi-scale loss function, thereby realizing the automatic extraction of the ocean internal wave crest line using the ocean internal wave extraction network. The present invention can greatly improve the efficiency of ocean internal wave remote sensing detection and realize efficient and real-time ocean internal wave detection; and the false color image synthesis method using the polarization SAR multi-channel combination can further increase the internal wave characterization information; by designing the internal wave automatic extraction network that integrates the spatiotemporal attention mechanism and the multi-scale loss function, it can realize efficient and accurate automatic extraction of the internal wave crest line under complex ocean background and atmospheric conditions, overcome the shortcomings of the traditional single-channel SAR internal wave detection method in terms of recognition accuracy and stability, broaden the application space of SAR technology in ocean internal wave monitoring, and is beneficial to marine disaster prevention and mitigation and ensuring the safety of submarine underwater navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1This is a schematic diagram of the steps of the method in Example 1 of the present invention;

[0059] Figure 2 (a) is the VV polarization channel image, Figure 2 (b) is the VH polarization channel image, Figure 2 (c) is the third polarization channel VV / VH image, Figure 2 (d) is a polarization false color image;

[0060] Figure 3 This is a schematic diagram of the internal wave crest line after manual annotation;

[0061] Figure 4 It is a schematic diagram of the structure of the U-Net network;

[0062] Figure 5 Schematic diagram of the structure of the spatiotemporal attention module;

[0063] Figure 6 A schematic diagram of the results of ocean internal wave extraction using an ocean internal wave extraction network;

[0064] Figure 7 (a) and Figure 7 (b) Schematic diagrams of two cases showing the differences between the internal wave extraction results of different channel images of polarimetric SAR data and the real samples;

[0065] Figure 8 This is a schematic diagram of the structure of the system in Example 2 of the present invention;

[0066] Fig. 9 Schematic diagram of the system structure in Example 3 of the present invention. DETAILED DESCRIPTION

[0067] It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the exemplary embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.

[0068] It should be understood that, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments can be understood according to specific circumstances.

[0069] In the following description, certain details are provided to facilitate a complete understanding of the example embodiments. However, it will be appreciated by those of ordinary skill in the art that the example embodiments may be implemented without these certain details. For example, devices may be shown in block diagrams to avoid obscuring the examples with unnecessary details. In other embodiments, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the embodiments.

[0070] Embodiment 1:

[0071] This embodiment provides an ocean internal wave extraction method for complex environment backgrounds, such as Figure 1 As shown, the method comprises the following steps:

[0072] S1. Collect a number of polarization SAR image data for the ocean, and perform image preprocessing on each polarization SAR image data to obtain corresponding preprocessed polarization SAR image data.

[0073] In the specific implementation, it is necessary to first collect a number of polarimetric SAR image data for the ocean, and then perform image preprocessing on each polarimetric SAR image data, including data format conversion, data correction, data reprojection, data splicing and mosaicking, image filtering and image enhancement processing, to obtain the corresponding preprocessed polarimetric SAR image data. For example, the polarimetric SAR image data takes the dual-polarization C-band Sentinel-1 data as an example, and the preprocessing process mainly includes:

[0074] (1) After acquiring polarimetric SAR image data, steps including data decompression, data format conversion, and data correction are required. Data decompression is to decompress the acquired data file to obtain the original data; data format conversion is to convert the original data into a commonly used data format, such as GeoTIFF format; data correction is to correct the original data to eliminate the influence of factors such as instruments and atmosphere on the data.

[0075] (2) Data reprojection: mapping the corrected remote sensing data to a specific geographic coordinate system. Data on the earth’s surface are usually expressed in longitude and latitude coordinates, while the original coordinates of Sentinle-1 data may be expressed in row and column numbers. Therefore, it is necessary to convert remote sensing images from the original coordinate system to the longitude and latitude coordinate system for subsequent analysis and application.

[0076] (3) Sentinle-1 data are usually distributed in different areas in the form of multiple files. In order to facilitate subsequent analysis, these data files need to be spliced ​​and mosaicked. Data splicing is to merge multiple data files into one file, while data mosaicking is to splice data from different areas together to form a complete surface coverage data set.

[0077] (4) SAR data is affected by speckle noise, which makes SAR image interpretation difficult and is not conducive to SAR image target detection and recognition. Therefore, SAR data needs to be filtered to reduce the influence of coherent speckle noise. This embodiment uses a 7×7 window refined Lee filter to perform fine filtering and noise reduction on SAR image data.

[0078] (5) Image enhancement: Image enhancement can purposefully emphasize the overall or local characteristics of an image, make an originally unclear image clear or emphasize certain interesting features, expand the differences between the features of different objects in the image, suppress uninteresting features, improve image quality, enrich information, and enhance image interpretation and recognition effects, thereby improving the accuracy of internal wave recognition in remote sensing images. This embodiment uses five operations: vertical flip, horizontal flip, and rotation of 90 / 180 / 270 degrees.

[0079] S2. Perform multi-channel polarization data false color image synthesis on each pre-processed polarization SAR image data to obtain a corresponding polarization false color image.

[0080] During specific implementation, the polarimetric SAR image data may use dual-polarization C-band Sentinel-1 data, including VV polarization channel data and VH polarization channel data. Compared with single-channel SAR images, polarimetric SAR data can characterize the scattering type and geometric structure of the target object to a certain extent, which is beneficial for distinguishing the ocean internal waves from the complex background environment, thereby improving the accuracy of internal wave extraction of SAR images. In order to make full use of the dual-channel polarization data of Sentinel-1, this embodiment also uses the ratio VV / VH of the VV polarization channel data to the VH polarization channel data as the third polarization channel data, and uses the VV polarization channel data, the VH polarization channel data and the third polarization channel data to perform multi-channel polarization data false color image synthesis to obtain the corresponding polarization false color image. Exemplarily, the VV polarization channel image, the VH polarization channel image, the third polarization channel VV / VH image and the synthesized polarization false color image are respectively as follows: Figure 2 (a) Figure 2 (b) Figure 2 (c) and Figure 2 (d) as shown.

[0081] S3. Obtaining the manual labeling instruction of the user, and labeling the internal wave peak lines of each polarization false color image in response to the manual labeling instruction, to obtain the corresponding labeled internal wave polarization image samples.

[0082] In specific implementation, according to the internal wave principle of SAR images, that is, when internal waves propagate in seawater, they will cause the sea surface to converge and diverge, thereby changing the roughness of the sea surface. The change in the roughness of the sea surface affects the backscattering intensity of the sea surface radar, thereby causing the SAR image intensity to change in the internal wave area, and the intensity image will appear as light and dark stripes. Visual interpretation methods can be used to manually interpret SAR images, and the internal wave crest lines can be manually marked on SAR images containing internal wave crest lines. Figure 3 As shown in the figure, taking the polarization false color image constructed by dual-polarization C-band Sentinel-1 data as an example, when manually annotating the internal wave peak line in the image, it is necessary to interpret and annotate the RGB three color channels of the false color image respectively, and finally take the union of the internal wave peak lines annotated by the RGB three color channels, and annotate the pixel values ​​of the pixels in the internal wave peak line image area as 1, that is, it appears in white, and annotate the pixel values ​​of the pixels outside the internal wave peak line image area as 0, that is, it appears in black.

[0083] S4. Divide all internal wave polarization image samples into training set and test set according to the set ratio.

[0084] In specific implementation, all internal wave polarization image samples can be divided into training sets and test sets according to a set ratio. For example, if a total of 1008 internal wave polarization image samples are manually interpreted and annotated, 808 of them are randomly divided as training sets, and the remaining 200 are used as test sets.

[0085] S5. Use the training set to train the preset internal wave automatic extraction network until the set training conditions are met to obtain the trained internal wave automatic extraction network, wherein the internal wave automatic extraction network includes a U-Net network, and a spatiotemporal attention module is connected in series after each convolution block of the encoder part of the U-Net network, and the spatiotemporal attention module includes a spatial attention module and a channel attention module, and the loss function of the internal wave automatic extraction network adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function and a Matthews correlation coefficient loss function.

[0086] During specific implementation, the training set can be used to train the preset internal wave automatic extraction network to obtain the trained internal wave automatic extraction network. The automatic extraction of ocean internal waves can be regarded as a binary classification problem of images. The pixel values ​​belonging to the internal wave crest lines in the SAR image are divided into one category, and other areas are divided into another category. However, SAR images are difficult to interpret due to the influence of SAR speckle noise. At the same time, the ocean environment and atmospheric conditions are complex and diverse. Phenomena including typhoons and mesoscale vortices may be misjudged as ocean internal wave lines, affecting the accuracy of internal wave extraction in SAR images. In order to improve the accuracy of automatic extraction of internal waves in complex background environments, the automatic extraction network of internal waves can be used as follows. Figure 4The U-Net network shown in the figure is constructed, and the spatiotemporal attention mechanism is integrated on the basis of the U-Net network to further improve the feature focusing ability of the U-Net network and help the internal wave extraction network focus on the features related to the internal wave. The construction process of the internal wave automatic extraction network includes:

[0087] Construct the basic framework of the U-Net network. The U-Net network adopts the classic encoder-decoder structure and consists of two parts: the encoder and the decoder. Like a typical convolutional network, the encoder part consists of four convolutional layers in series, each of which includes two 3×3 convolution (Conv) blocks, and each convolution block is followed by a batch normalization (BN) layer and a linear activation function (ReLU). BN normalizes the distribution of each input feature in each small batch to avoid gradient disappearance. ReLU sets the threshold of non-positive values ​​to 0 and keeps positive values ​​unchanged. After each convolution block operation, the length and width of the feature map are halved, and the channels of the feature map are doubled. The decoder part consists of four deconvolution layers, each of which includes a 2×2 deconvolution (DConv) block to implement upsampling operations and is connected in series with the corresponding feature maps of the encoder part. This series operation can combine low-level and high-level features. Fusion of features of different scales can improve the accuracy of the extraction network. Finally, each layer connects two 3×3 Conv, and connects BN layers and ReLU layers in series. After each decoder deconvolution layer operation, the length and width of the feature map are doubled, and the channel is halved. After the last deconvolution layer, the feature map is sampled back to the original size of the input image, with a channel size of 64, and then a 1×1 Conv layer with a softmax activation function is used to obtain the final internal wave extraction result. In this prediction layer, pixel 1 also represents the internal wave peak line, and pixel 0 represents the non-internal wave area.

[0088] When building the encoder part, such as Figure 5 As shown, a spatiotemporal attention module is connected in series after each convolution block, and the spatiotemporal attention module includes a spatial attention module and a channel attention module;

[0089] The spatial attention mechanism expression of the spatial attention module is:

[0090] W c =softmax((MaxPool(F))+MLP(AvgPool(F)))

[0091] Among them, the input feature layer is F, and the feature layer F is compressed to one dimension through the maximum pooling operation MaxPool() and the average pooling operation AvgPool() respectively, and then the two compressed features are input into the shared perceptron MLP respectively, and finally the weight W of the feature layer F in space is calculated by the normalized exponential function softmax() c ;

[0092] The channel attention mechanism expression of the channel attention module is:

[0093] W s =softmax((concat((F),AvgPool(F)))))

[0094] Among them, the input feature layer is F, and the feature layer F is compressed to one dimension through the maximum pooling operation MaxPool() and the average pooling operation AvgPool() respectively. Then the two compressed features are concat-connected and input into a 7×7 Conv() convolutional layer together. Finally, the weight W of the feature layer F on the channel is calculated by the normalized exponential function softmax() s .

[0095] Construct a loss function for the internal wave automatic extraction network. The loss function is mainly used to measure the difference or error between the prediction result of the network model and the actual result, so as to guide the network model to achieve the best training effect during the training process. However, the internal wave extraction task is unique. On the one hand, the target area of ​​the internal wave crest line accounts for a small proportion of the entire image, and there is a significant class imbalance problem between the internal wave area and the non-internal wave area. In addition, the overall linear relationship of the internal wave crest line cannot be ignored. Each pixel on the crest line does not exist in isolation, but is interdependent with the surrounding pixels to form a linear structure. This embodiment proposes a targeted multi-scale loss function for the particularity of the internal wave extraction task, which consists of three parts: a binary cross entropy loss function at the pixel level, a structural similarity loss function at the patch level, and a Matthews correlation coefficient loss function at the image level. The expression of the multi-scale loss function is:

[0096] L=L BCE + SSIM + MCC

[0097] Among them, L represents the multi-scale loss function, L BCE Characterizes the binary cross entropy loss function, SCIM Characterize the structural similarity loss function, L MCC represents the Matthews correlation coefficient loss function; and

[0098]

[0099] Among them, x i Represents the predicted value of the i-th pixel, y i represents the true value of the i-th pixel, x represents the predicted value of the pixel, y represents the true value of the pixel, i represents the pixel number, and N is the total number of pixels in the image; and

[0100] L SSIM =1-[l(x,y)α (x,y) β (x,y) γ ]

[0101]

[0102]

[0103]

[0104] Among them, μ X and μ Y are the means of the pixel prediction value set X and the pixel true value set Y, σ X and σ Y are the variances of the pixel prediction value set X and the pixel true value set Y, σ XY is the covariance of the pixel prediction value set X and the pixel true value set Y, α, β and γ are the first exponent, second exponent and third exponent respectively. Assume that α=β=γ=1, then the structural similarity loss function is SSIM Can be simplified to C1, C2 and C3 are the first constant, the second constant and the third constant respectively; and

[0105]

[0106] Among them, TP represents the positive samples predicted as positive by the internal wave automatic extraction network, TN represents the negative samples predicted as negative by the internal wave automatic extraction network, FP represents the negative samples predicted as positive by the internal wave automatic extraction network, and FN represents the positive samples predicted as negative by the internal wave automatic extraction network.

[0107] During the training process of the internal wave automatic extraction network, the training image samples were randomly cropped into 480×480 images, and the original SAR image size was kept unchanged during the test. The framework used in the training process was Pytorch 1.11 version. The number of data iterations, batch size, and learning rate set during training were 120 epochs, 4, and 0.001, respectively. After the iterative training was completed, the network model parameters with the smallest test error were selected as the optimal parameters to obtain the trained internal wave automatic extraction network.

[0108] S6. Use the test set to test the trained internal wave automatic extraction network until the set test conditions are met, and obtain the tested internal wave automatic extraction network, and use the tested internal wave automatic extraction network as the ocean internal wave extraction network.

[0109] During the specific implementation, the trained automatic internal wave extraction network is tested with a test set until the set test conditions are met, so as to evaluate whether the training effect of the network meets the standards and obtain the tested automatic internal wave extraction network, which is then used as the ocean internal wave extraction network.

[0110] S7. Input the target ocean polarimetric SAR image into the ocean internal wave extraction network to extract ocean internal waves, obtain the corresponding ocean internal wave extraction result, and output the ocean internal wave extraction result of the target ocean polarimetric SAR image.

[0111] In specific implementation, after obtaining the ocean internal wave extraction network, the ocean internal wave extraction network can be used for practical application, the target ocean polarimetric SAR image is input into the ocean internal wave extraction network to extract the ocean internal waves, and the corresponding ocean internal wave extraction result, that is, the spatiotemporal distribution of the internal waves in the target sea area, is obtained, and then the ocean internal wave extraction result of the target ocean polarimetric SAR image is output. Figure 6 As shown, the ocean internal wave extraction network is used to extract ocean internal waves from a target ocean polarimetric SAR image to obtain the corresponding ocean internal wave extraction result. Figure 7 (a) and Figure 7 (b) Two cases showing the difference between the internal wave extraction results of different channel images of polarimetric SAR data and the real samples. The multi-channel polarimetric false color image extraction structure proposed in this embodiment is significantly better than other single-channel extraction results.

[0112] Embodiment 2:

[0113] This embodiment provides an ocean internal wave extraction system for complex environment backgrounds, such as Figure 8 As shown, it includes a data acquisition unit, an image synthesis unit, an image annotation unit, a sample division unit, a network training unit, a network testing unit and an internal wave extraction unit, wherein:

[0114] A data acquisition unit, used for acquiring a plurality of polarization SAR image data for the ocean, and performing image preprocessing on each polarization SAR image data to obtain corresponding preprocessed polarization SAR image data;

[0115] An image synthesis unit is used to perform multi-channel polarization data false color image synthesis on each pre-processed polarization SAR image data to obtain a corresponding polarization false color image;

[0116] An image annotation unit is used to obtain a manual annotation instruction from a user, and annotate the internal wave peak lines of each polarization false color image in response to the manual annotation instruction to obtain a corresponding annotated internal wave polarization image sample;

[0117] A sample division unit is used to divide all internal wave polarization image samples into a training set and a test set according to a set ratio;

[0118] A network training unit is used to train a preset internal wave automatic extraction network using a training set until a set training condition is met to obtain a trained internal wave automatic extraction network, wherein the internal wave automatic extraction network includes a U-Net network, and a spatiotemporal attention module is connected in series after each convolution block of the encoder part of the U-Net network, wherein the spatiotemporal attention module includes a spatial attention module and a channel attention module, and a loss function of the internal wave automatic extraction network adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function, and a Matthews correlation coefficient loss function;

[0119] A network testing unit is used to test the trained internal wave automatic extraction network using a test set until the set test conditions are met, thereby obtaining the tested internal wave automatic extraction network, and using the tested internal wave automatic extraction network as an ocean internal wave extraction network;

[0120] The internal wave extraction unit is used to input the target ocean polarimetric SAR image into the ocean internal wave extraction network to extract the ocean internal waves, obtain the corresponding ocean internal wave extraction result, and output the ocean internal wave extraction result of the target ocean polarimetric SAR image.

[0121] Embodiment 3:

[0122] This embodiment provides an ocean internal wave extraction system for complex environment backgrounds, such as Fig. 9 As shown, at the hardware level, it includes:

[0123] Data interface, used to establish data connection between the processor and the external data terminal;

[0124] A memory for storing instructions;

[0125] The processor is used to read the instructions stored in the memory and execute the ocean internal wave extraction method in Example 1 according to the instructions.

[0126] Optionally, the system further includes an internal bus, through which the processor, the memory and the data interface can be interconnected, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0127] The memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO) memory, etc. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0128] Embodiment 4:

[0129] This embodiment provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed on a computer, the computer executes the ocean internal wave extraction method in Embodiment 1. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0130] This embodiment also provides a computer program product, which, when executed on a computer, executes the ocean internal wave extraction method in Embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0131] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Ocean internal wave extraction method for complex environment background, characterized by: include: Collecting a number of polarization SAR image data for the ocean, and performing image preprocessing on each polarization SAR image data to obtain corresponding preprocessed polarization SAR image data; Perform multi-channel polarization data false color image synthesis on each pre-processed polarization SAR image data to obtain a corresponding polarization false color image; Obtaining a manual marking instruction from a user, and marking the internal wave peak lines of each polarization false color image in response to the manual marking instruction, to obtain a corresponding marked internal wave polarization image sample; Divide all internal wave polarization image samples into training set and test set according to the set ratio; A preset internal wave automatic extraction network is trained using a training set until a set training condition is met, thereby obtaining a trained internal wave automatic extraction network, wherein the internal wave automatic extraction network includes a U-Net network, and a spatiotemporal attention module is connected in series after each convolution block of an encoder part of the U-Net network, wherein the spatiotemporal attention module includes a spatial attention module and a channel attention module, and a loss function of the internal wave automatic extraction network adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function, and a Matthews correlation coefficient loss function; The trained automatic internal wave extraction network is tested using a test set until the set test conditions are met, thereby obtaining a tested automatic internal wave extraction network, and the tested automatic internal wave extraction network is used as an ocean internal wave extraction network; The target ocean polarimetric SAR image is input into the ocean internal wave extraction network to extract the ocean internal waves, obtain the corresponding ocean internal wave extraction result, and output the ocean internal wave extraction result of the target ocean polarimetric SAR image.

2. The method for extracting ocean internal waves in a complex environment according to claim 1, characterized in that: The image preprocessing of each polarization SAR image data includes: data format conversion, data correction, data reprojection, data splicing and mosaicking, image filtering and image enhancement processing of each polarization SAR image data.

3. The method for extracting ocean internal waves in a complex environment according to claim 1, characterized in that: The step of marking the internal wave peak lines of each polarization false color image in response to the manual marking instruction includes: According to the manual annotation instructions, the image area of ​​the internal wave crest line is annotated on the polarization false color image, and the pixel values ​​of the pixels in the image area of ​​the internal wave crest line are annotated as 1, and the pixel values ​​of the pixels outside the image area of ​​the internal wave crest line are annotated as 0.

4. The method for extracting ocean internal waves in a complex environment according to claim 1, characterized in that: Before using the training set to train the preset internal wave automatic extraction network, the method further includes: Construct a U-Net network, wherein the U-Net network includes an encoder part and a decoder part, wherein the encoder part includes four convolutional layers connected in series, and each convolutional layer includes two 3×3 convolutional blocks; When constructing the encoder part, a spatiotemporal attention module is connected in series after each convolution block, and the spatiotemporal attention module includes a spatial attention module and a channel attention module; The spatial attention mechanism expression of the spatial attention module is: W c =softmax(MLP(MaxPool(F))+MLP(AvgPool(F))) Among them, the input feature layer is F, and the feature layer F is compressed to one dimension through the maximum pooling operation MaxPool() and the average pooling operation AvgPool() respectively, and then the two compressed features are input into the shared perceptron MLP respectively, and finally the weight W of the feature layer F in space is calculated by the normalized exponential function softmax() c ; The channel attention mechanism expression of the channel attention module is: W s =softmax(Conv(concat(MaxPool(F),AvgPool(F))))) Among them, the input feature layer is F, and the feature layer F is compressed to one dimension through the maximum pooling operation MaxPool() and the average pooling operation AvgPool() respectively. Then the two compressed features are concat-connected and input into a 7×7 Conv() convolutional layer together. Finally, the weight W of the feature layer F on the channel is calculated by the normalized exponential function softmax() s .

5. The method for extracting ocean internal waves in a complex environment according to claim 1, characterized in that: Before using the training set to train the preset internal wave automatic extraction network, the method further includes: Construct a loss function for the internal wave automatic extraction network, wherein the loss function adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function, and a Matthews correlation coefficient loss function, and the expression of the multi-scale loss function is L = L BCE +L SSIM +L MCC Among them, L represents the multi-scale loss function, L BCE Characterize the binary cross entropy loss function, L SSIM Characterize the structural similarity loss function, L MCC represents the Matthews correlation coefficient loss function; and Among them, x i Represents the predicted value of the i-th pixel, y i represents the true value of the i-th pixel, x represents the predicted value of the pixel, y represents the true value of the pixel, i represents the pixel number, and N is the total number of pixels in the image; and L SSIM =1-[l(x,y) α ·c(x,y) β ·s(x,y) γ ] Among them, μ X and μ Y are the means of the pixel prediction value set X and the pixel true value set Y, σ X and σ Y are the variances of the pixel prediction value set X and the pixel true value set Y, σ XY is the covariance of the pixel prediction value set X and the pixel true value set Y, α, β and γ are the set first exponent, second exponent and third exponent respectively, C1, C2 and C3 are the set first constant, second constant and third constant respectively; and Among them, TP represents the positive samples predicted as positive by the internal wave automatic extraction network, TN represents the negative samples predicted as negative by the internal wave automatic extraction network, FP represents the negative samples predicted as positive by the internal wave automatic extraction network, and FN represents the positive samples predicted as negative by the internal wave automatic extraction network.

6. The method for extracting ocean internal waves in a complex environment according to claim 1, characterized in that: The method of using the training set to train the preset internal wave automatic extraction network until the set training conditions are met to obtain the trained internal wave automatic extraction network includes: The trained internal wave automatic extraction network is trained using the training set until the number of training iterations reaches 120 or the batch size of the trained internal wave automatic extraction network is 4 or the learning rate of the trained internal wave automatic extraction network is 0.001, thereby obtaining the trained internal wave automatic extraction network.

7. The method for extracting ocean internal waves in a complex environment according to claim 1, characterized in that: The polarimetric SAR image data adopts dual-polarized C-band Sentinel-1 data, and the multi-channel polarimetric data false color image synthesis is performed on each pre-processed polarimetric SAR image data to obtain a corresponding polarimetric false color image, including: The VV polarization channel data and the VH polarization channel data are extracted from the preprocessed polarization SAR image data, and the ratio of the VV polarization channel data to the VH polarization channel data is used as the third polarization channel data. The VV polarization channel data, the VH polarization channel data and the third polarization channel data are used to perform multi-channel polarization data false color image synthesis to obtain the corresponding polarization false color image.

8. The ocean internal wave extraction system for complex environment background is characterized by: It includes a data acquisition unit, an image synthesis unit, an image annotation unit, a sample division unit, a network training unit, a network testing unit and an internal wave extraction unit, wherein: A data acquisition unit, used for acquiring a plurality of polarization SAR image data for the ocean, and performing image preprocessing on each polarization SAR image data to obtain corresponding preprocessed polarization SAR image data; An image synthesis unit is used to perform multi-channel polarization data false color image synthesis on each pre-processed polarization SAR image data to obtain a corresponding polarization false color image; An image annotation unit is used to obtain a manual annotation instruction from a user, and annotate the internal wave peak lines of each polarization false color image in response to the manual annotation instruction to obtain a corresponding annotated internal wave polarization image sample; A sample division unit is used to divide all internal wave polarization image samples into a training set and a test set according to a set ratio; A network training unit is used to train a preset internal wave automatic extraction network using a training set until a set training condition is met to obtain a trained internal wave automatic extraction network, wherein the internal wave automatic extraction network includes a U-Net network, and a spatiotemporal attention module is connected in series after each convolution block of the encoder part of the U-Net network, wherein the spatiotemporal attention module includes a spatial attention module and a channel attention module, and a loss function of the internal wave automatic extraction network adopts a multi-scale loss function including a binary cross entropy loss function, a structural similarity loss function, and a Matthews correlation coefficient loss function; A network testing unit is used to test the trained internal wave automatic extraction network using a test set until the set test conditions are met, thereby obtaining the tested internal wave automatic extraction network, and using the tested internal wave automatic extraction network as an ocean internal wave extraction network; The internal wave extraction unit is used to input the target ocean polarimetric SAR image into the ocean internal wave extraction network to extract the ocean internal waves, obtain the corresponding ocean internal wave extraction result, and output the ocean internal wave extraction result of the target ocean polarimetric SAR image.

9. The ocean internal wave extraction system for complex environment background is characterized by: include: A memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the ocean internal wave extraction method described in any one of claims 1-7 according to the instructions.

10. A computer program product, characterized in that When the computer program product is run on a computer, the ocean internal wave extraction method according to any one of claims 1 to 7 is executed.

Images