Water surface height field inversion method and system based on monocular optical image

Abstract

The invention discloses a monocular optical image-based water surface height field inversion method and system, and the method comprises the steps: inputting a water surface monocular optical image into a trained water surface height field inversion model, and obtaining a predicted water surface height field of each pixel point in the water surface monocular optical image; the method comprises the following steps: calculating a simulated water surface height field as model label data by simulating waves, calculating the image brightness of each pixel point to establish a water surface composite image, preprocessing the water surface composite image, and inputting the preprocessed water surface composite image into a U-Net network for training to obtain a trained water surface height field inversion model; inversion of the water surface height field is achieved based on monocular optical images and an artificial intelligence method, the problems that in the binocular stereo imaging technology, the implementation process is complex, and the result is prone to being influenced by errors are effectively solved, data of numerical simulation are adopted for making a synthetic image to conduct water surface height field inversion model training, and the accuracy of the water surface height field inversion is improved. The problem that a truth value label cannot be obtained due to the fact that observation data of the water surface height are not easy to obtain is solved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, in particular to a water surface height field inversion method and system based on monocular optical images. Background Art

[0002] The inversion of water surface height field from optical image data can be used for wave parameter measurement, tsunami warning and ocean dynamics research. The traditional inversion method is mainly based on stereo imaging technology. This method uses binocular or multi-eye optical cameras to obtain water surface optical images, and obtains the one-to-one correspondence between pixel points in multi-eye images at the same time after matching the water surface feature points. Then, it is substituted into the stereo imaging equation to realize the coordinate inversion of any point on the water surface to obtain the water surface height information of the corresponding point. However, this method depends on the calibration of camera parameters and the matching of feature points of multi-eye images. In the observation practice of natural water surface environment, firstly, it is difficult to directly obtain the measured control points required for camera calibration, and secondly, the water surface usually lacks matching points with obvious features, which makes it easy to have large errors in the multi-eye image matching step. Overall, the process of water surface height field inversion based on stereo imaging is relatively complex and difficult.

[0003] In recent years, artificial intelligence has been gradually applied to the extraction of marine environmental elements based on optical video images, including the extraction of sea surface wave height, overshore waves, coastal water level, etc. However, in the problem of water surface height inversion, some existing work still relies on binocular images to realize the inversion of water level field, or can only realize the inversion of single-point water level through monocular images. If stereo imaging technology based on binocular or multi-eye imaging system is used, the implementation process is complicated and the accuracy of the results depends largely on the errors in these two key steps due to the need for camera calibration and image matching; if monocular optical imaging and artificial intelligence technology are used, only single-point water level inversion can be realized, and the inversion of full-field information cannot be realized. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a method and system for inverting the water surface height field of a monocular optical image with accurate prediction and without the need for camera calibration.

[0005] Technical solution: The water surface height field inversion method based on monocular optical image described in the present invention inputs the water surface monocular optical image into a trained water surface height field inversion model to obtain the predicted water surface height field of each pixel point in the water surface monocular optical image;

[0006] The training method of the water surface height field inversion model comprises: simulating the water surface height field by simulating waves and using it as model label data; calculating the image brightness of each pixel to establish a water surface synthetic image, and inputting the water surface synthetic image into a U-Net network for training after preprocessing to obtain a trained water surface height field inversion model.

[0007] Furthermore, the calculation of the image brightness of each pixel to establish a water surface composite image includes: the image brightness of each pixel (x, y)

[0008] in Z is the simulated water surface height field, τ is the illumination azimuth, and σ is the illumination incident angle.

[0009] Furthermore, the preprocessing of the water surface synthetic image and inputting it into the U-Net network for training includes: cutting the water surface synthetic image into a plurality of individual image samples so that they match the size of the U-Net network input layer, and normalizing each individual image sample.

[0010] Furthermore, the preprocessing of the water surface synthetic image and inputting it into the U-Net network for training includes: calculating the average sea condition h corresponding to each individual image sample, and randomly extracting individual image samples based on the principle of uniform distribution of different sea conditions to establish a training set and a test set;

[0011] Average sea conditions Var represents the calculated variance, Z(x,y) is the water surface height field of the pixel point (x,y) in a single image sample, and the size of a single image sample is N×N.

[0012] Furthermore, the step of inputting the water surface synthetic image into the U-Net network for training after preprocessing includes: the U-Net network includes an input layer, an encoder, a bridge layer, a decoder, a convolution layer and an output layer connected in sequence;

[0013] The encoder includes three encoding modules connected in sequence, and the encoding module includes a convolution module and a pooling layer connected in sequence;

[0014] The bridge layer includes two convolution modules connected in sequence;

[0015] The decoder comprises three decoding modules connected in sequence, wherein the decoding module comprises a transposed convolution layer, a feature concatenation layer and a convolution module connected in sequence;

[0016] The convolution module includes a convolution layer, a batch normalization layer, an activation layer, a convolution layer, a batch normalization layer, an activation layer, and a discard layer which are connected in sequence.

[0017] Furthermore, the preprocessing of the water surface synthetic image and inputting it into the U-Net network for training includes: setting the training loss function to the mean square error between the simulated water surface height field and the predicted water surface height field.

[0018] Furthermore, the step of inputting the water surface monocular optical image into the trained water surface height field inversion model includes: converting the water surface monocular optical image taken by the monocular camera into a grayscale image, cutting it into a plurality of image samples to match the size of the U-Net network input layer, and normalizing each image sample.

[0019] The water surface height field inversion system based on monocular optical images of the present invention comprises:

[0020] A monocular camera image acquisition unit, used to acquire monocular optical images on the water surface;

[0021] A water surface height field inversion unit is used to input the water surface monocular optical image into the trained water surface height field inversion model to obtain a predicted water surface height field for each pixel point in the water surface monocular optical image;

[0022] The water surface height field inversion model training unit is used to simulate the water surface height field by simulating waves as model label data; the image brightness of each pixel is calculated to establish a water surface composite image, and the water surface composite image is input into the U-Net network for training after preprocessing to obtain a trained water surface height field inversion model.

[0023] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the water surface height field inversion method based on monocular optical images is implemented.

[0024] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the water surface height field inversion method based on monocular optical images is implemented.

[0025] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) The present invention uses numerical simulation data to make synthetic images for water surface height field inversion model training, which effectively solves the problem that the water surface height observation data is not easy to obtain and the true value label required for artificial intelligence training cannot be provided. (2) The present invention realizes the inversion of water surface height field based on monocular optical images and artificial intelligence methods, which effectively solves the problem that the implementation process is complicated and the results are easily affected by errors in binocular stereo imaging technology. (3) The present invention can have a large application prospect in wave monitoring. After the water surface height field is inverted from the continuous video by the method of the present invention, the wave height, period and other wave elements can be calculated based on wave theory, and the peak / trough position can be further extracted to realize the inversion of other wave elements such as wavelength, wave steepness, wave number, wave direction, etc.; the present invention can realize the monitoring of wave field information compared with the traditional single-point measurement method of water level gauge. (4) In recent years, a large number of optical video monitoring cameras have been set up in coastal areas of my country. However, the use of video images is still in the stage of manual visual monitoring. The present invention has important potential value for improving the utilization of existing optical image resources and monitoring and early warning of wave disasters in coastal areas of my country. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flow chart of a method for inverting a water surface height field according to an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the structure of a water surface height field inversion model according to an embodiment of the present invention;

[0028] Figure 3 This is a graph of loss function training results of an embodiment of the present invention;

[0029] Figure 4 A single image sample and a simulated water surface height field and a predicted water surface height field diagram under high sea conditions in an embodiment of the present invention;

[0030] Figure 5 It is a single image sample and a simulated water surface height field and a predicted water surface height field map under low sea conditions in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, the water surface height field inversion method based on monocular optical image described in this embodiment includes the following steps:

[0033] Step 1: Generate a synthetic image by simulating the propagation of irregular waves through a numerical Boussinesq wave model and calculating the water surface height field, and then generate a synthetic image of the water surface based on the Lambertian reflection model.

[0034] Step 1.1, use FUNWAVE-TVD wave mode to simulate the change of water surface height during the propagation of irregular waves, modify the terrain, grid resolution, boundary conditions in the model setting file, and select different wave makers to simulate multiple groups of wave propagation processes representing different sea conditions, and save the water surface height field Z of the simulation results.

[0035] Step 1.2, using the orthogonal projection mode and the Lambertian reflection model formula, setting the illumination azimuth angle to τ and the illumination incident angle to σ, calculate the brightness of each grid point coordinate (x, y), which is the image brightness E of the corresponding point on the water surface composite image:

[0036] Calculating the light vector

[0037] Calculating surface gradients

[0038] Calculate image intensity

[0039] Step 2: Preprocess the synthetic image and establish a training data set and a test data set according to the sea conditions corresponding to the water surface height field.

[0040] Step 2.1, cut the composite image into a single-channel image of N×N pixels and save it as a separate image sample.

[0041] In step 2.2, the grayscale value of each individual image sample is normalized to the maximum and minimum values ​​and then saved as the model input data.

[0042] Step 2.3, extract the water surface height field corresponding to the coordinates of each individual image sample and save it as model label data.

[0043] Step 2.4, calculate the average sea condition h corresponding to each sample, randomly select samples based on the principle of uniform distribution of different sea conditions to establish training sets and test sets, and the average sea condition h is

[0044] Step 3: Build a water surface height field inversion model. Use the U-Net deep learning network structure to build an artificial intelligence network model for inverting the water surface height field from a monocular image.

[0045] like Figure 2 As shown, the water surface height field inversion model of this embodiment includes an input layer, followed by an encoder consisting of three encoding modules, followed by a bridge layer, followed by a decoder consisting of three decoding modules, followed by a convolution layer consisting of a 1×1 convolution kernel, and finally an output layer.

[0046] Input layer: The input image is a single-channel grayscale image with an image size of N×N pixels.

[0047] Encoding module: It consists of a convolutional module connected to a 2×2 maximum pooling layer.

[0048] Decoding module: It consists of a transposed convolution layer connected to a feature concatenation layer, followed by a convolution module.

[0049] Bridge layer: consists of two convolutional modules.

[0050] Output layer: Linear activation is used to output N×N pixel water surface height field data.

[0051] The structure of the convolution module includes a convolution layer, a batch normalization layer, a RELU activation layer, a convolution layer, a batch normalization layer, a RELU activation layer, and a dropout layer, which are connected in sequence; the dropout rate in the dropout layer is 0.1, and the size of the convolution kernel in the two convolution layers is 3×3.

[0052] The number of convolution kernels is set to different values ​​according to the different layers they are in, as follows: the two convolution kernels used in the first encoding module are both 64, the two convolution kernels used in the second encoding module are both 128, and the two convolution kernels used in the third encoding module are both 256; the two convolution kernels of the first convolution module in the bridge layer are both 512, and the two convolution kernels of the second convolution module in the bridge layer are both 1024; the two convolution kernels used in the first decoding module are both 512, the two convolution kernels used in the second decoding module are both 256, and the two convolution kernels used in the third decoding module are both 128.

[0053] Step 4: training the water surface height field inversion model, using the data set established in step 2 to train the water surface height field inversion model constructed in step 3.

[0054] Step 4.1: Randomly divide the training and validation data prepared in step 2 in proportion.

[0055] Step 4.2, set the training loss function to simulate the water surface height field Z model and the predicted water surface height field Z pred The mean square error between .

[0056] In step 4.3, set the learning rate adjustment strategy, that is, the learning rate is halved when the loss on the validation set does not improve for 10 consecutive rounds.

[0057] In step 4.4, set the early stopping strategy, that is, stop training early when the loss of the training set does not improve significantly (less than 1e-5) within 10 consecutive rounds.

[0058] Step 4.5, start training and wait for the training to end.

[0059] Step 5, applying the water surface height field inversion model, using the optimal model trained in step 4 and the test data set established in step 2, to perform water surface height inversion application.

[0060] Step 5.1, convert the water surface image captured by the monocular optical camera into a grayscale image, cut out an N×N pixel image from it, and perform maximum-minimum value normalization on it.

[0061] Step 5.2, using the image processed in step 5.1 as input and using the model trained in step 4 to perform prediction, to obtain a predicted water surface height field corresponding to each pixel position in the input image.

[0062] In addition, the synthetic optical image can be used as input, and an N×N pixel image can be cut out from it. The image is normalized to the maximum and minimum values ​​to obtain the predicted water surface height field corresponding to each pixel position in the input image. The correlation coefficient and root mean square error between the simulated water surface height field and the predicted water surface height field are calculated to verify the performance of the model.

[0063] The method of the present invention is verified by taking a specific experiment as an example below.

[0064] (1) The FUNWAVE-TVD model was built, with uniform terrain along the coast and a seabed slope of 0.03 across the coast. The simulation area was 500 m along the coast and 512 m across the coast. A wave maker based on the TMA spectrum was used, located 400 m offshore. The spectral parameter range listed in Table 1 was used to simulate wave propagation and the corresponding water surface height field under different sea conditions. The simulation time for each simulation case was 10 minutes, and the water level information at each grid point was saved every 1 second.

[0065] Table 1. Wave simulation example settings used to generate synthetic image training and test sets.

[0066]

[0067] Using the orthogonal projection mode assumption and the Lambertian reflection model formula, the illumination azimuth angle is set to τ = 0, and the illumination incident angle is set to σ = 60°, and a water surface composite image is obtained.

[0068] (2) The synthetic image is cut into a single-channel image of 32×32 pixels, and the grayscale values ​​of the cut individual image samples are normalized to the maximum and minimum values ​​and saved as the model input data; the sea surface height field corresponding to the coordinates of each individual image sample is extracted and saved as the model label data; the average sea condition corresponding to each sample is calculated, 46080 samples are randomly selected from the examples in Table 1 to establish a training set, and 23040 samples are randomly selected from the examples in Table 1 to establish a test set, which are divided into 16 groups at intervals of 0.1m (that is, the average sea conditions corresponding to the samples are <0.1m, 0.1-0.2m, …>1.5m), and the number of samples in each group is the same.

[0069] (3) Construct a water surface height field inversion model based on the U-Net deep learning network.

[0070] (4) The training data is randomly divided into 8:2 for training and validation respectively; the learning rate adjustment strategy is set to halve the learning rate when the loss on the validation set has not improved for 10 consecutive rounds, and the Adam optimizer is used with an initial learning rate of 0.00001; the early stopping strategy is set to stop training early when the loss of the training set has not improved significantly (less than 1e-5) for 10 consecutive rounds, and the maximum training round is 300 rounds; the NVIDIA RTX4090 24GB GPU is used to start training, and the training batch size is set to 16. In this embodiment, the training stops automatically after 160 rounds, which takes 1 hour, 7 minutes and 17 seconds. The training results are as follows Figure 3 shown.

[0071] (5) After preprocessing the test data, the water surface height field is predicted using the trained water surface height field inversion model. Figure 4 The following are diagrams of individual image samples, simulated water surface height field, and predicted water surface height field under high sea conditions (h = 1.0-1.1m). Figure 5 Illustration of individual image samples and simulated and predicted water surface height fields under low sea conditions (h=0.4-0.5m).

[0072] The correlation coefficient and root mean square error between the simulated water surface height field and the predicted water surface height field are calculated. Table 2 shows the verification results of the test data.

[0073] Table 2 Error statistics of water surface height field inverted from the test set

[0074]

[0075] The water surface height field inversion system based on monocular optical images of the present invention comprises:

[0076] A monocular camera image acquisition unit, used to acquire monocular optical images on the water surface;

[0077] A water surface height field inversion unit is used to input the water surface monocular optical image into the trained water surface height field inversion model to obtain a predicted water surface height field for each pixel point in the water surface monocular optical image;

[0078] The water surface height field inversion model training unit is used to simulate the water surface height field by simulating waves as model label data; the image brightness of each pixel is calculated to establish a water surface composite image, and the water surface composite image is input into the U-Net network for training after preprocessing to obtain a trained water surface height field inversion model.

[0079] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the water surface height field inversion method based on monocular optical images is implemented.

[0080] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the water surface height field inversion method based on monocular optical images is implemented.

[0081] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store program code in the form of instructions or data structures and that can be accessed by a computer.

[0082] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A water surface height field inversion method based on monocular optical images, characterized in that: The water surface monocular optical image is input into the trained water surface height field inversion model to obtain the predicted water surface height field of each pixel point in the water surface monocular optical image; The training method of the water surface height field inversion model comprises: simulating the water surface height field by simulating waves and using it as model label data; calculating the image brightness of each pixel to establish a water surface synthetic image, and inputting the water surface synthetic image into a U-Net network for training after preprocessing to obtain a trained water surface height field inversion model.

2. The water surface height field inversion method based on monocular optical images according to claim 1 is characterized in that: The step of calculating the image brightness of each pixel point to establish a water surface composite image comprises: The image brightness at each pixel (x,y) in Z is the simulated water surface height field, τ is the illumination azimuth, and σ is the illumination incident angle.

3. The water surface height field inversion method based on monocular optical images according to claim 1 is characterized in that: The preprocessing of the water surface synthetic image and inputting it into the U-Net network for training comprises: The synthetic water surface image is cut into several individual image samples to match the size of the U-Net network input layer, and each individual image sample is normalized.

4. The water surface height field inversion method based on monocular optical images according to claim 3 is characterized in that: The preprocessing of the water surface synthetic image and inputting it into the U-Net network for training comprises: Calculate the average sea state h corresponding to each individual image sample, and randomly select individual image samples based on the principle of uniform distribution of different sea conditions to establish training sets and test sets; Average sea conditions Var represents the calculated variance, Z(x,y) is the water surface height field of the pixel point (x,y) in a single image sample, and the size of a single image sample is N×N.

5. The water surface height field inversion method based on monocular optical images according to claim 1 is characterized in that: The preprocessing of the water surface synthetic image and inputting it into the U-Net network for training comprises: The U-Net network consists of an input layer, encoder, bridge layer, decoder, convolutional layer, and output layer connected in sequence; The encoder includes three encoding modules connected in sequence, and the encoding module includes a convolution module and a pooling layer connected in sequence; The bridge layer includes two convolution modules connected in sequence; The decoder comprises three decoding modules connected in sequence, wherein the decoding module comprises a transposed convolution layer, a feature concatenation layer and a convolution module connected in sequence; The convolution module includes a convolution layer, a batch normalization layer, an activation layer, a convolution layer, a batch normalization layer, an activation layer, and a discard layer which are connected in sequence.

6. The water surface height field inversion method based on monocular optical images according to claim 1 is characterized in that: The preprocessing of the water surface synthetic image and inputting it into the U-Net network for training comprises: The loss function of the training is set to the mean square error between the simulated water surface height field and the predicted water surface height field.

7. The water surface height field inversion method based on monocular optical images according to claim 1 is characterized in that: The step of inputting the water surface monocular optical image into the trained water surface height field inversion model comprises: The monocular optical image of the water surface taken by the monocular camera is converted into a grayscale image, cut into several image samples to match the size of the U-Net network input layer, and each image sample is normalized.

8. A water surface height field inversion system based on monocular optical images, characterized in that: include: A monocular camera image acquisition unit, used to acquire monocular optical images on the water surface; A water surface height field inversion unit is used to input the water surface monocular optical image into the trained water surface height field inversion model to obtain a predicted water surface height field for each pixel point in the water surface monocular optical image; The water surface height field inversion model training unit is used to simulate the water surface height field by simulating waves as model label data; the image brightness of each pixel is calculated to establish a water surface composite image, and the water surface composite image is input into the U-Net network for training after preprocessing to obtain a trained water surface height field inversion model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the method for inverting the water surface height field based on monocular optical images according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for inverting the water surface height field based on monocular optical images according to any one of claims 1 to 7 is implemented.

Images