Three-dimensional wave surface real-time reconstruction method under ship motion

Abstract

The invention discloses a three-dimensional wave surface real-time reconstruction method under ship motion. The method comprises the following steps: acquiring a ship-borne wave image and a virtual wave image; enhancement of the shipborne wave image under a low-brightness or high-brightness condition is realized based on an unsupervised generative adversarial network; based on the enhanced image and the virtual wave image, carrying out wave feature identification and parameter extraction under ship control motion; and performing three-dimensional wave surface real-time reconstruction on the basis of the wave verification data and the parameters after wave identification and extraction. According to the method, by integrating advanced image processing and artificial intelligence algorithms, wave feature recognition and parameter extraction can be carried out on wave surface images collected when a ship operates and moves under the high sea condition in real time, and then near-field and far-field wave surface three-dimensional reconstruction of the ship is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a real-time reconstruction method of three-dimensional wave surface under ship motion. Background Art

[0002] High-precision water surface wave reconstruction algorithms have always been a hot topic of research at home and abroad, and their implementation process faces many challenges. The most significant problem is that the exposure intensity of the wavefront image collected in the real scene is often not ideal. Too bright or too dark light will directly affect the accuracy of the wavefront reconstruction algorithm. Therefore, how to normalize the real scene wavefront to a unified light intensity is the key to the research. In recent years, with the development of artificial intelligence technology, image processing technology is no longer limited to simple filtering and denoising technologies. Low / high light image enhancement technology based on computer vision has flourished and has been widely used and studied in fields such as night monitoring, autonomous driving, and medical imaging. However, low / high light images usually have problems such as too low / high brightness, low contrast, and high noise, which brings huge challenges to image enhancement. Traditional methods including histogram equalization and Retinex theory have limited effects when dealing with complex scenes. Deep learning methods perform well in the field of image enhancement. It is worth noting that because a large amount of paired data is required to train the model, these algorithms often need to shoot wavefront data with different exposure intensities at the same time and place, which is difficult to do under real-life shooting conditions. This makes unsupervised models the first choice for three-dimensional wavefront reconstruction. It no longer requires paired data to complete model training, and the construction of neural networks becomes simpler. This allows us to pay more attention to the scope of use of the algorithm when improving the model. Specifically, it is to pay more attention to the impact of the spectral type of waves on the accuracy of three-dimensional reconstruction.

[0003] On the other hand, the ship's maneuvering state will also affect the wave surface state, especially the incoming wave surface. Computational fluid dynamics (CFD) technology can be used to simulate the interaction between ship motion and waves, and actual ocean test data can be used for model correction to verify and optimize the wave surface reconstruction model. The Massachusetts Institute of Technology (MIT) has successfully achieved real-time reconstruction of complex water surface waves. They use technologies such as Multi-View Stereo Vision and Optical Flow to capture water surface images from different angles, and use deep learning models to perform data fusion and feature extraction to generate high-precision three-dimensional wave models. In addition, some European research institutions such as ETH Zurich have also achieved important results in this field, especially in proposing innovative methods in combining physical models and data-driven methods.

[0004] In China, research teams from many universities are also actively exploring the application of computer vision in the reconstruction of water surface waves. Researchers from Shandong University of Science and Technology proposed a deep learning-based water surface wave reconstruction method by combining computer vision with physical models, which significantly improved the reconstruction accuracy and computational efficiency. The research team of Harbin Engineering University focused on multi-sensor fusion technology, and improved the robustness and real-time performance of water surface wave reconstruction by combining lidar and camera data. In addition, the Institute of Automation of the Chinese Academy of Sciences has also carried out a lot of research in this field, especially in the dynamic monitoring and prediction of water surface waves, and has made important progress. Overall, domestic and foreign research in the field of water surface wave reconstruction with computer vision has different focuses. The international community pays more attention to algorithm innovation and multi-view data fusion technology, while China has unique advantages in combining physical models and multi-sensor data. With the continuous development of technology, research in this field will continue to deepen and play an increasingly important role in fields such as marine engineering, environmental monitoring and virtual reality. Summary of the invention

[0005] In order to solve the above problems, the technical solution adopted by the present invention is: a real-time reconstruction method of three-dimensional wave surface under ship motion, comprising the following steps:

[0006] Obtain shipborne wave images and virtual wave images;

[0007] Enhancement of shipborne wave images under low or high brightness conditions based on unsupervised generative adversarial networks;

[0008] Based on the enhanced images and virtual wave images, wave feature recognition and parameter extraction under ship maneuvering motion are performed;

[0009] Based on the wave verification data and the wave identification and extracted parameters, the three-dimensional wave surface is reconstructed in real time.

[0010] Further: the unsupervised generative adversarial network includes a generator for converting a low-light image or a high-light image into an enhanced image,

[0011] The discriminator is used to distinguish the enhanced image from the high-quality reference image;

[0012] By jointly training the generator and the discriminator, the generator can generate realistic high-quality images, and the trained generator can be applied to complex lighting images to achieve enhanced effects.

[0013] Further: The standard function of the discriminator is

[0014]

[0015]

[0016] Where C is the discriminator network, x r To sample from the true distribution, x f is sampling from a spurious distribution.

[0017] Further: the process of wave feature recognition and parameter extraction under ship maneuvering motion based on the enhanced image and virtual wave image is as follows:

[0018] The data collected by the binocular camera is preprocessed and corrected using the ship's attitude data to correct the effect of the ship's pitch on wave height measurement and the effect of roll / pitch on wave direction measurement;

[0019] Based on the wave reflection, diffraction and interference effects caused by ship motion, a quantitative model between stern wave, bow wave, side wave and ship motion state is established;

[0020] The Harris corner detection algorithm is used to select points with large intensity changes in the image as feature points, that is, to facilitate subsequent optical flow tracking. Between each pair of adjacent image frames, the Lucas-Kanade optical flow method is used to calculate the motion vector of the feature point. The optical flow value is estimated by minimizing the error in the local area, and finally the wave characteristic parameters of wave height, wavelength, wave speed and wave direction at each point on the sea surface are obtained.

[0021] Further: a quantitative model between the stern wave, bow wave, side wave and the ship motion state;

[0022]

[0023] in is the velocity potential function, A is the wave amplitude coefficient to be determined, ω is the circular frequency of the corresponding wave, is the initial phase, and F(y) is a function used to describe the distribution characteristics of the wave in the y direction and is only related to y.

[0024] Further: the process of real-time reconstruction of the three-dimensional wave surface based on the wave verification data and the wave identification and extracted parameters is as follows:

[0025] According to the conversion relationship between the pixel coordinate system, image coordinate system, camera coordinate system and world coordinate system, the two-dimensional points on the image are converted into three-dimensional points in the world coordinate system. Combined with the actual measured wave parameters, a three-dimensional model of the wave surface is preliminarily constructed.

[0026] The three-dimensional model of the initial wave surface is corrected and optimized by the gradient descent method and the ensemble Kalman filter method, and the wave components are reconstructed;

[0027] By performing fast Fourier transform on the reconstructed wave components, real-time reconstruction of the three-dimensional wave surface is achieved.

[0028] Further: the process of calculating the motion vector of the feature point using the Lucas-Kanade optical flow method is as follows:

[0029] By tracking the feature points in the image sequence, the motion trajectory of the feature points in the time series is calculated. The basic constraint equation is:

[0030] I x u+I y v+I t =0

[0031] Among them I x ,I y ,I t are the partial derivatives of the grayscale of the speed limit point in the image along the X-axis, Y-axis, and T time dimension, respectively. u and v are the velocity vectors of the optical flow along the X-axis and Y-axis, respectively.

[0032] A device for real-time reconstruction of three-dimensional wave surface under ship motion, comprising:

[0033] Acquisition module: used to acquire shipborne wave images and virtual wave images;

[0034] Enhancement module: used to enhance shipborne wave images under low or high brightness conditions based on unsupervised generative adversarial networks;

[0035] Wave feature recognition and parameter extraction: used to identify wave features and extract parameters under ship maneuvering motion based on enhanced images and virtual wave images;

[0036] Reconstruction module: used to reconstruct the three-dimensional wave surface in real time based on wave verification data and wave identification and extracted parameters.

[0037] The present invention provides a real-time reconstruction method for three-dimensional wave surface under ship motion. Aiming at the demand for intelligent prediction of ship maneuverability under high sea conditions, the present invention uses artificial intelligence technology as a guide to develop a full-process wave field intelligent restoration method covering wave surface image enhancement, wave feature recognition, wave parameter extraction and wave surface three-dimensional reconstruction. The method integrates advanced image processing and artificial intelligence algorithms to perform wave feature recognition and parameter extraction on wave surface images collected when the ship is maneuvering under high sea conditions in real time, thereby realizing three-dimensional reconstruction of the ship's near-field and far-field wave surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of a method for real-time reconstruction of three-dimensional wave surface under ship motion;

[0040] Figure 2 The enhanced processing diagram of the real wavefront image, where (a) is a low-light image and (b) is a high-light image;

[0041] Figure 3 This is a 3D reconstruction of the wave surface based on a binocular camera. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Figure 1 It is a flow chart of a method for real-time reconstruction of three-dimensional wave surface under ship motion;

[0045] A method for real-time reconstruction of three-dimensional wave surface under ship motion comprises the following steps:

[0046] S1: Obtain shipborne wave images and virtual wave images;

[0047] S2: Enhancement of shipborne wave images under low or high brightness conditions based on unsupervised generative adversarial networks;

[0048] S3: Based on the enhanced image and the virtual wave image, wave feature recognition and parameter extraction under the ship maneuvering motion are performed;

[0049] S4: Based on the wave verification data and the wave identification and extracted parameters, the three-dimensional wave surface is reconstructed in real time.

[0050] Steps S1 / S2 / S3 / S4 are executed sequentially;

[0051] Furthermore, the unsupervised generative adversarial network includes a generator for converting a low-light image or a high-light image into an enhanced image as follows:

[0052] The unsupervised generative adversarial network includes a generator for converting a low-light image or a high-light image into an enhanced image,

[0053] The discriminator is used to distinguish the enhanced image from the high-quality reference image;

[0054] Enable the generator to generate realistic high-quality images, and apply the trained generator to complex lighting images to achieve enhanced effects;

[0055] Normally, constructing an unsupervised generative adversarial network training dataset requires collecting low-light images / high-light images and their corresponding high-quality reference images. However, the unsupervised generative adversarial network Enlightengan does not require paired data for training and can perform well in a variety of scenarios. It improves model performance through methods such as global-local discriminator structure, self-regularized perceptual loss, and self-regularized attention mechanism, compensates for the shortcomings caused by unpaired data, and reduces the model's dependence on training datasets.

[0056] Furthermore, the wave feature recognition and parameter extraction under the ship maneuvering motion are studied, and the relative position change of the sensor caused by the ship motion and the deviation between the actual observation data of the wave and the wave characteristics under the ideal state are studied;

[0057] S21: Preprocess and correct the images collected by the binocular camera using the ship attitude data, mainly to correct the influence of the ship pitch on the wave height measurement and the influence of the roll / pitch on the wave direction measurement;

[0058] S22: Aiming at the effects of wave reflection, diffraction and interference caused by ship motion, a quantitative model between stern wave, bow wave, side wave and ship motion state is established, and verified and optimized using numerical simulation methods and experimental data.

[0059] S23: Research on wave feature recognition and parameter extraction technology based on Lucas-Kanade optical flow method. Harris corner detection algorithm is used to select points with large intensity changes in the image as feature points, that is, to facilitate the subsequent optical flow tracking. Between each pair of adjacent image frames, Lucas-Kanade optical flow method is used to calculate the motion vector of the feature point, and the optical flow value is estimated by minimizing the error in the local area. Finally, the wave characteristic parameters such as wave height, wavelength, wave speed and wave direction of each point on the sea surface are obtained.

[0060] Furthermore, the wavefront image and wave parameters are combined to reconstruct the wavefront in three dimensions. The specific process is as follows:

[0061] S31: Research on the three-dimensional reconstruction technology of wave surface based on binocular stereo vision. A series of image data are captured by a binocular camera system. The two-dimensional points on the image are converted into three-dimensional points in the world coordinate system according to the conversion relationship between the pixel coordinate system, the image coordinate system, the camera coordinate system and the world coordinate system. Combined with the actual measured wave parameters, a three-dimensional model of the wave surface is preliminarily constructed.

[0062] S32: The three-dimensional model of the wave surface initially constructed is corrected and optimized by the gradient descent method and the ensemble Kalman filter method, and the wave components are reconstructed. The three-dimensional wave surface is reconstructed in real time by performing fast Fourier transform on the reconstructed wave components.

[0063] Combining these technical means, three-dimensional reconstruction of wave surfaces based on computer vision can provide high-precision description of wave morphology, providing important data support for fields such as marine engineering, ship navigation and environmental monitoring.

[0064] A real-time reconstruction device for three-dimensional wave surface under ship motion, characterized by comprising:

[0065] Acquisition module: used to acquire shipborne wave images and virtual wave images;

[0066] Enhancement module: used to enhance shipborne wave images under low or high brightness conditions based on unsupervised generative adversarial networks;

[0067] Wave feature recognition and parameter extraction: used to identify wave features and extract parameters under ship maneuvering motion based on enhanced images and virtual wave images;

[0068] Reconstruction module: used to reconstruct the three-dimensional wave surface in real time based on wave verification data and wave identification and extracted parameters.

[0069] Embodiment 1: A method for real-time reconstruction of three-dimensional wave surface under ship motion, comprising the following steps:

[0070] First, according to the distance between the selected wavefront data collection location and the wavefront, a dual-camera sea surface image acquisition system is set up. Generally speaking, a larger camera spacing can provide more perspective differences, thereby more accurately calculating the depth of the object. Therefore, the farther the camera spacing is, the higher the accuracy of the three-dimensional reconstruction. However, a larger camera spacing will also lead to difficulties in feature matching during reconstruction. Therefore, the camera spacing should be flexibly adjusted according to the distance between the selected wavefront data collection location and the wavefront.

[0071] Figure 2 These are the enhanced processing images of the real wavefront image; (a) is a low-light image, and (b) is a high-light image;

[0072] The wavefront data collected by the binocular camera can be preprocessed to form a training data set. The preprocessing mainly includes filtering, classification and obstacle recognition. The images will be divided into two groups: underexposed and overexposed. The wavefront images containing obstacles or occlusions will be processed separately and the occlusions will be removed to avoid their impact on network training. At the same time, an improved Enlightengan model is established. By introducing multi-scale convolutional layers, the features of different scales, especially the wavefront details, are better captured, the adversarial loss function is improved, the training effect of the generator and the discriminator is enhanced, and the generated images are more realistic. The standard function relative to the discriminator is:

[0073]

[0074]

[0075] Where C is the discriminator network, x r To sample from the true distribution, x f To sample from a spurious distribution;

[0076] S2: To study the impact of ship motion on wave observation data, it is first necessary to use the ship attitude data to preprocess and correct the sensor data. When a ship moves in the ocean, its pitch, roll, and pitch will have a significant impact on the measurement of wave height and wave direction. In order to correct these effects, high-precision attitude sensors (such as inertial measurement units IMU) can be used to monitor the attitude changes of the ship in real time. By fusing the attitude data with the wave observation data, the wave sensor data is corrected using the attitude compensation algorithm, thereby reducing the interference of the ship motion on the observation results.

[0077] It is key to establish a quantitative model for the wave reflection, diffraction and interference effects caused by ship motion.

[0078] Firstly, a mathematical model of the interaction between waves and hull is constructed through theoretical analysis and experimental data.

[0079]

[0080] in: is the velocity potential function, A is the wave amplitude coefficient to be determined, ω is the circular frequency of the corresponding wave, is the initial phase, and F(y) is a function used to describe the distribution characteristics of the wave in the y direction and is only related to y.

[0081] Next, the model is numerically simulated using the finite volume method and spectral method in computational fluid dynamics (CFD) methods to simulate the wave reflection, diffraction and interference effects of ships under different sea conditions. In order to improve the accuracy of the model, physical model tests are also required to verify and optimize the numerical model through experimental data. In the experiment, a wave tank or a large ocean test pool can be used to simulate real sea conditions and obtain a large amount of high-precision data for model correction.

[0082] In terms of wave feature recognition, the Harris corner detection algorithm is used to select feature points in the image. The Harris corner detection algorithm is a corner detection method based on image grayscale changes, which can effectively identify significant feature points in the image; the weighted SSD, that is, the mathematical expression of grayscale change, is:

[0083]

[0084] The image function is I, the pixel coordinates are (x, y), the sliding variable of the sliding window is (u, v), and ω(x, y) is the weighting function or window function.

[0085] After selecting the feature points, the Lucas-Kanade optical flow method is used to calculate the motion vector of the feature points. The Lucas-Kanade optical flow method is a classic optical flow calculation method that calculates the motion trajectory of the feature points in the time series by tracking the feature points in the image sequence. Its basic constraint equation is:

[0086] I x u+I y v+I t =0

[0087] Among them I x ,I y ,I t are the partial derivatives of the grayscale of the speed limit point in the image along the X-axis, Y-axis, and T time dimension, respectively. u and v are the velocity vectors of the optical flow along the X-axis and Y-axis, respectively.

[0088] Combined with these motion vectors, wave characteristic parameters such as wave height, wavelength, wave speed and wave direction can be further extracted. Specifically, by analyzing the vertical motion amplitude and period of the feature points, the wave height and wavelength can be calculated; by the horizontal motion speed of the feature points, the wave speed and wave direction can be inferred.

[0089] S3: Based on binocular stereo vision technology, the wavefront image data is captured by a binocular camera system, and the two-dimensional points are converted into three-dimensional points using feature point detection and matching technology. Combined with the actual measured wave parameters, a three-dimensional model of the wavefront is preliminarily constructed. In the calibration part, Zhang's calibration method is proposed to simultaneously establish the coordinate transformation relationship from the world coordinate system, to the camera coordinate system, to the image coordinate system, and finally to the pixel coordinate system. Figure 3 This is a three-dimensional reconstruction of the wave surface based on a binocular camera;

[0090] The matrix expression for converting the world coordinate system to the camera coordinate system is:

[0091]

[0092] The expression for converting the camera coordinate system to the image coordinate system is:

[0093]

[0094] The matrix expression for converting the image coordinate system to the pixel coordinate system is:

[0095]

[0096] The iterative optimization algorithm and data assimilation technology are further applied to correct and optimize the three-dimensional model of the preliminary wave surface through the gradient descent method and the ensemble Kalman filter method, and the wave components are reconstructed. The three-dimensional wave surface is reconstructed in real time by performing fast Fourier transform on the reconstructed wave components.

[0097] This technical route integrates artificial intelligence and computer vision technologies to provide high-precision wave form description and data support for fields such as marine engineering, ship navigation and environmental monitoring.

[0098] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for real-time reconstruction of three-dimensional wave surface under ship motion, characterized in that: The following steps are involved: Obtain shipborne wave images and virtual wave images; Enhancement of shipborne wave images under low or high brightness conditions based on unsupervised generative adversarial networks; Based on the enhanced images and virtual wave images, wave feature recognition and parameter extraction under ship maneuvering motion are performed; Based on the wave verification data and the wave identification and extracted parameters, the three-dimensional wave surface is reconstructed in real time.

2. The method for real-time reconstruction of three-dimensional wave surface under ship motion according to claim 1, characterized in that: The unsupervised generative adversarial network includes a generator for converting a low-light image or a high-light image into an enhanced image, The discriminator is used to distinguish the enhanced image from the high-quality reference image; By jointly training the generator and the discriminator, the generator can generate realistic high-quality images, and the trained generator can be applied to complex lighting images to achieve enhanced effects.

3. The method for real-time reconstruction of three-dimensional wave surface under ship motion according to claim 2, characterized in that: The standard function of the discriminator is Where C is the discriminator network, x r To sample from the true distribution, x f is sampling from a spurious distribution.

4. The method for real-time reconstruction of three-dimensional wave surface under ship motion according to claim 1, characterized in that: The process of wave feature recognition and parameter extraction under ship maneuvering motion based on the enhanced image and virtual wave image is as follows: The data collected by the binocular camera is preprocessed and corrected using the ship's attitude data to correct the effect of the ship's pitch on wave height measurement and the effect of roll / pitch on wave direction measurement; Based on the wave reflection, diffraction and interference effects caused by ship motion, a quantitative model between stern wave, bow wave, side wave and ship motion state is established; The Harris corner detection algorithm is used to select points with large intensity changes in the image as feature points, that is, to facilitate subsequent optical flow tracking. Between each pair of adjacent image frames, the Lucas-Kanade optical flow method is used to calculate the motion vector of the feature point. The optical flow value is estimated by minimizing the error in the local area, and finally the wave characteristic parameters of wave height, wavelength, wave speed and wave direction at each point on the sea surface are obtained.

5. The method for real-time reconstruction of three-dimensional wave surface under ship motion according to claim 1, characterized in that: A quantitative model between the stern wave, bow wave, side wave and the ship motion state; in is the velocity potential function, A is the wave amplitude coefficient to be determined, ω is the circular frequency of the corresponding wave, is the initial phase, and F(y) is a function used to describe the distribution characteristics of the wave in the y direction and is only related to y.

6. The method for real-time reconstruction of three-dimensional wave surface under ship motion according to claim 1, characterized in that: The process of real-time reconstruction of the three-dimensional wave surface based on the wave verification data and the wave identification and extracted parameters is as follows: According to the conversion relationship between the pixel coordinate system, image coordinate system, camera coordinate system and world coordinate system, the two-dimensional points on the image are converted into three-dimensional points in the world coordinate system. Combined with the actual measured wave parameters, a three-dimensional model of the wave surface is preliminarily constructed. The three-dimensional model of the initial wave surface is corrected and optimized by the gradient descent method and the ensemble Kalman filter method, and the wave components are reconstructed; By performing fast Fourier transform on the reconstructed wave components, real-time reconstruction of the three-dimensional wave surface is achieved.

7. The method for real-time reconstruction of three-dimensional wave surface under ship motion according to claim 1, characterized in that: The process of calculating the motion vector of the feature point using the Lucas-Kanade optical flow method is as follows: By tracking the feature points in the image sequence, the motion trajectory of the feature points in the time series is calculated. The basic constraint equation is: I x u+I y v+I t =0 Among them I x ,I y ,I t are the partial derivatives of the grayscale of the speed limit point in the image along the X-axis, Y-axis, and T time dimension, respectively. u and v are the velocity vectors of the optical flow along the X-axis and Y-axis, respectively.

8. A real-time reconstruction device for three-dimensional wave surface under ship motion, characterized in that: include: Acquisition module: used to acquire shipborne wave images and virtual wave images; Enhancement module: used to enhance shipborne wave images under low or high brightness conditions based on unsupervised generative adversarial networks; Wave feature recognition and parameter extraction: used to identify wave features and extract parameters under ship maneuvering motion based on enhanced images and virtual wave images; Reconstruction module: used to reconstruct the three-dimensional wave surface in real time based on wave verification data and wave identification and extracted parameters.

Images