A wave field reconstruction method based on optical perception

Abstract

The present invention relates to a wave field reconstruction method based on optical perception, comprising the following steps: step 1, building a virtual wave field data generation platform; step 2, using virtual wave field color image data and real water surface image data captured by a camera as training sets to pre-train a constructed domain converter module; step 3, using the generated pairs of virtual wave field color image data and depth image data as training sets to construct a depth estimation module for pre-training; step 4, using the domain converter module to convert the style of the real water surface image into a virtual water surface style image; step 5, outputting a virtual water surface style depth image that records the distance between the wave surface sampling point and the camera optical center; and step 6, generating surface point cloud data of the water surface wave field with the camera optical center as the coordinate origin. The wave field reconstruction method of the present invention can achieve real-time reconstruction of the water surface wave field depth information by capturing the water surface image with a camera, thereby ensuring the reliability and authenticity of the wave field reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional wave field reconstruction and reconstruction, and in particular to a wave field reconstruction method based on optical perception. Background Art

[0002] With the development and application of artificial intelligence technology in the field of equipment, the intelligent system of unmanned boats has also ushered in a wave of intelligence. Different from unmanned vehicles and drones, unmanned boats face complex and changeable water surface environments. Due to the lack of perception of the surrounding water surface conditions, the motion control accuracy of traditional unmanned boats is very limited, making it difficult to control unmanned boats to complete tasks that require high motion accuracy.

[0003] In the field of fluid research, previous research on wave field reconstruction technology has been carried out mainly from three aspects. The first is to perform special treatment on the water body, dye the water body with fluorescent substances, analyze the fluorescence intensity of different areas of the captured image to reconstruct the water surface, or paste a known image texture under the transparent water body, and analyze the relationship between texture distortion and water surface shape based on the principle of refraction. This method is only suitable for laboratory use; the second is to use a camera to photograph the ocean surface, and combine physical equations for numerical settlement during settlement, but the amount of calculation is too large and the calculation is time-consuming, and the effect of real-time reconstruction cannot be achieved; the third is to use the SFS algorithm to reconstruct the wave field, but the wave reconstruction algorithm based on the SFS algorithm does not change the assumption of orthogonal projection, and cannot deal with the projection occlusion problem of the wave field well. Summary of the invention

[0004] In view of the above situation, the present invention provides a wave field reconstruction method based on optical perception, which can intuitively show the fluctuation information of the water surface wave field; by shooting the water surface image with a camera, the depth information of the water surface wave field can be reconstructed in real time with fast calculation speed; there is no special requirement for the water body and it has strong adaptability; the projection occlusion problem of the wave field is taken into consideration to ensure the reliability and authenticity of the wave field reconstruction.

[0005] The specific technical solution of the present invention is a wave field reconstruction method based on optical perception, which is characterized by comprising the following steps:

[0006] Step 1: Build a virtual wave field data generation platform, use a composite illumination model to simulate the optical characteristics of the water surface, and generate pairs of virtual wave field color image data and depth image data;

[0007] Step 2, constructing a domain converter module, using virtual wave field color image data and real water surface image data taken by a camera as training sets to pre-train the constructed domain converter module;

[0008] Step 3, constructing a depth estimation module, using the generated pairs of virtual wave field color image data and depth image data as a training set to construct a depth estimation module for pre-training;

[0009] Step 4, take a real water surface picture, and use the domain converter module pre-trained in step 2 to convert the style of the real water surface picture into a virtual water surface style picture;

[0010] Step 5, using a depth estimation module to output the virtual water surface style image as a virtual water surface style depth image that records the distance between the wave surface sampling point and the camera optical center;

[0011] Step 6: Use a point cloud mapping algorithm to perform coordinate mapping on the virtual water surface style depth image to generate surface point cloud data of the water surface wave field with the camera optical center as the coordinate origin.

[0012] Furthermore, the virtual wave field data generation platform described in step 1 is constructed using the Gerstner wave model, and the Gerstner wave model is as follows (I):

[0013]

[0014] Where t is time, (x, y, z) is the position at time t, (x0, y0, z0) is the position at rest, Q i is the sharpness of the peak of the superimposed i-th wave, and its value range is A i is the amplitude of the superimposed ith wave, k i is the wave number of the superimposed ith wave, θ i is the direction angle of the superimposed ith wave, ω i is the angular velocity of the superimposed ith wave, is the initial phase of the superimposed ith wave.

[0015] Furthermore, the composite illumination model described in step 1 is a diffuse reflection model, a specular reflection model, and a subsurface scattering model, which are respectively shown in the following equations (II), (III), and (IV):

[0016] I d =k d I in cosα......(II)

[0017] Among them I d is the observed color brightness, k d is the diffuse reflectance, I in is the incident light brightness, α is the angle between the plane normal and the incident light;

[0018] I m =km I in (cos(φ-θ)) n ......(III)

[0019] Among them I m is the observed color brightness, k m is the specular reflection coefficient, I in is the incident light brightness, φ is the angle between the plane normal and the incident light, θ is the angle between the plane normal and the observation line of sight, and n is the specular coefficient;

[0020]

[0021] Where L i is the radiance of the incident light, L o is the radiance of the outgoing light, p i is the point of contact between the incident light and the object, p o is the contact point between the outgoing light and the object, r i is the direction vector of the incident light, r o is the direction vector of the outgoing light, is the angle between the incident light and the normal vector of the microplane at the contact point, S(p o ,r o ,p i ,r i ) is the bidirectional subsurface reflection distribution function, F t is the Fresnel coefficient, R d is the diffusion approximation function, and d is the distance.

[0022] Furthermore, the process of generating paired wave field color image data and depth image data in step 1 is as follows:

[0023] Step a, using a virtual wave field data generation platform to generate a two-dimensional image of a virtual wave field, and writing the minimum value of the depth in the same pixel into a depth buffer;

[0024] Step b, according to the depth of the virtual wave field two-dimensional image, select the color closest to the camera to render each pixel in turn, and generate the required virtual wave field color image;

[0025] Step c, performing an inverse perspective transformation on the value in the depth buffer, converting the image coordinate system into the world coordinate system, obtaining the true depth value, and generating the required virtual wave field depth image.

[0026] Furthermore, the specific process of the point cloud mapping algorithm described in step 6 is as follows:

[0027] Step A, by translating the pixel coordinate system, translating the center point of the coordinate system from the upper left corner of the image to the center of the image, and then multiplying the conversion factor, the pixel coordinate system is converted to the image coordinate system, see the following formula (V),

[0028]

[0029] Among them, uv is the pixel coordinate system, the origin is the upper left corner of the image, w and h are the pixel values ​​of the image width and height respectively, o-xy is the image coordinate system, the origin is the center point of the image, r x 、r y is the conversion factor, which represents the absolute distance represented by one pixel in the horizontal and vertical directions of the image. The calculation formula is as follows (VI):

[0030]

[0031] Among them, α is the vertical opening angle of the camera's visual cone, β is the horizontal opening angle of the camera's visual cone, and f is the focal length of the camera;

[0032] Step B: normalize the vector (x, y, f) from the camera optical center to the pixel point according to the following formula (VII) to obtain (x e ,y e ,z e ), which represents the direction from the camera optical center to the point on the image, and is also the direction from the camera optical center to the real point in the world coordinate system:

[0033]

[0034] Step C, transform the vector (x e ,y e ,z e ) is multiplied by the distance between the camera optical center and the point in the world coordinate system stored in the depth map to obtain the position of the point in the camera coordinate system with the camera optical center as the origin, as shown in the following formula (VIII):

[0035]

[0036] Advantages or beneficial effects of the present invention:

[0037] 1) By introducing virtual wave field data to solve the problem of data acquisition, unsupervised learning is used to connect virtual data and real data, and a feasible water surface wave field perception technology framework that can be implemented under existing technical means is proposed;

[0038] 2) In order to solve the problem that it is difficult to obtain the true value of water surface wave field data, a large amount of virtual wave field data can be generated for algorithm training by using Gerstner wave superposition to simulate the wave field shape, using a combination of multiple illumination models to simulate the wave field optical characteristics, and using a rendering pipeline to generate the required color image data and depth image data, thus solving the problem of obtaining training data sets;

[0039] 3) The relationship between the color image and the depth image is regarded as an image translation problem, which avoids the "edge blur" problem caused by the artificial setting of the loss function in traditional monocular depth estimation;

[0040] 4) The virtual wave field data generation platform built takes into account the projection occlusion problem of the wave field when generating the virtual wave field image, ensuring the reliability and authenticity of the wave field reconstruction;

[0041] 5) This method has no special requirements for water bodies, has strong adaptability, and the wave field reconstruction speed reaches milliseconds, and can perform real-time wave field reconstruction in outdoor waters. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the flow of the wave field reconstruction method based on optical perception of the present invention;

[0043] Figure 2 Schematic diagram of the Gerstner wave superposition simulation under different wave conditions;

[0044] Figure 3 To add the wave field effect diagram of diffuse reflection model, specular reflection model and subsurface scattering model;

[0045] Figure 4a-4b is the rendered virtual wave field image, where Figure 4a is a color image of the virtual wave field, Figure 4b is the depth image of the virtual wave field;

[0046] Figure 5 This is a schematic diagram of camera coordinate mapping;

[0047] Figure 6a-Figure 6b This is the input and output diagram of the domain converter module, where: Figure 6a The input of the domain converter module is the real water surface image. Figure 6b The output of the domain converter module is the virtual water surface style image;

[0048] Figure 7a-Figure 7b This is a schematic diagram of the input and output of the depth estimation module, where: Figure 7a The input of the depth estimation module is the virtual water surface style picture. Figure 7b The output of the depth estimation module is the virtual water surface style depth image;

[0049] Figure 8a-8c Schematic diagram of input and output data of point cloud mapping algorithm, where: Figure 8a , Figure 8b The input of the point cloud mapping module is the virtual water surface style image and the virtual water surface style depth image. Figure 8c The output of the point cloud mapping module is the wave field point cloud data image. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0051] like Figure 1 As shown, the specific steps of a wave field reconstruction method based on optical perception of the present invention are as follows:

[0052] Step 1, build a virtual wave field data generation platform. Use the Unity development platform to construct the shape model and optical model of the virtual wave field respectively, and use the Unity engine to generate virtual wave field images. Unity is a real-time 3D interactive content creation and operation platform that provides a complete set of software solutions for services including game development, art, architecture, automotive design, film and television, and can be used to create, operate and monetize any real-time interactive 2D and 3D content. Use Gerstner wave superposition to simulate the shape of the wave field, obtain the height and normal information of each point, use a composite lighting model to simulate the optical characteristics of the water surface, use the Gerstner wave output information to calculate the color brightness of each point, and render a color image based on the brightness information to generate pairs of virtual wave field color image data and depth image data. The specific steps are as follows:

[0053] The range of the model parameters is Q i ∈[0,1], k i ∈[0,512],θ i ∈[0,2π],ω i ∈[0,20]. Figure 2 As shown, in this example, n=6, that is, 6 groups of waveforms are superimposed, where Q i All values ​​are 1, t, All values ​​are 0, θ i The values ​​are 0.187, 0.867, 1.590, 0.006, 2.625, and 1.328 respectively. i The values ​​are 0.1, 0.2, 0.5, 0.01, 0.01, 0.01, respectively. i The values ​​are 0.125, 0.25, 0.5, 1, 2, and 4, respectively. The permutations and combinations generate different Gerstner wave superposition simulation wave field shapes, where the Gerstner wave formula is as follows:

[0054]

[0055] Where t is time, (x, y, z) is the position at time t, (x0, y0, z0) is the position at rest, Q i is the sharpness of the peak of the superimposed i-th wave, and its value range is A i is the amplitude of the superimposed ith wave, k i is the wave number of the superimposed ith wave, θ i is the direction angle of the superimposed ith wave, ω i is the angular velocity of the superimposed ith wave, is the initial phase of the superimposed Ith wave;

[0056] like Figure 3 As shown, the composite illumination model is a diffuse reflection model, a specular reflection model and a subsurface scattering model, which are respectively shown in the following formulas (II), (III) and (IV):

[0057] I d =k d I in cosα......(II)

[0058] Among them I d is the observed color brightness, k d is the diffuse reflectance, I in is the incident light brightness, α is the angle between the plane normal and the incident light;

[0059] I m =k m I in (cos(φ-θ)) n ......(III)

[0060] Among them I m is the observed color brightness, k m is the reflection coefficient, I in is the incident light brightness, φ is the angle between the plane normal and the incident light, θ is the angle between the plane normal and the observation line of sight, and n is the specular coefficient;

[0061]

[0062] Where L i is the radiance of the incident light, L o is the radiance of the outgoing light, p i is the point of contact between the incident light and the object, p o is the contact point between the outgoing light and the object, r i is the direction vector of the incident light, r ois the direction vector of the outgoing light, is the angle between the incident light and the normal vector of the microplane at the contact point, S(p o ,r o ,p i ,r i ) is the bidirectional subsurface reflection distribution function, F t is the Fresnel coefficient, R d is the diffusion approximation function, and d is the distance.

[0063] The color brightness values ​​calculated by the three sets of illumination models are superimposed to obtain the final brightness value of the point. The process of generating paired wave field color image data and depth image data is as follows:

[0064] Step a, using a virtual wave field data generation platform to generate a two-dimensional image of a virtual wave field, and writing the minimum value of the depth in the same pixel into a depth buffer;

[0065] Step b, according to the depth of the virtual wave field two-dimensional image, use the Unity rendering function to perform the rendering process, according to the calculation results of the composite illumination model, select the color closest to the camera to render each pixel in turn, and generate the required virtual wave field color image, such as Figure 4a As shown;

[0066] Step c, perform inverse perspective transformation on the value in the depth buffer, convert the image coordinate system into the world coordinate system, obtain the true depth value, and generate the required virtual wave field depth image, such as Figure 4b shown.

[0067] Step 2, construct a domain converter module and perform pre-training. The domain converter module is a picture style conversion technology, which is a technology that converts a picture into a new picture with the same content but other styles. Dual Gan, Cycle Gan, Pix2Pix and SSIM-Gan models are usually used. The method of the present invention adopts the existing Cycle Gan model of Pytorch software. The training set of the model specifically adopts the real water surface image taken by the camera and the virtual wave field color image generated in step 1 to learn the style transfer from the real water surface image to the virtual water surface image. The network is composed of a generator and a discriminator. In this example, the structure of the generator is an encoder composed of two convolutional layers, a converter composed of 9 Resnet blocks, and a decoder composed of a deconvolution layer and a convolutional layer. The discriminator uses a PatchGAN network, and the structure is a feature extractor composed of four convolutional layers and a classifier composed of one convolutional layer.

[0068] Step 3, construct a depth estimation module and perform pre-training. Depth estimation is a computer vision task that aims to estimate depth from a 2D image. This task requires an RGB image as input and a depth image as output. The depth image includes information about the distance from the viewpoint to the object in the image. The viewpoint is usually the camera that took the image. This patent uses the existing Pix2Pix model of the Pytorch software, and specifically uses the paired virtual wave field color image data and depth image data generated in step 1 as a training set to learn the transformation of the wave field color image to the depth image. The model consists of a generator 2 and a discriminator 2. In this example, the generator 2 uses a U-Net model. In this example, the U-Net model consists of 4 layers of convolution and deconvolution layers, and jump connections are added between symmetrical convolution layers. The discriminator 2 uses a PatchGAN network, which is a feature extractor 2 composed of four convolution layers and a classifier 2 composed of one convolution layer.

[0069] Step 4, such as Figure 6a-Figure 6b As shown, a real water surface picture is taken, and the domain converter module is used to convert the real water surface picture into a virtual water surface style picture.

[0070] Step 5, such as Figure 7a-Figure 7b As shown, the converted virtual water surface style image is input, and the depth estimation module is used to output a virtual water surface style depth image that records the distance between the wave surface sampling point and the camera optical center.

[0071] Step 6, such as Figure 8a-8c As shown, the point cloud mapping algorithm module is used to map the virtual water surface style depth image. The camera coordinate mapping diagram is shown in Figure 5 As shown in the figure, the surface point cloud data of the water surface wave field with the camera optical center as the coordinate origin is generated. The specific steps are as follows:

[0072] Step A, by translating the pixel coordinate system, translating the center point of the coordinate system from the upper left corner of the image to the center of the image, and then multiplying the conversion factor, the pixel coordinate system is converted to the image coordinate system, see the following formula (V),

[0073]

[0074] Among them, uv is the pixel coordinate system, the origin is the upper left corner of the image, w and h are the pixel values ​​of the image width and height respectively, o-xy is the image coordinate system, the origin is the center point of the image, r x 、r y is the conversion factor, which represents the absolute distance represented by one pixel in the horizontal and vertical directions of the image. The calculation formula is as follows (VI):

[0075]

[0076] Among them, α is the vertical opening angle of the camera's visual cone, β is the horizontal opening angle of the camera's visual cone, and f is the focal length of the camera;

[0077] Step B: normalize the vector (x, y, f) from the camera optical center to the pixel point according to the following formula (VII) to obtain (x e ,y e ,z e ), which represents the direction from the camera optical center to the point on the image, and is also the direction from the camera optical center to the real point in the world coordinate system:

[0078]

[0079] Step C, transform the vector (x e ,y e ,z e ) is multiplied by the distance between the camera optical center and the point in the world coordinate system stored in the depth map to obtain the position of the point in the camera coordinate system with the camera optical center as the origin, as shown in the following formula (VIII):

[0080]

[0081] After the above steps, the three-dimensional reconstruction of the waves can be achieved through a water surface picture taken by a camera, and the point cloud data of the water surface wave field can be obtained.

[0082] Although the present invention has been disclosed as above with preferred embodiments, the embodiments are not intended to limit the present invention. Any equivalent changes or modifications made without departing from the spirit and scope of the present invention also belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the content defined by the claims of this application.

Claims

1. A wave field reconstruction method based on optical perception, characterized in that: The steps include: Step 1: Build a virtual wave field data generation platform, use a composite illumination model to simulate the optical characteristics of the water surface, and generate pairs of virtual wave field color image data and depth image data; Step 2, constructing a domain converter module, using virtual wave field color image data and real water surface image data taken by a camera as training sets to pre-train the constructed domain converter module; Step 3, constructing a depth estimation module, using the generated pairs of virtual wave field color image data and depth image data as a training set to construct a depth estimation module for pre-training; Step 4, take a real water surface picture, and use the domain converter module pre-trained in step 2 to convert the style of the real water surface picture into a virtual water surface style picture; Step 5, using a depth estimation module to output the virtual water surface style image as a virtual water surface style depth image that records the distance between the wave surface sampling point and the camera optical center; Step 6, using a point cloud mapping algorithm to perform coordinate mapping on the virtual water surface style depth image, and generate surface point cloud data of the water surface wave field with the camera optical center as the coordinate origin; The composite illumination model described in step 1 is a diffuse reflection model, a specular reflection model, and a subsurface scattering model, which are respectively shown in the following equations (II), (III), and (IV): I d =k d I in cosα......(II) Among them I d is the observed color brightness, k d is the diffuse reflectance, I in is the incident light brightness, α is the angle between the plane normal and the incident light; I m =k m I in (cos(φ-θ)) n ......(III) Among them I m is the observed color brightness, k m is the specular reflection coefficient, I in is the incident light brightness, φ is the angle between the plane normal and the incident light, θ is the angle between the plane normal and the observation line of sight, and n is the specular coefficient; Where L i is the radiance of the incident light, L o is the radiance of the outgoing light, p i is the point of contact between the incident light and the object, p o is the contact point between the outgoing light and the object, r i is the direction vector of the incident light, r o is the direction vector of the outgoing light, is the angle between the incident light and the normal vector of the microplane at the contact point, S(p o ,r o ,p i ,r i ) is the bidirectional subsurface reflection distribution function, F t is the Fresnel coefficient, R d is the diffusion approximation function, d is the distance; The specific process of the point cloud mapping algorithm described in step 6 is as follows: Step A, by translating the pixel coordinate system, translating the center point of the coordinate system from the upper left corner of the image to the center of the image, and then multiplying the conversion factor, the pixel coordinate system is converted to the image coordinate system, see the following formula (V), Among them, uv is the pixel coordinate system, the origin is the upper left corner of the image, w and h are the pixel values ​​of the image width and height respectively, o-xy is the image coordinate system, the origin is the center point of the image, r x 、r y is the conversion factor, which represents the absolute distance represented by one pixel in the horizontal and vertical directions of the image. The calculation formula is as follows (VI): Among them, α is the vertical opening angle of the camera's visual cone, β is the horizontal opening angle of the camera's visual cone, and f is the focal length of the camera; Step B: normalize the vector (x, y, f) from the camera optical center to the pixel point according to the following formula (VII) to obtain (x e ,y e ,z e ), which represents the direction from the camera optical center to the point on the image, and is also the direction from the camera optical center to the real point in the world coordinate system: Step C, transform the vector (x e ,y e ,z e ) is multiplied by the distance between the camera optical center and the point in the world coordinate system stored in the depth map to obtain the position of the point in the camera coordinate system with the camera optical center as the origin, as shown in the following formula (VIII):

2. A wave field reconstruction method based on optical perception according to claim 1, characterized in that: The virtual wave field data generation platform described in step 1 is constructed using the Gerstner wave model. The Gerstner wave model is as follows (I): Where t is time, (x, y, z) is the position at time t, (x0, y0, z0) is the position at rest, Q i is the sharpness of the peak of the superimposed i-th wave, and its value range is A i is the amplitude of the superimposed ith wave, k i is the wave number of the superimposed i-th wave, θ i is the direction angle of the superimposed ith wave, ω i is the angular velocity of the superimposed ith wave, is the initial phase of the superimposed ith wave.

3. The wave field reconstruction method based on optical perception according to claim 1, characterized in that: The process of generating paired wave field color image data and depth image data in step 1 is as follows: Step a, using a virtual wave field data generation platform to generate a two-dimensional image of a virtual wave field, and writing the minimum value of the depth in the same pixel into a depth buffer; Step b, according to the depth of the virtual wave field two-dimensional image, select the color closest to the camera to render each pixel in turn, and generate the required virtual wave field color image; Step c, performing an inverse perspective transformation on the value in the depth buffer, converting the image coordinate system into the world coordinate system, obtaining the true depth value, and generating the required virtual wave field depth image.

Images