Autonomous Underwater Vehicle (AUV) autonomous docking semi-physical simulation guide image generation method

Abstract

The invention belongs to the technical field of deep sea detection, and particularly relates to an AUV autonomous docking semi-physical simulation guide image generation method. The autonomous docking guide image generation mainly comprises three parts of guide lamp imaging position calculation, binary guide image construction and color guide image generation. Based on a camera imaging principle, according to the pose of an AUV relative docking device, the imaging position of a guide lamp is calculated, a guide lamp image is created, the condition that the imaging of the guide lamp is not only one point is considered, a guide lamp imaging area estimation network is constructed, the area of an overexposure dot is estimated, a binary guide image is constructed, and the result is taken as an input condition. A Pix2pix generative network is adopted to simulate guide image color distribution, a Pix2pix network training data set is constructed by utilizing a guide image obtained by exploring a 11000AUV sea test, the generative network can map the constructed binary guide image to generate a color guide image through training, the image can be used as a docking guide image for simulating camera perception, and the image can be used as a docking guide image for simulating camera perception. The method is used in AUV system autonomous docking and long-term residence full-process operation semi-physical simulation, the development and debugging efficiency of the long-term residence AUV system is improved, and the test cost is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of deep sea exploration, and in particular relates to a method for generating a semi-physical simulation guidance image for autonomous docking of an AUV. Background Art

[0002] The deep seabed contains a large amount of mineral resources and key ecosystems of the earth. It is an underwater area that humans know little about and urgently need to explore. Autonomous underwater vehicles (AUVs) can carry various payloads to perform various detection operations. They are widely used in deep-sea detection fields such as mineral resource exploration, biological surveys, and environmental monitoring. Due to the limited energy carried by AUVs, they need to return to auxiliary ships for energy replenishment, data exchange and maintenance after completing a single detection mission, interrupting the continuity of operations. With the urgent need for long-term, time- and space-continuous detection data in the study of deep-sea scientific issues (such as the operating mechanism of marine ecosystems), deep-sea long-term resident AUVs (RAUVs) came into being. RAUVs ​​can dock with deep-sea base stations through autonomous docking technology, replenish energy and exchange data in the base station, and enter and exit the deep-sea base station multiple times on the deep seabed for a long time, conduct large-scale, long-term detection operations, and obtain detection data with time and space continuity. Since RAUVs ​​stay on the seabed for a long time, their reliability and stability need to be fully tested and verified during the development stage. Semi-physical simulation can help RAUV systems carry out tasks such as full-process mission demonstration and verification, fault simulation, algorithm parameter debugging, and closed-loop dynamic acceptance, reduce test costs and risks, and improve RAUV R&D efficiency. However, the current semi-physical simulation lacks docking guidance image simulation technology, which makes it impossible for the semi-physical simulation system to complete the RAUV full-process operation simulation test including autonomous docking. Autonomous docking is the basis for the RAUV system to achieve long-term resident detection. In order to simulate the color guidance image output by the camera to test the autonomous docking algorithm, it is necessary to study the autonomous docking semi-physical simulation image generation method. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a method for generating semi-physical simulation guidance images for AUV autonomous docking, which utilizes the AUV posture information obtained by solving the AUV dynamic model during the semi-physical simulation process and the preset posture information of the docking device to generate a color docking guidance image corresponding to the relative posture relationship between the two, and provides simulated perception data for semi-physical simulation of autonomous docking and full-process operations, so as to facilitate the demonstration and verification of RAUV's operation mission and the debugging of algorithm parameters.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for generating a semi-physical simulation guidance image for autonomous docking of an AUV executes the following steps to simulate a docking guidance image perceived by a camera, which is used for semi-physical simulation of the full-process operation of autonomous docking and long-term residence of an AUV system; the method includes:

[0006] Step 1) Based on the camera imaging principle, the pixel position of the guidance light of the known world coordinates in the guidance image is calculated according to the relative position of the AUV to the docking device, and the guidance light image is created;

[0007] Step 2) constructing a guide light imaging area estimation network, estimating the overexposed dot area considering the overexposure of the color guide image, and generating a binary guide image;

[0008] Step 3) Pix2pix is ​​used to construct a color guidance image generation network, and real color images obtained from the sea trial are used for iterative learning and training to simulate the color distribution of the guidance image and obtain an ideal network model for color guidance image generation;

[0009] Step 4) Taking the binary guidance image as input, generate a color guidance image using the ideal network model.

[0010] The method is used to generate a guidance image containing a plurality of guidance lights.

[0011] The guide light imaging position calculation method comprises: constructing a guide light imaging model based on the camera imaging principle, calculating the guide light imaging pixel position and constructing a guide light image in a semi-physical simulation according to the AUV relative docking device posture calculated by dynamics and the preset position of the guide light on the docking device.

[0012] In the binary guidance image construction method, the guidance light in the color guidance image usually appears as an overexposed dot, and the area of ​​the overexposed dot is in a nonlinear inverse proportional relationship with the distance between the guidance lights; in order to estimate the area of ​​the overexposed dot, a single-input single-output overexposed dot estimation network is constructed to fit the mapping relationship between the area of ​​the overexposed dot and the distance between the guidance lights.

[0013] The overexposed dot estimation network includes three hidden layers, each of which is composed of different numbers of neurons. The activation function of the overexposed pixel estimation network is the ReLU function. The network input is the distance parameter x between the guide lights in the guide image, and the network output is the pixel area y′ of a single guide light dot:

[0014]

[0015] Where x is the distance parameter between the guidance lights in the guidance image, They are the weights between the input node, three neuron hidden layers, and the output node.

[0016] The distance parameter x between the guide lights in the guide image is: the average value of each side in the regular polygon image composed of a number of guide lights, or the average value of the distance from the center of the regular polygon image to each vertex.

[0017] The sea trial guidance image of the Exploration 11000 AUV is an image taken during the sea trial and includes a docking guidance light, which is installed on the docking device of the Exploration 11000 AUV.

[0018] The Pix2pix network is used to simulate the color distribution of the guidance image, the input of the model is the constructed binary guidance image, and the output is the color guidance image.

[0019] The training data set is constructed using color guidance images of the Exploration 11000AUV deep-sea autonomous docking test. The data set includes image pairs consisting of color guidance images and their corresponding binary guidance images. The binary guidance images are obtained using an adaptive threshold method. The Pix2pix network is trained using the data set so that its generated network has the ability to simulate the color distribution of guidance images.

[0020] AUV autonomous docking semi-physical simulation guidance image generation system, including a computing backend and a visualization interface front end mounted on a computer;

[0021] The computing backend is provided with a guide light imaging position calculation module, a binary guide image construction module, a color guide image generation module, and a real-time color image generation module. Each module is called and calculated to obtain an ideal network model for color guide image generation, and finally a color guide image is generated using the ideal network model. Among them, the guide light imaging position calculation module: based on the camera imaging principle, calculates the pixel position of the guide light of known world coordinates in the guide image according to the relative position of the AUV to the docking device, and creates a guide light image; the binary guide image construction module: constructs a guide light imaging area estimation network, estimates the overexposed dot area considering the overexposure of the color guide image, and generates a binary guide image; the color guide image generation module: uses Pix2pix to construct a color guide image generation network, uses the real color image obtained in the sea trial for iterative learning and training, and is used to simulate the color distribution of the guide image to obtain an ideal network model for color guide image generation; the real-time color image generation module uses the binary guide image as input and generates a color guide image using the ideal network model;

[0022] The visualization interface front end is used for human-computer interaction, loading training images, inputting model network parameters and training parameters; it is also provided with an image display area for displaying the rendering calculation background processing process image and the final generated color guidance image.

[0023] The advantages and beneficial effects of the present invention are:

[0024] The present invention provides a method for generating a semi-physical simulation guidance image for AUV autonomous docking, which can generate an autonomous docking guidance image that can reflect the posture relationship of the AUV relative to the docking device. The image can be used to perform semi-physical simulation tests of the entire process of AUV autonomous docking and long-term residence operations, which helps the RAUV system to improve the efficiency of algorithm development, debugging and testing and reduce test costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the semi-physical simulation guidance image generation strategy for AUV autonomous docking;

[0026] Figure 2 This is a schematic diagram of camera imaging;

[0027] Figure 3 A schematic diagram of the constructed guide light image;

[0028] Figure 4 Sample color guidance images collected for the Exploration 11000AUV sea trial;

[0029] Figure 5 Schematic diagram of the network for estimating the area of ​​imaging for the guide light;

[0030] Figure 6 Schematic diagram for guiding image generation based on Pix2pix network;

[0031] Figure 7 Schematic diagram of the constructed Pix2pix network training data set;

[0032] Figure 8 Schematic diagram of the generated color guidance image. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0035] like Figure 1As shown, the present invention provides a method for generating a semi-physical simulation guidance image for AUV autonomous docking, which mainly includes four steps:

[0036] Step 1) Calculation of the imaging position of the guidance light: Based on the camera imaging principle, the pixel position of the guidance light of the known world coordinates in the guidance image is calculated according to the relative position of the AUV to the docking device, and the guidance light image is created;

[0037] Step 2) Binary guidance image construction: Considering that the guidance light imaging is not just a point, a guidance light imaging area estimation network is constructed to estimate the area of ​​the overexposed dot and generate a binary guidance image;

[0038] Step 3) Color guidance image generation: Pix2pix's generative network is used to simulate the color distribution of the guidance image, and the network training is performed using the images obtained from the sea trial of the Exploration 11000AUV to obtain an ideal network. The Exploration 11000AUV sea trial guidance image is an image taken during the sea trial that includes a docking guidance light, and the guidance light is installed on the docking device of the Exploration 11000AUV.

[0039] Step 4) Using the binary guidance image as input, the trained ideal network is used to generate a color guidance image.

[0040] The present invention is described below by taking a guidance image formed by four guidance lights as an example.

[0041] Specific implementation method 1: Combination Figure 2-3 The following describes the method for calculating the imaging position of the guide light.

[0042] The camera imaging principle is as follows Figure 2 As shown, assuming that the position of a point in the image plane is (u, v), and the position of the point in the docking device plane is (x, y, z), then the imaging model of the point in the guidance image is:

[0043]

[0044] Where R is the camera coordinate system O c -X c Y c Z c The rotation matrix relative to the docking device coordinate system o-xyz, t is O c -X c Y c Z c The translation matrix relative to o-xyz, F is the camera internal parameter, Z c The z-axis position of the guide light in the camera coordinate system, 1 matrix calculation supplementary position.

[0045] The coordinates of the four guide lights in the docking device coordinate system o-xyz are known. In the semi-physical simulation, the AUV dynamics calculation is used to give the relative position relationship of the camera to the docking device, that is, R and t, so that the pixel coordinates of the four guide lights can be calculated, and then the guide light image can be constructed, such as Figure 3 shown.

[0046] Specific implementation method 2: Combination Figure 4-5 Describe the binary guided image construction method.

[0047] like Figure 4 As shown in the figure, the guide light in the color guide image usually presents an overexposed dot. When the guide light spacing is smaller, the overexposed dot area is larger. The overexposed dot area and the guide light spacing are in a complex nonlinear relationship. In order to estimate the overexposed dot area, we construct the following Figure 5 The single-input single-output guide light imaging area estimation network shown fits the mapping relationship between the overexposed dot area and the guide light spacing. The network contains three hidden layers, which are composed of 10, 20 and 40 neurons respectively.

[0048] The activation function of the overexposed pixel estimation network is the ReLU function:

[0049] Re LU(x)=max(0,x)

[0050] The output of the network is the pixel area y′ of a single guide light:

[0051]

[0052] Where x is the distance parameter between the guidance lights in the guidance image, They are the weights between the input node, three hidden layers of neurons, and the output node, respectively. i and j are the numbers of neurons in each layer.

[0053] The distance parameter x between the guide lights in the guide image may be: the mean value of each edge in a regular polygon image composed of a number of guide lights, or the mean value of the distance from the center of the regular polygon image to each vertex.

[0054] Specific implementation method three: Combination Figure 6-8 A method for generating a color guidance image is described.

[0055] Goodfellow et al. proposed that the Generative Adversarial Network (GAN) is widely used in image generation such as medical image simulation, automatic animation generation and coloring, but the GAN network used for guided image generation cannot control the guided imaging position. Pix2pix proposed by Isola et al. is a classic conditional GAN ​​network that can generate color images based on input conditions.

[0056] like Figure 6As shown, the pix2pix network is used to simulate the color distribution of the guidance image. In practical applications, the input of the model is the binary guidance image obtained in the second specific implementation method, and the output is a color guidance image.

[0057] In order to reliably generate color guidance images, the generation model G needs to be trained. The color guidance images of the Exploration 11000AUV deep-sea autonomous docking test are used to construct a training dataset, such as Figure 7 As shown in the figure, the dataset includes image pairs consisting of color guide images and their corresponding binary guide images, where the binary guide images are obtained using an adaptive threshold method. The color guide images are used as the real label images, and the corresponding binary guide images are used as the input data of the model. The model is trained through iterative learning so that its output color images are close to the real label images.

[0058] During the iterative learning and training process, the optimizer uses the Adam optimizer, sets the training batch and initial network parameters, uses the loss function to calculate the error, and obtains an ideal model for error convergence, so that the simulated color map output by the network is infinitely close to the real label color map of the sea trial.

[0059] In order to achieve color simulation, the encoding and decoding of the generative model G adopt the U-Net architecture. The kernel dimensions of the convolutional layers of the encoder and decoder are both 4×4, and the generator has 15 layers. The discriminator is designed to evaluate the consistency between the generated color-guided image and the actual color-guided image. The discriminator has 5 layers. The objective function of the network is:

[0060]

[0061] L pix2pix (G,D)=L cGan (G,D)+λ·L L1 (G)

[0062]

[0063] L L1 (G) = E a,b,c [||aG(c|b)||1]

[0064] Among them, G * is the trained generator, G is the generator, D is the discriminator, a is the real guidance image obtained by the sea trial, and p data is the real image dataset, c is random noise, G(c|b) is the generated image, D(a|b) is the discrimination result of the real guided image under control conditions, p c is the generated image data set, b is the control condition (binary guidance image), λ is the coefficient, set to 100, the purpose of network training is to find G through optimization solution * After training is completed, G* Then, it is used as a color guide image generator, and the binary guide image obtained in the second embodiment is input into G * , generate a color guidance image such as Figure 8 shown.

[0065] The guiding image generating method can be used to generate a guiding image including two or more guiding lights.

[0066] The present invention also provides an AUV autonomous docking semi-physical simulation guidance image generation system, including a computing backend and a visualization interface front end mounted on a computer;

[0067] The computing backend is provided with a guide light imaging position computing module, a binary guide image building module, and a color guide image generating module. Each module is called upon to calculate and obtain an ideal network model for color guide image generation, and finally a color guide image is generated using the ideal network model.

[0068] Among them, the guidance light imaging position calculation module: based on the camera imaging principle, calculates the pixel position of the guidance light of the known world coordinates in the guidance image according to the relative position of the AUV to the docking device, and creates the guidance light image; the binary guidance image construction module: constructs the guidance light imaging area estimation network, considers the overexposure of the color guidance image to estimate the overexposed dot area, and generates the binary guidance image; the color guidance image generation module: uses Pix2pix to build the color guidance image generation network, uses the real color images obtained in the sea trial for iterative learning and training, and is used to simulate the color distribution of the guidance image to obtain the ideal network model for color guidance image generation; the real-time color image generation module takes the binary guidance image as input and generates the color guidance image using the ideal network model;

[0069] The visualization interface front end is used for human-computer interaction, loading training images, inputting model network parameters and training parameters; it is also provided with an image display area for displaying the rendering calculation background processing process image and the final generated color guidance image.

[0070] The above is a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. AUV autonomous docking semi-physical simulation guidance image generation method, characterized in that: The following steps are performed to simulate the docking guidance image perceived by the camera, which is used for semi-physical simulation of the full-process operation of the AUV system autonomous docking and long-term residence; the method includes: Step 1) Based on the camera imaging principle, the pixel position of the guidance light of the known world coordinates in the guidance image is calculated according to the relative position of the AUV to the docking device, and the guidance light image is created; Step 2) constructing a guide light imaging area estimation network, estimating the overexposed dot area considering the overexposure of the color guide image, and generating a binary guide image; Step 3) Pix2pix is ​​used to construct a color guidance image generation network, and real color images obtained from the sea trial are used for iterative learning and training to simulate the color distribution of the guidance image and obtain an ideal network model for color guidance image generation; Step 4) Taking the binary guidance image as input, generate a color guidance image using the ideal network model.

2. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 1 is characterized in that: The method is used to generate a guidance image including a plurality of guidance lights.

3. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 1 is characterized in that: The guide light imaging position calculation method comprises: constructing a guide light imaging model based on the camera imaging principle, calculating the guide light imaging pixel position and constructing a guide light image in a semi-physical simulation according to the AUV relative docking device posture calculated by dynamics and the preset position of the guide light on the docking device.

4. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 1 is characterized in that: In the binary guidance image construction method, the guidance light in the color guidance image usually appears as an overexposed dot, and the area of ​​the overexposed dot is in a nonlinear inverse proportional relationship with the distance between the guidance lights; in order to estimate the area of ​​the overexposed dot, a single-input single-output overexposed dot estimation network is constructed to fit the mapping relationship between the area of ​​the overexposed dot and the distance between the guidance lights.

5. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 4 is characterized in that: The overexposed dot estimation network includes three hidden layers, each of which is composed of different numbers of neurons. The activation function of the overexposed pixel estimation network is the ReLU function. The network input is the distance parameter x between the guide lights in the guide image, and the network output is the pixel area y′ of a single guide light dot: Where x is the distance parameter between the guidance lights in the guidance image, They are the weights between the input node, three neuron hidden layers, and the output node.

6. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 5 is characterized in that: The distance parameter x between the guide lights in the guide image is: the average value of each side in the regular polygon image composed of a number of guide lights, or the average value of the distance from the center of the regular polygon image to each vertex.

7. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 1 is characterized in that: The sea trial guidance image of the Exploration 11000 AUV is an image taken during the sea trial and includes a docking guidance light, which is installed on the docking device of the Exploration 11000 AUV.

8. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 1 is characterized in that: The Pix2pix network is used to simulate the color distribution of the guidance image, the input of the model is the constructed binary guidance image, and the output is the color guidance image.

9. The AUV autonomous docking semi-physical simulation guidance image generation method according to claim 1, characterized in that: The training data set is constructed using color guidance images of the Exploration 11000AUV deep-sea autonomous docking test. The data set includes image pairs consisting of color guidance images and their corresponding binary guidance images. The binary guidance images are obtained using an adaptive threshold method. The Pix2pix network is trained using the data set so that its generated network has the ability to simulate the color distribution of guidance images.

10. AUV autonomous docking semi-physical simulation guidance image generation system, characterized by: Including the computing background and visual interface front end installed on the computer; The computing backend is provided with a guide light imaging position calculation module, a binary guide image construction module, a color guide image generation module, and a real-time color image generation module. Each module is called and calculated to obtain an ideal network model for color guide image generation, and finally a color guide image is generated using the ideal network model. Among them, the guide light imaging position calculation module: based on the camera imaging principle, calculates the pixel position of the guide light of known world coordinates in the guide image according to the relative position of the AUV to the docking device, and creates a guide light image; the binary guide image construction module: constructs a guide light imaging area estimation network, estimates the overexposed dot area considering the overexposure of the color guide image, and generates a binary guide image; the color guide image generation module: uses Pix2pix to construct a color guide image generation network, uses the real color image obtained in the sea trial for iterative learning and training, and is used to simulate the color distribution of the guide image to obtain an ideal network model for color guide image generation; the real-time color image generation module uses the binary guide image as input and generates a color guide image using the ideal network model; The visualization interface front end is used for human-computer interaction, loading training images, inputting model network parameters and training parameters; it is also provided with an image display area for displaying the rendering calculation background processing process image and the final generated color guidance image.

Images