Visual motion model construction method, underwater scene construction method based on visual motion model and underwater robot

Abstract

The invention discloses a visual motion model construction method, an underwater scene construction method based on a visual motion model and an underwater robot, and belongs to the field of machine vision, and the method comprises the steps: predicting a parallax image according to a target binocular image and a current visual motion model; performing bilinear sampling on a right eye image in the target binocular image according to the parallax image to obtain a first synthetic left eye image; according to the target binocular image, the binocular image of the target binocular image at the previous moment, the parallax image, the underwater camera internal reference and the current visual motion model, predicting to obtain a pose transformation value, and calculating a second synthetic left eye image according to the pose transformation value; calculating a loss function value according to a left eye image, the first synthetic left eye image and the second synthetic left eye image in the target binocular image, and adjusting network parameters of the model according to the loss function value; and iterative training is repeatedly executed until a final visual motion model is obtained. By implementing the application, the problem of low sensing precision in an underwater complex environment in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and in particular to a method for constructing a visual motion model, a method for constructing an underwater scene based on a visual motion model, and an underwater robot. Background Technology

[0002] In the field of underwater robot environmental perception, constructing an accurate underwater scene is crucial for applications such as navigation and positioning.

[0003] In existing technologies, stereo matching methods mainly rely on manually designed features (such as edges, textures, and corners) and similarity metrics between images (such as grayscale matching and phase matching). They calculate disparity maps by searching for matching relationships between corresponding pixels in the left and right eye images, thereby deriving the depth information of the scene. The core of these methods lies in establishing pixel correspondences through manually preset feature templates and matching rules. Their performance is highly dependent on the rationality of feature design and the adaptability of similarity metrics. In complex underwater environments (such as uneven lighting, water scattering, and target occlusion), the robustness of manually designed features often leads to decreased matching accuracy. Summary of the Invention

[0004] This invention provides a method for constructing a visual motion model, a method for constructing an underwater scene based on a visual motion model, and an underwater robot. The method can solve the problem of low perception accuracy in complex underwater environments in existing technologies.

[0005] An embodiment of the present invention provides a method for constructing a visual motion model, comprising:

[0006] Acquire an underwater stereo image dataset and underwater camera intrinsic parameters; wherein, the underwater stereo image dataset includes stereo images at several consecutive time points and the timestamps corresponding to each stereo image;

[0007] Based on the underwater binocular image dataset and the underwater camera intrinsic parameters, iterative training is repeatedly performed on the preset initial visual motion model until the loss function value of the initial visual motion model converges, and the iterative training is terminated to obtain the trained visual motion model.

[0008] The iterative training includes:

[0009] Each binocular image in the underwater binocular image dataset is sequentially used as the target binocular image;

[0010] Based on the target binocular image and the current visual motion model, a disparity image is predicted; wherein, during the first iteration of training, the current visual motion model is the initial visual motion model.

[0011] Based on the disparity image, the right eye image in the target binocular image is bilinearly sampled to obtain the first synthesized left eye image;

[0012] Based on the target binocular image, the binocular image at the moment before the target binocular image, the parallax image, the underwater camera intrinsic parameters, and the current visual motion model, the pose transformation value is predicted, and the second synthetic left eye image is calculated based on the pose transformation value.

[0013] Based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, the loss function value is calculated, and the network parameters of the current visual motion model are adjusted according to the loss function value to obtain the visual motion model for the next iteration of training.

[0014] Furthermore, the initial visual motion model includes an initial stereo matching model;

[0015] The step of predicting the disparity image based on the target binocular image and the current visual motion model includes:

[0016] The target binocular image is input into the current stereo matching model to obtain the disparity image output by the current stereo matching model; wherein, during the first iteration of training, the current stereo matching model is the initial stereo matching model.

[0017] Furthermore, the initial visual motion model also includes an initial pose model;

[0018] The step of predicting pose transformation values ​​based on the target binocular image, the binocular image at the moment preceding the target binocular image, the disparity image, the underwater camera intrinsic parameters, and the current visual motion model, and calculating the second synthetic left-eye image based on the pose transformation values, includes:

[0019] The target binocular image, the disparity image, and the binocular image at the moment preceding the target binocular image are input into the current pose model to obtain the pose transformation value output by the current pose model; wherein, during the first iteration of training, the current pose model is the initial pose model;

[0020] Based on the parallax image, the pose transformation value, the target binocular image, and the underwater camera intrinsic parameters, the pixel coordinate correspondence between the target binocular image and the binocular image at the previous moment of the target binocular image is calculated.

[0021] Based on the pixel coordinate correspondence, the binocular image at the moment preceding the target binocular image is bilinearly sampled to calculate the second synthesized left eye image.

[0022] Further, the step of calculating the pixel coordinate correspondence between the target binocular image and the binocular image at the moment preceding the target binocular image based on the disparity image, the pose transformation value, the target binocular image, and the underwater camera intrinsic parameters includes:

[0023] Based on the parallax image, the target binocular image, and the underwater camera intrinsic parameters, a back-projection transformation is performed on the left eye image in the target binocular image to obtain a first three-dimensional point cloud.

[0024] Based on the pose transformation value, the first three-dimensional point cloud is transformed to obtain the second three-dimensional point cloud;

[0025] Based on the second three-dimensional point cloud and the underwater camera intrinsic parameters, a projection transformation is performed to calculate the pixel coordinate correspondence between the target binocular image and the binocular image at the moment preceding the target binocular image.

[0026] Further, the step of calculating a loss function value based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, and adjusting the network parameters of the current visual motion model based on the loss function value to obtain the visual motion model for the next iteration of training includes:

[0027] Based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, the loss function value is calculated, and the network parameters of the current stereo matching model and the current pose model are adjusted according to the loss function value to obtain the stereo matching model and pose model for the next iteration of training.

[0028] Furthermore, before acquiring the underwater stereo image dataset, the following steps are also included:

[0029] Obtain the raw underwater binocular image dataset;

[0030] The original stereo images in the original underwater stereo image dataset are preprocessed to obtain preprocessed stereo images.

[0031] The underwater binocular image dataset is generated based on the preprocessed binocular images.

[0032] Further, the preprocessing of the original stereo images in the original underwater stereo image dataset to obtain preprocessed stereo images includes:

[0033] The original binocular image is subjected to binocular correction to obtain the corrected binocular image;

[0034] The corrected binocular images are subjected to joint normalization processing to obtain jointly normalized binocular images;

[0035] The stereo image after joint normalization is used as the preprocessed stereo image.

[0036] An embodiment of the present invention also provides a method for constructing underwater scenes based on a visual motion model, comprising:

[0037] Acquire real-time stereo images, and obtain the camera intrinsic and extrinsic parameters of the underwater stereo camera used to acquire real-time stereo images;

[0038] The real-time binocular images and the camera intrinsic parameters are input into the visual motion model to obtain the actual disparity image and actual pose transformation value output by the visual motion model; wherein, the visual motion model is determined by the visual motion model construction method.

[0039] An underwater scene is constructed based on the actual parallax image, the actual pose transformation value, the camera extrinsic parameters, and the camera intrinsic parameters.

[0040] Furthermore, the visual motion model includes a stereo matching model and a pose model;

[0041] The step of inputting the real-time binocular image and the camera intrinsic parameters into the visual motion model to obtain the actual disparity image and actual pose transformation value output by the visual motion model includes:

[0042] The real-time binocular image is input into the stereo matching model to obtain the actual disparity image output by the stereo matching model;

[0043] The real-time binocular image and the camera intrinsic parameters are input into the pose model to obtain the actual pose transformation value output by the pose model.

[0044] This application also provides an underwater robot, comprising:

[0045] One or more processors;

[0046] A memory, coupled to the processor, for storing one or more programs;

[0047] When the one or more programs are executed by the one or more processors, the one or more processors implement the underwater scene construction method based on the visual motion model as described in the above embodiments of the invention.

[0048] The following benefits can be obtained by implementing the present invention:

[0049] This invention provides a method for constructing a visual motion model, a method for constructing underwater scenes based on the visual motion model, and an underwater robot. The visual motion model construction method utilizes a continuous underwater binocular image dataset and underwater camera intrinsic parameters to iteratively train an initial visual motion model. During training, a first synthetic left-eye image is generated by combining disparity prediction and a second synthetic left-eye image is generated based on pose transformation values. The model parameters are optimized by using a loss function with the real left-eye image. Thus, the combination of disparity prediction and bilinear sampling improves the robustness of stereo matching, the introduction of pose transformation enhances the accuracy of motion construction across time steps, and the continuously optimized network parameters during iterative training enable the model to adapt to changes in the features of underwater images. Ultimately, this solves the problem of low perception accuracy in complex underwater environments in existing technologies, effectively improving the accuracy of disparity estimation, pose prediction, and overall perception capability of underwater scenes. Attached Figure Description

[0050] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating a visual motion model construction method provided in a certain embodiment of this application;

[0052] Figure 2 This is a flowchart illustrating an underwater scene construction method based on a visual motion model provided in a certain embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of an underwater robot provided in one embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0056] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0059] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0060] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0061] Example 1

[0062] See Figure 1To address the problem of low perception accuracy in complex underwater environments using existing technologies, an embodiment of the present invention provides a method for constructing a visual motion model, comprising:

[0063] S101. Obtain the underwater stereo image dataset and the underwater camera intrinsic parameters; wherein, the underwater stereo image dataset includes stereo images at several consecutive times and the timestamps corresponding to each stereo image;

[0064] In a preferred embodiment, prior to acquiring the underwater stereo image dataset, the method further includes:

[0065] Obtain the raw underwater binocular image dataset;

[0066] The original stereo images in the original underwater stereo image dataset are preprocessed to obtain preprocessed stereo images.

[0067] The underwater binocular image dataset is generated based on the preprocessed binocular images.

[0068] Specifically, the original underwater stereo image dataset includes several original stereo images acquired at consecutive times. In order to eliminate underwater environmental interference (such as image distortion), it is necessary to preprocess the original stereo images in the original underwater stereo image dataset to obtain preprocessed stereo images, so that the generated underwater stereo image dataset meets the training requirements of the visual motion model.

[0069] In a preferred embodiment, the step of preprocessing the original stereo images in the original underwater stereo image dataset to obtain preprocessed stereo images includes:

[0070] The original binocular image is subjected to binocular correction to obtain the corrected binocular image;

[0071] The corrected binocular images are subjected to joint normalization processing to obtain jointly normalized binocular images;

[0072] The binocular image after joint normalization is used as the preprocessed binocular image;

[0073] Specifically, the original binocular image is subjected to binocular correction, including distortion removal and epipolar alignment, so that the corrected left and right binocular images are aligned in the horizontal direction, and then the corrected binocular image is subjected to joint normalization processing.

[0074] Specifically, the corrected binocular image (I l ,I r Including the corrected left eye image I l And the corrected right eye image I rTo ensure that the two images are compatible with the subsequent stereo matching requirements in terms of statistical characteristics such as brightness and contrast, the corrected left eye image I... l With right eye image I r Joint standardization is performed (channel-by-channel operation: first calculate the mean of pixels in both images, then calculate the standard deviation based on the mean, and finally normalize). The specific processing formula is shown below:

[0075]

[0076] In the formula, This represents the right eye image after joint normalization processing; Represents the left eye image after joint normalization processing; I l (h,w,c) represents the corrected left eye image; I r (h,w,c) Corrected right eye image; μ c σ represents the joint mean of the corrected left and right eye images in channel c (the average brightness of all pixels in that channel in both images); c This represents the joint standard deviation of the corrected left and right eye images in channel c (reflecting the dispersion of pixel brightness in that channel between the two images); This represents the joint standard deviation σ of channel c. c and minimum value (To avoid the standard deviation being meaningless when it is 0) larger values ​​are used to normalize the denominator; h represents the row dimension index of the stereo image; w represents the horizontal dimension index of the stereo image; H represents the height of the stereo image (number of pixel rows); W represents the width of the stereo image (number of pixel columns).

[0077] S102. Based on the underwater binocular image dataset and the underwater camera intrinsic parameters, repeatedly perform iterative training on the preset initial visual motion model until the loss function value of the initial visual motion model converges, then terminate the iterative training to obtain the trained visual motion model.

[0078] The iterative training includes:

[0079] Each binocular image in the underwater binocular image dataset is sequentially used as the target binocular image;

[0080] Based on the target binocular image and the current visual motion model, a disparity image is predicted; wherein, during the first iteration of training, the current visual motion model is the initial visual motion model.

[0081] Based on the disparity image, the right eye image in the target binocular image is bilinearly sampled to obtain the first synthesized left eye image;

[0082] Based on the target binocular image, the binocular image at the moment before the target binocular image, the parallax image, the underwater camera intrinsic parameters, and the current visual motion model, the pose transformation value is predicted, and the second synthetic left eye image is calculated based on the pose transformation value.

[0083] Based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, the loss function value is calculated, and the network parameters of the current visual motion model are adjusted according to the loss function value to obtain the visual motion model for the next iteration of training.

[0084] Specifically, after obtaining the underwater binocular image dataset in step S1, in order to solve the problems of insufficient robustness of manually designed features and difficulty in adapting to complex underwater environments in the prior art, this embodiment constructs a visual motion model, takes the underwater binocular image dataset and underwater camera intrinsic parameters as input, integrates stereo matching and pose estimation tasks, uses self-supervised loss to iteratively optimize network parameters, trains the visual motion model, and thus accurately predicts the disparity map and pose transformation of the underwater scene based on the trained visual motion model;

[0085] In a preferred embodiment, the initial visual motion model includes an initial stereo matching model;

[0086] The step of predicting the disparity image based on the target binocular image and the current visual motion model includes:

[0087] The target binocular image is input into the current stereo matching model to obtain the disparity image output by the current stereo matching model; wherein, during the first iteration of training, the current stereo matching model is the initial stereo matching model;

[0088] Specifically, firstly, an initial deep neural network based on a convolutional neural network architecture is constructed and set as the network structure for the stereo matching task, serving as the initial stereo matching model;

[0089] It should be noted that this deep neural network is not limited to a specific network framework (such as an improved ResNet, Hourglass, or other common network architectures used for feature extraction and disparity prediction). It only needs to have the basic function of extracting features from binocular image pairs and predicting disparity, and be able to adapt to the feature extraction and disparity inference requirements of underwater binocular images.

[0090] Specifically, the target binocular image is input into the current stereo matching model to obtain the disparity image d output by the current stereo matching model;

[0091] Schematic, based on the disparity image d, the right eye image in the target binocular image is bilinearly sampled to obtain the first synthesized left eye image I. geo ;

[0092] It should be noted that since bilinear sampling is a mature technology in the fields of computer vision and image processing, the detailed principles and calculation process of the bilinear sampling algorithm will not be elaborated here.

[0093] In a preferred embodiment, the initial visual motion model further includes an initial pose model;

[0094] The step of predicting pose transformation values ​​based on the target binocular image, the binocular image at the moment preceding the target binocular image, the disparity image, the underwater camera intrinsic parameters, and the current visual motion model, and calculating the second synthetic left-eye image based on the pose transformation values, includes:

[0095] The target binocular image, the disparity image, and the binocular image at the moment preceding the target binocular image are input into the current pose model to obtain the pose transformation value output by the current pose model; wherein, during the first iteration of training, the current pose model is the initial pose model;

[0096] Based on the parallax image, the pose transformation value, the target binocular image, and the underwater camera intrinsic parameters, the pixel coordinate correspondence between the target binocular image and the binocular image at the previous moment of the target binocular image is calculated.

[0097] Based on the pixel coordinate correspondence, the binocular image at the moment before the target binocular image is bilinearly sampled to calculate the second synthesized left eye image;

[0098] Similarly, an initial deep neural network based on a convolutional neural network architecture is constructed and set as the network structure for the pose estimation task, as the initial pose model;

[0099] It should be noted that this deep neural network is not limited to a specific network framework (such as an improved ResNet, Hourglass, or other common network architectures used for feature extraction and pose prediction). It only needs to have the basic function of extracting features from binocular image pairs and predicting poses, and be able to adapt to the feature extraction and pose inference needs of underwater binocular images.

[0100] Specifically, the target binocular image, the disparity image, and the binocular image at the moment before the target binocular image are input into the current pose model to obtain the pose transformation value [R,t] output by the current pose model;

[0101] In a preferred embodiment, calculating the pixel coordinate correspondence between the target binocular image and the binocular image at the moment preceding the target binocular image, based on the disparity image, the pose transformation value, the target binocular image, and the underwater camera intrinsic parameters, includes:

[0102] Based on the parallax image, the target binocular image, and the underwater camera intrinsic parameters, a back-projection transformation is performed on the left eye image in the target binocular image to obtain a first three-dimensional point cloud.

[0103] Based on the pose transformation value, the first three-dimensional point cloud is transformed to obtain the second three-dimensional point cloud;

[0104] Based on the second three-dimensional point cloud and the underwater camera intrinsic parameters, a projection transformation is performed to calculate the pixel coordinate correspondence between the target binocular image and the binocular image at the previous moment before the target binocular image.

[0105] Schematic illustration, with known parallax image at d(w,h) and underwater camera intrinsic parameters [f] x ,f y ,c x ,c y ], where f x f represents the horizontal focal length of the camera. y c represents the camera's vertical focal length. x c represents the horizontal coordinates of the principal point; y This represents the coordinates of the principal point in the vertical direction.

[0106] Specifically, assuming the timestamp of the target binocular image corresponds to time t; based on the disparity image, the target binocular image, and the underwater camera intrinsic parameters, a back-projection transformation is performed on the left-eye image in the target binocular image to obtain the first three-dimensional point cloud p in the underwater camera coordinate system at time t. T The specific calculation formula is as follows:

[0107]

[0108] In the formula, p T Represents the first three-dimensional point cloud, X T Y represents the horizontal coordinate of the first 3D point cloud in the underwater camera coordinate system. T Z represents the vertical coordinate of the first 3D point cloud in the underwater camera coordinate system. T d(w,h) represents the depth of the first 3D point cloud in the underwater camera coordinate system; d(w,h) represents the parallax image; w represents the column pixel index of the left eye image in the target binocular image; h represents the row pixel index of the left eye image in the target binocular image.

[0109] Specifically, based on the pose transformation value [R,t], the first three-dimensional point cloud p T Perform pose transformation to obtain the second 3D point cloud p in the underwater camera coordinate system at time t-1. T1 The specific calculation formula is as follows:

[0110] P T1 =R·P T +t;

[0111] Specifically, based on the second three-dimensional point cloud p T1 and the underwater camera intrinsic parameters [f] x ,f y ,c x ,c y The projection transformation is performed to calculate the pixel coordinate correspondence (w,h)→(w',h') between the target binocular image and the binocular image at the previous moment before the target binocular image. The specific calculation formula is as follows:

[0112]

[0113] In the formula, X T1 Y represents the horizontal coordinate of the second 3D point cloud in the underwater camera coordinate system. T1 Z represents the vertical coordinates of the second 3D point cloud in the underwater camera coordinate system; T1 This represents the depth of the second 3D point cloud in the underwater camera coordinate system. This represents the unit depth plane that normalizes the 3D point coordinates to the camera coordinate system; w' represents the column pixel index of the left eye image in the stereo image at time t-1; h' represents the row pixel index of the left eye image in the stereo image at time t-1.

[0114] Specifically, based on the pixel coordinate correspondence, bilinear sampling is performed on the binocular image at the moment preceding the target binocular image to calculate the second synthesized left-eye image I. move ;

[0115] It should be noted that since bilinear sampling is a mature technology in the fields of computer vision and image processing, the detailed principles and calculation process of the bilinear sampling algorithm will not be elaborated here.

[0116] In a preferred embodiment, the step of calculating a loss function value based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, and adjusting the network parameters of the current visual motion model based on the loss function value to obtain the visual motion model for the next iteration of training includes:

[0117] Based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, calculate the loss function value, and adjust the network parameters of the current stereo matching model and the current pose model according to the loss function value to obtain the stereo matching model and pose model for the next iteration of training.

[0118] Specifically, the loss function value L self The specific calculation formula is as follows:

[0119] L self =abs(I l -I move )+abs(I l -I geo );

[0120] In the formula, abs represents the absolute value;

[0121] Specifically, the loss function value measures the difference between the synthetic image and the real image. The loss function value is backpropagated to adjust the network parameters of the current stereo matching model and the current pose model respectively, so as to achieve self-supervised training of the stereo matching model and the pose model.

[0122] Example 2

[0123] See Figure 2 This invention provides a method for constructing underwater scenes based on a visual motion model, comprising:

[0124] S201. Acquire real-time stereo images, and obtain the camera intrinsic and extrinsic parameters of the underwater stereo camera used to acquire real-time stereo images;

[0125] Specifically, real-time binocular images are continuously acquired at a preset frequency at several consecutive time points, and the camera intrinsic parameters of the underwater binocular camera are obtained.

[0126] S202. Input the real-time binocular image and the camera intrinsic parameters into the visual motion model to obtain the actual disparity image and actual pose transformation value output by the visual motion model; wherein, the visual motion model is determined by the visual motion model construction method.

[0127] In a preferred embodiment, the visual motion model includes a stereo matching model and a pose model;

[0128] The step of inputting the real-time binocular image and the camera intrinsic parameters into the visual motion model to obtain the actual disparity image and actual pose transformation value output by the visual motion model includes:

[0129] The real-time binocular image is input into the stereo matching model to obtain the actual disparity image output by the stereo matching model;

[0130] The real-time binocular image and the camera intrinsic parameters are input into the pose model to obtain the actual pose transformation value output by the pose model.

[0131] Specifically, the real-time binocular images and the camera intrinsic parameters are input into the visual motion model to obtain the actual disparity image and the actual pose transformation value;

[0132] S203. Based on the actual parallax image, the actual pose transformation value, the camera extrinsic parameters, and the camera intrinsic parameters, an underwater scene is constructed.

[0133] Specifically, triangulation calculation is performed based on the actual parallax image and the camera intrinsic parameters to generate a depth image. Then, a three-dimensional point cloud is generated based on the depth image, the camera extrinsic parameters, the actual pose transformation value, and the camera intrinsic parameters. Finally, an underwater scene is constructed based on the depth image and the three-dimensional point cloud.

[0134] It should be noted that triangulation is a key step in binocular vision to convert two-dimensional parallax into three-dimensional depth. Since triangulation is a mature technology in the fields of computer vision and image processing, the detailed triangulation process will not be elaborated here.

[0135] Example 3

[0136] See Figure 3 One embodiment of this application also provides an underwater robot, including:

[0137] One or more processors;

[0138] A memory, coupled to the processor, for storing one or more programs;

[0139] When the one or more programs are executed by the one or more processors, the one or more processors implement the underwater scene construction method based on the visual motion model as described above.

[0140] The processor controls the overall operation of the underwater robot to complete all or part of the steps of the aforementioned underwater scene construction method based on a vision motion model. The memory stores various types of data to support the operation of the underwater robot. This data may include, for example, instructions for any application or method used to operate on the underwater robot, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0141] In an exemplary embodiment, the underwater robot may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the underwater scene construction method based on the visual motion model as described in any of the foregoing embodiments, and achieve the same technical effect as the methods described above.

[0142] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for constructing a visual motion model, characterized in that, include: Acquire an underwater stereo image dataset and underwater camera intrinsic parameters; wherein, the underwater stereo image dataset includes stereo images at several consecutive time points and the timestamps corresponding to each stereo image; Based on the underwater binocular image dataset and the underwater camera intrinsic parameters, iterative training is repeatedly performed on the preset initial visual motion model until the loss function value of the initial visual motion model converges, and the iterative training is terminated to obtain the trained visual motion model. The iterative training includes: Each binocular image in the underwater binocular image dataset is sequentially used as the target binocular image; Based on the target binocular image and the current visual motion model, a disparity image is predicted; wherein, during the first iteration of training, the current visual motion model is the initial visual motion model. Based on the disparity image, the right eye image in the target binocular image is bilinearly sampled to obtain the first synthesized left eye image; Based on the target binocular image, the binocular image at the moment before the target binocular image, the parallax image, the underwater camera intrinsic parameters, and the current visual motion model, the pose transformation value is predicted, and the second synthetic left eye image is calculated based on the pose transformation value. Based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, the loss function value is calculated, and the network parameters of the current visual motion model are adjusted according to the loss function value to obtain the visual motion model for the next iteration of training.

2. The visual motion model construction method as described in claim 1, characterized in that, The initial visual motion model includes an initial stereo matching model; The step of predicting the disparity image based on the target binocular image and the current visual motion model includes: The target binocular image is input into the current stereo matching model to obtain the disparity image output by the current stereo matching model; wherein, during the first iteration of training, the current stereo matching model is the initial stereo matching model.

3. The visual motion model construction method as described in claim 2, characterized in that, The initial visual motion model also includes an initial pose model; The step of predicting pose transformation values ​​based on the target binocular image, the binocular image at the moment preceding the target binocular image, the disparity image, the underwater camera intrinsic parameters, and the current visual motion model, and calculating the second synthetic left-eye image based on the pose transformation values, includes: The target binocular image, the disparity image, and the binocular image at the moment preceding the target binocular image are input into the current pose model to obtain the pose transformation value output by the current pose model; wherein, during the first iteration of training, the current pose model is the initial pose model; Based on the parallax image, the pose transformation value, the target binocular image, and the underwater camera intrinsic parameters, the pixel coordinate correspondence between the target binocular image and the binocular image at the previous moment of the target binocular image is calculated. Based on the pixel coordinate correspondence, the binocular image at the moment preceding the target binocular image is bilinearly sampled to calculate the second synthesized left eye image.

4. The visual motion model construction method as described in claim 3, characterized in that, The step of calculating the pixel coordinate correspondence between the target binocular image and the binocular image at the previous moment of the target binocular image based on the disparity image, the pose transformation value, the target binocular image, and the underwater camera intrinsic parameters includes: Based on the parallax image, the target binocular image, and the underwater camera intrinsic parameters, a back-projection transformation is performed on the left eye image in the target binocular image to obtain a first three-dimensional point cloud. Based on the pose transformation value, the first three-dimensional point cloud is transformed to obtain the second three-dimensional point cloud; Based on the second three-dimensional point cloud and the underwater camera intrinsic parameters, a projection transformation is performed to calculate the pixel coordinate correspondence between the target binocular image and the binocular image at the moment preceding the target binocular image.

5. The visual motion model construction method as described in claim 4, characterized in that, The step of calculating a loss function value based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, and adjusting the network parameters of the current visual motion model based on the loss function value to obtain the visual motion model for the next iteration of training includes: Based on the left eye image, the first synthesized left eye image, and the second synthesized left eye image in the target binocular image, the loss function value is calculated, and the network parameters of the current stereo matching model and the current pose model are adjusted according to the loss function value to obtain the stereo matching model and pose model for the next iteration of training.

6. The visual motion model construction method as described in claim 1, characterized in that, Before acquiring the underwater stereo image dataset, the following steps are also included: Obtain the raw underwater binocular image dataset; The original stereo images in the original underwater stereo image dataset are preprocessed to obtain preprocessed stereo images. The underwater binocular image dataset is generated based on the preprocessed binocular images.

7. The visual motion model construction method as described in claim 6, characterized in that, The step of preprocessing the original stereo images in the original underwater stereo image dataset to obtain preprocessed stereo images includes: The original binocular image is subjected to binocular correction to obtain the corrected binocular image; The corrected binocular images are subjected to joint normalization processing to obtain jointly normalized binocular images; The stereo image after joint normalization is used as the preprocessed stereo image.

8. A method for constructing underwater scenes based on a visual motion model, characterized in that, include: Acquire real-time stereo images, and obtain the camera intrinsic and extrinsic parameters of the underwater stereo camera used to acquire real-time stereo images; The real-time binocular image and the camera intrinsic parameters are input into the visual motion model to obtain the actual disparity image and actual pose transformation value output by the visual motion model; wherein, the visual motion model is determined by the visual motion model construction method according to any one of claims 1-7; An underwater scene is constructed based on the actual parallax image, the actual pose transformation value, the camera extrinsic parameters, and the camera intrinsic parameters.

9. The underwater scene construction method based on a visual motion model as described in claim 8, characterized in that, The visual motion model includes a stereo matching model and a pose model; The step of inputting the real-time binocular image and the camera intrinsic parameters into the visual motion model to obtain the actual disparity image and actual pose transformation value output by the visual motion model includes: The real-time binocular image is input into the stereo matching model to obtain the actual disparity image output by the stereo matching model; The real-time binocular image and the camera intrinsic parameters are input into the pose model to obtain the actual pose transformation value output by the pose model.

10. An underwater robot, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the underwater scene construction method based on the visual motion model as described in any one of claims 8-9.

Images