Underwater robot environment sensing method and device based on feature propagator

Abstract

The invention provides an underwater robot environment sensing method and device based on a feature propagator. The method comprises the following steps: acquiring a binocular image acquired by an underwater robot and a pre-established deep neural network; the binocular images are corrected, the corrected binocular images are respectively input into a feature extractor of the network, feature maps under different scales are obtained, and a feature pyramid is formed; and performing cost body construction, cost body aggregation and parallax regression based on each feature pyramid by using a cost aggregator. And initializing a multi-stage convolution gating cycle unit in the iterative optimizer based on the original cost body and the aggregated cost body by a feature propagator. And optimizing parallax prediction based on a multi-level convolution gating circulation unit initialized by the feature propagator until a parallax prediction value reaches a convergence condition, and obtaining a final parallax map. Internal and external parameters of the binocular camera are obtained, a scene depth map and a three-dimensional point cloud of the underwater environment are calculated based on the internal and external parameters and the disparity map, and robust and efficient underwater environment perception is achieved.

Description

Technical Field

[0001] The present invention relates to the field of underwater visual perception technology, and in particular to an underwater robot environment perception method and device based on a feature propagator. Background Art

[0002] In the field of underwater environment perception, binocular vision systems have been widely studied and applied because they can provide rich depth information. The binocular vision system simulates the human binocular stereoscopic vision mechanism and calculates the depth information of objects through the images captured by two cameras, thereby achieving three-dimensional perception of the underwater environment. However, the particularity of the underwater environment, such as light attenuation and turbid water, poses challenges to the accuracy and reliability of the binocular vision system. Summary of the invention

[0003] The present invention provides an underwater robot environment perception method and device based on a feature propagator, which is used to solve the defect of poor accuracy of underwater environment perception in the prior art and realize robust and efficient underwater environment perception. The technical solution proposed by the present invention is as follows: In a first aspect, the present invention provides an underwater robot environment perception method based on a feature propagator, comprising: Acquire a binocular image collected by an underwater robot and a pre-established deep neural network; wherein the deep neural network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer; Correcting the binocular images, and inputting the corrected binocular images into the feature extractors respectively to obtain feature maps at different scales and form a feature pyramid; Using a cost aggregator to perform cost volume construction, cost volume aggregation and disparity regression based on each of the feature pyramids, to obtain an original cost volume, an aggregated cost volume and an initial disparity respectively; Initializing, by a feature propagator, a multi-level convolutional gated recurrent unit in an iterative optimizer based on the original cost body and the aggregated cost body; Optimize the disparity prediction based on the multi-level convolutional gated recurrent unit initialized by the feature propagator until the disparity prediction value reaches the convergence condition to obtain the final disparity map; The internal and external parameters of the binocular camera of the underwater robot are obtained, and the scene depth map and the three-dimensional point cloud of the underwater environment are calculated based on the internal and external parameters and the final disparity map.

[0004] Optionally, the cost aggregator includes a cost volume construction network, a cost aggregation network and a disparity regressor; The cost aggregator is used to perform cost volume construction, cost volume aggregation and disparity regression based on each of the feature pyramids to obtain an original cost volume, an aggregated cost volume and an initial disparity, respectively, including: The cost volume construction network constructs a cost volume based on the feature graph at the maximum scale in each of the feature pyramids to obtain an original cost volume; The cost aggregation network filters and concentrates the local cost information in the original cost volume to obtain an aggregated cost volume; The disparity regressor performs an initial disparity estimate based on the aggregated cost volume to obtain the initial disparity.

[0005] Optionally, the feature propagator includes a plurality of parallel residual modules, and the binocular image includes a left-eye image and a right-eye image; The step of initializing the multi-level convolution gated recurrent unit in the iterative optimizer based on the original cost body and the aggregated cost body by the feature propagator includes: For the top recurrent unit in the multi-level convolutional gated recurrent unit, the feature map of the left image at the maximum scale, the original cost volume and the aggregated cost volume are concatenated and input into the top residual module to extract the aggregated features, and the hidden layer of the top recurrent unit is initialized with the aggregated features; Furthermore, for the recurrent units in other layers except the top layer in the multi-level convolutional gated recurrent unit, the feature map of the left-eye image is input into the residual module of the corresponding number of layers at the corresponding scale to extract the image context features, and the hidden layer of the corresponding recurrent unit is initialized with the image context features.

[0006] Optionally, the binocular image includes a left-eye image; the disparity prediction is optimized based on the multi-level convolution gated recurrent unit initialized by the feature propagator and the initial disparity until the disparity prediction value reaches a convergence condition to obtain a final disparity map, including: Constructing a correlation volume based on a feature map of the left image at the maximum scale and a feature map of the right image at the maximum scale; According to the current disparity map as an index, the local information related to the current pixel is extracted from the correlation volume, and the current disparity map, the feature map of the left image at the maximum scale and the local information are input into the multi-level convolution gated recurrent unit initialized by the feature propagator; The multi-level convolutional gated recurrent unit generates the disparity update value of the current pixel based on the current disparity map, the feature map of the left image at the maximum scale, and local information; The current disparity of each current pixel is superimposed with the corresponding disparity update value to obtain a disparity prediction value; wherein, the current disparity in the first iteration is the initial disparity; The disparity prediction process is iterated until the disparity prediction value reaches a convergence state, and the final disparity map is obtained.

[0007] Optionally, the multi-stage convolution gated recurrent unit includes a reset gate and an update gate; The multi-level convolution gated recurrent unit generates a disparity update value of a current pixel based on the current disparity map, the feature map of the left-eye image at the maximum scale, and local information, including: The local related information is obtained by splicing the current disparity map, the feature map of the left image at the maximum scale, and the local information, and then input into the reset gate and the update gate. The reset gate and the update gate are updated based on the local related information and the stored historical hidden layer features, respectively, to obtain the reset gate feature and the update gate feature; Combining the reset gate features, local related information and historical hidden layer features to obtain candidate hidden layer features; Calculate updated hidden layer features by combining candidate hidden layer features, update gate features and historical hidden layer features; The disparity update value of the current pixel is calculated based on the updated hidden layer features.

[0008] Optionally, the calculating the scene depth map and the three-dimensional point cloud of the underwater environment based on the internal and external parameters and the final disparity map includes: Using the internal and external parameters, the disparity value of each pixel in the disparity map is converted into a corresponding depth value to generate a scene depth map; The scene depth map is back-projected in combination with internal and external parameters to obtain a three-dimensional point cloud of the underwater environment.

[0009] In a second aspect, the present invention further provides an underwater robot environment perception device based on a feature propagator, comprising the following modules: An image acquisition module, used to acquire binocular images collected by the underwater robot and a pre-established deep neural network; wherein the deep neural network includes a feature extractor, a cost aggregator, a feature propagator and an iterative optimizer; A feature extraction module, used to correct the binocular image, input the corrected binocular image into the feature extractor respectively, obtain feature maps at different scales and form a feature pyramid; A cost aggregation module, used for performing cost volume construction, cost volume aggregation and disparity regression based on each of the feature pyramids using a cost aggregator to obtain an original cost volume, an aggregated cost volume and an initial disparity respectively; An initialization module, configured to initialize a multi-level convolutional gated recurrent unit in an iterative optimizer based on the original cost body and the aggregated cost body by a feature propagator; A disparity prediction module, used to optimize the disparity prediction based on the multi-level convolution gated recurrent unit initialized by the feature propagator until the disparity prediction value reaches a convergence condition to obtain a final disparity map; The calculation module is used to obtain the internal and external parameters of the binocular camera of the underwater robot, and calculate the scene depth map and the three-dimensional point cloud of the underwater environment based on the internal and external parameters and the final disparity map.

[0010] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the underwater robot environment perception method based on the feature propagator as described in the first aspect above is implemented.

[0011] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the underwater robot environment perception method based on a feature propagator as described in the first aspect above.

[0012] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the underwater robot environment perception method based on a feature propagator as described in the first aspect above.

[0013] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: The underwater robot environment perception method and device based on feature propagator provided by the present invention first extracts high-dimensional features from binocular images through feature extractor, obtains aggregated cost volume and initial disparity estimation through cost aggregator, and then iteratively updates the initial disparity using iterative optimizer. The present invention combines two neural network structures of cost aggregator and iterative optimizer. The method uses feature propagator to bridge the compactly designed cost aggregator and iterative optimizer, and efficiently updates the initial disparity estimated by cost aggregator through iterative optimizer to perform disparity prediction. Based on the predicted disparity map, the scene depth map and three-dimensional point cloud of the underwater environment are obtained by triangulating and back-projecting the internal and external parameters of the binocular camera, thereby realizing robust and efficient underwater environment perception. Before iteratively updating the initial disparity, the present invention creatively introduces the multi-level convolution gated recurrent unit in the feature propagator initialization iterative optimizer, provides the hidden layer initial value for the iterative optimizer, and improves the efficiency of subsequent optimization. The present invention can be deployed on an underwater robot to efficiently calculate the disparity information in the binocular image, and then obtain the depth and point cloud information of the underwater environment.

[0014] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0015] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the underwater robot environment perception method based on feature propagator provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the deep neural network structure provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the structure of the feature propagator provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the underwater robot environment perception device based on the feature propagator provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 4 The present invention describes an underwater robot environment perception method and device based on a feature propagator.

[0024] Reference Figure 1 As shown, the method includes the following: Step S101: Acquire binocular images collected by an underwater robot and a pre-established deep neural network.

[0025] The underwater robot collects image pairs of the underwater environment through the binocular camera carried by the robot. These image pairs contain the same scene observed from different perspectives. The binocular images in the present invention include left-eye images and right-eye images.

[0026] Pre-train a deep neural network, refer to Figure 2 and Figure 3As shown in Figure 1, the network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer. This network is used to extract features from stereo images, construct cost volumes, and perform disparity estimation and optimization.

[0027] Step S102: correcting the binocular image, inputting the corrected binocular image into the feature extractor respectively, obtaining feature maps at different scales and forming a feature pyramid.

[0028] The acquired binocular images are binocularly rectified according to the pinhole camera model so that the camera optical centers of the left and right images are parallel after correction, which is helpful for the subsequent parallax calculation.

[0029] The rectified binocular images are input into the feature extractor respectively. The feature extractor extracts features in the image through structures such as convolutional neural networks (CNN). These features are organized into feature pyramids at different scales. The feature pyramid provides a multi-scale feature representation from coarse to fine, which helps to process objects and details of different scales. The feature extractor obtains feature maps of the left and right images at different scales, and composes feature pyramids with feature maps of the same image at different scales.

[0030] The head of the feature extractor in the present invention is composed of 6 convolutional layers with batch normalization and ReLu activation function. Among them, the convolution kernel step size of the 1st, 3rd and 5th layers is set to 2 to achieve downsampling of the feature map. The rest of the feature extractor is composed of multiple continuous pooling layers with a step size of 2, which is used to quickly extract multi-scale feature pyramids. After the corrected left and right eye images are respectively input into the feature extractor, a 3-fold downsampled feature image is obtained after multi-layer convolution of the feature extractor head. Subsequently, the remaining multi-layer pooling layers are used to further downsample the obtained feature image to obtain feature maps at different scales and form a feature pyramid.

[0031] Step S103: Use a cost aggregator to perform cost volume construction, cost volume aggregation, and disparity regression based on each of the feature pyramids to obtain an original cost volume, an aggregated cost volume, and an initial disparity, respectively.

[0032] The original cost volume is constructed based on the feature pyramid using a cost aggregator. The cost volume is a three-dimensional data structure where each element represents the cost of matching two feature points at different disparities. The original cost volume is aggregated to reduce noise and ambiguity to obtain an aggregated cost volume. The initial disparity map is estimated from the aggregated cost volume by disparity regression.

[0033] Step S104: The feature propagator initializes the multi-level convolutional gated recurrent unit in the iterative optimizer based on the original cost volume and the aggregated cost volume.

[0034] Reference Figure 2 As shown, the feature propagator uses the original cost volume (i.e. Figure 2 The cost body in the CNN is used to initialize the multi-level convolutional gated recurrent unit (GRU) in the iterative optimizer. GRU is a variant of a recurrent neural network (RNN) that can capture long-term dependencies in sequence data and is used in the present invention to iteratively optimize disparity prediction.

[0035] Step S105 , optimizing the disparity prediction based on the multi-level convolutional gated recurrent unit initialized by the feature propagator until the disparity prediction value reaches a convergence condition, thereby obtaining a final disparity map.

[0036] Through the iterative optimization of multi-level GRU, the disparity prediction is gradually refined until the disparity prediction value tends to be stable and reaches the convergence condition. The convergence condition can be set to the change in the disparity prediction value is less than a certain threshold. When the iterative optimization is completed, the reference Figure 2 The final disparity shown is the final disparity map mentioned above. The final disparity map represents the disparity of each pixel between the binocular image pair, that is, the relative position difference under the perspectives of the two cameras.

[0037] Step S106: Acquire internal and external parameters of the binocular camera of the underwater robot, and calculate a scene depth map and a three-dimensional point cloud of the underwater environment based on the internal and external parameters and the final disparity map.

[0038] The intrinsic and extrinsic parameters of a stereo camera describe the position, orientation, and lens properties of the camera. By calibrating the stereo camera, we can obtain the intrinsic parameters such as the focal length and principal point coordinates of the camera, as well as the extrinsic parameters such as the relative position and attitude between the cameras. Intrinsic parameters are parameters that describe the internal properties of the camera, and extrinsic parameters are parameters that describe the position and attitude of the camera in the world coordinate system. Based on the intrinsic and extrinsic parameters and the final disparity map, the three-dimensional point cloud of the underwater environment is calculated by the principle of triangulation. The three-dimensional point cloud is a set of three-dimensional coordinates of objects in the underwater environment, which can be used for further tasks such as environmental perception, navigation, and obstacle avoidance.

[0039] The present invention extracts multi-scale features through a feature extractor and uses a cost aggregator to reduce noise and ambiguity, thereby improving the robustness and accuracy of disparity estimation. Moreover, the combined use of a feature propagator and an iterative optimizer can further refine disparity prediction and reduce errors. The generated three-dimensional point cloud provides accurate three-dimensional information of the underwater environment, which helps underwater robots to perform more accurate navigation, obstacle avoidance and terrain reconstruction. The three-dimensional point cloud can also be used to identify and classify underwater objects, improving the autonomous operation capability of underwater robots. The method is based on a deep neural network and has strong learning and generalization capabilities. Through training and optimization, it can adapt to different underwater environments and lighting conditions.

[0040] In an optional embodiment, the cost aggregator in the above step S103 includes three parts: a cost volume construction network, a cost aggregation network and a disparity regressor.

[0041] The cost aggregator described in step S103 is used to perform cost volume construction, cost volume aggregation and disparity regression based on each feature pyramid to obtain the original cost volume, the aggregated cost volume and the initial disparity, respectively. Figure 2 As shown, including: S1031. The cost volume construction network constructs a cost volume based on the feature graph at the largest scale in each of the feature pyramids to obtain an original cost volume.

[0042] The cost volume construction network constructs a cost volume based on the feature pyramid obtained above, using the feature map at the maximum scale (i.e., at the maximum resolution), which is referred to as the original cost volume below.

[0043] First, the feature maps at the largest scale are extracted from the feature pyramid. These feature maps usually contain the most significant and important information in the image and are the basis for constructing the cost volume.

[0044] Next, the cost volume construction network uses these feature maps to construct a three-dimensional cost volume through a series of convolution, pooling and other operations. Each element of this cost volume represents the matching cost between two feature points under different parallaxes.

[0045] Finally, the original cost volume is obtained, which is a three-dimensional data structure containing rich matching information.

[0046] S1032: The cost aggregation network filters and concentrates the local cost information in the original cost volume to obtain an aggregated cost volume.

[0047] Although the original cost body contains rich matching information, it may also contain noise and redundancy. The present invention uses the cost aggregation network to further filter and centralize the original cost body to reduce noise and redundancy and improve the accuracy and robustness of the cost body.

[0048] The above-mentioned cost aggregation network in the present invention is an hourglass-shaped three-dimensional convolutional neural network, and the feature bodies at the same scale in the network contain cross-layer connections. The original cost body is input into the hourglass-shaped three-dimensional convolutional neural network, and the local cost information therein is filtered and concentrated through a series of operations such as convolution, pooling, weighted averaging, etc., to obtain the aggregated cost body (hereinafter referred to as the aggregated cost body).

[0049] The cost aggregation network can adopt a variety of strategies to optimize the cost body, such as using larger convolution kernels to capture more extensive contextual information, or using attention mechanisms to focus on important feature points.

[0050] S1033 . The disparity regressor performs initial disparity estimation based on the aggregated cost volume to obtain an initial disparity.

[0051] The aggregated cost volume is already a relatively accurate and robust cost volume that can be used for disparity estimation. The aggregated cost volume is then traversed and regressed through the disparity regressor to estimate the disparity value of each pixel. The initial disparity map is finally obtained by finding the minimum cost value corresponding to each pixel and using the corresponding disparity as the disparity estimate of the pixel.

[0052] The above disparity regressor includes a SoftMax operation and a mean operation, which is used to predict the disparity from the aggregated cost volume. Specifically, the disparity regressor can be used to perform a SoftMax operation on the aggregated cost volume on the depth channel and calculate the average to obtain the initial disparity of the pixel point, and the initial disparity of each pixel point is finally obtained by combining the initial disparity of each pixel point.

[0053] The present invention can filter and centrally process the original cost volume through the cost aggregation network, reduce the influence of noise and redundant information, and thus improve the robustness of the algorithm. Even in complex and noisy environments, the algorithm can maintain good performance. Disparity estimation based on the aggregated cost volume can significantly improve the accuracy of disparity estimation and reduce the influence of mismatching and noise. Using the feature pyramid and the cost aggregation network, the feature information in the image can be more effectively extracted and integrated, thereby obtaining a more accurate and robust cost volume. Moreover, the construction of the feature pyramid and the aggregation of the cost volume can be performed at different scales, thereby reducing the amount of calculation and improving the calculation efficiency.

[0054] In an optional embodiment, the feature propagator in the above step S104 includes multiple parallel residual modules. At the top layer, the residual module takes the spliced ​​original cost volume, the aggregated cost volume and the feature map at the maximum scale of the left-eye image as input to extract aggregated features; at other layers, the residual module only takes the feature map at the corresponding scale as input to extract features.

[0055] The step S104 described above, wherein the feature propagator initializes the multi-level convolution gated recurrent unit in the iterative optimizer based on the original cost volume and the aggregated cost volume, comprises: S1041, reference Figure 3 As shown, for the top recurrent unit in the multi-level convolutional gated recurrent unit, the feature map of the left image at the maximum scale, the original cost body and the aggregated cost body are concatenated and input into the top residual module to extract the aggregated features, and the hidden layer of the top recurrent unit is initialized with the aggregated features.

[0056] First, a feature map at the largest scale is extracted from the left image, and this feature map is concatenated with the original cost volume and the aggregated cost volume. The concatenation operation is performed along a specific dimension (such as the channel dimension) to fuse information from different sources. The concatenated features are input into the residual module at the top level. The residual module extracts and enhances features through convolution operations and skip connections to obtain aggregated features. Finally, the aggregated features are used to initialize the hidden layer of the top recurrent unit in the multi-level convolutional gated recurrent unit. In this way, the top recurrent unit has comprehensive information from the left image, the original cost volume, and the aggregated cost volume.

[0057] S1042, reference Figure 3 As shown, for the recurrent units in other layers except the top layer in the multi-level convolutional gated recurrent unit, the feature map of the left-eye image is input into the residual module of the corresponding number of layers at the corresponding scale to extract the image context features, and the hidden layer of the corresponding recurrent unit is initialized with the image context features.

[0058] For the recurrent units in the multi-level convolutional gated recurrent unit except the top layer, first extract the feature map from the left-eye image at the corresponding scale. Then, input the feature map into the residual module of the corresponding layer to extract the image context features. For example, for the second-layer recurrent unit, input the feature map of the second scale in the left-eye image into the residual module of the second layer to extract the image context features. Use the extracted image context features to initialize the hidden layer of the corresponding recurrent unit. In this way, each recurrent unit has the feature information from the left-eye image at the corresponding scale.

[0059] By using multi-level convolutional gated recurrent units and residual modules, the feature propagator of the present invention can more effectively propagate feature information between different levels and scales. This helps to reduce information loss and redundancy and improve the efficiency of feature propagation. The convolution operation and jump connection in the residual module can enhance the expressiveness of features, making the extracted features richer and more accurate. This helps to improve the performance of subsequent disparity estimation, stereo matching, three-dimensional reconstruction, etc. The method is based on a deep neural network and has strong learning and generalization capabilities. Through training and optimization, it can adapt to the needs of different scenes and tasks and achieve effective processing of various complex scenes.

[0060] In the multi-level convolution gated loop unit, the hidden layer is updated and optimized in an iterative manner. Each iteration uses the result of the previous iteration (i.e., the current disparity map) and the current input feature information to update the state of the hidden layer. During the iteration, a gating mechanism (such as a reset gate and an update gate) can be used to control the flow and update of information to achieve more effective feature propagation and optimization. In an optional embodiment, the iterative optimizer uses a multi-level convolution gated loop unit to gradually optimize the disparity prediction. This process first constructs a correlation body on the feature map of the largest scale, which contains the global information of the image and provides a basis for subsequent disparity prediction. In each iteration, the local information related to the current pixel point is extracted from the correlation body based on the current predicted disparity value (i.e., the current disparity map) as an index. This local information is then fed into the multi-level convolution gated loop unit initialized by the feature propagator to predict the disparity update amount of the current pixel point.

[0061] The above step S105 optimizes the disparity prediction based on the multi-level convolution gated recurrent unit initialized by the feature propagator and the initial disparity until the disparity prediction value reaches the convergence condition to obtain the final disparity map, including: S1051, reference Figure 2 As shown, a correlation volume is constructed based on the feature map of the left image at the maximum scale and the feature map of the right image at the maximum scale.

[0062] The cost volume is constructed by calculating the cost values ​​under different disparity assumptions for the feature maps of the left and right images at the maximum scale and combining them into a 3D disparity cost volume. Specifically, the correlation between pixels at different positions is calculated for the feature maps of the left and right images at the maximum scale to form a 3D correlation volume. Each element of this correlation volume represents the correlation or matching cost between the corresponding feature points in the left and right images at different disparities.

[0063] S1052. Using the current disparity map as an index, extract local information related to the current pixel from the correlation volume, and input the current disparity map, the feature map of the left-eye image at the maximum scale, and the local information into a multi-stage convolutional gated recurrent unit initialized by a feature propagator.

[0064] From the constructed correlation volume, local information related to the current pixel is extracted. This information usually includes feature points, disparity values, matching costs, etc. in the current pixel and its neighborhood. The extracted local information is input into the multi-level convolutional gated recurrent unit initialized by the feature propagator. The multi-level convolutional gated recurrent unit can capture long-term dependencies in sequence data and effectively process spatial information in the image.

[0065] S1053, the multi-level convolution gated recurrent unit generates a disparity update value of the current pixel based on the current disparity map, the feature map of the left-eye image at the maximum scale, and the local information.

[0066] The multi-level convolutional gated recurrent unit generates a disparity update value for the current pixel based on the input current disparity map, the feature map of the left image at the maximum scale, and local information. This update value reflects the adjustment amount to the disparity prediction value in the current iteration step.

[0067] S1054 , superimposing the corresponding disparity update value on the current disparity of each current pixel to obtain a disparity prediction value.

[0068] Reference Figure 2 As shown, the current disparity (i.e., the initial disparity or the disparity prediction value obtained in the previous iteration) of each current pixel is superimposed with the corresponding disparity update value to obtain a new disparity prediction value. In the first iteration, the initial disparity of each current pixel is superimposed with the corresponding disparity update value, and in subsequent iterations, the disparity prediction value obtained in the previous iteration of each current pixel is superimposed with the corresponding disparity update value.

[0069] S1055, iterating the disparity prediction process until the disparity prediction value reaches a convergence state, and obtaining a final disparity map.

[0070] The disparity update value will be used to optimize the global disparity prediction. Repeat the above steps S1051-S1054 to iterate the disparity prediction process. In each iteration, the disparity prediction value is updated according to the local information of the current pixel and the output of the multi-level convolution gated recurrent unit. When the disparity prediction value reaches the convergence condition (such as the number of iterations reaches the preset upper limit, the change in the disparity prediction value is less than the preset threshold, etc.), the iteration process is stopped and the final disparity map is output.

[0071] The present invention can capture the spatial information in the image and the long-term dependencies in the sequence data by using a multi-level convolutional gated recurrent unit, so as to more accurately predict the disparity value. The process of iterative disparity prediction can gradually optimize the disparity prediction value to make it closer to the true value. The multi-level convolutional gated recurrent unit has a certain robustness to the noise and changes of the input data, and can maintain good performance under different lighting, occlusion and noise conditions. Through the iterative optimization process, the influence of mismatching and noise can be gradually reduced, and the robustness of the algorithm can be improved. Moreover, the multi-level convolutional gated recurrent unit can reduce the computational complexity to a certain extent by performing feature extraction and disparity prediction at the maximum scale.

[0072] In an optional embodiment, the multi-level convolution gated recurrent unit includes a reset gate and an update gate. The multi-level convolution gated recurrent unit described in step S1053 generates a disparity update value of the current pixel based on the current disparity map, the feature map of the left eye image at the maximum scale, and the local information, including: S10531. The current disparity map, the feature map of the left-eye image at the maximum scale, and the local information are concatenated to obtain the local related information, and then input into the reset gate and the update gate. Inside the multi-level convolutional gated recurrent unit, the reset gate and the update gate are updated based on the local related information and the stored historical hidden layer features, respectively, to obtain the reset gate feature and the update gate feature.

[0073] First, the current disparity map, the feature map of the left image at the maximum scale, and the local information are concatenated to form a vector containing local related information, which is used as the input of the GRU. The above local information includes the disparity value and color information of the neighboring pixels.

[0074] Inside the multi-level convolutional gated recurrent unit, the reset gate and update gate are updated based on the local relevant information of the current input and the historical hidden layer features, respectively, to determine the information retention and update strategies.

[0075] The multi-level convolution gated recurrent unit in the present invention may also include a multi-level convolution layer, which is composed of three convolution layers with batch normalization and ReLU activation functions. These convolution layers are used to extract feature information from the input image, i.e., the above-mentioned local related information. Each layer of convolution gradually abstracts and refines the information in the image, so that higher-level convolution layers can capture more complex features.

[0076] The reset gate includes a single convolution layer for extracting the reset coefficient (reset gate). The role of the reset coefficient is to weaken the historical memory to generate candidate memory. When calculating the candidate memory, the reset gate uses the feature information extracted by the previous convolution layer and the memory state at the previous moment. By multiplying the reset coefficient with the memory state at the previous moment, the historical memory can be screened and weakened. The reset gate takes the input vector and the historical hidden layer state as input and obtains the reset gate feature through a sigmoid activation function. The update gate also includes a single convolution layer for extracting the update coefficient (update gate). The role of the update coefficient is to combine the candidate memory with the new feature to generate the latest memory. When calculating the latest memory, the update gate uses the feature information extracted by the previous convolution layer, the candidate memory and the memory state at the previous moment. By combining the update coefficient with the candidate memory and the new feature, the memory can be updated and fused. The update gate controls how much of the hidden layer state at the previous moment should be retained and how much of the candidate hidden layer state should be introduced. The update gate takes the input vector and the historical hidden layer state as input, and obtains the update gate feature through the sigmoid activation function. In a single iteration, the multi-level convolutional layer first extracts the feature information of the input image. Then, this feature information is passed to the reset gate and the update gate. The reset gate uses the feature information and the memory state of the previous moment to calculate the reset coefficient and generate candidate memories. The update gate uses the feature information, candidate memories, and the memory state of the previous moment to calculate the update coefficient and generate the latest memory. The latest memory is then used to predict the disparity update amount in a single iteration.

[0077] S10532. Combining the reset gate feature, the local related information and the historical hidden layer feature, a candidate hidden layer feature is obtained.

[0078] The historical hidden layer features are obtained through convolution operations or other transformations based on the previous reset gate features and the previous input vector (i.e., the above-mentioned local related information). After obtaining the historical hidden layer features, the reset gate features (i.e., the output value of the reset gate, which determines the degree of retention of historical information) and the input vector (i.e., the above-mentioned local related information) are combined to obtain candidate hidden layer features through convolution operations or other transformations. The candidate hidden layer features represent the possible hidden layer state updates under the combined action of the current input and historical information. The candidate hidden layer features combine the reset gate features, the local related information, and the historical hidden layer features, representing the ideal state after considering the new information. Specifically, the hidden layer first multiplies the historical hidden layer state with the reset gate features, and then obtains the candidate hidden layer features together with the local related information through a linear transformation and a tanh activation function.

[0079] S10533. Calculate updated hidden layer features by combining candidate hidden layer features, update gate features, and historical hidden layer features.

[0080] The updated hidden layer features are calculated by combining the candidate hidden layer features, the reset gate features and the historical hidden layer features through weighted summation or other combination methods. This step realizes the update of the hidden layer state, so that the updated hidden layer features contain both historical information and the new local information of the current input. For example, the updated hidden layer features are obtained by taking the weighted average of the historical hidden layer state and the candidate hidden layer state, and the weight is determined by the update gate feature. The weight of the historical hidden layer state is 1-the update gate feature, and the weight of the candidate hidden layer state is the update gate feature.

[0081] S10534. Calculate the disparity update value of the current pixel based on the updated hidden layer features.

[0082] Based on the updated hidden layer features, the disparity update value of the current pixel is calculated through linear transformation or other prediction models. This update value reflects the adjustment amount of the disparity prediction value in the current iteration step, and is used for the subsequent update of the disparity prediction value. By introducing the reset gate and update gate mechanism, the multi-level convolution gated recurrent unit can more flexibly handle long-term dependencies in sequence data, while effectively retaining and updating historical information. This enables the model to more accurately capture the spatial structure and depth information in the image in the disparity prediction task, thereby improving the accuracy of disparity prediction. The introduction of the reset gate and update gate enables the model to more flexibly adjust the information flow when facing complex situations such as noise and occlusion, reducing the impact of mismatching and noise. This enhances the robustness of the algorithm, enabling it to maintain good performance under different lighting, occlusion and noise conditions.

[0083] In an optional embodiment, the method of the present invention can quickly predict the disparity map based on the binocular image captured by the underwater binocular camera to obtain the scene depth. The above step S106 calculates the scene depth map and three-dimensional point cloud of the underwater environment based on the internal and external parameters and the final disparity map, including: S1061 . Convert the disparity value of each pixel in the disparity map into a corresponding depth value by using the internal and external parameters to generate a scene depth map.

[0084] The above-mentioned intrinsic parameters mainly include focal length (fx, fy, corresponding to the focal length in the x-axis and y-axis directions, respectively), principal point coordinates (x0, y0, i.e., optical center coordinates, relative to the imaging plane), distortion coefficients, etc. The above-mentioned extrinsic parameters are parameters that describe the position and posture of the camera in the world coordinate system, such as the baseline distance of the binocular camera. Based on the stereo matching result obtained in the above step S105, i.e., the above-mentioned disparity map, the depth of each pixel is calculated by triangulation.

[0085] Specifically, the disparity value of each pixel in the disparity map can be converted into a corresponding depth value by using the internal and external parameters of the camera, especially the focal length and the coordinates of the principal point, as well as the baseline distance of the binocular camera (i.e., the distance between the optical centers of the two cameras). The depth value represents the distance between the scene point corresponding to the pixel and the camera, thereby generating a depth map of the entire scene. The formula for converting disparity to depth is: depth = (f * baseline) / disp

[0086] Among them, depth is the depth value, f is the focal length, baseline is the baseline distance, and disp is the disparity value. Through this conversion, a scene depth map can be generated, that is, each pixel corresponds to a depth value representing its distance from the camera.

[0087] S1062: Back-project the scene depth map in combination with internal and external parameters to obtain a three-dimensional point cloud of the underwater environment.

[0088] After obtaining the scene depth map, the depth map can be back-projected in combination with the internal and external parameters of the camera, and the pixels on the two-dimensional image can be back-projected into three-dimensional space. This process corresponds each pixel on the two-dimensional image to the coordinates in the real three-dimensional space, thereby obtaining a three-dimensional point cloud of the underwater environment. The point cloud structure contains the three-dimensional position information of each visible point in the scene, and provides a complete scene geometry structure for subsequent applications such as environmental perception or three-dimensional reconstruction. The process of back-projecting the depth map from two-dimensional space to three-dimensional space can refer to the description in the prior art and will not be repeated here.

[0089] The present invention can measure the scene structure and depth information in an underwater environment through an underwater robot carrying a binocular camera. By combining the internal and external parameters of the camera and the disparity map to calculate the three-dimensional point cloud, the ranging ability of the binocular camera can be fully utilized to improve the accuracy of three-dimensional reconstruction. Especially in an underwater environment, due to complex lighting conditions and large noise interference, traditional three-dimensional reconstruction methods are often difficult to obtain high-precision results. The method based on internal and external parameters and disparity maps can more accurately capture the spatial structure and depth information in the image, thereby generating a more refined and accurate three-dimensional point cloud. The method can adapt to different lighting conditions and noise interference in the underwater environment. Since the internal and external parameters of the camera are used for accurate coordinate transformation and depth calculation, even if there are problems such as lighting changes and occlusion in the underwater environment, good performance can be maintained. This enhances the robustness of the algorithm, and reliable three-dimensional reconstruction results can be obtained in different scenes.

[0090] The generated 3D point cloud can provide reliable basic data for subsequent underwater environment perception, object recognition, topographic measurement and other tasks. These subsequent tasks usually require high-precision 3D information as input, and the method based on internal and external parameters and disparity map just meets this demand. Therefore, this method plays an important role in promoting the development of underwater robot technology, marine engineering, underwater archaeology and other fields.

[0091] The underwater robot environment perception device based on the feature propagator provided by the present invention is described below. The underwater robot environment perception device based on the feature propagator described below and the underwater robot environment perception method based on the feature propagator described above can be referenced to each other.

[0092] The underwater robot environment perception device based on the feature propagator provided by the present invention refers to Figure 4 As shown, it includes the following modules: An image acquisition module 201 is used to acquire a binocular image collected by an underwater robot and a pre-established deep neural network; wherein the deep neural network includes a feature extractor, a cost aggregator, a feature propagator and an iterative optimizer; A feature extraction module 202 is used to correct the binocular image, input the corrected binocular image into the feature extractor, obtain feature maps at different scales and form a feature pyramid; A cost aggregation module 203 is used to use a cost aggregator to perform cost volume construction, cost volume aggregation and disparity regression based on each of the feature pyramids to obtain an original cost volume, an aggregated cost volume and an initial disparity respectively; An initialization module 204 is configured to initialize the multi-level convolution gated recurrent unit in the iterative optimizer based on the original cost body and the aggregated cost body by the feature propagator; A disparity prediction module 205, configured to optimize the disparity prediction based on the multi-level convolution gated recurrent unit initialized by the feature propagator until the disparity prediction value reaches a convergence condition, thereby obtaining a final disparity map; The calculation module 206 is used to obtain the internal and external parameters of the binocular camera of the underwater robot, and calculate the scene depth map and the three-dimensional point cloud of the underwater environment based on the internal and external parameters and the final disparity map.

[0093] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the underwater robot environment perception method based on the feature propagator.

[0094] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the underwater robot environment perception method based on feature propagators provided by the above-mentioned methods.

[0096] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the underwater robot environment perception method based on feature propagators provided by the above-mentioned methods.

[0097] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An underwater robot environment perception method based on feature propagator, characterized in that: include: Acquire a binocular image collected by an underwater robot and a pre-established deep neural network; wherein the deep neural network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer; Correcting the binocular images, and inputting the corrected binocular images into the feature extractors respectively to obtain feature maps at different scales and form a feature pyramid; Using a cost aggregator to perform cost volume construction, cost volume aggregation and disparity regression based on each of the feature pyramids, to obtain an original cost volume, an aggregated cost volume and an initial disparity respectively; Initializing, by a feature propagator, a multi-level convolutional gated recurrent unit in an iterative optimizer based on the original cost body and the aggregated cost body; Optimize the disparity prediction based on the multi-level convolutional gated recurrent unit initialized by the feature propagator and the initial disparity until the disparity prediction value reaches the convergence condition to obtain the final disparity map; The internal and external parameters of the binocular camera of the underwater robot are obtained, and the scene depth map and the three-dimensional point cloud of the underwater environment are calculated based on the internal and external parameters and the final disparity map.

2. The underwater robot environment perception method based on feature propagator according to claim 1 is characterized in that: The cost aggregator includes a cost volume construction network, a cost aggregation network and a disparity regressor; The cost aggregator is used to perform cost volume construction, cost volume aggregation and disparity regression based on each of the feature pyramids to obtain an original cost volume, an aggregated cost volume and an initial disparity, respectively, including: The cost volume construction network constructs a cost volume based on the feature graph at the maximum scale in each of the feature pyramids to obtain an original cost volume; The cost aggregation network filters and concentrates the local cost information in the original cost volume to obtain an aggregated cost volume; The disparity regressor performs an initial disparity estimate based on the aggregated cost volume to obtain the initial disparity.

3. The underwater robot environment perception method based on feature propagator according to claim 1 is characterized in that: The feature propagator includes a plurality of parallel residual modules, and the binocular image includes a left-eye image and a right-eye image; The step of initializing the multi-level convolution gated recurrent unit in the iterative optimizer based on the original cost body and the aggregated cost body by the feature propagator includes: For the top recurrent unit in the multi-level convolutional gated recurrent unit, the feature map of the left image at the maximum scale, the original cost volume and the aggregated cost volume are concatenated and input into the top residual module to extract the aggregated features, and the hidden layer of the top recurrent unit is initialized with the aggregated features; Furthermore, for the recurrent units in other layers except the top layer in the multi-level convolutional gated recurrent unit, the feature map of the left-eye image is input into the residual module of the corresponding number of layers at the corresponding scale to extract the image context features, and the hidden layer of the corresponding recurrent unit is initialized with the image context features.

4. The underwater robot environment perception method based on feature propagator according to claim 1 is characterized in that: The binocular image includes a left eye image; the multi-level convolution gated recurrent unit initialized by the feature propagator and the initial disparity optimization disparity prediction are performed until the disparity prediction value reaches a convergence condition to obtain a final disparity map, including: Constructing a correlation volume based on a feature map of the left image at the maximum scale and a feature map of the right image at the maximum scale; According to the current disparity map as an index, the local information related to the current pixel is extracted from the correlation volume, and the current disparity map, the feature map of the left image at the maximum scale and the local information are input into the multi-level convolution gated recurrent unit initialized by the feature propagator; The multi-level convolutional gated recurrent unit generates the disparity update value of the current pixel based on the current disparity map, the feature map of the left image at the maximum scale, and local information; The current disparity of each current pixel is superimposed with the corresponding disparity update value to obtain a disparity prediction value; wherein, the current disparity in the first iteration is the initial disparity; The disparity prediction process is iterated until the disparity prediction value reaches a convergence state, and the final disparity map is obtained.

5. The underwater robot environment perception method based on feature propagator according to claim 4 is characterized in that: The multi-stage convolution gated recurrent unit includes a reset gate and an update gate; The multi-level convolution gated recurrent unit generates a disparity update value of a current pixel based on the current disparity map, the feature map of the left-eye image at the maximum scale, and local information, including: The local related information is obtained by splicing the current disparity map, the feature map of the left image at the maximum scale, and the local information, and then input into the reset gate and the update gate. The reset gate and the update gate are updated based on the local related information and the stored historical hidden layer features, respectively, to obtain the reset gate feature and the update gate feature; Combining the reset gate features, local related information and historical hidden layer features to obtain candidate hidden layer features; Calculate updated hidden layer features by combining candidate hidden layer features, update gate features and historical hidden layer features; The disparity update value of the current pixel is calculated based on the updated hidden layer features.

6. The underwater robot environment perception method based on feature propagator according to claim 1 is characterized in that: The method of calculating a scene depth map and a three-dimensional point cloud of an underwater environment based on internal and external parameters and the final disparity map includes: Using the internal and external parameters, the disparity value of each pixel in the disparity map is converted into a corresponding depth value to generate a scene depth map; The scene depth map is back-projected in combination with internal and external parameters to obtain a three-dimensional point cloud of the underwater environment.

7. An underwater robot environment perception method based on feature propagator, characterized in that: include: An image acquisition module, used to acquire binocular images collected by the underwater robot and a pre-established deep neural network; wherein the deep neural network includes a feature extractor, a cost aggregator, a feature propagator and an iterative optimizer; A feature extraction module, used to correct the binocular image, input the corrected binocular image into the feature extractor respectively, obtain feature maps at different scales and form a feature pyramid; A cost aggregation module, used for performing cost volume construction, cost volume aggregation and disparity regression based on each of the feature pyramids using a cost aggregator to obtain an original cost volume, an aggregated cost volume and an initial disparity respectively; An initialization module, configured to initialize a multi-level convolutional gated recurrent unit in an iterative optimizer based on the original cost body and the aggregated cost body by a feature propagator; A disparity prediction module, configured to optimize the disparity prediction based on the multi-level convolutional gated recurrent unit initialized by the feature propagator and the initial disparity until the disparity prediction value reaches a convergence condition to obtain a final disparity map; The calculation module is used to obtain the internal and external parameters of the binocular camera of the underwater robot, and calculate the scene depth map and the three-dimensional point cloud of the underwater environment based on the internal and external parameters and the final disparity map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the underwater robot environment perception method based on the feature propagator as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the underwater robot environment perception method based on a feature propagator as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the underwater robot environment perception method based on a feature propagator as described in any one of claims 1 to 6 is implemented.

Images