An underwater robot environment perception method and device based on feature propagator

Abstract

This invention provides an underwater robot environment perception method and apparatus based on a feature propagator. The method includes: acquiring binocular images collected by the underwater robot and a pre-established deep neural network; correcting the binocular images and inputting the corrected images into the network's feature extractor to obtain feature maps at different scales and assembling feature pyramids; using a cost aggregator to construct cost volumes, aggregate cost volumes, and perform disparity regression based on each feature pyramid; initializing multi-level convolutional gated recurrent units in the iterative optimizer based on the original and aggregated cost volumes using the feature propagator; optimizing disparity prediction based on the multi-level convolutional gated recurrent units initialized by the feature propagator until the disparity prediction values ​​reach convergence, obtaining the final disparity map; acquiring the intrinsic and extrinsic parameters of the binocular camera; and calculating the scene depth map and 3D point cloud of the underwater environment based on the intrinsic and extrinsic parameters and the disparity map, achieving robust and efficient underwater environment perception.

Description

Technical Field

[0001] This invention relates to the field of underwater visual perception technology, and in particular to an underwater robot environmental perception method and device based on a feature propagator. Background Technology

[0002] In the field of underwater environmental perception, binocular vision systems are widely studied and applied due to their ability to provide rich depth information. Binocular vision systems mimic the human binocular stereoscopic vision mechanism, calculating the depth information of objects from images captured by two cameras, thereby achieving three-dimensional perception of the underwater environment. However, the unique characteristics of the underwater environment, such as light attenuation and water turbidity, pose challenges to the accuracy and reliability of binocular vision systems. Summary of the Invention

[0003] This invention provides an underwater robot environment perception method and device based on a feature propagator, to overcome the shortcomings of poor accuracy in existing underwater environment perception technologies and achieve robust and efficient underwater environment perception. The technical solution proposed by this invention is as follows:

[0004] In a first aspect, the present invention provides an underwater robot environmental perception method based on a feature propagator, comprising:

[0005] The system acquires binocular images collected by an underwater robot and a pre-established deep neural network; wherein the deep neural network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer.

[0006] The binocular images are corrected, and the corrected binocular images are input into the feature extractor to obtain feature maps at different scales and form a feature pyramid.

[0007] The cost aggregator is used to construct the cost body, aggregate the cost body, and regress the disparity based on each of the feature pyramids, to obtain the original cost body, the aggregated cost body, and the initial disparity, respectively.

[0008] The feature propagator initializes the multi-level convolutional gated recurrent units in the iterative optimizer based on the original cost body and the aggregated cost body;

[0009] The disparity prediction is optimized based on a multi-level convolutional gated recurrent unit initialized by a feature propagator until the disparity prediction value reaches the convergence condition, and the final disparity map is obtained.

[0010] Acquire the intrinsic and extrinsic parameters of the underwater robot's binocular camera, and calculate the scene depth map and 3D point cloud of the underwater environment based on the intrinsic and extrinsic parameters and the final disparity map.

[0011] Optionally, the cost aggregator includes a cost volume construction network, a cost aggregation network, and a disparity regressor;

[0012] The process of using a cost aggregator to construct cost volumes, aggregate cost volumes, and perform disparity regression based on each of the feature pyramids yields the original cost volume, the aggregated cost volume, and the initial disparity, respectively, including:

[0013] The cost body construction network constructs the cost body based on the feature map at the largest scale in each of the feature pyramids, thus obtaining the original cost body.

[0014] The cost aggregation network filters and concentrates the local cost information in the original cost volume to obtain the aggregated cost volume.

[0015] The disparity regressor obtains the initial disparity by performing initial disparity estimation based on the aggregated cost volume.

[0016] Optionally, the feature propagator includes multiple parallel residual modules, and the binocular image includes a left-eye image and a right-eye image;

[0017] The process of initializing the multi-level convolutional gated recurrent unit in the iterative optimizer by the feature propagator based on the original cost body and the aggregated cost body includes:

[0018] For the top-level recurrent unit in the multi-level convolutional gated recurrent unit, the feature map at the maximum scale of the left eye image, the original cost volume, and the aggregated cost volume are concatenated and input into the top-level residual module to extract the aggregated features, and the hidden layer of the top-level recurrent unit is initialized with the aggregated features.

[0019] Furthermore, for the recurrent units other than the top layer in the multi-level convolutional gated recurrent unit, the feature map of the left eye image is input into the residual module of the corresponding layer at the corresponding scale to extract the image context features, and the hidden layer of the corresponding recurrent unit is initialized with the image context features.

[0020] Optionally, the binocular image includes the left eye image; the step of optimizing disparity prediction based on multi-level convolutional gated recurrent units initialized by feature propagators and initial disparity until the disparity prediction value reaches the convergence condition to obtain the final disparity map includes:

[0021] A correlation volume is constructed based on the feature maps of the left and right eyes at the maximum scale.

[0022] Using the current disparity map as an index, local information related to the current pixel is extracted from the relevant volume, and the current disparity map, the feature map of the left eye image at the maximum scale, and the local information are input into a multi-level convolutional gated recurrent unit initialized by the feature propagator.

[0023] The multi-level convolutional gated recurrent unit generates the disparity update value of the current pixel based on the current disparity map, the feature map of the left eye image at the maximum scale, and local information;

[0024] For each current pixel, the current disparity is superimposed with the corresponding disparity update value to obtain the disparity prediction value; where the current disparity in the first iteration is the initial disparity;

[0025] The process of iteratively predicting disparity continues until the predicted disparity values ​​converge, resulting in the final disparity map.

[0026] Optionally, the multi-level convolutional gated loop unit includes a reset gate and an update gate;

[0027] The multi-level convolutional gated recurrent unit generates the disparity update value for the current pixel based on the current disparity map, the feature map of the left eye image at the maximum scale, and local information, including:

[0028] The current disparity map, the feature map of the left eye image at the maximum scale, and local information are concatenated to obtain local related information, which is then input into the reset gate and the update gate. The reset gate and the update gate are updated based on the local related information and the stored historical hidden layer features, respectively, to obtain the reset gate features and the update gate features.

[0029] Candidate hidden layer features are obtained by combining reset gate features, local relevant information, and historical hidden layer features;

[0030] The updated hidden layer features are calculated by combining candidate hidden layer features, update gate features, and historical hidden layer features;

[0031] The disparity update value of the current pixel is calculated based on the updated hidden layer features.

[0032] Optionally, the calculation of the scene depth map and 3D point cloud of the underwater environment based on intrinsic and extrinsic parameters and the final disparity map includes:

[0033] Using intrinsic and extrinsic parameters, the disparity value of each pixel in the disparity map is converted into a corresponding depth value to generate a scene depth map;

[0034] By combining intrinsic and extrinsic parameters, the depth map of the scene is back-projected to obtain a three-dimensional point cloud of the underwater environment.

[0035] Secondly, the present invention also provides an underwater robot environmental perception device based on a feature propagator, comprising the following modules:

[0036] An image acquisition module is used to acquire binocular images collected by an underwater robot and a pre-established deep neural network; wherein, the deep neural network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer;

[0037] The feature extraction module is used to correct the binocular images, and input the corrected binocular images into the feature extractor to obtain feature maps at different scales and form a feature pyramid.

[0038] The cost aggregation module is used to construct cost bodies, aggregate cost bodies, and regress disparity based on each of the feature pyramids using a cost aggregator, to obtain the original cost body, aggregated cost body, and initial disparity, respectively.

[0039] An initialization module is used to initialize the multi-level convolutional gated recurrent unit in the iterative optimizer based on the original cost body and the aggregated cost body by the feature propagator;

[0040] The disparity prediction module is used to optimize disparity prediction based on multi-level convolutional gated recurrent units initialized by the feature propagator until the disparity prediction value reaches the convergence condition and the final disparity map is obtained.

[0041] The calculation module is used to acquire the intrinsic and extrinsic parameters of the underwater robot's binocular camera, and calculate the scene depth map and 3D point cloud of the underwater environment based on the intrinsic and extrinsic parameters and the final disparity map.

[0042] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the underwater robot environmental perception method based on feature propagators as described in the first aspect above.

[0043] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the underwater robot environmental perception method based on feature propagators as described in the first aspect above.

[0044] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the underwater robot environmental perception method based on a feature propagator as described in the first aspect above.

[0045] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0046] This invention provides an underwater robot environment perception method and apparatus based on a feature propagator. First, high-dimensional features are extracted from binocular images using a feature extractor. Then, a cost aggregator is used to obtain an aggregated cost volume and an initial disparity estimate. Subsequently, an iterative optimizer is used to iteratively update the initial disparity. This invention combines two neural network structures: a cost aggregator and an iterative optimizer. The method uses a feature propagator to bridge the compactly designed cost aggregator and iterative optimizer. The iterative optimizer efficiently updates the initial disparity estimated by the cost aggregator, performing disparity prediction. Based on the predicted disparity map, triangulation and back-projection are performed using the intrinsic and extrinsic parameters of the binocular cameras to obtain the underwater environment's scene depth map and 3D point cloud, achieving robust and efficient underwater environment perception. Before iteratively updating the initial disparity, this invention creatively introduces a feature propagator to initialize the multi-level convolutional gated recurrent units in the iterative optimizer, providing initial values ​​for the hidden layers and improving the efficiency of subsequent optimization. This invention can be deployed on underwater robots to efficiently calculate disparity information in binocular images, thereby obtaining underwater environment depth and point cloud information.

[0047] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the underwater robot environmental perception method based on feature propagators provided by the present invention.

[0051] Figure 2 This is a schematic diagram of the deep neural network structure provided by the present invention.

[0052] Figure 3 This is a schematic diagram of the feature propagator provided by the present invention.

[0053] Figure 4 This is a schematic diagram of the underwater robot environmental perception device based on a feature propagator provided by the present invention.

[0054] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The following is combined Figures 1-4 This invention describes an underwater robot environmental perception method and apparatus based on a feature propagator.

[0057] Reference Figure 1 As shown, the method includes the following:

[0058] Step S101: Acquire binocular images and pre-established deep neural networks collected by the underwater robot.

[0059] The underwater robot acquires image pairs of the underwater environment using its onboard binocular cameras. These image pairs contain the same scene observed from different perspectives; the binocular images described above in this invention include a left-eye image and a right-eye image.

[0060] Pre-train a deep neural network, refer to Figure 2 and Figure 3 As shown, the network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer. This network is used to extract features from stereo images, construct a cost volume, perform disparity estimation, and optimize.

[0061] Step S102: Correct the binocular images, and input the corrected binocular images into the feature extractor to obtain feature maps at different scales and form a feature pyramid.

[0062] The acquired binocular images are corrected according to the pinhole camera model to make the camera optical centers of the left and right eyes parallel after correction, which helps with subsequent parallax calculation.

[0063] The corrected binocular images are input into a feature extractor, which extracts features from the images using structures such as convolutional neural networks (CNNs). These features are organized into a feature pyramid at different scales. The feature pyramid provides multi-scale feature representations from coarse to fine, which helps in handling objects and details at different scales. For both the left and right binocular images, the feature extractor obtains feature maps of the same image at different scales, and then combines these feature maps at different scales to form a feature pyramid.

[0064] The head of the feature extractor in this invention consists of six convolutional layers with batch normalization and ReLU activation functions. The stride of the convolutional kernels in layers 1, 3, and 5 is set to 2 to downsample the feature maps. The remaining part of the feature extractor consists of multiple consecutive pooling layers with a stride of 2, used for rapid extraction of multi-scale feature pyramids. After the corrected left and right eye images are input into the feature extractor, they are passed through multiple convolutional layers in the feature extractor head to obtain feature images downsampled by 3 times. Subsequently, the remaining pooling layers further downsample the obtained feature images to obtain feature maps at different scales and form feature pyramids.

[0065] Step S103: Using the cost aggregator, cost bodies are constructed, cost bodies are aggregated, and disparity is regressed based on each of the feature pyramids to obtain the original cost body, aggregated cost body, and initial disparity, respectively.

[0066] The original cost volume is constructed based on the feature pyramid using a cost aggregator. The cost volume is a three-dimensional data structure where each element represents the cost of matching two feature points under different disparities. The original cost volumes are aggregated to reduce noise and ambiguity, resulting in an aggregated cost volume. An initial disparity map is estimated from the aggregated cost volume using disparity regression.

[0067] Step S104: The feature propagator initializes the multi-level convolutional gated recurrent unit in the iterative optimizer based on the original cost body and the aggregated cost body.

[0068] Reference Figure 2 As shown, the feature propagator utilizes the original cost body (i.e. Figure 2 The cost body and aggregate cost body are used to initialize the multi-level convolutionally gated recurrent units (GRUs) in the iterative optimizer. GRUs are a variant of recurrent neural networks (RNNs) that can capture long-term dependencies in sequential data and are used in this invention to iteratively optimize disparity prediction.

[0069] Step S105: Optimize disparity prediction based on multi-level convolutional gated recurrent units initialized by the feature propagator until the disparity prediction value reaches the convergence condition, and obtain the final disparity map.

[0070] Through iterative optimization using a multi-level GRU, disparity prediction is gradually refined until the disparity prediction values ​​stabilize and convergence is achieved. The convergence condition can be set to ensure that the change in disparity prediction values ​​is less than a certain threshold. After the iterative optimization is complete, a reference value is obtained. Figure 2The final disparity shown is the final disparity map mentioned above. The final disparity map represents the disparity of each pixel in the binocular image pair, that is, the relative positional difference between the two camera viewpoints.

[0071] Step S106: Obtain the intrinsic and extrinsic parameters of the underwater robot's binocular camera, and calculate the scene depth map and 3D point cloud of the underwater environment based on the intrinsic and extrinsic parameters and the final disparity map.

[0072] The intrinsic and extrinsic parameters of a stereo camera describe its position, orientation, and lens properties. By calibrating the stereo cameras, intrinsic parameters such as focal length and principal point coordinates, as well as extrinsic parameters such as relative position and attitude between cameras, are obtained. Intrinsic parameters describe the camera's internal properties, while extrinsic parameters describe the camera's position and attitude in the world coordinate system. Based on the intrinsic and extrinsic parameters and the final disparity map, a 3D point cloud of the underwater environment is calculated using triangulation principles. The 3D point cloud is a set of 3D coordinates of objects in the underwater environment and can be used for further tasks such as environmental perception, navigation, and obstacle avoidance.

[0073] This invention extracts multi-scale features using a feature extractor and reduces noise and ambiguity using a cost aggregator, improving the robustness and accuracy of disparity estimation. Furthermore, the combined use of a feature propagator and an iterative optimizer further refines disparity prediction and reduces errors. The generated 3D point cloud provides accurate 3D information about the underwater environment, aiding underwater robots in more accurate navigation, obstacle avoidance, and terrain reconstruction. The 3D point cloud can also be used for underwater object recognition and classification, enhancing the autonomous operation capabilities of underwater robots. This method is based on deep neural networks, possessing strong learning and generalization abilities. Through training and optimization, it can adapt to different underwater environments and lighting conditions.

[0074] In an optional embodiment, the cost aggregator in step S103 above includes three parts: a cost volume construction network, a cost aggregation network, and a disparity regressor.

[0075] Step S103 above describes using a cost aggregator to construct cost bodies, aggregate cost bodies, and perform disparity regression based on each of the feature pyramids, obtaining the original cost body, aggregated cost body, and initial disparity, respectively. (Refer to...) Figure 2 As shown, it includes:

[0076] S1031. The cost body construction network constructs the cost body based on the feature map at the largest scale in each of the feature pyramids, thus obtaining the original cost body.

[0077] The cost body construction network is based on the feature pyramid obtained above, and constructs the cost body using the feature map at the maximum scale (i.e., at the maximum resolution), which is referred to as the original cost body.

[0078] First, feature maps at the largest scale are extracted from the feature pyramid. These feature maps typically contain the most salient and important information in the image and form the basis for constructing the cost volume.

[0079] Next, the cost volume construction network uses these feature maps to construct a three-dimensional cost volume through a series of convolutions, pooling, and other operations. Each element of this cost volume represents the matching cost between two feature points under different disparities.

[0080] Finally, the original cost body is obtained, which is a three-dimensional data structure containing rich matching information.

[0081] S1032. The cost aggregation network filters and concentrates the local cost information in the original cost volume to obtain the aggregated cost volume.

[0082] While the original cost volume contains rich matching information, it may also contain noise and redundancy. This invention utilizes a cost aggregation network to further filter and centrally process the original cost volume, thereby reducing noise and redundancy and improving the accuracy and robustness of the cost volume.

[0083] The cost aggregation network described above in this invention is an hourglass-shaped three-dimensional convolutional neural network, in which feature volumes at the same scale contain cross-layer connections. The original cost volume is input into the hourglass-shaped three-dimensional convolutional neural network, and the local cost information is filtered and concentrated through a series of operations such as convolution, pooling, and weighted averaging to obtain the aggregated cost volume (hereinafter referred to as the aggregated cost volume).

[0084] Cost aggregation networks can employ various strategies to optimize the cost volume, such as using larger convolutional kernels to capture broader contextual information, or using attention mechanisms to focus on important feature points.

[0085] S1033, The disparity regressor obtains the initial disparity by performing initial disparity estimation based on the aggregated cost volume.

[0086] The aggregated cost volume is already a relatively accurate and robust cost volume, which can be used for disparity estimation. A disparity regressor is then used to traverse and regress the aggregated cost volume to estimate the disparity value of each pixel. By finding the minimum cost value corresponding to each pixel and using its corresponding disparity as the disparity estimate for that pixel, the initial disparity map is finally obtained.

[0087] The disparity regressor described above includes a SoftMax operation and a mean operation, used to predict disparity from the aggregated cost volume. Specifically, the disparity regressor can be used to perform a SoftMax operation on the aggregated cost volume in the depth channel and calculate the mean to obtain the initial disparity of each pixel. The initial disparity map is then obtained by combining the initial disparities of each pixel.

[0088] This invention utilizes a cost aggregation network to filter and centrally process the original cost volume, reducing the impact of noise and redundant information, thereby improving the algorithm's robustness. Even in complex and noisy environments, the algorithm maintains good performance. Disparity estimation based on the aggregated cost volume significantly improves the accuracy of disparity estimation and reduces the impact of mismatches and noise. Using a feature pyramid and cost aggregation network, feature information in the image can be extracted and integrated more effectively, resulting in a more accurate and robust cost volume. Furthermore, the construction of the feature pyramid and the aggregation of the cost volume can be performed at different scales, thereby reducing computational load and improving computational efficiency.

[0089] In an optional embodiment, the feature propagator in step S104 above includes multiple parallel residual modules. At the top layer, the residual module takes the stitched original cost body, the aggregated cost body, and the feature map at the maximum scale of the left eye image as input to extract aggregated features; in other layers, the residual module extracts features only with the feature map at the corresponding scale as input.

[0090] The step S104 above, which involves the feature propagator initializing the multi-level convolutional gated recurrent unit in the iterative optimizer based on the original cost body and the aggregated cost body, includes:

[0091] S1041, reference Figure 3 As shown, for the top-level recurrent unit in the multi-level convolutional gated recurrent unit, the feature map at the maximum scale of the left eye image, the original cost volume, and the aggregated cost volume are concatenated and input into the top-level residual module to extract the aggregated features, and the hidden layer of the top-level recurrent unit is initialized with the aggregated features.

[0092] First, a feature map at the maximum scale is extracted from the left-eye image, and this feature map is concatenated with the original cost volume and the aggregated cost volume. The concatenation operation is performed along a specific dimension (such as the channel dimension) to fuse information from different sources. The concatenated features are then fed into the top-level residual module. The residual module extracts and enhances features through convolution operations and skip connections to obtain aggregated features. Finally, the aggregated features are used to initialize the hidden layer of the top-level recurrent unit in a multi-level convolutional gated recurrent unit. In this way, the top-level recurrent unit possesses comprehensive information from the left-eye image, the original cost volume, and the aggregated cost volume.

[0093] S1042, reference Figure 3 As shown, for the recurrent units other than the top layer in the multi-level convolutional gated recurrent unit, the feature map of the left eye image is input into the residual module of the corresponding layer at the corresponding scale to extract the image context features, and the hidden layer of the corresponding recurrent unit is initialized with the image context features.

[0094] For recurrent units (ROUs) in a multi-level convolutional gated recurrent unit (GU), except for the top layer, feature maps are first extracted from the left-eye image at the corresponding scale. Then, these feature maps are input into the residual module of the corresponding layer to extract image context features. For example, for the second-layer GU, the feature map at the second scale of the left-eye image is input into the residual module of the second layer to extract image context features. The extracted image context features are then used to initialize the hidden layer of the corresponding GU. In this way, each GU possesses feature information from the left-eye image at the corresponding scale.

[0095] This invention utilizes multi-level convolutional gated recurrent units and residual modules to enable the feature propagator to more effectively propagate feature information across different levels and scales. This helps reduce information loss and redundancy, improving the efficiency of feature propagation. The convolutional operations and skip connections in the residual modules enhance the expressive power of features, resulting in richer and more accurate extracted features. This contributes to improved performance in subsequent disparity estimation, stereo matching, and 3D reconstruction. This method is based on deep neural networks and possesses strong learning and generalization capabilities. Through training and optimization, it can adapt to the needs of different scenarios and tasks, achieving effective processing of various complex scenarios.

[0096] In the multi-level convolutional gated recurrent unit (MLU), the hidden layer is updated and optimized iteratively. Each iteration updates the hidden layer's state using the result of the previous iteration (i.e., the current disparity map) and the current input feature information. During iteration, gating mechanisms (such as reset and update gates) can be used to control the flow and updating of information, achieving more efficient feature propagation and optimization. In an optional embodiment, the iterative optimizer uses the multi-level convolutional gated recurrent unit to progressively optimize disparity prediction. This process first constructs a correlation volume on the largest-scale feature map, which contains global information about the image, providing a foundation for subsequent disparity prediction. In each iteration, local information related to the current pixel is extracted from the correlation volume using the currently predicted disparity value (i.e., the current disparity map) as an index. This local information is then fed into the multi-level convolutional gated recurrent unit initialized by the feature propagator to predict the disparity update amount for the current pixel.

[0097] Step S105 above, which involves optimizing disparity prediction based on multi-level convolutional gated recurrent units initialized by the feature propagator and initial disparity, until the disparity prediction value reaches the convergence condition to obtain the final disparity map, includes:

[0098] S1051, reference Figure 2 As shown, a correlation volume is constructed based on the feature maps of the left and right eyes at the maximum scale.

[0099] The cost volume is constructed by calculating the cost under different disparity assumptions for the feature maps of the left and right eye images at maximum scale and combining them into a three-dimensional disparity cost volume. Specifically, the correlation between pixels at different locations is calculated from the feature maps of the left and right eye images at maximum scale to form a three-dimensional correlation volume. Each element of this correlation volume represents the correlation or matching cost between corresponding feature points in the left and right eye images under different disparities.

[0100] S1052. Using the current disparity map as an index, extract local information related to the current pixel from the relevant volume, and input the current disparity map, the feature map of the left eye image at the maximum scale, and the local information into the multi-level convolutional gated recurrent unit initialized by the feature propagator.

[0101] From the constructed correlation volume, local information related to the current pixel is extracted. This information typically includes feature points, disparity values, and matching costs within the current pixel and its neighborhood. The extracted local information is then input into a multi-level convolutional gated recurrent unit initialized by a feature propagator. This multi-level convolutional gated recurrent unit can capture long-term dependencies in sequential data and effectively process spatial information in images.

[0102] S1053, the multi-level convolutional gated recurrent unit generates the disparity update value of the current pixel based on the current disparity map, the feature map of the left eye image at the maximum scale, and local information.

[0103] The multi-level convolutional gated recurrent unit generates a disparity update value for the current pixel based on the input current disparity map, the feature map of the left eye image at the maximum scale, and local information. This update value reflects the adjustment amount of the disparity prediction value in the current iteration step.

[0104] S1054. For each current pixel, the current disparity is superimposed with the corresponding disparity update value to obtain the disparity prediction value.

[0105] Reference Figure 2 As shown, a new disparity prediction value is obtained by superimposing the current disparity (i.e., the initial disparity or the disparity prediction value obtained from the previous iteration) of each current pixel with the corresponding disparity update value. In the first iteration, the initial disparity of each current pixel is superimposed with the corresponding disparity update value, and in subsequent iterations, the disparity prediction value obtained from the previous iteration of each current pixel is superimposed with the corresponding disparity update value.

[0106] S1055. The process of iteratively predicting disparity continues until the disparity prediction values ​​reach a convergence state, resulting in the final disparity map.

[0107] The disparity update value will be used to optimize the global disparity prediction. Repeat steps S1051-S1054 to iterate the disparity prediction process. In each iteration, the disparity prediction value is updated based on the local information of the current pixel and the output of the multi-level convolutional gated recurrent unit. When the disparity prediction value reaches the convergence condition (e.g., the number of iterations reaches a preset upper limit, the change in the disparity prediction value is less than a preset threshold, etc.), the iteration process stops, and the final disparity map is output.

[0108] This invention, by using multi-level convolutional gated recurrent units, can capture spatial information in images and long-term dependencies in sequential data, thereby predicting disparity values ​​more accurately. The iterative process of disparity prediction gradually optimizes the predicted disparity values, making them closer to the true values. The multi-level convolutional gated recurrent units are robust to noise and variations in the input data, maintaining good performance under different lighting, occlusion, and noise conditions. Through iterative optimization, the impact of mismatches and noise can be gradually reduced, improving the algorithm's robustness. Furthermore, by performing feature extraction and disparity prediction at the maximum scale, the multi-level convolutional gated recurrent units can reduce computational complexity to some extent.

[0109] In an optional embodiment, the multi-level convolutional gated recurrent unit described above includes a reset gate and an update gate. The multi-level convolutional gated recurrent unit in step S1053 generates a disparity update value for the current pixel based on the current disparity map, the feature map of the left-eye image at the maximum scale, and local information, including:

[0110] S10531. After concatenating the current disparity map, the feature map of the left eye image at the maximum scale, and local information to obtain local related information, input it into the reset gate and the update gate. Inside the multi-level convolutional gated recurrent unit, the reset gate and the update gate are updated based on the local related information and the stored historical hidden layer features, respectively, to obtain the reset gate features and the update gate features.

[0111] First, the current disparity map, the feature map of the left-eye image at its maximum scale, and local information are concatenated to form a vector containing local information. This vector serves as the input to the GRU. The aforementioned local information includes disparity values ​​and color information of neighboring pixels.

[0112] Inside the multi-level convolutional gated recurrent unit, the Reset Gate and Update Gate are updated based on the local relevant information of the current input and the historical hidden layer features, respectively, to determine the information retention and update strategy.

[0113] The multi-level convolutional gated recurrent unit described above in this invention may further include multi-level convolutional layers, each consisting of three convolutional layers with batch normalization and ReLU activation functions. These convolutional layers are used to extract feature information from the input image, i.e., the aforementioned local related information. Each convolutional layer progressively abstracts and refines the information in the image, enabling higher-level convolutional layers to capture more complex features.

[0114] The reset gate described above contains a single convolutional layer for extracting reset coefficients. These reset coefficients weaken historical memories, generating candidate memories. When calculating candidate memories, the reset gate uses features extracted from previous convolutional layers and the memory state from the previous time step. By multiplying the reset coefficients by the memory state from the previous time step, historical memories are filtered and weakened. The reset gate takes the input vector and historical hidden layer states as input, passing them through a sigmoid activation function to obtain reset gate features. The update gate also contains a single convolutional layer for extracting update coefficients. These update coefficients combine candidate memories with new features to generate new memories. When calculating new memories, the update gate uses features extracted from previous convolutional layers, candidate memories, and the memory state from the previous time step. By combining the update coefficients with candidate memories and new features, memories are updated and merged. The update gate controls how much of the hidden layer state from the previous time step should be retained and how much of the candidate hidden layer state should be introduced. The update gate takes the input vector and historical hidden layer states as input and obtains update gate features through a sigmoid activation function. In a single iteration, multi-level convolutional layers first extract feature information from the input image. This feature information is then passed to the reset and update gates. The reset gate uses the feature information and the memory state from the previous time step to calculate reset coefficients and generate candidate memories. The update gate uses the feature information, candidate memories, and the memory state from the previous time step to calculate update coefficients and generate the latest memory. The latest memory is then used to predict the disparity update amount in a single iteration.

[0115] S10532. Combine reset gate features, local relevant information and historical hidden layer features to obtain candidate hidden layer features.

[0116] Historical hidden layer features are obtained through convolution or other transformations based on the previous reset gate features and the previous input vector (i.e., the previous local relevant information). After obtaining the historical hidden layer features, they are combined with the reset gate features (i.e., the output value of the reset gate, which determines the degree of retention of historical information) and the input vector (i.e., the aforementioned local relevant information) through convolution or other transformations to obtain candidate hidden layer features. Candidate hidden layer features represent possible hidden layer state updates under the combined influence of the current input and historical information. Candidate hidden layer features combine reset gate features, local relevant information, and historical hidden layer features, representing the ideal state after considering new information. Specifically, the hidden layer first multiplies the historical hidden layer state with the reset gate features, and then combines it with local relevant information through a linear transformation and a tanh activation function to obtain candidate hidden layer features.

[0117] S10533. Calculate the updated hidden layer features by combining the candidate hidden layer features, the update gate features, and the historical hidden layer features.

[0118] By combining candidate hidden layer features, reset gate features, and historical hidden layer features, and through weighted summation or other combinations, the updated hidden layer features are calculated. This step updates the hidden layer state, ensuring that the updated hidden layer features incorporate both historical information and new local information from the current input. For example, the updated hidden layer features can be obtained by weighted averaging of historical and candidate hidden layer states, with the weights determined by the update gate features. The weight of the historical hidden layer state is 1 - the update gate feature, and the weight of the candidate hidden layer state is the update gate feature.

[0119] S10534. Calculate the disparity update value of the current pixel based on the updated hidden layer features.

[0120] Based on the updated hidden layer features, the disparity update value for the current pixel is calculated through linear transformation or other prediction models. This update value reflects the adjustment to the disparity prediction value in the current iteration step and is used for subsequent updates to the disparity prediction value. This invention introduces reset and update gate mechanisms, enabling multi-level convolutional gated recurrent units to more flexibly handle long-term dependencies in sequential data while effectively preserving and updating historical information. This allows the model to more accurately capture spatial structure and depth information in images during disparity prediction tasks, thereby improving the accuracy of disparity prediction. The introduction of reset and update gates allows the model to more flexibly adjust information flow when facing complex situations such as noise and occlusion, reducing the impact of mismatches and noise. This enhances the robustness of the algorithm, enabling it to maintain good performance under different lighting, occlusion, and noise conditions.

[0121] In an optional embodiment, the method of the present invention can quickly predict the disparity map based on binocular images captured by an underwater binocular camera, thereby obtaining the scene depth. Step S106 above, which involves calculating the scene depth map and 3D point cloud of the underwater environment based on intrinsic and extrinsic parameters and the final disparity map, includes:

[0122] S1061. Using intrinsic and extrinsic parameters, convert the disparity value of each pixel in the disparity map into the corresponding depth value to generate a scene depth map.

[0123] The aforementioned intrinsic parameters mainly include focal length (fx, fy, corresponding to the focal lengths along the x and y axes, respectively), principal point coordinates (x0, y0, i.e., the coordinates of the optical center relative to the imaging plane), and distortion coefficients. The aforementioned extrinsic parameters describe the camera's position and orientation in the world coordinate system, such as the baseline distance of a stereo camera. Based on the stereo matching result obtained in step S105, i.e., the disparity map, the depth of each pixel is calculated using triangulation.

[0124] Specifically, using the camera's intrinsic and extrinsic parameters, especially the focal length and principal point coordinates, as well as the baseline distance between the two cameras (i.e., the distance between the optical centers of the two cameras), the disparity value of each pixel in the disparity map can be converted into a corresponding depth value. The depth value represents the distance between the scene point corresponding to that pixel and the camera, thus generating a depth map of the entire scene. The formula for converting disparity to depth is:

[0125] depth = (f * baseline) / disp

[0126] Where depth is the depth value, f is the focal length, baseline is the baseline distance, and disp is the parallax value. This transformation generates a scene depth map, where each pixel corresponds to a depth value representing its distance from the camera.

[0127] S1062. Combine the internal and external parameters to back-project the scene depth map to obtain the three-dimensional point cloud of the underwater environment.

[0128] After obtaining the scene depth map, and combining it with the camera's intrinsic and extrinsic parameters, the depth map can be back-projected, projecting the pixels on the 2D image into 3D space. This process maps each pixel on the 2D image to coordinates in the real 3D space, thus obtaining a 3D point cloud of the underwater environment. This point cloud structure contains the 3D position information of every visible point in the scene, providing a complete scene geometry for subsequent applications such as environmental perception or 3D reconstruction. The process of back-projecting the depth map from 2D space to 3D space can be referred to in existing technologies and will not be elaborated here.

[0129] This invention utilizes an underwater robot equipped with a binocular camera to measure scene structure and depth information in underwater environments. By combining the camera's intrinsic and extrinsic parameters with a disparity map to calculate 3D point clouds, the ranging capability of the binocular camera can be fully utilized to improve the accuracy of 3D reconstruction. Especially in underwater environments, due to complex lighting conditions and significant noise interference, traditional 3D reconstruction methods often struggle to achieve high-precision results. However, the method based on extrinsic and extrinsic parameters and a disparity map can more accurately capture spatial structure and depth information in images, thereby generating more detailed and accurate 3D point clouds. This method can adapt to different lighting conditions and underwater environmental noise interference. Because it utilizes the camera's extrinsic and extrinsic parameters for precise coordinate transformation and depth calculation, it maintains good performance even when encountering problems such as lighting changes and occlusion in underwater environments. This enhances the robustness of the algorithm, enabling reliable 3D reconstruction results in various scenarios.

[0130] The generated 3D point cloud can provide reliable foundational data for subsequent tasks such as underwater environment perception, object recognition, and topographic surveying. These subsequent tasks typically require high-precision 3D information as input, and the method based on intrinsic and extrinsic parameters and disparity maps perfectly meets this requirement. Therefore, this method plays an important role in promoting the development of underwater robotics, marine engineering, underwater archaeology, and other fields.

[0131] The underwater robot environment perception device based on feature propagator provided by the present invention is described below. The underwater robot environment perception device based on feature propagator described below can be referred to in correspondence with the underwater robot environment perception method based on feature propagator described above.

[0132] The underwater robot environment perception device based on feature propagator provided by this invention refers to... Figure 4 As shown, it includes the following modules:

[0133] The image acquisition module 201 is used to acquire binocular images collected by the underwater robot and a pre-established deep neural network; wherein, the deep neural network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer;

[0134] Feature extraction module 202 is used to correct the binocular images, and input the corrected binocular images into the feature extractor to obtain feature maps at different scales and form a feature pyramid.

[0135] Cost aggregation module 203 is used to construct cost bodies, aggregate cost bodies, and regress disparity based on each of the feature pyramids using a cost aggregator, so as to obtain the original cost body, aggregated cost body, and initial disparity, respectively.

[0136] Initialization module 204 is used to initialize the multi-level convolutional gated recurrent unit in the iterative optimizer by the feature propagator based on the original cost body and the aggregated cost body;

[0137] The disparity prediction module 205 is used to optimize disparity prediction based on a multi-level convolutional gated recurrent unit initialized by the feature propagator until the disparity prediction value reaches the convergence condition and the final disparity map is obtained.

[0138] The calculation module 206 is used to acquire the intrinsic and extrinsic parameters of the binocular camera of the underwater robot, and calculate the scene depth map and 3D point cloud of the underwater environment based on the intrinsic and extrinsic parameters and the final disparity map.

[0139] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions from the memory 330 to execute an underwater robot environmental perception method based on a feature propagator.

[0140] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the underwater robot environmental perception method based on the feature propagator provided by the above methods.

[0142] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the underwater robot environmental perception method based on the feature propagator provided by the above methods.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for environmental perception of underwater robots based on feature propagators, characterized in that, include: The system acquires binocular images collected by an underwater robot and a pre-established deep neural network; wherein the deep neural network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer. The binocular images are corrected, and the corrected binocular images are input into the feature extractor to obtain feature maps at different scales and form a feature pyramid. The cost aggregator is used to construct the cost body, aggregate the cost body, and regress the disparity based on each of the feature pyramids, to obtain the original cost body, the aggregated cost body, and the initial disparity, respectively. The feature propagator initializes the multi-level convolutional gated recurrent units in the iterative optimizer based on the original cost volume and the aggregated cost volume. The feature propagator includes multiple parallel residual modules, and the binocular image includes a left-eye image and a right-eye image. The initialization of the multi-level convolutional gated recurrent units in the iterative optimizer based on the original cost volume and the aggregated cost volume by the feature propagator includes: for the top-level recurrent unit in the multi-level convolutional gated recurrent unit, concatenating the feature map of the left-eye image at the largest scale, the original cost volume, and the aggregated cost volume, and inputting them into the residual module of the top level to extract aggregated features, and initializing the hidden layer of the top-level recurrent unit with the aggregated features; and for other recurrent units in the multi-level convolutional gated recurrent unit except the top level, inputting the feature map of the left-eye image at the corresponding scale into the residual module of the corresponding layer to extract image context features, and initializing the hidden layer of the corresponding recurrent unit with the image context features. Based on the multi-level convolutional gated recurrent units initialized by the feature propagator and the initial disparity optimization disparity prediction, the final disparity map is obtained until the disparity prediction value reaches the convergence condition. Acquire the intrinsic and extrinsic parameters of the underwater robot's binocular camera, and calculate the scene depth map and 3D point cloud of the underwater environment based on the intrinsic and extrinsic parameters and the final disparity map.

2. The underwater robot environment perception method based on feature propagator according to claim 1, characterized in that, The cost aggregator includes a cost volume construction network, a cost aggregation network, and a disparity regressor; The process of using a cost aggregator to construct cost volumes, aggregate cost volumes, and perform disparity regression based on each of the feature pyramids yields the original cost volume, the aggregated cost volume, and the initial disparity, respectively, including: The cost body construction network constructs the cost body based on the feature map at the largest scale in each of the feature pyramids, thus obtaining the original cost body. The cost aggregation network filters and concentrates the local cost information in the original cost volume to obtain the aggregated cost volume. The disparity regressor obtains the initial disparity by performing initial disparity estimation based on the aggregated cost volume.

3. The underwater robot environment perception method based on feature propagator according to claim 1, characterized in that, The binocular image includes the left eye image; the disparity prediction based on the multi-level convolutional gated recurrent unit initialized by the feature propagator and the initial disparity optimization until the disparity prediction value reaches the convergence condition, to obtain the final disparity map, includes: A correlation volume is constructed based on the feature maps of the left and right eyes at the maximum scale. Using the current disparity map as an index, local information related to the current pixel is extracted from the relevant volume, and the current disparity map, the feature map of the left eye image at the maximum scale, and the local information are input into a multi-level convolutional gated recurrent unit initialized by the feature propagator. The multi-level convolutional gated recurrent unit generates the disparity update value of the current pixel based on the current disparity map, the feature map of the left eye image at the maximum scale, and local information; For each current pixel, the current disparity is superimposed with the corresponding disparity update value to obtain the disparity prediction value; where the current disparity in the first iteration is the initial disparity; The process of iteratively predicting disparity continues until the predicted disparity values ​​converge, resulting in the final disparity map.

4. The underwater robot environment perception method based on feature propagator according to claim 3, characterized in that, The multi-level convolutional gated loop unit includes a reset gate and an update gate; The multi-level convolutional gated recurrent unit generates the disparity update value for the current pixel based on the current disparity map, the feature map of the left eye image at the maximum scale, and local information, including: The current disparity map, the feature map of the left eye image at the maximum scale, and local information are concatenated to obtain local related information, which is then input into the reset gate and the update gate. The reset gate and the update gate are updated based on the local related information and the stored historical hidden layer features, respectively, to obtain the reset gate features and the update gate features. Candidate hidden layer features are obtained by combining reset gate features, local relevant information, and historical hidden layer features; The updated hidden layer features are calculated by combining candidate hidden layer features, update gate features, and historical hidden layer features; The disparity update value of the current pixel is calculated based on the updated hidden layer features.

5. The underwater robot environment perception method based on feature propagator according to claim 1, characterized in that, The calculation of the underwater environment scene depth map and 3D point cloud based on intrinsic and extrinsic parameters and the final disparity map includes: Using intrinsic and extrinsic parameters, the disparity value of each pixel in the disparity map is converted into a corresponding depth value to generate a scene depth map; By combining intrinsic and extrinsic parameters, the depth map of the scene is back-projected to obtain a three-dimensional point cloud of the underwater environment.

6. An underwater robot environmental perception device based on a feature propagator, characterized in that, include: An image acquisition module is used to acquire binocular images collected by an underwater robot and a pre-established deep neural network; wherein, the deep neural network includes a feature extractor, a cost aggregator, a feature propagator, and an iterative optimizer; The feature extraction module is used to correct the binocular images, and input the corrected binocular images into the feature extractor to obtain feature maps at different scales and form a feature pyramid. The cost aggregation module is used to construct cost bodies, aggregate cost bodies, and regress disparity based on each of the feature pyramids using a cost aggregator, to obtain the original cost body, aggregated cost body, and initial disparity, respectively. An initialization module is used to initialize the multi-level convolutional gated recurrent units in the iterative optimizer based on the original cost body and the aggregated cost body by the feature propagator; the feature propagator includes multiple parallel residual modules, and the binocular image includes a left-eye image and a right-eye image; the initialization of the multi-level convolutional gated recurrent units in the iterative optimizer based on the original cost body and the aggregated cost body by the feature propagator includes: for the top-level recurrent unit in the multi-level convolutional gated recurrent unit, concatenating the feature map of the left-eye image at the largest scale, the original cost body, and the aggregated cost body, and inputting it into the residual module of the top level to extract aggregated features, and initializing the hidden layer of the top-level recurrent unit with the aggregated features; and for other recurrent units in the multi-level convolutional gated recurrent unit except the top level, inputting the feature map of the left-eye image at the corresponding scale into the residual module of the corresponding layer to extract image context features, and initializing the hidden layer of the corresponding recurrent unit with the image context features; The disparity prediction module is used to optimize disparity prediction based on multi-level convolutional gated recurrent units initialized by the feature propagator and the initial disparity until the disparity prediction value reaches the convergence condition, and obtain the final disparity map. The calculation module is used to acquire the intrinsic and extrinsic parameters of the underwater robot's binocular camera, and calculate the scene depth map and 3D point cloud of the underwater environment based on the intrinsic and extrinsic parameters and the final disparity map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the underwater robot environment perception method based on a feature propagator as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the underwater robot environment perception method based on the feature propagator as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the underwater robot environment perception method based on the feature propagator as described in any one of claims 1 to 5.

Images