Binocular vision depth estimation method, device, equipment, medium and product

Abstract

The invention discloses a binocular vision depth estimation method and device, equipment, a medium and a product, and relates to the technical field of robot environment perception, and the method comprises the steps: obtaining a binocular image sequence which comprises a plurality of continuous target image pairs; each target image pair comprises a left eye image and a right eye image, inputting the target image pairs into the trained parallax estimation model to obtain a final prediction parallax image of the target image pairs, and calculating depth information according to the final prediction parallax image of the target image pairs and camera parameters of the binocular camera. And when the robot cannot carry a laser radar or a depth camera, depth perception based on pure vision is realized.

Description

Technical Field

[0001] The present application relates to the field of robot environment perception technology, and in particular to a binocular vision depth estimation method, device, equipment, medium and product. Background Art

[0002] Depth perception is of great significance in the field of mobile robots. It can provide three-dimensional information about the environment, which is crucial for robot navigation, obstacle avoidance and task execution. Through depth perception, robots can understand the three-dimensional structure of the environment, so that they can better plan paths, identify and avoid obstacles, and choose the best route to reach the target location. This allows robots to move safely in complex and dynamic environments and avoid collisions. In addition, through depth sensors, robots can create three-dimensional maps of the environment, which is very important for autonomous navigation and task execution. These maps help robots locate and navigate in unknown or changing environments. Robots can also use depth perception technology to autonomously explore unknown environments, collect data and learn the characteristics of the environment, which is particularly important for robots that perform tasks in unknown or dangerous environments. Depth perception technology makes mobile robots more intelligent and autonomous, and can efficiently perform various tasks in complex and dynamic environments, thereby improving the functionality and reliability of robots.

[0003] At present, the depth perception method based on the fusion of laser radar, binocular stereo camera, inertial sensor (IMU) and global positioning system (GPS) has made a breakthrough. Robots such as BigDog, LittleDog and Cheetah are equipped with a rich sensor system (such as laser radar, binocular stereo camera and IMU), which can obtain three-dimensional information of the environment and generate cost maps for path planning and autonomous navigation. The depth perception method based on the fusion of laser radar, binocular stereo camera and IMU has realized functions such as environmental perception and autonomous navigation. At present, domestic and foreign research on robot depth perception has made certain progress, mainly relying on multiple sensors such as laser radar, camera and IMU to obtain the three-dimensional point cloud of the surrounding environment, but the above methods still have some problems: 1) The fusion efficiency of multi-view, multi-platform and multi-source point cloud data in complex scenes is low; 2) The description of depth information is incomplete, making it difficult to achieve real-time response and decision-making of robots; 3) Under the constraints of narrow space, robots cannot be equipped with laser radar or depth camera, so the above depth perception methods are constrained by size and price, large high-end perception equipment is not easy to apply, and the depth perception accuracy based on pure vision is low. Summary of the invention

[0004] The purpose of this application is to provide a binocular vision depth estimation method, device, equipment, medium and product, which can realize depth perception based on pure vision and improve the accuracy of depth information extraction under the constraints of narrow space when the robot cannot be equipped with lidar or depth camera.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a binocular vision depth estimation method, comprising:

[0007] Acquire a binocular image sequence; the binocular image sequence is obtained by photographing the target area by a binocular camera on the robot; the binocular image sequence includes a plurality of continuous target image pairs; each of the target image pairs includes a left-eye image and a right-eye image;

[0008] For each of the target image pairs, inputting the target image pair into a trained disparity estimation model to obtain a final predicted disparity map of the target image pair;

[0009] Wherein, the disparity estimation model includes a feature extraction module, a correlation pyramid and a recursive update module;

[0010] The feature extraction module includes a first encoder, a second encoder and a third encoder;

[0011] The first encoder is used to extract features from the left-eye image to obtain features of the left-eye image;

[0012] The second encoder is used to extract features from the right-eye image to obtain features of the right-eye image;

[0013] The third encoder is used to extract features from the left-eye image to obtain context coding features;

[0014] The related pyramid is used to: calculate the similarity between the left-eye image feature and the right-eye image feature to obtain a cost volume; and generate an initial disparity map according to an initial estimation of the cost volume;

[0015] The recursive updating module is used to: use a recurrent neural network to perform multiple iterative updates according to the current disparity map, the cost volume and the context encoding features to obtain a final predicted disparity map of the target image pair; the current disparity map in the first iteration process is the initial disparity map;

[0016] Depth information is calculated according to the final predicted disparity map of the target image pair and the camera parameters of the binocular camera; the depth information is used to create a three-dimensional scene of the target area.

[0017] Optionally, before inputting the target image pair into a trained disparity estimation model, the binocular vision depth estimation method further includes:

[0018] Acquire a sample set; the sample set includes a plurality of sample image pairs and a real disparity map corresponding to each of the sample image pairs;

[0019] The sample set is used to train a disparity estimation model to obtain a trained disparity estimation model.

[0020] Optionally, using the sample set to train a disparity estimation model to obtain a trained disparity estimation model specifically includes:

[0021] Input the sample image pair into the disparity estimation model to obtain a sample predicted disparity map;

[0022] Preprocessing the sample predicted disparity map to obtain a preprocessed disparity map;

[0023] Calculate the loss function value according to the preprocessed disparity map and the real disparity map;

[0024] The model parameters of the disparity estimation model are updated according to the loss function value to obtain a trained disparity estimation model.

[0025] Optionally, the preprocessing is smoothing and thinning processing.

[0026] Optionally, the camera parameters of the binocular camera include camera focal length and camera baseline.

[0027] Optionally, the depth information is calculated as follows:

[0028]

[0029] Among them, S d is the depth information, J is the camera focal length, G is the camera baseline, S g is the final predicted disparity map.

[0030] In a second aspect, the present application provides a binocular vision depth estimation device, comprising:

[0031] A binocular image sequence acquisition module is used to: acquire a binocular image sequence; the binocular image sequence is obtained by photographing a target area with a binocular camera on the robot; the binocular image sequence includes a plurality of continuous target image pairs; each target image pair includes a left-eye image and a right-eye image;

[0032] A disparity estimation module, used for: for each target image pair, inputting the target image pair into a trained disparity estimation model to obtain a final predicted disparity map of the target image pair; wherein the disparity estimation model includes a feature extraction module, a correlation pyramid and a recursive update module; the feature extraction module includes a first encoder, a second encoder and a third encoder; the first encoder is used for performing feature extraction on the left-eye image to obtain left-eye image features; the second encoder is used for performing feature extraction on the right-eye image to obtain right-eye image features; the third encoder is used for performing feature extraction on the left-eye image to obtain context coding features; the correlation pyramid is used for: performing similarity calculation on the left-eye image features and the right-eye image features to obtain a cost volume; generating an initial disparity map according to an initial estimate of the cost volume; the recursive update module is used for: using a recurrent neural network, performing multiple iterative updates according to the current disparity map, the cost volume and the context coding features to obtain a final predicted disparity map of the target image pair; the current disparity map in the first iteration process is the initial disparity map;

[0033] The depth information calculation module is used to calculate the depth information according to the final predicted disparity map of the target image pair and the camera parameters of the binocular camera; the depth information is used to create a three-dimensional scene of the target area.

[0034] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned binocular vision depth estimation method.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned binocular vision depth estimation method when executed by a processor.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned binocular vision depth estimation methods.

[0037] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0038] The present application provides a binocular vision depth estimation method, apparatus, device, medium and product, which inputs a target image pair into a trained disparity estimation model to obtain a final predicted disparity map of each target image pair, and calculates depth information based on the final predicted disparity map and camera parameters, thereby solving the problem of insufficient depth perception capability of current robots based on pure vision. Under the constraints of narrow space, when the robot cannot be equipped with a lidar or a depth camera, depth perception based on pure vision can be achieved, thereby improving the accuracy of depth information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is a diagram of an application environment of a binocular vision depth estimation method in one embodiment of the present application;

[0041] Figure 2 A schematic diagram of a process flow of a binocular vision depth estimation method provided in one embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of a disparity estimation model provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a training process of a disparity estimation model provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of functional modules of a binocular vision depth estimation device provided in one embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0048] The binocular vision depth estimation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send a binocular image sequence of the target area to the server 104, and the binocular image sequence includes several continuous target image pairs, each target image pair includes a left eye image and a right eye image. After the server 104 receives the target image pair, the target image pair is input into the trained disparity estimation model to obtain the final predicted disparity map of the target image pair, and the depth information is calculated according to the final predicted disparity map of the target image pair and the camera parameters of the binocular camera. The server 104 can feed back the obtained depth information for the target area to the terminal 102. In addition, in some embodiments, the binocular vision depth estimation method can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly perform depth estimation processing on the target image pair, or the server 104 can obtain the target image pair from the data storage system and perform depth estimation processing on the target image pair.

[0049] The terminal 102 may be, but is not limited to, various desktop computers and laptop computers. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0050] In an exemplary embodiment, Figure 2 As shown, a binocular vision depth estimation method is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 203. Among them:

[0051] Step 201: Acquire a binocular image sequence; the binocular image sequence is obtained by photographing a target area by a binocular camera on a robot; the binocular image sequence includes a plurality of continuous target image pairs; each target image pair includes a left-eye image and a right-eye image.

[0052] Step 202: For each of the target image pairs, input the target image pair into a trained disparity estimation model to obtain a final predicted disparity map of the target image pair.

[0053] Among them, Figure 3As shown, the disparity estimation model includes a feature extraction module, a correlation pyramid and a recursive update module.

[0054] The feature extraction module includes a first encoder, a second encoder and a third encoder; the first encoder is used to extract features from the left image to obtain left image features; the second encoder is used to extract features from the right image to obtain right image features; the third encoder is used to extract features from the left image to obtain context coding features;

[0055] The related pyramid is used to: calculate the similarity between the left-eye image feature and the right-eye image feature to obtain a cost volume; and generate an initial disparity map according to an initial estimation of the cost volume;

[0056] The recursive update module is used to: use a recurrent neural network to perform multiple iterative updates according to the current disparity map, the cost volume and the context encoding features to obtain a final predicted disparity map of the target image pair; the current disparity map in the first iteration process is the initial disparity map.

[0057] Step 203: Calculate depth information according to the final predicted disparity map of the target image pair and the camera parameters of the binocular camera; the depth information is used to create a three-dimensional scene of the target area.

[0058] Implement the above steps 201 to 203, input the target image pair into the trained disparity estimation model, obtain the final predicted disparity map of each target image pair, calculate the depth information according to the final predicted disparity map and camera parameters, solve the problem of insufficient depth perception ability of robots based on pure vision at present, and realize depth perception based on pure vision under the constraints of narrow space when the robot cannot be equipped with laser radar or depth camera, so as to improve the accuracy of depth information extraction. A pure vision depth perception system suitable for robots equipped with binocular cameras provides support for the practical application of robots in complex environments. Based on the attention mechanism in the Transformer module, a depth estimation network TRAFT for stereo matching of binocular images is proposed. Through the steps of feature extraction, cost volume construction, recursive update, etc., the disparity estimation is gradually refined, and finally a high-precision depth map is generated. By introducing the full-to-full association and recursive update modules, the depth estimation is made more accurate and efficient.

[0059] It is designed for small and low-cost robots whose perception systems only have binocular cameras due to cost and size constraints.

[0060] Before inputting the target image pair into the trained disparity estimation model, the binocular vision depth estimation method further includes the following steps 301 to 302 .

[0061] Step 301: Obtain a sample set; the sample set includes a number of sample image pairs and a real disparity map corresponding to each of the sample image pairs. The sample image pair includes a sample left-eye image and a sample right-eye image.

[0062] Step 302: using the sample set to train a disparity estimation model to obtain a trained disparity estimation model.

[0063] The above step 302 may include the following steps 401 to 404.

[0064] Step 401: input the sample image pair into the disparity estimation model to obtain a sample predicted disparity map;

[0065] Step 402: preprocessing the sample predicted disparity map to obtain a preprocessed disparity map; the preprocessing includes smoothing and thinning processing.

[0066] Step 403: Calculating a loss function value according to the preprocessed disparity map and the real disparity map;

[0067] Step 404: updating the model parameters of the disparity estimation model according to the loss function value to obtain a trained disparity estimation model.

[0068] The training process is as follows Figure 4 As shown, the details are as follows:

[0069] The first step is to obtain the binocular image sequence through the binocular camera, and then process each frame of the image sequence.

[0070] In the second step, the first encoder and the second encoder are used to extract features from the sample left image and the sample right image respectively to obtain left image features and right image features. The first encoder and the second encoder use a feature extraction network with shared weights to ensure that the extracted features are consistent between the two views. As an implementable approach, the first encoder and the second encoder can use a convolutional neural network.

[0071] The first encoder and the second encoder are applied to the left and right images and generate dense feature maps (i.e., left and right image features) for constructing the relevant volume. The third encoder has the same structure as the first and second encoders. The first, second, and third encoders are all composed of 5 convolutional layers. Each convolution operation uses a 3×3 convolution kernel. After each convolution layer, a ReLU activation function layer is connected. The specific configuration of the convolution layer is as follows:

[0072] The first convolutional layer: the input image has 3 channels and the output has 64 channels.

[0073] Second convolutional layer: 64 channels, output 128 channels.

[0074] The third convolutional layer has 128 channels and outputs 256 channels.

[0075] Fourth convolutional layer: 256 channels, output 512 channels.

[0076] Fifth convolutional layer: 512 channels, output 512 channels.

[0077] The third encoder is only applied on the left image, that is, the third encoder only extracts features from the left image. The context encoded features are used to initialize the hidden state of the update operator and are injected into the recurrent neural network in each iteration.

[0078] The third step is feature integration based on the attention mechanism. The extracted features undergo a series of feature enhancement operations to improve the expressiveness of the features. At the same time, position encoding is added to retain the position information in the image, which is a key step in processing sequence data in Transformer. After the position encoding is established, the self-attention mechanism is first used to globally model the features of the left and right eyes respectively. The self-attention mechanism can capture the relationship between any two pixels in the image, so as to better find corresponding point pairs in stereo matching. In the self-attention mechanism, the input features are linearly transformed through three updateable parameters: query (Q), key (K), and value (V), and the similarity between the query and the key is calculated to obtain the attention weight, and then the output features are obtained by weighted summation. In addition, through the cross-view attention mechanism, the features of the left and right images are interacted, and the matching relationship between the left and right images is calculated, so as to establish the matching relationship of the same features between the left and right images.

[0079] The fourth step is to generate a four-dimensional cost volume by calculating the similarity of the features of the left and right images. Each position of the cost volume represents the matching cost of the left and right image features under different disparities.

[0080] Correlation pyramids, used to improve the accuracy and robustness of rectified stereo vision. The correlation volume is based on the dot product between feature vectors and is used to calculate the similarity between the left and right images; the correlation pyramid is constructed by average pooling the last dimension, which has an increased receptive field, but by pooling only the last dimension, high-resolution information in the original image can be retained, allowing very subtle structures to be recovered.

[0081] Step 5: Generate an initial disparity map. Generate an initial disparity map using the initial estimate of the cost volume. This initial disparity map is the starting point for subsequent iterative optimization.

[0082] Step 6, recursive update module: The recursive update module repeatedly refines the disparity estimate through a recurrent neural network. Preferably, Figure 3 As shown, the recurrent neural network can be a GRU model. In each iteration, the recurrent neural network uses the current disparity map, the cost volume, and the context encoding features to update and generate a more accurate disparity map.

[0083] The input of the recursive update module includes: the current disparity map, the cost volume and the context encoding features. Through multiple iterations, the recursive update module gradually refines the disparity map to make it more accurate.

[0084] A series of disparity fields are predicted starting from an initial starting point 0. In each iteration, the cost volume, current disparity map and context features are concatenated and input into the RNN. The RNN updates the hidden state and the new hidden state is used to predict the disparity update.

[0085] In the seventh step, after multiple iterations, the final predicted disparity map is further smoothed and refined to remove noise and discontinuous areas to obtain a preprocessed disparity map.

[0086] In the eighth step, supervised learning is used for training, using the real disparity map as the label. The loss function calculates the difference between the preprocessed disparity map and the real disparity map, and the network parameters are optimized through back propagation to obtain the trained disparity estimation model.

[0087] After obtaining the trained disparity estimation model, a binocular image sequence is obtained by shooting with the binocular camera on the robot, and then each target image pair in the binocular image sequence is processed as follows: the target image pair is input into the trained disparity estimation model to obtain the final predicted disparity map of each target image pair, and the refined final predicted disparity map is used to construct the depth information of the scene.

[0088] The camera parameters of the binocular camera include the camera focal length and the camera baseline. The disparity map represents the horizontal pixel offset of the same scene point in the left and right camera images, while the depth map represents the actual distance from the scene point to the camera. Therefore, the conversion process from the disparity map to the depth map, that is, the calculation formula of the depth information is as follows:

[0089]

[0090] Among them, S d is the depth information, J is the camera focal length, G is the camera baseline, S g is the final predicted disparity map.

[0091] This application uses the Transformer's attention mechanism to solve the stereo matching problem, thereby obtaining a more accurate depth estimation; this application requires fewer hardware resources, has low computing power requirements, and places a light load on the robot's computing platform.

[0092] The present application also provides an application scenario, which applies the above-mentioned binocular vision depth estimation method. Specifically: the binocular vision depth estimation method provided in this embodiment can be applied in a three-dimensional map construction scenario. The three-dimensional map construction scenario includes an image acquisition link, a depth estimation link and a map construction link; several continuous image pairs of the target area enter the depth estimation link from the image acquisition link, obtain corresponding depth information, and enter the downstream map construction link. The binocular vision depth estimation method provided in this embodiment belongs to the depth estimation link. Specifically, in the depth estimation link process for the target area, the target image pair can be input into the trained disparity estimation model to obtain the final predicted disparity map of the target image pair, and the depth information is calculated based on the final predicted disparity map of the target image pair and the camera parameters of the binocular camera.

[0093] Based on the same inventive concept, the embodiment of the present application also provides a binocular vision depth estimation device for implementing the binocular vision depth estimation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more binocular vision depth estimation device embodiments provided below can refer to the limitations of the binocular vision depth estimation method above, and will not be repeated here.

[0094] In an exemplary embodiment, Figure 5 As shown, a binocular vision depth estimation device is provided, comprising:

[0095] The binocular image sequence acquisition module T1 is used to: acquire a binocular image sequence; the binocular image sequence is obtained by photographing the target area by a binocular camera on the robot; the binocular image sequence includes a plurality of continuous target image pairs; each target image pair includes a left-eye image and a right-eye image;

[0096] The disparity estimation module T2 is used to: for each target image pair, input the target image pair into the trained disparity estimation model to obtain the final predicted disparity map of the target image pair; wherein the disparity estimation model includes a feature extraction module, a correlation pyramid and a recursive update module; the feature extraction module includes a first encoder, a second encoder and a third encoder; the first encoder is used to extract features of the left-eye image to obtain left-eye image features; the second encoder is used to extract features of the right-eye image to obtain right-eye image features; the third encoder is used to extract features of the left-eye image to obtain context coding features; the correlation pyramid is used to: calculate the similarity between the left-eye image features and the right-eye image features to obtain a cost volume; generate an initial disparity map according to an initial estimate of the cost volume; the recursive update module is used to: use a recurrent neural network to perform multiple iterative updates according to the current disparity map, the cost volume and the context coding features to obtain the final predicted disparity map of the target image pair; the current disparity map in the first iteration is the initial disparity map;

[0097] The depth information calculation module T3 is used to calculate the depth information according to the final predicted disparity map of the target image pair and the camera parameters of the binocular camera; the depth information is used to create a three-dimensional scene of the target area.

[0098] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store depth estimation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a binocular vision depth estimation method is implemented.

[0099] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0101] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0102] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0103] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0104] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0105] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A binocular vision depth estimation method, characterized in that: The binocular vision depth estimation method comprises: Acquire a binocular image sequence; the binocular image sequence is obtained by photographing the target area by a binocular camera on the robot; the binocular image sequence includes a plurality of continuous target image pairs; each of the target image pairs includes a left-eye image and a right-eye image; For each of the target image pairs, inputting the target image pair into a trained disparity estimation model to obtain a final predicted disparity map of the target image pair; Wherein, the disparity estimation model includes a feature extraction module, a correlation pyramid and a recursive update module; The feature extraction module includes a first encoder, a second encoder and a third encoder; The first encoder is used to extract features from the left-eye image to obtain features of the left-eye image; The second encoder is used to extract features from the right-eye image to obtain features of the right-eye image; The third encoder is used to extract features from the left-eye image to obtain context coding features; The related pyramid is used to: calculate the similarity between the left-eye image feature and the right-eye image feature to obtain a cost volume; and generate an initial disparity map according to an initial estimation of the cost volume; The recursive updating module is used to: use a recurrent neural network to perform multiple iterative updates according to the current disparity map, the cost volume and the context encoding features to obtain a final predicted disparity map of the target image pair; the current disparity map in the first iteration process is the initial disparity map; Depth information is calculated according to the final predicted disparity map of the target image pair and the camera parameters of the binocular camera; the depth information is used to create a three-dimensional scene of the target area.

2. The binocular vision depth estimation method according to claim 1, characterized in that: Before inputting the target image pair into the trained disparity estimation model, the binocular vision depth estimation method further includes: Acquire a sample set; the sample set includes a plurality of sample image pairs and a real disparity map corresponding to each of the sample image pairs; The sample set is used to train a disparity estimation model to obtain a trained disparity estimation model.

3. The binocular vision depth estimation method according to claim 2, characterized in that: The disparity estimation model is trained using the sample set to obtain a trained disparity estimation model, specifically including: Input the sample image pair into the disparity estimation model to obtain a sample predicted disparity map; Preprocessing the sample predicted disparity map to obtain a preprocessed disparity map; Calculate the loss function value according to the preprocessed disparity map and the real disparity map; The model parameters of the disparity estimation model are updated according to the loss function value to obtain a trained disparity estimation model.

4. The binocular vision depth estimation method according to claim 3, characterized in that: The preprocessing includes smoothing and thinning processing.

5. The binocular vision depth estimation method according to claim 1, characterized in that: The camera parameters of the binocular camera include camera focal length and camera baseline.

6. The binocular vision depth estimation method according to claim 5, characterized in that: The calculation formula of depth information is as follows: Among them, S d is the depth information, J is the camera focal length, G is the camera baseline, S g is the final predicted disparity map.

7. A binocular vision depth estimation device, characterized in that: The binocular vision depth estimation device comprises: A binocular image sequence acquisition module is used to: acquire a binocular image sequence; the binocular image sequence is obtained by photographing a target area with a binocular camera on the robot; the binocular image sequence includes a plurality of continuous target image pairs; each target image pair includes a left-eye image and a right-eye image; A disparity estimation module, used for: for each target image pair, inputting the target image pair into a trained disparity estimation model to obtain a final predicted disparity map of the target image pair; wherein the disparity estimation model includes a feature extraction module, a correlation pyramid and a recursive update module; the feature extraction module includes a first encoder, a second encoder and a third encoder; the first encoder is used for performing feature extraction on the left-eye image to obtain left-eye image features; the second encoder is used for performing feature extraction on the right-eye image to obtain right-eye image features; the third encoder is used for performing feature extraction on the left-eye image to obtain context coding features; the correlation pyramid is used for: performing similarity calculation on the left-eye image features and the right-eye image features to obtain a cost volume; generating an initial disparity map according to an initial estimate of the cost volume; the recursive update module is used for: using a recurrent neural network, performing multiple iterative updates according to the current disparity map, the cost volume and the context coding features to obtain a final predicted disparity map of the target image pair; the current disparity map in the first iteration process is the initial disparity map; The depth information calculation module is used to calculate the depth information according to the final predicted disparity map of the target image pair and the camera parameters of the binocular camera; the depth information is used to create a three-dimensional scene of the target area.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the binocular vision depth estimation method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the binocular vision depth estimation method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the binocular vision depth estimation method described in any one of claims 1 to 6 is implemented.

Images