Binocular Stereo Image Disparity Image Acquisition Method, Device, Equipment and Medium

Abstract

An embodiment of this specification discloses a method for obtaining a binocular stereo image disparity image. The method includes: obtaining a binocular stereo image to be processed and lidar point cloud; performing feature extraction on the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image; generating a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image; projecting the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image; extracting a specified side image in the binocular stereo image to determine an affinity propagation map and context features; generating a disparity map according to the affinity propagation map and the lidar disparity image; finding corresponding relevant features in the feature correlation pyramid according to the disparity map; inputting the relevant features and context features into a GRU update module to obtain a binocular stereo disparity image in an iterative manner.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and particularly to a method, apparatus, device, and medium for obtaining a disparity image of a binocular stereo image. Background Art

[0002] Binocular Stereo Vision is an important form of machine vision. It is a method based on the principle of disparity and uses imaging devices to obtain two images of the object to be measured from different positions. By calculating the position deviation between corresponding points in the images, the three-dimensional geometric information of the object can be obtained. Binocular stereo vision fuses the images obtained by two eyes and observes the differences between them, which can obtain an obvious sense of depth, establish the corresponding relationship between features, and correspond the image points of the same physical point in space in different images. This difference is called the Disparity image.

[0003] In the prior art, during the disparity matching of binocular stereo images, it will be interfered by the external environment. Especially in the outdoor environment, factors such as sunlight will interfere with the disparity matching of binocular stereo images, and then the finally generated disparity image may not be accurate, affecting the final result. Summary of the Invention

[0004] One or more embodiments of this specification provide a method, apparatus, device, and medium for obtaining a disparity image of a binocular stereo image, which are used to solve the following technical problems:

[0005] During the disparity matching of binocular stereo images, it will be interfered by the external environment. Especially in the outdoor environment, factors such as sunlight will interfere with the disparity matching of binocular stereo images, and then the finally generated disparity image may not be accurate, affecting the final result.

[0006] One or more embodiments of this specification adopt the following technical solutions:

[0007] One or more embodiments of this specification provide a method for obtaining a disparity image of a binocular stereo image, and the method includes:

[0008] Obtain the binocular stereo image to be processed and the lidar point cloud, where the binocular stereo image includes a left image and a right image captured by a left camera and a right camera;

[0009] Extract features from the binocular stereo image to determine the feature correlation map corresponding to the left image and the right image in the binocular stereo image;

[0010] Generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image;

[0011] Project the lidar point cloud onto the specified side image in the binocular stereo image to generate an associated lidar disparity image;

[0012] Extract the specified side image in the binocular stereo image to determine the affinity propagation map and context features;

[0013] Generate a disparity map based on the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature-related pyramid according to the disparity map;

[0014] Input the relevant features and the context features into the GRU update module to obtain the binocular stereo disparity image in an iterative manner.

[0015] Further, the feature extraction of the binocular stereo image to determine the feature correlation map corresponding to the left image and the right image in the binocular stereo image specifically includes:

[0016] Input the binocular stereo image into the feature extraction network;

[0017] Extract the feature correlation map corresponding to the left image and the right image in the binocular stereo image through the feature extraction network; wherein, the feature extraction network consists of a residual module and a downsampling layer.

[0018] Further, the extraction of the specified side image in the binocular stereo image to determine the affinity propagation map and context features specifically includes:

[0019] Input the specified side image in the binocular stereo image into the context extraction network;

[0020] Extract the affinity propagation map and context features of the specified side image in the binocular stereo image through the context extraction network.

[0021] Further, multiple GRU update modules are provided;

[0022] The input of the relevant features and the context features into the GRU update module to obtain the binocular stereo disparity image in an iterative manner specifically includes:

[0023] Input the relevant features and the context features into the GRU update module, and obtain the binocular stereo disparity image through iterative calculation of multiple GRU update modules.

[0024] Further, the input of the relevant features and the context features into the GRU update module and obtaining the binocular stereo disparity image through iterative calculation of multiple GRU update modules specifically includes:

[0025] Take the features of the image on the specified side as the first hidden feature, and input the first hidden feature, the relevant features, and the context features into the GRU update module to output the updated hidden feature and the disparity change amount. Add the disparity change amount to the disparity map, and then perform disparity propagation on the added disparity using the affinity propagation map and the lidar disparity map to obtain the iterated disparity map. Perform iterative processing according to the remaining GRU update modules to obtain the binocular stereo disparity image.

[0026] Further, the loss function of the binocular stereo image disparity matching model is one or more of a sparse disparity loss function, a left-right consistency loss function, and a smooth loss function.

[0027] Further, the resolution of the feature correlation map is 1 / 4 or 1 / 8 of the binocular stereo image.

[0028] One or more embodiments of this specification provide a device for obtaining a binocular stereo image disparity image, and the device includes:

[0029] An acquisition unit, configured to acquire a binocular stereo image to be processed and lidar point cloud, wherein the binocular stereo image includes a left image and a right image captured by left and right cameras;

[0030] An extraction unit, configured to perform feature extraction on the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image;

[0031] A first generation unit, configured to generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image;

[0032] A second generation unit, configured to project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image;

[0033] A determination unit, configured to extract a specified side image in the binocular stereo image to determine an affinity propagation map and context features;

[0034] A third generation unit, configured to generate a disparity map according to the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map;

[0035] An acquisition unit, configured to input the relevant features and the context features into a GRU update module to obtain a binocular stereo disparity image in an iterative manner.

[0036] One or more embodiments of this specification provide a device for obtaining a binocular stereo image disparity image, including:

[0037] At least one processor; and,

[0038] A memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to:

[0040] Obtain a binocular stereo image and a lidar point cloud to be processed, wherein the binocular stereo image includes a left image and a right image captured by left and right cameras;

[0041] Extract features from the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image;

[0042] Generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image;

[0043] Project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image;

[0044] Extract a specified side image in the binocular stereo image to determine an affinity propagation map and context features;

[0045] Generate a disparity map according to the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map;

[0046] Input the relevant features and the context features into a GRU update module to obtain a binocular stereo disparity image in an iterative manner.

[0047] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are configured to:

[0048] Obtain a binocular stereo image and a lidar point cloud to be processed, wherein the binocular stereo image includes a left image and a right image captured by left and right cameras;

[0049] Extract features from the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image;

[0050] Generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image;

[0051] Project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image;

[0052] Extract a specified side image in the binocular stereo image to determine an affinity propagation map and context features;

[0053] Generate a disparity map based on the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map;

[0054] Input the relevant features and the context features into the GRU update module to obtain a binocular stereo disparity image in an iterative manner.

[0055] The above at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0056] The embodiments of this specification simultaneously obtain a binocular stereo image to be processed and a lidar point cloud, and perform feature extraction on the binocular stereo image to obtain a feature correlation map of the left image and the right image, so as to subsequently generate a correlation pyramid through the feature correlation map; at the same time, project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image, then extract a specified side image in the binocular stereo image to determine an affinity propagation map and context features, and then generate a disparity map based on the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map, and finally input the relevant features and the context features into the GRU update module to obtain a binocular stereo disparity image in an iterative manner. The disparity image obtained in the above manner combines the lidar point cloud, which can weaken the interference of the external environment and make the binocular stereo image disparity image more accurate. Description of the Drawings

[0057] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0058] Figure 1 It is a schematic flowchart of a method for obtaining a binocular stereo image disparity image provided by one or more embodiments of this specification;

[0059] Figure 2 It is a schematic structural diagram of a binocular stereo image disparity matching model provided by one or more embodiments of this specification;

[0060] Figure 3 A structural schematic diagram of a binocular stereo image disparity image acquisition device is provided for one or more embodiments of this specification;

[0061] Figure 4 A structural schematic diagram of a binocular stereo image disparity image acquisition device is provided for one or more embodiments of this specification. Specific implementation manners

[0062] Embodiments of this specification provide a method, device, equipment, and medium for acquiring a binocular stereo image disparity image.

[0063] When performing disparity matching on binocular stereo images, it will be interfered by the external environment. Especially in outdoor environments, external factors such as sunlight will interfere with the disparity matching of binocular stereo images, and thus the finally generated disparity image may not be accurate, affecting the final result.

[0064] To enable those skilled in the art to better understand the technical solutions in this specification, the following will clearly and completely describe the technical solutions in the embodiments of this specification with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments of this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0065] Figure 1 A flowchart of a method for acquiring a binocular stereo image disparity image provided for one or more embodiments of this specification. This method flow can be applied to a binocular stereo image disparity matching system, which can be combined with the application of binocular stereo image disparity, enabling the binocular stereo image disparity matching model constructed by this system to better generate the disparity image of binocular stereo images. Some input parameters or intermediate results in the flow allow manual intervention and adjustment to help improve accuracy.

[0066] The method flow steps of the embodiments of this specification are as follows:

[0067] S102. Obtain the binocular stereo image to be processed and the lidar point cloud, where the binocular stereo image includes a left image and a right image captured by left and right cameras respectively.

[0068] In the embodiments of this specification, the lidar image and the binocular stereo image are captured in the same scene, at the same time, and at the same angle.

[0069] Due to the limitations of the existing binocular stereo images in acquiring data, especially in outdoor environments, most of them are indoor synthetic datasets for training supervised networks, which greatly limits the generalization ability of the network. However, lidar point clouds can minimize the influence of the environment as much as possible. Therefore, self-supervised training using binocular stereo images and lidar point clouds can well solve the above problems.

[0070] S104. Extract features from the binocular stereo image to determine the feature correlation map corresponding to the left image and the right image in the binocular stereo image.

[0071] For computational considerations, the resolution of the feature correlation map is set to 1 / 4 or 1 / 8 of the original binocular stereo image to improve the computational speed.

[0072] Extracting features from the binocular stereo image to determine the feature correlation map corresponding to the left image and the right image in the binocular stereo image specifically includes:

[0073] Input the binocular stereo image into the feature extraction network;

[0074] Through the feature extraction network, extract the feature correlation map corresponding to the left image and the right image in the binocular stereo image; wherein, the feature extraction network is composed of a residual module and a downsampling layer.

[0075] It should be noted that for efficiency improvement, the calculation of the feature correlation map can be implemented by GPU matrix multiplication.

[0076] S106. Generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image.

[0077] S108. Project the lidar point cloud onto the specified side image in the binocular stereo image to generate an associated lidar disparity image.

[0078] S110. Extract the specified side image in the binocular stereo image to determine the affinity propagation map and context features.

[0079] Extracting the specified side image in the binocular stereo image to determine the affinity propagation map and context features specifically includes:

[0080] Input the specified side image in the binocular stereo image into the context extraction network;

[0081] Through the context extraction network, extract the affinity propagation map and context features of the specified side image in the binocular stereo image.

[0082] S112. Generate a disparity map based on the affinity propagation map and the lidar disparity image, and find corresponding correlation features in the feature correlation pyramid according to the disparity map.

[0083] S114. Input the correlation features and the context features into the GRU update module to obtain a binocular stereo disparity image in an iterative manner.

[0084] In the embodiments of this specification, multiple GRU update modules can be provided.

[0085] The step of inputting the correlation features and the context features into the GRU update module to obtain a binocular stereo disparity image in an iterative manner specifically includes:

[0086] Input the correlation features and the context features into the GRU update module, and obtain a binocular stereo disparity image through iterative calculations of multiple GRU update modules.

[0087] The step of inputting the correlation features and the context features into the GRU update module, and obtaining a binocular stereo disparity image through iterative calculations of multiple GRU update modules specifically includes:

[0088] Take the features of the image on a specified side as the first hidden feature, input the first hidden feature, the correlation features and the context features into the GRU update module, output the updated hidden feature and the disparity change amount, add the disparity change amount to the disparity map, and then perform disparity propagation on the added disparity using the affinity propagation map and the lidar disparity map to obtain the iterated disparity map, and perform iterative processing according to the remaining GRU update modules to obtain a binocular stereo disparity image.

[0089] Construction of the correlation pyramid: Similar to the construction of the 4D correlation volume in RAFT, in the embodiments of this specification, in binocular stereo image matching, a 3D correlation volume is constructed by calculating the dot product of the feature maps of the left and right images, and the last dimension of the 3D correlation volume is downsampled through a pooling layer to implement a correlation pyramid containing different scales. The correlation volume at each level has a different receptive field, but the correlation volume with the original image resolution is still retained to complete the construction of the correlation pyramid.

[0090] The embodiments of this specification can also define a search operator. Given the currently estimated disparity, the cost element at the corresponding pixel position can be searched backward in the correlation volume. A 1D grid is constructed in the correlation volume at each level to define the search range, and then the 1D grid elements at different levels are spliced to form a single feature map.

[0091] The loss function of the binocular stereo image disparity matching model is a sparse disparity loss function, a left-right consistency loss function, or a smooth loss function.

[0092] Among them, when the sparse disparity loss function is selected, it can not only help obtain dense disparity but also supervise the disparity estimation. When the left-right consistency loss function is selected, it is to ensure that the estimated left disparity is consistent with the right disparity. If only appearance and sparse supervision are used for training, it may lead to inaccurate and non-smooth estimated disparity. When the smooth loss function is introduced, the above problems can be alleviated.

[0093] The embodiments of this specification can train the binocular stereo image disparity matching model through the synthetic dataset SceneFlow. SceneFlow is a synthetic image dataset that includes multiple binocular stereo images.

[0094] The embodiments of this specification can solve the binocular stereo matching problem based on the optical flow estimation network RAFT. The basic idea is still the process of constructing the correlation volume and the multi-level convolutional GRU iterative optimization of RAFT, which can well propagate global information on the image.

[0095] GRU (Gate Recurrent Unit) is a type of recurrent neural network (RNN). Like LSTM (Long-Short Term Memory), it is also proposed to solve problems such as long-term memory and gradients in backpropagation.

[0096] Figure 2 It is a schematic structural diagram of the binocular stereo image disparity matching model provided by the embodiments of this specification. On the left are the right and left images of the binocular stereo image to be processed, as well as the lidar disparity map. After the binocular stereo image to be processed passes through the feature extraction network, it generates a feature correlation pyramid through the feature correlation module (C, Correlation). After the left image of the binocular stereo image passes through the context extraction network, an affinity propagation map and context features are generated. The lidar disparity map d lAs the initial disparity, the initial disparity is propagated (P, Propagation) in combination with the affinity propagation map to obtain the disparity map d0 before the first iteration. Among them, during the propagation, it passes through the disparity propagation module. Next, the GRU update module will iteratively update the disparity map d. In the t-th iteration, the following steps will be executed: Look up (L, Lookup) the corresponding correlation features at the disparity position of d t-1 in the feature correlation pyramid. Among them, during the lookup, it passes through the associated feature query module and is input into the GRU. At the same time, the context features output by the context extraction and the hidden features of the previous GRU iteration are also input into the GRU. (In the first iteration of the GRU, the features output by the feature extraction network are used as the hidden features). The GRU outputs the updated hidden features and outputs the required disparity change amount Δ. The output disparity change amount Δ is added to the disparity map d of the previous iteration t-1 by addition (+, Addition). Then, the affinity propagation map and the lidar disparity map d l are used to perform disparity propagation (P, Propagation) on the added disparity, that is, to obtain the disparity map d of this iteration t . Among them, the disparity propagation (P, Propagation) is also carried out iteratively: one iteration is divided into two steps. First, the disparity map is propagated once using the affinity propagation map, and then the disparity at the valid lidar data position is updated to the lidar disparity; after M propagations, the result of the disparity map propagation can be obtained. After N GRU iterative updates of the disparity map, the output disparity is upsampled (U, Upsampling) to the original image resolution from the low-resolution image, which is the rightmost image. Among them, during the upsampling, it passes through the upsampling module.

[0097] In the embodiments of this specification, the binocular stereo image and the lidar point cloud to be processed are obtained at the same time, and the binocular stereo image is subjected to feature extraction to obtain the feature correlation map of the left image and the right image, so as to generate a correlation pyramid through the feature correlation map in the subsequent process; at the same time, the lidar point cloud is projected onto the specified side image in the binocular stereo image to generate an associated lidar disparity image. Subsequently, the specified side image in the binocular stereo image is extracted to determine the affinity propagation map and the context features. Then, a disparity map is generated according to the affinity propagation map and the lidar disparity image, and the corresponding correlation features are found in the feature correlation pyramid according to the disparity map. Finally, the correlation features and the context features are input into the GRU update module to obtain the binocular stereo disparity image in an iterative manner. The disparity image obtained through the above method combines the lidar point cloud, which can weaken the interference of the external environment and make the binocular stereo image disparity image more accurate.

[0098] Figure 3A structural schematic diagram of a binocular stereo image disparity image acquisition device is provided for one or more embodiments of this specification. The device includes: an acquisition unit 302, an extraction unit 304, a first generation unit 306, a second generation unit 308, a determination unit 310, a third generation unit 312, and an acquisition unit 314.

[0099] The acquisition unit 302 is configured to acquire a binocular stereo image to be processed and lidar point cloud, wherein the binocular stereo image includes a left image and a right image captured by left and right cameras;

[0100] The extraction unit 304 is configured to perform feature extraction on the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image;

[0101] The first generation unit 306 is configured to generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image;

[0102] The second generation unit 308 is configured to project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image;

[0103] The determination unit 310 is configured to extract a specified side image in the binocular stereo image to determine an affinity propagation map and context features;

[0104] The third generation unit 312 is configured to generate a disparity map according to the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map;

[0105] The acquisition unit 314 is configured to input the relevant features and the context features into a GRU update module to obtain a binocular stereo disparity image in an iterative manner.

[0106] Figure 4 A structural schematic diagram of a binocular stereo image disparity image acquisition device is provided for one or more embodiments of this specification, including:

[0107] At least one processor; and,

[0108] A memory communicatively connected to the at least one processor; wherein,

[0109] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0110] Acquire a binocular stereo image to be processed and lidar point cloud, wherein the binocular stereo image includes a left image and a right image captured by left and right cameras;

[0111] Extract features from the binocular stereo image to determine the feature correlation map corresponding to the left image and the right image in the binocular stereo image;

[0112] Generate a feature correlation pyramid based on the feature correlation map corresponding to the left image and the right image;

[0113] Project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image;

[0114] Extract a specified side image in the binocular stereo image to determine the affinity propagation map and context features;

[0115] Generate a disparity map based on the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map;

[0116] Input the relevant features and the context features into the GRU update module to obtain the binocular stereo disparity image in an iterative manner.

[0117] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set as follows:

[0118] Obtain a binocular stereo image and a lidar point cloud to be processed, where the binocular stereo image includes a left image and a right image captured by left and right cameras;

[0119] Extract features from the binocular stereo image to determine the feature correlation map corresponding to the left image and the right image in the binocular stereo image;

[0120] Generate a feature correlation pyramid based on the feature correlation map corresponding to the left image and the right image;

[0121] Project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image;

[0122] Extract a specified side image in the binocular stereo image to determine the affinity propagation map and context features;

[0123] Generate a disparity map based on the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map;

[0124] Input the relevant features and the context features into the GRU update module to obtain the binocular stereo disparity image in an iterative manner.

[0125] In the 1990s, it was obvious to distinguish whether an improvement to a technology was an improvement in hardware (e.g., improvement to the circuit structures such as diodes, transistors, switches, etc.) or an improvement in software (improvement to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Almost all designers obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented with a hardware entity module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. The designer can program by himself to "integrate" a digital system on a PLD without asking the chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating the integrated circuit chip, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called Hardware Description Language (HDL). And there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply making a little logical programming of the method flow with the above-mentioned several hardware description languages and programming it into the integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0126] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0127] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0128] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0129] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, the embodiments of this specification can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0130] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0131] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0133] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0134] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0138] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0139] The specific embodiments of this specification are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] The above description is only for one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various changes and modifications can be made to one or more embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of one or more embodiments of this specification shall be included within the scope of the claims of this specification.

Claims

1. A method for obtaining a binocular stereo image disparity image, characterized in that, The method includes: Obtaining a binocular stereo image and a lidar point cloud to be processed, where the binocular stereo image includes a left image and a right image captured by left and right cameras; Performing feature extraction on the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image; Generating a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image; Projecting the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image; Extracting a specified side image in the binocular stereo image to determine an affinity propagation map and context features; Generating a disparity map according to the affinity propagation map and the lidar disparity image, and searching for corresponding relevant features in the feature correlation pyramid according to the disparity map; Inputting the relevant features and the context features into a GRU update module to obtain a binocular stereo disparity image in an iterative manner; There are multiple GRU update modules, specifically including: Taking the features of a specified side image as the first hidden feature, inputting the first hidden feature, the relevant features and the context features into the GRU update module, outputting an updated hidden feature and a disparity change amount, adding the disparity change amount to the disparity map, and then performing disparity propagation on the added disparity using the affinity propagation map and the lidar disparity map to obtain an iterated disparity map, and performing iterative processing according to the remaining GRU update modules to obtain a binocular stereo disparity image.

2. The method according to claim 1, characterized in that, The performing feature extraction on the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image specifically includes: Inputting the binocular stereo image into a feature extraction network; Extracting, through the feature extraction network, a feature correlation map corresponding to the left image and the right image in the binocular stereo image; where the feature extraction network is composed of a residual module and a downsampling layer.

3. The method according to claim 1, characterized in that, The extracting a specified side image in the binocular stereo image to determine an affinity propagation map and context features specifically includes: Inputting a specified side image in the binocular stereo image into a context extraction network; Extracting, through the context extraction network, the affinity propagation map and context features of a specified side image in the binocular stereo image.

4. The method according to claim 1, characterized in that, The loss function of the binocular stereo image disparity matching model is one or more of a sparse disparity loss function, a left-right consistency loss function, and a smoothness loss function.

5. The method according to claim 1, wherein the resolution of the feature correlation map is 1 / 4 or 1 / 8 of the binocular stereo image.

6. A device for obtaining a binocular stereo image disparity image, characterized in that, The device includes: An acquisition unit for acquiring a binocular stereo image and a lidar point cloud to be processed, where the binocular stereo image includes a left image and a right image captured by left and right cameras; An extraction unit for performing feature extraction on the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image; A first generation unit for generating a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image; A second generation unit, configured to project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image; A determination unit, configured to extract a specified side image in the binocular stereo image to determine an affinity propagation map and context features; A third generation unit, configured to generate a disparity map according to the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map; An acquisition unit, configured to input the relevant features and the context features into a GRU update module to obtain a binocular stereo disparity image in an iterative manner; There are multiple GRU update modules, specifically including: Taking the features of a specified side image as the first hidden feature, and inputting the first hidden feature, the relevant features and the context features into the GRU update module, outputting an updated hidden feature and a disparity change amount, adding the disparity change amount to the disparity map, and then performing disparity propagation on the added disparity using the affinity propagation map and the lidar disparity map to obtain an iterated disparity map, and performing iterative processing according to the remaining GRU update modules to obtain a binocular stereo disparity image.

7. A device for obtaining a binocular stereo image disparity image, characterized in that, Including: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: Obtain a binocular stereo image and a lidar point cloud to be processed, wherein the binocular stereo image includes a left image and a right image captured by left and right cameras; Extract features from the binocular stereo image to determine a feature correlation map corresponding to the left image and the right image in the binocular stereo image; Generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image; Project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image; Extract a specified side image in the binocular stereo image to determine an affinity propagation map and context features; Generate a disparity map according to the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map; Input the relevant features and the context features into a GRU update module to obtain a binocular stereo disparity image in an iterative manner; There are multiple GRU update modules, specifically including: Taking the features of a specified side image as the first hidden feature, and inputting the first hidden feature, the relevant features and the context features into the GRU update module, outputting an updated hidden feature and a disparity change amount, adding the disparity change amount to the disparity map, and then performing disparity propagation on the added disparity using the affinity propagation map and the lidar disparity map to obtain an iterated disparity map, and performing iterative processing according to the remaining GRU update modules to obtain a binocular stereo disparity image.

8. A non - volatile computer storage medium stores computer - executable instructions, characterized in that, The computer-executable instructions are set to: Obtain the binocular stereo image and lidar point cloud to be processed, where the binocular stereo image includes a left image and a right image captured by a left and a right camera respectively; Extract features from the binocular stereo image to determine the feature correlation map corresponding to the left image and the right image in the binocular stereo image; Generate a feature correlation pyramid according to the feature correlation map corresponding to the left image and the right image; Project the lidar point cloud onto a specified side image in the binocular stereo image to generate an associated lidar disparity image; Extract a specified side image in the binocular stereo image to determine the affinity propagation map and context features; Generate a disparity map according to the affinity propagation map and the lidar disparity image, and find corresponding relevant features in the feature correlation pyramid according to the disparity map; Input the relevant features and the context features into the GRU update module to obtain a binocular stereo disparity image in an iterative manner; There are multiple GRU update modules, specifically including: Take the features of a specified side image as the first hidden feature, input the first hidden feature, the relevant features and the context features into the GRU update module, output the updated hidden feature and the disparity change amount, add the disparity change amount to the disparity map, and then perform disparity propagation on the added disparity using the affinity propagation map and the lidar disparity map to obtain the iterated disparity map. Perform iterative processing according to the remaining GRU update modules to obtain the binocular stereo disparity image.

Images