Depth image acquisition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a depth image acquisition method and device, electronic equipment and a storage medium. The method comprises the following steps: carrying out image matching on an obtained first image and an obtained second image shot by a binocular camera to obtain a parallax image; performing pixel remapping on the second image based on the parallax image to obtain a pseudo image corresponding to the second image; calling a pre-trained depth image prediction model to process the first image, the second image and the pseudo image to obtain correlation features between the first image and the second image and between the first image and the pseudo image, and processing the correlation features and the image features of the first image to obtain a depth image, the depth image prediction model is deployed in a deep learning processing unit of a programmable logic system of an SOC architecture. According to the embodiment of the invention, the real-time performance of depth image acquisition can be improved, the computing power demand is reduced, and the system energy consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a depth image acquisition method, device, electronic device and storage medium. Background Art

[0002] There are two types of binocular matching algorithms: local binocular matching algorithm and global binocular matching algorithm. Usually, they include four steps: matching cost, cost aggregation, disparity regression, and disparity refinement. The similarity is evaluated by calculating the matching cost of the left and right views pixel by pixel.

[0003] With the rapid development of convolutional neural network technology and the emergence of large standard stereo data sets, neural network-based binocular matching algorithms such as FADNet (Fast And Detailed Network) have developed rapidly. This type of method trains neural networks to achieve end-to-end result output, achieving higher computational accuracy and efficiency than traditional methods.

[0004] In the field of autonomous driving, binocular vision solutions have always attracted much attention because they can obtain very reliable, high-precision, dense depth information. The key technology is to use the baseline information of the left and right views to achieve depth calculation through binocular matching. However, this process has a large amount of data, high computing power requirements, is often very time-consuming, increases system energy consumption, and has poor real-time performance. At the same time, it cannot meet the requirements of real-time and fast response of autonomous driving. Summary of the invention

[0005] The embodiments of the present application provide a depth image acquisition method, device, electronic device and storage medium to solve the problem that the existing depth estimation technology has high computing power requirements, increases system energy consumption, and has poor real-time performance and cannot meet the requirements of real-time and rapid response of autonomous driving.

[0006] In order to solve the above technical problems, the embodiments of the present application are implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a depth image acquisition method, the method comprising:

[0008] Performing image matching on the first image and the second image captured by the binocular camera to obtain a disparity image;

[0009] Remapping pixels of the second image based on the disparity image to obtain a pseudo image corresponding to the second image;

[0010] A pre-trained deep image prediction model is called to process the first image, the second image and the pseudo image to obtain correlation features between the first image and the second image and the pseudo image respectively, and the correlation features and image features of the first image are processed to obtain a depth image. The depth image prediction model is deployed in a deep learning processing unit of a programmable logic system of a SOC architecture.

[0011] Optionally, performing image matching on the first image and the second image captured by the binocular camera to obtain a disparity image includes:

[0012] Calling the image preprocessing module to obtain a first original image and a second original image taken by the binocular camera, and preprocessing the first original image and the second original image respectively to obtain a first preprocessed image and a second preprocessed image;

[0013] Calling the image preprocessing module to send the first preprocessed image and the second preprocessed image to a processor system of a SOC architecture through an AXI bus;

[0014] Calling the processor system to perform image correction processing on the first preprocessed image and the second preprocessed image respectively to obtain the first image and the second image;

[0015] The deep learning processing unit is called to adopt a semi-global matching algorithm to perform feature matching on the first image and the second image to obtain the disparity image.

[0016] Optionally, the processor system comprises: a memory and an ARM processor,

[0017] The calling of the processor system to perform image correction processing on the first preprocessed image and the second preprocessed image respectively to obtain the first image and the second image includes:

[0018] Calling the ARM processor to read the first pre-processed image and the second pre-processed image in the memory;

[0019] Image correction processing is performed on the first preprocessed image and the second preprocessed image according to a correction parameter matrix corresponding to the binocular camera acquired in advance to obtain the first image and the second image.

[0020] Optionally, the depth image prediction model includes: a feature extraction network, a correlation network and a depth estimation network,

[0021] The calling of a pre-trained depth image prediction model, processing the first image, the second image, and the pseudo image, obtaining correlation features between the first image and the second image and the pseudo image, respectively, and processing the correlation features and image features of the first image to obtain a depth image, includes:

[0022] Calling the feature extraction network to perform feature extraction on the first image, the second image, and the pseudo image respectively to obtain corresponding first feature maps, second feature maps, and pseudo feature maps;

[0023] Calling the correlation network to perform correlation calculation on the first feature map and the second feature map, and the first feature map and the pseudo feature map, respectively, to obtain a first correlation feature map and a second correlation feature map, and performing feature splicing on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map;

[0024] The depth estimation network is called to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and feature fusion is performed on the depth feature map and the first image to obtain and output the depth image.

[0025] Optionally, the calling of the correlation network to perform correlation calculation on the first feature map and the second feature map, and the first feature map and the pseudo feature map, respectively, to obtain a first correlation feature map and a second correlation feature map, respectively, includes:

[0026] Calling the one-dimensional correlation calculation method based on the vector inner product of the correlation network to calculate a first similarity score of each matching pixel point in the first feature map and the second feature map, and a second similarity score of each matching pixel point in the first feature map and the pseudo feature map;

[0027] A first correlation feature graph including a similarity score matrix is ​​determined based on the first similarity score, and a second correlation feature graph including a similarity score matrix is ​​determined based on the second similarity score.

[0028] Optionally, the performing feature splicing on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map includes:

[0029] An operation of adding similarity score matrices is performed on the first correlation feature graph and the second correlation feature graph to obtain the correlation splicing feature graph.

[0030] Optionally, the depth estimation network comprises: an upsampling convolution layer, a convolution layer and a residual block,

[0031] The calling of the depth estimation network to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and performing feature fusion on the depth feature map and the first image to obtain and output the depth image includes:

[0032] Calling the convolutional layer and the residual block to perform feature extraction on the correlation splicing feature map to obtain a correlation feature map;

[0033] Calling the upsampling convolution layer to process the correlation feature map to obtain a depth feature map with the same size as the original image;

[0034] Feature fusion processing is performed on the image features of the depth feature map and the image features of the first image to obtain and output the depth image.

[0035] Optionally, after the correlation feature and the image feature of the first image are processed to obtain a depth image, the method further includes:

[0036] The programmable logic system is called to send the depth image to a memory in a processor system of the SOC architecture through an AXI bus, so that other devices can read the depth image from the memory.

[0037] In a second aspect, an embodiment of the present application provides a depth image acquisition device, the device comprising:

[0038] A disparity map acquisition module is used to perform image matching on a first image and a second image captured by a binocular camera to obtain a disparity image;

[0039] a pseudo image acquisition module, configured to perform pixel remapping on the second image based on the parallax image to obtain a pseudo image corresponding to the second image;

[0040] A depth image acquisition module is used to call a pre-trained depth image prediction model, process the first image, the second image and the pseudo image, obtain correlation features between the first image and the second image and the pseudo image respectively, and process the correlation features and image features of the first image to obtain a depth image. The depth image prediction model is deployed in a deep learning processing unit of a programmable logic system of a SOC architecture.

[0041] Optionally, the disparity map acquisition module includes:

[0042] a preprocessed image acquisition unit, configured to call the image preprocessing module to acquire a first original image and a second original image captured by the binocular camera, and preprocess the first original image and the second original image respectively to obtain a first preprocessed image and a second preprocessed image;

[0043] A preprocessed image sending unit, configured to call the image preprocessing module to send the first preprocessed image and the second preprocessed image to a processor system of a SOC architecture through an AXI bus;

[0044] an image correction unit, configured to call the processor system to perform image correction processing on the first preprocessed image and the second preprocessed image respectively, so as to obtain the first image and the second image;

[0045] The disparity map acquisition unit is used to call the deep learning processing unit to use a semi-global matching algorithm to perform feature matching on the first image and the second image to obtain the disparity image.

[0046] Optionally, the processor system comprises: a memory and an ARM processor,

[0047] The image correction unit comprises:

[0048] A pre-processed image reading subunit, used for calling the ARM processor to read the first pre-processed image and the second pre-processed image in the memory;

[0049] The image correction subunit is used to perform image correction processing on the first preprocessed image and the second preprocessed image according to the correction parameter matrix corresponding to the binocular camera acquired in advance, so as to obtain the first image and the second image.

[0050] Optionally, the depth image prediction model includes: a feature extraction network, a correlation network and a depth estimation network,

[0051] The depth image acquisition module comprises:

[0052] A feature map acquisition unit, used for calling the feature extraction network to perform feature extraction on the first image, the second image and the pseudo image respectively, to obtain corresponding first feature maps, second feature maps and pseudo feature maps;

[0053] A correlation feature map acquisition unit, used for calling the correlation network to perform correlation calculation on the first feature map and the second feature map, and the first feature map and the pseudo feature map, respectively, to obtain a first correlation feature map and a second correlation feature map, and to perform feature splicing on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map;

[0054] A depth image acquisition unit is used to call the depth estimation network to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and perform feature fusion on the depth feature map and the first image to obtain and output the depth image.

[0055] Optionally, the correlation feature graph acquisition unit includes:

[0056] A similarity score calculation subunit, used for calling the one-dimensional correlation calculation method based on the vector inner product of the correlation network to calculate a first similarity score of each matching pixel point in the first feature map and the second feature map, and a second similarity score of each matching pixel point in the first feature map and the pseudo feature map;

[0057] The correlation feature graph determination subunit is used to determine a first correlation feature graph including a similarity score matrix according to the first similarity score, and to determine a second correlation feature graph including a similarity score matrix according to the second similarity score.

[0058] Optionally, the correlation feature graph acquisition unit includes:

[0059] The splicing feature map acquisition subunit is used to perform a similarity score matrix addition operation on the first correlation feature map and the second correlation feature map to obtain the correlation splicing feature map.

[0060] Optionally, the depth estimation network comprises: an upsampling convolution layer, a convolution layer and a residual block,

[0061] The depth image acquisition unit comprises:

[0062] A correlation feature map acquisition subunit, used for calling the convolutional layer and the residual block to perform feature extraction on the correlation splicing feature map to obtain a correlation feature map;

[0063] A depth feature map acquisition subunit, used to call the upsampling convolution layer to process the correlation feature map to obtain a depth feature map with the same size as the original image;

[0064] The depth image acquisition subunit is used to perform feature fusion processing on the image features of the depth feature map and the image features of the first image to obtain and output the depth image.

[0065] Optionally, the device further comprises:

[0066] The depth image sending module is used to call the programmable logic system to send the depth image to the memory in the processor system of the SOC architecture through the AXI bus, so that other devices can read the depth image from the memory.

[0067] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0068] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements any of the above-mentioned depth image acquisition methods when executed by the processor.

[0069] In a fourth aspect, an embodiment of the present application provides a readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any of the depth image acquisition methods described above.

[0070] In an embodiment of the present application, a disparity image is obtained by performing image matching on a first image and a second image captured by a binocular camera. Pixels of the second image are remapped based on the disparity image to obtain a pseudo image corresponding to the second image. A pre-trained depth image prediction model is called to process the first image, the second image, and the pseudo image to obtain correlation features between the first image and the second image and the pseudo image, respectively, and the correlation features and the image features of the first image are processed to obtain a depth image. The depth image prediction model is deployed in a deep learning processing unit of a programmable logic system of a SOC architecture. By generating a pseudo image and two original images as inputs to the model, an embodiment of the present application can achieve a significant improvement in robustness at a relatively low computational cost. At the same time, making full use of the processor system and programmable logic system of a multi-core on-chip processor, as well as a deep learning processor based on programmable logic resources to implement an efficient neural network algorithm, can improve the real-time performance of deep image acquisition, reduce computing power requirements, and thus reduce system energy consumption.

[0071] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A flowchart of a method for acquiring a depth image provided by an embodiment of the present application;

[0074] Figure 2 A schematic diagram of an image preprocessing process provided in an embodiment of the present application;

[0075] Figure 3A schematic diagram of a SOC architecture provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of a PhiFAD network structure provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of an overall depth estimation process provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a depth map estimation process provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of the structure of a depth image acquisition device provided in an embodiment of the present application;

[0080] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] Reference Figure 1 , shows a flowchart of the steps of a depth image acquisition method provided by an embodiment of the present application. Figure 1 As shown, the depth image acquisition method may include: step 101, step 102 and step 103.

[0083] Step 101: performing image matching on a first image and a second image captured by a binocular camera to obtain a disparity image.

[0084] In the embodiment of the present application, a binocular camera refers to a stereoscopic vision system composed of two cameras, which can perceive the geometric structure and surface grayscale value of an object in a three-dimensional manner and convert it into a digital signal. The two cameras are placed in two independent positions and shoot different perspectives respectively, so the positions of the objects in their pictures are different. By calculating the images of the two cameras, the three-dimensional information of the object, that is, the depth information, can be obtained.

[0085] In this example, the binocular camera can be applied to computer vision (such as 3D reconstruction, scene understanding, object recognition, etc.), autonomous driving, robot navigation, smart home and other fields.

[0086] The first image and the second image refer to two images obtained after processing two initial images taken by the binocular camera at the same time.

[0087] The disparity image is a key output in the binocular vision system. It records the disparity values ​​of corresponding pixels in the left and right images (i.e., the first image and the second image in this embodiment). Disparity is defined as the column coordinates of the same-name point pair in the left view minus the column coordinates in the right view, in pixels. The disparity image is a two-dimensional image of the same size as the original image. Each position stores the disparity value of the pixel at that position in pixels.

[0088] After the original image captured by the binocular camera is processed to obtain the first image and the second image, the first image and the second image may be matched to obtain a parallax image. The implementation process may be described in detail in conjunction with the following specific implementation manner.

[0089] In a specific implementation of the present application, the above step 101 may include:

[0090] Sub-step A1: calling the image preprocessing module to obtain a first original image and a second original image taken by the binocular camera, and preprocessing the first original image and the second original image respectively to obtain a first preprocessed image and a second preprocessed image.

[0091] In this embodiment, the image processing and depth map prediction process can be implemented based on the processor system (PS) and programmable logic system (PL) in the SOC architecture. Figure 3 As shown in the figure, PL and PS can communicate using the AXI protocol.

[0092] The AXI HP (AXI High Performance) interface is suitable for data-intensive and real-time applications, and has the advantages of high bandwidth and low latency in image processing scenarios. It supports multiple channels and multiple data widths, can efficiently transmit image data, ensure fast data exchange between the processor system (PS) and programmable logic (PL), and improve overall processing performance and efficiency.

[0093] PL has the advantages of low power consumption, strong flexibility and easy integration of all algorithms and modules. It is good at handling parallel tasks. For image processing algorithms, it can process convolution, interpolation and other calculations in parallel, which is very suitable for data-intensive and real-time applications.

[0094] An image preprocessing module is pre-set in the PL, and the image preprocessing module can communicate with the binocular vision system using MIPI (Mobile Industry Processor Interface) to read the input original image (ie, the first original image and the second original image).

[0095] Furthermore, the image preprocessing module can preprocess the first original image and the second original image respectively to obtain the corresponding first preprocessor and second preprocessed image. For example, the two original images can be resized and cropped respectively, wherein the resize operation refers to changing the size of the image, that is, the width and height of the image, which can be implemented by an interpolation algorithm (such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, etc.) to maintain the quality of the image. The image resize operation is suitable for FPGA (Field-Programmable Gate Array) implementation. In terms of algorithm, the module adopts linear splitting to calculate, which reduces the number of multiplication operations in the calculation, reduces the implementation area and power consumption, and the calculation clock frequency of the system increases due to the reduction of operations. The crop operation refers to selecting a rectangular area from the original image as a new image to remove unnecessary parts of the image or focus on a specific area of ​​the image. Among them, cropping is used to remove image areas with little effective information or high noise, which facilitates the feature extraction of the deep image prediction network. Scaling scales the cropped image to a preset size to ensure the input structure of the adaptation network.

[0096] It can be understood that by utilizing the high flexibility of FPGA, an image resizing algorithm can be performed for any target image that is smaller than or equal to the original image. As a result, the system can process any position in the scene with any resolution, significantly improving the flexibility and compatibility of the system.

[0097] After the first original image and the second original image are preprocessed respectively to obtain the first preprocessed image and the second preprocessed image, sub-step A2 is performed.

[0098] Sub-step A2: calling the image preprocessing module to send the first preprocessed image and the second preprocessed image to a processor system of a SOC architecture through an AXI bus.

[0099] After the first original image and the second original image are preprocessed to obtain the first preprocessed image and the second preprocessed image, the first preprocessed image and the second preprocessed image can be sent to the processor system of the SOC architecture through the AXI bus. In this example, the processor system (PS) may include: a memory (such as DDR (double data rate synchronous dynamic random access memory)) and an ARM (Advanced RISC Machines) processor. Figure 3 As shown, the image preprocessing module can send the first preprocessed image and the second preprocessed image to the DDR in the PS through the AXI bus, so as to cache and transfer the results of the ARM processor or the image preprocessing module.

[0100] After the image preprocessing module is called to send the first preprocessed image and the second preprocessed image to the processor system of the SOC architecture through the AXI bus, sub-step A3 is performed.

[0101] Sub-step A3: calling the processor system to perform image correction processing on the first pre-processed image and the second pre-processed image respectively to obtain the first image and the second image.

[0102] After the image preprocessing module is called to send the first preprocessed image and the second preprocessed image to the processor system of the SOC architecture through the AXI bus, the processor system can perform image correction processing on the first preprocessed image and the second preprocessed image respectively to obtain the first image and the second image. Specifically, the ARM processor in the PS can be called to read the first preprocessed image and the second preprocessed image in the memory, and perform image correction processing on the first preprocessed image and the second preprocessed image according to the correction parameter matrix corresponding to the pre-acquired binocular camera to obtain the first image and the second image. Figure 3 As shown, ARM can read the resized and corp image from DDR, perform correction processing, and then send the corrected image back to DDR.

[0103] In practical applications, the stereo camera can obtain the calibration parameter matrix through Zhang calibration method, and this parameter matrix is ​​known. The preprocessed image is corrected using the parameter matrix so that the same object in the left and right images has the same size and is on the same horizontal line. This ensures that the image matching cost only needs to be calculated in the horizontal direction and improves the final depth accuracy.

[0104] After the processor system is called to perform image correction processing on the first preprocessed image and the second preprocessed image respectively to obtain the first image and the second image, sub-step A4 is executed.

[0105] Sub-step A4: calling the deep learning processing unit to adopt a semi-global matching algorithm to perform feature matching on the first image and the second image to obtain the disparity image.

[0106] After the processor system is called to perform image correction processing on the first preprocessed image and the second preprocessed image respectively to obtain the first image and the second image, the ARM processor can send the corrected first image and the second image to the DDR for storage, and the image preprocessing module in the PL can read the first image and the second image from the DDR. Specifically, the image preprocessing module can read the corrected image from the DDR through AXI and extract the grayscale information. The grayscale image obtained at this time is the first image and the second image. Then, the deep learning processing unit in the PL can perform subsequent processing on the first image and the second image.

[0107] The Deep Learning Processing Unit (DPU) is a dedicated chip designed to accelerate deep learning inference, aiming to help system designers achieve low-latency, high-performance deep learning inference on edge devices.

[0108] The Semi-Global Matching (SGM) algorithm is a matching algorithm between local matching and global matching. It selects the disparity of each pixel to form a disparity map, and sets a global energy function related to the disparity map to minimize this energy function in order to achieve the purpose of solving the optimal disparity for each pixel.

[0109] After the DPU obtains the first image and the second image, the DPU may use a semi-global matching algorithm to perform feature matching on the first image and the second image to obtain a disparity image. The specific implementation process may be:

[0110] 1. Cost calculation: Perform census transformation on the first image and the second image, and use the transformed images to calculate the matching cost pixel by pixel within the disparity range.

[0111] 2. Cost aggregation: Adopt 4-direction (not limited to 4) aggregation, aggregate the cost of the previous step in 4 directions respectively, and obtain the aggregated matching cost.

[0112] 3. Disparity calculation: For each pixel of the first image, select the disparity that minimizes the matching cost after aggregation.

[0113] 4. Post-processing: Improve the quality of the final disparity map through uniqueness detection, median filtering and other methods.

[0114] It can be understood that the above implementation process is only an example of a disparity image acquisition method listed in order to better understand the technical solution of the embodiment of the present application. In a specific implementation, other algorithms can also be used to acquire disparity images (such as block matching algorithm (BM), etc.), and this embodiment does not limit this.

[0115] After performing image matching on the first image and the second image captured by the binocular camera to obtain a disparity image, step 102 is executed.

[0116] Step 102: remapping pixels of the second image based on the disparity image to obtain a pseudo image corresponding to the second image.

[0117] The pseudo image refers to an image generated after pixel remapping of the second image by the parallax image.

[0118] Pixel remapping is the process of mapping the coordinates of a pixel or a group of pixels in an image to a new position in another image or the same image. In pixel remapping based on disparity images, the disparity images provide information about the depth or position difference of corresponding pixels between the two images. With this information, the second image can be pixel remapped to simulate a scene observed from a different perspective or depth, or to generate a pseudo image with a specific effect.

[0119] After performing image matching on the first image and the second image captured by the binocular camera to obtain a disparity image, the second image can be pixel-remapped based on the disparity image to obtain a pseudo image corresponding to the second image. The specific implementation process can be:

[0120] 1. Calculate the mapping coordinates: Based on the disparity values ​​in the disparity image, calculate the coordinates of each pixel in the second image that should be located in the new image after remapping. This usually involves performing a geometric transformation based on the disparity values ​​and the internal and external parameters of the camera.

[0121] 2. Pixel value interpolation: Since the mapped coordinates may not be integers, an interpolation method (such as bilinear interpolation, nearest neighbor interpolation, etc.) is needed to calculate the pixel values ​​of the corresponding pixels in the new image.

[0122] 3. Generate a pseudo image: Generate a remapped image based on the calculated mapping coordinates and interpolated pixel values. This image can be a simulated image of the second image observed from a different perspective or depth, or a pseudo image obtained through some specific processing.

[0123] It can be understood that the generation of a pseudo image is not limited to the pixel remapping method. It can also be achieved by translating or otherwise transforming the pixels in the second image according to the disparity image to align them with the first image to obtain a pseudo image, etc. The method for generating the pseudo image can also be determined according to business needs, and this embodiment does not limit this.

[0124] The generation process of pseudo images can be combined with Figure 2 This is described in detail as follows.

[0125] like Figure 2 As shown in the figure, the binocular camera captures the left original image and the right original image at the same time, and then the left original image and the right original image can be resized, corp, corrected, etc. to obtain the left image and the right image. Then, the SGM algorithm is used to match the left image and the right image to obtain the SGM disparity map. The SGM disparity map is used to warp the right image to generate a pseudo right image.

[0126] After performing pixel remapping on the second image based on the disparity image to obtain a pseudo image corresponding to the second image, step 103 is performed.

[0127] Step 103: Call a pre-trained depth image prediction model to process the first image, the second image and the pseudo image to obtain correlation features between the first image and the second image and the pseudo image respectively, and process the correlation features and image features of the first image to obtain a depth image. The depth image prediction model is deployed in a deep learning processing unit of a programmable logic system of a SOC architecture.

[0128] The depth image prediction model may be a model for performing depth image prediction in a deep learning processing unit deployed in a programmable logic system of a SOC architecture.

[0129] After performing pixel remapping on the second image based on the disparity image to obtain a pseudo image corresponding to the second image, a pre-trained depth image prediction model can be called to process the first image, the second image and the pseudo image to obtain correlation features between the first image and the second image and the pseudo image respectively, and the correlation features and image features of the first image are processed to obtain a depth image.

[0130] The structure of the depth image prediction model can be as follows Figure 4As shown, the depth image prediction model may include three parts: I, feature extraction network, II, correlation network (i.e., correlation network), and III, depth estimation network. A pseudo right image can be generated through the SGM algorithm and the Warp layer, and the left image (i.e., the first image), the right image (i.e., the second image), and the pseudo right image (i.e., the pseudo image) are used as inputs to the depth image prediction model. The processing process of the three input images by the depth image prediction model can be described in detail in combination with the following specific implementation method.

[0131] In a specific implementation of the present application, the above step 103 may include:

[0132] Sub-step B1: calling the feature extraction network to perform feature extraction on the first image, the second image and the pseudo image respectively, to obtain corresponding first feature maps, second feature maps and pseudo feature maps.

[0133] In this embodiment, after obtaining the pseudo image, the first image, the second image and the pseudo image are used as inputs of the depth image prediction model. First, the feature extraction network can be called to extract features from the first image, the second image and the pseudo image respectively to obtain the corresponding first feature map, the second feature map and the pseudo feature map.

[0134] In this example, the feature extraction network can be composed of a convolution layer and a residual block, and the main computational operations are convolution and downsampling. Through multiple convolution and downsampling operations, image features are extracted and the size is reduced. The first image, the second image, and the pseudo image can be input into the feature extraction network with the same parameters to obtain their corresponding feature maps.

[0135] After calling the feature extraction network to perform feature extraction on the first image, the second image and the pseudo image respectively to obtain the corresponding first feature map, the second feature map and the pseudo feature map, sub-step B2 is executed.

[0136] Sub-step B2: Call the correlation network to perform correlation calculation on the first feature map and the second feature map, and the first feature map and the pseudo feature map, respectively, to obtain a first correlation feature map and a second correlation feature map, and perform feature splicing on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map.

[0137] After calling the feature extraction network to extract features from the first image, the second image, and the pseudo image respectively to obtain the corresponding first feature map, the second feature map, and the pseudo feature map, the correlation network can be called to perform correlation calculations on the first feature map and the second feature map, and the first feature map and the pseudo feature map respectively to obtain the first correlation feature map and the second correlation feature map respectively, and feature splicing is performed on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map.

[0138] Specifically, the one-dimensional correlation calculation method based on the vector inner product of the correlation network can be called to calculate the first similarity score of each matching pixel in the first feature map and the second feature map, and the second similarity score of each matching pixel in the first feature map and the pseudo feature map. Then, according to the first similarity score, the first correlation feature map containing the similarity score matrix can be determined, and according to the second similarity score, the second correlation feature map containing the similarity score matrix can be determined. Finally, the similarity score matrix can be added to the first correlation feature map and the second correlation feature map to obtain the correlation splicing feature map.

[0139] In this embodiment, for two given multi-channel feature maps f1 and f2, the role of the correlation layer is to find the correspondence between the left and right images. Since the input image of stereo matching is usually a corrected row-aligned image, the disparity search range can be constrained in the horizontal direction, within the range of the maximum disparity D, and the one-dimensional correlation based on the vector inner product is used to calculate the similarity between the left and right feature vectors. The calculation formula is as follows:

[0140]

[0141] In the above formula (1), f l and f r is the multi-channel feature map corresponding to the two images extracted by the convolutional neural network, (x, y) and (xd, y) are the pixel coordinates of the two feature maps, N c is the number of feature channels, d∈[0,D-1] represents the disparity value, <> represents the vector dot product, C corr (d,x,y) represents the similarity score matrix composed of the similarity scores corresponding to each pixel in the two feature maps.

[0142] After obtaining two correlation feature graphs including similarity score matrices, the similarity score matrices may be added to the first correlation feature graph and the second correlation feature graph to obtain a correlation splicing feature graph.

[0143] After obtaining the correlation splicing feature map, execute sub-step B3.

[0144] Sub-step B3: calling the depth estimation network to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and performing feature fusion on the depth feature map and the first image to obtain and output the depth image.

[0145] After obtaining the correlation splicing feature map, the depth estimation network can be called to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and the depth feature map and the first image are feature fused to obtain and output a depth image. In this example, the depth estimation network may include: an upsampling convolution layer, a convolution layer, and a residual block. When processing the correlation splicing feature map, the convolution layer and the residual block can be called to extract features from the correlation splicing feature map to obtain a correlation feature map. Then, the upsampling convolution layer can be called to process the correlation feature map to obtain a depth feature map of the same size as the original image. Finally, the image features of the depth feature map and the image features of the first image can be feature fused to obtain and output a depth image. Specifically, the total similarity matrix is ​​input into the network, and the features of the correlation map are extracted using the convolution layer and the residual block. Through multiple convolution and upsampling convolution operations, the scale of the feature map is restored, and the disparity is optimized. Among them, the upsampling convolution is used to gradually restore the spatial resolution, and the convolution is used to further extract and enhance the features. Fusion of left image information. During the disparity optimization process, the information of the left image is fused to further improve the accuracy of disparity estimation.

[0146] In this example, the convolution layer can intercept a part of the input image through a sliding window and perform convolution calculations. Different convolution kernels can be used to extract features such as edges and textures. After the sliding window traverses the entire image and calculates the convolution, the image features can be extracted and the data size of the result is smaller than the original image. The convolution layer can be used to extract features from the input data and reduce the data size.

[0147] The residual block can contain multiple convolutional layers and pass the input directly to the output through jump connections. The residual block improves the training efficiency and performance of the network by learning the residual between the input and output. It can maintain the effective transfer of gradients and realize deeper neural networks to extract higher-level features. The residual block can assist in extracting higher-level features while maintaining the effective transfer of gradients.

[0148] The upsample convolution layer can restore the spatial resolution of the feature map to a higher level through interpolation or deconvolution operations. It is usually used to restore the low-resolution feature map to the original resolution of the input image. In this example, the upsample convolution layer is used to gradually restore the resolution of the extracted and streamlined feature map and generate a disparity map that is consistent with the size of the input image.

[0149] The embodiment of the present application generates a pseudo right image from the right image through SGM and Warp operations as the third image input of the depth estimation algorithm, which significantly improves the robustness with less computational cost. The processor system and programmable logic system of the multi-core on-chip processor are fully utilized, as well as the deep learning processor (Deep Learning Processing Unit) based on programmable logic resources to implement an efficient FAD neural network algorithm (i.e., the depth image prediction model in this embodiment). At the same time, the resources of the FPGA can be fully utilized to achieve efficient storage resource utilization based on the AXI HP port, and realize an efficient stereoscopic vision depth estimation method from both storage and computing aspects.

[0150] After the depth image is obtained, it can be sent by the DPU to the memory (such as DDR, etc.) in the PS through the AXI bus, and then read by other devices.

[0151] Next, combine Figure 5 and Figure 6 The depth map estimation process is described in detail.

[0152] like Figure 6 As shown, the depth map estimation process may include:

[0153] 1. Image preprocessing to obtain the left image, right image and pseudo right image. Figure 5 As shown, the binocular camera captures images to obtain the left original image and the right original image at the same time. The image preprocessing module in the SOC preprocesses the left original image and the right original image respectively (such as resizing, cropping, etc.) to obtain the left image and the right image. Then, the left image and the right image can be matched based on the SGM algorithm to obtain a disparity map, and the disparity map is used to perform a warp operation on the right image to obtain a pseudo right image.

[0154] 2. The feature extraction layer extracts features and reduces the size to obtain the left feature map, right feature map and pseudo right feature map. Figure 5 As shown, the left image, the right image and the pseudo right image can be input into three feature extraction networks with the same parameters to perform operations such as feature extraction and scale reduction, and output the corresponding three feature maps.

[0155] 3. The Correlation layer calculates the correlation between the left feature map and the pseudo right feature map, and the correlation between the left feature map and the right feature map, and obtains two correlation maps.

[0156] 4. Splice two correlation graphs. Figure 5 As shown, two correlation graphs can be spliced ​​through a contact layer to obtain a spliced ​​graph.

[0157] 5. The depth estimation network estimates the depth map based on the spliced ​​correlation map and the left image. Figure 5 As shown in the figure, the left image can be used as the input of the depth estimation network. After the spliced ​​correlation map is input into the depth estimation network, the convolution layer and residual block can be used to extract the features of the correlation map. Through multiple convolution and upsampling convolution operations, the scale of the feature map is restored and the disparity is optimized. The upsampling convolution is used to gradually restore the spatial resolution, and the convolution is used to further extract and enhance the features. And the feature information of the left image is fused. In the process of disparity optimization, the information of the left image is fused to further improve the accuracy of disparity estimation.

[0158] The depth image acquisition method provided in the embodiment of the present application obtains a disparity image by performing image matching on the first image and the second image captured by the binocular camera. The second image is pixel-remapped based on the disparity image to obtain a pseudo image corresponding to the second image. The pre-trained depth image prediction model is called to process the first image, the second image and the pseudo image to obtain the correlation features between the first image and the second image and the pseudo image, respectively, and the correlation features and the image features of the first image are processed to obtain a depth image. The depth image prediction model is deployed in the deep learning processing unit of the programmable logic system of the SOC architecture. The embodiment of the present application can obtain a significant improvement in robustness at a lower computational cost by generating a pseudo image and two original images as the input of the model. At the same time, the processor system and programmable logic system of the multi-core on-chip processor are fully utilized, and the deep learning processor based on programmable logic resources is used to realize an efficient neural network algorithm, which can improve the real-time performance of deep image acquisition, reduce computing power requirements, and thus reduce system energy consumption.

[0159] Reference Figure 7 , which shows a schematic diagram of the structure of a depth image acquisition device provided in an embodiment of the present application, such as Figure 7 As shown, the depth image acquisition device 700 may include the following modules:

[0160] The disparity map acquisition module 710 is used to perform image matching on the first image and the second image captured by the binocular camera to obtain a disparity image;

[0161] A pseudo image acquisition module 720, configured to perform pixel remapping on the second image based on the parallax image to obtain a pseudo image corresponding to the second image;

[0162] The depth image acquisition module 730 is used to call a pre-trained depth image prediction model, process the first image, the second image and the pseudo image, obtain the correlation features between the first image and the second image and the pseudo image respectively, and process the correlation features and the image features of the first image to obtain a depth image. The depth image prediction model is deployed in the deep learning processing unit of the programmable logic system of the SOC architecture.

[0163] Optionally, the disparity map acquisition module includes:

[0164] a preprocessed image acquisition unit, configured to call the image preprocessing module to acquire a first original image and a second original image captured by the binocular camera, and preprocess the first original image and the second original image respectively to obtain a first preprocessed image and a second preprocessed image;

[0165] A preprocessed image sending unit, configured to call the image preprocessing module to send the first preprocessed image and the second preprocessed image to a processor system of a SOC architecture through an AXI bus;

[0166] an image correction unit, configured to call the processor system to perform image correction processing on the first preprocessed image and the second preprocessed image respectively, so as to obtain the first image and the second image;

[0167] The disparity map acquisition unit is used to call the deep learning processing unit to use a semi-global matching algorithm to perform feature matching on the first image and the second image to obtain the disparity image.

[0168] Optionally, the processor system comprises: a memory and an ARM processor,

[0169] The image correction unit comprises:

[0170] A pre-processed image reading subunit, used for calling the ARM processor to read the first pre-processed image and the second pre-processed image in the memory;

[0171] The image correction subunit is used to perform image correction processing on the first preprocessed image and the second preprocessed image according to the correction parameter matrix corresponding to the binocular camera acquired in advance, so as to obtain the first image and the second image.

[0172] Optionally, the depth image prediction model includes: a feature extraction network, a correlation network and a depth estimation network,

[0173] The depth image acquisition module comprises:

[0174] A feature map acquisition unit, used for calling the feature extraction network to perform feature extraction on the first image, the second image and the pseudo image respectively, to obtain corresponding first feature maps, second feature maps and pseudo feature maps;

[0175] A correlation feature map acquisition unit, used for calling the correlation network to perform correlation calculation on the first feature map and the second feature map, and the first feature map and the pseudo feature map, respectively, to obtain a first correlation feature map and a second correlation feature map, and to perform feature splicing on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map;

[0176] A depth image acquisition unit is used to call the depth estimation network to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and perform feature fusion on the depth feature map and the first image to obtain and output the depth image.

[0177] Optionally, the correlation feature graph acquisition unit includes:

[0178] A similarity score calculation subunit, used for calling the one-dimensional correlation calculation method based on the vector inner product of the correlation network to calculate a first similarity score of each matching pixel point in the first feature map and the second feature map, and a second similarity score of each matching pixel point in the first feature map and the pseudo feature map;

[0179] The correlation feature graph determination subunit is used to determine a first correlation feature graph including a similarity score matrix according to the first similarity score, and to determine a second correlation feature graph including a similarity score matrix according to the second similarity score.

[0180] Optionally, the correlation feature graph acquisition unit includes:

[0181] The splicing feature map acquisition subunit is used to perform a similarity score matrix addition operation on the first correlation feature map and the second correlation feature map to obtain the correlation splicing feature map.

[0182] Optionally, the depth estimation network comprises: an upsampling convolution layer, a convolution layer and a residual block,

[0183] The depth image acquisition unit comprises:

[0184] A correlation feature map acquisition subunit, used for calling the convolutional layer and the residual block to perform feature extraction on the correlation splicing feature map to obtain a correlation feature map;

[0185] A depth feature map acquisition subunit, used to call the upsampling convolution layer to process the correlation feature map to obtain a depth feature map with the same size as the original image;

[0186] The depth image acquisition subunit is used to perform feature fusion processing on the image features of the depth feature map and the image features of the first image to obtain and output the depth image.

[0187] Optionally, the device further comprises:

[0188] The depth image sending module is used to call the programmable logic system to send the depth image to the memory in the processor system of the SOC architecture through the AXI bus, so that other devices can read the depth image from the memory.

[0189] The depth image acquisition device provided in the embodiment of the present application obtains a disparity image by performing image matching on the first image and the second image captured by the binocular camera. The second image is pixel-remapped based on the disparity image to obtain a pseudo image corresponding to the second image. The pre-trained depth image prediction model is called to process the first image, the second image and the pseudo image to obtain the correlation features between the first image and the second image and the pseudo image respectively, and the correlation features and the image features of the first image are processed to obtain a depth image. The depth image prediction model is deployed in the deep learning processing unit of the programmable logic system of the SOC architecture. The embodiment of the present application can obtain a significant improvement in robustness at a lower computational cost by generating a pseudo image and two original images as the input of the model. At the same time, the processor system and programmable logic system of the multi-core on-chip processor are fully utilized, and the deep learning processor based on programmable logic resources is used to realize an efficient neural network algorithm, which can improve the real-time performance of deep image acquisition, reduce computing power requirements, and thus reduce system energy consumption.

[0190] An embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned depth image acquisition method when executed by the processor.

[0191] Figure 8 FIG. 8 is a schematic diagram showing the structure of an electronic device 800 according to an embodiment of the present invention. Figure 8 As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0192] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, a microphone, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0193] The various processes and processing described above may be performed by the processing unit 801. For example, the method of any of the above embodiments may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the CPU 801, one or more actions in the method described above may be performed.

[0194] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned depth image acquisition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0195] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0196] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0197] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0198] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0200] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0203] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0204] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A depth image acquisition method, characterized in that: The method comprises: Performing image matching on the first image and the second image captured by the binocular camera to obtain a disparity image; Remapping pixels of the second image based on the disparity image to obtain a pseudo image corresponding to the second image; A pre-trained deep image prediction model is called to process the first image, the second image and the pseudo image to obtain correlation features between the first image and the second image and the pseudo image respectively, and the correlation features and image features of the first image are processed to obtain a depth image. The depth image prediction model is deployed in a deep learning processing unit of a programmable logic system of a SOC architecture.

2. The method according to claim 1, characterized in that The step of performing image matching on the first image and the second image captured by the binocular camera to obtain a disparity image includes: Calling the image preprocessing module to obtain a first original image and a second original image taken by the binocular camera, and preprocessing the first original image and the second original image respectively to obtain a first preprocessed image and a second preprocessed image; Calling the image preprocessing module to send the first preprocessed image and the second preprocessed image to a processor system of a SOC architecture through an AXI bus; Calling the processor system to perform image correction processing on the first preprocessed image and the second preprocessed image respectively to obtain the first image and the second image; The deep learning processing unit is called to adopt a semi-global matching algorithm to perform feature matching on the first image and the second image to obtain the disparity image.

3. The method according to claim 2, characterized in that The processor system comprises: a memory and an ARM processor, The calling of the processor system to perform image correction processing on the first preprocessed image and the second preprocessed image respectively to obtain the first image and the second image includes: Calling the ARM processor to read the first pre-processed image and the second pre-processed image in the memory; Image correction processing is performed on the first preprocessed image and the second preprocessed image according to a correction parameter matrix corresponding to the binocular camera acquired in advance to obtain the first image and the second image.

4. The method according to claim 1, characterized in that: The depth image prediction model includes: a feature extraction network, a correlation network and a depth estimation network. The calling of a pre-trained depth image prediction model, processing the first image, the second image, and the pseudo image, obtaining correlation features between the first image and the second image and the pseudo image, respectively, and processing the correlation features and image features of the first image to obtain a depth image, includes: Calling the feature extraction network to perform feature extraction on the first image, the second image, and the pseudo image respectively to obtain corresponding first feature maps, second feature maps, and pseudo feature maps; Calling the correlation network to perform correlation calculation on the first feature map and the second feature map, and the first feature map and the pseudo feature map, respectively, to obtain a first correlation feature map and a second correlation feature map, and performing feature splicing on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map; The depth estimation network is called to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and feature fusion is performed on the depth feature map and the first image to obtain and output the depth image.

5. The method according to claim 4, characterized in that The calling of the correlation network to perform correlation calculation on the first feature map and the second feature map, and the first feature map and the pseudo feature map, respectively, to obtain a first correlation feature map and a second correlation feature map, respectively, includes: Calling the one-dimensional correlation calculation method based on the vector inner product of the correlation network to calculate a first similarity score of each matching pixel point in the first feature map and the second feature map, and a second similarity score of each matching pixel point in the first feature map and the pseudo feature map; A first correlation feature graph including a similarity score matrix is ​​determined based on the first similarity score, and a second correlation feature graph including a similarity score matrix is ​​determined based on the second similarity score.

6. The method according to claim 5, characterized in that The step of performing feature splicing on the first correlation feature map and the second correlation feature map to obtain a correlation splicing feature map includes: An operation of adding similarity score matrices is performed on the first correlation feature graph and the second correlation feature graph to obtain the correlation splicing feature graph.

7. The method according to claim 4, characterized in that The depth estimation network includes: an upsampling convolution layer, a convolution layer and a residual block, The calling of the depth estimation network to perform depth estimation on the correlation splicing feature map to obtain a depth feature map, and performing feature fusion on the depth feature map and the first image to obtain and output the depth image includes: Calling the convolutional layer and the residual block to perform feature extraction on the correlation splicing feature map to obtain a correlation feature map; Calling the upsampling convolution layer to process the correlation feature map to obtain a depth feature map with the same size as the original image; Feature fusion processing is performed on the image features of the depth feature map and the image features of the first image to obtain and output the depth image.

8. The method according to claim 1, characterized in that After the correlation feature and the image feature of the first image are processed to obtain a depth image, the method further includes: The programmable logic system is called to send the depth image to a memory in a processor system of the SOC architecture through an AXI bus, so that other devices can read the depth image from the memory.

9. A depth image acquisition device, characterized in that: The device comprises: A disparity map acquisition module is used to perform image matching on a first image and a second image captured by a binocular camera to obtain a disparity image; a pseudo image acquisition module, configured to perform pixel remapping on the second image based on the parallax image to obtain a pseudo image corresponding to the second image; A depth image acquisition module is used to call a pre-trained depth image prediction model, process the first image, the second image and the pseudo image, obtain correlation features between the first image and the second image and the pseudo image respectively, and process the correlation features and image features of the first image to obtain a depth image. The depth image prediction model is deployed in a deep learning processing unit of a programmable logic system of a SOC architecture.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the depth image acquisition method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the depth image acquisition method according to any one of claims 1 to 8.

Images