Binocular image depth estimation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a binocular image depth estimation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a binocular image shot by a binocular camera; the binocular image comprises a first image and a second image; the first image comprises a first region of interest; the second image comprises a second region of interest; the first region of interest and the second region of interest are used for representing the same object; obtaining a disparity map between the first image and the second image as a first disparity map, and obtaining a disparity map between the first region of interest and the second region of interest as a second disparity map; updating the first disparity map by using the second disparity map to obtain the updated first disparity map; and generating a depth map corresponding to the binocular image based on the updated first disparity map.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a binocular image depth estimation method, device, electronic device and storage medium. Background Art

[0002] In the field of modeling, it is necessary to perform depth estimation on the binocular images taken by the binocular camera, obtain a depth map, and perform modeling based on the depth map. When performing depth estimation on the binocular images, one of the binocular images can be used as a reference image, and a depth map of the reference image can be generated as the depth map corresponding to the binocular image. At present, the scheme for depth estimation of binocular images has the defect of low accuracy in depth estimation of key areas in the binocular images. Summary of the invention

[0003] The embodiments of the present disclosure provide a binocular image depth estimation method, device, electronic device and storage medium, which can improve the accuracy of depth estimation of key areas in a binocular image.

[0004] In a first aspect, an embodiment of the present disclosure provides a binocular image depth estimation method, comprising:

[0005] Acquire a binocular image captured by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object;

[0006] Acquire a disparity map between the first image and the second image as a first disparity map, and acquire a disparity map between the first region of interest and the second region of interest as a second disparity map;

[0007] Using the second disparity map, updating the first disparity map to obtain an updated first disparity map;

[0008] A depth map corresponding to the binocular image is generated based on the updated first disparity map.

[0009] In a second aspect, an embodiment of the present disclosure provides a binocular image depth estimation device, comprising:

[0010] A first acquisition unit is used to acquire a binocular image taken by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object;

[0011] A second acquiring unit, configured to acquire a disparity map between the first image and the second image as a first disparity map, and acquire a disparity map between the first region of interest and the second region of interest as a second disparity map;

[0012] a disparity updating unit, configured to update the first disparity map by using the second disparity map to obtain an updated first disparity map;

[0013] An image generating unit is configured to generate a depth map corresponding to the binocular image based on the updated first disparity map.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0016] In one or more embodiments of the present disclosure, first, a binocular image captured by a binocular camera is obtained, the binocular image includes a first image and a second image, the first image includes a first region of interest, the second image includes a second region of interest, the first region of interest and the second region of interest are used to represent the same object, then, a disparity map between the first image and the second image is obtained as a first disparity map, a disparity map between the first region of interest and the second region of interest is obtained as a second disparity map, and the first disparity map is updated using the second disparity map to obtain an updated first disparity map, and finally, a depth map corresponding to the binocular image is generated based on the updated first disparity map. It can be seen that through this embodiment, the disparity map between the first region of interest and the second region of interest can be used as the second disparity map, the disparity map between the first image and the second image can be used as the first disparity map, and the first disparity map can be updated using the second disparity map, so as to achieve the effect of updating the disparity value between the first region of interest and the second region of interest recorded in the first disparity map using the disparity value between the first region of interest and the second region of interest recorded in the second disparity map, so that a more accurate disparity value between the first region of interest and the second region of interest is recorded in the updated first disparity map, so that in the depth map corresponding to the binocular image generated based on the updated first disparity map, more accurate depth values ​​of the first region of interest and the second region of interest are recorded, thereby improving the accuracy of depth estimation of the first region of interest and the second region of interest, and improving the accuracy of depth estimation of key areas in the binocular image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0018] Figure 1 A schematic diagram of a process for binocular image depth estimation provided by an embodiment of the present disclosure;

[0019] Figure 2 A schematic diagram of the principle of a disparity estimation model provided by an embodiment of the present disclosure;

[0020] Figure 3 A schematic diagram of the structure of a binocular image depth estimation device provided by an embodiment of the present disclosure;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0023] To facilitate understanding of the various embodiments of the present disclosure, the technical terms involved are explained here.

[0024] Disparity value: For the binocular images taken by the binocular camera, determine the pixel point corresponding to any physical space point in each image of the binocular image to obtain two pixel points. Based on the imaging principle of the binocular camera, the two pixel points have the same ordinate value but different abscissa values. The difference between the two abscissa values ​​is the disparity value of the pixel point.

[0025] Disparity map: Taking one of the binocular images as the reference image, an image is generated with pixels corresponding to the pixels in the reference image one by one. The pixel value of each pixel in the generated image is equal to the difference between the horizontal coordinate value of the corresponding pixel in the reference image and the horizontal coordinate value of the corresponding pixel in the other image in the binocular image. The generated image is the disparity map between the binocular images generated from the perspective of the reference image. It can be seen that the disparity map is an image generated by taking one of the binocular images as the reference image to record the disparity value between the reference image and the other image in the binocular image. It can be understood that the pixel value of each pixel in the disparity map is the disparity value, and when the disparity value is calculated according to a certain rule, such as the disparity value is equal to the horizontal coordinate value of the pixel in the reference image minus the horizontal coordinate value of the pixel in the other image, if the reference images selected in the binocular image are different, the disparity values ​​recorded in the generated disparity map are also different.

[0026] Depth map corresponding to the binocular image: After generating a disparity map between the binocular images with one of the binocular images as the reference image, a depth map of the reference image can be generated based on the disparity map and the parameters of the camera in the binocular camera that captures the reference image. The depth map of the reference image is the depth map corresponding to the binocular image.

[0027] Region of interest: ROI (region of interest). In the fields of machine vision and image processing, the area to be processed in the processed image is outlined in the form of a box, circle, ellipse, irregular polygon, etc., which is called the region of interest ROI. The region of interest is usually the area that needs to be focused on during image processing.

[0028] Taking into account the defect of low accuracy in depth estimation of key areas in binocular images in current solutions for depth estimation of binocular images, the embodiments of the present disclosure provide a binocular image depth estimation method, which can improve the accuracy of depth estimation of key areas in binocular images. The method can be applied on a dedicated server and executed by the server.

[0029] Figure 1 A schematic diagram of a process flow of a binocular image depth estimation method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:

[0030] Step S102, acquiring a binocular image captured by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object;

[0031] Step S104, obtaining a disparity map between the first image and the second image as a first disparity map, and obtaining a disparity map between the first region of interest and the second region of interest as a second disparity map;

[0032] Step S106, using the second disparity map to update the first disparity map to obtain an updated first disparity map;

[0033] Step S108: generating a depth map corresponding to the binocular image based on the updated first disparity map.

[0034] In the disclosed embodiment, first, a binocular image taken by a binocular camera is obtained, the binocular image includes a first image and a second image, the first image includes a first region of interest, the second image includes a second region of interest, the first region of interest and the second region of interest are used to represent the same object, then, a disparity map between the first image and the second image is obtained as a first disparity map, a disparity map between the first region of interest and the second region of interest is obtained as a second disparity map, and the first disparity map is updated using the second disparity map to obtain an updated first disparity map, and finally, based on the updated first disparity map, a depth map corresponding to the binocular image is generated. It can be seen that through this embodiment, the disparity map between the first region of interest and the second region of interest can be used as the second disparity map, the disparity map between the first image and the second image can be used as the first disparity map, and the first disparity map can be updated using the second disparity map, so as to achieve the effect of updating the disparity value between the first region of interest and the second region of interest recorded in the first disparity map using the disparity value between the first region of interest and the second region of interest recorded in the second disparity map, so that a more accurate disparity value between the first region of interest and the second region of interest is recorded in the updated first disparity map, so that in the depth map corresponding to the binocular image generated based on the updated first disparity map, more accurate depth values ​​of the first region of interest and the second region of interest are recorded, thereby improving the accuracy of depth estimation of the first region of interest and the second region of interest, and improving the accuracy of depth estimation of key areas in the binocular image.

[0035] The binocular image depth estimation method provided in the disclosed embodiment can be applied in the scene of real-time depth estimation and real-time modeling of binocular images, improve the accuracy of real-time depth estimation of key areas in binocular images, and improve the modeling accuracy of key areas in the real-time modeling process. For example, in various scenes such as live broadcast scenes of volumetric videos and live broadcast scenes of virtual objects, the binocular image depth estimation method in this embodiment can generate a depth map in real time based on the binocular image taken by the binocular camera, so as to perform modeling in real time based on the depth map.

[0036] In the above step S102, a binocular image captured by a binocular camera is obtained. In this embodiment, the binocular camera can capture any scene to obtain a binocular image, such as capturing a person, a landscape, a building, etc. to obtain a binocular image. The binocular image includes a first image and a second image, the first image is captured by the first camera in the binocular camera, and the second image is captured by the second camera in the binocular camera. In an example, the first camera is a left camera, the second camera is a right camera, the first image is a left image, and the second image is a right image. The first image includes a first region of interest, and the second image includes a second region of interest. The first region of interest and the second region of interest are used to represent the same object, for example, the first region of interest and the second region of interest are both used to represent a face. It can be understood that the object represented by the first region of interest and the second region of interest is the object that needs to be focused on during depth estimation, and the accuracy of depth estimation of the object needs to be improved so as to accurately model the object.

[0037] In one embodiment, the method further comprises the following steps to determine a first region of interest in the first image, and to determine a second region of interest in the second image:

[0038] Determine a first area where the object is located in the first image, and determine a second area where the object is located in the second image;

[0039] Based on the disparity value between the first area and the second area, the first area is shifted in the first image, and the shifted first area is a first area of ​​interest;

[0040] Based on the disparity value between the first region and the second region, the second region is shifted in the second image, and the shifted second region is the second region of interest.

[0041] In this embodiment, since the first image and the second image are binocular images captured by a binocular camera, the image contents of the first image and the second image are substantially the same, and therefore, the object exists in both the first image and the second image. Based on this, a first region where the object is located is determined in the first image, and a second region where the object is located is determined in the second image.

[0042] In one embodiment, the first image and the second image are images that have been epipolar corrected. After the epipolar correction, the same point in the photographed physical space, the two corresponding pixels in the first image and the second image are on the same horizontal line. First, the rectangular area where the above-mentioned object is located is determined in the first image, and the rectangular area where the above-mentioned object is located is determined in the second image. Considering that the width and height of the two rectangular areas may be different, the two rectangular areas are also expanded so that the width and height of the two expanded rectangular areas are the same. The expanded rectangular area in the first image is the first area, and the expanded rectangular area in the second image is the second area. It can be seen that the width of the first area is the same as that of the second area.

[0043] Next, determine the disparity value between the first area and the second area, that is, determine the difference between the horizontal coordinates of the pixel points in the first area and the horizontal coordinates of the corresponding pixel points in the second area as the disparity value, and according to the disparity value, offset the first area in the first image, and the first area after the offset is the first area of ​​interest, and according to the disparity value, offset the second area in the second image, and the second area after the offset is the second area of ​​interest.

[0044] It can be seen that through this embodiment, the first area where the above-mentioned object is located can be determined in the first image, the second area where the above-mentioned object is located can be determined in the second image, and based on the disparity value between the first area and the second area, the first area and the second area are offset respectively to obtain the first area of ​​interest and the second area of ​​interest. Since the disparity value between the first area and the second area is taken into account when determining the first area of ​​interest and the second area of ​​interest, the accuracy of determining the first area of ​​interest and the second area of ​​interest is improved, and the accuracy of subsequently generating the second disparity map is improved.

[0045] In one embodiment, the method further comprises the following steps to determine the disparity value between the first area and the second area:

[0046] A disparity value between a central pixel point of the first area and a central pixel point of the second area is obtained as the disparity value between the first area and the second area.

[0047] In this embodiment, the horizontal coordinate value of the central pixel point of the first area is determined in the first area of ​​the first image as the first horizontal coordinate value, and the horizontal coordinate value of the central pixel point of the second area is determined in the second area of ​​the second image as the second horizontal coordinate value. The first image is used as a reference, and the second horizontal coordinate value is subtracted from the first horizontal coordinate value to obtain the disparity value between the central pixel point of the first area and the central pixel point of the second area, and the disparity value is used as the disparity value between the first area and the second area. The above process can be expressed by the following formula (1):

[0048] disp approx =center1-center2 formula (1)

[0049] In formula (1), disp approx represents the disparity value between the first area and the second area, center1 represents the horizontal coordinate value of the central pixel point of the first area, and center2 represents the horizontal coordinate value of the central pixel point of the second area.

[0050] It can be seen that through this embodiment, the central pixel point of the first area and the central pixel point of the second area can be used as representatives of the first area and the second area, and the disparity value between the central pixel point of the first area and the central pixel point of the second area can be obtained as the disparity value between the first area and the second area, thereby efficiently determining the disparity value between the first area and the second area.

[0051] In one embodiment, the method of shifting the first area in the first image based on the disparity value between the first area and the second area includes:

[0052] determining an offset of the first area based on a disparity value between the first area and the second area, a width of the first area in the first image, and an offset coefficient;

[0053] The first region is shifted in the first image according to the shift amount of the first region.

[0054] In this embodiment, the offset of the first area is first determined based on the disparity value between the first area and the second area, the width of the first area in the first image, and the offset coefficient. This process can be expressed by the following formula (2).

[0055]

[0056] In formula (2), offset org1 is the offset of the first area, disp approx is the disparity value between the first area and the second area, is the width of the first area in the first image, and A is the offset coefficient.

[0057] In one example, Among them, width infer The disparity estimation model to be introduced later requires an input width for the input image when performing disparity estimation. The image to be disparity estimated needs to be transformed to the required input width before the disparity estimation model can perform disparity estimation. datasetis the average disparity value of a training sample set used by the disparity estimation model in a training process, the training sample set includes a plurality of binocular sample images and a disparity map of each binocular sample image, and each binocular sample image includes two sample images.

[0058] In one example, Where i represents the i-th binocular sample image in the training sample set used by the disparity estimation model during the training process, disp i represents the average disparity value of each pixel in the i-th binocular sample image, and N is the number of binocular sample images included in the training sample set. Among them, for the i-th binocular sample image, one sample image in the i-th binocular sample image is used as the reference image, and the disparity value of each pixel in the reference image relative to the corresponding pixel in another sample image in the i-th binocular sample image is calculated, and the average disparity value of each pixel in the reference image is obtained, and the average value is used as disparity. i Substitute the above formula to calculate disp dataset .

[0059] Next, the first region is offset in the first image according to the offset of the first region. The offset of the first region is the offset of the first region in the horizontal direction. Therefore, according to the offset of the first region, the first region is offset in the horizontal direction in the first image, for example, the horizontal coordinates of the four vertices of the first region are offset according to the offset of the first region, and the offset first region is the first region of interest.

[0060] It can be seen that through this embodiment, the offset of the first region can be determined based on the disparity value between the first region and the second region, the width of the first region in the first image and the offset coefficient, and the first region is offset in the first image according to the offset of the first region, so as to accurately obtain the first region of interest in the first image.

[0061] In one embodiment, the method of offsetting the second area in the second image based on the disparity value between the first area and the second area includes:

[0062] determining an offset of the second area based on a disparity value between the first area and the second area, a width of the second area in the second image, and an offset coefficient;

[0063] The second area is shifted in the second image according to the shift amount of the second area.

[0064] In this embodiment, the process of shifting the second area is the same as the process of shifting the first area. First, based on the disparity value between the first area and the second area, the width of the second area in the second image and the shift coefficient, the shift amount of the second area is determined, and the process can be expressed by the following formula (3).

[0065]

[0066] In formula (3), offset org2 is the offset of the second area, disp approx is the disparity value between the first area and the second area, is the width of the second area in the second image, and A is the offset coefficient.

[0067] In one example, Among them, width infer The disparity estimation model to be introduced later requires an input width for the input image when performing disparity estimation. The image to be disparity estimated needs to be transformed to the required input width before the disparity estimation model can perform disparity estimation. dataset is the average disparity value of a training sample set used by the disparity estimation model in a training process, the training sample set includes a plurality of binocular sample images and a disparity map of each binocular sample image, and each binocular sample image includes two sample images.

[0068] In one example, Where i represents the i-th binocular sample image in the training sample set used by the disparity estimation model during the training process, disp i represents the average disparity value of each pixel in the i-th binocular sample image, and N is the number of binocular sample images included in the training sample set. Among them, for the i-th binocular sample image, one sample image in the i-th binocular sample image is used as the reference image, and the disparity value of each pixel in the reference image relative to the corresponding pixel in another sample image in the i-th binocular sample image is calculated, and the average disparity value of each pixel in the reference image is obtained, and the average value is used as disparity. i Substitute the above formula to calculate disp dataset .

[0069] It can be seen that since the width of the second area in the second image is the same as the width of the first area in the first image, and the offset coefficients in formula (2) and formula (3) are also the same, the offset value of the second area calculated by formula (3) is equal to the offset value of the first area calculated by formula (2).

[0070] Next, the second area is offset in the second image according to the offset of the second area. The offset of the second area is the offset of the second area in the horizontal direction. Therefore, according to the offset of the second area, the second area is offset in the horizontal direction in the second image, for example, the horizontal coordinates of the four vertices of the second area are offset according to the offset of the second area, and the second area after the offset is the second region of interest.

[0071] It can be seen that through this embodiment, the offset of the second area can be determined based on the disparity value between the first area and the second area, the width of the second area in the second image and the offset coefficient, and the second area is offset in the second image according to the offset of the second area, so as to accurately obtain the second area of ​​interest in the second image.

[0072] In the above step S104, a disparity map between the first image and the second image is obtained as a first disparity map, and a disparity map between the first region of interest and the second region of interest is obtained as a second disparity map.

[0073] In one embodiment, obtaining a disparity map between a first image and a second image as a first disparity map includes:

[0074] A disparity map between the first image and the second image is generated as a first disparity map through a disparity estimation model; the disparity estimation model is trained using binocular video frames that are sequentially arranged in time sequence.

[0075] In this embodiment, a disparity estimation model is pre-trained. Figure 2 A schematic diagram of the principle of a disparity estimation model provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the disparity estimation model includes a feature extractor and a disparity iterator, the feature extractor can be a 2D fully connected network (FCN) and its variants, and the disparity iterator can be a gated recurrent unit convolutional network (GRU) and its variants. In this embodiment, the first image and the second image can be input into the disparity estimation model, and the feature extractor in the disparity estimation model can extract image features of the first image to obtain first image features, and extract image features of the second image to obtain second image features.

[0076] The disparity estimation model can analyze the feature mutual correlation between the first image feature and the second image feature to obtain a feature mutual correlation matrix between the first image feature and the second image feature, in which the feature similarity between each feature in the first image feature and each feature in the second image feature is recorded through multiple matrix values.

[0077] Next, the disparity estimation model can randomly generate an initial disparity map for the first image and the second image, and input the initial disparity map and the above-mentioned feature cross-correlation matrix into the disparity iterator. The disparity iterator can adjust the disparity value in the initial disparity map according to the above-mentioned feature cross-correlation matrix. The adjusted initial disparity map is subjected to an upsampling process to become the first disparity map.

[0078] According to the foregoing introduction, the disparity estimation model has a required input width for the input image, and the widths of the first image and the second image need to be transformed into the required input width before they can be input into the disparity estimation model for disparity estimation.

[0079] When the method in the embodiment of the present disclosure is used to generate volumetric video in real time, the disparity estimation model can obtain multiple binocular images continuously captured by a binocular camera, each binocular image includes a first image and a second image, and each binocular image is a sequence of video frames continuously captured in time sequence. Based on this, when each binocular image is input into the disparity estimation model for disparity estimation, the disparity estimation model can use the disparity map estimated for the previous binocular image in time sequence as the initial disparity map of the next binocular image, thereby improving the efficiency of estimating the disparity map of the next binocular image based on the initial disparity map, so that the disparity iterator can obtain the disparity map of the next binocular image after a smaller number of iterations.

[0080] When the method in the embodiment of the present disclosure is used to generate volumetric video in real time, the feature extractor and disparity iterator in the disparity estimation model can be accelerated by TensorRT (Nvidia inference engine acceleration tool), and the above-mentioned process of generating the feature correlation matrix and the process of upsampling the adjusted initial disparity map can be accelerated using CUDA (Compute Unified Device Architecture), thereby improving the disparity estimation speed and modeling speed in the scenario of real-time generation of volumetric video and meeting the real-time requirements of modeling.

[0081] The following is a brief introduction to the training process of the disparity estimation model. Training the disparity estimation model means training the parameters in the feature extractor and the disparity iterator. According to the previous introduction, the disparity estimation model is trained based on a training sample set, which includes a plurality of binocular sample images and a disparity map of each binocular sample image, and each binocular sample image includes two sample images. For each binocular sample image, one of the sample images is used as a reference image, and the disparity value of each pixel in the reference image relative to the corresponding pixel in another sample image in the binocular sample image is calculated, and the disparity map of the binocular sample image is generated according to the calculated disparity value. During training, the disparity estimation model is trained based on a plurality of binocular sample images and the disparity map of each binocular sample image.

[0082] In this embodiment, the disparity estimation model is trained using binocular video frames arranged sequentially in time. As an example, each time the disparity estimation model is trained, multiple binocular sample images obtained by continuous shooting of a binocular camera are selected from the training sample set, and the multiple binocular sample images are multiple binocular video frames arranged sequentially in time. For example, the 3rd to 15th binocular video frames taken continuously are selected, and the disparity estimation model is trained according to the selected binocular video frames, so that the disparity map estimated by the disparity estimation model has high stability in the time domain, so that when the disparity estimation model estimates the disparity map of the binocular video frames taken continuously, the disparity value of the same point in the physical space in each binocular video frame can be guaranteed to change continuously and stably. In addition, the parameters of the disparity estimation model after each training are used as the initial parameters of the model for the next training, and the loss is returned. In addition, random Gaussian noise is added to each binocular sample image in the training sample set to improve the stability of the model.

[0083] It can be seen that through this embodiment, a disparity map between the first image and the second image can be generated as a first disparity map through a disparity estimation model. Since the disparity estimation model is trained using binocular video frames that are sequentially arranged in time sequence, the disparity map estimated by the disparity estimation model has high stability in the time domain. When estimating the disparity maps of continuously shot binocular video frames, the disparity estimation model can ensure that the disparity values ​​of the same point in the physical space in each binocular video frame change continuously and stably.

[0084] In one embodiment, obtaining a disparity map between the first region of interest and the second region of interest as a second disparity map includes:

[0085] Cropping the first region of interest in the first image to obtain a first region image, and cropping the second region of interest in the second image to obtain a second region image;

[0086] A disparity estimation model is used to generate a disparity map between the first region image and the second region image as a second disparity map; the disparity estimation model is trained using binocular video frames that are sequentially arranged in time sequence.

[0087] In this embodiment, the first region of interest is firstly cropped in the first image to obtain a first region image, and the second region of interest is cropped in the second image to obtain a second region image. Next, the widths of the first region image and the second region image are transformed into the required input widths of the disparity estimation model, and then the transformed first region image and the transformed second region image are input into the disparity estimation model for disparity estimation to obtain a second disparity map. The disparity estimation process and the introduction of the disparity estimation model can be found in the previous description, which will not be repeated here.

[0088] It can be seen that through this embodiment, a second disparity map can be generated by a disparity estimation model. Since the disparity estimation model is trained using binocular video frames that are arranged in sequence in time, the disparity map estimated by the disparity estimation model has high stability in the time domain. When estimating the disparity maps of continuously shot binocular video frames, the disparity estimation model can ensure that the disparity values ​​of the same point in the physical space in each binocular video frame change continuously and stably.

[0089] In one embodiment, the size of the first disparity map is an output size of a disparity estimation model; and generating a disparity map between the first region image and the second region image as a second disparity map by using the disparity estimation model includes:

[0090] Processing the first region image and the second region image through a disparity estimation model to obtain an output image; the size of the output image is the above output size;

[0091] According to the size of the first region of interest in the first image after the first image is transformed into the output size, the above-mentioned output size, the size of the first image and the transformation coefficient, the output image is resized to obtain a second disparity map; the size of the second disparity map is the same as the size of the first region of interest in the first disparity map.

[0092] According to the previous description, the disparity estimation model has a required input width for the input image, and the width of the image needs to be transformed into the required input width before it can be input into the disparity estimation model for disparity estimation. Correspondingly, the disparity estimation model also has an output size, which includes an output width and an output height. The output width is equal to the required input width of the disparity estimation model for the input image, and the output height is equal to the required input height of the disparity estimation model for the input image. Therefore, the width and height of the first disparity map output by the disparity estimation model are equal to the output width and height of the disparity estimation model.

[0093] Based on this, in the process of generating a disparity map between a first region image and a second region image as a second disparity map through a disparity estimation model, the first region image and the second region image are first processed through the disparity estimation model to obtain an output image, the output image represents the disparity value between the first region image and the second region image, and the width and height of the output image are equal to the output width and height of the disparity estimation model. In this embodiment, the first disparity map is an image generated with the first image as a reference, so the output image is also resized according to the size of the first region of interest in the first image after the size of the first image is transformed into the output size of the disparity estimation model, the output size of the disparity estimation model, the size of the first image, and the transformation coefficient to obtain the second disparity map. This process can be performed outside the disparity estimation model. In an example, the output image can be resized according to the width of the first region of interest in the first image after the width of the first image is transformed into the output width of the disparity estimation model, the output width of the disparity estimation model, the width of the first image, and the transformation coefficient to obtain the second disparity map. The width and height of the second disparity map are the same as the width and height of the first region of interest in the first disparity map, and the second disparity map is an image generated with the first region of interest as a reference.

[0094] By transforming the output image to obtain the second disparity map, the size of the second disparity map is made the same as the size of the first region of interest in the first disparity map, which can improve the accuracy of subsequently using the second disparity map to update the first disparity map, and generate a more accurate updated first disparity map.

[0095] In one embodiment, the output image is resized according to the size of the first region of interest in the first image after the first image is transformed into the output size, the above output size, the size of the first image and the transformation coefficient through the following formula (4) and formula (5) to obtain a second disparity map.

[0096]

[0097]

[0098] In formulas (4) and (5), offset org It represents the transformation coefficient, which is the offset of the first area or the second area mentioned above. According to the previous records, the offset values ​​of the first area and the second area are the same, so they are represented by the same symbol here. Indicates the width of the first image after the width of the first image is transformed into the output width of the disparity estimation model, width infer Indicates the output width of the disparity estimation model, width org Indicates the width of the first image, represents the above output image, represents the output image after the width and height are transformed to the width and height of the first region of interest, represents the second disparity map.

[0099] The principle of formulas (4) and (5) is as follows: for the pixel coordinates of each pixel point in the above-mentioned output image, substitute them into formulas (4) and (5) in turn for calculation, obtain each calculated pixel coordinate, and generate a second disparity map according to the pixel value of the calculated pixel coordinate. For the first type of pixel points in the second disparity map, if the pixel coordinates of the first type of pixel points are obtained by calculating the pixel coordinates of the second type of pixel points in the output image through formulas (4) and (5), then the pixel value (i.e., disparity value) of the first type of pixel points is equal to the pixel value of the second type of pixel points in the output image. For other pixel points in the second disparity map except the first type of pixel points, the pixel values ​​(i.e., disparity values) of other pixel points can be obtained by sampling in the second disparity map.

[0100] An upsampling operation or a downsampling operation is performed on each calculated pixel value to generate a second disparity map.

[0101] It can be understood that if the first disparity map is an image generated based on the second image, the output image can be resized according to the width of the second image after the width of the second image is transformed into the output width of the disparity estimation model, the output width of the disparity estimation model, the width of the second image, and the transformation coefficient to obtain the second disparity map. The width and height of the second disparity map are the same as the width and height of the second region of interest in the second disparity map, and the second disparity map is an image generated based on the second region of interest.

[0102] It can be seen that through this embodiment, the first area image and the second area image can be processed by the disparity estimation model to obtain an output image, and the output image is resized according to the size of the first region of interest in the first image after the size of the first image is transformed into the output size of the disparity estimation model, the output size of the disparity estimation model, the size of the first image and the transformation coefficient to obtain a second disparity map. Since the size of the second disparity map is the same as the size of the first region of interest in the first disparity map, the accuracy of subsequently using the second disparity map to update the first disparity map can be improved, thereby generating a more accurate updated first disparity map.

[0103] After obtaining the first disparity map and the second disparity map, in one embodiment, using the second disparity map to update the first disparity map to obtain an updated first disparity map includes:

[0104] The disparity value between the first region of interest and the second region of interest recorded in the first disparity map is updated by using the disparity value between the first region of interest and the second region of interest recorded in the first disparity map to obtain an updated first disparity map.

[0105] In this embodiment, the second disparity map is an image generated with the first region of interest as a reference, and the second disparity map records the disparity value between each pixel in the first region of interest and the corresponding pixel in the second region of interest. The first disparity map is an image generated with the first image as a reference, and the first disparity map records the disparity value between each pixel in the first image and the corresponding pixel in the second image. Since the first image includes the first region of interest, in this embodiment, the disparity value between the first region of interest and the second region of interest recorded in the second disparity map is used to update the disparity value between the first region of interest and the second region of interest recorded in the first disparity map to obtain an updated first disparity map, so that the updated first disparity map records a more accurate disparity value between the first region of interest and the second region of interest, so that in the depth map corresponding to the binocular image generated based on the updated first disparity map, more accurate depth values ​​of the first region of interest and the second region of interest are recorded, so as to improve the accuracy of depth estimation of the first region of interest and the second region of interest, and improve the accuracy of depth estimation of the key area in the binocular image.

[0106] It can be seen that through this embodiment, the disparity value between the first region of interest and the second region of interest recorded in the first disparity map can be used to update the disparity value between the first region of interest and the second region of interest recorded in the first disparity map, so as to obtain an updated first disparity map, so that the updated first disparity map records more accurate disparity values ​​between the first region of interest and the second region of interest, thereby improving the accuracy of disparity estimation and depth estimation for key areas.

[0107] In one embodiment, the first disparity map is used to record the disparity values ​​between the pixels in the first image and the pixels in the second image; the second disparity map is used to record the disparity values ​​between the pixels in the first region of interest and the pixels in the second region of interest; accordingly, the disparity values ​​between the first region of interest and the second region of interest recorded in the first disparity map are updated by using the disparity values ​​between the first region of interest and the second region of interest recorded in the second disparity map, to obtain the updated first disparity map, including:

[0108] Identify a pixel to be updated in the first image; the pixel to be updated is located in the first region of interest, and a difference between a disparity value corresponding to the pixel to be updated recorded in the first disparity map and a disparity value corresponding to the pixel to be updated recorded in the second disparity map meets a difference requirement;

[0109] The disparity values ​​corresponding to the pixels to be updated recorded in the first disparity map are updated to the disparity values ​​corresponding to the pixels to be updated recorded in the second disparity map, so as to obtain an updated first disparity map.

[0110] In this embodiment, the first disparity map is an image generated based on the first image, and the disparity value between each pixel in the first image and the corresponding pixel in the second image is recorded in the first disparity map. The second disparity map is an image generated based on the first region of interest, and the disparity value between each pixel in the first region of interest and the corresponding pixel in the second region of interest is recorded in the second disparity map. Based on this, in this embodiment, the pixel to be updated is identified in the first image. The pixel to be updated is located in the first region of interest, and the difference between the disparity value corresponding to the pixel recorded in the first disparity map and the disparity value corresponding to the pixel recorded in the second disparity map meets the difference requirement. Then, in the first disparity map, the disparity value corresponding to the pixel to be updated is updated to the disparity value corresponding to the pixel to be updated recorded in the second disparity map, and the updated first disparity map is obtained.

[0111] In one embodiment, the process of updating the first disparity map can be expressed by the following formula (6).

[0112]

[0113] In formula (6), disp final is the updated first disparity map, is the first disparity map, is the second disparity map, pixel∈crop means that the pixel is located in the first region of interest, is the disparity value between the first region of interest and the second region of interest recorded in the first disparity map, threshold diff is the preset disparity threshold.

[0114] It can be seen from formula (6) that the pixel to be updated is located in the first region of interest, and the absolute value of the difference between the disparity value corresponding to the pixel recorded in the first disparity map and the disparity value corresponding to the pixel recorded in the second disparity map is less than the disparity threshold. For the pixel to be updated, the disparity value of the pixel to be updated in the first disparity map is updated to the disparity value recorded in the second disparity map.

[0115] It can be seen that through this embodiment, the pixel points to be updated can be identified by analyzing the positions of the pixel points and the disparity values ​​of the pixel points in the first disparity map and the second disparity map, and the disparity values ​​corresponding to the pixel points to be updated recorded in the first disparity map are updated to the disparity values ​​corresponding to the pixel points to be updated recorded in the second disparity map, so as to accurately obtain the updated first disparity map.

[0116] In one embodiment, after obtaining the updated first disparity map, a depth map corresponding to the binocular image is also generated based on the updated first disparity map. For example, the updated first disparity map is an image generated based on the first image, and the updated first disparity map records the disparity value between each pixel in the first image and the corresponding pixel in the second image. Then, according to the updated first disparity map and the focal length of the camera in the binocular camera that shoots the first image, and the distance between the two cameras in the binocular camera, a depth map of the first image is generated as the depth map corresponding to the binocular image.

[0117] In one embodiment, generating a depth map corresponding to the binocular image based on the updated first disparity map includes:

[0118] In the updated first disparity map, remove the problematic disparity value;

[0119] Generate a depth map corresponding to the binocular image based on the first disparity map after removing the problematic disparity value;

[0120] Among them, the problematic disparity value includes one or more of the following disparity values: a disparity value whose confidence is less than a confidence threshold, a disparity value whose value does not meet the disparity value requirement, and a disparity value whose value is greater than the horizontal coordinate value of the pixel point to which the disparity value belongs.

[0121] In this embodiment, in the updated first disparity map, the problematic disparity value is identified and deleted, that is, in the updated first disparity map, the pixel points with wrong pixel values ​​are identified and deleted. Then, based on the first disparity map after removing the problematic disparity value, a depth map corresponding to the binocular image is generated. For example, the updated first disparity map is an image generated based on the first image, and the updated first disparity map records the disparity value between each pixel in the first image and the corresponding pixel in the second image. Then, based on the first disparity map after removing the problematic disparity value and the focal length of the camera in the binocular camera that shoots the first image, and the distance between the two cameras in the binocular camera, a depth map of the first image is generated as the depth map corresponding to the binocular image.

[0122] In this embodiment, the problematic disparity value includes one or more of the following disparity values: a disparity value whose confidence is less than a confidence threshold, a disparity value whose value does not meet the disparity value requirements, and a disparity value whose value is greater than the horizontal coordinate value of the pixel point to which the disparity value belongs.

[0123] As an example, in the case where the updated first disparity map is generated based on the first image, the updated first disparity map, the first image, and the projection map of the updated first disparity map can be input into a pre-trained confidence estimation model, and the confidence of the pixel value (i.e., the disparity value) of each pixel in the updated first disparity map is determined by the confidence estimation model, so as to determine the disparity value with a confidence less than the confidence threshold in the updated first disparity map as the problem disparity value. The projection map refers to the image obtained by projecting the first image according to the first disparity map. For example, the horizontal coordinate of a pixel in the first image is 5, and the disparity value of the pixel is recorded as 3 in the disparity map. Then, the horizontal coordinate of the projected pixel obtained by projecting the pixel in the projection map is 5-3=2.

[0124] As an example, if there is a preset disparity value requirement, such as a preset disparity value interval, then according to the requirement, a disparity value that does not meet the disparity value requirement can be determined in the updated first disparity map as a problem disparity value.

[0125] As an example, each pixel point in the updated first disparity map corresponds one-to-one to each pixel point in the first image. If a disparity value in the first disparity map is greater than the horizontal coordinate value of the pixel point corresponding to the disparity value in the first image, then the disparity value is a disparity value that is greater than the horizontal coordinate value of the pixel point to which the disparity value belongs. In the updated first disparity map, the disparity value that is greater than the horizontal coordinate value of the pixel point to which the disparity value belongs is determined as the problem disparity value.

[0126] In one embodiment, the problematic disparity value may be removed from the updated first disparity map using the following formula (7).

[0127]

[0128] In formula (7), disp final `For the first disparity map after removing the problematic disparity value, disp final is the updated first disparity map, conf net is the confidence level of the disparity value, threshold net is the preset confidence threshold, the horizontal coordinate value of the pixel point to which the cid disparity value belongs, threshold disp The disparity value threshold corresponding to the disparity value requirement. The disparity value threshold can be obtained in the following way: determine the maximum depth value that may appear in the depth map corresponding to the binocular image, and reversely deduce the disparity value threshold based on the maximum depth value. If the disparity value of the pixel point is greater than the disparity value threshold, the depth value of the pixel point is also greater than the maximum depth value, which is an unreasonable depth value.

[0129] It can be seen that through this embodiment, the problematic disparity values ​​can be identified and removed in the updated first disparity map, and a depth map corresponding to the binocular image is generated based on the first disparity map after removing the problematic disparity values, thereby improving the accuracy of depth estimation for the binocular image.

[0130] In summary, a method for depth estimation of a binocular image is provided, and the method embodiment has at least the following technical effects:

[0131] 1. The first disparity map and the second disparity map are efficiently and quickly generated through the disparity estimation model, so that the depth estimation process meets the real-time requirements, and the estimated disparity map has high temporal stability;

[0132] 2. More accurate disparity and depth values ​​can be obtained for key areas, improving the accuracy of depth estimation in key areas;

[0133] 3. It can eliminate the incorrectly estimated disparity values ​​and improve the accuracy of depth estimation.

[0134] Figure 3 A schematic diagram of the structure of a binocular image depth estimation device provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the device comprises:

[0135] A first acquisition unit 31 is used to acquire a binocular image taken by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object;

[0136] A second acquiring unit 32 is configured to acquire a disparity map between the first image and the second image as a first disparity map, and acquire a disparity map between the first region of interest and the second region of interest as a second disparity map;

[0137] A disparity updating unit 33, configured to update the first disparity map by using the second disparity map to obtain an updated first disparity map;

[0138] The image generating unit 34 is configured to generate a depth map corresponding to the binocular image based on the updated first disparity map.

[0139] Optionally, the method further includes a region determination unit, configured to:

[0140] Determine a first area where the object is located in the first image, and determine a second area where the object is located in the second image;

[0141] Based on the disparity value between the first region and the second region, the first region is shifted in the first image, and the first region after the shift is the first region of interest;

[0142] Based on the disparity value between the first region and the second region, the second region is offset in the second image, and the offset second region is the second region of interest.

[0143] Optionally, the area determination unit is specifically used to:

[0144] A disparity value between a central pixel point of the first area and a central pixel point of the second area is obtained as the disparity value between the first area and the second area.

[0145] Optionally, the area determination unit is specifically used to:

[0146] determining an offset of the first area based on a disparity value between the first area and the second area, a width of the first area in the first image, and an offset coefficient;

[0147] The first region is shifted in the first image according to the shift amount of the first region.

[0148] Optionally, the area determination unit is specifically used to:

[0149] determining an offset of the second area based on a disparity value between the first area and the second area, a width of the second area in the second image, and an offset coefficient;

[0150] The second area is shifted in the second image according to the shift amount of the second area.

[0151] Optionally, the second acquiring unit 32 is specifically configured to:

[0152] A disparity map between the first image and the second image is generated as a first disparity map through a disparity estimation model; the disparity estimation model is trained using binocular video frames that are sequentially arranged in time sequence.

[0153] Optionally, the second acquiring unit 32 is specifically configured to:

[0154] Cropping the first region of interest in the first image to obtain a first region image, and cropping the second region of interest in the second image to obtain a second region image;

[0155] A disparity map between the first region image and the second region image is generated as a second disparity map by using the disparity estimation model; the disparity estimation model is obtained by training using binocular video frames sequentially arranged in time sequence.

[0156] Optionally, the size of the first disparity map is an output size of the disparity estimation model; the second acquisition unit 32 is further specifically configured to:

[0157] Processing the first region image and the second region image by the disparity estimation model to obtain an output image; the size of the output image is the output size;

[0158] The output image is resized according to the size of the first region of interest in the first image after the first image is transformed into the output size, the output size, the size of the first image and the transformation coefficient to obtain the second disparity map; the size of the second disparity map is the same as the size of the first region of interest in the first disparity map.

[0159] Optionally, the disparity updating unit 33 is specifically configured to:

[0160] The disparity value between the first region of interest and the second region of interest recorded in the first disparity map is updated by using the disparity value between the first region of interest and the second region of interest recorded in the first disparity map to obtain an updated first disparity map.

[0161] Optionally, the first disparity map is used to record disparity values ​​between pixels in the first image and pixels in the second image; the second disparity map is used to record disparity values ​​between pixels in the first region of interest and pixels in the second region of interest; the disparity updating unit 33 is further specifically used to:

[0162] Identifying a pixel to be updated in the first image; the pixel to be updated is located in the first region of interest, and a difference between a disparity value corresponding to the pixel to be updated recorded in the first disparity map and a disparity value corresponding to the pixel to be updated recorded in the second disparity map meets a difference requirement;

[0163] The disparity value corresponding to the to-be-updated pixel recorded in the first disparity map is updated to the disparity value corresponding to the to-be-updated pixel recorded in the second disparity map, so as to obtain the updated first disparity map.

[0164] Optionally, the image generating unit 34 is specifically configured to:

[0165] In the updated first disparity map, removing the problematic disparity value;

[0166] Generating a depth map corresponding to the binocular image based on the first disparity map after removing the problematic disparity value;

[0167] Among them, the problematic disparity value includes one or more of the following disparity values: a disparity value whose confidence is less than a confidence threshold, a disparity value whose value does not meet the disparity value requirement, and a disparity value whose value is greater than the horizontal coordinate value of the pixel point to which the disparity value belongs.

[0168] The binocular image depth estimation device in the embodiment of the present disclosure can implement each process of the above-mentioned binocular image depth estimation method embodiment and achieve the same effect and function, which will not be repeated here.

[0169] An embodiment of the present disclosure further provides an electronic device, Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 4 As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 401 and memory 402, and one or more applications or data may be stored in the memory 402. Among them, the memory 402 may be a temporary storage or a permanent storage. The application stored in the memory 402 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 401 may be configured to communicate with the memory 402 to execute a series of computer executable instructions in the memory 402 on the electronic device. The electronic device may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input or output interfaces 405, one or more keyboards 406, etc.

[0170] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0171] Acquire a binocular image captured by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object;

[0172] Acquire a disparity map between the first image and the second image as a first disparity map, and acquire a disparity map between the first region of interest and the second region of interest as a second disparity map;

[0173] Using the second disparity map, updating the first disparity map to obtain an updated first disparity map;

[0174] A depth map corresponding to the binocular image is generated based on the updated first disparity map.

[0175] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned binocular image depth estimation method embodiment and achieve the same effect and function, which will not be repeated here.

[0176] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0177] Acquire a binocular image captured by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object;

[0178] Acquire a disparity map between the first image and the second image as a first disparity map, and acquire a disparity map between the first region of interest and the second region of interest as a second disparity map;

[0179] Using the second disparity map, updating the first disparity map to obtain an updated first disparity map;

[0180] A depth map corresponding to the binocular image is generated based on the updated first disparity map.

[0181] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned binocular image depth estimation method embodiment and achieve the same effect and function, which will not be repeated here.

[0182] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0183] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0184] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0185] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0186] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0187] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0192] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0193] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0194] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A binocular image depth estimation method, characterized in that: include: Acquire a binocular image captured by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object; Acquire a disparity map between the first image and the second image as a first disparity map, and acquire a disparity map between the first region of interest and the second region of interest as a second disparity map; Using the second disparity map, updating the first disparity map to obtain an updated first disparity map; A depth map corresponding to the binocular image is generated based on the updated first disparity map.

2. The method according to claim 1, characterized in that The method further comprises: Determine a first area where the object is located in the first image, and determine a second area where the object is located in the second image; Based on the disparity value between the first region and the second region, the first region is shifted in the first image, and the first region after the shift is the first region of interest; Based on the disparity value between the first region and the second region, the second region is offset in the second image, and the offset second region is the second region of interest.

3. The method according to claim 2, characterized in that Also includes: A disparity value between a central pixel point of the first area and a central pixel point of the second area is obtained as the disparity value between the first area and the second area.

4. The method according to claim 2, characterized in that: The offsetting of the first area in the first image based on the disparity value between the first area and the second area includes: determining an offset of the first area based on a disparity value between the first area and the second area, a width of the first area in the first image, and an offset coefficient; The first region is shifted in the first image according to the shift amount of the first region.

5. The method according to claim 2, characterized in that: The offsetting of the second area in the second image based on the disparity value between the first area and the second area includes: determining an offset of the second area based on a disparity value between the first area and the second area, a width of the second area in the second image, and an offset coefficient; The second area is shifted in the second image according to the shift amount of the second area.

6. The method according to claim 1, characterized in that The acquiring a disparity map between the first image and the second image as a first disparity map includes: A disparity map between the first image and the second image is generated as a first disparity map through a disparity estimation model; the disparity estimation model is trained using binocular video frames that are sequentially arranged in time sequence.

7. The method according to claim 6, characterized in that The acquiring the disparity map between the first region of interest and the second region of interest as a second disparity map includes: Cropping the first region of interest in the first image to obtain a first region image, and cropping the second region of interest in the second image to obtain a second region image; A disparity map between the first region image and the second region image is generated as a second disparity map by using the disparity estimation model; the disparity estimation model is obtained by training using binocular video frames sequentially arranged in time sequence.

8. The method according to claim 7, characterized in that The size of the first disparity map is an output size of the disparity estimation model; and the generating, by using the disparity estimation model, a disparity map between the first region image and the second region image as a second disparity map comprises: Processing the first region image and the second region image by the disparity estimation model to obtain an output image; the size of the output image is the output size; The output image is resized according to the size of the first region of interest in the first image after the first image is transformed into the output size, the output size, the size of the first image and the transformation coefficient to obtain the second disparity map; the size of the second disparity map is the same as the size of the first region of interest in the first disparity map.

9. The method according to claim 1, characterized in that: The updating the first disparity map by using the second disparity map to obtain the updated first disparity map includes: The disparity value between the first region of interest and the second region of interest recorded in the first disparity map is updated by using the disparity value between the first region of interest and the second region of interest recorded in the first disparity map to obtain an updated first disparity map.

10. The method according to claim 9, characterized in that The first disparity map is used to record the disparity values ​​between the pixels in the first image and the pixels in the second image; the second disparity map is used to record the disparity values ​​between the pixels in the first region of interest and the pixels in the second region of interest; and the disparity values ​​between the first region of interest and the second region of interest recorded in the first disparity map are updated by using the disparity values ​​between the first region of interest and the second region of interest recorded in the first disparity map to obtain the updated first disparity map, including: Identifying a pixel to be updated in the first image; the pixel to be updated is located in the first region of interest, and a difference between a disparity value corresponding to the pixel to be updated recorded in the first disparity map and a disparity value corresponding to the pixel to be updated recorded in the second disparity map meets a difference requirement; The disparity value corresponding to the to-be-updated pixel recorded in the first disparity map is updated to the disparity value corresponding to the to-be-updated pixel recorded in the second disparity map, so as to obtain the updated first disparity map.

11. The method according to claim 1, characterized in that: The step of generating a depth map corresponding to the binocular image based on the updated first disparity map includes: In the updated first disparity map, removing the problematic disparity value; Generating a depth map corresponding to the binocular image based on the first disparity map after removing the problematic disparity value; Among them, the problematic disparity value includes one or more of the following disparity values: a disparity value whose confidence is less than a confidence threshold, a disparity value whose value does not meet the disparity value requirement, and a disparity value whose value is greater than the horizontal coordinate value of the pixel point to which the disparity value belongs.

12. A binocular image depth estimation device, characterized in that: include: A first acquisition unit is used to acquire a binocular image taken by a binocular camera; the binocular image includes a first image and a second image; the first image includes a first region of interest; the second image includes a second region of interest; the first region of interest and the second region of interest are used to represent the same object; A second acquiring unit, configured to acquire a disparity map between the first image and the second image as a first disparity map, and acquire a disparity map between the first region of interest and the second region of interest as a second disparity map; a disparity updating unit, configured to update the first disparity map by using the second disparity map to obtain an updated first disparity map; An image generating unit is configured to generate a depth map corresponding to the binocular image based on the updated first disparity map.

13. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 11.

Images