Binocular depth estimation method and apparatus, computer-readable medium, and electronic device

Abstract

The disclosure provides a binocular depth estimation method and device, computer readable medium and electronic equipment, and relates to the technical field of depth estimation. The method comprises: acquiring a first image and a second image, and determining image key points corresponding to the first image and the second image respectively; determining key point feature data corresponding to each image key point, and determining fusion distance features between the image key points based on the key point feature data; performing pixel alignment on the first image and the second image according to the fusion distance features, and determining a depth estimation image based on the first image and the second image after pixel alignment. The disclosure can obtain fusion distance features based on key point feature data, and can significantly improve the accuracy of key point matching through the fusion distance features, thereby completing the precise pixel row alignment task.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of depth estimation, in particular to a binocular depth estimation method, a binocular depth estimation device, a computer readable medium and an electronic device. BACKGROUND

[0002] With the continuous improvement of people's living standards, image has become the core function of mobile phone products. In order to improve the imaging effect of image, the depth information of each pixel in the image needs to be determined in some scenes. The hardware-based depth detection has high hardware cost and wide application range. Therefore, a scheme of realizing depth estimation on software is proposed in the image field, that is, monocular depth estimation and binocular depth estimation.

[0003] In the related binocular depth estimation scheme, it is more susceptible to the influence of objective conditions such as light intensity, noise, repeated texture and no texture, resulting in poor accuracy of key point matching results, and further poor accuracy of depth estimation results. SUMMARY

[0004] The purpose of the present disclosure is to provide a binocular depth estimation method, a binocular depth estimation device, a computer readable medium and an electronic device, thereby at least to some extent improving the accuracy of key point matching in the binocular depth estimation scheme, and further ensuring the accuracy of the depth estimation result.

[0005] According to a first aspect of the present disclosure, a binocular depth estimation method is provided, comprising:

[0006] obtaining a first image and a second image, and determining image key points corresponding to the first image and the second image respectively;

[0007] determining key point feature data corresponding to each of the image key points, and determining fusion distance features between the image key points based on the key point feature data;

[0008] performing pixel alignment on the first image and the second image according to the fusion distance features, and determining a depth estimation image based on the first image and the second image after pixel alignment.

[0009] According to a second aspect of the present disclosure, a binocular depth estimation device is provided, comprising:

[0010] An image key point determination module is configured to obtain a first image and a second image, and determine image key points corresponding to the first image and the second image respectively;

[0011] A fusion distance feature determination module is configured to determine key point feature data corresponding to each of the image key points, and determine fusion distance features between the image key points based on the key point feature data.

[0012] pixel alignment module, configured to perform pixel alignment on the first image and the second image according to the fusion distance feature, and determine a depth estimation image based on the pixel-aligned first image and second image.

[0013] According to a third aspect of the present disclosure, a computer readable medium is provided, which stores a computer program. The computer program is executed by a processor to implement the method described above.

[0014] According to a fourth aspect of the present disclosure, an electronic device is provided, which comprises:

[0015] a processor; and

[0016] a memory configured to store one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0017] The binocular depth estimation method provided by an embodiment of the present disclosure can determine image key points corresponding to the first image and the second image respectively, and then determine key point feature data corresponding to the image key points, and determine fusion distance features between the image key points based on the key point feature data. Then, the first image and the second image can be pixel-aligned according to the fusion distance features, and a depth estimation image can be determined based on the pixel-aligned first image and second image. The fusion distance features determined based on the key point feature data can effectively improve the accuracy of pixel alignment of the first image and the second image, thereby improving the accuracy of the depth estimation image.

[0018] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. In the drawings:

[0020] Figure 1 a schematic diagram showing an exemplary system architecture to which the embodiments of the present disclosure can be applied;

[0021] Figure 2 a flowchart schematically showing a binocular depth estimation method in an exemplary embodiment of the present disclosure;

[0022] Figure 3A schematic diagram illustrating a principle of implementing portrait background blurring in an example embodiment of the present disclosure is shown.

[0023] Figure 4 A schematic diagram illustrating a function of a depth estimation module in an example embodiment of the present disclosure is shown.

[0024] Figure 5 A schematic diagram illustrating a flow of calculating a fusion distance feature in an example embodiment of the present disclosure is shown.

[0025] Figure 6 A schematic diagram illustrating a composition of a binocular depth estimation apparatus in an example embodiment of the present disclosure is shown.

[0026] Figure 7 A schematic diagram of an electronic device to which an embodiment of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0027] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. Features described in the description, examples, or claims can be combined in any suitable manner in one or more implementations.

[0028] In addition, the accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure. Like reference numerals refer to like elements throughout the accompanying drawings. It will be described in detail below.

[0029] Figure 1 A schematic diagram of a system architecture of an example application environment of a binocular depth estimation method and apparatus to which an embodiment of the present disclosure can be applied is shown.

[0030] As Figure 1As shown, the system architecture 100 can include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. The terminal devices 101, 102, 103 can be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, and tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system architecture 100 is only illustrative. Depending on the implementation needs, there can be any number of terminal devices, networks, and servers. For example, the server 105 can be a server cluster composed of multiple servers, etc.

[0031] The binocular depth estimation method provided by the embodiments of the present disclosure is generally executed by the terminal devices 101, 102, 103, and accordingly, the binocular depth estimation apparatus is generally provided in the terminal devices 101, 102, 103. However, it is easy for those skilled in the art to understand that the binocular depth estimation method provided by the embodiments of the present disclosure can also be executed by the server 105, and accordingly, the binocular depth estimation apparatus can also be provided in the server 105, which is not specially limited in the present exemplary embodiment. For example, in an exemplary embodiment, a user can collect a first image and a second image through a rear camera module of the terminal device 101, 102, 103, which includes at least two cameras, and upload the first image and the second image to the server 105. The server generates a depth estimation image through the binocular depth estimation method provided by the embodiments of the present disclosure, and then transmits the depth estimation image to the terminal device 101, 102, 103, etc.

[0032] In the binocular depth estimation technology implemented based on the rear camera module of a smart phone (which can include a main camera and a sub camera), although the offline calibration can provide initial values, the line alignment still mainly depends on the positioning and matching accuracy of the image key points. The key point detection and matching mainly include two parts. The first part is to detect the coordinates (positions) of the key points, and to construct a descriptor according to the geometric and grayscale information around the key points. The second part is to establish a matching corresponding relationship between the image key points according to the descriptor. In the application scenario of smart phone portraits, due to the influence of objective conditions such as light intensity, noise, repeated texture, and no texture, the distinguishability of the feature descriptor will significantly decrease. The traditional algorithm mainly relies on the feature descriptor to find the matching corresponding relationship between the image key points, which will lead to the generation of error matching, and these error matching will affect the accuracy of the line alignment, and further affect the accuracy of the depth estimation result and the portrait virtualization effect.

[0033] Based on one or more problems in the related art, the present disclosure first provides a binocular depth estimation method. The binocular depth estimation method and the binocular depth estimation apparatus of the exemplary embodiments of the present disclosure are described in detail below with the terminal device executing the method as an example.

[0034] Figure 2 A flowchart of the binocular depth estimation method in the present exemplary embodiment is shown, which can include the following steps S210 to S230:

[0035] In step S210, a first image and a second image are acquired, and image key points corresponding to the first image and the second image are determined respectively.

[0036] In an exemplary embodiment, the first image refers to image data collected by a primary camera (or a secondary camera) in the terminal device, and the second image refers to image data collected by a secondary camera (or a primary camera) in the terminal device. The first image and the second image are images of the same scene collected by different cameras at different angles at the same time.

[0037] The primary camera and the secondary camera can be started simultaneously to collect the first image and the second image in real time. As understood by those skilled in the art, the first image and the second image can also be acquired from a storage unit (such as a local storage unit or a cloud storage unit), which are pre-collected by the primary camera and the secondary camera and stored in the storage unit. Of course, the first image and the second image can also be collected by the primary camera and the secondary camera of another terminal device, and transmitted to the current terminal device through wired or wireless transmission, and the present exemplary embodiment does not make any special limitation on the source of the first image and the second image.

[0038] The image key point refers to an image pixel point in an image for representing image features. For example, the image key point can be a Harris key point, an Oriented FAST and Rotated BRIEF (ORB) key point, or a Scale Invariant Feature Transform (SIFT) key point, and the present exemplary embodiment does not make any special limitation on the type of the image key point.

[0039] The first image and the second image can be subjected to key point detection respectively to obtain image key points corresponding to the first image and the second image respectively. For example, the first image and the second image can be subjected to key point detection respectively by using an ORB feature description algorithm to obtain image key points corresponding to the first image and the second image respectively. Alternatively, the first image and the second image can be subjected to key point detection respectively by using a SIFT feature description algorithm to obtain image key points corresponding to the first image and the second image respectively. Of course, the first image and the second image can also be subjected to key point detection respectively by using a robust feature description algorithm (BRIEF), a histogram of oriented gradient (HOG) description algorithm, a Harris corner feature description algorithm, and the like, which are not specially limited in the present example embodiment.

[0040] In step S220, key point feature data corresponding to the image key points is determined, and a fusion distance feature between the image key points is determined based on the key point feature data.

[0041] In an example embodiment, the key point feature data refers to data used to describe relevant image features of an image key point. For example, the key point feature data can include a key point feature descriptor and monocular depth information, where the monocular depth information can include key point depth information and key point depth statistical data. The key point feature data can include at least two of a key point feature descriptor, key point depth information, and key point depth statistical data.

[0042] The fusion feature data refers to feature data describing a distance between image key points of two images by using key point feature data. The fusion distance feature between image key points of the first image and the second image can be determined according to the key point feature data of the image key points, and the matching accuracy between image key points of two images can be significantly improved by fusing the key point feature data compared to the related art in which only a key point feature descriptor is used to determine a matching corresponding relationship between image key points of two images.

[0043] In step S230, the first image and the second image are subjected to pixel alignment according to the fusion distance feature, and a depth estimation image is determined based on the first image and the second image after pixel alignment.

[0044] In an example embodiment, the pixel alignment refers to ensuring that corresponding pixels (i.e. homonymous pixels) in the first image and the second image are on the same row, i.e. the row alignment task of the pixels. By performing the row alignment of the pixels of the first image and the second image based on the fused distance feature, the accuracy of the matching corresponding relationship between the key points of the images can be effectively improved, and thus the accuracy of the depth information in the depth estimation image can be ensured.

[0045] The first image and the second image can be pixel-aligned based on the fused distance feature, and then the disparity image can be determined based on the pixel-aligned first image and second image, and the depth information corresponding to each pixel point can be estimated based on the disparity image to obtain the depth estimation image.

[0046] Next, steps S210 to S230 are described in detail.

[0047] In an example application scenario, the binocular depth estimation method in the embodiments of the present disclosure can be applied to a portrait background blurring scene, Figure 3 An example schematic diagram illustrating the principle of implementing portrait background blurring in the example embodiments of the present disclosure is shown in FIG. 3. Figure 3 As shown in FIG. 3, the main function modules used in the portrait background blurring scene can include a depth estimation module 301 and a portrait blurring module 302. The first image 303 captured by the main camera and the second image 304 captured by the auxiliary camera can be input into the depth estimation module 301 for binocular depth estimation, to estimate the disparity of each pixel in the first image 303 and to obtain a depth estimation image based on the disparity. The depth estimation image can be input into the portrait blurring module 302 to determine a blurring parameter distribution image based on certain calculation criteria, to simulate the blurring and spot effects of a single-lens reflex camera. Then, the pixels in the first image 303 can be blurred and rendered based on the blurring parameter distribution image to obtain an effect image 305 with a clear portrait and a blurred background. The binocular depth estimation method in the embodiments of the present disclosure is mainly applied in the depth estimation module 301.

[0048] Specifically, Figure 4 An example functional diagram of a depth estimation module in the example embodiments of the present disclosure is shown in FIG. 4. Figure 4As shown, the depth estimation module 301 can mainly include steps S410 to S440, in which: in step S410, the first image 401 and the second image 402 can be aligned in rows according to the extremum constraint principle, i.e., to ensure that corresponding pixels (i.e., homonymous pixels) in the first image 401 and the second image 402 are on the same row coordinate, and through the row alignment, the calculation amount of subsequent steps can be effectively reduced; in step S420, a sparse disparity map can be calculated according to the first image 401 and the second image 402 after the row alignment, for example, the sparse disparity map can be calculated based on SGM, AD-Census, etc.; in step S430, the sparse disparity map calculated in step S420 can be densified, i.e., the sparse disparity map is converted into a dense disparity map; in step S440, the pixel points in the first image 401 after the row alignment can be de-corrected through the dense disparity map to recover the corresponding disparity of each pixel of the first image 401, and then the depth information of each pixel can be estimated according to the disparity of each pixel in the first image 401, and finally a depth estimation image is generated.

[0049] In the row alignment step, online row alignment is generally adopted, such as using the detected key points, taking the offline calibration parameters as initial values, calculating new internal and external parameters by using a numerical optimization method, and then determining the matching corresponding relationship between the image key points according to the new internal and external parameters, and then realizing row alignment. That is, the traditional method mainly uses feature descriptors to find the matching corresponding relationship between the image key points, but due to the influence of objective conditions such as light intensity, noise, repeated texture, and no texture, the discriminability of the feature descriptors will significantly decrease, which will lead to the generation of error matching, and these error matching will affect the row alignment accuracy, and then affect the accuracy of depth estimation and the portrait blurring effect.

[0050] In the embodiment of the present disclosure, the fusion distance feature between the image key points is determined based on the key point feature data, and then the row alignment of the pixels of the first image and the second image can be performed according to the fusion distance feature, which can effectively improve the accuracy of the matching corresponding relationship between the image key points, and then ensure the accuracy of the depth information in the depth estimation image.

[0051] In an exemplary embodiment, the image key points can include first image key points corresponding to the first image and second image key points corresponding to the second image. The first image can be subjected to key point detection to obtain the first image key points, and the second image can be subjected to key point detection to obtain the second image key points, wherein the type of the first image key points is consistent with the type of the second image key points. For example, the first image key points and the second image key points can both be Harris key points, and the present exemplary embodiment does not specially limit this.

[0052] It should be noted that the "first" and "second" in the "first image key point" and "second image key point" in the present embodiment and subsequent embodiments are only used to distinguish the image key points corresponding to different images, and do not have any special meaning and should not cause any special limitation to the present embodiment.

[0053] Specifically, the key point feature data can include a key point feature descriptor, a first key point feature descriptor of the first image key point can be constructed, and a second key point feature descriptor of the second image key point can be constructed; wherein the key point feature descriptor can be any one of a histogram of oriented gradients (HOG) descriptor, a scale-invariant feature transform (SIFT) descriptor, a binary robust independent elementary features (BRIEF) descriptor, and an oriented FAST and rotated BRIEF (ORB) descriptor.

[0054] In an exemplary embodiment, the key point feature data can include key point depth information, and the key point depth information can be determined by the following steps:

[0055] Monocular depth estimation can be performed on the first image to determine a first depth image, and the first key point depth information of the first image key point can be determined according to the first depth image; and monocular depth estimation can be performed on the second image to determine a second depth image, and the second key point depth information of the second image key point can be determined according to the second depth image.

[0056] In an exemplary embodiment, the key point feature data can include key point depth statistical data, which can be gradient statistical data corresponding to depth values of surrounding pixel points of the current image key point. For example, the key point depth statistical data can be variance data of the depth values of the surrounding pixel points, or can be histogram data, and the present exemplary embodiment does not make special limitation thereto.

[0057] Specifically, the key point depth statistical data can be determined by the following steps:

[0058] The first neighboring depth values corresponding to the neighboring pixel points corresponding to the first image key point can be determined according to the first depth image, and the first key point depth statistical data of the first image key point can be determined based on the first neighboring depth values; and the second neighboring depth values corresponding to the neighboring pixel points corresponding to the second image key point can be determined according to the second depth image, and the second key point depth statistical data of the second image key point can be determined based on the second neighboring depth values.

[0059] In an exemplary embodiment, the fusion distance feature between the image key points can be determined based on the key point feature data by the steps in Figure 5 , and reference is made to FIG. 4, which specifically can include: Figure 5

[0060] ​Step S510, determining descriptor distance data according to the first key point feature descriptor of the first image key point and the second key point feature descriptor corresponding to the second image key point;

[0061] Step S520, determining depth distance data according to the first key point depth information of the first image key point and the second key point depth information of the second image key point;

[0062] Step S530, determining depth statistical distance data according to the first key point depth statistical data of the first image key point and the second key point depth statistical data of the second image key point;

[0063] Step S540, determining the fusion distance feature between the first image key point and the second image key point based on the descriptor distance data, the depth distance data and the depth statistical distance data.

[0064] Wherein, the descriptor distance data can be data for measuring the size or length between the first key point feature descriptor of the first image key point and the second key point feature descriptor corresponding to the second image key point; the depth distance data can be data for measuring the size or length between the first key point depth information of the first image key point and the second key point depth information of the second image key point; and the depth statistical distance data can be data for measuring the size or length between the first key point depth statistical data of the first image key point and the second key point depth statistical data of the second image key point.

[0065] For example, the first image and the second image can be obtained by the main camera and the auxiliary camera in the rear camera module of the terminal device; the image key points in the first image and the second image can be detected by using the key point detection algorithm, and are respectively marked as the first image key points X={x1,...,xM} and the second image key points Y={y1,...,yN}, wherein M and N can respectively represent the number of the first image key points and the number of the second image key points; meanwhile, the key point feature descriptor corresponding to each image key point can be respectively represented as M} and N} respectively.

[0066] The depth information of each pixel point in the first image and the second image can be estimated by using the monocular depth estimation network, and the key point depth information of each image key point is marked as } and

[0067] ​The depth statistics of the neighboring pixels of each image key point can be calculated based on the monocular depth images corresponding to the first image and the second image, such as variance statistics, and the depth statistics of the key points around each image key point can be marked as and

[0068] After obtaining the key point feature descriptor, key point depth information and key point depth statistics, the distance between the image key points in the two images is calculated by fusing the monocular depth information (i.e. key point depth information and key point depth statistics) and the key point feature descriptor information, i.e. the fused distance feature. i and the keypoint y in the second image j , then the key point x i and key point y j The descriptor distance data between can be defined as the relationship (1):

[0069]

[0070] in, It can be expressed as descriptor distance data, Can represent the key point x i The corresponding key point feature descriptor, Can represent the key point y j The corresponding key point feature descriptor.

[0071] Key point x i and key point y j The depth distance data between can be defined as the relationship (2):

[0072]

[0073] in, Can be expressed as depth distance data, Can represent the key point x i The corresponding key point depth information, Can represent the key point y j The corresponding key point depth information.

[0074] Key point x i and key point y j The depth statistical distance data between can be defined as the relationship (3):

[0075]

[0076] in, It can be expressed as depth statistical distance data, Can represent the key point xi corresponding key point depth statistics, The key point y j corresponding key point depth statistics.

[0077] The three distance data in the relationship (1)-(3) can be fused to obtain the fused distance feature between the image key points, which can be represented by relationship (4):

[0078]

[0079] Wherein, η can represent the weight coefficient, which can be set by statistical analysis of a large amount of actual data, and is used to control the role of the sub-distance data and the depth distance data ; and the depth statistical distance data is mainly used to overcome the difficulty of depth estimation in the area without texture or repeated texture. If the key point is located in the area without texture or repeated texture, the variance is small, at this time, the reliability of the key point feature descriptor is low, and the role of the depth information in establishing the corresponding relationship can be improved; in the area with rich texture, the variance is large, and the matching corresponding relationship between the image key points is mainly determined by the feature descriptor; the fused distance feature adopts the addition method, which requires that the sub-distance data and the depth distance data between the image key points are close, which further improves the standard of establishing the corresponding relationship between the image key points; exp() can be an exponential function with natural constant e as the base, and σ can represent the normal distribution parameter.

[0080] In an example embodiment, the matching relationship matrix between the image key points can be determined according to the fused distance feature; the target internal and external parameters are determined based on the matching relationship matrix, and the first image and the second image are aligned in pixels based on the target internal and external parameters.

[0081] Specifically, the matching relationship matrix between the image key points can be determined according to the fused distance feature, and the matching corresponding relationship between the image key points can be calculated according to the nearest neighbor criterion or the Hungarian algorithm and other matching algorithms, and then the target internal and external parameters can be determined based on the matching corresponding relationship between the image key points and the epipolar constraint through the Gauss-Newton method or the LM algorithm, and the first image and the second image are aligned in pixels based on the target internal and external parameters.

[0082] In an example embodiment, the key point depth information corresponding to the image key points can be obtained; the image key points are clustered based on the key point depth information, and at least one image key point region is determined; the first image and the second image are aligned in pixels based on the target internal and external parameters.

[0083] By clustering the image key points through the key point depth information, and taking the image key point region after clustering as a unit, the first image and the second image are aligned in pixels based on the target internal and external parameters, the image transformation of the local region can be realized, and abnormal distortion of other regions caused by global alignment during the alignment in rows is avoided, which leads to errors.

[0084] In summary, in the example embodiment, the image key points corresponding to the first image and the second image can be determined, and then the key point feature data corresponding to each image key point can be determined, and the fusion distance feature between the image key points can be determined based on the key point feature data. Then, the first image and the second image can be aligned in pixels according to the fusion distance feature, and the depth estimation image can be determined based on the first image and the second image after the alignment in pixels. Through the fusion distance feature determined based on the key point feature data, the accuracy of the alignment in pixels of the first image and the second image can be effectively improved, thereby improving the accuracy of the depth estimation image.

[0085] The binocular depth estimation method proposed by the present disclosure is mainly applied to the key point matching step in the row alignment technology in the binocular depth estimation module. By fusing the key point information and the monocular depth information, the key point matching accuracy can be significantly improved, thereby completing the precise row alignment task and providing a good foundation for the subsequent depth calculation step, which helps to improve the virtualization effect and product competitiveness.

[0086] Due to the obvious progress and development of monocular depth estimation, the depth information in the image can be accurately and quickly obtained, especially in the textureless and repeated texture region, smooth and stable depth information can be provided. The binocular depth estimation method proposed by the present disclosure fuses the monocular depth information and the descriptor information of the key points, adaptively adjusts the status of the two features in calculating the matching corresponding relationship between the image key points and the image key points, and cooperatively determines the corresponding relationship, which can significantly improve the registration accuracy of the image key points, thereby completing the precise row alignment task.

[0087] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the example embodiments of the present disclosure, and are not for limitation purposes. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0088] Further, referring to Figure 6 In the example embodiment, a binocular depth estimation device 600 is also provided, which includes an image key point determination module 610, a fusion distance feature determination module 620, and a pixel alignment module 630. Wherein:

[0089] The image key point determination module 610 is configured to acquire a first image and a second image, and determine image key points corresponding to the first image and the second image respectively.

[0090] The fusion distance feature determination module 620 is configured to determine key point feature data corresponding to the image key points, and determine fusion distance features between the image key points based on the key point feature data.

[0091] The pixel alignment module 630 is configured to perform pixel alignment on the first image and the second image according to the fusion distance features, and determine a depth estimation image based on the pixel-aligned first image and second image.

[0092] In an example embodiment, the image key points can include first image key points corresponding to the first image and second image key points corresponding to the second image, and the key point feature data can include key point feature descriptors.

[0093] The fusion distance feature determination module 620 can be configured to:

[0094] construct first key point feature descriptors of the first image key points;

[0095] construct second key point feature descriptors of the second image key points;

[0096] The key point feature descriptors can include any one of a histogram of oriented gradients descriptor, a scale-invariant feature transform descriptor, a robust feature descriptor, and a speeded-up robust features descriptor.

[0097] In an example embodiment, the key point feature data can include key point depth information, and the fusion distance feature determination module 620 can be configured to:

[0098] perform monocular depth estimation on the first image to determine a first depth image, and determine first key point depth information of the first image key points according to the first depth image;

[0099] perform monocular depth estimation on the second image to determine a second depth image, and determine second key point depth information of the second image key points according to the second depth image.

[0100] In an example embodiment, the key point feature data can include key point depth statistical data, and the fusion distance feature determination module 620 can be configured to:

[0101] determine first neighboring depth values corresponding to neighboring pixel points corresponding to the first image key points according to the first depth image, and determine first key point depth statistical data of the first image key points based on the first neighboring depth values;

[0102] determine a second neighboring depth value corresponding to a neighboring pixel point corresponding to the second image key point according to the second depth image, and determine second key point depth statistical data of the second image key point based on the second neighboring depth value.

[0103] In an example embodiment, the fusion distance feature determination module 620 can be configured to:

[0104] determine descriptor distance data according to the first key point feature descriptor of the first image key point and the second key point feature descriptor corresponding to the second image key point;

[0105] determine depth distance data according to the first key point depth information of the first image key point and the second key point depth information of the second image key point;

[0106] determine depth statistical distance data according to the first key point depth statistical data of the first image key point and the second key point depth statistical data of the second image key point;

[0107] determine the fusion distance feature between the first image key point and the second image key point based on the descriptor distance data, the depth distance data and the depth statistical distance data.

[0108] In an example embodiment, the pixel alignment module 630 can be configured to:

[0109] determine a matching relationship matrix between the image key points according to the fusion distance feature;

[0110] determine target intrinsic and extrinsic parameters based on the matching relationship matrix, and perform row alignment of pixels of the first image and the second image based on the target intrinsic and extrinsic parameters.

[0111] In an example embodiment, the pixel alignment module 630 can be configured to:

[0112] obtain key point depth information corresponding to the image key points;

[0113] cluster the image key points based on the key point depth information, and determine at least one image key point region;

[0114] perform row alignment of pixels of the first image and the second image based on the target intrinsic and extrinsic parameters in units of the image key point region.

[0115] The specific details of each module in the above apparatus have been described in detail in the method part embodiments, and the undisclosed details can be referred to the embodiment contents of the method part, thus no further description is given.

[0116] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied in the form of a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation of hardware and software in combination, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0117] The exemplary embodiments of the present disclosure provide an electronic device for implementing a binocular depth estimation method, which can be a terminal device 101, 102, 103 or a server 105 in Figure 1 The electronic device at least includes a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the binocular depth estimation method by executing the executable instructions.

[0118] The following will take the electronic device 700 in Figure 7 as an example to exemplarily illustrate the structure of the electronic device in the present disclosure. Figure 7 The electronic device 700 shown is only an example, and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

[0119] As shown in Figure 7 , the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 can include but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), a display unit 740.

[0120] Among them, the storage unit 720 stores program code, which can be executed by the processing unit 710, so that the processing unit 710 executes the binocular depth estimation method in the present disclosure.

[0121] The storage unit 720 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 721 and / or a cache memory unit 722, and can further include a read-only memory (ROM) 723.

[0122] The storage unit 720 can also include program / utilities 724 with a set of (at least one) program modules 725, such as operating systems, one or more application programs, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0123] Bus 730 can be one of several types of bus structures including a storage bus or a memory bus, a peripheral bus, a graphics bus, a processor bus, or a local bus using any of a variety of bus architectures.

[0124] Electronic device 700 can also communicate with one or more external devices 770, such as a sensor device, a Bluetooth device, or the like, and / or one or more accessories coupled thereto, such as a headphone or the like. Additionally, electronic device 700 can communicate with one or more electronic devices 770, such as a sensor device, a Bluetooth device, or the like, and / or one or more accessories coupled thereto, such as a headphone or the like. Furthermore, electronic device 700 can include one or more input / output (I / O) interfaces 750 for wired or wireless communication with other devices. Electronic device 700 can also include one or more network interfaces 760 for communicating with one or more networks, such as a local-area network (LAN), a wide-area network (WAN), and / or the Internet, for example. As illustrated, network interface 760 is in communication with the other modules of electronic device 700 through bus 730. It should be appreciated that other hardware and / or software modules can be used in conjunction with electronic device 700, such as microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archival storage systems, and the like, as well as sensors modules (e.g., gyroscopes, magnetic sensors, acceleration sensors, distance sensors, proximity light sensors, and the like).

[0125] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, or the like) or a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the disclosure.

[0126] The example embodiments of the disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the methods described above in the specification. In some possible embodiments, various aspects of the disclosure can also be implemented in the form of a program product including a program code for causing an end device to perform the steps described in the above “Example Method” section according to various example embodiments of the disclosure when the program product is run on the end device.

[0127] Note that the computer-readable medium shown in the disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In the disclosure, the computer-readable signal medium can include a data signal that propagates in a baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal can take many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can be used to carry or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0129] In addition, the program code for carrying out operations of the disclosure can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., and a conventional procedural programming language such as the "C" programming language or a similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0130] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0131] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A binocular depth estimation method, characterized in that: include: Acquire a first image and a second image, and determine image key points corresponding to each of the first image and the second image; The image key points include first image key points corresponding to the first image and second image key points corresponding to the second image, and the key point feature data include key point feature descriptors, key point depth information, and key point depth statistics; Determining key point feature data corresponding to each of the image key points, and determining fusion distance features between the image key points based on the key point feature data; performing pixel alignment on the first image and the second image according to the fused distance feature, and determining a depth estimation image based on the pixel-aligned first image and the second image; The determining of the fusion distance feature between the image key points based on the key point feature data includes: Determining descriptor distance data based on a first key point feature descriptor of the first image key point and a second key point feature descriptor corresponding to the second image key point; Determining depth distance data based on first keypoint depth information of the first image keypoint and second keypoint depth information of the second image keypoint; determining depth statistical distance data based on first keypoint depth statistical data of the first image keypoint and second keypoint depth statistical data of the second image keypoint; A fused distance feature between the first image key point and the second image key point is determined based on the descriptor distance data, the depth distance data, and the depth statistical distance data.

2. The method according to claim 1, characterized in that The determining of key point feature data corresponding to each of the image key points includes: Constructing a first key point feature descriptor of the first image key point; Constructing a second key point feature descriptor of the second image key point; The key point feature descriptor includes any one of a histogram of oriented gradients descriptor, a scale-invariant feature transformation descriptor, a robust feature descriptor, and a fast feature extraction descriptor.

3. The method according to claim 2, characterized in that The determining of key point feature data corresponding to each of the image key points includes: Performing monocular depth estimation on the first image to determine a first depth image, and determining first key point depth information of key points of the first image based on the first depth image; Performing monocular depth estimation on the second image to determine a second depth image, and determining second key point depth information of key points of the second image based on the second depth image.

4. The method according to claim 3, characterized in that The determining of key point feature data corresponding to each of the image key points includes: Determining, according to the first depth image, first neighboring depth values ​​corresponding to neighboring pixels corresponding to the first image key point, and determining first key point depth statistics of the first image key point based on the first neighboring depth values; Determine second neighboring depth values ​​corresponding to neighboring pixels corresponding to the second image key point according to the second depth image, and determine second key point depth statistics of the second image key point based on the second neighboring depth values.

5. The method according to claim 1, wherein The performing pixel alignment on the first image and the second image according to the fusion distance feature includes: Determine a matching relationship matrix between the image key points according to the fusion distance feature; The internal and external parameters of the target are determined based on the matching relationship matrix, and the first image and the second image are aligned in pixel rows based on the internal and external parameters of the target.

6. The method according to claim 5, characterized in that The performing pixel row alignment on the first image and the second image based on the target internal and external parameters includes: Obtaining key point depth information corresponding to the image key point; Clustering the image key points based on the key point depth information to determine at least one image key point area; Taking the image key point area as a unit, pixel rows of the first image and the second image are aligned based on the target internal and external parameters.

7. A binocular depth estimation device, characterized in that: include: An image key point determination module, configured to acquire a first image and a second image, and determine image key points corresponding to each of the first image and the second image; The image key points include first image key points corresponding to the first image and second image key points corresponding to the second image, and the key point feature data include key point feature descriptors, key point depth information, and key point depth statistics; a fusion distance feature determination module, configured to determine key point feature data corresponding to each of the image key points, and determine a fusion distance feature between the image key points based on the key point feature data; a pixel alignment module, configured to perform pixel alignment on the first image and the second image according to the fused distance feature, and determine a depth estimation image based on the pixel-aligned first image and the second image; The method of determining the fusion distance feature between the image key points based on the key point feature data includes: determining descriptor distance data based on the first key point feature descriptor of the first image key point and the second key point feature descriptor corresponding to the second image key point; determining depth distance data based on the first key point depth information of the first image key point and the second key point depth information of the second image key point; determining depth statistical distance data based on the first key point depth statistics of the first image key point and the second key point depth statistics of the second image key point; and determining the fusion distance feature between the first image key point and the second image key point based on the descriptor distance data, the depth distance data and the depth statistical distance data.

8. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 6 by executing the executable instructions.

Images