A deep learning active binocular vision measurement method and device

Abstract

The present invention proposes a deep learning active binocular vision measurement method and device, belonging to the field of sensor technology. The method comprises: using cameras 1 and 2 of the active binocular vision measurement system to simultaneously capture images of the measured scene after a coded pattern is projected; based on the captured images, using a binocular prediction network based on a cost volume to calculate the probabilistic cost volume and initial disparity prediction of the measured scene to generate a disparity confidence level and thereby obtain a confidence mask; using a normal prediction network to obtain a normal prediction; using an object contour detection network to obtain a contour detection prediction; and obtaining a final disparity prediction through global optimization calculation to obtain a three-dimensional measurement result of the measured scene. The present invention utilizes confidence levels to globally optimize and complement the deep learning active binocular disparity prediction, achieving complete and accurate three-dimensional measurement of objects with complex optical characteristics, thus having high application value.

Description

Technical Field

[0001] The present invention belongs to the field of sensor technology, and in particular relates to a deep learning active binocular vision measurement method and device. Background Art

[0002] Transparent, translucent, and reflective objects are collectively referred to as objects with complex optical properties. Deep learning active binocular vision measurement methods use neural networks to extract features from images captured by two cameras and perform matching calculations to achieve depth measurement. This method offers the advantages of non-contact, simple structure, low cost, and high spatial resolution. However, the technology's drawback is that it struggles to extract effective image features in areas with complex optical properties, making accurate matching calculations impossible. This reduces depth measurement accuracy and makes it unsuitable for depth measurement of objects with complex optical properties. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies by proposing a deep learning active binocular vision measurement method and device. This method improves the measurement accuracy of objects with complex optical properties using deep learning active binocular vision measurement methods. By using confidence levels to globally optimize and complete the deep learning active binocular disparity predictions, this method enables complete and accurate three-dimensional measurement of objects with complex optical properties. This method is applicable to fields such as robotic grasping and industrial quality inspection, and has high application value.

[0004] The first embodiment of the present invention provides a deep learning active binocular vision measurement method, comprising:

[0005] Using cameras 1 and 2 of the active binocular vision measurement system to simultaneously capture images of the measured scene after the set coding pattern is projected, which are recorded as image 1 and image 2 respectively; wherein the measured scene is in the measurement field of view of the active binocular vision measurement system;

[0006] Inputting the image 1 and the image 2 simultaneously into a pre-trained binocular disparity prediction network based on a cost volume to obtain a probabilistic cost volume and an initial disparity prediction corresponding to the measured scene, so as to obtain a disparity confidence corresponding to the measured scene;

[0007] Inputting the image 1 and the image 2 into the pre-trained normal prediction network at the same time to obtain the normal prediction corresponding to the measured scene;

[0008] Input the image 1 into a pre-trained object contour detection network to obtain a contour detection prediction corresponding to the detected scene;

[0009] Calculating a confidence mask based on the disparity confidence;

[0010] Calculating a completed disparity prediction corresponding to the detected scene based on the initial disparity prediction, the normal prediction, and the contour detection prediction;

[0011] A final disparity prediction is calculated based on the initial disparity prediction, the confidence mask, and the completed disparity prediction to obtain a three-dimensional measurement result of the measured scene.

[0012] In a specific embodiment of the present invention, the coding pattern is a random speckle pattern projected by a structured light projection module of the active binocular vision measurement system.

[0013] In a specific embodiment of the present invention, the measured scene includes objects with complex optical characteristics.

[0014] In a specific embodiment of the present invention, the method further comprises:

[0015] The intrinsic parameter matrices of the camera 1 and the camera 2 and the coordinate transformation matrix between the camera 1 and the camera 2 are obtained through calibration.

[0016] In a specific embodiment of the present invention, the cost volume disparity sequence of the cost volume-based binocular disparity prediction network is d=[d0+Δd*i], i=1, 2, ..., D, where d0 is the minimum disparity, Δd is the difference between two adjacent disparities in the disparity sequence, and D is the length of the disparity sequence;

[0017] The probability cost is denoted as The initial disparity prediction is denoted as H and W are the vertical and horizontal resolutions of image 1, respectively.

[0018] In a specific embodiment of the present invention, obtaining the disparity confidence corresponding to the measured scene includes:

[0019] The disparity confidence C(i,j) of any coordinate (i,j), i=1,2,...,H,j=1,2,...,W in the disparity confidence is calculated as:

[0020]

[0021] Where P(i, j, l) is the value of the probability cost volume P at the coordinate (i, j, l);

[0022] D0(i,j) is the value of the initial disparity prediction D0 at the coordinate (i,j), and k is the third-dimensional coordinate corresponding to D0(i,j) in the probability cost volume P; is the rounding symbol, c is the neighborhood range selected when calculating the disparity confidence, and c is a positive integer;

[0023] After calculating C(i, j) corresponding to all coordinates, the disparity confidence C is obtained.

[0024] In a specific embodiment of the present invention, calculating a confidence mask based on the disparity confidence includes:

[0025] For any coordinate (i, j) in the confidence mask, i = 1, 2, ..., H, j = 1, 2, ..., W, the confidence U (i, j) is calculated as follows:

[0026]

[0027] Among them, μ c is the confidence threshold, and its value range is (0,1);

[0028] After calculating U(i, j) corresponding to all coordinates, the confidence mask U is obtained.

[0029] In a specific embodiment of the present invention, the calculating the complementary disparity prediction corresponding to the measured scene includes:

[0030]

[0031] Where:

[0032]

[0033]

[0034]

[0035] NE((i,j),(i′,j′))=||<v((i,j),(i′,j′)),N(i,j)>·E(i,j)|| 2

[0036] Where, is the complete disparity prediction corresponding to the tested scene, E D is the parallax distance energy function, E S is the disparity continuous energy function, E N is the normal difference energy function, NE((i,j),(i′,j′)) is the normal difference between two pixel coordinates (i,j),(i′,j′), Neigh(i,j) is the neighborhood of pixel coordinate (i,j); λ D is the parallax distance energy function weight, λ S is the weight of the disparity continuous energy function, λ N is the normal difference energy function weight, λ D ,λ S ,λ Nis a positive number;

[0037] D(i,j) is the value of the disparity prediction variable D at (i,j); N(i,j) is the value of the normal prediction N at (i,j) corresponding to the measured scene, E(i,j) is the value predicted by the E contour detection at (i,j) corresponding to the tested scene,

[0038] In a specific embodiment of the present invention, the calculating the final disparity prediction includes:

[0039]

[0040] Among them, D * is the final disparity prediction.

[0041] The second embodiment of the present invention provides a deep learning active binocular vision measurement device, comprising:

[0042] An image acquisition module is configured to project a set coded pattern onto a scene under test by means of a structured light projection module of the active binocular vision measurement system, and then simultaneously capture images of the scene under test using cameras 1 and 2 of the active binocular vision measurement system, which are recorded as image 1 and image 2, respectively; the scene under test is in a common field of view of the structured light projection module, the camera 1, and the camera 2;

[0043] An initial disparity prediction module is used to input the image 1 and the image 2 simultaneously into a pre-trained binocular disparity prediction network based on a cost volume, obtain a probabilistic cost volume and an initial disparity prediction corresponding to the measured scene, and obtain a disparity confidence corresponding to the measured scene;

[0044] A normal prediction module, configured to simultaneously input the image 1 and the image 2 into a pre-trained normal prediction network to obtain a normal prediction corresponding to the measured scene;

[0045] A contour detection prediction module, configured to input the image 1 into a pre-trained object contour detection network to obtain a contour detection prediction corresponding to the detected scene;

[0046] A confidence mask module, configured to calculate a confidence mask based on the disparity confidence;

[0047] A supplementary disparity prediction module, configured to calculate a supplementary disparity prediction corresponding to the detected scene based on the initial disparity prediction, the normal prediction, and the contour detection prediction;

[0048] The three-dimensional measurement module is configured to calculate a final disparity prediction based on the initial disparity prediction, the confidence mask, and the completed disparity prediction, so as to obtain a three-dimensional measurement result of the measured scene.

[0049] In a specific embodiment of the present invention, the device further comprises:

[0050] The calibration module is used to obtain the intrinsic parameter matrices of the camera 1 and the camera 2, and the coordinate transformation matrix between the camera 1 and the camera 2 through calibration.

[0051] In a specific embodiment of the present invention, the cost volume disparity sequence of the cost volume-based binocular disparity prediction network is d=[d0+Δd*i], i=1, 2, ..., D, where d0 is the minimum disparity, Δd is the difference between two adjacent disparities in the disparity sequence, and D is the length of the disparity sequence;

[0052] The probability cost is denoted as The initial disparity prediction is denoted as H and W are the vertical and horizontal resolutions of image 1, respectively.

[0053] In a specific embodiment of the present invention, obtaining the disparity confidence corresponding to the measured scene includes:

[0054] The disparity confidence C(i,j) of any coordinate (i,j), i=1,2,...,H,j=1,2,...,W in the disparity confidence is calculated as:

[0055]

[0056] Where P(i, j, l) is the value of the probability cost volume P at the coordinate (i, j, l);

[0057] D0(i,j) is the value of the initial disparity prediction D0 at the coordinate (i,j), and k is the third-dimensional coordinate corresponding to D0(i,j) in the probability cost volume P; is the rounding symbol, c is the neighborhood range selected when calculating the disparity confidence, and c is a positive integer;

[0058] After calculating C(i, j) corresponding to all coordinates, the disparity confidence C is obtained.

[0059] In a specific embodiment of the present invention, calculating a confidence mask based on the disparity confidence includes:

[0060] For any coordinate (i, j) in the confidence mask, i = 1, 2, ..., H, j = 1, 2, ..., W, the confidence U (i, j) is calculated as follows:

[0061]

[0062] Among them, μc is the confidence threshold, and its value range is (0,1);

[0063] After calculating U(i, j) corresponding to all coordinates, the confidence mask U is obtained.

[0064] In a specific embodiment of the present invention, the calculating the complementary disparity prediction corresponding to the measured scene includes:

[0065]

[0066] Where:

[0067]

[0068]

[0069]

[0070] NE((i,j),(i′,j′))=||<v((i,j),(i′,j′)),N(i,j)>·E(i,j)|| 2

[0071] Where, is the complete disparity prediction corresponding to the tested scene, E D is the parallax distance energy function, E S is the disparity continuous energy function, E N is the normal difference energy function, NE((i,j),(i′,j′)) is the normal difference between two pixel coordinates (i,j),(i′,j′), Neigh(i,j) is the neighborhood of pixel coordinate (i,j); λ D is the parallax distance energy function weight, λ S is the weight of the disparity continuous energy function, λ N is the normal difference energy function weight, λ D ,λ S ,λ N is a positive number;

[0072] D(i,j) is the value of the disparity prediction variable D at (i,j); N(i,j) is the value of the normal prediction N at (i,j) corresponding to the measured scene, E(i,j) is the value predicted by the E contour detection at (i,j) corresponding to the tested scene,

[0073] In a specific embodiment of the present invention, the calculating the final disparity prediction includes:

[0074]

[0075] Among them, D * is the final disparity prediction.

[0076] A third embodiment of the present invention provides an electronic device, including:

[0077] at least one processor; and a memory communicatively coupled to the at least one processor;

[0078] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned deep learning active binocular vision measurement method.

[0079] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned deep learning active binocular vision measurement method.

[0080] Features and beneficial effects of the present invention:

[0081] The present invention solves the problem of decreased accuracy in areas with complex optical properties of active binocular vision measurement methods based on deep learning. The present invention is based on an active binocular vision measurement system, in which a structured light projection module projects a structured light pattern onto the measured scene, and two cameras capture the structured light pattern reflected by the measured scene; a binocular prediction network based on a cost body is used to generate disparity confidence and initial disparity prediction, and the normal and object contour are predicted through deep learning; a confidence mask is calculated using the disparity confidence; and the final disparity prediction is obtained through global optimization calculation using the initial disparity prediction, confidence mask, normal prediction, and contour detection prediction. The present invention uses disparity confidence to automatically identify areas with poor accuracy in the deep learning binocular vision measurement results and deletes the measurement results in this area, and complements the depth of this area through normal prediction and object contour prediction, thereby improving the measurement accuracy of this area. According to this invention, complete and accurate three-dimensional measurement of objects with complex optical properties can be achieved, which can promote the development of related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is an overall flow chart of a deep learning active binocular vision measurement method in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The present invention proposes a deep learning active binocular vision measurement method and device, which is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0084] The first embodiment of the present invention provides a deep learning active binocular vision measurement method, comprising:

[0085] Using cameras 1 and 2 of the active binocular vision measurement system to simultaneously capture images of the measured scene after the set coding pattern is projected, which are recorded as image 1 and image 2 respectively; wherein the measured scene is in the measurement field of view of the active binocular vision measurement system;

[0086] Inputting the image 1 and the image 2 simultaneously into a pre-trained binocular disparity prediction network based on a cost volume to obtain a probabilistic cost volume and an initial disparity prediction corresponding to the measured scene, so as to obtain a disparity confidence corresponding to the measured scene;

[0087] Inputting the image 1 and the image 2 into the pre-trained normal prediction network at the same time to obtain the normal prediction corresponding to the measured scene;

[0088] Input the image 1 into a pre-trained object contour detection network to obtain a contour detection prediction corresponding to the detected scene;

[0089] Calculating a confidence mask based on the disparity confidence;

[0090] Calculating a completed disparity prediction corresponding to the detected scene based on the initial disparity prediction, the normal prediction, and the contour detection prediction;

[0091] A final disparity prediction is calculated based on the initial disparity prediction, the confidence mask, and the completed disparity prediction to obtain a three-dimensional measurement result of the measured scene.

[0092] In a specific embodiment of the present invention, the overall process of the deep learning active binocular vision measurement method is as follows: Figure 1 As shown, the following steps are included:

[0093] 1) Build an active binocular vision measurement system and perform calibration.

[0094] In this embodiment, the active binocular vision measurement system includes a structured light projection module and two cameras, so that the measured scene is completely within the common field of view of the structured light projection module and the two cameras. The common field of view is the measurement field of view of the active binocular vision measurement system. The structured light projection module is used to project a coded pattern onto the measured scene, and the cameras are used to capture the coded pattern projected by the structured light projection module.

[0095] It should be noted that there are no specific requirements for the placement order of the structured light projection module and the two cameras. In this embodiment, the two cameras are designated as Camera 1 and Camera 2, respectively. Camera 1 and Camera 2 can be determined independently. In a specific embodiment of the present invention, the positional relationship between the structured light projection module and the cameras is required to ensure a public field of view of no less than 30 cm × 30 cm × 20 cm.

[0096] In this embodiment, both the structured light projection module and the camera can be conventional models, and the two cameras do not need to be identical models. The structured light projection module is required to be able to project a random speckle pattern. In one specific embodiment of the present invention, the structured light projection module is a vertical cavity surface emitting laser. The camera is required to have a resolution of no less than 500×500 pixels. In one specific embodiment of the present invention, the camera model used is the OV2740, with a resolution of 1920×1080 pixels.

[0097] The intrinsic parameter matrices of the two cameras and the coordinate transformation matrix between the two cameras are obtained through system calibration.

[0098] 2) Using the structured light projection module in the active binocular vision measurement system constructed in step 1), a random speckle pattern is projected onto a measured scene. In this embodiment, the measured scene may include multiple objects to be measured, including ordinary objects and objects with complex optical properties. Camera 1 and camera 2 are used to simultaneously capture an image containing the projected random speckle pattern, which are recorded as image 1 and image 2, respectively.

[0099] In a specific embodiment of the present invention, the objects to be tested in the tested scene include objects three-dimensionally printed with gypsum powder, and various objects with complex optical properties, such as metal balls, transparent mineral water bottles, etc.

[0100] 3) Input image 1 and image 2 into the pre-trained binocular disparity prediction network based on cost volume at the same time to obtain the probability cost volume corresponding to the measured scene and initial disparity prediction H and W are the vertical resolution and horizontal resolution of image 1 respectively (the vertical resolution and horizontal resolution of image 2 can be different from those of image 1, but are generally the same), and D is the length of the disparity sequence.

[0101] The cost volume disparity sequence of the network is d = [d0 + Δd*i], i = 1, 2, ..., D, where d0 is the minimum disparity and Δd is the difference between two adjacent disparities in the disparity sequence. In a specific embodiment of the present invention, d0 = 16, Δd = 0.5, and D = 192.

[0102] In this embodiment, the cost-volume-based binocular disparity prediction network structure is from Chang, Jia-Ren, and Yong-Sheng Chen. "Pyramid stereo matching network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. The pre-trained binocular disparity prediction network can be obtained from Liu I, Yang E, Tao J, et al. ActiveZero: Mixed Domain Learning for Active Stereovision with Zero Annotation [C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2022: 13033-13042.

[0103] 4) Input image 1 and image 2 into the pre-trained normal prediction network at the same time to obtain the normal prediction corresponding to the tested scene

[0104] The normal prediction network structure described in this embodiment is from Sajjan S, Moore M, Pan M, et al. Cleargrasp: 3D shape estimation of transparent objects for manipulation [C] / / 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020: 3634-3642. The pre-trained normal prediction network can be obtained from https: / / github.com / Shreeyak / cleargrasp Download and obtain.

[0105] 5) Input image 1 into the pre-trained object contour detection network to obtain the contour detection prediction corresponding to the detected scene

[0106] The object contour detection network structure described in this embodiment is from Sajjan S, Moore M, Pan M, et al. Clear grasp: 3D shape estimation of transparent objects for manipulation [C] / / 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020: 3634-3642. The pre-trained object contour detection network can be obtained from https: / / github.com / Shreeyak / cleargrasp 6) Use the probability cost volume P and the initial disparity prediction D0 to calculate the disparity confidence corresponding to the measured scene The specific method is as follows:

[0107] For any coordinate (i, j) in the disparity confidence C, i = 1, 2, ..., H, j = 1, 2, ..., W, the disparity confidence C(i, j) calculation expression is:

[0108]

[0109] Where P(i, j, l) is the value of the probability cost volume P at the coordinate (i, j, l).

[0110] D0(i,j) is the value of the initial disparity prediction D0 at the coordinate (i,j), and k is the third-dimensional coordinate corresponding to D0(i,j) in the probability cost volume P; is the rounding down symbol, c is the neighborhood range selected when calculating the disparity confidence, and c is a positive integer; in a specific embodiment of the present invention, c=2.

[0111] After calculating C(i, j) corresponding to all coordinates, the disparity confidence C is obtained.

[0112] 7) Calculate the confidence mask U using the disparity confidence C calculated in step 3).

[0113] Among them, for any coordinate (i, j) in the confidence mask U, i = 1, 2, ..., H, j = 1, 2, ..., W, the confidence U (i, j) calculation expression is:

[0114]

[0115] If the value of C(i,j) is greater than the preset confidence threshold μ c , then U(i,j)=1; otherwise U(i,j)=0. μ cis the confidence threshold, and its value range is (0,1). In a specific embodiment of the present invention, μ c The value is 0.7.

[0116] After calculating U(i, j) corresponding to all coordinates, the confidence mask U is obtained.

[0117] 8) Use global optimization to calculate the complete disparity prediction corresponding to the measured scene

[0118]

[0119] Where:

[0120]

[0121]

[0122]

[0123] NE((i,j),(i′,j′))=||<v((i,j),(i′,j′)),N(i,j)>·E(i,j)|| 2

[0124] Where, E D is the parallax distance energy function, E S is the disparity continuous energy function, E N is the normal difference energy function, NE((i,j),(i′,j′)) is the normal difference between two pixel coordinates (i,j) and (i′,j′), Neigh(i,j) is the neighborhood of the pixel coordinate (i,j). In a specific embodiment of the present invention, pixels with a distance less than 5 pixels are considered neighbors. D is the parallax distance energy function weight, λ S is the weight of the disparity continuous energy function, λ N is the weight of the normal difference energy function. D ,λ S ,λ N is a positive number. In a specific embodiment of the present invention, D =1,λ S =1,λ N =0.1.

[0125] D(i,j) is the value of the disparity prediction variable D at (i,j), and D is a temporary variable for the optimization calculation; N(i,j) is the value of the normal prediction N at (i,j) corresponding to the measured scene; E(i,j) is the value of the contour detection prediction E at (i,j) corresponding to the measured scene.

[0126] 9) Use the initial disparity prediction D0 calculated in step 3), the confidence mask U calculated in step 7), and the completed disparity prediction calculated in step 8) Calculate the final disparity prediction D * :

[0127]

[0128] The intrinsic parameter matrices of camera 1 and camera 2 obtained by system calibration, as well as the coordinate transformation matrix between the two cameras, can be used to convert D * Converted into the final three-dimensional measurement results of the measured scene.

[0129] To implement the above embodiment, a second embodiment of the present invention proposes a deep learning active binocular vision measurement device, comprising:

[0130] An image acquisition module is configured to simultaneously acquire images of the measured scene after the set coding pattern is projected using cameras 1 and 2 of the active binocular vision measurement system, which are respectively recorded as image 1 and image 2; wherein the measured scene is within the measurement field of view of the active binocular vision measurement system;

[0131] An initial disparity prediction module is used to input the image 1 and the image 2 simultaneously into a pre-trained binocular disparity prediction network based on a cost volume, obtain a probabilistic cost volume and an initial disparity prediction corresponding to the measured scene, and obtain a disparity confidence corresponding to the measured scene;

[0132] A normal prediction module, configured to simultaneously input the image 1 and the image 2 into a pre-trained normal prediction network to obtain a normal prediction corresponding to the measured scene;

[0133] A contour detection prediction module, configured to input the image 1 into a pre-trained object contour detection network to obtain a contour detection prediction corresponding to the detected scene;

[0134] A confidence mask module, configured to calculate a confidence mask based on the disparity confidence;

[0135] A supplementary disparity prediction module, configured to calculate a supplementary disparity prediction corresponding to the detected scene based on the initial disparity prediction, the normal prediction, and the contour detection prediction;

[0136] The three-dimensional measurement module is configured to calculate a final disparity prediction based on the initial disparity prediction, the confidence mask, and the completed disparity prediction, so as to obtain a three-dimensional measurement result of the measured scene.

[0137] It should be noted that the above explanation of an embodiment of a deep learning active binocular vision measurement method is also applicable to a deep learning active binocular vision measurement device of this embodiment, and will not be repeated here. According to a deep learning active binocular vision measurement device proposed in an embodiment of the present invention, after projecting a set coding pattern to the measured scene through the structured light projection module of the active binocular vision measurement system, the camera 1 and camera 2 of the active binocular vision measurement system are used to simultaneously capture images of the measured scene, which are recorded as image 1 and image 2 respectively; the measured scene is in the common field of view of the structured light projection module, the camera 1 and the camera 2; the image 1 and the image 2 are simultaneously input into the pre-trained binocular disparity prediction network based on the cost body, and the probability cost body and initial disparity prediction corresponding to the measured scene are obtained to obtain the measured scene. The method comprises the following steps: calculating the disparity confidence corresponding to the measured scene; inputting the image 1 and the image 2 simultaneously into a pre-trained normal prediction network to obtain a normal prediction corresponding to the measured scene; inputting the image 1 into a pre-trained object contour detection network to obtain a contour detection prediction corresponding to the measured scene; calculating a confidence mask based on the disparity confidence; calculating a completed disparity prediction corresponding to the measured scene based on the initial disparity prediction, the normal prediction, and the contour detection prediction; and calculating a final disparity prediction based on the initial disparity prediction, the confidence mask, and the completed disparity prediction to obtain a three-dimensional measurement result of the measured scene. This method can improve the measurement accuracy of objects with complex optical properties using the deep learning active binocular measurement method, achieve complete and accurate three-dimensional measurement of objects with complex optical properties, and is applicable to fields such as robot grasping and industrial quality inspection, with high application value.

[0138] In a specific embodiment of the present invention, the device further comprises:

[0139] The calibration module is used to obtain the intrinsic parameter matrices of the camera 1 and the camera 2, and the coordinate transformation matrix between the camera 1 and the camera 2 through calibration.

[0140] In a specific embodiment of the present invention, the cost volume disparity sequence of the cost volume-based binocular disparity prediction network is d=[d0+Δd*i], i=1, 2, ..., D, where d0 is the minimum disparity, Δd is the difference between two adjacent disparities in the disparity sequence, and D is the length of the disparity sequence;

[0141] The probability cost is denoted as The initial disparity prediction is denoted as H and W are the vertical and horizontal resolutions of image 1, respectively.

[0142] In a specific embodiment of the present invention, obtaining the disparity confidence corresponding to the measured scene includes:

[0143] The disparity confidence C(i,j) of any coordinate (i,j), i=1,2,...,H,j=1,2,...,W in the disparity confidence is calculated as:

[0144]

[0145] Where P(i, j, l) is the value of the probability cost volume P at the coordinate (i, j, l);

[0146] D0(i,j) is the value of the initial disparity prediction D0 at the coordinate (i,j), and k is the third-dimensional coordinate corresponding to D0(i,j) in the probability cost volume P; is the rounding symbol, c is the neighborhood range selected when calculating the disparity confidence, and c is a positive integer;

[0147] After calculating C(i, j) corresponding to all coordinates, the disparity confidence C is obtained.

[0148] In a specific embodiment of the present invention, calculating a confidence mask based on the disparity confidence includes:

[0149] For any coordinate (i, j) in the confidence mask, i = 1, 2, ..., H, j = 1, 2, ..., W, the confidence U (i, j) is calculated as follows:

[0150]

[0151] Among them, μ c is the confidence threshold, and its value range is (0,1);

[0152] After calculating U(i, j) corresponding to all coordinates, the confidence mask U is obtained.

[0153] In a specific embodiment of the present invention, the calculating the complementary disparity prediction corresponding to the measured scene includes:

[0154]

[0155] Where:

[0156]

[0157]

[0158]

[0159] NE((i,j),(i′,j′))=||<v((i,j),(i′,j′)),N(i,j)>·E(i,j)|| 2

[0160] Where, is the complete disparity prediction corresponding to the tested scene, E D is the parallax distance energy function, E S is the disparity continuous energy function, E N is the normal difference energy function, NE((i,j),(i′,j′)) is the normal difference between two pixel coordinates (i,j),(i′,j′), Neigh(i,j) is the neighborhood of pixel coordinate (i,j); λ D is the parallax distance energy function weight, λ S is the weight of the disparity continuous energy function, λ N is the normal difference energy function weight, λ D ,λ S ,λ N is a positive number;

[0161] D(i,j) is the value of the disparity prediction variable D at (i,j); N(i,j) is the value of the normal prediction N at (i,j) corresponding to the measured scene, E(i,j) is the value predicted by the E contour detection at (i,j) corresponding to the tested scene,

[0162] In a specific embodiment of the present invention, the calculating the final disparity prediction includes:

[0163]

[0164] Among them, D * is the final disparity prediction.

[0165] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0166] at least one processor; and a memory communicatively coupled to the at least one processor;

[0167] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above-mentioned deep learning active binocular vision measurement method.

[0168] To implement the above embodiment, the fourth aspect of the present invention proposes a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned deep learning active binocular vision measurement method.

[0169] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0170] The computer-readable medium may be included in the electronic device or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When executed by the electronic device, the one or more programs cause the electronic device to perform the deep learning active binocular vision measurement method described in the above embodiment.

[0171] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0172] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0173] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0174] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0175] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0176] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0177] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0178] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0179] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A deep learning active binocular vision measurement method, characterized in that: include: The cameras 1 and 2 of the active binocular vision measurement system are used to simultaneously capture images of the measured scene after the set coding pattern is projected, which are recorded as image 1 and image 2 respectively; The measured scene is in the measurement field of view of the active binocular vision measurement system; Inputting the image 1 and the image 2 simultaneously into a pre-trained binocular disparity prediction network based on a cost volume to obtain a probabilistic cost volume and an initial disparity prediction corresponding to the measured scene, so as to obtain a disparity confidence corresponding to the measured scene; Inputting the image 1 and the image 2 into the pre-trained normal prediction network at the same time to obtain the normal prediction corresponding to the measured scene; Input the image 1 into a pre-trained object contour detection network to obtain a contour detection prediction corresponding to the detected scene; Calculating a confidence mask based on the disparity confidence; Calculating a completed disparity prediction corresponding to the detected scene based on the initial disparity prediction, the normal prediction, and the contour detection prediction; Calculating a final disparity prediction based on the initial disparity prediction, the confidence mask, and the completed disparity prediction to obtain a three-dimensional measurement result of the measured scene; The cost volume disparity sequence of the cost volume-based binocular disparity prediction network is d=[d0+Δd*i], i=1, 2, ..., D, where d0 is the minimum disparity, Δd is the difference between two adjacent disparities in the disparity sequence, and D is the length of the disparity sequence; The probability cost is denoted as The initial disparity prediction is denoted as H and W are the vertical and horizontal resolutions of Image 1, respectively; Obtaining the disparity confidence corresponding to the measured scene includes: The disparity confidence C(i,j) of any coordinate (i,j), i=1,2,...,H,j=1,2,...,W in the disparity confidence is calculated as: Where P(i, j, l) is the value of the probability cost volume P at the coordinate (i, j, l); D0(i,j) is the value of the initial disparity prediction D0 at the coordinate (i,j), and k is the third-dimensional coordinate corresponding to D0(i,j) in the probability cost volume P; is the rounding symbol, c is the neighborhood range selected when calculating the disparity confidence, and c is a positive integer; After calculating C(i,j) corresponding to all coordinates, we get the disparity confidence C; The calculating the complementary disparity prediction corresponding to the measured scene includes: Where: NE((i,j),(i′,j′))=||<v((i,j),(i′,j′)),N(i,j)>·E(i,j)|| 2 Where, is the complete disparity prediction corresponding to the tested scene, E D is the parallax distance energy function, E S is the disparity continuous energy function, E N is the normal difference energy function, NE((i,j),(i′,j′)) is the normal difference between two pixel coordinates (i,j),(i′,j′), Neigh(i,j) is the neighborhood of pixel coordinate (i,j); λ D is the parallax distance energy function weight, λ S is the weight of the disparity continuous energy function, λ N is the normal difference energy function weight, λ D ,λ S ,λ N is a positive number; D(i,j) is the value of the disparity prediction variable D at (i,j); N(i,j) is the value of the normal prediction N at (i,j) corresponding to the measured scene, E(i,j) is the value predicted by the E contour detection at (i,j) corresponding to the tested scene, 2. The method according to claim 1, characterized in that The coding pattern is a random speckle pattern projected by the structured light projection module of the active binocular vision measurement system.

3. The method according to claim 1, characterized in that The measured scene contains objects with complex optical characteristics.

4. The method according to claim 1, wherein The method further comprises: The intrinsic parameter matrices of the camera 1 and the camera 2 and the coordinate transformation matrix between the camera 1 and the camera 2 are obtained through calibration.

5. The method according to claim 1, wherein The calculating of a confidence mask based on the disparity confidence comprises: For any coordinate (i, j) in the confidence mask, i = 1, 2, ..., H, j = 1, 2, ..., W, the confidence U (i, j) is calculated as follows: Among them, μ c is the confidence threshold, and its value range is (0,1); After calculating U(i, j) corresponding to all coordinates, the confidence mask U is obtained.

6. The method according to claim 5, characterized in that The calculating of the final disparity prediction comprises: Among them, D * is the final disparity prediction.

7. A deep learning active binocular vision measurement device, characterized in that: include: An image acquisition module is used to simultaneously acquire images of the measured scene after the set coding pattern is projected using cameras 1 and 2 of the active binocular vision measurement system, which are respectively recorded as image 1 and image 2; The measured scene is in the measurement field of view of the active binocular vision measurement system; An initial disparity prediction module is used to input the image 1 and the image 2 simultaneously into a pre-trained binocular disparity prediction network based on a cost volume, obtain a probabilistic cost volume and an initial disparity prediction corresponding to the measured scene, and obtain a disparity confidence corresponding to the measured scene; A normal prediction module, configured to simultaneously input the image 1 and the image 2 into a pre-trained normal prediction network to obtain a normal prediction corresponding to the measured scene; A contour detection prediction module, configured to input the image 1 into a pre-trained object contour detection network to obtain a contour detection prediction corresponding to the detected scene; A confidence mask module, configured to calculate a confidence mask based on the disparity confidence; A supplementary disparity prediction module, configured to calculate a supplementary disparity prediction corresponding to the detected scene based on the initial disparity prediction, the normal prediction, and the contour detection prediction; a three-dimensional measurement module, configured to calculate a final disparity prediction based on the initial disparity prediction, the confidence mask, and the completed disparity prediction, so as to obtain a three-dimensional measurement result of the measured scene; The cost volume disparity sequence of the cost volume-based binocular disparity prediction network is d=[d0+Δd*i], i=1, 2, ..., D, where d0 is the minimum disparity, Δd is the difference between two adjacent disparities in the disparity sequence, and D is the length of the disparity sequence; The probability cost is denoted as The initial disparity prediction is denoted as H and W are the vertical and horizontal resolutions of Image 1, respectively; Obtaining the disparity confidence corresponding to the measured scene includes: The disparity confidence C(i,j) of any coordinate (i,j), i=1,2,...,H,j=1,2,...,W in the disparity confidence is calculated as: Where P(i, j, l) is the value of the probability cost volume P at the coordinate (i, j, l); D0(i,j) is the value of the initial disparity prediction D0 at the coordinate (i,j), and k is the third-dimensional coordinate corresponding to D0(i,j) in the probability cost volume P; is the rounding symbol, c is the neighborhood range selected when calculating the disparity confidence, and c is a positive integer; After calculating C(i,j) corresponding to all coordinates, we get the disparity confidence C; The calculating the complementary disparity prediction corresponding to the measured scene includes: Where: NE((i,j),(i′,j′))=||<v((i,j),(i′,j′)),N(i,j)>·E(i,j)|| 2 Where, is the complete disparity prediction corresponding to the tested scene, E D is the parallax distance energy function, E S is the disparity continuous energy function, E N is the normal difference energy function, NE((i,j),(i′,j′)) is the normal difference between two pixel coordinates (i,j),(i′,j′), Neigh(i,j) is the neighborhood of pixel coordinate (i,j); λ D is the parallax distance energy function weight, λ S is the weight of the disparity continuous energy function, λ N is the normal difference energy function weight, λ D ,λ S ,λ N is a positive number; D(i,j) is the value of the disparity prediction variable D at (i,j); N(i,j) is the value of the normal prediction N at (i,j) corresponding to the measured scene, E(i,j) is the value predicted by the E contour detection at (i,j) corresponding to the tested scene, 8. The device according to claim 7, characterized in that The device further comprises: The calibration module is used to obtain the intrinsic parameter matrices of the camera 1 and the camera 2, and the coordinate transformation matrix between the camera 1 and the camera 2 through calibration.

9. The device according to claim 7, characterized in that The calculating of a confidence mask based on the disparity confidence comprises: For any coordinate (i, j) in the confidence mask, i = 1, 2, ..., H, j = 1, 2, ..., W, the confidence U (i, j) is calculated as follows: Among them, μ c is the confidence threshold, and its value range is (0,1); After calculating U(i, j) corresponding to all coordinates, the confidence mask U is obtained.

10. The device according to claim 9, characterized in that The calculating of the final disparity prediction comprises: Among them, D * is the final disparity prediction.

Images