Disparity map determination method, three-dimensional reconstruction method and related device

Abstract

The invention provides a disparity map determination method, a three-dimensional reconstruction method and a related device. The disparity map determination method comprises the following steps: acquiring images of a to-be-detected object under projection of a plurality of images with different illumination intensities acquired by a camera module in the camera equipment, and obtaining a plurality of structured light object images of the camera module; the camera shooting equipment at least comprises two camera shooting modules; based on the plurality of structured light object images of the camera module, separating out a specular reflection image and / or a diffuse reflection image of the camera module; searching matching points based on the specular reflection images and / or diffuse reflection images of the two camera modules to obtain a first matching point pair; and based on the first matching point pair, obtaining a disparity map of the two camera modules. According to the invention, the quality of the disparity map on the highly reflective surface can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a disparity map determination method, a three-dimensional reconstruction method and related devices. Background Art

[0002] In application scenarios such as 3D reconstruction and object detection and tracking, a disparity map between images captured by two camera modules in a camera device is determined. However, the quality of the disparity map on a highly reflective surface determined using a related disparity map determination scheme is relatively poor. Summary of the invention

[0003] The present application provides a disparity map determination method, a three-dimensional reconstruction method and related devices, which can improve the quality of the disparity map on a highly reflective surface.

[0004] To achieve the above object, the present application provides a disparity map determination method, the method comprising:

[0005] Acquire images of the object to be detected under the projection of multiple images with different light intensities captured by the camera module in the camera device, and obtain multiple structured light object images of the camera module; the camera device includes at least two camera modules;

[0006] Separating a specular reflection image and / or a diffuse reflection image of the camera module based on the multiple structured light object images of the camera module;

[0007] Searching for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules to obtain a first matching point pair;

[0008] Based on the first matching point pair, a disparity map of the two camera modules is obtained.

[0009] To achieve the above object, the present application provides a three-dimensional reconstruction method, which includes:

[0010] Determine, based on the above method, a disparity map of two camera modules in the camera device when the camera device photographs the object to be detected;

[0011] The three-dimensional reconstruction of the object to be detected is performed based on the disparity map.

[0012] To achieve the above-mentioned purpose, the present application provides an electronic device, which includes a memory and a processor; a computer program is stored in the memory, and the processor is used to execute the computer program to implement the steps of the above-mentioned method.

[0013] To achieve the above objectives, the present application provides a computer-readable storage medium having a computer program stored thereon, and the steps in the above method are implemented when the computer program is executed by a processor.

[0014] The method of the present application is: obtaining images of the object to be detected under the projection of multiple images with different light intensities captured by the camera module in the camera device, and obtaining multiple structured light object images, so that the mirror reflection images and / or diffuse reflection images of the two camera modules of the camera device can be separated based on the multiple structured light object images, and then it is convenient to perform a search for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules to determine the disparity maps of the two camera modules. In this way, the matching of points with the same name can be performed based on the mirror reflection images and / or diffuse reflection images, which effectively compats the point cloud sparseness problem caused by external light interference and reflective surfaces, thereby improving the quality of the disparity map on the highly reflective surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the implementation modes of the present application, the drawings required for use in the description of the implementation modes will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 It is a flowchart of the first implementation method of the disparity map determination method of the present application;

[0017] Figure 2 A schematic diagram of multiple images with different projection light intensities for this application;

[0018] Figure 3 It is a flowchart of the first implementation method of the disparity map determination method of the present application;

[0019] Figure 4 It is a schematic diagram of the structure of the electronic device of the present application;

[0020] Figure 5 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solution of the present application, the disparity map determination method, three-dimensional reconstruction method and related devices provided by the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] In the related art, the disparity map determination system mainly includes four modules: fringe projection module, binocular camera module, computer processing module and mechanical control module. Among them, the fringe projection module is used to generate structured light and project the structured light onto the object to be measured; the binocular camera module is used to collect image data of the object to be measured; the computer processing module is used to process and analyze the collected data; the mechanical control module is used to control the position and posture of the fringe projection module and the binocular camera module. A combination of fringe projection structured light and binocular camera is adopted, and the binocular camera is used to collect several groups of patterns for correlation calculations and three-dimensional reconstruction, thereby achieving high-precision three-dimensional measurement. In this related art, due to the difference in the surface characteristics of the measured object, the point cloud quality of the solution for the highly reflective surface is poor, the point cloud missing phenomenon is more serious, and the quality of the disparity map on the determined highly reflective surface is relatively poor, which is not suitable for three-dimensional reconstruction of high-reflective measurement.

[0023] Based on this, the present application proposes a method for determining a disparity map, which first obtains an image of an object to be detected under the projection of multiple images with different light intensities captured by a camera module in a camera device, and obtains multiple structured light object images of the camera module; then uses the multiple structured light object images of the camera module to separate the mirror reflection image and / or diffuse reflection image of the camera module; performs a matching point search based on the mirror reflection image and / or diffuse reflection image of the two camera modules to obtain a first matching point pair; based on the first matching point pair, obtains a disparity map of the two camera modules. In this way, the mirror reflection image and / or diffuse reflection image representing different light components on the surface of the object can be obtained. When matching, the matching of the same-name points is performed based on the mirror reflection image and / or diffuse reflection image, which effectively complies with the problem of sparse point clouds caused by external light interference and reflective surfaces, thereby improving the quality of the disparity map on the highly reflective surface.

[0024] See also Figure 1 , Figure 1 : is a flow chart of the first embodiment of the disparity map determination method of the present application. Optionally, the camera device in the disparity map determination method of the present embodiment may include at least two camera modules. Among them, the camera device may be a binocular camera, so that the two camera modules of the camera device are two cameras in the binocular camera. In other embodiments, the camera device may also be a trinocular camera. Among them, the disparity map determination method of the present embodiment may include the following steps.

[0025] S101: Acquire multiple images of an object to be detected projected from images with different light intensities captured by a camera module in a camera device to obtain multiple structured light object images from the camera module.

[0026] The images of the object to be detected under the projection of multiple images with different light intensities captured by the camera module in the camera device can be obtained to obtain multiple structured light object images, so that the mirror reflection images and / or diffuse reflection images of the two camera modules of the camera device can be separated based on the multiple structured light object images, thereby facilitating the subsequent search for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules to determine the disparity maps of the two camera modules. In this way, the matching of points with the same name can be performed based on the mirror reflection images and / or diffuse reflection images, which effectively complies with the problem of sparse point clouds caused by external light interference and reflective surfaces, thereby improving the quality of the disparity map on highly reflective surfaces.

[0027] The contents of the multiple images with different light intensities are not limited.

[0028] Optionally, the projection illumination intensity of each pixel point in each image is consistent, and the projection illumination intensity of the pixel points in different images is different.

[0029] Alternatively, the multiple images with different illumination intensities may also be at least two groups of structured light images, each group of structured light images includes at least two structured light images, among all the structured light images in each group of structured light images corresponding to the same pixel position, only one structured light image has a projection illumination intensity that is not 0, and the image contents of any two structured light images in at least two groups of structured light images are different. The projection illumination intensity of pixels whose projection illumination intensity is not 0 in each structured light image is consistent, for example, all are preset illumination intensities. Of course, in other embodiments, the projection illumination intensity of multiple pixels whose projection illumination intensity is not 0 in the structured light image may also be different. The projection illumination intensities of different structured light images are different, that is, the projection illumination intensities of pixels whose projection illumination intensity is not 0 in different structured light images are different.

[0030] In one embodiment, there are two groups of structured light images, each group of structured light images includes two structured light images, and the projection light intensity of half of the pixels in each structured light image is not 0. In this way, step S101 can obtain the image of the object to be detected under the projection of each structured light image in the four structured light images captured by the camera device, so as to obtain four structured light object images.

[0031] For example, Figure 2 As shown, in the first group of structured light images: the projection illumination intensity of the pixels in the right half of the image area of ​​one of the structured light images is not 0 (i.e. Figure 2 The projection illumination intensity of the pixels in the left half of the image area is 0 (i.e. Figure 2 The projection illumination intensity of the pixels in the right half of the image area of ​​the other structured light image is 0 (i.e. Figure 2 The dark area of ​​the first image in the figure), the projection illumination intensity of the pixels in the left half of the image area is not 0 (i.e. Figure 2 In the second set of structured light images: the projection illumination intensity of the pixels in the upper half of the image area of ​​one of the structured light images is not 0 (i.e. Figure 2 The projection illumination intensity of the pixels in the lower half of the image area is 0 (i.e. Figure 2 The projection illumination intensity of the pixels in the upper half of the image area of ​​the other structured light image is 0 (i.e. Figure 2 The projection illumination intensity of the pixels in the lower half of the image area is not 0 (i.e. Figure 2 In this example, the encoding method of the four structured light images of the two sets of structured light images can be shown as follows:

[0032] Lj(x,y)=sign(x·cos(2πj / N)+y·sin(2πj / N));

[0033] Wherein, N=4 is the number of projected images, j=1,2,3,4, Lj(x,y) is the light intensity of the jth image at the (x,y) pixel position, when Lj(x,y)=-1, the projected light intensity is 0, and when Lj(x,y)=1, the projected light intensity is 255.

[0034] For another example, in the first group of structured light images: the projection illumination intensity of the pixels in the upper right triangular image area of ​​one of the structured light images is not 0, and the projection illumination intensity of the pixels in the lower left triangular image area is 0; the projection illumination intensity of the pixels in the upper right triangular image area of ​​another structured light image is 0, and the projection illumination intensity of the pixels in the lower left triangular image area is not 0. In the second group of structured light images: the projection illumination intensity of the pixels in the upper left triangular image area of ​​one of the structured light images is not 0, and the projection illumination intensity of the pixels in the lower right triangular image area is 0; the projection illumination intensity of the pixels in the upper left triangular image area of ​​another structured light image is 0, and the projection illumination intensity of the pixels in the lower right triangular image area is not 0.

[0035] Optionally, a projection module can be used to project each of multiple images with different projection light intensities onto the surface of the object to be detected, and then a camera module in a camera device can be used to capture the image of the object to be detected under each image projection to obtain each structured light object image captured by the camera module in the camera device.

[0036] The projection module may be a projection unit on the camera device. In other embodiments, the projection module and the camera device are independent of each other.

[0037] The projection module may be an LCOS sensor or a DLP projection unit, etc.

[0038] S102: Separate a specular reflection image and / or a diffuse reflection image of the camera module based on a plurality of structured light object images of the camera module.

[0039] After obtaining a plurality of structured light object images captured by the camera module, the specular reflection image and / or diffuse reflection image of the camera module may be separated based on the plurality of structured light object images.

[0040] Optionally, a method of polarization and orthogonal decomposition can be used to establish a linear constraint model based on the distribution relationship of specular reflection light and diffuse reflection light in the vertical and parallel directions on the surface of a smooth object, so as to solve the specular reflection component and the diffuse reflection component, so as to separate the specular reflection image and / or diffuse reflection image of the two camera modules of the camera device based on multiple structured light object images.

[0041] Among them, under ideal conditions, the light in a scene can be divided into two types: diffuse reflection and specular reflection. Diffuse reflection follows the Lambert model, that is, the intensity of the reflected light is proportional to the intensity of the incident light and the cosine of the incident angle; while specular reflection follows the Specular model, that is, the intensity of the reflected light depends on the incident light and the angle between the observer's line of sight and the surface normal. Assuming that there are (n) images, each image corresponds to a different lighting condition, the pixel value of each image can be expressed as the superposition of diffuse reflection and specular reflection, as shown in formula (1). The pixel value of a pixel point in each image can be equal to the product of the diffuse reflection intensity of the pixel and the illumination intensity vector plus the specular reflection intensity of the pixel:

[0042] I i (x) = D(x) × L i +S i (x) Formula (1);

[0043] in:

[0044] I i (x) is the intensity of pixel (x) in the (i)th image.

[0045] D(x) is the diffuse intensity of pixel (x), which is constant in all images.

[0046] L i is the projection illumination intensity of the (i)th image.

[0047] S i (x) is the specular intensity of pixel (x) in the (i)th image, which depends on the light source position, the observer position and the surface properties.

[0048] Based on the principle that the pixel value of each image can be expressed as the superposition of diffuse reflection and specular reflection, a system equation can be constructed based on multiple structured light images (i.e., images of the object to be detected under the projection of multiple images with different light intensities) collected by each camera module. The system equation characterizes the relationship between the pixel intensity matrix, the diffuse reflection intensity matrix and the specular reflection intensity matrix of the structured light image; then, the diffuse reflection intensity matrix and the specular reflection intensity matrix in the system equation can be optimized and solved with the purpose of minimizing the error, wherein the error is positively correlated to the difference between the actual pixel value of the structured light image and the solved pixel value, and the solved pixel value of the structured light image is equal to the value obtained by adding the product of the diffuse reflection intensity of each pixel in the image and the projection light intensity to the specular reflection intensity of the pixel.

[0049] In a specific example, assuming that each camera module has n structured light images and each structured light image has m pixels, we can construct an (n×m) image pixel intensity matrix (I), an (m×1) diffuse reflection intensity matrix (D), and (n) (m×1) specular reflection intensity matrices (S1, S2, ..., S n ). At the same time, there are (n) light intensity vectors (L1, L2, ..., L n ). In this way, the following system equation (2) can be constructed:

[0050] I=[D×L1|D×L2|……|D×L n ]+[S1|S2|……|S n ]Formula (2).

[0051] The goal is to solve (D) and (S i ). Since (D) is the same for all structured light images of the camera module, and (S i ) can be adjusted according to the lighting conditions (L i ) to calculate, so we can get formula (3):

[0052] I=L·D T +S T Formula (3);

[0053] Where I is an (n×m) matrix, each column corresponds to the intensity of a single pixel in different images; L is a (n×1) vector, each row corresponds to the corresponding light intensity contribution; D T is a (1×m) vector, each column of which corresponds to the diffuse reflection contribution of a pixel; S T is an (n×m) matrix, where each column corresponds to the specular contribution of a single pixel in different images.

[0054] Thus, in this example, D can be solved by minimizing the error ET and S T As shown in formula (4), E may be equal to the square norm of the difference between the actual pixel value of the structured light image and the solved pixel value. In other embodiments, E may also be equal to the square difference between the actual pixel value of the structured light image and the solved pixel value.

[0055] minimizeE=||I-(L·D T +S T )|| 2 Formula (4);

[0056] Alternatively, D can be solved by minimizing the error E using the alternating minimum optimization method T and S T For example, it can be solved by adaptive alternating minimization algorithms, that is, first fix (D T )Solve (S T ), then fix (S T )Solve (D T ), repeat this process until convergence. Of course, other optimization methods can also be used to minimize the error E to solve D, such as gradient descent, genetic algorithm, and simulated annealing. T and S T Therefore, the present application proposes a method for separating the specular reflection component and the diffuse reflection component based on a special projection image. The method realizes the separation of the optical reflection components of high-reflective surface defect types, separates different types of illumination components into two different images, and realizes the texture imaging visualization of the high-reflective surface.

[0057] Among them, due to D T is a (1×m) vector, each column of which corresponds to the diffuse reflection contribution (ie, diffuse reflection intensity) of a pixel point. The pixel value of the pixel point in the diffuse reflection image of the camera module can be equal to the value corresponding to the pixel point in the diffuse reflection intensity matrix of the camera module.

[0058] Due to S Tis an (n×m) matrix, each column corresponds to the specular reflection contribution (i.e., specular reflection intensity) of a single pixel in different images, and the pixel value of a pixel in the specular reflection image of the camera module can be equal to the value corresponding to the pixel in the i row of the specular reflection intensity matrix of the camera module, and the value range of i is 1-n. Of course, in other embodiments, the pixel value of a pixel in the specular reflection image of the camera module can be equal to the weighted value of the values ​​corresponding to the pixel in multiple rows of the specular reflection intensity matrix of the camera module, wherein the weighting coefficient of the values ​​corresponding to the pixel in multiple rows can be set according to actual conditions, and is not limited here. Alternatively, in other embodiments, a specular reflection image of the camera module can be determined based on each row in the specular reflection intensity matrix of the camera module, so that multiple specular reflection images of the camera module can be obtained.

[0059] S103: searching for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules to obtain a first matching point pair.

[0060] After the mirror reflection images and / or diffuse reflection images of the two camera modules are separated, a matching point search may be performed based on the mirror reflection images and / or diffuse reflection images of the two camera modules to obtain a first matching point pair.

[0061] In one implementation, corresponding point pairs may be searched in the mirror reflection images of the two camera modules to obtain a first matching point pair.

[0062] In another implementation, corresponding point pairs may be searched in the diffuse reflection images of the two camera modules to obtain a first matching point pair.

[0063] In another implementation, matching point search may be performed based on the mirror reflection images and diffuse reflection images of the two camera modules. For example, corresponding point pairs may be searched out in the mirror reflection images of the two camera modules, and corresponding point pairs may be searched out in the diffuse reflection images of the two camera modules, and then the first matching point pair may be determined based on the search results of the matching point pairs of the mirror reflection images and the diffuse reflection images. For a pixel, if the matching points determined by the mirror reflection image and the diffuse reflection image are the same, the pixel and its matching point may be used as the first matching point pair; and for a pixel, if the matching points determined by the mirror reflection image and the diffuse reflection image are different, the pixel and the matching points respectively determined by the mirror reflection image and the diffuse reflection image may be used together as the first matching point pair. When determining the disparity map later, the first disparity and the second disparity of the pixel may be weighted to obtain the final disparity of the pixel, wherein the first disparity of the pixel is the disparity of the pixel and the matching point determined by the mirror reflection image, and the second disparity of the pixel is the disparity of the pixel and the matching point determined by the diffuse reflection image. The weighting coefficients of the first disparity and the second disparity may be determined according to their respective signal-to-noise ratios and / or matching confidences, etc. In other embodiments, for a pixel, if the matching points determined using the specular reflection image and the diffuse reflection image are different, one of the matching points and the pixel may be used as a first matching point pair, or a matching point with a higher matching confidence and / or signal-to-noise ratio and the pixel may be used as a first matching point pair.

[0064] In yet another implementation, corresponding point pairs may be searched out in the mirror reflection images of the two camera modules first, and it may be determined whether the matching point pairs searched out in the mirror reflection images meet preset conditions, for example, whether the matching point pairs searched out meet similarity measurement conditions, geometric consistency conditions, uniqueness conditions, bidirectional consistency check passed, continuity and smoothness conditions, brightness consistency conditions and / or no occlusion conditions, and the matching point pairs searched out in the mirror reflection images that meet the preset conditions may be used as the first matching point pairs; then, the pixel points where the matching fails may be traversed, and corresponding point pairs may be searched out in the diffuse reflection images of the two camera modules, and it may be determined whether the matching point pairs searched out in the diffuse reflection images meet preset conditions, for example, whether the matching point pairs searched out meet similarity measurement conditions, geometric consistency conditions, uniqueness conditions, bidirectional consistency check passed, continuity and smoothness conditions, brightness consistency conditions and / or no occlusion conditions, and the matching point pairs searched out in the diffuse reflection images that meet the preset conditions may be used as the first matching point pairs.

[0065] In another implementation, corresponding point pairs can be first searched for in the diffuse reflection images of the two camera modules, and it is determined whether the matching point pairs searched for in the diffuse reflection images meet preset conditions, for example, whether the matching point pairs searched for meet similarity measurement conditions, geometric consistency conditions, uniqueness conditions, bidirectional consistency check passed, continuity and smoothness conditions, brightness consistency conditions and / or no occlusion conditions, and the matching point pairs searched for in the diffuse reflection images that meet the preset conditions are used as the first matching point pairs; then the pixel points where the matching fails can be traversed, and corresponding point pairs can be searched for in the mirror reflection images of the two camera modules, and it is determined whether the matching point pairs searched for in the mirror reflection images meet preset conditions, for example, whether the matching point pairs searched for meet similarity measurement conditions, geometric consistency conditions, uniqueness conditions, bidirectional consistency check passed, continuity and smoothness conditions, brightness consistency conditions and / or no occlusion conditions, and the matching point pairs searched for in the mirror reflection images that meet the preset conditions are used as the first matching point pairs.

[0066] The corresponding point pairs may be searched for in the mirror reflection images of the two camera modules and / or in the diffuse reflection images of the two camera modules by using a window matching method, a limited search matching method or a global optimization method.

[0067] For example, the field of view that needs to be matched in the first image can be limited. For point A in the field of view of the first image, in the case of row calibration, all points in the second image are traversed row by row to find the best matching point A'; according to the principle of continuity, among the point B adjacent to point A in the first image, a given threshold range adjacent to A' is selected to find the local optimal point B. If point B meets the preset conditions, the match is successful, otherwise it is discarded; wherein the first image can be a mirror reflection image of one of the two camera modules, and correspondingly, the second image is a mirror reflection image of the other of the two camera modules, so that the method can be used to search for corresponding point pairs in the mirror reflection images of the two camera modules; in addition, the first image can be a diffuse reflection image of one of the two camera modules, and correspondingly, the second image is a diffuse reflection image of the other of the two camera modules, so that the method can be used to search for corresponding point pairs in the diffuse reflection images of the two camera modules.

[0068] When the mirror reflection image is used to search for matching points in the above implementation, if there are multiple mirror reflection images of the camera module, one of them can be selected as the mirror reflection image for searching for matching points.

[0069] S104: Determine a disparity map of two camera modules based on the first matching point pair.

[0070] Optionally, the successfully matched point pairs may be summarized to obtain a disparity map of the two camera modules.

[0071] For each first matching point pair, its disparity value can be calculated; and then the disparity values ​​of all first matching point pairs are aggregated into a disparity map, so that the disparity map between the two camera modules can be determined.

[0072] Based on the above disparity map determination method, the present application further improves to provide another disparity map determination method, such as Figure 3 As shown, the disparity map determination method includes the following steps.

[0073] S201: Searching for corresponding point pairs in the images of the two camera modules to obtain second matching point pairs.

[0074] Corresponding point pairs may be searched in the images of the two camera modules to obtain second matching point pairs.

[0075] Among them, corresponding point pairs can be searched in the captured images or absolute phase images of the two camera modules to obtain a second matching point pair. Among them, the captured content corresponding to the captured image or absolute phase image of the shooting module is the same as the captured content of the structured light object image of the shooting module. More preferably, when obtaining the captured image, the original captured image and the structured light image corresponding to the absolute phase image, the relative position relationship between each camera module and the photographed object (i.e., the above-mentioned object to be detected) and the parameters of the camera module remain unchanged, so as to ensure that the captured content corresponding to the captured image or absolute phase image of the shooting module is the same as the captured content of the structured light object image of the shooting module.

[0076] In one embodiment, corresponding point pairs may be searched in the images captured by the two camera modules respectively to obtain second matching point pairs.

[0077] That is, the images collected by the two camera modules can be used to search for matching points. As described in step S103, a window matching method, a feature matching method, or a convolutional neural matching method can be used to search for corresponding point pairs in the images collected by the two camera modules.

[0078] In another embodiment, corresponding point pairs may be searched in the absolute phase images of the two camera modules to obtain second matching point pairs.

[0079] That is, the absolute phase images of the two camera modules can be used to search for matching points. As described in step S103, the window matching method, the limited search matching method or the global optimization method can be used to search for corresponding point pairs in the absolute phase images of the two camera modules. In the example, assuming a binocular camera, the field of view of the left image that needs to be matched can be limited during matching. For point A in the field of view of the absolute phase image of the left camera, in the case of row calibration, all points of the absolute phase image of the right camera are traversed row by row to find the best matching point A'; according to the principle of continuity, among the points B adjacent to point A in the left image, a given threshold range adjacent to A' is selected to find the local optimal point B. If point B meets the conditions, the match is successful, otherwise it is discarded; traverse the pixel points where the match fails, search for corresponding point pairs in the mirror reflection and diffuse reflection images of the left and right cameras, find the local optimal matching point, and summarize the successfully matched point pairs to obtain a complete disparity map.

[0080] In this embodiment, the absolute phase images of the two camera modules may be determined first.

[0081] In one implementation, the absolute phase images of the two camera modules may be determined by a phase shift method.

[0082] Exemplarily, a set of sinusoidal fringe patterns with different phase shifts can be projected. Each phase shift corresponds to a captured image. Phase information is calculated from these images using a phase resolution algorithm (such as a four-step phase shift method or a Gray code algorithm), thereby obtaining absolute phase images of the two camera modules.

[0083] In a specific example, a four-step phase-shift pattern with three different frequencies can be projected onto the surface of the object to be detected, and then the surface of the object to be detected can be photographed by a camera device to obtain four-step phase-shift images with three different frequencies; then, the multi-frequency interpolation principle can be used to phase-solve multiple images based on the obtained four-step phase-shift images with three different frequencies to obtain the absolute phase images of the two camera modules respectively.

[0084] Among them, the intensity distribution of the four-step phase-shifted image with three different frequencies, f1, f2, and f3, is shown in formula (5):

[0085]

[0086] I i n (x,y) represents the light intensity value of the nth image with frequency i at pixel (x,y);

[0087] A n (x,y) represents the modulated light intensity value at pixel (x,y) of the nth image;

[0088] Bn (x,y) represents the background light intensity value at pixel (x,y) of the nth image;

[0089] i=1,2,3,n=1,2,3,4.

[0090] According to the principle that the frequency f=f1f2 / (f1-f2) after the extrapolation of the frequencies f1 and f2, the frequency after the extrapolation of the three frequencies must be 1.

[0091] The absolute phase image can be obtained by formula (6).

[0092]

[0093] in, is the pixel value of the absolute phase image of the camera module at pixel (x, y).

[0094] In another implementation, the absolute phase images of the two camera modules may be determined by Fourier transform method.

[0095] Alternatively, a random or pseudo-random fringe pattern can be projected, the image acquired and a two-dimensional Fourier transform performed, and the phase information can then be extracted in the frequency domain.

[0096] In yet another implementation, the absolute phase images of the two camera modules may be determined by a spatial phase modulation method. In this implementation, a spatial light modulator (SLM) may be used to dynamically change the projected fringe pattern.

[0097] S202: traverse pixel points that fail to match in the images of the two camera modules, search for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules, and obtain a first matching point pair.

[0098] During the matching search through step S201, if the searched matching point pair does not meet preset conditions such as similarity measurement conditions, geometric consistency conditions, uniqueness conditions, passing of bidirectional consistency check, continuity and smoothness conditions, brightness consistency conditions and / or no occlusion conditions, then step S202 can be used to further perform matching point search on the pixel points in the matching point pair to determine the first matching point pair; if the preset conditions are met, the matching point pair searched through step S201 is the second matching point pair.

[0099] The specific process can be found in step S103 and will not be described in detail here.

[0100] S203: Obtain the disparity map based on the first matching point pair and the second matching point pair.

[0101] Optionally, the successfully matched point pairs (including the first matching point pair and the second matching point pair) may be summarized to obtain a disparity map of the two camera modules.

[0102] For each first matching point pair and each second matching point pair, the disparity value thereof can be calculated; and then the disparity values ​​of all matching point pairs are aggregated into a disparity map, so that the disparity map between the two camera modules can be determined.

[0103] In this embodiment, an initial search for matching points is performed based on images of two camera modules (e.g., original captured images or absolute phase images); for pixels where the matching point search fails, the mirror reflection images and / or diffuse reflection images of the two camera modules are used to perform a supplementary search for matching points. In this way, the mirror reflection images and / or diffuse reflection images can be used to supplement the matching points when there is external lighting interference and the stereo matching error of the reflective surface is large, thereby improving the disparity map determination effect, and optimizing the reconstruction of the point cloud of the external lighting and reflective surface from the perspective of separating the lighting components of the object surface, which can solve the problem of sparse point cloud, without the need to construct near-field photometric stereo, and more convenient and faster deployment. Furthermore, the present application can use the projection chip of the LCOS optical machine, which can effectively reduce the development cost. Furthermore, when matching the same-name points, the absolute phase image, the mirror reflection map and the diffuse reflection image are fused together to match the same-name points, which effectively accommodates the problem of sparse point cloud caused by external lighting interference and reflective surfaces.

[0104] Optionally, in order to improve the disparity map determination effect, the camera equipment can also be calibrated, such as calibrating the camera intrinsic parameters of the camera equipment, the distortion coefficient, the rotation matrix and translation vector of the two camera modules of the camera equipment, and / or the limit correction of the two camera modules of the camera equipment.

[0105] Optionally, a variety of calibration methods can be used to calibrate the camera device, which is not limited here. For example, the Zhang Zhengyou calibration method is used to calibrate the internal parameters fx, fy, u, v, radial distortion and tangential distortion k1, k2, k3, p1, p2 of a single camera.

[0106] The rotation matrix R=Rr·RlT and the translation vector t=tr-R·tl between the coordinate systems of the two camera modules of the camera device can be calibrated so that the imaging planes of the two camera modules of the camera device are rotated by a certain angle respectively to ensure that the imaging planes of the two camera modules of the camera device are parallel.

[0107] The extreme correction is to make the epipolar line parallel to the baseline on the basis of the parallel imaging planes of the two camera modules of the camera device, so that the corresponding points of the same name of the two camera modules of the camera device are in the same row. This process can use the Bouguet algorithm to achieve row alignment of the left and right camera images.

[0108] In addition, the disparity map of two camera modules determined by the disparity map determination method of the present application can be used in application scenarios such as three-dimensional reconstruction and target detection and tracking.

[0109] For example, in a three-dimensional reconstruction scenario, after determining the disparity map of two camera modules in the camera device when the camera device photographs the object to be detected based on the above implementation, the three-dimensional reconstruction of the object to be detected can be performed based on the disparity map.

[0110] The disparity map can be processed and mapped to a 3D space to obtain a 3D point cloud of the object to be detected, thereby completing the three-dimensional reconstruction of the object to be detected.

[0111] A binocular structured light stereo imaging system can be used to achieve the above-mentioned disparity map determination and three-dimensional reconstruction, among which the binocular structured light stereo imaging system is a three-dimensional imaging technology that combines structured light technology and binocular stereo vision. The system is mainly composed of two cameras, a structured light projector, an image acquisition card, a computer and image processing software, etc. It mainly projects a specific light pattern (structured light) in the scene, and uses a binocular camera to capture the deformation of the light pattern on the object, and restores the three-dimensional shape of the object through the binocular parallax principle. System and method.

[0112] See also Figure 4 , Figure 4 The electronic device 10 includes a memory 11 and a processor 12 coupled to each other, the memory 11 is used to store program instructions, and the processor 12 is used to execute the program instructions to implement the method of any of the above-mentioned embodiments.

[0113] The logical process of the above method is presented in a computer program. In terms of the computer program, if it is sold or used as an independent software product, it can be stored in a computer storage medium. Therefore, this application proposes a computer-readable storage medium. Figure 5 , Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. In this embodiment, a computer-readable storage medium 20 stores a computer program 21, and when the computer program 21 is executed by a processor, the steps in the above-mentioned video encoding method are implemented.

[0114] The computer-readable storage medium 20 may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., which can store computer programs, or may be a server storing the computer program, which can send the stored computer program to other devices for execution, or can also execute the stored computer program itself. From a physical entity point of view, the computer-readable storage medium 20 may be a combination of multiple entities, such as multiple servers, a server plus a memory, or a memory plus a mobile hard disk, etc.

[0115] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for determining a disparity map, characterized in that: The method comprises: Acquire images of the object to be detected under the projection of multiple images with different light intensities captured by the camera module in the camera device, and obtain multiple structured light object images of the camera module; the camera device includes at least two camera modules; Separating a specular reflection image and / or a diffuse reflection image of the camera module based on the multiple structured light object images of the camera module; Searching for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules to obtain a first matching point pair; Based on the first matching point pair, a disparity map of the two camera modules is obtained.

2. The disparity map determination method according to claim 1, characterized in that: The method further comprises: Searching for corresponding point pairs in the captured images or absolute phase images of the two camera modules to obtain second matching point pairs; the captured content corresponding to the captured image or absolute phase image of the camera module is the same as the captured content of the structured light object image of the camera module; The searching for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules to obtain a first matching point pair includes: Traversing pixel points that fail to match in the captured images or absolute phase images of the two camera modules, searching for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules, and obtaining a first matching point pair; The obtaining, based on the first matching point pair, a disparity map of the two camera modules, comprises: The disparity map is obtained based on the first matching point pair and the second matching point pair.

3. The disparity map determination method according to claim 2, characterized in that: The two camera modules include a first camera module and a second camera module, the second matching point pair includes a preliminary matching point pair and a local matching point pair, and the searching for corresponding point pairs in the captured images or absolute phase images of the two camera modules to obtain the second matching point pair includes: For pixel points in the image of the first camera module, searching for matching points in a row corresponding to the image of the second camera module to obtain preliminary matching point pairs; For the neighboring points of the preliminary matching point pair in the image of the first camera module, searching for a local optimal matching point within a given threshold range of the neighboring points of the preliminary matching point pair in the image of the second camera module; If the local optimal matching point meets the preset conditions, the matching is successful and the local matching point pair is obtained; otherwise, the matching fails.

4. The method for determining a disparity map according to any one of claims 1 to 3, characterized in that: The searching for matching points based on the mirror reflection images and / or diffuse reflection images of the two camera modules to obtain a first matching point pair includes: Searching for matching points in the mirror reflection images of the two camera modules, and searching for matching points in the diffuse reflection images of the two camera modules; A first matching point pair is determined based on the matching point pair search results of the specular reflection image and the diffuse reflection image.

5. The disparity map determination method according to claim 4, characterized in that: The step of comprehensively determining a first matching point pair based on the matching point pair search results of the specular reflection image and the diffuse reflection image comprises: For a pixel, if the matching points determined by using the specular reflection image and the diffuse reflection image are the same, the pixel and its matching point are taken as a first matching point pair; For a pixel, if the matching points determined by using the specular reflection image and the diffuse reflection image are different, the pixel and the matching points respectively determined by using the specular reflection image and the diffuse reflection image are taken together as a first matching point pair; The obtaining, based on the first matching point pair, a disparity map of the two camera modules, comprises: For the first matching point pair including the matching points respectively determined by using the mirror reflection image and the diffuse reflection image, the first disparity and the second disparity of the pixel are weighted to obtain the disparity of the pixel in the disparity map, wherein the first disparity of the pixel is the disparity between the pixel and the matching point determined by using the mirror reflection image, and the second disparity of the pixel is the disparity between the pixel and the matching point determined by using the diffuse reflection image.

6. The disparity map determination method according to claim 1, characterized in that: The separating of the specular reflection image and / or diffuse reflection image of the camera module based on the multiple structured light object images of the camera module includes: A system equation is constructed based on the multiple structured light images captured by the camera module, wherein the system equation represents that the intensity of a pixel in the structured light image is equal to the sum of a specified product of the pixel in the structured light image and a mirror reflection intensity of the pixel, and the specified product of the pixel in the structured light image is equal to the product of the diffuse reflection intensity of the pixel in the structured light image and a projection illumination intensity of the structured light image; With the optimization purpose of minimizing the error of the system equation, the diffuse reflection intensity matrix and the specular reflection intensity matrix in the system equation of the camera module are optimized and solved to obtain the diffuse reflection image and the specular reflection image of the camera module.

7. The disparity map determination method according to claim 6, characterized in that: The optimization objective is to minimize the error of the system equation, and to optimize and solve the diffuse reflection intensity matrix and the specular reflection intensity matrix in the system equation of the camera module to obtain the diffuse reflection image and the specular reflection image of the camera module, including: By using an alternating minimum optimization method, the diffuse reflection intensity matrix and the specular reflection intensity matrix in the system equation of the camera module are optimized and solved; or, Each column in the mirror reflection intensity matrix corresponds to the mirror reflection intensity of a single pixel in different images, and the pixel value of the pixel in the mirror reflection image of the camera module is determined based on the value corresponding to the pixel in at least one row of the mirror reflection intensity matrix of the camera module; each column in the diffuse reflection intensity matrix corresponds to the diffuse reflection intensity of a single pixel, and the pixel value of the pixel in the diffuse reflection image of the camera module is equal to the value corresponding to the pixel in the diffuse reflection intensity matrix of the camera module.

8. A three-dimensional reconstruction method, characterized in that: The method comprises: Determine, based on the method described in any one of claims 1 to 7, a disparity map of two camera modules in the camera device when the camera device photographs the object to be detected; The three-dimensional reconstruction of the object to be detected is performed based on the disparity map.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor; a computer program is stored in the memory, and the processor is used to execute the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images