Structured light three-dimensional reconstruction method, device, electronic device and storage medium

Abstract

The present application provides a structured light three-dimensional reconstruction method, device, electronic device and storage medium. In an example of the present application, the structured light three-dimensional reconstruction method includes: acquiring an image of the object to be measured; performing phase solution processing on a first type of image in the image of the object to be measured to obtain a first phase solution result; performing phase solution processing on a second type of image in the image of the object to be measured to obtain a second phase solution result; performing unwrapping processing on the first phase solution result based on the second phase solution result to obtain an absolute phase map; performing three-dimensional reconstruction processing based on the absolute phase map. This method can achieve high-precision three-dimensional reconstruction.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a structured light three-dimensional reconstruction method, device, electronic device and storage medium. Background Art

[0002] Structured light 3D reconstruction is a common 3D imaging technology that uses a specific light spot generated by a structured light source (such as a laser or projector) to capture the shape and depth information of an object's surface.

[0003] Structured light 3D reconstruction has broad application prospects in industrial manufacturing, virtual reality, cultural heritage protection and other fields. Summary of the invention

[0004] In view of this, the present application provides a structured light three-dimensional reconstruction method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present application, a structured light 3D reconstruction method is provided, comprising:

[0006] Acquire an image of the object to be measured; wherein the object to be measured is projected with a specified pattern, the specified pattern includes multiple frames of integer coded patterns and multiple frames of line-shifted fringe patterns projected in sequence, and the number of the integer coded patterns is consistent with the period of the line-shifted fringe pattern in the spatial domain;

[0007] Performing phase solution processing on a first type of image in the image of the object to be measured to obtain a first phase solution result; wherein the first type of image is an image of the object to be measured obtained when the multiple frames of line-shifted fringe patterns are sequentially projected;

[0008] Performing phase solution processing on a second type of image in the image of the object to be measured to obtain a second phase solution result; wherein the second type of image is an image of the object to be measured obtained when the multiple frames of integer coded patterns are sequentially projected;

[0009] According to the second phase solution result, performing unwrapping processing on the first phase solution result to obtain an absolute phase diagram;

[0010] A three-dimensional reconstruction process is performed according to the absolute phase map.

[0011] According to a second aspect of an embodiment of the present application, a structured light 3D reconstruction device is provided, comprising:

[0012] An acquisition unit, configured to acquire an image of the object under test; wherein the object under test is projected with a designated pattern, the designated pattern comprising a plurality of frames of integer coded patterns and a plurality of frames of line-shifted fringe patterns projected in sequence, the number of the integer coded patterns being consistent with the period of the line-shifted fringe pattern in the spatial domain;

[0013] A dephasing unit, configured to perform dephasing on a first type of image in the image of the object to be measured, to obtain a first dephasing result; wherein the first type of image is an image of the object to be measured obtained by sequentially projecting the multiple frames of line-shifted fringe patterns;

[0014] The dephasing unit is further used to perform dephasing on a second type of image in the image of the object to be measured to obtain a second dephasing result; wherein the second type of image is an image of the object to be measured obtained when the multiple frames of integer coded patterns are sequentially projected;

[0015] an unwrapping unit, configured to perform an unwrapping process on the first phase solution result according to the second phase solution result to obtain an absolute phase map;

[0016] A reconstruction unit is used to perform three-dimensional reconstruction processing according to the absolute phase map.

[0017] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method provided in the first aspect.

[0018] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0019] The structured light 3D reconstruction method of the embodiment of the present application obtains an image of the object to be measured with a specified pattern projected thereon to obtain multiple frames of first type images and multiple frames of second type images, respectively performs phase solution processing on the first type images to obtain a first phase solution result, and performs phase solution processing on the second type images to obtain a second phase solution result, and based on the second phase solution result, performs unwrapping processing on the first phase solution result to obtain an absolute phase map, and then performs 3D reconstruction processing based on the absolute phase map, and through the coordinated application of integer coded patterns and line-shifted stripe patterns, uses integer coded patterns to assist line-shifted stripe patterns in phase unwrapping, thereby improving the accuracy of phase solution, and thereby achieving high-precision 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a structured light 3D reconstruction method provided in an embodiment of the present application;

[0021] Figure 2A to Figure 2C is a schematic diagram of a line-shifted stripe pattern provided in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a single pixel template integral calculation process provided in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a single pixel time domain grayscale and template integration result provided in an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of an integral result period extension provided in an embodiment of the present application;

[0025] Figure 6 It is a flowchart of a structured light 3D reconstruction method provided in an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of the structure of a structured light 3D reconstruction device provided in an embodiment of the present application;

[0027] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, some terms involved in the embodiments of the present application are briefly explained below.

[0029] 1. Integer coded pattern: The image is encoded by binary numbers, and the code value corresponding to each pixel in the image is an integer.

[0030] 2. Gray code: An integer coding pattern, also called reflected binary coding, is a special binary coding method. In Gray code, two adjacent values ​​differ by only one binary digit.

[0031] 3. XOR code: An integer coding pattern, also known as exclusive OR code or parity code, is a special binary coding method. It is similar to Gray code, and only one bit is different between two adjacent values.

[0032] 4. Phase solution: The process of calculating the code value of the collected coded image according to the structured light pattern coding method, also known as decoding.

[0033] 5. Time domain template integration: Calculate the integral result of the gray value of the acquired serial line-shifted fringe image in the time domain and the given template (which can be called the time domain template).

[0034] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0035] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0036] See also Figure 1 , is a flow chart of a structured light 3D reconstruction method provided in an embodiment of the present application, such as Figure 1 As shown, the structured light 3D reconstruction method may include the following steps:

[0037] Step S100, acquiring an image of the object under test; wherein a specified pattern is projected onto the object under test, the specified pattern comprising multiple frames of integer coded patterns and multiple frames of line-shifted fringe patterns projected sequentially, the number of the integer coded patterns being consistent with the period of the line-shifted fringe patterns in the spatial domain.

[0038] In an embodiment of the present application, in order to achieve three-dimensional reconstruction of the object to be measured, a plurality of frames of integer coded patterns and a plurality of frames of line-shifted fringe patterns may be sequentially projected onto the object to be measured through a structured light source.

[0039] Exemplarily, for any line-shifted stripe pattern, pixel values ​​in the same column are the same, pixel values ​​in the same row are not completely the same, and are periodic (and may include multiple periods).

[0040] For example, a schematic diagram of a line-shifted fringe pattern can be as follows Figure 2A As shown ( Figure 2A Take the horizontal direction including 3 cycles as an example). That is, the line shifting stripe pattern can include multiple cycles in the spatial domain (i.e., in the horizontal direction of a frame of line shifting stripe image), and there are multiple pixels in one cycle, for example, Figure 2A 8 shown.

[0041] Among them, Figure 2A As shown, the period of the line-shifted stripe pattern in the spatial domain is 8, that is, within one period, the line-shifted stripe pattern can be cyclically shifted 8 times.

[0042] Exemplarily, the multiple frames of line-shifted stripe patterns projected sequentially may include a group of line-shifted stripe patterns (which may be referred to as sequential line-shifted stripe patterns) obtained by cyclic shifting.

[0043] Exemplarily, the sequential line-shifted fringe pattern shifts one pixel at a time in the time domain.

[0044] For example, the above-mentioned multi-frame line shift stripe pattern may include 8-frame line shift stripe pattern, which includes Figure 2A The line shift fringe pattern shown, and Figure 2A The line shift stripe pattern shown in FIG. 1 is a new line shift stripe pattern obtained by cyclic shifting, wherein the schematic diagram of the two frames can be shown as follows Figure 2B and Figure 2C shown.

[0045] Exemplarily, an image of the object under test projected with the above-mentioned designated pattern may be acquired.

[0046] Exemplarily, the image of the object under test obtained includes: an image of the object under test obtained when a line-shifted stripe pattern is projected (which may be referred to as a first type of image), and an image of the object under test obtained when an integer coded pattern is projected (which may be referred to as a second type of image).

[0047] Exemplarily, images of the object under test obtained when projecting different line-shifted stripe patterns correspond to different first-type images; images of the object under test obtained when projecting different integer coding patterns correspond to different second-type images.

[0048] Exemplarily, the image of the object to be measured can be acquired by a monocular camera or a binocular camera.

[0049] Exemplarily, the first type of image and the second type of image have the same resolution.

[0050] Exemplarily, the first type of image and the second type of image are acquired by the same camera, and during the process of acquiring the first type of image and the second type of image, the object to be measured and the structured light system must be relatively still.

[0051] Step S110: performing phase solution processing on a first type of image in the image of the measured object to obtain a first phase solution result.

[0052] Step S120: performing phase solution processing on the second type of image in the image of the measured object to obtain a second phase solution result.

[0053] In the embodiment of the present application, a phase solution process may be performed on the acquired first type of image to obtain a corresponding phase solution result (referred to herein as a first phase solution result).

[0054] The acquired second type of image may be subjected to phase solution processing to obtain a corresponding phase solution result (referred to herein as a second phase solution result).

[0055] It should be noted that, in order to improve the accuracy of subsequent processing, the acquired image of the measured object may be pre-processed, for example, filtering, distortion correction, etc., before subsequent processing such as phase solution.

[0056] For example, the acquired image can be preprocessed by Gaussian filtering to reduce image noise interference, and the image distortion can be corrected according to the camera calibration result.

[0057] Exemplarily, for a binocular system, epipolar correction may also be performed to obtain an undistorted image in which the left and right camera images are aligned.

[0058] Step S130: perform unwrapping processing on the first phase solution result according to the second phase solution result to obtain an absolute phase diagram.

[0059] In the embodiment of the present application, the line-shifted fringe image is used to implement sub-pixel level encoding.

[0060] Since the line-shifted fringe image is periodic, the de-phased result will also be periodic. For example, if the line-shifted fringe image has 128 periods in the horizontal direction, the de-phased result will also have 128 periods. This requires the phase to be wrapped in order to better represent and understand the sub-pixel phase information, that is, to use appropriate mathematical methods to constrain the phase value within a specific range to obtain more accurate sub-pixel encoding results.

[0061] It can be seen that the phase solution result obtained by phase solution of the first type of image is a wrapped phase, and the wrapped phase needs to be unwrapped to obtain the absolute phase.

[0062] The integer coded image is used to determine the line shift fringe order (i.e., the integer part of the absolute phase corresponding to the wrapped phase), and assists the line shift fringe pattern in phase unwrapping processing.

[0063] Correspondingly, when the first solution phase result and the second solution phase result are obtained in the above manner, the first solution phase result can be unwrapped according to the second solution phase result to obtain an absolute phase diagram.

[0064] Exemplarily, the absolute phase map includes the absolute phase of each pixel position.

[0065] Step S140: Perform three-dimensional reconstruction processing according to the obtained absolute phase map.

[0066] In the embodiment of the present application, when an absolute phase map is obtained in the above manner, three-dimensional reconstruction processing can be performed based on the obtained absolute phase map.

[0067] For example, for a binocular system (i.e., the image of the object to be measured is captured by a binocular camera in step S100), the point cloud and depth map can be calculated by binocular stereo matching and triangulation, and its specific implementation can be described in detail below.

[0068] It can be seen that in Figure 1In the method flow shown, by acquiring an image of the object to be measured with a specified pattern projected thereon, multiple frames of first type images and multiple frames of second type images are obtained, and phase solution processing is performed on the first type images to obtain a first phase solution result, and phase solution processing is performed on the second type images to obtain a second phase solution result, and based on the second phase solution result, the first phase solution result is unwrapped to obtain an absolute phase map, and then three-dimensional reconstruction processing is performed based on the absolute phase map, thereby achieving high-precision three-dimensional reconstruction.

[0069] In some embodiments, performing phase solution processing on the first type of image in the image of the measured object to obtain a first phase solution result may include:

[0070] For any pixel position in the first type of image, an integration operation is performed according to the grayscale values ​​of different first type images at the pixel position and the time domain template to obtain the integral response value corresponding to each time domain template; wherein the time domain template includes an initial template and a new template obtained by cyclically shifting the initial template, and the number of the time domain templates is consistent with the period of the line shift fringe pattern in the spatial domain; the initial template is determined according to the theoretical coding value of the specified pixel position in the time domain;

[0071] According to the position of the maximum value of the integral response value in the time domain, the first phase solution result corresponding to the pixel position is determined.

[0072] Exemplarily, in order to improve the decoding accuracy of line-shifted fringe patterns, the theoretical coding value of a single pixel position in the time domain can be used as a template. By performing time domain template integration on the serial line-shifted fringe images, the phase solution of the line-shifted fringe image is realized based on the sub-pixel position corresponding to the maximum value of the integral response value, thereby improving the phase solution accuracy and thus improving the accuracy of structured light three-dimensional reconstruction.

[0073] Exemplarily, the time domain template may include an initial template and a new template obtained by cyclically shifting the initial template.

[0074] The initial template is determined based on the theoretical encoding value of the specified pixel position in the time domain.

[0075] by Figure 2A to Figure 2C Taking the line-shifted stripe pattern shown in the figure as an example, assuming that the initial template corresponds to the pixel position in the upper left corner of the image, the theoretical coding values ​​of this pixel position in the time domain are 1, 0, 0, 0, 0, 1, 1, 1, respectively, so the initial template can be [1, 0, 0, 0, 0, 1, 1, 1].

[0076] The time domain template may also include a new template obtained by cyclically shifting the initial template, which may include [1, 1, 0, 0, 0, 0, 1, 1], [1, 1, 1, 0, 0, 0, 0, 1], [1, 1, 1, 1, 0, 0, 0, 0], [0, 1, 1, 1, 1, 0, 0, 0], [0, 0, 1, 1, 1, 1, 0, 0], [0, 0, 1, 1, 1, 1, 0, 0], and [0, 0, 0, 0, 1, 1, 1, 1].

[0077] It should be noted that in the embodiment of the present application, considering that the integral result is similar to a Gaussian distribution, expanding the amplitude of the integral response value can more accurately determine the maximum value and its corresponding position; when the template uses 1 and 0, the integration process is equivalent to accumulating only the grayscale value corresponding to 1. Therefore, in order to expand the amplitude of the response value, 0 in the template can be set to -1, and when the amplitude of the response value is expanded, it is consistent with 1. That is, the initial template can be [1, -1, -1, -1, -1, 1, 1, 1].

[0078] Exemplarily, for any pixel position in the first type of image, an integration operation is performed according to the grayscale values ​​of different first type images at the pixel position and the time domain template to obtain the integral response value corresponding to each time domain template.

[0079] For example, taking the above time domain template as an example, a set of line shift stripe patterns may include 8 frames (3 of which may be as follows Figure 2A to Figure 2C As shown, the remaining 5 frames can be obtained by shifting), and the images of the object to be measured (that is, the first type of image mentioned above) can be collected when the 8 frames of line-shifted stripe patterns are projected on the object to be measured. For any pixel position, the grayscale values ​​of the 8 frames of the first type of image at the pixel position are obtained respectively, and according to the grayscale values ​​of the 8 frames of the first type of image at the pixel position, integration operations are performed with each time domain template respectively to obtain 8 integral response values.

[0080] Taking the initial template as [1, -1, -1, -1, -1, 1, 1, 1] as an example, assuming that for a certain pixel position, the grayscale values ​​of the 8 frames of the first type of image at the pixel position are 28, 50, 24, 0, 0, 0, 0, 10 respectively, [28, 50, 24, 0, 0, 0, 0, 10] can be integrated with each time domain template respectively to obtain the integral response value corresponding to each time domain template. The specific implementation can be seen in Figure 3 .

[0081] For example, for the initial template, the integrated response value = 28*1+50*(-1)+24*(-1)+0*(-1)+0*(-1)+0*(-1)+0*(-1)+10*1=-36.

[0082] Exemplarily, when the integral response values ​​corresponding to each time domain template are determined, the first phase solution result corresponding to the pixel position may be determined according to the position of the maximum value of the integral response value in the time domain.

[0083] For example, assuming that the non-initial template in the time domain template is obtained by cyclically right shifting the initial template, a new time domain template can be obtained by cyclically right shifting the initial template, and the corresponding integral response value is calculated. The position with the largest integral response value represents that when the template moves to this position, it is closest to the actual pixel intensity in the time domain (that is, the actual collected pixel grayscale). That is, the position with the largest integral response value can uniquely identify the pixels in a single cycle.

[0084] For example, Figure 2A As shown, for 8 pixels in one cycle, a unique decimal value can be calculated for the 8 pixels.

[0085] In an example, determining the first phase solution result corresponding to the pixel position according to the position of the maximum value of the integral response value in the time domain may include:

[0086] Determine the search range based on the location of the maximum value of the integral response value in the time domain and the preset window size;

[0087] In the search range, the grayscale centroid method is used to determine the exact position of the maximum value;

[0088] The precise position of the maximum value is normalized to obtain the first solution phase result corresponding to the pixel position.

[0089] Exemplarily, when the position of the maximum value of the integral response value in the time domain is determined, the grayscale centroid method can be used to determine the first phase solution result.

[0090] Accordingly, the search range may be determined according to the location of the maximum value of the integral response value in the time domain and the preset window size.

[0091] The actual single pixel time domain grayscale and template integration results can be found in Figure 4 ,like Figure 4 As shown, the lower curve is the gray value, and the upper curve is the integral response value. The maximum value of the integral response value corresponds to the edge where the pixel changes from bright to dark in the time domain of the line shift fringe.

[0092] Considering that the maximum value of the integral response value may appear at the two boundaries of the period, the period must be expanded first when using the grayscale centroid method. That is, the second half of the integral result data is extended to the beginning and the first half is extended to the end, so that the data on both sides of the period form a loop, which is convenient for grayscale centroid method calculation.

[0093] Exemplarily, the schematic diagram of the periodic expansion of the integration result can be as follows: Figure 5 shown.

[0094] The grayscale centroid method can be used to calculate the maximum value formula as follows:

[0095]

[0096] Where h is the half window width, is the position with the largest integral response value, i corresponds to the template movement position, and pos is the integral result (the integral result after period expansion).

[0097] for , can be normalized to obtain the corresponding first solution phase result, where:

[0098]

[0099] Wherein, N is the number of line shift steps, that is, the number of projected images (consistent with the number of values ​​in the time domain integration template). For example, for an 8-step line shift, that is, N=8, the time domain integration template includes 8 values.

[0100] It should be noted that the grayscale centroid method is only an example of a method for calculating the maximum value of the integral response in the embodiment of the present application, and does not limit the range of calculating the maximum value of the integral response. In the embodiment of the present application, other methods for calculating the maximum value after integrating the line-shifted fringes can also be used, for example, the quadratic curve fitting method, and its specific implementation will not be described here.

[0101] In some embodiments, performing phase solution processing on the second type of image in the image of the measured object to obtain a second phase solution result may include:

[0102] For any pixel position in the second type image, binarization processing is performed on the grayscale values ​​of different second type images at the pixel position according to the binarization threshold value of the pixel position to obtain binarization results of different second type images at the pixel position;

[0103] A second phase solution result at the pixel position is determined according to the binarization results of different second type images at the pixel position.

[0104] Exemplarily, for any pixel position in the second type image, the grayscale values ​​of different second type images at the pixel position are binarized according to the binarization threshold of the pixel position to obtain the binarization results of different second type images at the pixel position.

[0105] For example, assuming that the integer coding pattern projected in sequence includes 8 frames, the number of second type images is also 8 frames. For any pixel position, the grayscale values ​​of the 8 frames of second type images at the pixel position can be binarized according to the binarization threshold of the pixel position.

[0106] For example, for any grayscale value of the 8 frames of the second type image at the pixel position, if the grayscale value is greater than the binarization threshold of the pixel position, the grayscale value of the pixel position can be set to 1; otherwise, it can be set to 0.

[0107] Exemplarily, the second phase solution result at the pixel position may be determined based on the binarization results of different second type images at the pixel position.

[0108] For example, assuming that the binarization results of 8 frames of the second type of image at the pixel position are 00001111 respectively, the mapping rule of the binarization result can be determined according to the encoding method of the integer coding pattern, and the binarization result can be mapped based on the mapping rule, for example, mapped to an on-demand arranged and increasing result, and then, the second phase solution result can be determined based on the mapped result.

[0109] In one example, for any pixel position in the second type of image, the binarization threshold of the pixel position may be determined in the following manner:

[0110] Determine a maximum grayscale value and a minimum grayscale value according to the grayscale value of each first type image at the pixel position;

[0111] The binarization threshold of the pixel position is determined based on the maximum grayscale value and the minimum grayscale value.

[0112] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below in conjunction with specific application scenarios.

[0113] In this embodiment, if Figure 6 As shown, structured light 3D reconstruction may include steps such as coded image projection and acquisition, image preprocessing, image correction, image phase solution, and point cloud and depth map calculation.

[0114] The implementation of each step is described below.

[0115] 1. Coded Image Projection and Collection

[0116] Through a structured light source such as a projector (or laser galvanometer), multiple frames of integer coded patterns and multiple frames of line-shifted fringe patterns are projected onto the surface of the object to be measured in sequence, and the image of the object to be measured is synchronously collected through a camera, such as a binocular camera, for three-dimensional reconstruction of the object to be measured.

[0117] Exemplarily, the integer coding pattern may include a projection pattern such as a Gray code or an XOR code for assisting the line-shifted fringe pattern in performing wrapped phase unwrapping.

[0118] 2. Image Preprocessing

[0119] Exemplarily, the captured image may be pre-processed by Gaussian filtering to reduce image noise interference.

[0120] 3. Image Correction

[0121] Exemplarily, the image distortion may be corrected according to the calibration result of the camera.

[0122] Exemplarily, for a binocular system, epipolar correction is also performed to obtain an undistorted image in which the left and right camera images are aligned.

[0123] 4. Image Phase Deconstruction

[0124] Exemplarily, the integer coded image (ie, the second type of image) and the line-shifted fringe image (ie, the second type of image) may be phase-decoded respectively.

[0125] Exemplarily, the line-shift fringe image is used to achieve sub-pixel encoding, and the phase unpacking result is the wrapped phase. The wrapped phase needs to be unpacked to obtain the absolute phase, and the phase unpacking accuracy directly affects the accuracy of three-dimensional reconstruction.

[0126] The integer coded image is used to determine the order of the line shift fringes and assist the line shift fringes in unwrapping the phase, and the unwrapping result will not affect the reconstruction accuracy.

[0127] Exemplarily, for phase dephasing of a line-shifted fringe image, the theoretical coding value of the line-shifted fringe image in the time domain can be used as an initial template, and an integration operation is performed on it and the grayscale value of the actual line-shifted fringe image captured by the camera. The template is shifted right one position each time to calculate the integral response value of each position in the time domain. The position with the largest integral response value represents the position that is closest to the actual pixel intensity in the time domain when the template moves to this position.

[0128] Exemplarily, the initial template may be set to be consistent with the theoretical encoding value, and the moving position of the template, that is, the position with the maximum response value, can uniquely identify the pixels in a single period.

[0129] Assuming that the theoretical encoding value of the line-shifted fringe image in the time domain is [1, 0, 0, 0, 0, 1, 1, 1], in order to expand the amplitude of the response value, the template can be set to [1, -1, -1, -1, -1, 1, 1, 1].

[0130] The single pixel template integral calculation process can be as follows Figure 3As shown, the position with the largest integral response value is traversed in the time domain. With the position with the largest integral response value as the center, a window size (for example, 5) is selected, and the exact position of the maximum value is calculated according to the grayscale centroid method. After normalization, it is the decoding result of the pixel.

[0131] like Figure 4 As shown in the figure, the maximum response value corresponds to the edge where the pixel changes from bright to dark in the time domain of the line shift stripe. The maximum value may appear at the two boundaries of the cycle, so when using the grayscale centroid method for calculation, the cycle must be expanded first so that the data on both sides of the cycle form a loop and can be calculated normally.

[0132] in, Figure 4 The horizontal axis represents the position of the template movement in the time domain; the vertical axis represents the integrated response value.

[0133] Exemplarily, in the process of dephasing the line-shifted fringe image, the maximum grayscale value and the minimum grayscale value of each pixel position in the time domain can be calculated to generate a maximum grayscale image and a minimum grayscale image as thresholds for binarization of the integer coded image.

[0134] Compared with the implementation method of projecting a completely black image and a completely white image respectively to determine the threshold value for binarization in the traditional scheme, the above scheme provided in the embodiment of the present application can reduce the number of projected images; in addition, the brightness and darkness of the integer-coded image actually projected in the traditional scheme and the brightness and darkness of the completely black / completely white image may not be completely consistent, resulting in errors in binarization in some complex scenes, while the above scheme provided in the embodiment of the present application can improve the accuracy of binarization while reducing the projection of black and white patterns.

[0135] Exemplarily, for any pixel position, the average value of the maximum grayscale value and the minimum grayscale value of the pixel position may be used as a threshold for binarization of the integer coding pattern.

[0136] After completing the integer coded image dephasing and the line-shifted fringe image dephasing, the wrapped phase is unfolded according to the following formula to obtain the absolute phase.

[0137]

[0138] Among them, k is the phase solution result (or integer decoding result) of the integer coded image, It is the dephasing result (or fractional decoding result) of the line-shifted fringe image. is the absolute phase value.

[0139] 5. Point cloud and depth map calculation

[0140] Exemplarily, when an absolute phase map is obtained, three-dimensional reconstruction can be performed in a variety of ways.

[0141] For example, for a monocular system, the point cloud depth map calculation can be completed based on the pre-calibrated phase-height or phase-disparity curve.

[0142] For the binocular system, point cloud and depth map can be calculated through binocular stereo matching and triangulation.

[0143] Take the binocular system as an example. First, binocular stereo matching can be performed. Since the binocular epipolar line correction has been completed, the left and right camera images are aligned in rows, and only the column direction needs to be matched.

[0144] First, the phase values ​​of the single row of the left and right camera phase images are sorted, and based on the sorting results, the left and right integer pixel coordinates corresponding to each pixel under the other camera are determined, and the sub-pixel coordinates are calculated by linear interpolation based on the absolute phase value, and then the binocular disparity value of each pixel is obtained. Finally, the three-dimensional coordinates of the point cloud are calculated based on the camera imaging model and triangulation principle, and the point cloud and depth map reconstruction are completed.

[0145] The imaging model of the camera can be expressed as:

[0146]

[0147] Where f is the focal length, ( , ) is the principal point coordinate, which can be obtained through camera calibration; (u, v) is the pixel coordinate; (X, Y, Z) is the point cloud three-dimensional coordinate.

[0148] According to the triangulation principle, we can get:

[0149]

[0150] Among them, b is the base distance and d is the parallax value.

[0151] Then the calculation formulas for X and Y are as follows:

[0152]

[0153] The method provided by the present application is described above. The device provided by the present application is described below:

[0154] See also Figure 7 , is a schematic diagram of the structure of a structured light 3D reconstruction device provided in an embodiment of the present application, such as Figure 7 As shown, the structured light 3D reconstruction device may include:

[0155] An acquisition unit 710 is used to acquire an image of the object under test; wherein the object under test is projected with a specified pattern, the specified pattern comprising multiple frames of integer coded patterns and multiple frames of line-shifted fringe patterns projected in sequence, and the number of the integer coded patterns is consistent with the period of the line-shifted fringe pattern in the spatial domain;

[0156] The dephasing unit 720 is used to perform dephasing on a first type of image in the image of the object to be measured to obtain a first dephasing result; wherein the first type of image is an image of the object to be measured obtained by sequentially projecting the multiple frames of line-shifted fringe patterns;

[0157] The dephasing unit 720 is further used to perform dephasing on a second type of image in the image of the object to be measured to obtain a second dephasing result; wherein the second type of image is an image of the object to be measured obtained when the multiple frames of integer coded patterns are sequentially projected;

[0158] An unwrapping unit 730, configured to perform unwrapping processing on the first phase solution result according to the second phase solution result to obtain an absolute phase map;

[0159] The reconstruction unit 740 is used to perform three-dimensional reconstruction processing according to the absolute phase map.

[0160] In some embodiments, the de-phase unit 720 performs de-phase processing on the first type of image in the image of the object to be measured to obtain a first de-phase result, including:

[0161] For any pixel position in the first type of image, an integration operation is performed according to the grayscale values ​​of different first type images at the pixel position and the time domain template to obtain an integral response value corresponding to each time domain template; wherein the time domain template includes an initial template and a new template obtained by cyclically shifting the initial template, and the number of the time domain templates is consistent with the period of the line-shifted fringe pattern in the spatial domain; the initial template is determined according to the theoretical coding value of the specified pixel position in the time domain;

[0162] According to the position of the maximum value of the integral response value in the time domain, the first phase solution result corresponding to the pixel position is determined.

[0163] In some embodiments, the phase solution unit 720 determines the first phase solution result corresponding to the pixel position according to the position of the maximum value of the integral response value in the time domain, including:

[0164] Determine the search range based on the location of the maximum value of the integral response value in the time domain and the preset window size;

[0165] In the search range, the grayscale centroid method is used to determine the exact position of the maximum value;

[0166] The precise position of the maximum value is normalized to obtain a first phase solution result corresponding to the pixel position.

[0167] In some embodiments, the de-phase unit 720 performs de-phase processing on the second type of image in the image of the measured object to obtain a second de-phase result, including:

[0168] For any pixel position in the second type image, binarization processing is performed on the grayscale values ​​of different second type images at the pixel position according to the binarization threshold value of the pixel position to obtain binarization results of different second type images at the pixel position;

[0169] A second phase solution result at the pixel position is determined according to the binarization results of different second type images at the pixel position.

[0170] In some embodiments, for any pixel position in the second type of image, the binarization threshold of the pixel position is determined by:

[0171] Determine a maximum grayscale value and a minimum grayscale value according to the grayscale value of each first type image at the pixel position;

[0172] A binarization threshold of the pixel position is determined according to the maximum grayscale value and the minimum grayscale value.

[0173] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the vehicle camera calibration method described above.

[0174] See also Figure 8 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 801 and a memory 802 storing machine executable instructions. The processor 801 and the memory 802 may communicate via a system bus 803. In addition, by reading and executing the machine executable instructions corresponding to the vehicle camera calibration logic in the memory 802, the processor 801 may execute the vehicle camera calibration method described above.

[0175] The memory 802 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0176] In some embodiments, a machine-readable storage medium is also provided. Figure 8The memory 802 in the machine-readable storage medium stores machine executable instructions, and when the machine executable instructions are executed by the processor, the vehicle-mounted camera calibration method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Claims

1. A structured light 3D reconstruction method, characterized in that: include: Acquire an image of the object to be measured; wherein the object to be measured is projected with a specified pattern, the specified pattern includes multiple frames of integer coded patterns and multiple frames of line-shifted fringe patterns projected in sequence, and the number of the integer coded patterns is consistent with the period of the line-shifted fringe pattern in the spatial domain; Performing phase solution processing on a first type of image in the image of the object to be measured to obtain a first phase solution result; wherein the first type of image is an image of the object to be measured obtained when the multiple frames of line-shifted fringe patterns are sequentially projected; Performing phase solution processing on a second type of image in the image of the object to be measured to obtain a second phase solution result; wherein the second type of image is an image of the object to be measured obtained when the multiple frames of integer coded patterns are sequentially projected; According to the second phase solution result, performing unwrapping processing on the first phase solution result to obtain an absolute phase diagram; A three-dimensional reconstruction process is performed according to the absolute phase map.

2. The method according to claim 1, characterized in that The performing phase solution processing on the first type of image in the image of the measured object to obtain a first phase solution result includes: For any pixel position in the first type of image, an integration operation is performed according to the grayscale values ​​of different first type images at the pixel position and the time domain template to obtain an integral response value corresponding to each time domain template; wherein the time domain template includes an initial template and a new template obtained by cyclically shifting the initial template, and the number of the time domain templates is consistent with the period of the line-shifted fringe pattern in the spatial domain; the initial template is determined according to the theoretical coding value of the specified pixel position in the time domain; According to the position of the maximum value of the integral response value in the time domain, the first phase solution result corresponding to the pixel position is determined.

3. The method according to claim 2, characterized in that The determining, according to the position of the maximum value of the integral response value in the time domain, a first solution phase result corresponding to the pixel position includes: Determine the search range based on the location of the maximum value of the integral response value in the time domain and the preset window size; In the search range, the grayscale centroid method is used to determine the exact position of the maximum value; The precise position of the maximum value is normalized to obtain a first phase solution result corresponding to the pixel position.

4. The method according to claim 1, characterized in that: The performing phase solution processing on the second type of image in the image of the measured object to obtain a second phase solution result includes: For any pixel position in the second type image, binarization processing is performed on the grayscale values ​​of different second type images at the pixel position according to the binarization threshold value of the pixel position to obtain binarization results of different second type images at the pixel position; A second phase solution result at the pixel position is determined according to the binarization results of different second type images at the pixel position.

5. The method according to claim 4, characterized in that For any pixel position in the second type of image, the binarization threshold of the pixel position is determined by: Determine a maximum grayscale value and a minimum grayscale value according to the grayscale value of each first type image at the pixel position; A binarization threshold of the pixel position is determined according to the maximum grayscale value and the minimum grayscale value.

6. A structured light 3D reconstruction device, characterized in that: include: An acquisition unit, configured to acquire an image of the object under test; wherein the object under test is projected with a designated pattern, the designated pattern comprising a plurality of frames of integer coded patterns and a plurality of frames of line-shifted fringe patterns projected in sequence, the number of the integer coded patterns being consistent with the period of the line-shifted fringe pattern in the spatial domain; A dephasing unit, configured to perform dephasing on a first type of image in the image of the object to be measured, to obtain a first dephasing result; wherein the first type of image is an image of the object to be measured obtained by sequentially projecting the multiple frames of line-shifted fringe patterns; The dephasing unit is further used to perform dephasing on a second type of image in the image of the object to be measured to obtain a second dephasing result; wherein the second type of image is an image of the object to be measured obtained when the multiple frames of integer coded patterns are sequentially projected; an unwrapping unit, configured to perform an unwrapping process on the first phase solution result according to the second phase solution result to obtain an absolute phase map; A reconstruction unit is used to perform three-dimensional reconstruction processing according to the absolute phase map.

7. The device according to claim 6, characterized in that The phase solution unit performs phase solution processing on the first type of image in the image of the measured object to obtain a first phase solution result, including: For any pixel position in the first type of image, an integration operation is performed according to the grayscale values ​​of different first type images at the pixel position and the time domain template to obtain an integral response value corresponding to each time domain template; wherein the time domain template includes an initial template and a new template obtained by cyclically shifting the initial template, and the number of the time domain templates is consistent with the period of the line-shifted fringe pattern in the spatial domain; the initial template is determined according to the theoretical coding value of the specified pixel position in the time domain; Determine the first solution phase result corresponding to the pixel position according to the position of the maximum value of the integral response value in the time domain; The phase solution unit determines the first phase solution result corresponding to the pixel position according to the position of the maximum value of the integral response value in the time domain, including: Determine the search range based on the location of the maximum value of the integral response value in the time domain and the preset window size; In the search range, the grayscale centroid method is used to determine the exact position of the maximum value; The precise position of the maximum value is normalized to obtain a first phase solution result corresponding to the pixel position.

8. The device according to claim 6, characterized in that The phase solution unit performs phase solution processing on the second type of image in the image of the measured object to obtain a second phase solution result, including: For any pixel position in the second type image, binarization processing is performed on the grayscale values ​​of different second type images at the pixel position according to the binarization threshold value of the pixel position to obtain binarization results of different second type images at the pixel position; Determine a second phase solution result at the pixel position according to the binarization results of different second type images at the pixel position; Wherein, for any pixel position in the second type of image, the binarization threshold of the pixel position is determined by the following method: Determine a maximum grayscale value and a minimum grayscale value according to the grayscale value of each first type image at the pixel position; A binarization threshold of the pixel position is determined according to the maximum grayscale value and the minimum grayscale value.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 5.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images