Three-dimensional reconstruction method, system and device and electronic equipment

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method, system and device and electronic equipment, and relates to the technical field of machine vision. The method comprises the steps of obtaining a plurality of first images and a plurality of second images for each exposure duration in a plurality of exposure durations; calculating to obtain an absolute phase diagram and a modulation degree diagram under the exposure duration by utilizing each first image and each second image under the exposure duration; based on the modulation degree and / or the stripe sinusoidal error of each pixel position under the exposure duration, calculating the confidence coefficient of the absolute phase of the pixel position under the exposure duration; selecting the corresponding absolute phase with the maximum confidence coefficient from the absolute phases of each pixel position under each exposure duration to obtain a fused absolute phase diagram; and performing three-dimensional reconstruction based on the fused absolute phase diagram to obtain a three-dimensional reconstruction result of the to-be-detected region. Therefore, the method can be suitable for a high-dynamic-range scene, and the accuracy and integrity of an obtained three-dimensional reconstruction result are improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a three-dimensional reconstruction method, system, device and electronic equipment. Background Art

[0002] In the field of machine vision technology, three-dimensional reconstruction technology based on structured light has the advantages of fast speed, high accuracy, and large data volume, and is widely used in industrial scenarios such as workpiece size measurement, defect detection, and robotic arm grasping and positioning. However, general industrial scenes are relatively complex, and the same scene may include workpieces with very different reflectivity, such as black and white plastic parts, dark, rusty metal workpieces, and shiny metal workpieces. For the above-mentioned high dynamic range scenes, there are often local overexposed or local dark areas in the images captured under a single exposure. In the process of dephasing the image captured by the image acquisition device, this part of the image is difficult to dephasize or the dephasing is inaccurate, resulting in missing or low reconstruction accuracy in the obtained three-dimensional reconstruction results.

[0003] Therefore, there is an urgent need for a 3D reconstruction method suitable for the above-mentioned high dynamic range scenes to improve the accuracy and completeness of the obtained 3D reconstruction results. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a 3D reconstruction method, system, device, and electronic device suitable for high dynamic range scenes, and to improve the accuracy and completeness of the obtained 3D reconstruction results. The specific technical solutions are as follows:

[0005] In a first aspect of an embodiment of the present application, a three-dimensional reconstruction method is provided, the method comprising:

[0006] For each of the plurality of exposure times, acquiring an image acquired during the exposure time when each binary-coded image is projected onto the area to be measured, thereby obtaining a plurality of first images; and acquiring an image acquired during the exposure time when each fringe image is projected onto the area to be measured, thereby obtaining a plurality of second images.

[0007] Using each first image and each second image under the exposure time, an absolute phase map and a modulation map under the exposure time are calculated; wherein the absolute phase map records the absolute phase of each pixel position; and the modulation map records the modulation of each pixel position;

[0008] Calculating a confidence level of the absolute phase of each pixel position at the exposure duration based on the modulation level and / or the fringe sinusoidal error at the exposure duration; wherein the fringe sinusoidal error at each pixel position at the exposure duration represents the degree to which the grayscale value of each pixel position in each second image at the exposure duration conforms to a sine function; and the confidence level of the absolute phase of a pixel position at the exposure duration is positively correlated with the modulation level of the pixel position at the exposure duration and negatively correlated with the fringe sinusoidal error at the exposure duration.

[0009] From the absolute phases of each pixel position at each exposure time, the absolute phase with the highest confidence is selected to obtain the fused absolute phase map;

[0010] Three-dimensional reconstruction is performed based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured.

[0011] In some embodiments, calculating the absolute phase map and the modulation map under the exposure time using each first image and each second image under the exposure time includes:

[0012] For each pixel position, calculating a decoded value of the pixel position at the exposure duration based on the grayscale values ​​of the pixel position in each first image at the exposure duration;

[0013] Using each second image under the exposure time, a relative phase map and a modulation map under the exposure time are calculated; wherein the projected fringe images are multiple line-shifted fringe images or multiple phase-shifted fringe images; and the relative phase map records the relative phase of each pixel position;

[0014] The decoded value of each pixel position at the exposure time is used to perform phase unwrapping on the relative phase map at the exposure time to obtain an absolute phase map at the exposure time; wherein the absolute phase of each pixel position is recorded in the absolute phase map.

[0015] In some embodiments, calculating the relative phase map and the modulation map for the exposure time using each second image for the exposure time includes:

[0016] For each pixel position, calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the sine value of the phase shift of the corresponding fringe image as the first inner product of the pixel position;

[0017] Calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the cosine value of the phase shift of the corresponding fringe image as the second inner product of the pixel position;

[0018] Calculate the ratio of the first inner product to the second inner product at the pixel position under the exposure duration, and use the arc tangent of the ratio as the relative phase of the pixel position under the exposure duration, to obtain a relative phase map under the exposure duration; calculate the sum of the squares of the first inner product and the second inner product at the pixel position under the exposure duration, and use the normalized value of the square root of the calculated sum of squares as the modulation degree of the pixel position under the exposure duration, to obtain a modulation degree map under the exposure duration.

[0019] In some embodiments, performing phase unwrapping on the relative phase map for the exposure duration using the decoded value of each pixel position for the exposure duration to obtain the absolute phase map for the exposure duration includes:

[0020] For each pixel position, the relative phase of the pixel position at the exposure time is calculated, and the sum of the phase change corresponding to the decoded value of the pixel position at the exposure time is used as the absolute phase of the pixel position at the exposure time to obtain the absolute phase map at the exposure time.

[0021] In some embodiments, the fringe sinusoidal error at each pixel position under the exposure duration is calculated by the following steps:

[0022] Using each second image under the exposure time, a background intensity map under the exposure time is calculated; wherein the background intensity map records the background intensity of each pixel position;

[0023] For each second image under the exposure time, calculating the relative phase of each pixel position under the exposure time and the cosine value of the difference between the phase shift amount of the fringe image corresponding to the second image;

[0024] The product of the calculated cosine value and the modulation degree of the pixel position under the exposure time;

[0025] The calculated product and the sum of the background intensity at the pixel position at the exposure time are used to obtain a theoretical grayscale value of the pixel position at the exposure time for the second image;

[0026] The difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time is calculated to obtain the fringe sinusoidal error of the pixel position at the exposure time.

[0027] In some embodiments, calculating the difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time to obtain the fringe sinusoidal error of the pixel position at the exposure time includes:

[0028] For each pixel position, calculating the absolute value of the difference between a theoretical grayscale value of the pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in the second image as the error component of the pixel position for the second image;

[0029] The average of the error components of the pixel position for each second image is calculated as the fringe sinusoidal error of the pixel position under the exposure time.

[0030] In some embodiments, calculating the confidence level of the absolute phase of each pixel position at the exposure duration based on the modulation degree and / or fringe sine error at the exposure duration includes:

[0031] Normalizing the fringe sinusoidal error at each pixel position under the exposure time to a range of 0 to 1, thereby obtaining a normalized value of the fringe sinusoidal error at each pixel position under the exposure time;

[0032] A weighted sum of the modulation degree and the normalized value of the fringe sinusoidal error at each pixel position under the exposure duration is calculated as the confidence level of the absolute phase of the pixel position under the exposure duration.

[0033] In some embodiments, each first image and each second image at each exposure time is acquired by using an image acquisition device;

[0034] The performing three-dimensional reconstruction based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured includes:

[0035] In the case where the image acquisition device is a binocular camera, based on a preset binocular 3D reconstruction algorithm, 3D reconstruction is performed on the fused absolute phase images obtained based on the images acquired by the two cameras to obtain a 3D reconstruction result of the area to be measured;

[0036] or,

[0037] When the image acquisition device is a monocular camera, three-dimensional reconstruction is performed on the obtained fused absolute phase image based on a preset monocular three-dimensional reconstruction algorithm to obtain a three-dimensional reconstruction result of the area to be measured.

[0038] According to a second aspect of an embodiment of the present application, a three-dimensional reconstruction system is provided, the system comprising: a projection device, an image acquisition device, and a processor;

[0039] The projection device is used to project the binary coded image and the fringe image onto the area to be measured;

[0040] The image acquisition device is used to acquire images of the area to be measured when the projection device projects each image onto the area to be measured at different exposure times;

[0041] The processor is used to execute any three-dimensional reconstruction method described in the first aspect above.

[0042] According to a third aspect of the present application, a three-dimensional reconstruction apparatus is provided, comprising:

[0043] an image acquisition module configured to acquire, for each of a plurality of exposure durations, an image acquired during the exposure duration when each binary-coded image is projected onto the area to be measured, thereby obtaining a plurality of first images; and to acquire, for each of a plurality of exposure durations, an image acquired during the exposure duration when each fringe image is projected onto the area to be measured, thereby obtaining a plurality of second images;

[0044] An image processing module, configured to calculate an absolute phase map and a modulation map for the exposure time using each of the first and second images for the exposure time; wherein the absolute phase map records the absolute phase of each pixel position; and the modulation map records the modulation of each pixel position;

[0045] A confidence acquisition module is configured to calculate a confidence level of the absolute phase of each pixel position at the exposure duration based on the modulation level and / or the fringe sinusoidal error at the exposure duration. The fringe sinusoidal error at each pixel position at the exposure duration represents the degree to which the grayscale value of each pixel position in each second image at the exposure duration conforms to a sine function. The confidence level of the absolute phase of a pixel position at the exposure duration is positively correlated with the modulation level of the pixel position at the exposure duration and negatively correlated with the fringe sinusoidal error at the exposure duration.

[0046] A fusion module is used to select the absolute phase with the highest confidence from the absolute phases of each pixel position under various exposure times to obtain a fused absolute phase map;

[0047] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured.

[0048] In some embodiments, the image processing module includes:

[0049] a decoding value calculation submodule, configured to calculate, for each pixel position, a decoding value of the pixel position at the exposure duration based on the grayscale value of the pixel position in each first image at the exposure duration;

[0050] a relative phase calculation submodule, configured to calculate a relative phase map and a modulation map for the exposure time using each second image for the exposure time; wherein the projected fringe images are multiple line-shifted fringe images or multiple phase-shifted fringe images; and the relative phase map records the relative phase of each pixel position;

[0051] The absolute phase calculation submodule is used to perform phase expansion on the relative phase map for the exposure duration using the decoded value of each pixel position for the exposure duration to obtain the absolute phase map for the exposure duration; wherein the absolute phase map records the absolute phase of each pixel position.

[0052] In some embodiments, the relative phase calculation submodule is specifically configured to:

[0053] For each pixel position, calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the sine value of the phase shift of the corresponding fringe image as the first inner product of the pixel position;

[0054] Calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the cosine value of the phase shift of the corresponding fringe image as the second inner product of the pixel position;

[0055] Calculate the ratio of the first inner product to the second inner product at the pixel position under the exposure duration, and use the arc tangent of the ratio as the relative phase of the pixel position under the exposure duration, to obtain a relative phase map under the exposure duration; calculate the sum of the squares of the first inner product and the second inner product at the pixel position under the exposure duration, and use the normalized value of the square root of the calculated sum of squares as the modulation degree of the pixel position under the exposure duration, to obtain a modulation degree map under the exposure duration.

[0056] In some embodiments, the absolute phase calculation submodule is specifically configured to:

[0057] For each pixel position, the relative phase of the pixel position at the exposure time is calculated, and the sum of the phase change corresponding to the decoded value of the pixel position at the exposure time is used as the absolute phase of the pixel position at the exposure time to obtain the absolute phase map at the exposure time.

[0058] In some embodiments, the fringe sinusoidal error at each pixel position under the exposure duration is calculated by the following steps:

[0059] Using each second image under the exposure time, a background intensity map under the exposure time is calculated; wherein the background intensity map records the background intensity of each pixel position;

[0060] For each second image under the exposure time, calculating the relative phase of each pixel position under the exposure time and the cosine value of the difference between the phase shift amount of the fringe image corresponding to the second image;

[0061] The product of the calculated cosine value and the modulation degree of the pixel position under the exposure time;

[0062] The calculated product and the sum of the background intensity at the pixel position at the exposure time are used to obtain a theoretical grayscale value of the pixel position at the exposure time for the second image;

[0063] The difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time is calculated to obtain the fringe sinusoidal error of the pixel position at the exposure time.

[0064] In some embodiments, calculating the difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time to obtain the fringe sinusoidal error of the pixel position at the exposure time includes:

[0065] For each pixel position, calculating the absolute value of the difference between a theoretical grayscale value of the pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in the second image as the error component of the pixel position for the second image;

[0066] The average of the error components of the pixel position for each second image is calculated as the fringe sinusoidal error of the pixel position under the exposure time.

[0067] In some embodiments, the confidence acquisition module is specifically configured to:

[0068] Normalizing the fringe sinusoidal error at each pixel position under the exposure time to a range of 0 to 1, thereby obtaining a normalized value of the fringe sinusoidal error at each pixel position under the exposure time;

[0069] A weighted sum of the modulation degree and the normalized value of the fringe sinusoidal error at each pixel position under the exposure duration is calculated as the confidence level of the absolute phase of the pixel position under the exposure duration.

[0070] In some embodiments, each first image and each second image at each exposure time is acquired by using an image acquisition device;

[0071] The three-dimensional reconstruction module is specifically used to:

[0072] In the case where the image acquisition device is a binocular camera, based on a preset binocular 3D reconstruction algorithm, 3D reconstruction is performed on the fused absolute phase images obtained based on the images acquired by the two cameras to obtain a 3D reconstruction result of the area to be measured;

[0073] or,

[0074] When the image acquisition device is a monocular camera, three-dimensional reconstruction is performed on the obtained fused absolute phase image based on a preset monocular three-dimensional reconstruction algorithm to obtain a three-dimensional reconstruction result of the area to be measured.

[0075] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, including:

[0076] Memory for storing computer programs;

[0077] The processor is configured to implement any of the above-mentioned three-dimensional reconstruction methods when executing the program stored in the memory.

[0078] In another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned three-dimensional reconstruction methods is implemented.

[0079] In another aspect of the embodiments of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to perform any of the above-mentioned three-dimensional reconstruction methods.

[0080] Beneficial effects of the embodiments of the present application:

[0081] Based on the three-dimensional reconstruction method provided by the embodiment of the present application, the first image and the second image acquired at different exposure times can be used and processed separately to obtain the absolute phase map at each exposure time, as well as the confidence level corresponding to each pixel position in the absolute phase map at each exposure time. Accordingly, after obtaining the absolute phase map at each exposure time, the absolute phase map at each exposure time can be fused with one pixel position as the fusion granularity. The confidence level corresponding to each pixel position in the absolute phase map at each exposure time is calculated based on the modulation and / or fringe sinusoidal error of the pixel position at the exposure time. Moreover, the greater the modulation of a pixel position at an exposure time, the greater the contrast between the light and dark fringes at the pixel position in each second image at the exposure time. The smaller the fringe sinusoidal error of a pixel position at an exposure time, the more the grayscale value of the pixel position in each second image at the exposure time conforms to the variation pattern of the light and dark fringes in the projected fringe image, and the more it conforms to the sine function. Therefore, the modulation degree and fringe sinusoidal error of a pixel position at the exposure time can represent the accuracy of the absolute phase of the pixel position in the absolute phase map at the exposure time from different dimensions. That is to say, based on the three-dimensional reconstruction method provided in the embodiment of the present application, at least one item can be selected from the above-mentioned different dimensions to calculate the accuracy of the absolute phase of each pixel position at the exposure time. Furthermore, for each pixel position, the absolute phase with the highest corresponding confidence is selected from the absolute phases of the pixel position at each exposure time to obtain a fused absolute phase map. In this way, the accuracy of the fused absolute phase map is improved. Furthermore, three-dimensional reconstruction based on the fused absolute phase map can be applicable to high dynamic range scenes and improve the accuracy of the obtained three-dimensional reconstruction results.

[0082] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0084] Figure 1 A first flow chart of the three-dimensional reconstruction method provided in an embodiment of the present application;

[0085] FIG2( a ) is a schematic diagram of the first image in a set of Gray code images provided in an embodiment of the present application;

[0086] FIG2( b ) is a schematic diagram of the second image in a set of Gray code images provided in an embodiment of the present application;

[0087] FIG2( c ) is a schematic diagram of the third image in a set of Gray code images provided in an embodiment of the present application;

[0088] FIG2( d ) is a schematic diagram of the fourth image in a set of Gray code images provided in an embodiment of the present application;

[0089] Figure 3 A schematic diagram of a three-dimensional reconstruction process provided in an embodiment of the present application;

[0090] FIG4( a ) is an example diagram of three-dimensional reconstruction performed at a first exposure time according to an embodiment of the present application;

[0091] FIG4( b ) is an example diagram of three-dimensional reconstruction performed at a second exposure time according to an embodiment of the present application;

[0092] FIG4( c ) is a schematic diagram of three-dimensional reconstruction based on the fused absolute phase image provided by an embodiment of the present application;

[0093] Figure 5 A schematic structural diagram of a three-dimensional reconstruction system provided in an embodiment of the present application;

[0094] Figure 6 A structural diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0095] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0097] The 3D reconstruction technology based on structured light has the advantages of fast speed, high accuracy, and large data volume, and is widely used in industrial scenarios such as workpiece size measurement, defect detection, and robotic arm grasping and positioning. However, general industrial scenes are relatively complex, and the same scene may include workpieces with very different reflectivity, such as black and white plastic parts, dark, rusty metal workpieces, and shiny metal workpieces. For the above-mentioned high dynamic range scenes, there are often local overexposed or local dark areas in the images captured under a single exposure. In the process of dephasing the images captured by the image acquisition device, this part of the image is difficult to dephasize or the dephasing is inaccurate, resulting in missing or low reconstruction accuracy in the obtained 3D reconstruction results.

[0098] Incomplete 3D reconstruction results (i.e., missing parts of the 3D reconstruction result) indicate that there are areas that cannot be reconstructed during 3D reconstruction using images acquired with a single exposure. Inaccurate 3D reconstruction results (i.e., poor quality 3D reconstruction results) mean that, during 3D reconstruction using images acquired with a single exposure, although 3D reconstruction results are obtained for some areas, the images may be overly bright or dark due to inappropriate exposure parameters selected during image acquisition. In this case, the 3D reconstruction results may be inaccurate.

[0099] Therefore, there is an urgent need for a 3D reconstruction method suitable for the above-mentioned high dynamic range scenes to improve the accuracy and completeness of the obtained 3D reconstruction results.

[0100] The present invention provides a 3D reconstruction method that can be applied to a 3D reconstruction system. The 3D reconstruction system includes a projection device, an image acquisition device, and a processor with image processing capabilities. For example, the processor can be a server or a computer; the image acquisition device can be a binocular camera or a monocular camera.

[0101] See also Figure 1 , Figure 1 This is a first flow chart of a 3D reconstruction method provided in an embodiment of the present application, the method comprising the following steps:

[0102] S101: For each exposure time among multiple exposure times, obtain the image when each binary coded image is projected onto the area to be tested, which is collected during the exposure time, to obtain multiple first images; and obtain the image when each stripe image is projected onto the area to be tested, which is collected during the exposure time, to obtain multiple second images.

[0103] S102: Calculate an absolute phase map and a modulation map under the exposure time using each first image and each second image under the exposure time.

[0104] The absolute phase map records the absolute phase of each pixel position; the modulation map records the modulation of each pixel position.

[0105] S103: Calculate the confidence level of the absolute phase of each pixel position at the exposure time based on the modulation degree and / or fringe sinusoidal error at the exposure time.

[0106] Among them, the fringe sinusoidal error of each pixel position at the exposure time represents: the degree to which the grayscale value of each pixel position in each second image at the exposure time conforms to the sine function; the confidence of the absolute phase of a pixel position at the exposure time is positively correlated with the modulation degree of the pixel position at the exposure time, and negatively correlated with the fringe sinusoidal error of the pixel position at the exposure time.

[0107] S104: Selecting the absolute phase with the highest confidence level from the absolute phases of each pixel position under each exposure duration to obtain a fused absolute phase map.

[0108] S105: Perform three-dimensional reconstruction based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured.

[0109] Based on the three-dimensional reconstruction method provided by the embodiment of the present application, the first image and the second image acquired at different exposure times can be used and processed separately to obtain the absolute phase map at each exposure time, as well as the confidence level corresponding to each pixel position in the absolute phase map at each exposure time. Accordingly, after obtaining the absolute phase map at each exposure time, the absolute phase map at each exposure time can be fused with one pixel position as the fusion granularity. The confidence level corresponding to each pixel position in the absolute phase map at each exposure time is calculated based on the modulation and / or fringe sinusoidal error of the pixel position at the exposure time. Moreover, the greater the modulation of a pixel position at an exposure time, the greater the contrast between the light and dark fringes at the pixel position in each second image at the exposure time. The smaller the fringe sinusoidal error of a pixel position at an exposure time, the more the grayscale value of the pixel position in each second image at the exposure time conforms to the variation pattern of the light and dark fringes in the projected fringe image, and the more it conforms to the sine function. Therefore, the modulation degree and fringe sinusoidal error of a pixel position at the exposure time can represent the accuracy of the absolute phase of the pixel position in the absolute phase map at the exposure time from different dimensions. That is to say, based on the three-dimensional reconstruction method provided in the embodiment of the present application, at least one item can be selected from the above-mentioned different dimensions to calculate the accuracy of the absolute phase of each pixel position at the exposure time. Furthermore, for each pixel position, the absolute phase with the highest corresponding confidence is selected from the absolute phases of the pixel position at each exposure time to obtain a fused absolute phase map. In this way, the accuracy of the fused absolute phase map is improved. Furthermore, three-dimensional reconstruction based on the fused absolute phase map can be applicable to high dynamic range scenes and improve the accuracy of the obtained three-dimensional reconstruction results.

[0110] It is understandable that for high-dynamic scenes, it is difficult to obtain a complete phase image by collecting images with a single exposure time. By fusing the phase images calculated from images collected with different exposure times, more complete and accurate phase information can be obtained.

[0111] Regarding step S101, the exposure duration represents the exposure time of the image acquisition device in the 3D reconstruction system when acquiring an image, and the exposure durations may have different values.

[0112] The total number of exposure durations and the value of each exposure duration are pre-set by a technician based on actual needs. Furthermore, the number of exposure durations is greater than or equal to 2. The images (first image and second image) captured by the image acquisition device are both grayscale images, meaning that the grayscale value of each pixel is recorded in the grayscale image.

[0113] In this application, the number of images that the image acquisition device needs to capture under different exposure times is fixed. The number of images that the image acquisition device needs to capture under one exposure time will be described in subsequent embodiments.

[0114] It is understandable that the greater the number of exposure times, the longer the total time it takes for the image acquisition device to capture images, and the greater the total number of images captured. Accordingly, in the process of performing three-dimensional reconstruction according to the three-dimensional reconstruction method provided by this application, the greater the amount of computing power required, the higher the accuracy of the three-dimensional reconstruction result. In other words, when the accuracy of the three-dimensional reconstruction results of the measured area is required to be higher, the technician can increase the number of exposure times set. For example, three different exposure times can be pre-set (e.g., 20 milliseconds, 40 milliseconds, and 60 milliseconds).

[0115] The fewer the number of exposure times, the shorter the total time it takes for the image acquisition device to capture images, and the fewer the total number of images captured. Accordingly, in the process of performing three-dimensional reconstruction according to the three-dimensional reconstruction method provided by this application, the less computing power is required, the more efficient the three-dimensional reconstruction results are. In other words, when the measurement cycle is fast, that is, when the efficiency of the three-dimensional reconstruction results of the measured area is high, technicians can reduce the number of exposure times set. For example, two different exposure times can be pre-set.

[0116] For ease of description, in the following embodiments, two exposure times will be used as an example. Of the two exposure times, the longer exposure time (referred to as exposure time 1) is 40 milliseconds, and the shorter exposure time (referred to as exposure time 2) is 20 milliseconds.

[0117] The "area to be measured" represents the spatial region requiring 3D reconstruction. For example, in a workpiece dimension measurement scenario, the "area to be measured" is the spatial region where the workpiece to be measured is placed. Accordingly, the image acquisition device can capture images of the "area to be measured." The location of the image acquisition device is not limited; for example, it can be mounted above the "area to be measured."

[0118] The three-dimensional reconstruction result of the area to be measured may be: a depth map of the area to be measured, or a three-dimensional point cloud of the area to be measured.

[0119] The set of binary-coded images to be projected onto the area to be measured includes multiple binary-coded images. The projection device projects one binary-coded image onto the area to be measured at a time. That is, during one projection process, the projection device can project one binary-coded image from the set of binary-coded images onto the area to be measured. Correspondingly, the image acquisition device can capture an image of the area to be measured when the binary-coded image is projected onto the area to be measured, thereby obtaining a first image.

[0120] For example, the binary coded image may be a Gray Code image or an XOR (Exclusive OR) code image.

[0121] Taking the above-mentioned exposure time 1 as an example, if there are 4 binary-coded images (which can be respectively recorded as binary-coded image 1, binary-coded image 2, binary-coded image 3, and binary-coded image 4), then after the projection device projects binary-coded image 1 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a first image (which can be recorded as first image 1); after the projection device projects binary-coded image 2 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a first image (which can be recorded as first image 2); after the projection device projects binary-coded image 3 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a first image (which can be recorded as first image 3); after the projection device projects binary-coded image 4 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a first image (which can be recorded as first image 4).

[0122] Similarly, for exposure time 2, after the projection device projects the binary coded image 1 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a first image (which can be recorded as first image 5); after the projection device projects the binary coded image 2 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a first image (which can be recorded as first image 6); after the projection device projects the binary coded image 3 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a first image (which can be recorded as first image 7); after the projection device projects the binary coded image 4 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a first image (which can be recorded as first image 8).

[0123] See also Figure 2(a) to Figure 2(d) , Figure 2(a) to Figure 2(d) Schematic diagram of a set of Gray code images that need to be projected.

[0124] Figure 2(a) is a schematic diagram of the first image in a set of Gray code images provided in an embodiment of the present application. Figure 2(b) is a schematic diagram of the second image in a set of Gray code images provided in an embodiment of the present application. Figure 2(c) is a schematic diagram of the third image in a set of Gray code images provided in an embodiment of the present application; and Figure 2(d) is a schematic diagram of the fourth image in a set of Gray code images provided in an embodiment of the present application. Figure 2(a) to Figure 2(d)In the image, the grayscale value of each pixel in the black area is 0; the grayscale value of each pixel in the white area is 255.

[0125] For a pixel position in the set of Gray code images, if the pixel position is located in a black area of ​​the Gray code image, the binary number of the pixel position at the corresponding bit sequence of the Gray code image is 0. If the pixel position is located in a white area of ​​the Gray code image, the binary number of the pixel position at the corresponding bit sequence of the Gray code image is 1. In other words, the binary number of each bit sequence of a pixel position in the set of Gray code images (i.e., the binary value of the pixel position) is fixed.

[0126] For example, for a region (such as region S1), if any pixel position in the region is located in the black region of the first image, the binary number of the first bit of the pixel position is 0; if the pixel position is located in the black region of the second image, the binary number of the second bit of the pixel position is 0; if the pixel position is located in the black region of the third image, the binary number of the third bit of the pixel position is 0; if the pixel position is located in the black region of the fourth image, the binary number of the fourth bit of the pixel position is 0. That is, the binary value of the pixel position is: 0000.

[0127] That is, the binary values ​​of all pixels in a region are consistent. Accordingly, the period of a binary coded image is the width of the region corresponding to the same binary value.

[0128] That is, for each exposure duration, the processor can obtain multiple first images for that exposure duration. The number of first images is consistent with the number of binary-coded images; each first image for that exposure duration corresponds to a binary-coded image.

[0129] It is understood that when an image acquisition device captures images, all internal parameters of the image acquisition device, except for the exposure duration, are fixed. For example, the internal parameters of the image acquisition device include focal length, principal image point coordinates, distortion parameters, etc. Accordingly, the size of each first image at a given exposure duration is consistent, and the size of each first image at different exposure durations is also consistent.

[0130] It is understandable that during the three-dimensional reconstruction process based on the images captured by the image acquisition device, the image size is not changed. In other words, the sizes of the first images, second images, relative phase maps, modulation maps, and absolute phase maps involved in steps S101 to S106 are all consistent. For example, if the size of the images captured by the image acquisition device (i.e., the first images and the second images) is 256×256, then the size of the relative phase map, modulation map, and absolute phase map obtained will also be 256×256.

[0131] The projected fringe images (which may be referred to as a group of fringe images) are a plurality of line-shifted fringe images or a plurality of phase-shifted fringe images. A group of fringe images may contain 3 or more fringe images.

[0132] Among them, multiple line-shifting fringe images (which can be called a group of line-shifting fringe images) represent: a group of images formed when a fringe image moves in a certain direction in space (such as any direction in the plane perpendicular to the lens direction of the image acquisition device in the measured space) according to a certain displacement interval.

[0133] A plurality of phase-shifted fringe images (which may be referred to as a group of line-shifted fringe images) indicates that the phase difference between every two adjacent images (such as sinusoidal fringe images) in the group of line-shifted fringe images is consistent.

[0134] In the embodiment of the present application, the image of the area to be measured that is projected with the line-shifted fringe image and captured by the image capture device can be approximately used as the image of the area to be measured that is projected with the phase-shifted fringe image.

[0135] The periodic size of the projected stripe image (i.e., the width of a dark stripe and a bright stripe in the stripe image) is consistent with the periodic size of the binary coded image (i.e., the width corresponding to a decoded value in the binary coded image).

[0136] The projection device projects one fringe image onto the test area at a time. That is, during one projection process, the projection device can project one fringe image from a set of fringe images onto the test area. Correspondingly, the image capture device can capture an image of the test area when the fringe image is projected onto the test area, obtaining a second image.

[0137] Taking the above-mentioned exposure time 1 as an example, if there are 4 line-shift fringe images in a group (which can be respectively recorded as fringe image 1, fringe image 2, fringe image 3 and fringe image 4), then after the projection device projects fringe image 1 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a second image (which can be recorded as second image 1); after the projection device projects fringe image 2 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a second image (which can be recorded as second image 2); after the projection device projects fringe image 3 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a second image (which can be recorded as second image 3); after the projection device projects fringe image 4 onto the area to be measured, the image acquisition device can capture the image of the area to be measured at an exposure time of 1 to obtain a second image (which can be recorded as second image 4).

[0138] Similarly, for exposure time 2, after the projection device projects the stripe image 1 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a second image (which can be recorded as second image 5); after the projection device projects the stripe image 2 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a second image (which can be recorded as second image 6); after the projection device projects the stripe image 3 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a second image (which can be recorded as second image 7); after the projection device projects the stripe image 4 onto the area to be measured, the image acquisition device can acquire the image of the area to be measured at exposure time 2 to obtain a second image (which can be recorded as second image 8).

[0139] With respect to step S102 , for each exposure duration, the processor may calculate the absolute phase map and the modulation map for the exposure duration using each first image and each second image for the exposure duration.

[0140] In some embodiments, step S102 includes:

[0141] Step S1021 : For each pixel position, based on the grayscale values ​​of the pixel position in each first image at the exposure time, calculate the decoded value of the pixel position at the exposure time.

[0142] Step S1022: using each second image under the exposure time, calculate and obtain a relative phase map and a modulation map under the exposure time.

[0143] The projected fringe images are multiple line-shifted fringe images or multiple phase-shifted fringe images; and the relative phase image records the relative phase of each pixel position.

[0144] Step S1023: performing phase unwrapping on the relative phase map for the exposure duration using the decoded values ​​of each pixel position for the exposure duration to obtain the absolute phase map for the exposure duration.

[0145] The absolute phase map records the absolute phase of each pixel position.

[0146] In this embodiment of the present application, for each pixel position, the processor may calculate a decoded value for that pixel position at that exposure duration based on the grayscale value of that pixel position in each first image at that exposure duration. The decoded value for a pixel position at a given exposure duration is an integer.

[0147] In some embodiments, for a pixel position, the processor may determine whether the grayscale value of the pixel position in each first image is greater than a preset grayscale value threshold according to the bit sequence of the binary-coded image corresponding to each first image in the plurality of binary-coded images. If the grayscale value of the pixel position in the first image is greater than the preset grayscale value threshold, the binary number of the pixel position at the bit sequence corresponding to the first image is determined to be 1. If the grayscale value of the pixel position in the first image is not greater than the preset grayscale value threshold, the binary number of the pixel position at the bit sequence corresponding to the first image is determined to be 0.

[0148] Furthermore, after obtaining the binary number of each bit sequence of the pixel position (i.e., the binary value of the pixel position), the processor converts the binary value of the pixel position to obtain the decimal value of the pixel position as the decoded value of the pixel position under the exposure time (also called the stripe secondary).

[0149] Taking exposure duration 1 as an example, the first images for this exposure duration include: first image 1, first image 2, first image 3, and first image 4. If the preset grayscale value threshold is 127, for a pixel position (for example, pixel position (100, 100)), the grayscale value of this pixel position in first image 1 is 220, then the binary number at the corresponding bit (first bit) of this pixel position (100, 100) in this first image is 1. If the grayscale value of this pixel position in first image 2 is 210, then the binary number at the corresponding bit (second bit) of this pixel position (100, 100) in this first image is 1. If the grayscale value of this pixel position in first image 3 is 10, then the binary number at the corresponding bit (third bit) of this pixel position (100, 100) in this first image is 0. If the grayscale value of this pixel position in first image 4 is 60, then the binary number at the corresponding bit (fourth bit) of this pixel position (100, 100) in this first image is 0. That is, the binary number of each bit sequence of the pixel position (i.e., the binary value of the pixel position) is: 1100. Accordingly, the processor converts the binary value of the pixel position to obtain the decimal value of the pixel position (i.e., 12), which is used as the decoded value of the pixel position at the exposure time.

[0150] Accordingly, the processor can obtain the decoded value of each pixel position under the exposure duration 1.

[0151] The grayscale value threshold may be preset by a technician, for example, 127.

[0152] In one implementation, the projection device can pre-project a completely black image onto the area to be measured, and the image acquisition device can obtain an image of the area to be measured when the completely black image is projected onto the area to be measured (i.e., the grayscale value of each pixel position in the image is 0), and calculate the mean of the grayscale values ​​of each pixel position in the acquired image (referred to as a first mean). Alternatively, the projection device can pre-project a completely white image onto the area to be measured, and the image acquisition device can obtain an image of the area to be measured when the completely white image is projected onto the area to be measured (i.e., the grayscale value of each pixel position in the image is 255), and calculate the mean of the grayscale values ​​of each pixel position in the acquired image (referred to as a second mean). Furthermore, the processor can calculate the mean of the first mean and the second mean as the grayscale value threshold.

[0153] Similarly, for exposure duration 2, the processor can obtain the decoded value of each pixel position under the exposure duration 2.

[0154] It can be understood that for projecting a binary coded image, due to the influence of the exposure time of the image acquisition device, the grayscale value in the first image captured at the same pixel position under different exposure times may be different. Correspondingly, the decoding value of the same pixel position under different exposure times may also be different.

[0155] For step S1022, for each exposure time, the processor can use each second image under the exposure time, combined with the functional relationship between grayscale value, relative phase and modulation, to calculate the relative phase map and modulation map under the exposure time.

[0156] In one implementation, the functional relationship between the grayscale value, relative phase, and modulation index is expressed as a first formula:

[0157]

[0158] I n (x, y) represents the grayscale value of the pixel position (x, y) in the second image corresponding to the nth fringe image at the exposure time, according to the relative order between the fringe images; A(x, y) represents the background intensity of the pixel position (x, y) at the exposure time; B(x, y) represents the modulation degree of the pixel position (x, y) at the exposure time; φ(x, y) represents the relative phase of the pixel position (x, y) at the exposure time; N represents the total number of second images collected at the exposure time; n represents the order of the fringe image corresponding to a second image in the fringe images; cos represents the cosine function.

[0159] In some embodiments, step S1022 includes:

[0160] Step S1022a: For each pixel position, calculate the inner product of the grayscale value of the pixel position in each second image under the exposure time and the sine value of the phase shift of the corresponding fringe image as the first inner product of the pixel position.

[0161] Step S1022b: Calculate the inner product of the grayscale value of the pixel position in each second image under the exposure time and the cosine value of the phase shift of the corresponding fringe image as the second inner product of the pixel position.

[0162] Step S1022c: Calculate the ratio of the first inner product to the second inner product of the pixel position under the exposure time, and use the arc tangent value of the ratio as the relative phase of the pixel position under the exposure time, to obtain the relative phase diagram under the exposure time; calculate the sum of the squares of the first inner product and the second inner product of the pixel position under the exposure time, and use the normalized value of the square root of the calculated sum of squares as the modulation degree of the pixel position under the exposure time, to obtain the modulation degree diagram under the exposure time.

[0163] In the embodiment of the present application, for each pixel position, the first inner product of the pixel position can be expressed as:

[0164]

[0165] The second inner product at this pixel position can be expressed as:

[0166]

[0167] I n (x, y) represents the grayscale value of the pixel position (x, y) in the second image corresponding to the nth fringe image at the exposure time, according to the relative position order between the fringe images. N represents the total number of second images collected at the exposure time. n represents the position order of the fringe image corresponding to a second image in the fringe images. Indicates the phase shift of the nth fringe image.

[0168] Accordingly, the process of obtaining the relative phase map under the exposure time can be expressed as the second formula. The second formula is:

[0169]

[0170] ∑ is the summation function; sin represents the sine function; tan -1 represents the inverse tangent function; φ(x,y) represents the relative phase of the pixel position (x,y) under the exposure time.

[0171] Accordingly, the process of obtaining the modulation map under the exposure time can be expressed as the third formula. The third formula is:

[0172]

[0173] B(x,y) represents the modulation degree of the pixel position (x,y) under the exposure time.

[0174] In the embodiment of the present application, for an exposure time, a second image under the exposure time records the grayscale value of each pixel position. That is, in the above first formula, the left side of the equation (i.e., I n The value of (x,y)) is known.

[0175] Taking the exposure time 1 as an example, based on the functional relationship between grayscale value, relative phase, and modulation, the second images (i.e., second image 1, second image 2, second image 3, and second image 4) under exposure time 1 can be expressed as the following set of equations:

[0176]

[0177] Among them, I1(x,y) is the grayscale value at the pixel position (x,y) in the second image 1; I2(x,y) is the grayscale value at the pixel position (x,y) in the second image 2; I3(x,y) is the grayscale value at the pixel position (x,y) in the second image 3; I4(x,y) is the grayscale value at the pixel position (x,y) in the second image 4.

[0178] Accordingly, based on the first formula, a second formula for calculating the relative phase of each pixel position at the exposure duration can be derived. Furthermore, the processor can use the second formula to calculate the relative phase of each pixel position at the exposure duration (i.e., obtain a relative phase map for the exposure duration).

[0179] The relative phase can also be called the wrapping phase. Furthermore, the relative phase value of a pixel at a given exposure time lies within the interval (0, 1).

[0180] Based on the first formula, a third formula can be derived for calculating the modulation degree at each pixel position for the exposure duration. Furthermore, the processor can use the third formula to calculate the modulation degree at each pixel position for the exposure duration (i.e., obtain a modulation degree map for the exposure duration).

[0181] Based on the above processing, the processor can calculate the relative phase map and modulation map under an exposure time according to the functional relationship between grayscale value, relative phase and modulation, combined with each second image under an exposure time.

[0182] That is, for a given exposure duration, the processor can calculate a relative phase map and a modulation map for that exposure duration based on each second image obtained during that exposure duration. The relative phase map records the relative phase at each pixel position, and the modulation map records the modulation at each pixel position.

[0183] Taking the exposure time 1 as an example, after obtaining the second images (i.e., second image 1, second image 2, second image 3, and second image 4) under the exposure time 1, the processor can calculate the relative phase map (which can be called relative phase) under the exposure time 1 by combining the functional relationship between the grayscale value, relative phase and modulation. Figure 1 ), and the modulation diagram under the exposure time 1 (which can be called the modulation Figure 1 ).

[0184] Taking the above-mentioned exposure time 2 as an example, for exposure time 2, after obtaining the second images under exposure time 2 (i.e., second image 5, second image 6, second image 7, and second image 8), the processor can combine the functional relationship between grayscale value, relative phase and modulation to calculate the relative phase map under exposure time 2 (which can be called relative phase map 2) and the modulation map under exposure time 2 (which can be called modulation map 2).

[0185] For step S1023, for each pixel position, the processor can use the decoded value of the pixel position at the exposure duration to perform phase expansion on the relative phase of the pixel position at the exposure duration to obtain the absolute phase of the pixel position at the exposure duration, that is, to obtain the absolute phase map at the exposure duration.

[0186] In one implementation, step S1023 includes:

[0187] For each pixel position, the relative phase of the pixel position at the exposure time is calculated, and the sum of the phase change corresponding to the decoded value of the pixel position at the exposure time is used as the absolute phase of the pixel position at the exposure time to obtain the absolute phase map at the exposure time.

[0188] In the embodiment of the present application, the phase change corresponding to the decoded value of a pixel position at the exposure duration can be expressed as: 2πk(x,y), where k(x,y) represents the decoded value of the pixel position (x,y) at the exposure duration.

[0189] Accordingly, the process of obtaining the absolute phase map under the exposure time can be expressed as the fourth formula. The fourth formula is:

[0190] Φ(x,y)=2πk(x,y)+φ(x,y);

[0191] Φ(x,y) represents the absolute phase of the pixel position (x,y) under the exposure duration, k(x,y) represents the decoded value of the pixel position (x,y) under the exposure duration; φ(x,y) represents the relative phase of the pixel position (x,y) under the exposure duration.

[0192] That is, the fourth formula is used to solve the absolute phase of each pixel position under the exposure time.

[0193] For each pixel position, the process of obtaining the decoded value (also called fringe secondary) of the pixel position under the exposure duration is as described in the above embodiment and will not be described in detail here.

[0194] Correspondingly, the absolute phase of a pixel position at the exposure duration is: the sum of the relative phase of the pixel position at the exposure duration and the product of the decoded value of the pixel position at the exposure duration and 2π.

[0195] In this way, for each pixel position, the processor can calculate the absolute phase of the pixel position under the exposure duration according to the fourth formula, and thus can obtain the absolute phase map under the exposure duration.

[0196] Taking the above exposure time 1 as an example, the relative phase under the exposure time 1 is obtained. Figure 1 , and the decoded value of each pixel position under the exposure time 1, the processor can calculate the absolute phase map under the exposure time 1 (which can be called absolute phase Figure 1 ).

[0197] Similarly, taking the above-mentioned exposure time 2 as an example, after obtaining the relative phase map 2 under the exposure time 2 and the decoded value of each pixel position under the exposure time 2, the processor can calculate the absolute phase map under the exposure time 2 (which can be called absolute phase map 2).

[0198] In step S103, the processor may calculate the confidence level of the absolute phase at each pixel position at an exposure duration using any of the following three methods. It will be appreciated that, to facilitate subsequent comparison of the confidence levels, the method for calculating the confidence level of the absolute phase at each pixel position at each exposure duration remains consistent.

[0199] In the process of calculating the confidence of the absolute phase of each pixel position under the exposure duration, if the processor uses the modulation degree of each pixel position under the exposure duration for calculation, then the confidence of the absolute phase of a pixel position under the exposure duration is positively correlated with the modulation degree of the pixel position under the exposure duration.

[0200] If the processor uses the fringe sinusoidal error of each pixel position under the exposure time for calculation, then the confidence of the absolute phase of a pixel position under the exposure time is negatively correlated with the fringe sinusoidal error of the pixel position under the exposure time.

[0201] Method 1: The processor may calculate the confidence level of the absolute phase of each pixel position at the exposure duration based solely on the modulation degree of the pixel position at the exposure duration.

[0202] For example, the processor may directly determine the modulation degree of each pixel position at the exposure duration as the confidence level of the absolute phase of the pixel position at the exposure duration.

[0203] For a pixel at a given exposure time, a greater degree of modulation at that pixel at that exposure time indicates a more pronounced contrast between the projected areas of the bright and dark fringes in each captured second image. Accordingly, the accuracy of the absolute phase at that pixel at that exposure time, determined based on each second image, is higher, and the confidence level of the absolute phase at that pixel at that exposure time is higher.

[0204] That is, after obtaining the modulation degree of each pixel position at the exposure duration, the processor can directly determine the modulation degree of each pixel position at the exposure duration as the confidence level of the absolute phase of the pixel position at the exposure duration.

[0205] Method 2: The processor may calculate the confidence level of the absolute phase of each pixel position at the exposure duration based only on the fringe sinusoidal error at the exposure duration.

[0206] Under an exposure time, for a pixel position, the fringe sinusoidal error of a pixel position under this exposure time represents: the difference between the actual grayscale value of the pixel position recorded in each second image under this exposure time and the theoretical grayscale value of the pixel position for each second image under this exposure time.

[0207] For example, the processor may determine the inverse of the fringe sinusoidal error at each pixel position at the exposure duration as the confidence level of the absolute phase at that pixel position at the exposure duration. Alternatively, the fringe sinusoidal error at each pixel position at the exposure duration may be normalized to a range of 0-1, and the difference between 1 and each normalized fringe sinusoidal error may be calculated as the confidence level of the absolute phase at each pixel position at the exposure duration.

[0208] In some embodiments, the fringe sinusoidal error at each pixel position under the exposure duration is calculated by the following steps:

[0209] Step 1: Calculate and obtain a background intensity map under the exposure time using each second image under the exposure time.

[0210] The background intensity map records the background intensity of each pixel position.

[0211] Step 2: For each second image under the exposure time, calculate the relative phase of each pixel position under the exposure time, and the cosine value of the difference between the phase shift amount of the fringe image corresponding to the second image.

[0212] Step 3: Multiply the calculated cosine value by the modulation depth of the pixel position at the exposure time.

[0213] Step 4: Calculate the sum of the product obtained and the background intensity at the pixel position under the exposure time to obtain a theoretical grayscale value of the pixel position for the second image under the exposure time.

[0214] Step 5: Calculate the difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time to obtain the fringe sinusoidal error of the pixel position at the exposure time.

[0215] In an embodiment of the present application, under an exposure time, the processor can use each second image under the exposure time in combination with the above functional relationship to calculate a background intensity map under the exposure time, wherein the background intensity map records the background intensity at each pixel position.

[0216] That is, based on the above functional relationship, the background intensity at each pixel position under the exposure time can be expressed as:

[0217]

[0218] Where A(x,y) represents the background intensity at the pixel position (x,y) under the exposure time, B(x,y) represents the modulation degree at the pixel position (x,y) under the exposure time; φ(x,y) represents the relative phase at the pixel position (x,y) under the exposure time; I n (x, y) represents the grayscale value of the pixel position (x, y) in the n-th second image; n represents the position order of the fringe image corresponding to a second image in each fringe image; N represents the total number of second images collected at the exposure time; cos represents the cosine function; ∑ is the summation factor.

[0219] Taking the above exposure time 1 as an example, the processor obtains the second images (i.e., second image 1, second image 2, second image 3, and second image 4) under the exposure time, the relative phase of the second image under the exposure time 1, and the relative phase of the second image under the exposure time 1. Figure 1, and the modulation degree under the exposure duration 1 Figure 1 Afterwards, the background intensity map (also called background intensity) under the exposure time can be calculated based on the above functional relationship. Figure 1 ).

[0220] Accordingly, for each second image under the exposure time, the process of obtaining the theoretical grayscale value of each pixel position for the second image under the exposure time can be expressed as:

[0221]

[0222] Where φ(x,y) represents the relative phase of the pixel position (x,y) at the exposure time. represents: the relative phase of the pixel position (x, y) under the exposure time, and the cosine value of the difference in the phase shift amount of the fringe image corresponding to the second image; A(x, y) represents the background intensity of the pixel position (x, y) under the exposure time; B(x, y) represents the modulation degree of the pixel position (x, y) under the exposure time; I ′ n (x, y) represents the theoretical grayscale value of the pixel position (x, y) for the second image under the exposure time.

[0223] That is, for each pixel position of the second image, the processor can substitute the position sequence of the fringe image corresponding to the second image, the relative phase of the pixel position at the exposure time, the modulation index, and the background intensity into the first formula to obtain the theoretical grayscale value of the pixel position for the second image at the exposure time, which can be expressed as: ′ n (x, y), n represents the position sequence of the fringe image corresponding to the second image.

[0224] Taking the aforementioned exposure duration 1 as an example, for each pixel position of second image 1, the processor can calculate the theoretical grayscale value of each pixel position for second image 1 at the exposure duration 1. For each pixel position of second image 2, the processor can calculate the theoretical grayscale value of each pixel position for second image 2 at the exposure duration 1. For each pixel position of second image 3, the processor can calculate the theoretical grayscale value of each pixel position for second image 3 at the exposure duration 1. For each pixel position of second image 4, the processor can calculate the theoretical grayscale value of each pixel position for second image 4 at the exposure duration 1.

[0225] In this way, the processor can calculate the theoretical grayscale value of each pixel position for each second image under the exposure time.

[0226] Accordingly, the fringe sinusoidal error at a pixel position at a specific exposure time represents the difference between the actual grayscale value at that pixel position recorded in each second image at that exposure time and the theoretical grayscale value for that pixel position at that exposure time. In other words, the smaller the fringe sinusoidal error at that pixel position at that exposure time, the more closely the grayscale value at that pixel position in each acquired second image conforms to the sine function corresponding to the light and dark fringes in the projected fringe image. Consequently, the confidence level in the absolute phase of that pixel position at that exposure time is higher.

[0227] In one implementation, step 5 includes:

[0228] For each pixel position, calculating the absolute value of the difference between a theoretical grayscale value of the pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in the second image as the error component of the pixel position for the second image;

[0229] The average of the error components of the pixel position for each second image is calculated as the fringe sinusoidal error of the pixel position under the exposure time.

[0230] In the embodiment of the present application, the process of obtaining the fringe sinusoidal error of each pixel position under the exposure time can be expressed as the fifth formula. Wherein, the fifth formula is:

[0231]

[0232] E(x,y) represents the fringe sinusoidal error at the pixel position (x,y) under the exposure time; I n (x, y) represents the true grayscale value of the pixel position (x, y) recorded in the second image corresponding to the nth fringe image according to the relative position order between the fringe images at the exposure time; n ′(x, y) represents the theoretical grayscale value of the pixel position (x, y) in the second image corresponding to the nth fringe image at the exposure time according to the relative position order between the fringe images; N represents the total number of second images collected at the exposure time; n represents the position order of the fringe image corresponding to a second image in the fringe images; |I ′ n (x,y)-I n (x, y)| represents the error component of the pixel position (x, y) for the n-th second image.

[0233] In an embodiment of the present application, at an exposure time, the processor can calculate the fringe sinusoidal error of the pixel position at the exposure time according to the fifth formula, based on the theoretical grayscale value of each pixel position for each second image at the exposure time, and the actual grayscale value of the pixel position recorded in each second image at the exposure time.

[0234] Taking the above-mentioned exposure time 1 as an example, after the processor obtains the second images (i.e., second image 1, second image 2, second image 3, and second image 4) under the exposure time, the theoretical grayscale value of each pixel position for second image 1 under the exposure time 1, the theoretical grayscale value of each pixel position for second image 2 under the exposure time 1, the theoretical grayscale value of each pixel position for second image 3 under the exposure time 1, and the theoretical grayscale value of each pixel position for second image 4 under the exposure time 1, for each pixel position, the fringe sinusoidal error of the pixel position under the exposure time 1 can be calculated according to the above-mentioned fifth formula.

[0235] In this way, the processor can calculate the fringe sinusoidal error of each pixel position under the exposure time.

[0236] Method three: The processor may calculate the confidence level of the absolute phase of each pixel position at the exposure time based on the modulation degree and fringe sinusoidal error at the exposure time.

[0237] The confidence of the absolute phase of a pixel position at the exposure time is positively correlated with the modulation degree of the pixel position at the exposure time, and negatively correlated with the fringe sinusoidal error of the pixel position at the exposure time.

[0238] In some embodiments, step S103 includes:

[0239] Step S1031 : normalizing the fringe sinusoidal error of each pixel position at the exposure time to a range of 0 to 1, thereby obtaining a normalized value of the fringe sinusoidal error of each pixel position at the exposure time.

[0240] Step S1032: Calculate the weighted sum of the modulation index and the normalized value of the fringe sinusoidal error at each pixel position at the exposure time as the confidence level of the absolute phase of the pixel position at the exposure time.

[0241] In the embodiment of the present application, for each pixel position, the process of obtaining the confidence of the absolute phase of the pixel position under the exposure time can be expressed as the sixth formula. The sixth formula is:

[0242] cfd(x,y)=ωB(x,y)+(1-ω)(1-E′(x,y));

[0243] cfd(x,y) represents the confidence of the absolute phase of the pixel position (x,y) under the exposure time; E′(x,y) represents the normalized value of the fringe sinusoidal error at the pixel position (x,y) under the exposure time; B(x,y) represents the modulation degree of the pixel position (x,y) under the exposure time; ω represents the weight.

[0244] Under an exposure time, for each pixel position, the process of obtaining the modulation degree of the pixel position under the exposure time, and the process of obtaining the normalized value of the fringe sinusoidal error of the pixel position under the exposure time can refer to the contents in the above embodiments and will not be repeated here.

[0245] Accordingly, for each pixel position, the processor may substitute the modulation degree of the pixel position at the exposure time and the normalized value of the fringe sinusoidal error at the exposure time into the sixth formula to calculate the confidence level of the absolute phase at the pixel position at the exposure time. The value of ω may range from (0.2 to 0.3).

[0246] Thus, the modulation degree and fringe sinusoidal error at a pixel position at a given exposure duration can represent the quality of the absolute phase at that pixel position in the absolute phase image at that exposure duration from different dimensions. The processor can combine the modulation degree and fringe sinusoidal error at a pixel position at that exposure duration to calculate the confidence level of the absolute phase at that pixel position at that exposure duration.

[0247] It can be understood that, when the number of projected fringe images is three, based on the functional relationship between grayscale value, relative phase, and modulation, for each pixel position, in the system of simultaneous equations based on the second images, the number of equations corresponding to that pixel position at that exposure time (i.e., three) is consistent with the number of unknown quantities corresponding to that pixel position at that exposure time (i.e., three). Accordingly, for a pixel position, the processor can calculate the background intensity at that pixel position at that exposure time, the modulation at that pixel position at that exposure time, and the relative phase at that pixel position at that exposure time based on the functional relationship between grayscale value, relative phase, and modulation.

[0248] At this point, for each pixel position, the background intensity at that pixel position at that exposure time, the modulation degree at that pixel position at that exposure time, and the relative phase at that pixel position at that exposure time, along with the grayscale value (i.e., the true grayscale value) at that pixel position in each second image, conform to the aforementioned functional relationship. Therefore, for each pixel position, the fringe sinusoidal error at that exposure time is zero. Accordingly, the processor can calculate the confidence level of the absolute phase at each pixel position at that exposure time using the aforementioned method 1.

[0249] When the number of projected fringe images is greater than three, based on the functional relationship between grayscale value, relative phase, and modulation, for each pixel position, in the system of simultaneous equations based on the second images, the number of equations corresponding to that pixel position at that exposure time (i.e., the number of projected fringe images) is greater than the number of unknown quantities corresponding to that pixel position at that exposure time (i.e., three). The unknown quantities corresponding to a pixel position at that exposure time include: the background intensity at that pixel position at that exposure time, the modulation at that pixel position at that exposure time, and the relative phase at that pixel position at that exposure time.

[0250] Furthermore, when the processor substitutes the calculated background intensity at the pixel location for the exposure duration, the modulation at the pixel location for the exposure duration, and the relative phase at the pixel location for the exposure duration into the aforementioned functional relationship, the calculated grayscale value (i.e., the theoretical grayscale value) may not be consistent with the grayscale value at the pixel location in each second image (i.e., the actual grayscale value). That is, for each pixel location, the background intensity at the pixel location for the exposure duration, the modulation at the pixel location for the exposure duration, and the relative phase at the pixel location for the exposure duration may not conform to the aforementioned functional relationship with the grayscale value at the pixel location in each second image (i.e., the actual grayscale value). In other words, the calculated fringe sinusoidal error at each pixel location for the exposure duration is typically not zero.

[0251] That is, when the number of projected fringe images is greater than three, the fringe sinusoidal error at each pixel location for that exposure duration can reflect the difference between the theoretical grayscale value at that exposure duration and the actual grayscale value at that exposure duration. In this case, the processor can calculate the confidence level of the absolute phase at each pixel location for that exposure duration based on the fringe sinusoidal error at that exposure duration. That is, the processor can calculate the confidence level of the absolute phase at each pixel location for that exposure duration using either of the aforementioned methods 2 and 3. Alternatively, the processor can also calculate the confidence level of the absolute phase at each pixel location for that exposure duration using the aforementioned method 1.

[0252] With respect to step S104 , for each pixel position, the processor may select the absolute phase with the highest confidence from the absolute phases of the pixel position at each exposure time to obtain a fused absolute phase map.

[0253] Taking the above exposure duration 1 and exposure duration 2 as an example, for a pixel position (for example, pixel position (x1, y1)), if the absolute phase Figure 1The absolute phase of the pixel position (x1, y1) recorded in absolute phase image 1 is 12.16, the absolute phase of the pixel position (x1, y1) recorded in absolute phase image 2 is 15.11, and the confidence level of the absolute phase of the pixel position (x1, y1) at exposure time 1 is less than the confidence level of the absolute phase of the pixel position (x1, y1) at exposure time 2. The processor can determine 15.11 as the absolute phase of the pixel position (x1, y1) in the fused absolute phase image.

[0254] With respect to step S105 , after obtaining the fused absolute phase image, the processor may perform three-dimensional reconstruction based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured.

[0255] In some embodiments, each first image and each second image at each exposure time is acquired by using an image acquisition device. Accordingly, step S105 includes:

[0256] When the image acquisition device is a binocular camera, based on a preset binocular 3D reconstruction algorithm, the fused absolute phase maps obtained based on the images acquired by the two cameras are 3D reconstructed to obtain a 3D reconstruction result of the area to be measured.

[0257] In an embodiment of the present application, if the image acquisition device is a binocular camera, the processor can perform distortion correction and epipolar line correction on the fused absolute phase map of the left camera and the fused absolute phase map of the right camera in the binocular camera according to the pre-calibrated internal and external parameters of the camera, to achieve phase point matching, that is, determine the corresponding points on the fused absolute phase map of the left camera and the fused absolute phase map of the right camera and the left camera, and reconstruct the three-dimensional point cloud according to the triangulation principle.

[0258] Alternatively, the processor can perform distortion correction and epipolar correction on the fused absolute phase map of the left camera and the fused absolute phase map of the right camera in the binocular camera based on the pre-calibrated camera internal and external parameters to achieve phase point matching, that is, determine the corresponding points on the fused absolute phase map of the left camera and the fused absolute phase map of the right camera and the left camera. And calculate the disparity map based on the matching phase points. The value of each pixel position in the disparity map represents the size of the disparity. Accordingly, the processor can use the disparity map and the camera internal and external parameters to calculate the depth map. The value of each pixel position in the depth map represents: the distance from the point in space corresponding to the pixel position to the camera.

[0259] Based on the above processing, the 3D reconstruction result can be a depth map or a 3D point cloud. The processor can perform 3D reconstruction based on the fused absolute phase images corresponding to the left and right cameras in the binocular camera to obtain the 3D reconstruction result of the test area.

[0260] In some embodiments, step S105 includes:

[0261] When the image acquisition device is a monocular camera, the fused absolute phase image is three-dimensionally reconstructed based on a preset monocular three-dimensional reconstruction algorithm to obtain a three-dimensional reconstruction result of the area to be measured.

[0262] In an embodiment of the present application, if the image acquisition device is a monocular camera, the processor can perform three-dimensional reconstruction on the obtained fused absolute phase image based on a preset monocular three-dimensional reconstruction algorithm to obtain a three-dimensional reconstruction result (depth map and / or three-dimensional point cloud) of the area to be measured.

[0263] For example, the processor can input the fused absolute phase image into a pre-calibrated phase-height conversion model. Accordingly, the phase-height conversion model can input the 3D reconstruction result of the fused absolute phase image. The processor can pre-calibrate the phase-height model. That is, the processor acquires images at different preset distances (i.e., heights), calculates the absolute phase values, and fits the phase-height function relationship.

[0264] Based on the above processing, the 3D reconstruction result can be a depth map or a 3D point cloud. The processor can perform 3D reconstruction based on the absolute phase map fused by the monocular camera to obtain a 3D reconstruction result of the area to be measured.

[0265] See also Figure 3 , Figure 3 A schematic diagram of a process for performing three-dimensional reconstruction provided in an embodiment of the present application.

[0266] The process of projecting and acquiring the encoded image, decoding the encoded image phase, and calculating the phase position reliability in the first set of exposures represents the process of obtaining the absolute phase map for the exposure duration corresponding to the first set of exposures. The process of projecting and acquiring the encoded image, decoding the encoded image phase, and calculating the phase position reliability in the second set of exposures represents the process of obtaining the absolute phase map for the exposure duration corresponding to the second set of exposures.

[0267] For each exposure duration, the processor may utilize an image acquisition device to capture images of the test area when each binary coded image is projected onto the test area during the exposure duration, thereby obtaining multiple first images. For each pixel position, based on the grayscale value of each first image at the pixel position during the exposure duration, a decoded value of the pixel position at the exposure duration is calculated. Furthermore, the processor utilizes the image acquisition device to capture images of the test area when each fringe image is projected onto the test area during the exposure duration, thereby obtaining multiple second images. Furthermore, utilizing each second image during the exposure duration, the processor may utilize an integrated functional relationship between grayscale value, relative phase, and modulation to calculate a relative phase map and a modulation map at the exposure duration. Furthermore, utilizing the decoded value of each pixel position during the exposure duration, the processor may utilize an image acquisition device to capture images of the test area when each fringe image is projected onto the test area during the exposure duration, thereby obtaining multiple first images ... second images. Furthermore, utilizing the decoded value of each pixel position during the exposure duration, the processor may utilize an image acquisition device to capture images of the test area when each fringe image is projected onto the test area, thereby obtaining multiple first images. Furthermore, utilizing an image acquisition device to capture images of the test area when each fringe image is projected onto the test area, the processor may utilize an image acquisition device to capture images of the test area when each fringe image is projected onto the test area, thereby obtaining multiple second images. Furthermore, utilizing the decoded value of each pixel position during the exposure duration, the processor may utilize an image acquisition device to capture images of the test area when each fringe image is projected onto the test area, thereby obtaining multiple second images. Furthermore, utilizing

[0268] After obtaining the absolute phase map for the exposure duration corresponding to the first set of exposures and the absolute phase map for the exposure duration corresponding to the second set of exposures, the processor can perform multi-exposure phase map fusion. That is, for each pixel position, the absolute phase with the highest confidence level is selected from the absolute phases at that pixel position for each exposure duration to obtain a fused absolute phase map (i.e., a fused phase map). Accordingly, the processor can perform 3D reconstruction based on the fused absolute phase maps to obtain a 3D reconstruction result of the area to be measured.

[0269] Refer to Figure 4(a), which is an example diagram of three-dimensional reconstruction under the first exposure time provided in an embodiment of the present application. In Figure 4(a), the leftmost image is a second image captured by the acquisition device under the first exposure time. In Figure 4(a), the middle image is the absolute phase map under the first exposure time. In Figure 4(a), the rightmost image is a three-dimensional point cloud obtained by three-dimensional reconstruction based on the absolute phase map under the first exposure time. It can be seen from the leftmost image in Figure 4(a) that there is a local area that is too dark in the image captured by the image acquisition device under the first exposure. Such as the area shown by the rectangular box 401 in Figure 4(a). For the area to be measured represented by Figure 4(a), due to the low reflectivity and dark color of the workpiece in the box on the right, the processor cannot perform three-dimensional reconstruction on this part of the area.

[0270] Refer to Figure 4(b), which is an example diagram of three-dimensional reconstruction under the second exposure time provided in an embodiment of the present application. Moreover, the second exposure time is greater than the first exposure time. In Figure 4(b), the leftmost image is a second image captured by the acquisition device under the second exposure time. In Figure 4(b), the middle image is the absolute phase map under the second exposure time. In Figure 4(b), the rightmost image is a three-dimensional point cloud obtained by three-dimensional reconstruction based on the absolute phase map under the second exposure time. It can be seen from the leftmost image in Figure 4(b) that there is a locally overexposed area in the image captured by the image acquisition device under the second exposure. Such as the area shown by the rectangular box 402 in Figure 4(b). For the area to be measured represented by Figure 4(a), due to the high reflectivity and lighter color of the workpiece in the left box, the processor cannot perform three-dimensional reconstruction on this part of the area.

[0271] Correspondingly, see Figure 4(c), which is a schematic diagram of three-dimensional reconstruction based on the fused absolute phase map provided in an embodiment of the present application. In Figure 4(c), the image on the left represents the fused absolute phase map obtained by fusing the absolute phase map under the first exposure time and the absolute phase map under the second exposure time based on the three-dimensional reconstruction method provided by the present application. In Figure 4(c), the image on the right represents the three-dimensional point cloud obtained by three-dimensional reconstruction based on the fused absolute phase map. It can be seen that the three-dimensional reconstruction method provided in an embodiment of the present application can be applied to high dynamic range scenes and improve the accuracy of the obtained three-dimensional reconstruction results.

[0272] Based on the same inventive concept, an embodiment of the present application provides a three-dimensional reconstruction system.

[0273] See also Figure 5 , Figure 5 The 3D reconstruction system 500 includes a projection device 501 , an image acquisition device 502 , and a processor 503 .

[0274] The projection device 501 is used to project the binary coded image and the fringe image onto the area to be measured.

[0275] The image acquisition device 502 is used to acquire images of the area to be measured when the projection device 501 projects each image onto the area to be measured at different exposure times.

[0276] The processor 503 is configured to execute any of the three-dimensional reconstruction methods in the above embodiments.

[0277] Based on the same inventive concept, the present application provides a three-dimensional reconstruction device. Figure 6 , Figure 6 This is a structural diagram of a three-dimensional reconstruction device provided in an embodiment of the present application, the device comprising:

[0278] The image acquisition module 601 is configured to acquire, for each of a plurality of exposure times, an image acquired during the exposure time when each binary-coded image is projected onto the area to be measured, thereby obtaining a plurality of first images; and acquire, for each of a plurality of exposure times when each fringe image is projected onto the area to be measured, thereby obtaining a plurality of second images.

[0279] An image processing module 602 is configured to calculate an absolute phase map and a modulation map for the exposure time using each of the first and second images for the exposure time; wherein the absolute phase map records the absolute phase of each pixel position; and the modulation map records the modulation of each pixel position.

[0280] The confidence acquisition module 603 is configured to calculate the confidence of the absolute phase of each pixel position at the exposure duration based on the modulation degree and / or fringe sinusoidal error at the exposure duration. The fringe sinusoidal error at each pixel position at the exposure duration represents the degree to which the grayscale value of each pixel position in each second image at the exposure duration conforms to a sine function. The confidence of the absolute phase of a pixel position at the exposure duration is positively correlated with the modulation degree of the pixel position at the exposure duration and negatively correlated with the fringe sinusoidal error at the exposure duration.

[0281] A fusion module 604 is configured to select the absolute phase with the highest confidence level from the absolute phases at each pixel position under each exposure duration to obtain a fused absolute phase map;

[0282] The three-dimensional reconstruction module 605 is configured to perform three-dimensional reconstruction based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured.

[0283] In some embodiments, the image processing module 602 includes:

[0284] a decoding value calculation submodule, configured to calculate, for each pixel position, a decoding value of the pixel position at the exposure duration based on the grayscale value of the pixel position in each first image at the exposure duration;

[0285] a relative phase calculation submodule, configured to calculate a relative phase map and a modulation map for the exposure time using each second image for the exposure time; wherein the projected fringe images are multiple line-shifted fringe images or multiple phase-shifted fringe images; and the relative phase map records the relative phase of each pixel position;

[0286] The absolute phase calculation submodule is used to perform phase expansion on the relative phase map for the exposure duration using the decoded value of each pixel position for the exposure duration to obtain the absolute phase map for the exposure duration; wherein the absolute phase map records the absolute phase of each pixel position.

[0287] In some embodiments, the relative phase calculation submodule is specifically configured to:

[0288] For each pixel position, calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the sine value of the phase shift of the corresponding fringe image as the first inner product of the pixel position;

[0289] Calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the cosine value of the phase shift of the corresponding fringe image as the second inner product of the pixel position;

[0290] Calculate the ratio of the first inner product to the second inner product at the pixel position under the exposure duration, and use the arc tangent of the ratio as the relative phase of the pixel position under the exposure duration, to obtain a relative phase map under the exposure duration; calculate the sum of the squares of the first inner product and the second inner product at the pixel position under the exposure duration, and use the normalized value of the square root of the calculated sum of squares as the modulation degree of the pixel position under the exposure duration, to obtain a modulation degree map under the exposure duration.

[0291] In some embodiments, the absolute phase calculation submodule is specifically configured to:

[0292] For each pixel position, the relative phase of the pixel position at the exposure time is calculated, and the sum of the phase change corresponding to the decoded value of the pixel position at the exposure time is used as the absolute phase of the pixel position at the exposure time to obtain the absolute phase map at the exposure time.

[0293] In some embodiments, the fringe sinusoidal error at each pixel position under the exposure duration is calculated by the following steps:

[0294] Using each second image under the exposure time, a background intensity map under the exposure time is calculated; wherein the background intensity map records the background intensity of each pixel position;

[0295] For each second image under the exposure time, calculating the relative phase of each pixel position under the exposure time and the cosine value of the difference between the phase shift amount of the fringe image corresponding to the second image;

[0296] The product of the calculated cosine value and the modulation degree of the pixel position under the exposure time;

[0297] The calculated product and the sum of the background intensity at the pixel position at the exposure time are used to obtain a theoretical grayscale value of the pixel position at the exposure time for the second image;

[0298] The difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time is calculated to obtain the fringe sinusoidal error of the pixel position at the exposure time.

[0299] In some embodiments, calculating the difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time to obtain the fringe sinusoidal error of the pixel position at the exposure time includes:

[0300] For each pixel position, calculating the absolute value of the difference between a theoretical grayscale value of the pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in the second image as the error component of the pixel position for the second image;

[0301] The average of the error components of the pixel position for each second image is calculated as the fringe sinusoidal error of the pixel position under the exposure time.

[0302] In some embodiments, the confidence acquisition module 603 is specifically configured to:

[0303] Normalizing the fringe sinusoidal error at each pixel position under the exposure time to a range of 0 to 1, thereby obtaining a normalized value of the fringe sinusoidal error at each pixel position under the exposure time;

[0304] A weighted sum of the modulation degree and the normalized value of the fringe sinusoidal error at each pixel position under the exposure duration is calculated as the confidence level of the absolute phase of the pixel position under the exposure duration.

[0305] In some embodiments, each first image and each second image at each exposure time is acquired by using an image acquisition device;

[0306] The three-dimensional reconstruction module 605 is specifically used to:

[0307] In the case where the image acquisition device is a binocular camera, based on a preset binocular 3D reconstruction algorithm, 3D reconstruction is performed on the fused absolute phase images obtained based on the images acquired by the two cameras to obtain a 3D reconstruction result of the area to be measured;

[0308] or,

[0309] When the image acquisition device is a monocular camera, three-dimensional reconstruction is performed on the obtained fused absolute phase image based on a preset monocular three-dimensional reconstruction algorithm to obtain a three-dimensional reconstruction result of the area to be measured.

[0310] The present application also provides an electronic device, such as Figure 7 Shown, including:

[0311] Memory 701, used for storing computer programs;

[0312] The processor 702 is configured to implement any of the steps of the above-mentioned three-dimensional reconstruction method when executing the program stored in the memory 701 .

[0313] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 702, the communication interface, and the memory 701 communicate with each other via the communication bus.

[0314] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0315] The communication interface is used for communication between the above electronic device and other devices.

[0316] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0317] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0318] In another embodiment provided in the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of any of the above-mentioned three-dimensional reconstruction methods are implemented.

[0319] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the three-dimensional reconstruction methods in the above embodiments.

[0320] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0321] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0322] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system, device, electronic device, and computer-readable storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0323] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: For each of the plurality of exposure times, acquiring an image acquired during the exposure time when each binary-coded image is projected onto the area to be measured, thereby obtaining a plurality of first images; and acquiring an image acquired during the exposure time when each fringe image is projected onto the area to be measured, thereby obtaining a plurality of second images. Using each first image and each second image under the exposure time, an absolute phase map and a modulation map under the exposure time are calculated; wherein the absolute phase map records the absolute phase of each pixel position; and the modulation map records the modulation of each pixel position; Calculating a confidence level of the absolute phase of each pixel position at the exposure duration based on the modulation level and / or the fringe sinusoidal error at the exposure duration; wherein the fringe sinusoidal error at each pixel position at the exposure duration represents the degree to which the grayscale value of each pixel position in each second image at the exposure duration conforms to a sine function; and the confidence level of the absolute phase of a pixel position at the exposure duration is positively correlated with the modulation level of the pixel position at the exposure duration and negatively correlated with the fringe sinusoidal error at the exposure duration. From the absolute phases of each pixel position at each exposure time, the absolute phase with the highest confidence is selected to obtain the fused absolute phase map; Three-dimensional reconstruction is performed based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured.

2. The method according to claim 1, characterized in that The step of calculating the absolute phase map and the modulation map under the exposure time using the first images and the second images under the exposure time includes: For each pixel position, calculating a decoded value of the pixel position at the exposure duration based on the grayscale values ​​of the pixel position in each first image at the exposure duration; Using each second image under the exposure time, a relative phase map and a modulation map under the exposure time are calculated; wherein the projected fringe images are multiple line-shifted fringe images or multiple phase-shifted fringe images; and the relative phase map records the relative phase of each pixel position; The decoded value of each pixel position at the exposure time is used to perform phase unwrapping on the relative phase map at the exposure time to obtain an absolute phase map at the exposure time; wherein the absolute phase of each pixel position is recorded in the absolute phase map.

3. The method according to claim 2, characterized in that The step of calculating the relative phase map and the modulation map under the exposure time using each second image under the exposure time includes: For each pixel position, calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the sine value of the phase shift of the corresponding fringe image as the first inner product of the pixel position; Calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the cosine value of the phase shift of the corresponding fringe image as the second inner product of the pixel position; Calculate the ratio of the first inner product to the second inner product at the pixel position under the exposure duration, and use the arc tangent of the ratio as the relative phase of the pixel position under the exposure duration, to obtain a relative phase map under the exposure duration; calculate the sum of the squares of the first inner product and the second inner product at the pixel position under the exposure duration, and use the normalized value of the square root of the calculated sum of squares as the modulation degree of the pixel position under the exposure duration, to obtain a modulation degree map under the exposure duration.

4. The method according to claim 2 or 3, characterized in that The method of performing phase unwrapping on the relative phase map under the exposure duration by using the decoded value of each pixel position under the exposure duration to obtain the absolute phase map under the exposure duration includes: For each pixel position, the relative phase of the pixel position at the exposure time is calculated, and the sum of the phase change corresponding to the decoded value of the pixel position at the exposure time is used as the absolute phase of the pixel position at the exposure time to obtain the absolute phase map at the exposure time.

5. The method according to claim 1, wherein The fringe sinusoidal error at each pixel position under the exposure time is calculated by the following steps: Using each second image under the exposure time, a background intensity map under the exposure time is calculated; wherein the background intensity map records the background intensity of each pixel position; For each second image under the exposure time, calculating the relative phase of each pixel position under the exposure time and the cosine value of the difference between the phase shift amount of the fringe image corresponding to the second image; The product of the calculated cosine value and the modulation degree of the pixel position under the exposure time; The calculated product and the sum of the background intensity at the pixel position at the exposure time are used to obtain a theoretical grayscale value of the pixel position at the exposure time for the second image; The difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time is calculated to obtain the fringe sinusoidal error of the pixel position at the exposure time.

6. The method according to claim 5, characterized in that Calculating the difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time to obtain the fringe sinusoidal error of the pixel position at the exposure time includes: For each pixel position, calculating the absolute value of the difference between a theoretical grayscale value of the pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in the second image as the error component of the pixel position for the second image; The average of the error components of the pixel position for each second image is calculated as the fringe sinusoidal error of the pixel position under the exposure time.

7. The method according to claim 1, characterized in that The step of calculating the confidence level of the absolute phase of each pixel position at the exposure duration based on the modulation degree and / or fringe sinusoidal error at each pixel position at the exposure duration includes: Normalizing the fringe sinusoidal error at each pixel position under the exposure time to a range of 0 to 1, thereby obtaining a normalized value of the fringe sinusoidal error at each pixel position under the exposure time; A weighted sum of the modulation degree and the normalized value of the fringe sinusoidal error at each pixel position under the exposure duration is calculated as the confidence level of the absolute phase of the pixel position under the exposure duration.

8. The method according to claim 1, characterized in that Each first image and each second image at each exposure time is acquired by using an image acquisition device; The performing three-dimensional reconstruction based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured includes: In the case where the image acquisition device is a binocular camera, based on a preset binocular 3D reconstruction algorithm, 3D reconstruction is performed on the fused absolute phase images obtained based on the images acquired by the two cameras to obtain a 3D reconstruction result of the area to be measured; or, When the image acquisition device is a monocular camera, three-dimensional reconstruction is performed on the obtained fused absolute phase image based on a preset monocular three-dimensional reconstruction algorithm to obtain a three-dimensional reconstruction result of the area to be measured.

9. A three-dimensional reconstruction system, characterized in that: The system includes: a projection device, an image acquisition device, and a processor; The projection device is used to project the binary coded image and the fringe image onto the area to be measured; The image acquisition device is used to acquire images of the area to be measured when the projection device projects each image onto the area to be measured at different exposure times; The processor is configured to execute the method according to any one of claims 1 to 8.

10. A three-dimensional reconstruction device, characterized in that: The device comprises: an image acquisition module configured to acquire, for each of a plurality of exposure durations, an image acquired during the exposure duration when each binary-coded image is projected onto the area to be measured, thereby obtaining a plurality of first images; and to acquire, for each of a plurality of exposure durations, an image acquired during the exposure duration when each fringe image is projected onto the area to be measured, thereby obtaining a plurality of second images; An image processing module, configured to calculate an absolute phase map and a modulation map for the exposure time using each of the first and second images for the exposure time; wherein the absolute phase map records the absolute phase of each pixel position; and the modulation map records the modulation of each pixel position; A confidence acquisition module is configured to calculate a confidence level of the absolute phase of each pixel position at the exposure duration based on the modulation level and / or the fringe sinusoidal error at the exposure duration. The fringe sinusoidal error at each pixel position at the exposure duration represents the degree to which the grayscale value of each pixel position in each second image at the exposure duration conforms to a sine function. The confidence level of the absolute phase of a pixel position at the exposure duration is positively correlated with the modulation level of the pixel position at the exposure duration and negatively correlated with the fringe sinusoidal error at the exposure duration. A fusion module is used to select the absolute phase with the highest confidence from the absolute phases of each pixel position under various exposure times to obtain a fused absolute phase map; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the fused absolute phase image to obtain a three-dimensional reconstruction result of the area to be measured.

11. The device according to claim 10, characterized in that The image processing module includes: a decoding value calculation submodule, configured to calculate, for each pixel position, a decoding value of the pixel position at the exposure duration based on the grayscale value of the pixel position in each first image at the exposure duration; a relative phase calculation submodule, configured to calculate a relative phase map and a modulation map for the exposure time using each second image for the exposure time; wherein the projected fringe images are multiple line-shifted fringe images or multiple phase-shifted fringe images; and the relative phase map records the relative phase of each pixel position; An absolute phase calculation submodule is configured to perform phase unwrapping on the relative phase map for the exposure duration using the decoded values ​​of each pixel position for the exposure duration, thereby obtaining an absolute phase map for the exposure duration; wherein the absolute phase map records the absolute phase of each pixel position; And / or, the relative phase calculation submodule is specifically configured to: For each pixel position, calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the sine value of the phase shift of the corresponding fringe image as the first inner product of the pixel position; Calculating the inner product of the grayscale value of the pixel position in each second image under the exposure time and the cosine value of the phase shift of the corresponding fringe image as the second inner product of the pixel position; Calculating the ratio of the first inner product to the second inner product at the pixel position under the exposure duration, and using the arc tangent of the ratio as the relative phase of the pixel position under the exposure duration, thereby obtaining a relative phase map under the exposure duration; calculating the sum of the squares of the first inner product and the second inner product at the pixel position under the exposure duration, and using the normalized value of the square root of the calculated sum of squares as the modulation index of the pixel position under the exposure duration, thereby obtaining a modulation index map under the exposure duration; And / or, the absolute phase calculation submodule is specifically used to: For each pixel position, calculate the relative phase of the pixel position at the exposure duration, and the sum of the phase change corresponding to the decoded value of the pixel position at the exposure duration as the absolute phase of the pixel position at the exposure duration, to obtain the absolute phase map for the exposure duration; And / or, the fringe sinusoidal error at each pixel position under the exposure time is calculated by the following steps: Using each second image under the exposure time, a background intensity map under the exposure time is calculated; wherein the background intensity map records the background intensity of each pixel position; For each second image under the exposure time, calculating the relative phase of each pixel position under the exposure time and the cosine value of the difference between the phase shift amount of the fringe image corresponding to the second image; The product of the calculated cosine value and the modulation degree of the pixel position under the exposure time; The calculated product and the sum of the background intensity at the pixel position at the exposure time are used to obtain a theoretical grayscale value of the pixel position at the exposure time for the second image; Calculating the difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time to obtain the fringe sinusoidal error of the pixel position at the exposure time; And / or, calculating the difference between the theoretical grayscale value of each pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in each second image at the exposure time to obtain the fringe sinusoidal error of the pixel position at the exposure time includes: For each pixel position, calculate the absolute value of the difference between the theoretical grayscale value of the pixel position for each second image at the exposure time and the actual grayscale value of the pixel position in the second image as the error component of the second image at the pixel position; calculate the average of the error components of the pixel position for each second image as the fringe sine error of the pixel position at the exposure time; And / or, the confidence acquisition module is specifically used to: Normalizing the fringe sinusoidal error at each pixel position under the exposure time to a range of 0 to 1, thereby obtaining a normalized value of the fringe sinusoidal error at each pixel position under the exposure time; Calculate the weighted sum of the modulation degree and the normalized value of the fringe sinusoidal error at each pixel position under the exposure time as the confidence level of the absolute phase of the pixel position under the exposure time; And / or, each first image and each second image at each exposure time is acquired by using an image acquisition device; The three-dimensional reconstruction module is specifically used to: In the case where the image acquisition device is a binocular camera, based on a preset binocular 3D reconstruction algorithm, 3D reconstruction is performed on the fused absolute phase images obtained based on the images acquired by the two cameras to obtain a 3D reconstruction result of the area to be measured; or, When the image acquisition device is a monocular camera, three-dimensional reconstruction is performed on the obtained fused absolute phase image based on a preset monocular three-dimensional reconstruction algorithm to obtain a three-dimensional reconstruction result of the area to be measured.

12. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 8 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images