Three-dimensional reconstruction method, three-dimensional reconstruction device, and computer-readable storage medium

Abstract

The application provides a three-dimensional reconstruction method, a three-dimensional reconstruction device and a computer readable storage medium. The three-dimensional reconstruction method comprises: acquiring a modulated fringe image, acquiring a wrapped phase based on the modulated fringe image; acquiring an infrared image based on the modulated fringe image; acquiring a first three-dimensional point cloud by using the infrared image and a visible light image; acquiring a first absolute phase by using the first three-dimensional point cloud and the wrapped phase; and reconstructing a second three-dimensional point cloud of an object according to the modulated fringe image and the first absolute phase. In this way, the three-dimensional reconstruction device uses an infrared projector and an infrared camera to collect a projected fringe image, so that non-invasive three-dimensional reconstruction can be realized. Two visual subsystems are used for three-dimensional reconstruction, the information obtained by the two subsystems is fully utilized, and the three-dimensional reconstruction results are fused, so that the speed and accuracy of three-dimensional reconstruction can be greatly improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method, a three-dimensional reconstruction device, and a computer-readable storage medium. Background Technology

[0002] 3D reconstruction is a fundamental technology in computer vision and is widely used in fields such as robotics, autonomous driving, industrial inspection, AR, and motion-sensing entertainment.

[0003] Among the current mainstream 3D measurement technologies, time-of-flight technology offers fast measurement speed, but the equipment is expensive, bulky, and lacks high accuracy; binocular stereo vision technology offers fast measurement speed, but its accuracy is limited and it is easily affected by lighting and texture; structured light measurement technology offers high accuracy and high point density, with speckle structured light technology requiring only one projection, but it cannot obtain dense 3D data. Fringe projection profilometry obtains dense 3D data, but to achieve high accuracy, it often requires projecting multiple coded patterns, significantly limiting the measurement speed.

[0004] Consumer-grade depth cameras such as Kinect V1 and iPhone's Face ID module typically consist of an infrared projector, an infrared camera, and a visible light camera. The infrared projector projects an infrared-coded image onto the target, the infrared camera captures the coded image modulated by the object's shape to reconstruct the 3D model, and the visible light camera is used to acquire color images and does not participate in 3D reconstruction. This results in a waste of sensors and information, and the accuracy is somewhat limited. Summary of the Invention

[0005] This application provides a three-dimensional reconstruction method, a three-dimensional reconstruction device, and a computer-readable storage medium.

[0006] This application provides a three-dimensional reconstruction method, the three-dimensional reconstruction method comprising:

[0007] Acquire a modulation stripe image, and obtain the wrapping phase based on the modulation stripe image;

[0008] Infrared images are obtained based on the modulated stripe image;

[0009] The first three-dimensional point cloud is obtained using the infrared and visible light images;

[0010] Using the first 3D point cloud and the wrapped phase, obtain the first absolute phase;

[0011] The second three-dimensional point cloud of the object is reconstructed based on the modulated stripe image and the first absolute phase.

[0012] The acquisition of the modulated stripe image includes:

[0013] An infrared projector is used to project N-step phase-shifted fringes onto the surface of an object, where N is an integer greater than 0;

[0014] Acquire a modulated stripe image of the phase-shifted stripes modulated by the surface of the object.

[0015] The step of obtaining the first three-dimensional point cloud using the infrared image and the visible light image includes:

[0016] Obtain the first spatially invariant common feature of the infrared image;

[0017] Obtain the second spatially invariant common feature of the visible light image;

[0018] Based on the first spatially invariant common feature and the second spatially invariant common feature, a disparity map is established;

[0019] The disparity map is converted into the first three-dimensional point cloud using infrared camera calibration parameters and visible light calibration parameters.

[0020] The step of establishing a disparity map based on the first spatially invariant common feature and the second spatially invariant common feature includes:

[0021] The first spatially invariant common feature and the second spatially invariant common feature are input into the stereo matching network, and the disparity map is output through the stereo matching network.

[0022] The stereo matching network is obtained by weighted training using a smoothing loss function and an L1 loss function.

[0023] The three-dimensional reconstruction method further includes, after obtaining the first three-dimensional point cloud using the infrared and visible light images:

[0024] The measurement range of the infrared camera is determined based on the spatial span of the fringe period of the modulated fringe image.

[0025] Based on the measurement range and the actual depth of the object, obtain the offset value;

[0026] The first three-dimensional point cloud is offset along a preset direction according to the offset value to obtain the third three-dimensional point cloud.

[0027] The step of obtaining the first absolute phase using the first 3D point cloud and the wrapped phase includes:

[0028] Obtain the first calibration parameters of the infrared projector;

[0029] The second absolute phase is solved using the first three-dimensional point cloud and the first calibration parameters;

[0030] The first absolute phase is solved using the second absolute phase and the wrapping phase.

[0031] The step of reconstructing the second three-dimensional point cloud of the object according to the modulated stripe image and the first absolute phase includes:

[0032] Obtain the second calibration parameters of the infrared camera;

[0033] A first equation is constructed using the second calibration parameters and the pixel coordinates of the modulated stripe image;

[0034] A second equation is constructed using the first calibration parameters and the pixel values ​​of the modulated stripe image;

[0035] A third-party program is constructed using the pixel values ​​of the modulated stripe image and the first absolute phase.

[0036] The first equation, the second equation, and the third-party program are solved together to obtain the second three-dimensional point cloud of the object.

[0037] The step of constructing a third-party program using the pixel values ​​of the modulated stripe image and the first absolute phase includes:

[0038] Obtain the third calibration parameters of the infrared projector, wherein the first calibration parameters include the projection matrix, and the third calibration parameters include the horizontal projection fringe frequency, the vertical projection fringe frequency, the horizontal resolution of the projector, and / or the vertical resolution of the projector.

[0039] The third-party program is constructed using the pixel values ​​of the modulated stripe image, the first absolute phase, and the third calibration parameter.

[0040] This application also provides a three-dimensional reconstruction apparatus, which includes a processor and a memory. The memory stores program data, and the processor executes the program data to implement the three-dimensional reconstruction method as described above.

[0041] This application also provides a computer-readable storage medium for storing program data, which, when executed by a processor, is used to implement the above-described three-dimensional reconstruction method.

[0042] The beneficial effects of this application are as follows: the 3D reconstruction device acquires a modulated fringe image, obtains a wrapping phase based on the modulated fringe image, acquires an infrared image based on the modulated fringe image, acquires a first 3D point cloud using the infrared image and the visible light image, acquires a first absolute phase using the first 3D point cloud and the wrapping phase, and reconstructs a second 3D point cloud of the object according to the modulated fringe image and the first absolute phase. Through the above method, the 3D reconstruction device uses an infrared projector and an infrared camera to acquire projected fringe patterns, enabling seamless 3D reconstruction; by utilizing two vision subsystems for 3D reconstruction and fully utilizing the information acquired by both, and fusing their 3D reconstruction results, the speed and accuracy of 3D reconstruction can be significantly improved. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0044] Figure 1 This is a flowchart illustrating an embodiment of the three-dimensional reconstruction method provided in this application;

[0045] Figure 2 This is a schematic diagram of the overall process of the three-dimensional reconstruction method provided in this application;

[0046] Figure 3 This is a schematic diagram of the framework of an embodiment of the depth camera device provided in this application;

[0047] Figure 4 yes Figure 1 The diagram shows the detailed process flow of step S13 in the 3D reconstruction method.

[0048] Figure 5 This is a schematic diagram of the cross-modal binocular stereo matching algorithm provided in this application;

[0049] Figure 6 This is a schematic diagram of the structure of an embodiment of the three-dimensional reconstruction device provided in this application;

[0050] Figure 7 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0052] In the 3D reconstruction method of this application, an infrared camera and an infrared projector can be used to form a monocular structured photonic system, while an infrared camera and an RGB camera (visible light camera) can form a pair of binocular stereo vision subsystems. Both subsystems can perform 3D reconstruction. By making full use of the information obtained by the two and fusing their 3D reconstruction results, the speed and accuracy of 3D reconstruction can be greatly improved.

[0053] Specifically, the 3D reconstruction device first projects N-step phase-shifted fringes (N>=3) onto the object using an infrared projector. Simultaneously, a synchronization module sends a trigger signal to control an infrared camera to acquire the image of the fringes modulated on the object's surface. Phase calculation is then performed to obtain the wrapped phase, and a fringe-free infrared image is calculated. Simultaneously, the synchronization module controls a visible light camera to acquire the visible light image of the object. A cross-modal binocular stereo matching algorithm is used to match the infrared and visible light images, resulting in a coarse disparity map, which is then converted into a 3D point cloud. The coarse 3D point cloud obtained in the previous step is offset, and the phase is inversely solved using the offset 3D point cloud to obtain a coarse absolute phase. Phase unfolding is then performed using the wrapped phase and the coarse absolute phase to obtain a precise absolute phase. Finally, calibration parameters are used to obtain a precise 3D point cloud.

[0054] Please refer to the details. Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the three-dimensional reconstruction method provided in this application. Figure 2 This is a schematic diagram of the overall process of the three-dimensional reconstruction method provided in this application.

[0055] The three-dimensional reconstruction method of this application is applied to a three-dimensional reconstruction device, which can be a server or a system consisting of a server and terminal devices working together. Accordingly, the various parts of the three-dimensional reconstruction device, such as units, sub-units, modules, and sub-modules, can all be located in the server, or they can be located separately in the server and terminal devices.

[0056] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed servers, or as a single software program or software module; no specific limitation is made here. In some possible implementations, the three-dimensional reconstruction method of this application embodiment can be implemented by a processor calling computer-readable instructions stored in memory.

[0057] Specifically, such as Figure 1 As shown, the three-dimensional reconstruction method of this application embodiment specifically includes the following steps:

[0058] Step S11: Obtain the modulation stripe image, and obtain the wrapping phase based on the modulation stripe image.

[0059] In this application embodiment, the hardware device for implementing the three-dimensional reconstruction method is first described below. Please refer to [link / reference] for details. Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the depth camera device provided in this application.

[0060] like Figure 3 As shown, the RGBD depth camera device provided in this application includes an infrared camera, a visible light camera, an infrared projector, a synchronization component, a processing component, etc.

[0061] The system comprises an infrared camera for acquiring infrared images in the infrared band, a visible light camera for acquiring visible light images in the visible light band, and an infrared projector for projecting infrared-coded structured light patterns. The infrared camera and infrared projector form a monocular structured light subsystem, while the infrared camera and visible light camera form a binocular stereo vision subsystem.

[0062] The synchronization component is used to control the synchronous operation of the infrared camera, visible light camera, and infrared projector. The processing component is used to control, process, and store the data generated during the operation of the device.

[0063] To improve the accuracy of 3D reconstruction, the infrared camera maintains a large baseline distance from both the infrared projector and the visible light camera, while there is no requirement for the distance between the infrared projector and the visible light camera. Therefore, the visible light camera and the infrared projector are relatively close together, while the infrared camera is relatively far from both.

[0064] When the 3D reconstruction device and the RGBD depth camera device start working, firstly, the infrared projector projects N-step phase-shifted fringes (N>=3) onto the object, and the synchronization module sends a trigger signal to control the infrared camera to acquire N images of the modulated fringes modulated by the object's surface; at the same time, the synchronization module controls the visible light camera to acquire a visible light image of the object.

[0065] Furthermore, a monocular structured photonic system consisting of an infrared camera and an infrared projector acquires an N-step phase-shifted fringe image modulated on the object's surface, and calculates the phase using the following formula to obtain the encapsulated phase:

[0066]

[0067] Where (x,y) represents the image pixel coordinates, I n δ is the image captured by the camera during the nth projection of each set of stripes. n =2π×(n-1) / N represents the phase shift number. This represents the phase of the package to be determined, with values ​​ranging from (-π, π].

[0068] Step S12: Obtain an infrared image based on the modulated stripe image.

[0069] In this embodiment of the application, the three-dimensional reconstruction device calculates a stripe-free infrared image based on the adjusted stripe image, and the calculation method is as follows:

[0070]

[0071] Among them, I IR This is a stripeless infrared image.

[0072] Step S13: Obtain the first three-dimensional point cloud using infrared and visible light images.

[0073] In this embodiment of the application, the three-dimensional reconstruction device uses a visible light camera to acquire a visible light image of the object, and forms a pair of heterogeneous binocular images with the infrared image in step S12; then, a cross-modal binocular stereo matching algorithm is used to perform heterogeneous image matching between the infrared and visible light images to obtain a coarse disparity map and convert it to obtain three-dimensional data.

[0074] Please refer to the following for details. Figure 4 , Figure 4 yes Figure 1 The diagram shows the specific process flow of step S13 in the three-dimensional reconstruction method.

[0075] Specifically, such as Figure 4 As shown, the three-dimensional reconstruction method of this application embodiment specifically includes the following steps:

[0076] Step S131: Obtain the first spatially invariant common features of the infrared image.

[0077] Step S132: Obtain the second spatially invariant common features of the visible light image.

[0078] In this embodiment, the 3D reconstruction device reconstructs a 3D point cloud using a cross-modal binocular stereo matching algorithm. Specifically, the cross-modal binocular stereo matching algorithm consists of two common feature extraction modules and a binocular stereo matching module. For details, please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic diagram of the cross-modal stereo matching algorithm provided in this application. Each module of the cross-modal stereo matching algorithm consists of several convolutional layers, batch normalization (BN) layers, pooling layers, and activation layers.

[0079] The common feature extraction module is based on the UNet network, with non-shared weights. The input is an infrared image or a visible light image, and the output is the spatially invariant common features of the infrared image and the spatially invariant common features of the visible light image.

[0080] Spatial invariant common features are the feature intersections of infrared and visible light images, and are spatially strictly aligned with both infrared and visible light images respectively. Therefore, using spatial invariant common features for stereo matching can solve the problem of heterogeneous image alignment, where visible light images have rich texture and color features, while infrared images are blurry and have weak texture, making matching difficult.

[0081] Step S133: Based on the first space-invariant common features and the second space-invariant common features, establish a disparity map.

[0082] In this embodiment of the application, the three-dimensional reconstruction device uses the first spatially invariant common feature extracted in step S131 and the second spatially invariant common feature extracted in step S132 to input the stereo matching network in the binocular stereo matching module, and outputs the disparity map of the infrared image and the visible light image.

[0083] in, Figure 5 The stereo matching network shown consists of an SPP (Spatial Pyramid Pooling) module that integrates global context and a stacked hourglass module for cost volume regularization. Determining contextual relationships solely from pixel intensity is difficult. The SPP module learns the relationship between an object and its sub-regions (windows, tires, hoods, etc.) to incorporate hierarchical contextual information. The SPP module uses adaptive average pooling to compress features to four scales, then upsamples the low-dimensional feature map to the same size as the original feature map using 1×1 convolutional dimensions and bilinear interpolation. A 4D cost volume is then formed using features from both images and fed into a 3D CNN for cost volume regularization and disparity regression.

[0084] Specifically, the stereo matching algorithm used in this application is an improvement on psmnet (Pyramid StereoMatching Network, an end-to-end stereo matching network). Its loss function uses a weighted combination of L1 Loss and Smooth Loss. Smooth Loss can reduce singularities in the disparity map and coarse 3D points, which is beneficial for subsequent steps to inversely solve the absolute phase using coarse 3D points. The formula for the loss function is as follows:

[0085] Loss = A * loss l1 +B*loss smooth

[0086] Where A and B are weights.

[0087]

[0088] in, is the second derivative of the visible light image in the horizontal and vertical directions, d is the parallax, and I is the second derivative of the visible light image in the horizontal and vertical directions. ivs It is a visible light image.

[0089] Step S134: Convert the disparity map into a first three-dimensional point cloud using infrared camera calibration parameters and visible light calibration parameters.

[0090] In this embodiment, the 3D reconstruction device uses infrared camera calibration parameters and visible light calibration parameters to convert the disparity map into a coarse 3D point cloud (x-axis). w ,y w ,z w ).

[0091] In other embodiments, the 3D reconstruction device may also use other 3D reconstruction techniques or other 3D reconstruction algorithms to process infrared images and visible light images to obtain reconstructed 3D point clouds.

[0092] Step S14: Obtain the first absolute phase using the first 3D point cloud and the wrapped phase.

[0093] In this embodiment of the application, the three-dimensional reconstruction device uses the coarse three-dimensional point cloud reconstructed in step S13 and the wrapping phase determined in step S11 to obtain an accurate absolute phase, namely the first absolute phase, and uses the absolute phase to obtain accurate three-dimensional reconstruction data, namely the second three-dimensional point cloud.

[0094] Previously, due to noise and calibration accuracy limitations, the obtained coarse 3D point cloud might not be located in front of the actual object, leading to ambiguity in subsequent phase unfolding. Therefore, it is necessary to adjust the coarse 3D point cloud (x w ,y w ,z w ) Shift Δz towards the camera along the z-direction.

[0095] The offset value Δz needs to satisfy the following conditions:

[0096] z true -z range <z w -Δz <z true

[0097] Among them, z true The actual depth of the object, z range This is the measurement range of monocular structured light. The measurement range can be calculated using the following formula:

[0098] z range ≈T p / tan(θ)

[0099] Among them, T p It is the spatial span of a fringe period, and θ is the angle between the infrared camera and the infrared projector.

[0100] Furthermore, the 3D reconstruction device uses the offset 3D point cloud and the calibration parameters of the infrared projector to inversely solve for the coarse absolute phase, i.e., the second absolute phase. The specific process is as follows:

[0101]

[0102]

[0103] Among them, S p M is a constant scale factor. P The projection matrix of the projector is a parameter obtained from calibration, Φ. ref To obtain the rough absolute phase, (u p ,v p ) represents the pixel coordinates of the infrared projector, which are determined by the pixel values ​​of the modulated stripe image.

[0104] Since the monocular structured light system and the binocular system share the same infrared camera, they can both be calibrated to the same world coordinate system, eliminating the need for further calibration of the coarse 3D point cloud (x) obtained by the binocular system. w ,y w ,z w Perform coordinate transformation.

[0105] The three-dimensional reconstruction device utilizes the rough absolute phase Φ ref Calculate the precise absolute phase Φ:

[0106]

[0107] Here, ceil[] is the floor function.

[0108] Step S15: Reconstruct the second three-dimensional point cloud of the object according to the modulated fringe image and the first absolute phase.

[0109] In this embodiment of the application, the three-dimensional reconstruction device uses the precise absolute phase Φ determined in step S14 and the calibration parameters of the infrared camera and infrared projector to obtain an accurate three-dimensional point cloud (X,Y,Z).

[0110]

[0111]

[0112]

[0113]

[0114] By solving the above equations simultaneously, the accurate three-dimensional coordinates can be obtained.

[0115] Among them, S c S p M is a constant scale factor. C M P These are the infrared camera mapping matrix and the projector projection matrix, respectively, and are parameters obtained from calibration. (u) c ,v c ) represents the camera pixel coordinates, (u p ,v p F represents the pixel coordinates of the projector; where F u F is the transverse projection fringe frequency. v W is the longitudinal projection fringe frequency. u For the horizontal resolution of the projector, W v This refers to the projector's vertical resolution. (u) c ,v c ), (u p ,v p All of them are corrected using pre-calibrated system distortion parameters, among which camera distortion types include, but are not limited to, pincushion distortion, barrel distortion, and linear distortion.

[0116] In this embodiment, the 3D reconstruction device acquires a modulated fringe image, obtains an enveloping phase based on the modulated fringe image, acquires an infrared image based on the modulated fringe image, acquires a first 3D point cloud using the infrared image and a visible light image, acquires a first absolute phase using the first 3D point cloud and the enveloping phase, and reconstructs a second 3D point cloud of the object according to the modulated fringe image and the first absolute phase. Through this method, the 3D reconstruction device uses an infrared projector and an infrared camera to acquire projected fringe patterns, enabling seamless 3D reconstruction. By utilizing two vision subsystems for 3D reconstruction and fully leveraging the information acquired by both, and fusing their 3D reconstruction results, the speed and accuracy of 3D reconstruction can be significantly improved.

[0117] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0118] To implement the three-dimensional reconstruction method described in the above embodiments, this application also proposes a three-dimensional reconstruction apparatus, for details please refer to [link / reference needed]. Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the three-dimensional reconstruction device provided in this application.

[0119] The three-dimensional reconstruction apparatus 300 of this application embodiment includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled together.

[0120] The memory 31 is used to store program data, and the processor 32 is used to execute the program data to implement the three-dimensional reconstruction method described in the above embodiments.

[0121] In this embodiment, processor 32 can also be referred to as a CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor.

[0122] To implement the three-dimensional reconstruction method of the above embodiments, this application also provides a computer-readable storage medium, such as... Figure 7As shown, the computer-readable storage medium 400 is used to store program data 41, which, when executed by a processor, is used to implement the three-dimensional reconstruction method as described in the above embodiments.

[0123] This application also provides a computer program product, wherein the computer program product includes a computer program operable to cause a computer to perform the three-dimensional reconstruction method as described in the embodiments of this application. The computer program product may be a software installation package.

[0124] The three-dimensional reconstruction method described in the above embodiments of this application, when implemented as a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0125] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A three-dimensional reconstruction method, characterized in that, The three-dimensional reconstruction method includes: Acquire a modulation stripe image, and obtain the wrapping phase based on the modulation stripe image; Infrared images are obtained based on the modulated stripe image; The first three-dimensional point cloud is obtained using the infrared and visible light images; Using the first 3D point cloud and the wrapped phase, obtain the first absolute phase; Reconstruct the second three-dimensional point cloud of the object based on the modulated stripe image and the first absolute phase; The step of obtaining the first absolute phase using the first 3D point cloud and the wrapped phase includes: Obtain the first calibration parameters of the infrared projector; The second absolute phase is solved using the first three-dimensional point cloud and the first calibration parameters; The first absolute phase is solved using the second absolute phase and the wrapping phase.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, The acquisition of the modulated stripe image includes: An infrared projector is used to project N-step phase-shifted fringes onto the surface of an object, where N is an integer greater than 0; Acquire a modulated stripe image of the phase-shifted stripes modulated by the surface of the object.

3. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of obtaining the first three-dimensional point cloud using the infrared image and the visible light image includes: Obtain the first spatially invariant common feature of the infrared image; Obtain the second spatially invariant common feature of the visible light image; Based on the first spatially invariant common feature and the second spatially invariant common feature, a disparity map is established; The disparity map is converted into the first three-dimensional point cloud using infrared camera calibration parameters and visible light calibration parameters.

4. The three-dimensional reconstruction method according to claim 3, characterized in that, The step of establishing a disparity map based on the first spatially invariant common feature and the second spatially invariant common feature includes: The first spatially invariant common feature and the second spatially invariant common feature are input into the stereo matching network, and the disparity map is output through the stereo matching network. The stereo matching network is obtained by weighted training using a smoothing loss function and an L1 loss function.

5. The three-dimensional reconstruction method according to claim 1, characterized in that, After obtaining the first three-dimensional point cloud using the infrared image and the visible light image, the three-dimensional reconstruction method further includes: The measurement range of the infrared camera is determined based on the spatial span of the fringe period of the modulated fringe image. Based on the measurement range and the actual depth of the object, obtain the offset value; The first three-dimensional point cloud is offset along a preset direction according to the offset value to obtain the third three-dimensional point cloud.

6. The three-dimensional reconstruction method according to claim 1, characterized in that, The reconstruction of the second three-dimensional point cloud of the object based on the modulated fringe image and the first absolute phase includes: Obtain the second calibration parameters of the infrared camera; A first equation is constructed using the second calibration parameters and the pixel coordinates of the modulated stripe image; A second equation is constructed using the first calibration parameters and the pixel values ​​of the modulated stripe image; A third-party program is constructed using the pixel values ​​of the modulated stripe image and the first absolute phase. The first equation, the second equation, and the third-party program are solved together to obtain the second three-dimensional point cloud of the object.

7. The three-dimensional reconstruction method according to claim 6, characterized in that, The third-party program constructed using the pixel values ​​of the modulated stripe image and the first absolute phase includes: Obtain the third calibration parameters of the infrared projector, wherein the first calibration parameters include the projection matrix, and the third calibration parameters include the horizontal projection fringe frequency, the vertical projection fringe frequency, the horizontal resolution of the projector, and / or the vertical resolution of the projector. The third-party program is constructed using the pixel values ​​of the modulated stripe image, the first absolute phase, and the third calibration parameter.

8. A three-dimensional reconstruction device, characterized in that, The three-dimensional reconstruction apparatus includes a processor and a memory, the memory storing program data, and the processor executing the program data to implement the three-dimensional reconstruction method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data, which, when executed by a processor, is used to implement the three-dimensional reconstruction method according to any one of claims 1-7.

Images