A structured light three-dimensional reconstruction device and method

Abstract

The present invention discloses a structured light three-dimensional reconstruction device and method, the device includes: a processor, an FPGA controller, a projection device, a memory, a shooting device and a serial high-speed interface, the shooting device includes left and right cameras, the FPGA controller reads the initial grating image stored in the memory, drives the projection device to complete the projection of a pair of initial grating images, and sends a trigger signal to the shooting device, so that the shooting device simultaneously shoots a deformed grating image modulated by the initial grating image; the processor is used to collect the deformed grating image and send it to the memory, so that the FPGA controller reconstructs the deformed grating image and obtains a depth map, and then sends the depth map to the memory, and the processor completes the three-dimensional reconstruction according to the depth map. The present invention solves the problem of low acquisition efficiency and long reconstruction time caused by all decoding calculations and projection acquisition picture control being completed by the PC end.

Description

Technical Field

[0001] The present invention relates to the technical field of structured light 3D scanning, and in particular to a structured light 3D reconstruction device and method. Background Art

[0002] In the fields of 3D vision, defect detection, 3D measurement, etc., structured light 3D scanning technology plays an increasingly important role. At present, structured light 3D scanning technology stores structured light grating images on the PC side, then outputs the images to the projection device for projection, and finally controls the camera to shoot, and reconstructs the deformed grating image data sent back by the camera to calculate the 3D information of the scanned object.

[0003] The prior art uses the PC as a controller. The controller sends the sinusoidal and Gray code stripe gratings to be projected to the projection device. The projection device projects a set of sinusoidal gratings and Gray code grating stripes with the same period onto the surface of the object to be measured. After the grating is projected onto the surface of the object, it will be distorted by the surface structure of the object. The camera synchronously collects the grating image that is deformed on the surface of the object after each phase shift. The collected image is sent to the controller, which reconstructs the three-dimensional shape of the object based on the deformed grating decoding. All decoding calculations and projection acquisition image control are completed by the PC. The efficiency of the entire process is restricted by the performance of the PC processor, and the acquisition control timing of the projection device and the camera cannot be accurately controlled. It can only ensure that the next grating projection and image acquisition will be carried out after a picture is correctly acquired. In order to ensure the correctness of the binocular camera image acquisition, the grating projection must be fixed within the exposure time. The projection and acquisition will not be carried out in parallel. Therefore, in order to ensure the correctness of the acquisition, the efficiency of the three-dimensional scanning will also be reduced. In addition, when the number of projected and captured images is large and binocular structured light scanning is required to capture images with the left and right cameras at the same time, due to the limitations of PC bandwidth and the current serial acquisition method, the next image can only be captured after one image is captured. In other words, only after one camera has captured the image can the other camera capture the image, which seriously restricts the speed of image acquisition. Summary of the invention

[0004] The purpose of the present invention is to provide a structured light 3D reconstruction device and method to solve the problem of low image acquisition efficiency and long reconstruction time in the prior art due to the fact that all decoding calculations and projection acquisition image control are completed by the PC.

[0005] In a first aspect, the present invention provides a structured light three-dimensional reconstruction device, comprising: a processor, an FPGA controller, a projection device, a memory, a shooting device and a serial high-speed interface, wherein the FPGA controller is connected to the memory, and the FPGA controller is connected to the serial high-speed interface, wherein:

[0006] The FPGA controller is used to read the multiple initial grating images stored in the memory according to the grating image coding sequence, and send a driving signal to the projection device;

[0007] The projection device is used to sequentially project the initial grating image onto the object to be scanned;

[0008] The FPGA controller is further configured to output a trigger signal to the shooting device when the projection device completes the projection of an initial grating image;

[0009] The photographing device is used to photograph a deformed grating image obtained by modulating the initial grating image with the object to be scanned after receiving the trigger signal;

[0010] The processor is used to collect the deformed grating image and send the deformed grating image to the memory through the serial high-speed interface;

[0011] The FPGA controller is further configured to read a plurality of the deformed grating images in the memory after the processor has acquired all the deformed grating images, perform calculations based on the plurality of the deformed grating images to obtain a depth map, and send the depth map to the memory;

[0012] The processor is further configured to read the depth map in the memory and complete three-dimensional reconstruction of the object to be scanned according to the depth map.

[0013] Preferably, the photographing device comprises a left camera and a right camera;

[0014] The FPGA controller comprises:

[0015] A bus connection module, used to realize communication with the processor, the projection device, the memory, the left camera and the right camera;

[0016] A bus adapter module, used to implement data interaction with the processor and the user through the bus connection module;

[0017] A storage control module, used to implement data reading and writing operations on the memory through the bus connection module;

[0018] An HLS IP module is used to convert a preset three-dimensional settlement algorithm implemented by the HLS language into a first hardware description language, so that the first hardware description language performs calculation according to the plurality of deformed grating images to obtain a depth map;

[0019] The projection and acquisition control module is used to convert the preset HDL language for realizing the projection and acquisition function into a second hardware description language, so that the second hardware description language can realize the preprocessing of the initial grating image stored in the memory, and drive the projection device to project the preprocessed initial grating image onto the object to be scanned, and after the projection device completes a grating projection, trigger the left camera and the right camera to take pictures at the same time.

[0020] Furthermore, the second hardware description language is specifically used for:

[0021] The initial grating image stored in the memory is converted into video stream data, and the projection device is driven to project the video stream data. After the projection device projects a frame of grating, a trigger signal is sent to the left camera and the right camera, so that the left camera and the right camera simultaneously shoot the deformed grating image obtained by modulating the initial grating image through the object to be scanned.

[0022] Furthermore, the first hardware description language is specifically used for:

[0023] According to the internal and external parameters of the left camera and the right camera, the deformed grating image is subjected to phase adjustment and stereo matching to obtain a depth map.

[0024] Preferably, the method by which the processor completes three-dimensional reconstruction of the object to be scanned according to the depth map is specifically:

[0025] After obtaining the three-dimensional point cloud of the object to be scanned according to the depth map, three-dimensional reconstruction of the object to be scanned is performed according to the three-dimensional point cloud.

[0026] In a second aspect, the present invention further provides a structured light 3D reconstruction method, which is applicable to the structured light 3D reconstruction device as described above, and comprises the following steps:

[0027] S1, the FPGA controller reads a plurality of initial grating images stored in the memory according to the grating image coding sequence, and sends a driving signal to the projection device;

[0028] S2, projecting the initial grating image onto the object to be scanned by the projection device;

[0029] S3, when the projection device completes the projection of an initial grating image, the FPGA controller sends a trigger signal to the shooting device;

[0030] S4, photographing, by the photographing device, a deformed grating image obtained by modulating the initial grating image through the object to be scanned;

[0031] S5, the processor simultaneously collects the deformed grating image, and sends the deformed grating image to the FPGA controller through the serial high-speed interface;

[0032] S6, repeat S2 to S5 until the processor has acquired all the deformed grating images;

[0033] S7, the FPGA controller reads the multiple deformed grating images in the memory, performs calculations based on the multiple deformed grating images to obtain a depth map, and sends the depth map to the memory;

[0034] S8. The processor reads the depth map in the memory, and completes three-dimensional reconstruction of the object to be scanned according to the depth map.

[0035] Preferably, the shooting device includes a left camera and a right camera, and the FPGA controller converts a preset three-dimensional settlement algorithm implemented by the HLS language into a first hardware description language, and calculates the plurality of deformed grating images by using the first hardware description language to obtain a depth map;

[0036] The FPGA controller converts a preset HDL language for realizing the projection and acquisition function into a second hardware description language, uses the second hardware description language to preprocess the initial grating image stored in the memory, and drives the projection device to project the preprocessed initial grating image onto the object to be scanned, and triggers the left camera and the right camera to take pictures when the projection device completes a grating projection.

[0037] Furthermore, the second hardware description language is specifically used for:

[0038] The initial grating image stored in the memory is converted into video stream data, and the projection device is driven to project the video stream data. When the projection device completes projecting a frame of grating, a trigger signal is sent to the left camera and the right camera, so that the left camera and the right camera capture the deformed grating image obtained by modulating the initial grating image with the object to be scanned.

[0039] Furthermore, the first hardware description language is specifically used for:

[0040] According to the internal and external parameters of the left camera and the right camera, the deformed grating image is subjected to phase adjustment and stereo matching to obtain a depth map.

[0041] Preferably, the step S5 specifically includes:

[0042] After obtaining the three-dimensional point cloud of the object to be scanned according to the depth map, three-dimensional reconstruction of the object to be scanned is performed according to the three-dimensional point cloud.

[0043] Compared with the prior art, the structured light 3D reconstruction device and method provided by the present invention adds an FPGA controller. The FPGA controller plays a connecting role in the entire system, connects the various devices in the system together, controls the operation of each device, and ensures the orderly and correct operation of the entire system. By moving the calculation process and the projection acquisition control process into the FPGA, the timing of the projection acquisition control can be precisely controlled, that is, the shooting device can be triggered to take pictures after each grating projection is completed. At the same time, the characteristics of FPGA parallel calculation can ensure a more efficient realization of a single 3D reconstruction settlement. The characteristics of FPGA pipeline work can shorten the waiting time between two 3D reconstructions, and the next work can be started without waiting for the previous work to be completed, thereby maximizing the efficiency of image acquisition, which is not restricted by the performance of the PC processor. Moreover, when the number of projected and acquired images is large, due to the characteristics of the parallel calculation of the FPGA controller, it is not affected by the interface bandwidth and will not restrict the speed of image acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the structure of a structured light 3D reconstruction device provided by an embodiment of the present invention;

[0045] Figure 2 It is a structural schematic diagram of the FPGA controller shown in an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of camera triggering shown in an embodiment of the present invention;

[0047] Figure 4 This is a flow chart of a structured light 3D reconstruction method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not intended to limit the scope of protection of the present invention in any way.

[0049] Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items. In the drawings, the thickness, size and shape of the objects have been slightly exaggerated for ease of explanation. The drawings are only examples and are not drawn strictly to scale.

[0050] It should also be understood that the terms “comprises,” “including,” “having,” “includes,” and / or “comprising,” when used in this specification, indicate the presence of stated features, steps, integers, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, steps, integers, operations, elements, parts, and / or combinations thereof.

[0051] As used in the specification, the terms "substantially," "approximately," and the like are intended to serve as terms of approximation rather than terms of degree, and are intended to account for the inherent deviations in measurements or calculations that would be recognized by those of ordinary skill in the art.

[0052] Unless otherwise defined, all terms (including technical terms and scientific terms) used in this article have the same meaning as commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that terms (such as terms defined in commonly used dictionaries) should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in this article.

[0053] Embodiment 1

[0054] like Figure 1 As shown, the present invention discloses a structured light three-dimensional reconstruction device, including a processor 1, an FPGA controller 2, a projection device 3, a memory 4, a shooting device 5 and a serial high-speed interface, wherein the processor 1, the projection device 3, the memory 4 and the shooting device 5 are all electrically connected to the FPGA controller 2, the shooting device 5 is also electrically connected to the FPGA controller 2, and the FPGA controller is connected to the serial high-speed interface.

[0055] Specifically, the FPGA controller 2 is used to read multiple initial grating images stored in the memory 4 according to the grating image encoding order, and send a driving signal to the projection device 3; the projection device 3 is used to project the multiple initial grating images onto the object to be scanned in sequence; the FPGA controller 2 is also used to output a trigger signal to the shooting device when the projection device 3 completes the projection of a pair of initial grating images; the shooting device 5 is used to shoot the deformed grating image obtained by modulating the initial grating image with the object to be scanned when receiving the trigger signal; the processor 1 is used to capture the deformed grating image and send the deformed grating image to the memory 4 through the serial high-speed interface; the FPGA controller 2 is also used to read the multiple deformed grating images in the memory 4 after the processor 1 has captured all the deformed grating images, perform calculations based on the multiple deformed grating images, process the depth map, and send the depth map to the memory 4; the processor is also used to read the depth map in the memory 4, and complete the three-dimensional reconstruction of the object to be scanned based on the depth map.

[0056] In this embodiment, the FPGA controller 2 first reads the initial grating image stored in the memory 3 according to the grating image coding order, wherein the initial grating image is presented in the form of sinusoidal and Gray code stripe gratings, and then the FPGA controller 2 sends a driving signal to the projection device 3, and the projection device 3 projects a group of sinusoidal gratings and Gray code grating stripes with the same period onto the surface of the object to be scanned. After the grating is projected onto the surface of the object to be scanned, it will be distorted by the surface structure of the object, thereby generating a deformed grating image. When the projection device 3 completes the projection, the FPGA controller 2 sends a trigger signal to the left camera and the right camera, and the left camera and the right camera simultaneously shoot the deformed grating image, and then the processor simultaneously collects the deformed grating images shot by the left camera and the right camera, and sends the deformed grating images to the memory 4. After the FPGA controller 2 calculates the multiple deformed grating images shot by the shooting device 5, a depth map is obtained, and then the FPGA controller 2 sends the depth map to the memory 4 for storage, and at the same time sends the calculated depth map to the processor 1, and the processor 1 completes the three-dimensional reconstruction according to the depth map for use by the user.

[0057] In the embodiment of the present invention, the FPGA plays a connecting role in the entire system, connecting various devices in the system together, controlling the operation of each device, and ensuring the orderly and correct operation of the entire system. By moving the calculation process and the projection acquisition control process into the FPGA, the timing of the projection acquisition control can be precisely controlled, that is, the shooting device 5 can be triggered to take a photo after each grating projection is completed. At the same time, the FPGA parallel calculation ensures that a single three-dimensional reconstruction settlement is more efficiently realized. The FPGA pipeline works to shorten the waiting time between two three-dimensional reconstructions. The deformed grating image is stored in the memory. While the FPGA reads the deformed grating image from the memory for calculation, it sends a driving signal to the projection device to start the next three-dimensional reconstruction calculation, that is, the next three-dimensional reconstruction calculation is started during the intermediate calculation process of the previous three-dimensional reconstruction, ensuring that there is no idle time in the calculation process, and the calculation circuit unit in the FPGA has no idle state, thereby improving the speed of calculation, and the next work can be started without waiting for the completion of the previous work, thereby maximizing the efficiency of image acquisition, and is not restricted by the performance of the PC-side processor. Moreover, when the number of projected and collected images is large, due to the characteristics of the parallel calculation of the FPGA controller 2, it is not affected by the interface bandwidth and will not restrict the speed of image acquisition.

[0058] In a specific embodiment, the processor may be implemented using ARM or CPU, and the present invention does not limit its specific implementation method.

[0059] In a specific embodiment, the memory 4 can adopt a DDR3 memory external to the FPGA controller 2. The FPGA external DDR3 can be used as an image cache space. The pipeline mode is adopted to start the next image acquisition while the three-dimensional reconstruction calculation process is in progress, so as to further improve the efficiency of three-dimensional scanning and achieve real-time effect.

[0060] In a specific embodiment, the shooting device 5 includes a left camera and a right camera, and the FPGA controller 2 can simultaneously process the images taken by the left and right cameras. Due to the parallel computing characteristics of the FPGA controller 2, the present invention can simultaneously reconstruct the deformed grating images taken by the left and right cameras, thereby speeding up the three-dimensional reconstruction when the number of projected and collected images is too large.

[0061] In a specific embodiment, see Figure 2 The FPGA controller 2 includes a bus connection module 21, a bus adapter module 22, a storage control module 23, an HLS IP module 24 and an acquisition control module 25. The bus adapter module 22, the storage control module 23, the HLS IP module 24 and the acquisition control module 25 are all connected to the bus connection module 21, wherein:

[0062] The bus connection module 21 is used to realize communication with the processor 1, the projection device 3, the memory 4, the left camera and the right camera. The bus connection module 21 is an AXI bus connection module, which is implemented by the IP core (a logic block or a data block for a product application-specific integrated circuit or an editable logic device) provided by XILINXVIVADO.

[0063] The bus adapter module 22 is used to realize data interaction with the processor 1 and the user through the bus connection module 21. The bus adapter module 22 adopts a PCIE adapter module, which can realize data interaction communication with the processor, the sending of user control commands and parameters required for FPGA settlement, and the high-speed interface between the bus adapter module 22 and the storage control module 23, which can ensure the maximization of image acquisition speed, thereby improving the working rhythm of the entire device.

[0064] The storage control module 23 is used to implement data reading and writing operations on the memory 4 through the bus connection module 21 .

[0065] The HLS IP module 24 is used to convert the preset three-dimensional settlement algorithm implemented by the HLS language into a first hardware description language, so that the first hardware description language is calculated according to the multiple deformed grating images to obtain a depth map; the HLS IP module is used to realize the function of three-dimensional settlement, and is designed by HLS (High Level Synthesis) and developed by C / C++ language. The three-dimensional settlement algorithm implemented by C / C++ can be directly converted into an IP core described by RTL for direct use on FPGA, without in-depth consideration of how the hardware circuit is implemented, especially the conversion efficiency of loop statements is higher, and the designer does not need to consider how the loop statement is implemented in the hardware circuit. The HLS software will automatically convert the loop statement into the hardware circuit in the best and fastest mode, and the HLS development method is more efficient than the traditional FPGA HDL language development method, with a short development cycle, which is particularly suitable for the implementation of image processing, and the HLS data interface is relatively rich, and the designer does not need to consider how the specific interface is implemented in hardware. HLS will automatically convert according to the comprehensive strategy provided by the designer. In addition, the HLS IP module in the present invention can directly interact with the memory for data, without the need to additionally design the interface data module of the AXI bus. This can simplify development work, quickly carry out FPGA development, and ensure rapid product implementation.

[0066] The projection and acquisition control module 25 is used to convert the preset HDL language for realizing the projection and acquisition function into a second hardware description language, so that the second hardware description language can realize the preprocessing of the initial grating image stored in the memory, and drive the projection device to project the preprocessed initial grating image onto the object to be scanned, and after the projection device completes a grating projection, trigger the left camera and the right camera to take pictures at the same time. The projection and acquisition control module 25 adopts the RTL description realized by the VerilogHDL language, which is used to realize the timing control, so that the initial grating image can be projected in an orderly manner, and after each projection is completed, the trigger pulse of the left camera and the right camera is provided, and the left camera and the right camera are triggered to take pictures at the same time, so as to achieve the synchronization of projection and picture acquisition, achieve the maximum acquisition efficiency, and make each frame of grating projection without waiting time.

[0067] In a preferred embodiment, the second hardware description language is specifically used for:

[0068] The initial grating image stored in the memory is converted into video stream data, and the projection device is driven to project the video stream data. After the projection device projects a frame of grating, a trigger signal is sent to the left camera and the right camera, so that the left camera and the right camera simultaneously shoot the deformed grating image obtained by modulating the initial grating image through the object to be scanned.

[0069] Specifically, in this embodiment, multiple initial grating images can be projected in an orderly manner. First, the grating image data pre-stored in the memory 4 is read in order, and converted into standard video stream data output according to the HDMI timing requirements, so that the projection device can project the initial grating image in the correct order, and at the same time, after each grating projection is completed, an external trigger pulse for the left and right cameras is provided to trigger the left and right cameras to take pictures. For example, after the projection device 3 projects the first frame of grating, the left camera and the right camera are triggered to take pictures, and then the projection device 3 projects the second grating, and the left camera and the right camera are triggered to take pictures again, and so on. Figure 3 As shown, the trigger time of the camera of the present invention is fixed and controllable relative to the timing of projection, while the PC board cannot do this. The use of an FPGA controller can synchronize projection and image acquisition to achieve maximum acquisition efficiency, so that there is no waiting time between projections of each frame of grating.

[0070] In a preferred embodiment, the first hardware description language is specifically used for:

[0071] According to the internal and external parameters of the left camera and the right camera, the deformed grating image is subjected to phase adjustment and stereo matching to obtain a depth map.

[0072] Specifically, before reconstruction, the left camera and the right camera are first jointly calibrated to calculate the internal and external parameters of the left and right cameras of the shooting device, and then the rotation matrix and displacement vector between the left and right cameras are obtained. The rotation matrix and the displacement vector are then used to analyze the grating image for phase solution and stereo matching to obtain a depth map. The specific process of analyzing the phase solution and stereo matching is a prior art and will not be repeated here.

[0073] In a preferred embodiment, the method for the processor to complete the three-dimensional reconstruction of the object to be scanned according to the depth map is specifically:

[0074] After obtaining the three-dimensional point cloud of the object to be scanned according to the depth map, three-dimensional reconstruction of the object to be scanned is performed according to the three-dimensional point cloud.

[0075] In this embodiment, after the depth map is obtained, multiple groups of three-dimensional point clouds are obtained through the depth map, and then the multiple groups of three-dimensional point clouds are spliced ​​to achieve three-dimensional reconstruction of the object to be scanned.

[0076] The structured light 3D reconstruction system provided by the embodiment of the present invention has the following advantages:

[0077] The invention moves the calculation process and projection acquisition control into the FPGA controller, and the timing of projection acquisition control can be precisely controlled, maximizing the efficiency of image acquisition.

[0078] The decoding and 3D reconstruction process is realized through hardware acceleration. The calculation speed is much higher than that of PC, and the calculation time is fixed and not restricted by hardware.

[0079] The FPGA implementation can achieve simultaneous left and right acquisition without being affected by the interface bandwidth.

[0080] Embodiment 2

[0081] like Figure 4 As shown, the present invention also discloses a structured light 3D reconstruction method, which is applicable to the structured light 3D reconstruction device as described in the first embodiment, and comprises the following steps:

[0082] S1, the FPGA controller reads a plurality of initial grating images stored in the memory according to the grating image coding sequence, and sends a driving signal to the projection device;

[0083] S2, projecting the initial grating image onto the object to be scanned by the projection device;

[0084] S3, when the projection device completes the projection of an initial grating image, the FPGA controller sends a trigger signal to the shooting device;

[0085] S4, photographing, by the photographing device, a deformed grating image obtained by modulating the initial grating image through the object to be scanned;

[0086] S5, the processor collects the deformed grating image and sends the deformed grating image to the memory;

[0087] S6, repeat S2 to S5 until the processor has acquired all the deformed grating images;

[0088] S4, the FPGA controller reads the multiple deformed grating images in the memory, performs calculations based on the multiple deformed grating images to obtain a depth map, and sends the depth map to the memory;

[0089] S5. The processor reads the depth map in the memory, and completes three-dimensional reconstruction of the object to be scanned according to the depth map.

[0090] In a preferred embodiment, the shooting device includes a left camera and a right camera, and the FPGA controller converts a preset three-dimensional settlement algorithm implemented by the HLS language into a first hardware description language, and calculates the plurality of deformed grating images by using the first hardware description language to obtain a depth map;

[0091] The FPGA controller converts a preset HDL language for realizing the projection and acquisition function into a second hardware description language, uses the second hardware description language to preprocess the initial grating image stored in the memory, and drives the projection device to project the preprocessed initial grating image onto the object to be scanned, and triggers the left camera and the right camera to take pictures when the projection device completes a grating projection.

[0092] In a preferred embodiment, the second hardware description language is specifically used for:

[0093] The initial grating image stored in the memory is converted into video stream data, and the projection device is driven to project the video stream data. When the projection device completes projecting a frame of grating, a trigger signal is sent to the left camera and the right camera, so that the left camera and the right camera capture the deformed grating image obtained by modulating the initial grating image with the object to be scanned.

[0094] In a preferred embodiment, the first hardware description language is specifically used for:

[0095] According to the internal and external parameters of the left camera and the right camera, the deformed grating image is subjected to phase adjustment and stereo matching to obtain a depth map.

[0096] In a preferred embodiment, the step S8 specifically includes:

[0097] After obtaining the three-dimensional point cloud of the object to be scanned according to the depth map, three-dimensional reconstruction of the object to be scanned is performed according to the three-dimensional point cloud.

[0098] In the present invention, FPGA plays a connecting role in the whole system, connects various devices in the system together, controls the operation of each device, and ensures the orderly and correct operation of the whole system. By moving the calculation process and the projection acquisition control process into FPGA, the timing of the projection acquisition control can be precisely controlled, that is, the shooting device 4 can be triggered to take pictures after each grating projection is completed. At the same time, the characteristics of FPGA parallel calculation can ensure a more efficient realization of a single three-dimensional reconstruction settlement. The characteristics of FPGA pipeline work can shorten the waiting time between two three-dimensional reconstructions, and the next work can be started without waiting for the previous work to be completed, thereby maximizing the efficiency of image acquisition, which is not restricted by the performance of the PC processor. Moreover, when the number of projected and collected images is large, due to the characteristics of the parallel calculation of the FPGA controller, the reconstruction processing of the deformed grating images taken by the left and right cameras can be carried out at the same time, and the left and right cameras can be collected at the same time, which is not affected by the interface bandwidth and will not restrict the speed of image acquisition.

[0099] It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein may be combined with each other.

[0100] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0101] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A structured light 3D reconstruction device, characterized in that: include: A processor, an FPGA controller, a projection device, a memory, a shooting device and a serial high-speed interface, wherein the FPGA controller is connected to the memory, and the FPGA controller is connected to the serial high-speed interface, wherein: The FPGA controller is used to read the multiple initial grating images stored in the memory according to the grating image coding sequence, and send a driving signal to the projection device; The projection device is used to sequentially project the plurality of initial grating images onto the object to be scanned; The FPGA controller is further configured to output a trigger signal to the shooting device when the projection device completes the projection of a pair of the initial grating images; The photographing device is used to photograph a deformed grating image obtained by modulating the initial grating image with the object to be scanned when receiving the trigger signal; The processor is used to collect the deformed grating image and send the deformed grating image to the memory through the serial high-speed interface; The FPGA controller is further configured to read a plurality of the deformed grating images in the memory after the processor has acquired all the deformed grating images, perform calculations based on the plurality of the deformed grating images to obtain a depth map, and send the depth map to the memory; The processor is further configured to read the depth map in the memory and complete three-dimensional reconstruction of the object to be scanned according to the depth map; The shooting device includes a left camera and a right camera; The FPGA controller comprises: A bus connection module, used to realize communication with the processor, the projection device, the memory, the left camera and the right camera; A bus adapter module, used to implement data interaction with the processor and the user through the bus connection module; A storage control module, used to implement data reading and writing operations on the memory through the bus connection module; An HLS IP module is used to convert a preset three-dimensional settlement algorithm implemented by the HLS language into a first hardware description language, so that the first hardware description language performs calculation according to the plurality of deformed grating images to obtain a depth map; A projection and acquisition control module, used for converting a preset HDL language for realizing the projection and acquisition function into a second hardware description language, so that the second hardware description language can realize preprocessing of the initial grating image stored in the memory, and drive the projection device to project the preprocessed initial grating image onto the object to be scanned, and after the projection device completes one grating projection, trigger the left camera and the right camera to take pictures at the same time; The second hardware description language is specifically used to: convert the initial grating image stored in the memory into video stream data, and drive the projection device to project the video stream data, and after the projection device projects a frame of grating, send a trigger signal to the left camera and the right camera, so that the left camera and the right camera simultaneously shoot the deformed grating image obtained by modulating the initial grating image through the object to be scanned; The first hardware description language is specifically used to: perform phase adjustment and stereo matching on the deformed grating image according to the internal and external parameters of the left camera and the right camera to obtain a depth map.

2. The structured light 3D reconstruction device according to claim 1, characterized in that: The method for the processor to complete the three-dimensional reconstruction of the object to be scanned according to the depth map is specifically: after obtaining the three-dimensional point cloud of the object to be scanned according to the depth map, the three-dimensional reconstruction of the object to be scanned is performed according to the three-dimensional point cloud.

3. A structured light 3D reconstruction method, characterized in that: The structured light 3D reconstruction device according to any one of claims 1 to 2 comprises the following steps: S1, the FPGA controller reads a plurality of initial grating images stored in the memory according to the grating image coding sequence, and sends a driving signal to the projection device; S2, projecting the initial grating image onto the object to be scanned by the projection device; S3, when the projection device completes the projection of an initial grating image, the FPGA controller sends a trigger signal to the shooting device; S4, photographing, by the photographing device, a deformed grating image obtained by modulating the initial grating image through the object to be scanned; S5, the processor simultaneously collects the deformed grating image, and sends the deformed grating image to the memory through the serial high-speed interface; S6, repeat S2 to S5 until the processor has acquired all the deformed grating images; S7, the FPGA controller reads the multiple deformed grating images in the memory, performs calculations based on the multiple deformed grating images to obtain a depth map, and sends the depth map to the memory; S8. The processor reads the depth map in the memory, and completes three-dimensional reconstruction of the object to be scanned according to the depth map.

4. The structured light 3D reconstruction method according to claim 3, characterized in that: The shooting device includes a left camera and a right camera, and the FPGA controller converts a preset three-dimensional settlement algorithm implemented by the HLS language into a first hardware description language, and calculates the plurality of deformed grating images by using the first hardware description language to obtain a depth map; The FPGA controller converts a preset HDL language for realizing the projection and acquisition function into a second hardware description language, uses the second hardware description language to preprocess the initial grating image stored in the memory, and drives the projection device to project the preprocessed initial grating image onto the object to be scanned, and triggers the left camera and the right camera to take pictures when the projection device completes a grating projection.

5. The structured light 3D reconstruction method according to claim 4, characterized in that: The second hardware description language is specifically used for: The initial grating image stored in the memory is converted into video stream data, and the projection device is driven to project the video stream data. When the projection device completes projecting a frame of grating, a trigger signal is sent to the left camera and the right camera, so that the left camera and the right camera capture the deformed grating image obtained by modulating the initial grating image with the object to be scanned.

6. The structured light 3D reconstruction method according to claim 4, characterized in that: The first hardware description language is specifically used for: According to the internal and external parameters of the left camera and the right camera, the deformed grating image is subjected to phase adjustment and stereo matching to obtain a depth map.

7. The structured light 3D reconstruction method according to claim 3, characterized in that: The step S8 specifically includes: after obtaining a three-dimensional point cloud of the object to be scanned according to the depth map, three-dimensional reconstruction of the object to be scanned according to the three-dimensional point cloud.

Images