Structured light 3D camera system and measurement method based on FPGA heterogeneous computing
The structured light 3D camera system using FPGA heterogeneous computing utilizes a hardware-accelerated processing pipeline to generate phase maps, solving the problems of insufficient real-time computation and dynamic scene adaptability in existing 3D reconstruction technologies, and achieving efficient 3D reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-03-31
AI Technical Summary
Existing structured light 3D reconstruction technology suffers from frequent decoding errors under complex lighting conditions, has high computational complexity, and struggles to meet real-time and low-power requirements. Furthermore, it lacks adaptability to dynamic scenes, limiting its application in industrial automation and large-scale scenarios.
A structured light 3D camera system based on FPGA heterogeneous computing is adopted. Through the combination of projection module, acquisition module and processing chip module, the hardware acceleration processing pipeline in the FPGA programmable logic unit is used to perform pipeline processing to generate phase map of target object, and send it to external host via Ethernet to perform 3D modeling.
It realizes pipelined processing of data acquisition and computation, reduces the time consumption of 3D reconstruction, improves the real-time performance of computation and adaptability to dynamic scenes, and solves the real-time performance problem in existing technologies.
Smart Images
Figure CN121010493B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a structured light 3D camera system and measurement method based on FPGA heterogeneous computing. Background Technology
[0002] Structured light 3D reconstruction technology obtains 3D information about an object's surface by projecting encoded patterns and analyzing their deformation. It is widely used in fields such as industrial inspection, medical imaging, and virtual reality. Traditional decoding methods combining Gray code and phase shift rely on multi-frame pattern projection and pixel-level processing, which can provide high-resolution phase information.
[0003] However, existing 3D reconstruction schemes are prone to decoding errors under complex lighting conditions (such as strong light, shadows, or reflective surfaces), and their high computational complexity makes it difficult to meet real-time and low-power requirements. Furthermore, existing schemes lack adaptability to dynamic scenes and support for multi-device collaborative decoding, limiting their application in industrial automation and large-scale scenarios.
[0004] Currently, no effective solution has been proposed for improving the real-time performance of 3D reconstruction calculations in related technologies. Summary of the Invention
[0005] This application provides a structured light 3D camera system and measurement method based on FPGA heterogeneous computing, in order to at least solve the problem of how to improve the real-time performance of 3D reconstruction computation in related technologies.
[0006] In a first aspect, embodiments of this application provide a structured light 3D camera system based on FPGA heterogeneous computing. The system includes a projection module, a data acquisition module, and a processing chip module, wherein the processing chip module includes an FPGA programmable logic unit.
[0007] The projection module is used to project a structured light coded pattern sequence containing multiple frames of patterns;
[0008] The acquisition module is used to acquire the structured light coded pattern sequence modulated by the target object and output it as a structured light coded pattern sequence in a preset format.
[0009] The processing chip module is used to perform pipelined processing on the structured light coded pattern sequence through the hardware acceleration processing pipelines in the FPGA programmable logic unit, generate a phase map of the target object, and send the phase map to an external host via Ethernet to perform 3D modeling of the target object.
[0010] In some embodiments, the hardware acceleration processing pipeline in the FPGA programmable logic unit is an image denoising pipeline, an ambient light calibration pipeline, a distortion calibration pipeline, an epipolar calibration pipeline, a local binarization pipeline, and a phase temporal decoding pipeline.
[0011] Each hardware acceleration pipeline works in conjunction with its respective algorithm core and buffer to perform pipelined processing of the structured light coded pattern sequence.
[0012] In some embodiments, the FPGA programmable logic unit includes an extraction interpolation subunit;
[0013] The projection module is used to project a sequence of structured light coded patterns according to a preset time sequence. Each set of structured light coded pattern sequences includes one full-bright pattern, one full-dark pattern, multiple Gray code patterns, and multiple phase-shifting grating patterns.
[0014] The acquisition module is used to acquire the structured light coded pattern sequence modulated by the target object and output it as a structured light coded pattern sequence in Bayer format.
[0015] Before the structured light encoded pattern sequence enters each hardware acceleration processing pipeline in the FPGA programmable logic unit, it first enters the extraction and interpolation subunit: the extraction and interpolation subunit is used to separate and extract the G1 and G2 green channels in the Bayer array of the structured light encoded pattern sequence, generate a grayscale image of a preset resolution by bilinear interpolation, and use the grayscale image to perform ambient light elimination processing on the structured light encoded pattern sequence.
[0016] In some embodiments, the FPGA programmable logic unit includes:
[0017] The image denoising pipeline is used to run the first algorithm kernel to obtain the structured light coded pattern sequence after ambient light cancellation processing, so as to denoise the pattern in the structured light coded pattern sequence through a filter window of a preset size, and store the denoising processing result in the first buffer area;
[0018] The ambient light calibration pipeline is used to run the second algorithm core to read the denoising processing result from the first buffer, so as to use the full-bright pattern and the full-dark pattern to perform ambient light calibration on the multi-frame Gray code pattern, and store the ambient light calibration result in the second buffer.
[0019] The distortion calibration pipeline is used to run a third algorithm core to read the ambient light calibration result from the second buffer, to perform distortion correction on the ambient light calibrated pattern through a first difference lookup table (LUT), and to store the distortion correction result in the third buffer. The first difference lookup table (LUT) is generated based on pre-calibrated camera intrinsic parameters.
[0020] The epipolar calibration pipeline is used to run a fourth algorithm core to read the distortion calibration result from the third buffer, perform epipolar calibration on the distorted pattern through a second difference lookup table (LUT), and store the epipolar calibration result in the fourth buffer. The second difference lookup table (LUT) is generated based on pre-calibrated camera extrinsic parameters.
[0021] In some embodiments, the FPGA programmable logic unit includes:
[0022] The local binarization pipeline is used to run the fifth algorithm core to read the epipolar calibration result from the fourth buffer, to segment the epipolar calibrated Gray code pattern into multiple image blocks, and to calculate the dynamic threshold based on the local statistical characteristics of each image block to complete the binarization of the Gray code pattern, and store the binarization result in the fourth buffer.
[0023] In some embodiments, the FPGA programmable logic unit includes:
[0024] The phase-time decoding pipeline is used to run the sixth algorithm core to read the binarization result from the fifth buffer, and to decode the binarized Gray code pattern and phase-shift grating pattern respectively to obtain coarse-grained phase information and fine-grained phase information. The coarse-grained phase information and fine-grained phase information are fused to generate a phase map representing the absolute phase, and the phase map is stored in the sixth buffer.
[0025] In some embodiments, the acquisition module includes a first acquisition unit and a second acquisition unit, which cooperate to acquire a complete structured light coded pattern;
[0026] The patterns acquired by the first and second acquisition units are divided into two regions respectively; the algorithm cores of each hardware acceleration processing pipeline in the FPGA programmable logic unit adopt a quad-core parallel architecture, and each parallel core processes the divided quarter pattern.
[0027] In some embodiments, the processing chip module further includes an ARM processor unit;
[0028] The ARM processor unit is used to schedule image processing tasks and control the hardware acceleration processing pipeline in the FPGA programmable logic unit, and send the processing results to an external host via Ethernet.
[0029] In some embodiments, the FPGA programmable logic unit further includes a data acquisition synchronization subunit;
[0030] The acquisition synchronization subunit is used to generate a hardware trigger signal to control the projection module and the acquisition module to perform synchronized exposure:
[0031] The projection module responds to the hardware trigger signal and synchronously projects a structured light coded pattern sequence; the acquisition module responds to the hardware trigger signal and synchronously acquires a structured light coded pattern sequence modulated by the target object, and outputs it as a structured light coded pattern sequence in a preset format.
[0032] Secondly, embodiments of this application provide a structured light 3D measurement method based on FPGA heterogeneous computing. The method is executed based on the system described in the first aspect of the claim, and the method includes:
[0033] Projecting a structured light coded pattern sequence containing multiple frames of patterns;
[0034] Acquire a structured light coded pattern sequence modulated by the target object and output it as a structured light coded pattern sequence in a preset format;
[0035] The structured light coded pattern sequence is processed in a pipeline using the hardware acceleration pipelines in the FPGA programmable logic unit to generate a phase map of the target object. The phase map is then sent to an external host via Ethernet to perform 3D modeling of the target object.
[0036] Compared to related technologies, this application provides a structured light 3D camera system and measurement method based on FPGA heterogeneous computing. The system includes: a projection module for projecting a sequence of structured light coded patterns containing multiple frames; an acquisition module for acquiring the structured light coded pattern sequence modulated by a target object and outputting it as a structured light coded pattern sequence in a preset format; and a processing chip module for pipelined processing of the structured light coded pattern sequence through various hardware-accelerated processing pipelines in the FPGA programmable logic unit, generating a phase map of the target object, and sending the phase map to an external host via Ethernet to perform 3D modeling of the target object. This achieves pipelined processing of the structured light coded pattern based on FPGA hardware-accelerated processing pipelines, enabling data acquisition and calculations to form a pipeline, i.e., data acquisition and various calculations are executed in parallel, greatly reducing the time consumption of 3D reconstruction. Furthermore, edge computing is implemented on the FPGA of the camera system to directly output the phase map, exhibiting high adaptability to dynamic scenes and solving the problem of how to improve the real-time performance of 3D reconstruction calculations. Attached Figure Description
[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0038] Figure 1 This is a schematic diagram of a structured light 3D camera system based on FPGA heterogeneous computing according to an embodiment of this application;
[0039] Figure 2 This is a schematic diagram of the FPGA pipeline processing according to an embodiment of this application;
[0040] Figure 3 This is a comparative schematic diagram of FPGA pipeline flow processing and non-flow processing according to embodiments of this application;
[0041] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.
[0043] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.
[0044] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0045] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.
[0046] This application provides a structured light 3D camera system based on FPGA heterogeneous computing. Figure 1 This is a schematic diagram of a structured light 3D camera system based on FPGA heterogeneous computing according to an embodiment of this application, as shown below. Figure 1 As shown, the system includes a projection module, a data acquisition module, and a processing chip module, wherein the processing chip module includes an FPGA programmable logic unit and an ARM processor unit.
[0047] The projection module is used to project a sequence of structured light coded patterns containing multiple frames.
[0048] Specifically, the projection module is used to project a sequence of structured light coded patterns according to a preset time sequence, wherein each set of structured light coded pattern sequences includes one full-bright pattern, one full-dark pattern, multiple Gray code patterns, and multiple phase-shifting grating patterns.
[0049] Preferably, each set of structured light coded pattern sequences includes one full-bright pattern, one full-dark pattern, six Gray code patterns, and four phase-shifting grating patterns, or includes one full-bright pattern, one full-dark pattern, eight Gray code patterns, and three phase-shifting grating patterns. Preferably, as... Figure 1 As shown, the projection module is preferably a DLP projector (1024x720, 150Hz); alternatively, the projection module can be a laser scanning projector (low power consumption but also low resolution).
[0050] The acquisition module is used to acquire the structured light coded pattern sequence modulated by the target object and output it as a structured light coded pattern sequence in a preset format.
[0051] Specifically, the acquisition module is used to acquire the structured light coded pattern sequence modulated by the target object and output it as a Bayer format structured light coded pattern sequence; wherein, the acquisition module includes a first acquisition unit and a second acquisition unit, the first acquisition unit and the second acquisition unit cooperate to acquire the complete structured light coded pattern;
[0052] Preferably, such as Figure 1 As shown, the first acquisition unit and the second acquisition unit are preferably two RGB cameras (2048x1024, 100fps) that output Bayer RGGB format images and capture green Gray code patterns.
[0053] The processing chip module is used to pipeline the structured light encoded pattern sequence through the hardware acceleration processing pipelines in the FPGA programmable logic unit to generate the phase map of the target object. The ARM processor unit schedules the image processing tasks and controls the hardware acceleration processing pipelines in the FPGA programmable logic unit, and sends the processing results to an external host via Ethernet to perform 3D modeling of the target object.
[0054] It should be noted that, as Figure 1 As shown, the hardware acceleration processing pipeline in the FPGA programmable logic unit consists of an image denoising pipeline, an ambient light calibration pipeline, a distortion calibration pipeline, an epipolar calibration pipeline, a local binarization pipeline, and a phase temporal decoding pipeline. Each hardware acceleration processing pipeline works in conjunction with its respective algorithm core and buffer to perform pipelined processing of the structured light coded pattern sequence.
[0055] It should be further noted that the processing chip module is preferably based on the Xilinx Zynq UltraScale+ MPSoC architecture. This chip integrates four ARM processor units and an FPGA programmable logic unit. The FPGA programmable logic unit can run algorithm cores, which read and write corresponding caches via the AXI-LITE bus. The cache and algorithm cores work together to form a hardware-accelerated processing pipeline. The ARM processor units can control these algorithm cores to complete image processing tasks via interrupts and the AXI-LITE bus. Figure 1As shown, the ARM processor unit is defined as the PS (Power Supply) section of the chip, and the FPGA (Programmable Logic Unit) is defined as the PL (Programmable Logic Unit) section. The ARM processor unit (PS section) integrates necessary hardware devices, such as a DDR4 controller, an Ethernet controller, and I2C / UART / SPI controllers. The FPGA (PL section) section uses logic resources to implement the DDR4 controller. For Ethernet, such as... Figure 1 As shown, the ARM processor unit sends the two phase images, decoded in the temporal domain, to an external host via Ethernet. The host, upon receiving the phase images from the two cameras, performs 3D modeling of the target object (such as various parallax matching algorithms and post-processing). For DDR4 memory, such as... Figure 1 As shown, DDR4 here includes both the PS's DDR4 memory and the PL's DDR4 memory. Most hardware acceleration processing pipelines use the PL's DDR4 memory. Only the "phase time domain decoding pipeline" outputs to the PS's DDR4 memory to avoid the algorithm core in the PL frequently accessing the OCM, which would lead to a decrease in the efficiency of ARM processing.
[0056] Through the projection module, acquisition module, and processing chip module in this embodiment, pipelined processing of structured light encoded patterns based on FPGA hardware acceleration is realized, enabling data acquisition and calculation of each pipeline to form a pipeline, that is, data acquisition and various calculations are executed in parallel, which greatly reduces the time consumption of 3D reconstruction. Furthermore, edge computing is implemented on the FPGA of the camera system to directly output phase maps, which has high dynamic scene adaptability and solves the problem of how to improve the real-time performance of 3D reconstruction calculation.
[0057] In some of these embodiments, such as Figure 1 As shown, the FPGA programmable logic unit includes an acquisition synchronization subunit and an extraction interpolation subunit;
[0058] The acquisition synchronization subunit is used to generate a hardware trigger signal to control the projection module and the acquisition module to perform synchronous exposure: the projection module responds to the hardware trigger signal and synchronously projects the structured light coded pattern sequence; the acquisition module responds to the hardware trigger signal and synchronously acquires the structured light coded pattern sequence modulated by the target object, and outputs the structured light coded pattern sequence in a preset format.
[0059] Before the structured light encoded pattern sequence enters each hardware acceleration processing pipeline in the FPGA programmable logic unit, it first enters the extraction and interpolation subunit: the extraction and interpolation subunit is used to separate and extract the G1 and G2 green channels in the Bayer array of the structured light encoded pattern sequence, generate a grayscale image of a preset resolution (2048×1024 resolution) by bilinear interpolation, and use the grayscale image to perform ambient light elimination processing on the structured light encoded pattern sequence.
[0060] In some of these embodiments, Figure 2 This is a schematic diagram of the FPGA pipeline processing according to an embodiment of this application, such as... Figure 2 As shown:
[0061] The image denoising pipeline is used to run the first algorithm kernel to obtain the structured light coded pattern sequence after ambient light cancellation processing, so as to denoise the pattern in the structured light coded pattern sequence through a filter window of a preset size (such as 3*3), and store the denoising result in the first buffer area.
[0062] An ambient light calibration pipeline is used to run a second algorithm core to read the denoising processing results from the first buffer, to perform ambient light calibration on multiple frames of Gray code patterns using full-bright and full-dark patterns, and to store the ambient light calibration results in the second buffer; preferably, the ambient light calibration formula is Corrected_img = (Code_img - Black_img) / (White_img - Black_img).
[0063] The distortion calibration pipeline is used to run the third algorithm core to read the ambient light calibration results from the second buffer, perform distortion correction on the ambient light calibrated pattern through the first difference lookup table (LUT), and store the distortion correction results in the third buffer. The first difference lookup table (LUT) is generated based on the pre-calibrated camera intrinsic parameters.
[0064] The epipolar calibration pipeline is used to run the fourth algorithm core to read the distortion calibration results from the third buffer, perform epipolar calibration on the distortion-calibrated pattern through the second difference lookup table (LUT), and store the epipolar calibration results in the fourth buffer. The second difference lookup table (LUT) is generated based on pre-calibrated camera extrinsic parameters.
[0065] The local binarization pipeline is used to run the fifth algorithm kernel to read the epipolar calibration results from the fourth buffer, segment the epipolar-calibrated Gray code pattern into multiple image blocks, and calculate a dynamic threshold based on the local statistical characteristics of each image block to complete the binarization of the Gray code pattern. The binarization result is then stored in the fourth buffer. Preferably, the Gray code pattern is segmented into several image blocks, each with a pixel size of 64x64. First, the local threshold of each image block is calculated, and then the image blocks are binarized. The dynamic threshold calculation formula is: T_local = μ + k * σ, where μ is the pixel mean within the block, σ is the standard deviation, and k is an adjustable coefficient.
[0066] It should be noted that each hardware-accelerated processing pipeline works in conjunction with its respective algorithm core and buffer to achieve pipelined processing of the light-encoded pattern sequence. In other words, the processing of each pipeline is not blocking. For example, if there is complete data to be processed in the "first buffer," then the "ambient light calibration algorithm core" is started to perform calculations. After the "ambient light calibration algorithm core" completes its calculations, an interrupt is sent (e.g., ...). Figure 2 As shown, this interrupt can be bound to the Arm Cortex-R5F, with these two CPUs dedicated to handling real-time tasks; while the two Arm Cortex-A53 CPUs run the Linux operating system to handle non-real-time tasks (such as Ethernet network transmission). At this time, the CPU identifies the "ambient light calibration cache," and after identification, the "distortion correction algorithm core" can be started for calculation. Meanwhile, the image denoising module can still write data to the "raw image cache." This is the pipeline process of the hardware acceleration processing pipeline. Figure 3 This is a comparative schematic diagram of FPGA pipeline flow processing and non-flow processing according to embodiments of this application, such as... Figure 3 As shown, the pipeline-based computation significantly improves system efficiency because data acquisition can occur while the algorithm core is performing calculations. Once all data acquisition is complete, the algorithm core has finished its computational task and can proceed to the next calculation. In other words, because the structured light 3D camera system needs to acquire a large sequence of images at a time, such as... Figure 3 As shown, by pipelined computation and data acquisition, the shortest processing time of the system becomes the acquisition time. Without pipelined processing, the shortest processing time of the system is "acquisition + algorithm computation".
[0067] In some embodiments, the FPGA programmable logic unit includes:
[0068] like Figure 2 As shown, the phase-temporal decoding pipeline is used to run the sixth algorithm core to read the binarization result from the fifth buffer, and to decode the binarized Gray code pattern and phase-shift grating pattern respectively to obtain coarse-grained phase information and fine-grained phase information. The coarse-grained phase information and fine-grained phase information are fused to generate a phase map representing the absolute phase, and the phase map is stored in the sixth buffer.
[0069] It should be noted that the phase-temporal decoding pipeline is the core hardware acceleration processing pipeline of this system, integrated into the FPGA programmable logic unit (PL part of the chip), and is specifically designed for processing structured light coded pattern sequences. This pipeline combines the coarse-grained phase range provided by the Gray code sequence with the fine phase information of the phase-shifting grating sequence through an innovative fusion algorithm, achieving the generation of a high-precision absolute phase map, thereby providing sub-pixel-level depth data for subsequent 3D reconstruction. The specific working principle is as follows:
[0070] The input data comes from the binarized Gray code pattern (6 frames) and phase-shifting grating pattern (4 frames) processed by the upstream pipeline. After local adaptive binarization, the Gray code sequence forms a binary code used to coarsely divide the phase period, avoiding the periodic ambiguity problem of the phase-shifting method. The phase-shifting method calculates the phase shift of each pixel based on the deformation of the sinusoidal grating pattern. The decoding process consists of three stages:
[0071] The coarse phase extraction stage involves pixel-by-pixel decoding of the binarized Gray code image to generate coarse phase codewords (Gray_code). A dynamic codeword verification mechanism is introduced, which detects and corrects decoding errors (such as jumps caused by shadows) by comparing the Gray code continuity of adjacent pixels. The verification rule is: if |Gray_code(i,j) - Gray_code(i,j+1)| > 1, then a mean square filter is applied to correct for the neighborhood average. This innovative design improves decoding accuracy on complex surfaces (such as highly reflective or unevenly textured surfaces), with experiments showing a 15%-20% reduction in error rate.
[0072] Fine phase calculation stage: Apply an N-step phase shift algorithm (e.g., N=4) to the four-frame phase-shifting grating pattern to calculate the wrapping phase. The specific algorithm formula is as follows: Where I1 to I4 are the phase-shifted image intensity values. Furthermore, to enhance robustness, adaptive weight adjustment is incorporated: the phase-shift weight w_k = 1 / (σ_k + ε) is dynamically adjusted based on the noise estimate σ from the upstream ambient light calibration, where ε is a small constant (to prevent division by zero). This innovative design ensures sub-pixel accuracy (<0.1 pixel error) is maintained even in low signal-to-noise ratio environments (such as indoor strong light interference).
[0073] Absolute phase fusion stage: Combining coarse and fine phases to generate absolute phase. Where N is the Gray code bit depth (in this example, N=6, providing 64 period divisions). For further optimization, a machine learning-assisted phase unfolding network is introduced: a lightweight neural network module (a multilayer perceptron implemented using a LUT, with <10K parameters) is embedded within the FPGA, pre-trained on a simulated dataset to predict potential blurred boundaries and automatically correct them. Furthermore, this neural network module interacts with the chip's PS section via the AXI-LITE interface, allowing for online fine-tuning, suitable for dynamic scenarios such as robot vision tracking.
[0074] In some of these embodiments, such as Figure 2 As shown, the patterns acquired by the first acquisition unit (RGB camera) and the second acquisition unit (RGB camera) are divided into two regions respectively; the algorithm cores of each hardware acceleration processing pipeline in the FPGA programmable logic unit adopt a quad-core parallel architecture, and each parallel core processes the divided quarter pattern.
[0075] It should be noted that the pipeline employs a quad-core parallel architecture (two Arm Cortex-R5F cores and two Arm Cortex-A53 cores): each camera image is segmented into two regions (top / bottom or left / right), each assigned an independent algorithm core (an IP core implemented in Verilog), and accesses the DDR4 memory of the chip's PL section in parallel via the AXI bus. Each core processes 1 / 4 of the image, with a computation time of <20ms / frame, supporting an overall frame rate of 8fps. The data flow is non-blocking; for example, an interrupt signal from the upstream binarization pipeline triggers decoding, while the downstream can prepare the output buffer in parallel. Furthermore, a load balancing mechanism dynamically allocates region sizes and adjusts the segmentation lines according to image complexity, ensuring that the load difference between cores is <5%.
[0076] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.
[0077] This application provides a structured light 3D measurement method based on FPGA heterogeneous computing. The method is executed based on the system described in the first aspect of the claim, and includes:
[0078] Projecting a structured light coded pattern sequence containing multiple frames of patterns;
[0079] Acquire a structured light coded pattern sequence modulated by the target object and output it as a structured light coded pattern sequence in a preset format;
[0080] The structured light encoded pattern sequence is processed in a pipeline through the hardware acceleration pipelines in the FPGA programmable logic unit to generate a phase map of the target object. The phase map is then sent to an external host via Ethernet to perform 3D modeling of the target object.
[0081] The method steps in this application embodiment realize the pipelined processing of structured light coding patterns based on FPGA hardware acceleration processing pipeline, so that data acquisition and calculation of each pipeline can form a pipeline, that is, data acquisition and various calculations are executed in parallel, which greatly reduces the time consumption of 3D reconstruction. Moreover, the edge computing is realized on the FPGA of the camera system to directly output the phase map, which has high dynamic scene adaptability and solves the problem of how to improve the real-time performance of 3D reconstruction calculation.
[0082] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0083] This embodiment provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0084] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0085] Optionally, the electronic device may further include a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a structured light 3D measurement method based on FPGA heterogeneous computing. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.
[0086] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.
[0087] Furthermore, in conjunction with the structured light 3D measurement method based on FPGA heterogeneous computing in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the structured light 3D measurement methods based on FPGA heterogeneous computing in the above embodiments.
[0088] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operating system and computer programs to run, the computer programs are executed by the processor to implement a structured light 3D measurement method based on FPGA heterogeneous computing, and the database stores data.
[0089] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0090] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0091] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0092] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A structured light 3D camera system based on FPGA heterogeneous computing, characterized in that, The system comprises a projection module, a collection module and a processing chip module, wherein the processing chip module comprises an FPGA programmable logic unit, and the FPGA programmable logic unit comprises an extraction and interpolation subunit; The projection module is configured to project a structured light coding pattern sequence according to a preset timing, wherein each group of structured light coding pattern sequence comprises one frame of full-bright pattern, one frame of full-dark pattern, multiple frames of gray code pattern and multiple frames of phase shift grating pattern; The collection module is configured to collect the structured light coding pattern sequence modulated by the target object and output the structured light coding pattern sequence in Bayer format; Before the structured light coding pattern sequence enters each hardware acceleration processing pipeline in the FPGA programmable logic unit, the structured light coding pattern sequence first enters the extraction and interpolation subunit, which is configured to separate and extract G1 and G2 green channels in the Bayer array of the structured light coding pattern sequence, generate a gray scale image with a preset resolution by using a bilinear interpolation method, and perform ambient light elimination processing on the structured light coding pattern sequence by using the gray scale image; The processing chip module is configured to perform pipeline processing on the structured light coding pattern sequence by using each hardware acceleration processing pipeline in the FPGA programmable logic unit, generate a phase map of the target object, and send the phase map to an external host through Ethernet to perform three-dimensional modeling of the target object. The hardware acceleration processing pipelines in the FPGA programmable logic unit are image denoising pipeline, ambient light calibration pipeline, distortion calibration pipeline, polar line calibration pipeline, local binarization pipeline and phase time domain decoding pipeline; each hardware acceleration processing pipeline cooperates between its own algorithm kernel and cache area to perform pipeline processing on the structured light coding pattern sequence.
2. The system of claim 1, wherein, The FPGA programmable logic unit comprises: The image denoising pipeline is configured to run a first algorithm kernel to obtain the structured light coding pattern sequence after ambient light elimination processing, to perform denoising processing on the patterns in the structured light coding pattern sequence by using a filter window with a preset size, and to store the denoising processing result in a first cache area; The ambient light calibration pipeline is configured to run a second algorithm kernel to read the denoising processing result from the first cache area, to perform ambient light calibration on the multiple frames of gray code pattern by using the full-bright pattern and the full-dark pattern, and to store the ambient light calibration result in a second cache area; The distortion calibration pipeline is configured to run a third algorithm kernel to read the ambient light calibration result from the second cache area, to perform distortion correction on the ambient light calibrated pattern by using a first differential lookup table (LUT), and to store the distortion correction result in a third cache area, wherein the first differential LUT is generated based on pre-calibrated camera intrinsic parameters; The polar line calibration pipeline is configured to run a fourth algorithm kernel to read the distortion calibration result from the third cache area, to perform polar line calibration on the distortion calibrated pattern by using a second differential LUT, and to store the polar line calibration result in a fourth cache area, wherein the second differential LUT is generated based on pre-calibrated camera extrinsic parameters.
3. The system of claim 2, wherein, The FPGA programmable logic unit comprises: The local binarization pipeline is configured to run a fifth algorithm kernel to read the epipolar rectification result from the fourth cache area, to split the epipolar rectified Gray code pattern into a plurality of image blocks, to calculate a dynamic threshold based on local statistical characteristics of each image block, to complete binarization of the Gray code pattern, and to store the binarization result into the fifth cache area.
4. The system of claim 3, wherein, The FPGA programmable logic unit comprises: The phase time domain decoding pipeline is configured to run a sixth algorithm kernel to read the binarization result from the fifth cache area, to respectively decode the binarized Gray code pattern and the phase shift grating pattern, to correspondingly obtain coarse-grained phase information and fine-grained phase information, to fuse the coarse-grained phase information and the fine-grained phase information, to generate a phase map representing absolute phase, and to store the phase map into the sixth cache area.
5. The system of claim 1, wherein, The acquisition module comprises a first acquisition unit and a second acquisition unit, which cooperate to acquire a complete structured light encoding pattern; For the partial patterns respectively acquired by the first acquisition unit and the second acquisition unit, the partial patterns are respectively split into two regions; algorithm kernels of each hardware acceleration processing pipeline in the FPGA programmable logic unit adopt a four-core parallel architecture, and each parallel core processes a quarter of the split pattern.
6. The system of claim 1, wherein, The processing chip module further comprises an ARM processor unit; The ARM processor unit is configured to schedule image processing tasks and control the hardware acceleration processing pipelines in the FPGA programmable logic unit, and to send processing results to an external host through Ethernet.
7. The system of claim 1, wherein, The FPGA programmable logic unit further comprises an acquisition synchronization subunit; The acquisition synchronization subunit is configured to generate a hardware trigger signal to control the projection module and the acquisition module to perform synchronous exposure: The projection module projects a sequence of structured light encoding patterns in response to the hardware trigger signal; and the acquisition module acquires a sequence of structured light encoding patterns modulated by a target object in response to the hardware trigger signal, and outputs the sequence of structured light encoding patterns in a preset format.
8. A structured light 3D measurement method based on FPGA heterogeneous computing, characterized in that, The method is executed based on the system according to any one of claims 1 to 7, and the method comprises: projecting a sequence of structured light encoding patterns comprising a plurality of patterns; acquiring a sequence of structured light encoding patterns modulated by a target object, and outputting the sequence of structured light encoding patterns in a preset format; performing pipeline processing on the sequence of structured light encoding patterns by each hardware acceleration processing pipeline in the FPGA programmable logic unit, generating a phase map of the target object, and sending the phase map to an external host through Ethernet to perform three-dimensional modeling of the target object.