An FPGA-based method for optimizing and accelerating structured light algorithms

Abstract

This invention discloses an FPGA-based method for optimizing and accelerating structured light algorithms. The method includes: a processor receiving image data acquired by a camera and sending the image data to the FPGA; using Gray code projection-based structured light imaging technology, coarsely locating the position of each camera pixel on the optical engine using Gray code, and then finely locating the position of the pixel on the optical engine within each Gray code period using moving fringe technology to reconstruct three-dimensional topographic information; image preprocessing, HDR fusion of multiple exposure images, and Gray code decoding are completed on the FPGA side, while point cloud reconstruction is completed on the PC side; by utilizing parallel processing and a pipelined approach on the FPGA, image preprocessing, HDR fusion, Gray code generation, and moving edge fringe subtraction are completed in real time while data is sent to the PC side; Gray code decoding is performed, and moving fringe decoding is performed simultaneously, and the two are combined to obtain the final phase data; the phase data is then uploaded to a host computer.

Description

Technical Field

[0001] This invention relates to the field of FPGA hardware algorithm acceleration, and more specifically, to an FPGA-based method for optimizing and accelerating structured light algorithms. Background Technology

[0002] 3D structured light is a non-contact measurement method based on structured light measurement. It projects structured light carrying coded information onto the object being measured using a projection device, records the structured light image sequence using a camera, and then uses specific algorithms to obtain the three-dimensional data of the object. 3D structured light offers advantages such as high detection speed and high accuracy, and has wide applications in online inspection in the consumer electronics industry, 3D scanning of automotive parts, acquisition of surface features, and research on complex structures. However, structured light algorithms typically involve large amounts of data acquisition and processing, and processor-based solutions require significant computation time, which cannot meet the requirements of rapid, real-time online measurement. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this invention proposes an FPGA-based method for optimizing and accelerating structured light algorithms.

[0004] The first aspect of this invention provides a method for optimizing and accelerating structured light algorithms based on FPGA, comprising the following steps:

[0005] S1, the processor receives image data captured by the camera and sends the image data to the FPGA;

[0006] S2, structured light imaging technology based on Gray code projection, uses Gray code to coarsely locate the position of each camera pixel on the optical engine, and then uses moving stripe technology to finely locate the position of the pixel on the optical engine within each Gray code period, thus reconstructing three-dimensional topography information.

[0007] S3, the FPGA side sequentially completes image preprocessing, HDR fusion of multiple exposure images, Gray code decoding, moving stripe decoding, and phase synthesis, and the PC side completes point cloud reconstruction;

[0008] S4, by utilizing FPGA parallel processing and pipelined approach, can perform image preprocessing, HDR fusion, Gray code generation, and moving edge stripe subtraction in real time while sending data to the PC.

[0009] S5, extract the Gray code for decoding, and simultaneously decode the moving stripes. Combine the two to obtain the final phase data;

[0010] S6 uploads the phase data to the host computer.

[0011] In a preferred embodiment of the present invention, the FPGA-side algorithm implementation steps in steps S4 and S5 are as follows:

[0012] The original image is first denoised in the FPGA by caching 5 lines of the image on the FPGA side and filtering out individual glitch data in the image.

[0013] By acquiring low-exposure and high-exposure images, and then combining them;

[0014] The FPGA buffers images with odd-numbered frame numbers. When an image with an even-numbered frame number is input, it synchronously reads the image with the odd-numbered frame number and subtracts it from the image to obtain a set of difference data.

[0015] Read the difference data and locate the sub-pixel edges of the moving stripes;

[0016] Obtain the fringe difference data at the same position, calculate the sub-pixel position, and obtain intermediate result data; read the Gray code data from memory, perform Gray code decoding in real time, and calculate and merge the intermediate result data and Gray code decoded data to obtain the final phase data.

[0017] In a preferred embodiment of the present invention, the FPGA-side algorithm uses 8-level Gray code, with a total of 9 images, one of which is the inverse color image of the level 1 Gray code. The threshold image and Gray code image are generated from the level 1 Gray code and its inverse color image and stored in the FPGA memory.

[0018] As data from the Gray code projection map flows in, the threshold map T and the generated Gray code map G are read out synchronously, and the Gray code map G is updated and written to memory.

[0019] After the input of 9 Gray code projection images is completed, the Gray code image G is generated and stored in the FPGA's memory.

[0020] In a preferred embodiment of the present invention, phase data is uploaded to the host computer via a PCIe interface for high-throughput data interaction, and the calculation result data is quickly uploaded to the processor.

[0021] The technical solution of the present invention has the following advantages compared with the prior art:

[0022] This application proposes an optimization and acceleration method for structured light algorithms based on FPGA. By implementing some operations on the FPGA for hardware acceleration, and then having the host computer obtain the results of the FPGA operations for final calculation, the pipelined approach greatly improves the algorithm's operation speed and meets the needs of online rapid detection in the industrial field. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, some of the drawings in the following description are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the structured light algorithm optimization and acceleration method based on FPGA according to an embodiment of the present invention. Detailed Implementation

[0025] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0027] Example 1

[0028] See Figure 1 As shown, this invention proposes an FPGA-based method for optimizing and accelerating structured light algorithms, comprising the following steps:

[0029] S1, the processor receives image data captured by the camera and sends the image data to the FPGA;

[0030] S2, structured light imaging technology based on Gray code projection, uses Gray code to coarsely locate the position of each camera pixel on the optical engine, and then uses moving stripe technology to finely locate the position of the pixel on the optical engine within each Gray code period, thus reconstructing three-dimensional topography information.

[0031] S3, the FPGA side sequentially completes image preprocessing, HDR fusion of multiple exposure images, Gray code decoding, moving stripe decoding, and phase synthesis, and the PC side completes point cloud reconstruction;

[0032] S4, by utilizing FPGA parallel processing and pipelined approach, can perform image preprocessing, HDR fusion, Gray code generation, and moving edge stripe subtraction in real time while sending data to the PC.

[0033] S5, extract the Gray code for decoding, and simultaneously decode the moving stripes. Combine the two to obtain the final phase data;

[0034] S6 uploads the phase data to the host computer.

[0035] Specifically, the core algorithm is processed on an FPGA. The algorithm flow is as follows: The structured light camera consists of a CMOS camera and an optomechanical system. The optomechanical system projects grating fringes onto the object being measured. The grating fringes are distorted and deformed due to the undulating shape of the object. The camera captures a photograph of the deformed grating. By utilizing the degree of curvature of the captured fringes modulated by the object, the phase of the curved fringes is demodulated. This phase is then converted into the height of the entire field to obtain the position and depth information of the object, thereby reconstructing the entire three-dimensional space.

[0036] Furthermore, structured light imaging technology based on Gray code projection is employed. Gray code-based 3D imaging technology exhibits good robustness and fast computation speed, facilitating its application in industrial fields. However, Gray codes only carry object information at their edge pixels, and high-level Gray code periods are very small, making accurate detection and positioning difficult. Therefore, Gray code-assisted fringe edge-moving technology is actually used. This method uses Gray codes to coarsely locate the position of each camera pixel on the optical engine, and then uses fringe-moving technology to finely locate the position of pixels on the optical engine within each Gray code period. This method can quickly and accurately reconstruct 3D topographic information.

[0037] According to an embodiment of the present invention, the FPGA-side algorithm implementation steps in steps S4 and S5 are as follows:

[0038] The original image is first denoised in the FPGA by caching 5 lines of the image on the FPGA side and filtering out individual glitch data in the image.

[0039] By acquiring low-exposure and high-exposure images, and then combining them;

[0040] The FPGA buffers images with odd-numbered frame numbers. When an image with an even-numbered frame number is input, it synchronously reads the image with the odd-numbered frame number and subtracts it from the image to obtain a set of difference data.

[0041] Read the difference data and locate the sub-pixel edges of the moving stripes;

[0042] Obtain the fringe difference data at the same position, calculate the sub-pixel position, and obtain intermediate result data; read the Gray code data from memory, perform Gray code decoding in real time, and calculate and merge the intermediate result data and Gray code decoded data to obtain the final phase data.

[0043] According to an embodiment of the present invention, the FPGA-side algorithm uses 8-level Gray code, with a total of 9 images, one of which is the inverse color image of the level 1 Gray code. The threshold image and Gray code image are generated from the level 1 Gray code and its inverse color image and stored in the FPGA memory.

[0044] As data from the Gray code projection map flows in, the threshold map T and the generated Gray code map G are read out synchronously, and the Gray code map G is updated and written to memory.

[0045] After the input of 9 Gray code projection images is completed, the Gray code image G is generated and stored in the FPGA's memory.

[0046] Specifically, the FPGA-side algorithm implementation steps are as follows: The FPGA side completes image preprocessing, HDR fusion of multiple exposure images, Gray code decoding, moving stripe decoding, and phase synthesis; the PC side completes point cloud reconstruction. By utilizing FPGA parallel processing and a pipelined approach, image preprocessing, HDR fusion, Gray code generation, and moving stripe subtraction can be performed in real time while data is sent from the PC. Then, the Gray code is extracted and decoded, and the moving stripes are decoded simultaneously. The two are then combined to obtain the final phase. The specific process is as follows:

[0047] (1) Convert the XDMA AXI-MM write interface to a general parallel port data. XDMA is 8 lanes with a speed configuration of 5GT / S. The port data width of AXI is 128 bits and the frequency is 250M. In order to reduce the overall operating frequency of the algorithm system, the data width is increased to 256 bits and the frequency is reduced to 125M. At this time, the data parallelism is 32, that is, 32 pixels are processed per clock cycle.

[0048] (2) Preprocessing operation of the original image. During the measurement process of structured light technology, interference from environmental and hardware factors will introduce certain noise, which will cause certain errors in solving the phase. Therefore, noise reduction operation is performed on the original image in the FPGA. By using Block Ram to cache 5 lines of image on the FPGA side, a 3x3 or 5x5 window can be used to filter out individual spur data in the image.

[0049] (3) In order to improve the image quality of the stripe pattern, low-exposure and high-exposure images are collected and then synthesized. In this mode, the first set of exposure image data sent by XDMA needs to be cached in DDR first. When the second set of exposure images is sent, the exposure data G1 of the first set is read from the DDR of the FPGA and synthesized with the pixels at the same position of the second set of exposure images G2 in real time to generate a set of HDR images M. The data stream flows directly into the next algorithm step.

[0050] (4) In this algorithm, an 8-level Gray code is used, with a total of 9 images, one of which is the inverse color image of the 1-level Gray code. First, the threshold image and the Gray code image are generated from the 1-level Gray code and its inverse color image and stored in the FPGA memory. Then, as the data of the Gray code projection images of other levels flow in, the threshold image T and the generated Gray code image G are read out synchronously, and the Gray code image G is updated and written into memory. After the input of 9 Gray code projection images is completed, the Gray code image G is also generated and stored in the memory of the FPGA.

[0051] (5) Moving stripes to obtain the difference. This step involves 20 frames of images. As the forward and reverse moving stripe projection images are input, the FPGA buffers the images with odd frame numbers. When the images with even frame numbers are input, the FPGA reads out the images with odd frame numbers synchronously and subtracts from them to obtain a set of difference data, totaling 10 frames, denoted as F1-F10, and stores them sequentially in memory.

[0052] (6) The previous steps, through the FPGA real-time pipeline processing, completed the corresponding processing when the host computer finished sending the data, obtaining some intermediate data and storing it in memory. This step begins non-real-time processing. By reading the difference data of the previous fringe pattern, the sub-pixel of the moving fringe edge is first located. This process requires reading data from the same position F1-F10 in the FPGA memory for calculation. To improve the data throughput efficiency, AXIDMA is designed to burst read a complete line of data from memory each time. There are 10 lines in 10 frames of images, which are stored in Block Ram. Every time 10 lines are stored, the calculation begins. Through ping-pong operation, the data can be continuously output from memory. By simultaneously obtaining the fringe difference data at the same position, the sub-pixel position is calculated to obtain the first result. At the same time, Gray code data is read from memory and Gray code decoding is performed in real time. The first result and the Gray code decoded data D are calculated and merged to obtain the final phase data.

[0053] (7) To avoid jagged edges on the point cloud, the phase data S2 obtained in the previous step is smoothed by 3x3 Gaussian and the smoothed data S3 is written back to memory for XDMA to read.

[0054] According to an embodiment of the present invention, phase data is uploaded to the host computer via the PCIe interface for high-throughput data interaction, and the calculation result data is quickly uploaded to the processor.

[0055] Specifically, after the phase data calculation is completed and written to the PS's memory, an interrupt signal is sent to the XDMA. The XDMA then sends a PCIe interrupt message to the host computer, informing it that the phase data calculation on the FPGA side has been completed. The phase data is then read from the FPGA's memory via a PCIe read operation and sent back to the host computer's memory for the next step of point cloud generation calculation.

[0056] Furthermore, after the processor receives the image data captured by the camera, it needs to send the image data to the FPGA. PCIe is chosen as the transmission interface, mainly based on the following advantages of PCIe:

[0057] (1) PCIe is a high-speed serial point-to-point transmission, which integrates a high-speed transceiver, has a large transmission bandwidth, and high reliability.

[0058] (2) PCIe has lower latency. PCIe devices are directly connected to the PCIe bus and share memory with the main processor. They do not occupy CPU bus resources, thus reducing protocol overhead.

[0059] (3) Easy to upgrade. The protocol speed has been greatly improved from 2.5GT / S for a single lane of PCIe 1.0 to 16GT / S for PCIe 4.0, while the interface for users remains the same.

[0060] In this application, the FPGA acts as the PCIe Endpoint, communicating with the root complex of the host computer. Xilinx FPGAs offer various PCIe-based IP cores, among which the XDMA IP core integrates DMA functionality, opening both AXI-Lite and AXI-MM as user-side access interfaces. This simplifies data interaction and offers superior performance. The unified use of the AXI bus ensures better compatibility with PCIe port designs of different chips and speeds. In this design, the AXI-MM interface is used for reading and writing image data, while the AXI-Lite interface is used for configuring FPGA-side device registers. Considering the large volume of image data throughput, the XDMA AXI-MM read / write signals are separated. When data is sent from the host computer (i.e., AXI-MM write access), after logical conversion, one path directly connects to the algorithm's input module, while the other connects to AXIInterConnect. The first set of exposed image data is written to memory. When the second set of exposed images arrives, the first set is simultaneously read from memory, synthesized into an HDR image, and then fed into the algorithm module. This reduces DDR bandwidth usage and improves overall processing speed.

[0061] In summary, this application proposes an FPGA-based method for optimizing and accelerating structured light algorithms. By implementing some computations on the FPGA for hardware acceleration, and then having the host computer acquire the results from the FPGA computation for final calculation, the pipelined approach significantly improves the algorithm's processing speed, meeting the needs of online rapid detection in industrial fields. Utilizing the pipelined processing characteristics of FPGAs greatly enhances the overall algorithm speed. The data transmission and computation time on the FPGA side is t1, and the cloud computing time on the PC side is t2. Cloud computing of the first frame requires t1 + t2 time. When data is continuously input, the time consumed is m. For a 1080p image, the actual processing time for ax(t1, t2) can be controlled within 300ms, greatly reducing the processing speed of the algorithm. The processing speed of this algorithm on the FPGA side is mainly limited by the XDMA transmission interface rate and the interaction bandwidth of the FPGA memory. With the XDMA2.0 IP, the PCIE download speed reaches 2.4GB / s and the return speed reaches 2.1GB / s. The overall DDR3 bandwidth of the FPGA is 9GB / s. If the FPGA chip is upgraded to support XDMA3.0 and DDR4, the speed of the phase dephase operation on the FPGA can be further improved.

[0062] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to the above embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An FPGA-based structured light algorithm optimization acceleration method, characterized in that, It comprises the following steps: S1, the processor receives the image data collected by the camera, and sends the image data to the FPGA; S2, based on the structure light imaging technology of Gray code projection, the position of each camera pixel on the light machine is roughly positioned by Gray code, and the position of the pixel in each Gray code period on the light machine is precisely positioned by moving stripe technology, and the three-dimensional topographic information is reconstructed; S3, the FPGA end completes image preprocessing, HDR fusion of multiple exposure pictures, Gray code decoding, moving stripe decoding, and phase synthesis in turn, and the PC end completes point cloud reconstruction; S4, by using the parallel processing and pipeline mode of FPGA, the image preprocessing, HDR fusion, Gray code generation, and moving edge stripe difference are completed in real time while the data is sent to the PC end; S5, the Gray code decoding is taken out, and the moving stripe decoding is performed at the same time, and the final phase data is obtained by combining the two; S6, the phase data is uploaded to the host computer; The FPGA end algorithm implementation steps in steps S4 and S5 are as follows: In the FPGA, the original image is first denoised, 5 rows of images are cached in the FPGA end, and individual burr data in the image is filtered out; By collecting low exposure and high exposure images, and then synthesizing; FPGA caches odd frame number pictures, when even frame number pictures are input, odd frame number pictures are read out synchronously, and difference value data is obtained by subtracting them; Read the difference value data, and locate the edge sub-pixel of the moving stripe; Obtain the stripe difference value data at the same position, calculate the sub-pixel position, and obtain the intermediate result data; Read out the Gray code data from the memory, decode the Gray code in real time, calculate and merge the intermediate result data and the Gray code decoding data to obtain the final phase data.

2. The FPGA-based structured light algorithm optimization acceleration method of claim 1, wherein, In the FPGA end algorithm, 8-level Gray code is used, and a total of 9 pictures are used, of which 1 is the anti-color image of 1-level Gray code, the threshold image and Gray code image are generated from 1-level Gray code and its anti-color image, and are stored in the FPGA memory; With the data flow of Gray code projection image, the threshold image T and the generated Gray code image G are read out synchronously, and the Gray code image G is updated and written into the memory; After the input of 9 Gray code projection images is completed, the Gray code image G is generated and saved in the FPGA memory.

3. The FPGA-based structured light algorithm optimization acceleration method of claim 1, wherein, The phase data is uploaded to the host computer through the PCIe interface, high-throughput data interaction is performed, and the result data is quickly uploaded to the processor.

Images