A real-time binocular stereo vision architecture method and system based on FPGA

Through the real-time binocular stereoscopic vision architecture based on FPGA, synchronous processing of visible light and infrared images is solved, and the existing technology has poor working performance in foggy and rainy weather is achieved, achieving low latency, high precision and cost-effective image processing.

CN116347055BActive Publication Date: 2024-11-22UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310244816.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-09
Publication Date
2024-11-22
Estimated Expiration
2043-03-09

AI Technical Summary

Technical Problem

The existing binocular stereoscopic vision system has poor working performance in foggy and rainy days, and is costly, making it difficult to meet the needs of real-time work around the clock.

Method used

Using a real-time binocular stereoscopic vision architecture based on FPGA, the visible light and infrared images are obtained synchronously, image correction and parallax calculation are performed, and semi-global matching algorithm and bilinear difference calculation are used, and the image processing module is controlled by combining frame blanking period judgment. Start and stop to achieve low latency and high precision image processing.

Benefits of technology

A binocular stereoscopic vision system that works in real time under all weather conditions is realized. The system has low system delay, high parallax image accuracy, small cost and high economic benefits. It can complete image correction and stereoscopic matching tasks in real time under different climate environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116347055B_ABST
    Figure CN116347055B_ABST
Patent Text Reader

Abstract

The invention discloses a real-time binocular stereoscopic vision architecture method and system based on FPGA, comprising the following steps: synchronously acquiring a visible light image and an infrared image and storing them in a sequence according to the order of the visible light image and the infrared image; in an original image sending mode, sending the stored visible light image and infrared image in a polling priority manner; in a correction image and disparity map sending mode, performing image correction on the visible light image and the infrared image through an image processing module to obtain a visible light corrected image and an infrared corrected image, obtaining a disparity map of the visible light corrected image and the infrared corrected image through a semi-global matching algorithm, and sending the visible light corrected image, the infrared corrected image and the disparity map, wherein the disparity map includes a visible light disparity image and an infrared disparity image; the architecture method and the system can work in real time under all-weather conditions, have low system delay, high disparity image accuracy, and low cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image acquisition technology, and in particular to a real-time binocular stereoscopic vision architecture method and system based on FPGA. Background Art

[0002] In recent years, binocular stereo vision systems have been widely used in intelligent robots, 3D scene reconstruction, drones, autonomous driving, etc. In order to achieve all-weather and real-time working requirements, the industry mostly uses LiDAR ranging to meet the scene requirements of all-weather work, which has good effects in foggy and rainy days, but the high cost is a big disadvantage of this solution. Summary of the invention

[0003] Based on the technical problems existing in the background technology, the present invention proposes a real-time binocular stereo vision architecture method and system based on FPGA, which can work in real time under all-weather conditions with low system delay, high parallax image accuracy, low construction cost and high economic benefit.

[0004] The present invention proposes a real-time binocular stereoscopic vision architecture method based on FPGA, comprising the following steps:

[0005] Synchronously acquiring visible light images and infrared images and storing them in sequence according to the order of the visible light images and the infrared images;

[0006] In the original image uploading mode, the stored visible light image and infrared image are uploaded in a polling priority manner;

[0007] In the correction image and disparity map sending mode, the image processing module performs image correction on the visible light image and the infrared image to obtain a visible light corrected image and an infrared corrected image, and obtains a disparity map of the visible light corrected image and the infrared corrected image through a semi-global matching algorithm, and sends the visible light corrected image, the infrared corrected image and the disparity map, wherein the disparity map includes a visible light disparity image and an infrared disparity image;

[0008] In the correction image and disparity image sending mode, the start and stop of the image processing module can be controlled in real time by judging whether the data amount in the current DRAM meets the consumption of the image processing module during the frame blanking period of the image processing module.

[0009] Furthermore, in the synchronously acquiring the visible light image and the infrared image and storing them in sequence according to the order of the visible light image and the infrared image, it specifically includes:

[0010] Synchronously receive visible light images and infrared images uploaded by the visible light binocular camera module and the infrared binocular camera module;

[0011] Two frame buffer spaces are provided in the DRAM for storing visible light images and infrared images respectively. According to the order of the visible light images and the infrared images, the two frame buffer spaces in the DRAM are written sequentially through the AXI4 bus interface.

[0012] Furthermore, in the synchronous reception of the visible light image and the infrared image uploaded by the visible light binocular camera module and the infrared binocular camera module, it specifically includes:

[0013] Synchronize the frame synchronization signal given by the visible light binocular camera module to the clock domain of the infrared binocular camera module;

[0014] An infrared image frame that matches the synchronized frame synchronization signal is acquired, and the visible light image and the infrared image are synchronously aligned to obtain the synchronized visible light image and the infrared image.

[0015] Furthermore, when using the bus to write data in sequence according to the visible light image and the infrared image, it is determined whether to write data or discard a complete frame of data by the number of written data packets, whether it is in the last stage of the frame blanking period, and the specific address of the current read address bus.

[0016] Further, in the correction image and disparity map uploading mode, image correction is performed on the visible light image and the infrared image to obtain a visible light corrected image and an infrared corrected image, specifically including:

[0017] Set the lens calibration table for the visible light binocular camera module and the infrared binocular camera module;

[0018] The 3-8 decoder divides the input visible light image and infrared image data into 8 groups of line buffers according to the lower three bits of the line number;

[0019] The visible light image and infrared image data are read out from 8 sets of line buffers by 8 16-to-1 multiplexers;

[0020] The four visible light images and infrared image data in the required two rows are selected through two 8-to-1 multiplexers;

[0021] Performing bilinear difference calculation on the four visible light images and infrared image data to obtain a corrected visible light corrected image and an infrared corrected image;

[0022] Furthermore, the specific bilinear difference calculation formula is as follows:

[0023] I[x,y]=I[x i ,y i ](1-x f )(1-y f )+I[x i +1,y i ]xf (1-y f )

[0024] +I[x i ,y i +1](1-x f )y f +I[x i +1,y i +1]x f y f

[0025] Among them, (x, y) represents the coordinates of the input pixel, which is divided into an integer part and a decimal part. i ,y i ) is the integer part, which is used to select the four nearest pixels. f ,y f ) is the decimal part, which is used to weight the pixel value.

[0026] Furthermore, in obtaining the disparity map of the visible light corrected image and the infrared corrected image by the semi-global matching algorithm, it specifically includes:

[0027] Denoising the grayscale images of the input visible light corrected image and the infrared corrected image by Gaussian filtering;

[0028] The initial cost of each pixel under all disparities is calculated through the Census transformation algorithm;

[0029] A semi-global cost aggregation method is used to calculate the aggregation cost of each pixel;

[0030] Finally, the WTA principle is used to select the disparity corresponding to each pixel;

[0031] The disparity accuracy is optimized through sub-pixel disparity calculation, speckle filtering, median filtering, and hole filling, and visible light disparity images and infrared disparity images are output.

[0032] Furthermore, in the correction image and disparity image uploading mode, the image processing module is started and stopped in real time by judging whether the current data amount in the DRAM meets the consumption of the image processing unit during the frame blanking period of the image processing module. The specific process is as follows:

[0033] Determine how many data packets are still stored in the current DRAM. If the data in the cached data packets reaches the set threshold, an enable signal is sent to the VTC;

[0034] Determine whether the pfull signal of the cache level 1 FIFO is pulled high and whether the pempty signal of the level 2 FIFO is pulled low. If the pfull signal of the level 1 FIFO is pulled high or the pempty signal of the level 2 FIFO is pulled low, send an enable signal to the VTC to start the image processing module.

[0035] Determine whether it is currently in the frame blanking period. If it is in the frame blanking period, send an enable signal to the VTC to start the image processing module;

[0036] If the image processing modules all meet the above conditions, the image processing modules will be in the startup state or remain in the startup state, continue to perform image correction on the visible light image and the infrared image, and continue to obtain the disparity map of the visible light corrected image and the infrared corrected image through the semi-global matching algorithm;

[0037] If the image processing module does not fully meet the above conditions, the image processing module will be in a paused state, and will suspend image correction of the visible light image and the infrared image, as well as suspend obtaining the disparity map of the visible light corrected image and the infrared corrected image through the semi-global matching algorithm, until the number of data packets corresponding to the original visible light image and the infrared image exceeds the set threshold, and then restart the image processing module.

[0038] A real-time binocular stereo vision architecture system based on FPGA, a real-time binocular stereo vision architecture method and system based on FPGA, including a visible light / infrared image synchronous receiving module, a frame buffer write controller module, a frame buffer read controller module, an image processing unit rate matching module, an image correction module, a binocular stereo matching module, and a USB transceiver controller module;

[0039] The visible light / infrared image synchronous receiving module is used to acquire the visible light image and the infrared image and store the visible light image and the infrared image in a fifo after synchronization;

[0040] The frame buffer write controller module is used to sequentially write the visible light image and the infrared image into the DRAM for frame buffering;

[0041] The frame buffer read controller module is used to interact with the USB transceiver controller module or the image processing unit rate matching module to obtain the post-stage status to determine whether to read out the visible light image and the infrared original image stored in the DRAM;

[0042] The image processing unit rate matching module is used to determine whether the amount of data in the current DRAM meets the consumption of the image processing unit during the frame blanking period of the image processing module, so as to control the start and stop of the image processing module in real time;

[0043] The image correction module is used to perform image correction on the visible light image and the infrared image and output the visible light corrected image and the infrared corrected image to the binocular stereo matching module;

[0044] The binocular stereo matching module is used to obtain a visible light parallax image and an infrared parallax image by processing the visible light corrected image and the infrared corrected image through a semi-global matching algorithm respectively;

[0045] The USB transceiver controller module is used to receive a startup mode instruction sent by a host computer, and to upload visible light images and infrared images in the original image upload mode, and to upload visible light corrected images, infrared corrected images, visible light parallax images and infrared parallax images in the correction image and parallax image upload mode.

[0046] A computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the real-time binocular stereoscopic vision architecture method as described above.

[0047] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0048] The advantages of the real-time binocular stereoscopic vision architecture method and system based on FPGA provided by the present invention are: the real-time binocular stereoscopic vision architecture method and system based on FPGA provided in the structure of the present invention can work in real time under all-weather conditions with low system delay, high parallax image accuracy, low cost and high economic benefit. Under the synchronous operation of the visible light binocular camera module and the infrared binocular camera module, the image processing tasks such as binocular image correction and binocular stereo matching can be completed in real time in different climatic environments and facing different working scenes, and the processed visible light and infrared corrected images and visible light and infrared parallax maps can be uploaded to the host computer through USB 3.0; finally, an image processing frame rate of 25fps and a system delay of less than 4 milliseconds are achieved on the Xilinx XZU15EG FPGA core board. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the process of the present invention;

[0050] Figure 2 A flowchart of the real-time binocular stereo vision architecture;

[0051] Figure 3A schematic diagram of the hardware architecture of the real-time binocular stereo vision architecture;

[0052] Figure 4 It is a structural diagram of a frame buffer write controller based on the AXI4 bus interface;

[0053] Figure 5 It is a structural diagram of a frame buffer read controller based on the AXI4 bus interface;

[0054] Figure 6 Schematic diagram of the structure of the rate matching module of the image processing unit. DETAILED DESCRIPTION

[0055] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0056] like Figures 1 to 6 As shown, the present invention proposes a real-time binocular stereoscopic vision architecture method and system based on FPGA, comprising the following steps:

[0057] S1: synchronously acquire visible light images and infrared images and store them in sequence according to the order of visible light images and infrared images;

[0058] S2: In the original image uploading mode, the stored visible light image and infrared image are uploaded in a polling priority manner;

[0059] S3: In the correction image and disparity map sending mode, image correction is performed on the visible light image and the infrared image to obtain a visible light corrected image and an infrared corrected image, and the disparity map of the visible light corrected image and the infrared corrected image is obtained by a semi-global matching algorithm. The visible light corrected image, the infrared corrected image and the disparity map are sent, and the disparity map includes a visible light disparity image and an infrared disparity image. At the same time, during the frame blanking period of the image processing module, it is determined whether the data amount in the current DRAM meets the consumption of the image processing unit, so as to control the start and stop of the image processing module in real time.

[0060] Through steps S1 to S3, by synchronizing the visible light image with the infrared image and storing the image cache in a limited time window through a reasonably designed image, the image processing of the visible light image and the infrared image is completed, so that the overall delay of the system is controlled below 4 milliseconds at a frame rate of 25fps, realizing all-weather real-time binocular stereo vision system hardware acceleration.

[0061] Moreover, under the synchronous operation of the visible light binocular camera module and the infrared binocular camera module, this architecture method can complete image processing tasks such as binocular image correction and binocular stereo matching in real time under different climatic environments and facing different working scenes, and upload the processed visible light and infrared corrected images and visible light and infrared disparity maps to the host computer through USB 3.0; finally, an image processing frame rate of 25fps and a system delay of less than 4 milliseconds are achieved on the Xilinx XZU15EG FPGA core board.

[0062] For the convenience of description, an architecture system is used for detailed description, wherein the architecture system is used to implement the above-mentioned architecture method.

[0063] like Figure 2 As shown, after the system synchronizes the image data of the visible light binocular camera module and the infrared binocular camera module, it obtains paired visible light images and infrared images, creates a frame buffer of two frames as a ping-pong buffer for image data, designs an efficient frame buffer read and write controller to match the image data speed of the visible light binocular camera module and the infrared binocular camera module with the speed of the USB transceiver controller, designs a low-latency image processing unit rate matching module to start and stop the image processing module in real time to match the slight changes in the original image data speed when the visible light and infrared binocular cameras are working, wherein the image processing module is used to implement the image correction and image binocular stereo matching in step S3 to output a visible light corrected image, an infrared corrected image, a visible light parallax image, and an infrared parallax image.

[0064] Specific description of the architecture system:

[0065] 1. Visible light / infrared image synchronous receiving module

[0066] The visible light / infrared image synchronous receiving module is used to receive the visible light images and infrared images uploaded by the visible light binocular camera module and the infrared binocular camera module, and synchronously align the visible light images and infrared images to facilitate the synchronization of the subsequent two-way image processing flow and the visible light / infrared data package sent on the USB; it is used to realize the synchronous acquisition of the visible light image and the infrared image in step S1.

[0067] The visible light / infrared image synchronization receiving module synchronizes the frame synchronization signal given by the visible light binocular camera module (25fps) to the clock domain of the infrared binocular camera module (50fps), and uses the synchronized frame synchronization signal to match the most recent infrared image frame, so that the visible light and infrared images are synchronized (the time difference between the visible light and infrared images ranges from 0s to 0.02s).

[0068] 2. Frame buffer write controller module

[0069] The frame buffer write controller module is used to safely write the received visible light image and infrared image into DRAM through the AXI4 bus interface, and controls the writing of new data by opening two frame buffer spaces for visible light and infrared images in DRAM and real-time monitoring the data reading status in DRAM, so as to avoid the data in DRAM being overwritten by new data before being used, and is used to realize the sequential storage in the order of visible light image and infrared image in step S1.

[0070] like Figure 4 As shown, the frame buffer write controller module is composed of visible light left and right image data buffer, infrared image left and right data buffer, two-channel polling read and write controller, write data counter, write data controller, frame blanking period counter, etc. The visible light left and right image data cache and the infrared image left and right data cache are used to cache the visible light image and the infrared image respectively; the two-channel polling read-write controller is used to arbitrate and allocate the right to use the write bus, and use the bus to write data in sequence according to the visible light image and the infrared image; the frame blanking period counter is used to record the frame blanking period of the original uploaded visible light image and infrared image data, and provide this data to the write data controller to determine whether to write data or discard data at the current moment; the write data counter is used to record the number of data packets written, and the write data controller is used to communicate and interact with the AXI4 bus interface bus to write data into the DRAM, and judge whether to write data or discard a complete frame of data according to the number of written data packets, whether it is in the last stage of the frame blanking period, and the specific address of the current read address bus. Specifically, when the size of the current write data packet reaches the size of a frame image and is at the end of the frame blanking period, the spatial position of the read address bus at this time is determined: 1. If the read address bus is still in the address space of the unread image data at this time, the upcoming complete frame of data will be completely discarded; 2. If the read address bus is in the address space of the read image data at this time, the upcoming image data will be written normally.

[0071] 3. Frame buffer read controller module

[0072] The frame buffer read controller module is used to read the image data and correction table data in the DRAM through the AXI4 bus interface in a polling priority manner, and write the visible light data into the intermediate level fifo in Y, U, and V formats respectively. In the original image upload mode, it is used to realize the step S2 of reading the stored visible light image and infrared image in a polling priority manner, and then uploading the read image through the USB read-write controller module; in the correction image and disparity image upload mode, it is used to realize the step S3 of reading the visible light image and infrared image before image correction for subsequent image processing, and then uploading the read image through the USB read-write controller module.

[0073] like Figure 5 As shown, the frame buffer read controller module is composed of a read data counter, a read data controller, a four-channel polling read controller, a visible light data cache (FIFO0), an infrared data cache (FIFO1), a visible light correction table data cache (FIFO2), and an infrared correction table data cache (FIFO3). The read data counter is used to count the number of data packets read out; the read data controller is used to communicate and interact with the AXI4 bus interface bus, read data from the DRAM, and determine whether to read data currently according to the number of read data packets and the read data request of the FIFO; the four-channel polling read controller is used to use the bus to read data in the order of visible light image data, infrared image data, visible light correction table data, and infrared correction table data. The visible light image data cache (FIFO0), the infrared image data cache (FIFO1), the visible light corrected image data cache (FIFO2), and the infrared corrected image data cache (FIFO3) are used to cache four channels of data read out from the DRAM respectively.

[0074] 4. Image processing unit rate matching module

[0075] The image processing unit rate matching module is used to adjust the start and pause of the image processing module in real time between frames according to the previous stage data rate, so that the image processing unit matches the slight speed difference between the visible light binocular camera module and the infrared binocular camera module during operation, and is used to realize the start and pause of the image processing module in the control step S3 during the image processing process, wherein the previous stage data specifically refers to the data packets corresponding to the original visible light image and the infrared image.

[0076] like Figure 6As shown, the rate matching module of the image processing unit is composed of functional modules such as a cached data packet counter, a VTC start-stop judgment, and a FIFO state judgment. The cached data packet counter determines how many data packets are still stored in the current DRAM according to the values ​​of the read and write data packet counters in the previous read and write cache controller module; the FIFO state judgment module is used to judge whether the pfull signal of the read cache 1st level FIFO is pulled high and whether the pempty signal of the secondary FIFO is pulled low. If the conditions are met, an enable signal is sent to the VTC start-stop judgment module; the VTC start-stop judgment module is used to judge whether it is currently in the frame blanking period, whether the value of the cached data packet counter reaches the set threshold, and whether the enable signal of the FIFO state judgment module is received. If it is in the frame blanking period, the value of the cached data packet counter reaches the set threshold, and the enable signal of the FIFO state judgment module is received, the VTC will start or remain in the start state, otherwise the VTC will be suspended (i.e., the operation of the image processing module will be suspended), and the VTC will be restarted when the number of previous image data packets exceeds the set threshold, and the image processing module continues to work, wherein the previous image data packets are specifically the data packets corresponding to the original visible light image and the infrared image.

[0077] 5. Image correction module

[0078] The image correction module is used to correct the original visible light image and infrared image to facilitate the improvement of the accuracy of the subsequent binocular stereo matching, and is used to implement the image correction of the visible light image and the infrared image in step S3 to obtain the visible light corrected image and the infrared corrected image.

[0079] The calculation principle of the image correction module is based on the bilinear interpolation algorithm, that is, the original image is corrected using bilinear interpolation according to the camera correction table data prepared in advance. The image correction module is mainly composed of three parts: a row buffer composed of 128 dual-port RAMs, a row buffer read-write controller, and a bilinear interpolation calculation. The row buffer is composed of 128 (a total of 8 groups, 16 in each group) 8-bit dual-port RAMs with a depth of 1280. The row buffer design size is determined by searching for the maximum row difference in the upper and lower rows in the correction table. Since the maximum row difference of the correction table in this embodiment is 64, the row buffer is designed to be 128 rows, which can search for image data with a maximum row difference of 64 upward and downward, ensuring the correctness of the algorithm data. The row buffer read-write controller is mainly composed of 1 3-8 decoder, 16 16-to-1 multiplexers, and 2 8-to-1 multiplexers. In the write control path, the 3-8 decoder divides the input image data into 8 groups of line caches according to the lower three bits of the line number. In the read control path, the data is read out from 8 groups by 8 16-to-1 multiplexers, and finally the four data in the two required lines are selected by 2 8-to-1 multiplexers. The bilinear interpolation calculation module is mainly composed of 6 multipliers and 6 adders. The bilinear interpolation algorithm is as follows:

[0080] I[x,y]=I[x i ,y i ](1-x f )(1-y f )+I[x i +1,y i ]x f (1-y f )

[0081] +I[x i ,y i +1](1-x f )y f +I[x i +1,y i +1]x f y f

[0082] Among them, (x, y) represents the coordinates of the input pixel, which is divided into an integer part and a decimal part. i ,y i ) is the integer part, which is used to select the four nearest pixels. f ,y f ) is the decimal part, which is used to weight the pixel value.

[0083] 6. Binocular Stereo Matching Module

[0084] The binocular stereo matching module is used to obtain a visible light disparity image and an infrared disparity image through the SGM algorithm according to the Y channel data of the corrected visible light image and the infrared image, and to obtain the disparity map of the visible light corrected image and the infrared corrected image through the semi-global matching algorithm in step S3.

[0085] The hardware architecture of the binocular stereo matching module is designed according to the improved algorithm based on the SGM algorithm. The binocular stereo matching algorithm is mainly composed of a preprocessing module, an initial cost calculation module, a semi-global cost aggregation module, a disparity selection module, and a disparity optimization module. In the preprocessing module, the input grayscale image is denoised by Gaussian filtering, and then the initial cost of each pixel under all disparities is calculated by the Census transformation algorithm with good robustness. In the cost aggregation stage, the semi-global cost aggregation method is used to calculate the aggregation cost of each pixel. Finally, the disparity corresponding to each pixel is selected using the WTA (winner takes all) principle. In the disparity optimization stage, the disparity accuracy is further improved by sub-pixel disparity calculation, speckle filtering, median filtering, and hole filling. The entire hardware design requires two clock frequencies. The input of the initial left and right grayscale images and the output of the final disparity map are performed at a clock frequency of 52MHz, and one data is input / output in each cycle. The stereo matching calculation module in the middle processes data under the control of a 104MHz clock frequency and processes one data every four cycles. The hardware architecture design of the entire stereo matching calculation module is a pipeline architecture with a high throughput rate.

[0086] 7. USB read-write controller module

[0087] The USB read-write controller module is used to send the four images generated by the image processing module, namely, the visible light corrected image, the infrared corrected image, the visible light parallax image and the infrared parallax image, to the host computer according to a fixed priority arbitration strategy, so as to realize the final image sending stage in step S2 and step S3.

[0088] The USB read-write controller module is used to receive instructions from the host computer or send corresponding instruction headers and image data to the host computer. The USB read-write controller receives and parses the working mode instructions from the host computer through custom instructions, sends receiving instructions to the visible light binocular camera module and the infrared binocular camera module, and sends working instructions to the image processing module, and the whole system starts working. The USB read-write controller defines the data packet instruction headers of the visible light corrected image, infrared corrected image, visible light parallax image, and infrared parallax image through custom instruction headers, and polls and sends image data according to the working mode.

[0089] In this embodiment, the main processing flow of the hardware architecture includes: the USB read-write controller module receives the mode start instruction sent by the host computer and parses it, and starts working when the initialization of the visible light binocular camera module and the infrared binocular camera module is completed. In the original image sending mode, the visible light / infrared image synchronization receiving module receives the left and right images from the visible light binocular camera module and the infrared binocular camera module and stores them in the fifo after synchronization; the frame buffer write controller module takes out the original visible light image and infrared image data in turn according to the fixed priority order of the original image data stored in the fifo and writes them safely into the DRAM (without overwriting the unread data in the DRAM); the frame buffer read controller module reads the original visible light image and infrared image data stored in the DRAM in turn according to the fixed priority order when the USB read-write controller module can send data, and writes them into the fifo interacting with the USB read-write controller module, waiting for the USB read-write controller module to send data; when the USB read-write controller module detects that there is data in the previous fifo, it alternately sends the original left and right images of the visible light image and infrared image in the order of visible light image and infrared image in frames. In the correction image and disparity image upload mode, the visible light / infrared image synchronization receiving module receives the left and right images from the visible light binocular camera module and the infrared binocular camera module and stores them in the fifo after synchronization; the frame buffer write controller module takes out the original visible light image and infrared image data stored in the fifo in turn according to the fixed priority order and writes them safely into the DRAM (without overwriting the unread data in the DRAM); when the number of data packets in the DRAM is greater than 1, the frame buffer read controller module starts to read the original visible light image and infrared image data in turn according to the fixed priority order, and writes the read data into the fifo according to the left and right images respectively The image processing unit rate matching module detects the number of data packets in the current DRAM and the data threshold signal in the previous fifo during the frame blanking period when the image processing module processes the image data, and determines whether the image processing module can be started at present. If the number of data packets in the current DRAM and the amount of data in the previous fifo do not reach the set threshold, the image processing module is suspended (at this time, the processing speed of the image processing module is slightly faster than the maximum rate at which the actual camera generates images), and the image processing module is started after the data is sufficient for the image processing module to completely process one frame. The image correction module uses the camera lens correction table data obtained in advance to correct the original visible light image and infrared image based on the bilinear interpolation algorithm to obtain the visible light corrected image and the infrared corrected image.The binocular stereo matching module uses the visible light corrected image and the infrared corrected image to obtain the visible light disparity image and the infrared disparity image (disparity is used to calculate the distance) according to the semi-global stereo matching algorithm; when the USB read-write controller module detects that there is data in the previous fifo, it sends the visible light corrected image and the infrared corrected image, the visible light disparity image and the infrared disparity image alternately in the order of frames. Finally, the host computer can obtain the distance information from the object in the image to the camera based on the disparity map.

[0090] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A real-time binocular stereo vision architecture method based on FPGA, comprising the following steps: S1, synchronously acquiring a visible light image and an infrared image and storing them in sequence according to the order of the visible light image and the infrared image; S2. In the original image uploading mode, the stored visible light image and infrared image are uploaded in a polling priority manner; S3, in the correction image and disparity map sending mode, the visible light image and the infrared image are corrected by the image processing module to obtain a visible light corrected image and an infrared corrected image, the disparity map of the visible light corrected image and the infrared corrected image is obtained by the semi-global matching algorithm, and the visible light corrected image, the infrared corrected image and the disparity map are sent, and the disparity map includes a visible light disparity image and an infrared disparity image, wherein, The process of obtaining the visible light corrected image and the infrared corrected image includes S31 to S35: S31, setting a lens calibration table for a visible light binocular camera module and an infrared binocular camera module; S32, 3-8 decoder divides the input visible light image and infrared image data into 8 groups of line buffers according to the lower three bits of the line number; S33, visible light image and infrared image data are read out from 8 sets of line buffers by 8 16-to-1 multiplexers; S34, selecting four visible light images and infrared image data in two required rows through two 8-to-1 multiplexers; S35, performing bilinear difference calculation on the four visible light images and the infrared image data to obtain a corrected visible light corrected image and an infrared corrected image; S4, in the correction image and disparity image sending mode, by judging whether the data amount in the current DRAM meets the consumption of the image processing module during the frame blanking period of the image processing module, so as to control the start and stop of the image processing module in real time, specifically including S41 to S45: S41, determining how many data packets are still stored in the current DRAM, and if the data in the cached data packets reaches a set threshold, sending an enable signal to the VTC; S42, judging whether the pfull signal of the cache level 1 FIFO is pulled high and whether the pempty signal of the level 2 FIFO is pulled low, if the pfull signal of the level 1 FIFO is pulled high or the pempty signal of the level 2 FIFO is pulled low, sending an enable signal to the VTC to start the image processing module; S43, determining whether it is currently in a frame blanking period, and if it is in the frame blanking period, sending an enable signal to the VTC to start the image processing module; S44, if the image processing modules all meet the above conditions, the image processing modules will be in the startup state or remain in the startup state, continue to perform image correction on the visible light image and the infrared image, and continue to obtain the disparity map of the visible light corrected image and the infrared corrected image through the semi-global matching algorithm; S45. If the image processing module does not fully meet the above conditions, the image processing module will be in a paused state, and will suspend image correction of the visible light image and the infrared image, and will suspend obtaining the disparity map of the visible light corrected image and the infrared corrected image through the semi-global matching algorithm.

2. The real-time binocular stereo vision architecture method based on FPGA according to claim 1, characterized in that: The synchronously acquiring the visible light image and the infrared image and sequentially storing the visible light image and the infrared image in the order of the visible light image and the infrared image specifically includes: Synchronously receive visible light images and infrared images uploaded by the visible light binocular camera module and the infrared binocular camera module; Two frame buffer spaces are provided in the DRAM for storing visible light images and infrared images respectively. According to the order of the visible light images and the infrared images, the two frame buffer spaces in the DRAM are written sequentially through the AXI4 bus interface.

3. The real-time binocular stereo vision architecture method based on FPGA according to claim 2, characterized in that: The visible light images and infrared images uploaded by the visible light binocular camera module and the infrared binocular camera module are received synchronously, including: Synchronize the frame synchronization signal given by the visible light binocular camera module to the clock domain of the infrared binocular camera module; An infrared image frame that matches the synchronized frame synchronization signal is acquired, and the visible light image and the infrared image are synchronously aligned to obtain the synchronized visible light image and the infrared image.

4. The real-time binocular stereo vision architecture method based on FPGA according to claim 2, characterized in that: When using the bus to write data in sequence according to the visible light image and the infrared image, the number of written data packets, whether it is in the last stage of the frame blanking period, and the specific address of the current read address bus are used to determine whether data is currently being written or whether a complete frame of data is discarded.

5. The real-time binocular stereo vision architecture method based on FPGA according to claim 1, characterized in that: The specific bilinear difference calculation formula is as follows: in, Represents the coordinates of the input pixel, divided into an integer part and a decimal part, is the integer part, which is used to select the four nearest pixels. It is the fractional part used to weight the pixel value.

6. The real-time binocular stereo vision architecture method based on FPGA according to claim 1, characterized in that: The disparity map of the visible light corrected image and the infrared corrected image obtained by the semi-global matching algorithm specifically includes: Denoising the grayscale images of the input visible light corrected image and the infrared corrected image by Gaussian filtering; The initial cost of each pixel under all disparities is calculated using the Census transformation algorithm; A semi-global cost aggregation method is used to calculate the aggregation cost of each pixel; Finally, the WTA principle is used to select the disparity corresponding to each pixel; The disparity accuracy is optimized through sub-pixel disparity calculation, speckle filtering, median filtering, and hole filling, and visible light disparity images and infrared disparity images are output.

7. A real-time binocular stereo vision architecture system based on FPGA, characterized in that: It includes a visible light / infrared image synchronous receiving module, a frame buffer write controller module, a frame buffer read controller module, an image processing unit rate matching module, an image correction module, a binocular stereo matching module, and a USB transceiver controller module; The visible light / infrared image synchronous receiving module is used to acquire the visible light image and the infrared image and store the visible light image and the infrared image in a fifo after synchronization; The frame buffer write controller module is used to sequentially write the visible light image and the infrared image into the DRAM for frame buffering; The frame buffer read controller module is used to interact with the USB transceiver controller module and the image processing unit rate matching module to obtain a subsequent state to determine whether to read out the visible light image and the infrared original image stored in the DRAM, wherein the subsequent state is the number of data packets read out and the FIFO read data request; The image processing unit rate matching module is used to determine whether the amount of data in the current DRAM meets the consumption of the image processing unit during the frame blanking period of the image processing module, so as to control the start and stop of the image processing module in real time; The image correction module is used to perform image correction on the visible light image and the infrared image and output the visible light corrected image and the infrared corrected image to the binocular stereo matching module; The binocular stereo matching module is used to obtain a visible light parallax image and an infrared parallax image by processing the visible light corrected image and the infrared corrected image through a semi-global matching algorithm respectively; The USB transceiver controller module is used to receive a startup mode instruction sent by a host computer, and to send a visible light image and an infrared image in an original image sending mode, and to send a visible light corrected image, an infrared corrected image, a visible light parallax image, and an infrared parallax image in a correction image and parallax image sending mode; Among them, the image correction module is used for S31 to S35: S31, setting a lens calibration table for a visible light binocular camera module and an infrared binocular camera module; S32, 3-8 decoder divides the input visible light image and infrared image data into 8 groups of line buffers according to the lower three bits of the line number; S33, visible light image and infrared image data are read out from 8 sets of line buffers by 8 16-to-1 multiplexers; S34, selecting four visible light images and infrared image data in two required rows through two 8-to-1 multiplexers; S35, performing bilinear difference calculation on the four visible light images and the infrared image data to obtain a corrected visible light corrected image and an infrared corrected image; In addition, the image processing unit rate matching module is specifically used in S41 to S45: S41, determining how many data packets are still stored in the current DRAM, and if the data in the cached data packets reaches a set threshold, sending an enable signal to the VTC; S42, judging whether the pfull signal of the cache level 1 FIFO is pulled high and whether the pempty signal of the level 2 FIFO is pulled low, if the pfull signal of the level 1 FIFO is pulled high or the pempty signal of the level 2 FIFO is pulled low, sending an enable signal to the VTC to start the image processing module; S43, determining whether it is currently in a frame blanking period, and if it is in the frame blanking period, sending an enable signal to the VTC to start the image processing module; S44, if the image processing modules all meet the above conditions, the image processing modules will be in the startup state or remain in the startup state, continue to perform image correction on the visible light image and the infrared image, and continue to obtain the disparity map of the visible light corrected image and the infrared corrected image through the semi-global matching algorithm; S45. If the image processing module does not fully meet the above conditions, the image processing module will be in a paused state, and will suspend image correction of the visible light image and the infrared image, and will suspend obtaining the disparity map of the visible light corrected image and the infrared corrected image through the semi-global matching algorithm.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the real-time binocular stereoscopic vision architecture method as described in any one of claims 1 to 6.