Multi-channel fusion polarization camera based on FPGA and multi-channel fusion method

Abstract

The invention discloses a multi-channel fusion polarization camera based on an FPGA and a multi-channel fusion method. The polarization camera comprises a power conversion circuit, and a polarized light image sensor peripheral circuit, an external synchronization signal processing circuit, a serial port conversion circuit, a video interface circuit, a memory circuit and a clock circuit which are interconnected with an FPGA circuit. According to the multi-channel fusion method, a focal plane splitting type polarization imaging technology is adopted, multi-group 4-channel near-infrared polarized light image information and multi-group 16-channel visible light polarized light image information are synchronously collected in real time, 16-channel image automatic exposure is achieved through a synchronization method, each image sensor can independently and automatically expose according to respective adaptive exposure parameters, and the image sensor can also independently and automatically expose according to the adaptive exposure parameters. Image data can be synchronously output; interpolation reconstruction of a 16-channel color visible light polarization image is achieved through an FPGA assembly line sliding window method, polarized light image data of multiple channels are spliced and fused, and after coding, the data are output to a user side through a video interface to be processed.

Description

Technical Field

[0001] The present invention relates to the technical field of polarized light imaging, and in particular to a multi-channel fusion polarization camera based on FPGA and a multi-channel fusion method. Background Art

[0002] Polarized light imaging technology is a technology that acquires and processes information on the surface morphology and physical properties of objects by collecting and analyzing polarized light reflected or transmitted by the objects. This technology helps to detect things that are difficult to visualize or identify. Polarized cameras, as front-end equipment for the engineering application of polarized light imaging technology, play an important role in polarized light collection.

[0003] However, the commonly used polarization cameras currently use only polarization grayscale sensors to collect 4-channel polarization brightness information at four polarization angles of 0°, 45°, 90° and 135°, and do not contain color information. To this end, we propose a multi-channel fusion polarization camera based on FPGA, which simultaneously fuses a total of 20 channels of polarization information from polarization grayscale sensors and polarization RGGB sensors (polarization image sensors coupled with polarization microarrays and RGGB Bayer filters). This can effectively make up for the lack of information obtained by a single polarization grayscale sensor, and provide more dimensional optical information for back-end applications such as defogging, anti-counterfeiting, and water surface reflection removal. An interpolation algorithm is used to reconstruct the 16-channel data of the polarization RGGB sensor, effectively reducing defects such as image boundary grid points and pseudo-color caused by the nearest neighbor linear interpolation method.

[0004] In addition, the collaborative work of more types of photoelectric sensors will also involve synchronization issues. For example, each load of the traditional imaging system independently handles processes such as exposure control, and uses the same synchronization source to make its output relatively synchronized, but the output video image needs to enter the same clock domain through the frame buffer to eliminate the phase difference and complete the fusion, and the frame buffer mechanism will introduce at least 1 frame delay time. For another example, an integrated imaging system that uses the same hardware platform to process multiple types of sensors can be exposed synchronously from the sensor end, but there are two problems with the existing implementation method. First, due to the difference in light intensity response rate of different types of sensors and the difference in transmittance of optical micro-transmittance elements on the pixel array, there are certain differences in the grayscale values ​​of the raw video data output by different types of sensors after the start and end of exposure, which affects the back-end image fusion effect. Eliminating the difference through the algorithm requires additional time and resources; second, multiple sensors share an automatic exposure control module. When the incident light of one of the sensors is blocked, the average brightness decreases, and the automatic exposure module will reversely increase the exposure time and gain, resulting in the output image of the other sensor with shared parameters being too bright. Therefore, we also propose a multi-channel fusion method based on the above-mentioned polarization camera to solve the synchronization problem involved in the collaborative work of multiple types of photoelectric sensors. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a polarization camera and a multi-channel fusion method, which uses FPGA to connect multiple polarization grayscale sensors and polarization RGGB sensors. Each polarization grayscale sensor can simultaneously obtain polarization information of 4 near-infrared channels, including but not limited to IMX250MZR-C; each polarization RGGB sensor can simultaneously obtain polarization information of 16 channels, including but not limited to IMX250MYR-C.

[0006] To achieve the above-mentioned purpose, one of the technical solutions adopted by the present invention is: a multi-channel fusion polarization camera based on FPGA, including a power conversion circuit, and a polarization light image sensor peripheral circuit interconnected with the FPGA circuit, an external synchronization signal processing circuit, a serial port conversion circuit, a video interface circuit, a memory circuit, and a clock circuit; the FPGA circuit serves as the core processing device of the camera, including a polarization light sensor drive module, a video acquisition module, a video processing module, a synchronization and exposure control module, a serial port data encoding and decoding module, a parameter configuration and management module, and a clock and reset management module.

[0007] Furthermore, the power conversion circuit converts the external power supply into the working power required for the internal circuit; the polarized light image sensor peripheral circuit is equipped with multiple polarized light grayscale image sensors, multiple polarized light RGGB sensors, and circuits to ensure the normal operation of the sensors; the external synchronization signal processing circuit converts the external synchronization signal of the differential or other level standards into the TTL level standard and outputs it to the FPGA circuit; the serial port conversion circuit converts the RS-422 or RS-232 level and the TTL level standard to realize the serial data reception and transmission between the user end and this system; the video interface circuit receives parallel digital video data and converts it into a video signal output in the SDI, cameralink, or CML interface format; the memory circuit includes an erasable read-only memory, a FLASH memory chip, and its peripheral circuits; the clock circuit includes a clock chip and its peripheral circuits.

[0008] Furthermore, the polarization sensor driving module encodes the configuration parameters of each register in each polarization sensor into serial data in I2C and SPI formats and sends them to the polarization image sensor. Since the configuration timing and data encoding can be programmably implemented, various sensors with non-standard interfaces can also be configured; the video acquisition module decodes the original video from the sensor according to the nominal video data format standard of the sensor, generates parallel video data and outputs it to the video processing module; the video processing module only uses black level correction, multi-channel video data merging, and cross overlay / blanking algorithm to process the parallel video data to retain more original polarization information obtained by the sensor, and adds one level to the image output by the polarization RGGB sensor. The interpolation algorithm reconstructs a 16-channel visible light polarized light color image; the synchronization and exposure control module runs an automatic / manual exposure algorithm, and uses an external synchronization signal and the time parameters calculated by the automatic / manual exposure algorithm to control the external trigger exposure of the polarization sensor, so as to realize independent exposure and synchronous output of each sensor; the serial port data encoding and decoding module executes the encoding and decoding of control instructions and return data between the user end and the FPGA; the parameter configuration and management module executes the initialization parameter configuration after power-on reset, saves the user-modified parameters, and restores the factory default parameters; the clock and reset management module converts the main clock input by the clock circuit into the clock signal required for the operation of each module inside the FPGA, and generates reset signals for each polarization light sensor and each module inside the FPGA.

[0009] The second technical solution adopted by the present invention is: based on the multi-channel fusion method of the above-mentioned polarization camera, it is realized by the following steps: SA1: The user end provides power and external synchronization signals for the camera; SA2: The camera's clock circuit oscillates to generate the main clock, which excites the clock and reset management modules to generate the clock and reset signals required by the polarization light sensor and other modules inside the FPGA that meet the timing requirements; SA3: The parameter configuration and management module reads parameter data from the memory and uses it to configure the registers of the sensor and all other modules inside the FPGA except the parameter configuration and management module; SA4: The synchronization and exposure control module of the camera calculates the average value of the grayscale data of a frame by continuous acquisition in each frame or in intervals of multiple frames, and generates the exposure parameter I using the automatic / manual exposure algorithm exp ; and continuously sample the external synchronization signal period T input by the user end to the camera frame , and generate an external trigger exposure signal XTRIG through the two calculations and input it to the polarization sensor. Through the iterative method, mean converges to the target grayscale value MEAN_TARGET; at the same time, the video acquisition module frameThe original video data output by multiple polarization sensors is collected once in a cycle, and then the multi-channel data is synchronously processed with black level correction and cross-grained superposition image processing; during the processing, the one-channel video image output by the polarized light RGGB sensor is processed by an additional level of interpolation and reconstruction algorithm, and the polarized light grayscale sensor video data stream needs to flow in and out of the interpolation and reconstruction algorithm module to ensure complete synchronization with other channel video streams, and the digital video generated is finally sent to the video output module; SA5: Fuse the data of polarized light images of each channel, perform non-compressed encoding on the digital video according to the interface protocol standard, and then send it to the video interface circuit for output.

[0010] Furthermore, in the synchronization and exposure control module, the automatic / manual exposure algorithm and the synchronization control algorithm work together, and the implementation steps of the automatic / manual exposure algorithm are as follows: SC1: Set the user-side external synchronization signal period to The next n+1th external synchronization signal period is set to T frame_n+1 , and so on frame_n+2 ...; Readout time T for each frame readout is a constant, and the delay time between the end of exposure and pixel readout is set to T delay , and assume that T delay =0; SC2: The automatic / manual exposure algorithm of sensor n generates exposure parameters independently in each external synchronization signal cycle, and sets the exposure parameters generated in the nth cycle as T expN_n ; SC3: Sensor n in T frame_n The exposure parameter T of the previous cycle in the cycle expN_n For reference, at T frame_n -T expN_n Start exposure independently at any time. frame_n The exposure ends at the cut-off time and at the next T frame_n Start time to output the original video.

[0011] Furthermore, the interpolation and reconstruction algorithm of the original video data of the polarized light RGGB sensor is completed by a video processing module, and its implementation steps are as follows: SD1: Building a mathematical model A two-dimensional model is established based on the microarray arrangement of polarization and RGGB Bayer filter coupling, where each pixel is a sampling point; before reconstruction, each sampling point (i, j) only has one-dimensional chromaticity information P of a polarization angle. (i，j) , where i is the number of rows, i∈[0~maximum number of rows); j is the number of columns, j∈[0~maximum number of columns); P (i，j)is the original grayscale data of the sampling point (i, j); after reconstruction, interpolation generates a combined color component {R_d5 (i，j) , G_d5 (i，j) , B_d5 (i，j)}; where R_d5 (i，j) 、G_d5 (i，j) , B_d5 (i，j) They are the red interpolation component, green interpolation component, and blue interpolation component of the sampling point (i, j) respectively; SD2: Sliding Window 9 lines of images are cached through the FPGA's FIFO buffer or random access memory, and 9 pixels are cached in each line using the REG register, and a 9×9 matrix of each pixel is generated in a pipeline manner; the center point of the matrix is ​​the current sampling point (i, j), the row coordinates of adjacent rows are i-1 and i+1, the column coordinates of adjacent columns are j-1 and j+1, and so on; SD3: The pipeline processing method of FPGA is used to match the 9×9 matrix of each pixel with the model one by one in real time, and the red, green and blue color space images of polarized light are reconstructed by interpolation using the following formula. The process is as follows: (1) The green interpolation components of the green component sampling points Gr and Gb are their original grayscale data P (i，j) , that is, G_d5 (i，j) =P (i，j) ; The red interpolation component of the red component sampling point R is its original grayscale data P (i，j) , that is, R_d5 (i，j) =P (i，j) ; The blue interpolation component of the blue component sampling point B is its original grayscale data P (i，j) , which is B_d5 (i，j) =P (i，j) ; (2) Reconstruct the green interpolation component G_d5 of the red component sampling point R and the blue component sampling point B based on the original grayscale data of the known green component sampling points Gr and Gb (i，j) ; First, use Formula 1 and Formula 2 to calculate Δh and Δv of the red component sampling point R and the blue component sampling point B at polarization angles of 0°, 45°, 90°, and 135° respectively: Δh=|P (i,j-2) -P (i,j+2) |+|2P (i,j) -P (i,j-4) -P (i,j+4) | (1) Δv=|P (i-2,j) -P (i+2,j) |+|2P (i,j) -P (i-4,j)-P (i+4,j) | (2) Where P (i,j) is the original grayscale data of the current sampling point (i, j) before interpolation and reconstruction, P (i,j-2) , P (i,j+2) , P (i-2,j) and P (i+2,j) are the original grayscale data of sampling points (i, j-2), (i, j+2), (i-2, j) and (i+2, j) before interpolation and reconstruction, P (i,j-4) , P (i,j+4) , P (i-4,j) and P (i+4,j) are the original grayscale data of sampling points (i, j-4), (i, j+4), (i-4, j) and (i+4, j) before interpolation and reconstruction; according to the mathematical model, if (i, j) is the red component sampling point, then P (i,j) , P (i,j-4) , P (i,j+4) , P (i-4,j) and P (i+4,j) | are all red components, P (i,j-2) , P (i,j+2) , P (i-2,j) and P (i+2,j) is the green component; if (i, j) is the blue component sampling point, then P (i,j) , P (i,j-4) , P (i,j+4) , P (i-4,j) and P (i+4,j) All are blue components, P (i,j-2) , P (i,j+2) , P (i-2,j) and P (i+2,j) is the green component; Then compare the size of Δh and Δv, and calculate the green interpolation component G_d5 of the red component sampling point R and the blue component sampling point B by formula 3 (i，j) : (3) synchronously calculate the residual components at other sampling points; Calculate the red interpolation component R_d5 of the green component sampling point Gb: Calculate the blue interpolation component B_d5 of the green component sampling point Gb: Calculate the red interpolation component R_d5 of the green component sampling point Gr: Calculate the blue interpolation component B_d5 of the green component sampling point Gr: Calculate the red interpolation component R_d5 of the blue component sampling point B: Δh=|P (i-2,j-2) -P (i+2,j+2) |+|2G_d5 (i,j) -G_d5 (i-2,j-2) -G_d5 (i+2,j+2) | (8) Δv=|P (i+2,j-2) -P (i-2,j+2) |+|2G_d5 (i,j) -G_d5 (i+2,j-2) -G_d5 (i-2,j+2) | (9) Calculate the blue component B_d5 of the red component sampling point R: Δh=|P (i-2,j-2) -P (i+2,j+2) |+|2G_d5 (i,j) -G_d5 (i-2,j-2) -G_d5 (i+2,j+2) | (11) Δv=|P (i+2,j-2) -P (i-2,j+2) |+|2G_d5 (i,j) -G_d5 (i+2,j-2) -G_d5 (i-2,j+2) | (12)

[0012] Compared with the prior art, the present invention adopts the focal plane polarization imaging technology to collect multiple groups of 4-channel near-infrared polarized light image information and multiple groups of 16-channel visible light polarized light image information in real time and synchronously, and realizes automatic exposure of 16-channel images through a synchronization method. Each image sensor can not only independently and automatically expose according to its own adapted exposure parameters, but also output image data synchronously; the FPGA pipeline sliding window method is used to realize interpolation and reconstruction of 16-channel color visible light polarized images, splice and fuse polarized light image data of multiple channels, and output them to the user end through a video interface for processing after encoding. The specific advantages are as follows: 1. Multiple polarized light RGGB sensors work together to obtain not only brightness and color information, but also polarized light information that is difficult for traditional image sensors to perceive; 2. An improved external synchronization method is proposed, which can not only converge the grayscale values ​​of each channel of video data to different target grayscale values ​​through independent exposure, but also eliminate the grayscale deviation from the sensor source, and ensure the pixel-level synchronization of the video image; taking the commonly used 74.25MHz pixel clock as an example, the synchronization phase difference of the multi-channel video image is less than 1 pixel clock cycle, about 0.13468ns; 3. The focal plane polarized light RGGB image sensor simultaneously outputs 16 channels of light polarization information. The defects such as image boundary grid points and pseudo-color caused by the nearest neighbor linear interpolation method are particularly obvious. The present invention provides an interpolation method that can effectively reduce the defects and can synchronously separate RGB images of 0°, 45°, 90° and 135° with low delay. The interpolation processing time is 8 lines of pixel clock cycles. Taking an image with 1080P resolution, 30Hz frame rate and 74.25MHz pixel clock frequency as an example, the delay is about 237μs. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the system framework and data flow of the present invention; Figure 2 is a flow chart of the multi-channel fusion method of the present invention; Figure 3 This is a schematic diagram of the timing of related signals of the polarization light sensor; Figure 4 Schematic diagram of the arrangement of polarized light RGGB sensor microarray; Figure 5 A schematic diagram of a 16-channel microarray arrangement; Figure 6 Schematic diagram of the model with the red component of 0° polarized light as the current pixel; Figure 7 Schematic diagram of the model with the Gr green component of 0° polarized light as the current pixel; Figure 8 Schematic diagram of the model with the blue component of 0° polarized light as the current pixel; Fig. 9 Schematic diagram of the model with the Gb green component of 0° polarized light as the current pixel; Fig.10 Schematic diagram of the model with the red component of 45° polarized light as the current pixel; Fig.11 Schematic diagram of the model with the Gr green component of 45° polarized light as the current pixel; Fig.12 Schematic diagram of the model with the blue component of 45° polarized light as the current pixel; Fig.13 Schematic diagram of the model with the Gb green component of 45° polarized light as the current pixel; Fig.14 Schematic diagram of a model with the red component of 90° polarized light as the current pixel; Fig.15 Schematic diagram of the model with the Gr green component of 90° polarized light as the current pixel; Fig.16Schematic diagram of a model with the blue component of 90° polarized light as the current pixel; Fig.17 Schematic diagram of a model with the Gb green component of 90° polarized light as the current pixel; Fig.18 Schematic diagram of the model with the red component of 135° polarized light as the current pixel; Fig.19 Schematic diagram of the model with the Gr green component of 135° polarized light as the current pixel; Fig. 20 Schematic diagram of the model with the blue component of 135° polarized light as the current pixel; Fig.21 Schematic diagram of a model with the Gb green component of 135° polarized light as the current pixel; Fig. 22 Generate a schematic diagram for the sliding window. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings.

[0015] Embodiment 1: like Figure 1 As shown, a multi-channel fusion polarization camera based on FPGA in this embodiment includes a power conversion circuit, a polarization light image sensor peripheral circuit, an external synchronization signal processing circuit, a serial port conversion circuit, a video interface circuit, a memory circuit, a clock circuit, and an FPGA circuit. The functions of the above hardware circuits are as follows: (1) Power conversion circuit: converts external power into the working power required for the internal circuit to operate; (2) Polarized light image sensor peripheral circuit: equipped with multiple polarized light grayscale image sensors, multiple polarized light RGGB sensors, and circuits to ensure the normal operation of the sensors; (3) External synchronization signal processing circuit: converts the differential or other level standard external synchronization signal into TTL (Transistor-Transistor Logic) level standard and outputs it to the FPGA circuit; (4) Serial port conversion circuit: converts RS-422 or RS-232 level to TTL level standard to realize serial data transmission and reception between the user end and this system; (5) Video interface circuit: receives parallel digital video data and converts it into video signal output in formats such as SDI, cameralink, and CML; (6) Memory circuit: including erasable read-only memory, FLASH and other memory chips and their peripheral circuits; (7) Clock circuit: including clock chip and its peripheral circuits; (8) FPGA circuit: As the core processing device of the entire camera, it is interconnected with circuits (2) to (7). The program running on the FPGA chip contains 6 modules. The functions of each module are as follows: a. Polarization sensor driver module: encodes the configuration data of each register in each polarization sensor into serial data in the format of I2C, SPI, etc. and sends it to the polarization image sensor. Since the configuration timing and data encoding can be programmed, it can also configure sensors with various non-standard interfaces; b. Video acquisition module: decodes the original video from the sensor according to the sensor's nominal video data format standard, generates parallel video data and outputs it to the video processing module; c. Video processing module: Only black level correction, multi-channel video data merging, and cross superposition / blanking algorithm are used for parallel video data to retain more original polarization information obtained by the sensor. For one channel of image output by the polarized light RGGB sensor, a first-level interpolation algorithm is added to reconstruct a 16-channel visible light polarized light color image; d. Synchronization and exposure control module: runs automatic / manual exposure algorithm, uses external synchronization signal and time parameters calculated by automatic exposure algorithm to control external trigger exposure of polarization sensor, and realizes independent exposure and synchronous output of each sensor; e. Serial port data encoding and decoding module: executes the encoding and decoding of control instructions and return data between the user end and FPGA; f. Parameter configuration and management module: perform initialization parameter configuration after power-on reset, save user-modified parameters, and restore factory default parameters; g. Clock and reset management module: converts the main clock input by the clock circuit into the clock signal required for the operation of each module inside the FPGA, and generates reset signals for each polarization light sensor and each module inside the FPGA.

[0016] Embodiment 2: like Figure 2 As shown, the multi-channel fusion method based on FPGA polarization camera of this embodiment is implemented by the following steps: SA1: The user end provides power and external synchronization signals for the camera; SA2: The camera's clock circuit oscillates to generate the main clock, which excites the clock and reset management modules to generate the clock and reset signals required by the polarization light sensor and other modules inside the FPGA that meet the timing requirements; SA3: The parameter configuration and management module reads parameter data from the memory and uses it to configure the registers of the sensor and all other modules inside the FPGA except the parameter configuration and management module; SA4: The synchronization and exposure control module of the camera calculates the average value of the grayscale data of a frame by continuous acquisition in each frame or in intervals of multiple frames, and generates the exposure parameter T using the automatic / manual exposure algorithm exp ; and continuously sample the external synchronization signal period T input by the user end to the camera frame , and generate an external trigger exposure signal XTRIG through the two calculations and input it to the polarization sensor. Through the iterative method, mean converges to the target grayscale value MEAN_TARGET; at the same time, the video acquisition module frame The original video data output by multiple polarization sensors is collected once in a cycle, and then the multi-channel data is synchronously processed with black level correction and cross-grained superposition image processing; during the processing, the one-channel video image output by the polarized light RGGB sensor is processed by an additional level of interpolation and reconstruction algorithm, and the polarized light grayscale sensor video data stream needs to flow in and out of the interpolation and reconstruction algorithm module to ensure complete synchronization with other channel video streams, and the digital video generated is finally sent to the video output module; SA5: Fuse the data of polarized light images of each channel, perform non-compressed encoding on the digital video according to the interface protocol standard, and send it to the video interface circuit for output; During steps SA1 to SA5 , the user end can modify parameter data through the serial port control parameter configuration and management module, and can also store parameter data into the memory or read parameter data from the memory.

[0017] like Figure 3 As shown, in the synchronization and exposure control module, the automatic / manual exposure algorithm and the synchronization control algorithm work together. Taking two polarized light sensors as an example, the implementation steps are as follows: SC1: Set the user-side external synchronization signal period to The next n+1th external synchronization signal period is set to T frame_n+1 , and so on frame_n+2 ...; Readout time T for each frame readout is a constant, and the delay time between the end of exposure and pixel readout is set to T delay , and assume that T delay =0; SC2: The automatic / manual exposure algorithms of sensor 1 and sensor 2 independently generate exposure parameters in each external synchronization signal cycle, and set the exposure parameters generated in the nth cycle to T exp1_n and T exp2_n ; SC3: Sensor 1 and Sensor 2 at T frame_n The exposure parameters T of the previous cycle are used in each cycle. exp1_n and T exp2_n For reference, respectively, at T frame_n -Texp1_n and T frame_n -T exp2_n Start exposure independently at any time. frame_n The exposure ends at the cut-off time and at the next T frame_n Start time sync output of original video.

[0018] The interpolation and reconstruction algorithm of the raw video data of the polarized light RGGB sensor is completed by the video processing module. The interpolation and reconstruction method of the raw video data is based on the split-focal-plane polarized light RGGB sensor, whose on-chip polarized light microarray and RGGB Bayer filter coupled microarray are arranged as follows Figure 4 As shown, Figure 5 For a set of 16-channel microarrays, the implementation steps of the interpolation reconstruction algorithm are as follows: SD1: Building a mathematical model like Figure 6-21 As shown in the figure, a two-dimensional model is established based on the microarray arrangement of polarization and RGGB Bayer filter coupling. Each pixel is a sampling point, and the red, green, and blue sampling points are arranged alternately at intervals. Before reconstruction, each sampling point only has one-dimensional chromaticity information of one polarization angle, that is, the original grayscale data P. After reconstruction, a combined color component {R_d5, G d5, B_d5} of a polarization angle can be generated for each sampling point. The relevant parameters in the mathematical model are explained as follows: (1) Color component sampling points: The red component sampling point at a polarization angle of 0° is represented by R_0, the blue component sampling point is represented by B_0, and the green component sampling point is represented by Gr_0 or Gb_0; the color component sampling points at polarization angles of 45°, 90°, and 270° are represented in the same way; (2) The coordinates of the current sampling point (i, j), where i is the row number, i∈[0 to the maximum number of rows); j is the column number, j∈[0 to the maximum number of columns); (3) Gray value P(i, j) of sampling point (i, j): the original gray data of the sampling point before interpolation and reconstruction; (4) Red interpolation component R_d5 of sampling point (i, j) (i，j) , green interpolation component G_d5 (i，j) , blue interpolation component B_d5 (i，j) : Color components generated by interpolation and reconstruction; If the current sampling point (i, j) is R_0, then the red interpolation component of the point is R_d5 (i，j) =P (i，j) , the green interpolation component G_d5 of this point needs to be reconstructed through the SD3 step (i，j) and the blue interpolation component B_d5 (i，j) ; SD2: Sliding Window 9 lines of images are cached through the FPGA's first-in-first-out buffer or random access memory, and 9 pixels are cached in each line using the REG register, and a 9×9 matrix of each pixel is generated in a pipelined manner, such as Fig. 22 As shown; the center point of the matrix is ​​the current sampling point (i, j), the row coordinates of adjacent rows are i-1 and i+1 respectively, the column coordinates of adjacent columns are j-1 and j+1 respectively, and so on; SD3: The pipeline processing method of FPGA is used to match the 9×9 matrix of each pixel with the model one by one in real time, and the red, green and blue color space images of polarized light are reconstructed by interpolation using the following formula. The process is as follows: (1) Green interpolation component G_d5 of the green component sampling points Gr and Gb at each polarization angle (i，j) =P (i，j) ; Red interpolation component R_d5 of red component sampling point R (i，j) =P (i，j) ; Blue interpolation component B_d5 of blue component sampling point B (i，j) =P (i，j) ; (2) Based on the known original grayscale data of the green component sampling points Gr and Gb, construct the green interpolation component G_d5 of the red component sampling point R and the blue component sampling point B (i，j) ; First, calculate the Δh and Δv of the sampling points of R_0, R_45, R_90, R_135 and B_0, B_45, B_90, B_135 respectively: Δh=|P (i，j-2) -P (i，j+2) |+|2P (i，j) -P (i，j-4) -P (i，j+4) |, Δv=|P (i-2，j) -P (i+2，j) |+|2P (i，j) -P (i-4，j) -P (i+4，j) |, Among them, P (i，j) is the original grayscale data of the current sampling point, that is, the red component or the blue component; P (i，j-2) , P (i，j+2) , P (i-2，j) and P (i+2，j) is the green component; P (i，j-4) , P (i，j+4) , P (i-4，j) and P (i+4，j) Color component attributes and P (i，j) Consistency; Then, by comparing the sizes of Δh and Δv, the green interpolation component at the red and blue component sampling points is calculated: (3) synchronously calculating the residual components at other pixel sampling points; Calculate the red interpolation component Rd5 and the blue interpolation component Bd5 at the green sampling point Gb: Calculate the red interpolation component R_d5 and the blue interpolation component B_d5 at the green sampling point Gr: Calculate the red interpolation component R_d5 at the blue sampling point: Δh=|P (i-2,j-2) -P (i+2,j+2) |+|2G_d5 (i,j) -G_d5 (i-2,j-2) -G_d5 (i+2,j+2) |, Δv=|P (i+2,j-2) -P (i-2,j+2) |+|2G_d5 (i,j) -G_d5 (i+2,j-2) -G_d5 (i-2,j+2) |, Calculate the blue interpolation component B_d5 at the red sampling point: Δh=|P (i-2,j-2) -P (i+2,j+2) |+|2G_d5 (i,j) -G_d5 (i-2,j-2) -G_d5 (i+2,j+2) |, Δv=|P (i+2,j-2) -P (i-2,j+2) |+|2G_d5 (i,j) -G_d5 (i+2,j-2) -G_d5 (i-2,j+2) |,

[0019] The present invention is not limited by the above-mentioned embodiments. The above-mentioned embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-channel fusion polarization camera based on FPGA, characterized in that: It includes a power conversion circuit, and a polarized light image sensor peripheral circuit interconnected with an FPGA circuit, an external synchronization signal processing circuit, a serial port conversion circuit, a video interface circuit, a memory circuit, and a clock circuit; the FPGA circuit serves as the core processing device of the camera, including a polarized light sensor drive module, a video acquisition module, a video processing module, a synchronization and exposure control module, a serial port data encoding and decoding module, a parameter configuration and management module, and a clock and reset management module.

2. The FPGA-based multi-channel fusion polarization camera according to claim 1, characterized in that: The power conversion circuit converts the external power supply into the working power required for the internal circuit to work; the polarized light image sensor peripheral circuit is equipped with multiple polarized light grayscale image sensors, multiple polarized light RGGB sensors, and circuits to ensure the normal operation of the sensors; the external synchronization signal processing circuit converts the external synchronization signal of the differential or other level standard into the TTL level standard and outputs it to the FPGA circuit; The serial port conversion circuit converts RS-422 or RS-232 level to TTL level standard to realize serial data transmission and reception between the user end and this system; the video interface circuit receives parallel digital video data and converts it into video signal output in SDI, cameralink, CML interface format; the memory circuit includes erasable read-only memory, FLASH memory chip and its peripheral circuits; the clock circuit includes clock chip and its peripheral circuits.

3. The FPGA-based multi-channel fusion polarization camera according to claim 2, characterized in that: The polarization light sensor driving module encodes the configuration parameters of each register in each polarization light sensor into serial data in I2C and SPI formats and sends them to the polarization light image sensor; the video acquisition module decodes the original video from the sensor according to the nominal video data format standard of the sensor, generates parallel video data and outputs it to the video processing module; the video processing module processes the parallel video data using black level correction, multi-channel video data merging, and cross superposition / blanking algorithm, and adds a first-level interpolation algorithm to reconstruct a 16-channel visible light polarization color image for one channel of image output by the polarization light RGGB sensor; the synchronization and exposure control The module runs the automatic / manual exposure algorithm, and uses the external synchronization signal and the time parameters calculated by the automatic / manual exposure algorithm to control the external trigger exposure of the polarization sensor, so as to realize independent exposure and synchronous output of each sensor; the serial port data encoding and decoding module executes the encoding and decoding of the control instructions and the returned data between the user end and the FPGA; the parameter configuration and management module executes the initialization parameter configuration after power-on reset, saves the user-modified parameters, and restores the factory default parameters; the clock and reset management module converts the main clock input by the clock circuit into the clock signal required for the operation of each module inside the FPGA, and generates reset signals for each polarization light sensor and each module inside the FPGA.

4. The multi-channel fusion method of the FPGA-based multi-channel fusion polarization camera according to claims 1-3, characterized in that: This method is implemented through the following steps: SA1: The user end provides power and external synchronization signals for the camera; SA2: The camera's clock circuit oscillates to generate the main clock, which excites the clock and reset management modules to generate the clock and reset signals required by the polarization light sensor and other modules inside the FPGA that meet the timing requirements; SA3: The parameter configuration and management module reads parameter data from the memory and uses it to configure the registers of the sensor and all other modules inside the FPGA except the parameter configuration and management module; SA4: The synchronization and exposure control module of the camera calculates the average value of the grayscale data of a frame by continuous acquisition in each frame or in intervals of multiple frames, and generates the exposure parameter T using the automatic / manual exposure algorithm exp ; and continuously sample the external synchronization signal period T input by the user end to the camera frame , and generate an external trigger exposure signal XTRIG through the two calculations and input it to the polarization sensor. Through the iterative method, the mean converges to the target grayscale value MEAN_TARGET; at the same time, the video acquisition module frame The original video data output by multiple polarization sensors is collected once in a cycle, and then the multi-channel data is synchronously processed with black level correction and cross-grained superposition image processing; during the processing, the one-channel video image output by the polarized light RGGB sensor is processed by an additional level of interpolation and reconstruction algorithm, and the polarized light grayscale sensor video data stream needs to flow in and out of the interpolation and reconstruction algorithm module to ensure complete synchronization with other channel video streams, and the digital video generated is finally sent to the video output module; SA5: Fuse the data of polarized light images of each channel, perform non-compressed encoding on the digital video according to the interface protocol standard, and then send it to the video interface circuit for output.

5. The multi-channel fusion method of the multi-channel fusion polarization camera based on FPGA according to claim 4, characterized in that: In the synchronization and exposure control module, the automatic / manual exposure algorithm and the synchronization control algorithm work together. The implementation steps of the automatic / manual exposure algorithm are as follows: SC1: Set the user-side external synchronization signal period to The next n+1th external synchronization signal period is set to T frame_n+1 , and so on frame_n+2 ...; Readout time T for each frame readout is a constant, and the delay time between the end of exposure and pixel readout is set to T delay , and assume that T delay =0; SC2: The automatic / manual exposure algorithm of sensor n generates exposure parameters independently in each external synchronization signal cycle, and sets the exposure parameters generated in the nth cycle as T expN_n ; SC3: Sensor n in T frame_n The exposure parameter T of the previous cycle in the cycle expN_n For reference, at T frame_n -T expN_n Start exposure independently at any time. frame_n The exposure ends at the cut-off time and at the next T frame_n Start time to output the original video.

6. The multi-channel fusion method of the multi-channel fusion polarization camera based on FPGA according to claim 5, characterized in that: The interpolation and reconstruction algorithm of the original video data of the polarized light RGGB sensor is completed by the video processing module, and its implementation steps are as follows: SD1: Building a mathematical model A two-dimensional model is established based on the microarray arrangement of polarization and RGGB Bayer filter coupling, where each pixel is a sampling point; before reconstruction, each sampling point (i, j) only has one-dimensional chromaticity information P of a polarization angle. (i，j) , where i is the number of rows, i∈[0~maximum number of rows); j is the number of columns, j∈[0~maximum number of columns); P (i，j) is the original grayscale data of the sampling point (i, j); after reconstruction, interpolation generates a combined color component {R_d5 (i，j) , G_d5 (i，j) , B_d5 (i，j) }; where R_d5 (i，j) 、G_d5 (i，j) , B_d5 (i，j) They are the red interpolation component, green interpolation component, and blue interpolation component of the sampling point (i, j) respectively; SD2: Sliding Window 9 lines of images are cached through the FPGA's FIFO buffer or random access memory, and 9 pixels are cached in each line using the REG register, and a 9×9 matrix of each pixel is generated in a pipeline manner; the center point of the matrix is ​​the current sampling point (i, j), the row coordinates of adjacent rows are i-1 and i+1, and the column coordinates of adjacent columns are j-1 and j+1; SD3: The pipeline processing method of FPGA is used to match the 9×9 matrix of each pixel with the model one by one in real time, and the red, green and blue color space images of polarized light are reconstructed by interpolation using the following formula. The process is as follows: (1) The green interpolation components of the green component sampling points Gr and Gb are their original grayscale data P (i，j) , that is, G_d5 (i，j) =P (i，j) ; The red interpolation component of the red component sampling point R is its original grayscale data P (i，j) , that is, R_d5 (i，j) =P (i，j) ; The blue interpolation component of the blue component sampling point B is its original grayscale data P (i，j) , which is B_d5 (i，j) =P (i，j) ; (2) Reconstruct the green interpolation component G_d5 of the red component sampling point R and the blue component sampling point B based on the original grayscale data of the known green component sampling points Gr and Gb (i，j) ; First, use Formula 1 and Formula 2 to calculate Δh and Δv of the red component sampling point R and the blue component sampling point B at polarization angles of 0°, 45°, 90°, and 135° respectively: Δh=|P (i，j-2) -P (i，j+2) |+|2P (i，j) -P (i，j-4) -P (i，j+4) | (1) Δv=|P (i-2，j) -P (i+2，j) |+|2P (i，j) -P (i-4，j) -P (i+4，j) | (2) Where P (i，j) is the original grayscale data of the current sampling point (i, j) before interpolation and reconstruction, P (i，j-2) , P (i，j+2) , P (i-2，j) and P (i+2，j) are the original grayscale data of sampling points (i, j-2), (i, j+2), (i-2, j) and (i+2, j) before interpolation and reconstruction, P (i，j-4) , P (i，j+4) , P (i-4，j) and P (i+4，j) are the original grayscale data of sampling points (i, j-4), (i, j+4), (i-4, j) and (i+4, j) before interpolation and reconstruction; according to the mathematical model, if (i, j) is the red component sampling point, then P (i,j) , P (i,j-4) , P (i,j+4) , P (i-4,j) and P (i+4,j) Both are red components, P (i,j-2) , P (i,j+2) , P (i-2,j) and P (i+2,j) is the green component; If (i, j) is a blue component sampling point, then P (i,j) , P (i,j-4) , P (i,j+4) , P (i-4,j) and P (i+4,j) All are blue components, P (i,j-2) , P (i,j+2) , P (i-2,j) and P (i+2,j) is the green component; Then compare the size of Δh and Δv, and calculate the green interpolation component G_d5 of the red component sampling point R and the blue component sampling point B by formula 3 (i，j) : (3) synchronously calculate the residual components at other sampling points; Calculate the red interpolation component R_d5 of the green component sampling point Gb: Calculate the blue interpolation component B_d5 of the green component sampling point Gb: Calculate the red interpolation component R_d5 of the green component sampling point Gr: Calculate the blue interpolation component B_d5 of the green component sampling point Gr: Calculate the red interpolation component R_d5 of the blue component sampling point B: Δh=|P (i-2,j-2) -P (i+2,j+2) |+|2G_d5 (i,j) -G_d5 (i-2,j-2) -G_d5 (i+2,j+2) | (8) Δv=|P (i+2,j-2) -P (i-2,j+2) |+|2G_d5 (i,j) -G_d5 (i+2,j-2) -G_d5 (i-2,j+2) | (9) Calculate the blue component B_d5 of the red component sampling point R: Δh=|P (i-2,j-2) -P (i+2,j+2) |+|2G_d5 (i,j) -G_d5 (i-2,j-2) -G_d5 (i+2,j+2) | (11) Δv=|P (i+2,j-2) -P (i-2,j+2) |+|2G_d5 (i,j) -G_d5 (i+2,j-2) -G_d5 (i-2,j+2) | (12)

Images