True color imaging method based on prism splitting 3CMOS camera

Abstract

The invention provides a true color imaging method based on a prism light splitting 3CMOS camera, three prism light splitting channels are respectively received by three image sensors, and the following steps are realized based on an FPGA: prism light splitting non-uniformity correction of RGB three channels, registration of three-channel images, synchronous splicing of three-channel pixel data, and application of an automatic white balance algorithm to color restoration. According to the invention, real-time registration, reconstruction and image processing of images acquired by the prism splitting 3CMOS camera are realized, and higher color precision and better imaging quality are brought.

Description

Technical Field

[0001] The present invention belongs to the technical field of color camera imaging, and specifically relates to a true color imaging method based on a prism-splitting 3CMOS camera. Background Art

[0002] In a traditional Bayer filter array, the color filters on each pixel block block the other two color components from reaching the photosensitive area, and only one-third of the photons can pass through the filter array. In contrast, the prism-splitting method uses three image sensors to greatly improve the energy utilization rate of the optical window. Its photosensitive area is three times that of a single-sensor camera, and the light signal intensity is retained as much as possible. Due to its optical structure, the prism-splitting method naturally has true color accuracy. The three-channel information is collected by separate image sensors, enabling each pixel to capture true color information with a complete bit depth. True color information can reduce color differences, avoid false color display, reduce heterogeneity, and bring better image contrast. The Bayer filter array estimates colors through interpolation calculations, and the demosaicing process often leads to false presentation of image quality. The estimated colors are difficult to accurately restore the object colors compared to the true colors collected by prism splitting.

[0003] The prism-splitting method can achieve better spectral separation and has the characteristic of low color crosstalk. The Bayer filter array is composed of color dyes or pigments. Due to the characteristics of these materials themselves, blue extends into green and red, green extends into blue and red, and red extends into green, resulting in color information crosstalk. The characteristics of CMOS sensors can cause color crosstalk when photons falling on a pixel are mis-sensed by the surrounding pixels. The erroneously responded pixel values are then used for interpolation calculations to demosaic the array, which will further cause the influence of color crosstalk. The 3CMOS prism camera can perform independent gain and exposure control for three channels, and each color channel can be optimally adjusted in exposure time and digital gain, enabling each channel to achieve the best signal-to-noise ratio. Summary of the Invention

[0004] The present invention proposes a true color imaging method based on a prism-splitting 3CMOS camera, which realizes real-time registration, reconstruction, and color restoration of the images collected by the prism-splitting 3CMOS camera, bringing higher color accuracy and richer image details.

[0005] The technical solution for achieving the object of the present invention is: a true color imaging method based on a prism-splitting 3CMOS camera, comprising the following steps:

[0006] Step 1: Construct a prism-splitting 3CMOS camera imaging system;

[0007] Step 2: Based on the FPGA, perform prism splitting non-uniformity correction on the RGB three-channel pixel data respectively;

[0008] Step 3: Based on the FPGA, perform registration of the three-channel images;

[0009] Step 4: Use the FPGA to construct a three-channel pixel data synchronization module. The three-channel pixel data synchronization module includes a data latch module, a three-channel pixel data cache module, and a line-field synchronization timing generation module. Use the data latch module to latch the registered image data; use the three-channel pixel data cache module to cache the latched registered image data. The line-field synchronization timing generation module generates the line-field synchronization signal timing after the three-channel pixel data is synchronized. Use the line-field synchronization signal timing after the three-channel pixel data is synchronized to synchronously read out the registered image data from the three-channel pixel data cache module, and splice the three-channel registered image data in the order of RGB three channels to obtain the registered RGB image;

[0010] Step 5: Based on the FPGA, design an automatic white balance module, and use the automatic white balance module to perform color restoration on the registered RGB image.

[0011] Preferably, the specific process of constructing a prism splitting 3CMOS camera imaging system is as follows:

[0012] Step 1.1: Use a beam splitting prism to separate visible light into three bands of red, green, and blue, which are respectively received by three image sensors;

[0013] Step 1.2: Use a slide rail to adjust the position of the image sensor to focus the image sensor's picture. Manually adjust the image sensor according to the focal length to focus the image sensor's picture to the best state respectively, and fix the position of the image sensor on the slide rail;

[0014] Step 1.3: Use a connector wire to connect the three CMOS image sensor adapter boards to the FPGA image processing board, and use a Cameralink wire to connect the FPGA image processing board to the Cameralink image acquisition card. Use the three image sensors to collect image data. The collected image data passes through the three CMOS image sensor adapter boards, the FPGA image processing board, and the Cameralink image acquisition card, and is finally collected and displayed on the host computer.

[0015] Preferably, the specific steps for performing prism splitting non-uniformity correction on the RGB three-channel pixel data based on the FPGA are as follows:

[0016] Step 2.1: Remove the prism camera lens, use an integrating sphere to generate a uniform light source to irradiate the prism, and the prism splitting 3CMOS camera imaging system collects three-channel non-uniformity grayscale images;

[0017] Step 2.2: Calculate the spatial average signal of the three-channel non-uniformity grayscale image respectively

[0018]

[0019] where R i,j (φ) is the grayscale value of the pixel at the coordinate (i, j) of the three-channel non-uniformity grayscale image, M is the number of horizontal pixels of the three-channel non-uniformity grayscale image, and N is the number of vertical pixels of the three-channel non-uniformity grayscale image;

[0020] Step 2.3: Substitute into the correction formula to solve the correction gain coefficient P of the (i, j)th pixel ij , and the specific formula is:

[0021]

[0022] After solving each of the M×N pixels one by one, the non-uniformity correction coefficient matrix P of this image can be obtained;

[0023]

[0024] The non-uniformity correction coefficient matrices of the RGB three channels are calculated respectively through the above steps.

[0025] Step 2.4: According to the up-down symmetry of the non-uniformity, calculate the average value of the values in the same column of the non-uniformity correction coefficient matrix P as the correction coefficient of the corresponding column. Construct a correction coefficient ROM in the FPGA to store the correction coefficient, obtain the ROM read address generation module according to the video line timing for pixel counting, and the ROM read address generation module generates a read address to read out the correction coefficient one by one, and perform image multiplication with the input image pixels in real time to obtain the corrected three-channel unregistered image.

[0026] Preferably, the registration of the three-channel image is performed based on the FPGA, and the specific steps are as follows:

[0027] Step 3.1: Adopt the method of calibration and then cropping. Take a cross-shaped standard point in the three-channel unregistered image as the calibration reference point. Taking one of the channels as the reference, record the horizontal and vertical coordinate displacements of the center pixels of the cross marks in the other two channels relative to the center pixel of the cross mark in the reference channel respectively, and determine the image range where the three channels coincide based on this;

[0028] Step 3.2: Write a row counting module and a column counting module on the FPGA to determine the horizontal and vertical coordinates of the pixels. Construct a cropping range judgment module through combinational logic, and use the cropping range judgment module to limit the cropping of the image range where the three channels coincide. Each of the three channels has a cropping module, so as to obtain the registered image of the three channels according to the image range where the three channels coincide.

[0029] Compared with the prior art, the significant advantages of the present invention are as follows: (1) The present invention realizes true-color imaging based on the prism spectroscopy method, with advantages such as high light energy utilization rate, true color accuracy, and low color crosstalk. (2) Based on FPGA, the non-uniformity correction of prism spectroscopy is realized, eliminating the problem of local color deviation in imaging caused by uneven three-channel spectroscopy. (3) Based on FPGA, real-time registration of three channels and synchronous stitching of registered images are realized, eliminating the color edges and halos caused by image misalignment due to assembly errors of three CMOS cameras. (4) Based on FPGA, an automatic white balance algorithm is realized to reproduce the true color of the target object.

[0030] The following further describes the present invention in detail with reference to the accompanying drawings. Description of the Drawings

[0031] Figure 1 It is the system principle structure diagram of the present invention.

[0032] Figure 2 It is the structure block diagram of the FPGA image processing algorithm design and implementation of the present invention.

[0033] Figure 3 It is the effect diagram of the FPGA implementation of the non-uniformity correction of prism spectroscopy of the present invention.

[0034] Figure 4 It is the principle diagram of the registration of the cropped image of the present invention.

[0035] Figure 5 It is the effect diagram of the FPGA implementation of the three-channel image registration of the present invention.

[0036] Figure 6 It is the effect diagram of the FPGA implementation of the automatic white balance algorithm of the present invention. Detailed Implementation Manner

[0037] To further explain the technical solution of the present invention, the present invention is described in conjunction with the accompanying drawings.

[0038] The specific steps of a true-color imaging method based on a prism spectroscopy 3CMOS camera are as follows:

[0039] Step 1: Construct a prism spectroscopy 3CMOS camera imaging system;

[0040] Step 1.1: As shown in the overall system principle diagram, a spectroscopic prism is used to separate visible light into three bands of red, green, and blue, which are respectively received by three image sensors; Figure 1 Step 1.2: Use a slide rail to adjust the position of the image sensor to focus the image sensor picture. Manually adjust the image sensor according to the focal length to focus the image sensor picture to the best state respectively, and fix the position of the image sensor on the slide rail.

[0041] ​

[0042] Step 1.3: Connect three CMOS image sensor adapter boards to the FPGA image processing board using a connector cable, connect the FPGA image processing board to the Cameralink image acquisition card using a Cameralink cable, use three image sensors to collect image data, and the collected image data passes through the three CMOS image sensor adapter boards, the FPGA image processing board, and the Cameralink image acquisition card, and is finally collected and displayed on the host computer. This step transfers the image data collected by the three image sensors into the FPGA image processing board with FPGA as the core device, providing conditions for subsequent real-time image processing using FPGA.

[0043] Step 2: Based on the FPGA, perform prism splitting non-uniformity correction on the RGB three-channel pixel data respectively;

[0044] Step 2.1: Figure 2 Design and implement a structural block diagram for the FPGA image processing algorithm. Due to the introduction of the prism, the three-channel splitting is non-uniform, resulting in local color deviation on the left and right of the image. To solve this problem, the process of prism splitting non-uniformity correction can be expressed as follows: First, disassemble the prism camera lens, use an integrating sphere to generate a uniform light source to irradiate the prism, and the prism splitting 3CMOS camera imaging system collects three-channel non-uniformity grayscale images;

[0045] Step 2.2: For the same irradiance φ, the output signal Y of each detector should be equal after non-uniform correction. Let the spatially averaged signal be Calculate the spatially averaged signals of the three-channel non-uniformity grayscale images respectively

[0046]

[0047] where R i,j (φ) is the grayscale value of the pixel at the coordinate (i, j) of the three-channel non-uniformity grayscale image, M is the number of horizontal pixels of the three-channel non-uniformity grayscale image, and N is the number of vertical pixels of the three-channel non-uniformity grayscale image.

[0048] Step 2.3: Substitute into the correction formula to solve the correction gain coefficient P of the (i, j)th pixel ij , and the specific formula is:

[0049]

[0050] After solving each of the M×N pixels one by one, the non-uniformity correction coefficient matrix P of this image can be obtained;

[0051]

[0052] For each of the RGB three channels, the non-uniformity correction coefficient matrix is calculated respectively through the above steps.

[0053] Step 2.4: According to the up-down symmetry of non-uniformity, calculate the average value of the values in the same column of the non-uniformity correction coefficient matrix P as the correction coefficient for the corresponding column. Construct a correction coefficient ROM in the FPGA to store the correction coefficients. Generate a ROM read address generation module based on the video line timing for pixel counting. The ROM read address generation module generates read addresses to read out the correction coefficients one by one, and perform image multiplication with the input image pixels to obtain the corrected three-channel unregistered image Y in real time. ij This non-uniformity correction algorithm performs the compensation operation of the correction coefficient matrix frame by frame for each frame of the three-channel input image. The corrected effect image is as Figure 3 shown.

[0054] Step 3: Perform registration of the three-channel images based on the FPGA;

[0055] Step 3.1: Adopt the method of calibration and then cropping. Take a cross-shaped standard point in the three-channel unregistered image Y ij as the calibration reference point, such as the cross-shaped standard point in Figure 4 . Take one channel as the reference, and record the horizontal and vertical coordinate displacements of the center pixels of the cross marks in the other two channels relative to the center pixels of the cross mark in the reference channel respectively. Determine the image range where the three channels coincide based on this.

[0056] Step 3.2: Write a row counting module and a column counting module on the FPGA to determine the horizontal and vertical coordinates of the pixels. Construct a cropping range judgment module through combinational logic, and use the cropping range judgment module to limit the cropping of the image range where the three channels coincide. Each of the three channels has a cropping module, so as to obtain the image after registration of the three-channel images according to the image range where the three channels coincide. The cropping start coordinates and the cropping frame size can be configured through the top-level parameter interface of this module. This registration method effectively eliminates the color halo and color fringes caused by the misalignment of the three channels due to assembly errors in real time, and improves the imaging quality. The FPGA implementation effect diagram is as Figure 5 shown.

[0057] Step 4: Use the FPGA to construct a three-channel pixel data synchronization module, and the three-channel pixel data synchronization module includes a data latch module, a three-channel pixel data cache module, and a line-field synchronization timing generation module.

[0058] Step 4.1: Use the data latch module to latch the registered image data, reduce data errors caused by signal jitter or timing problems, and improve the stability of the system;

[0059] Step 4.2: Use a three-channel pixel data cache module to cache the registered image after latching. The three channels respectively use FIFOs to construct the three-channel pixel data cache module, select the First Word Fall Through readout mode. The write bit width of the FIFO is the pixel data bit width of 12 bits, and the write depth is set to 16384. Taking the number of pixels in one row as 1920 as an example, the FIFO depth of 16384 can accommodate at least 8 rows of pixel data caching, leaving sufficient margin for the arrival time of the first row of pixel data of the three-channel images being different due to image row cropping during registration.

[0060] Step 4.3: The line-field synchronization timing generation module generates the line-field synchronization signal timing after synchronizing the three-channel pixel data, and uses the line-field synchronization signal timing after synchronizing the three-channel pixel data to synchronously read out the registered image data from the three-channel pixel data cache module. Taking the channel with the most row cropping (i.e., the channel where the first row of pixel data arrives the slowest) as the benchmark, the pixel count of one row is used as the core constraint condition for writing and reading. Write a state machine to generate the FIFO read enable signal, and use this read enable signal to simultaneously read out the data cached in the three-channel FIFOs, thus synchronously aligning the three-channel pixel data.

[0061] Step 4.4: Finally, splice the three-channel registered image data in the order of RGB three channels to obtain the registered RGB image;

[0062] Step 5: Design an automatic white balance module based on FPGA, and use the automatic white balance module to perform color restoration on the registered RGB image. The automatic white balance module includes a gain coefficient calculation module, an RGB color correction module, and a line-field timing buffering module.

[0063] The gain coefficient calculation module calculates the gain coefficient according to the registered RGB image, which is implemented using a divider on the FPGA. The specific formula is:

[0064]

[0065] where R sum 、G sum 、B sum are respectively the sum of the gray values of the three channels of the registered RGB image, and K r 、K g 、K b are respectively the correction coefficients of the three channels of the registered RGB image.

[0066] The RGB color correction module corrects the coefficients K r 、K g 、K bPerform RGB color correction on the next frame of the image, and perform image multiplication on each of the three channels with its own correction coefficient. The correction process is as follows:

[0067] R * = K r × R

[0068] G * = K r × G

[0069] B * = K r × B

[0070] Wherein, R * , G * , B * are the pixel gray values of the three corrected channels respectively, and R, G, and B are the pixel gray values of the three channels before correction respectively.

[0071] There is a delay in the divider and the correction calculation. It is necessary to delay and beat the line and field timing. Delay and beat the line and field timing on the FPGA, and delay synchronization with the output of the pixel data. Obtain the final output video image data, and finally display it in real time on the host computer through the encoding chip and the image acquisition card. The FPGA implementation effect diagram of the algorithm is as Figure 6 shown.

Claims

1. A true color imaging method based on prism spectroscopic 3CMOS camera, characterized in that: The steps include: Step 1: Build a prism spectrometer 3CMOS camera imaging system; Step 2: Based on FPGA, prism light splitting non-uniformity correction is performed on the RGB three-channel pixel data respectively; Step 3: Perform three-channel image registration based on FPGA; Step 4: Use FPGA to construct a three-channel pixel data synchronization module, the three-channel pixel data synchronization module includes a data latch module, a three-channel pixel data cache module, and a row-field synchronization timing generation module, and the data latch module is used to latch the registered image data; the three-channel pixel data cache module is used to cache the latched registered image data, and the row-field synchronization timing generation module generates a row-field synchronization signal timing after the three-channel pixel data is synchronized, and the row-field synchronization signal timing after the three-channel pixel data is synchronized is used to synchronously read out the registered image data from the three-channel pixel data cache module, and the three-channel registered image data is spliced ​​according to the RGB three-channel order to obtain a registered RGB image; Step 5: Design an automatic white balance module based on FPGA, and use the automatic white balance module to restore the color of the registered RGB image.

2. The true color imaging method based on prism spectrometry 3CMOS camera according to claim 1, characterized in that: The specific process of building a prism spectrometer 3CMOS camera imaging system is as follows: Step 1.1: Use a beam splitter prism to separate visible light into three bands: red, green, and blue, which are received by three image sensors respectively; Step 1.2: Use the slide rail to adjust the position of the image sensor to focus the image sensor screen, manually adjust the image sensor according to the focal length to focus the image sensor screen to the best state, and fix the position of the image sensor on the slide rail; Step 1.3: Use connector cables to connect the three CMOS image sensor adapter boards to the FPGA image processing board, and use Cameralink cables to connect the FPGA image processing board to the Cameralink image acquisition card. Use the three image sensors to collect image data. The collected image data is transmitted through the three CMOS image sensor adapter boards, the FPGA image processing board, and the Cameralink image acquisition card, and finally collected to the host computer for display.

3. The true color imaging method based on prism spectrometry 3CMOS camera according to claim 1, characterized in that: Based on FPGA, prism light splitting non-uniformity correction is performed on the RGB three-channel pixel data respectively. The specific steps are as follows: Step 2.1: Disassemble the prism camera lens, use an integrating sphere to generate a uniform light source to illuminate the prism, and use the prism spectrometer 3CMOS camera imaging system to collect three-channel non-uniform grayscale images; Step 2.2: Calculate the spatial average signal of the three-channel non-uniform grayscale image respectively In the formula, R i,j (φ) is the gray value of the pixel at the coordinate (i, j) of the three-channel non-uniform grayscale image, M is the number of horizontal pixels of the three-channel non-uniform grayscale image, and N is the number of vertical pixels of the three-channel non-uniform grayscale image; Step 2.3: Substitute into the correction formula and solve the correction gain coefficient P of the (i, j)th pixel ij , the specific formula is: After solving the M×N pixels one by one, the non-uniformity correction coefficient matrix P of this image can be obtained; The three channels of RGB calculate their respective non-uniformity correction coefficient matrices through the above steps respectively. Step 2.4: According to the upper and lower symmetry of the non-uniformity, the values ​​in the same column of the non-uniformity correction coefficient matrix P are averaged as the correction coefficients of the corresponding columns. A correction coefficient ROM is constructed in the FPGA to store the correction coefficients. The pixel count is performed according to the video row timing to obtain the ROM read address generation module. The ROM read address generation module generates a read address to read out the correction coefficients one by one, and performs image multiplication with the input image pixels to obtain the corrected three-channel unregistered image in real time.

4. The true color imaging method based on prism spectrometry 3CMOS camera according to claim 1, characterized in that: The specific steps for three-channel image registration based on FPGA are as follows: Step 3.1: Take the calibration and then cropping method, take a cross-shaped standard point in the three-channel unregistered image as the calibration reference point, take one channel as the reference, record the horizontal and vertical coordinate displacement of the center pixel of the cross mark of the other two channels relative to the center pixel of the cross mark of the reference channel, and use this as the standard to determine the image range where the three channels overlap; Step 3.2: Write a row counting module and a column counting module on the FPGA to determine the horizontal and vertical coordinates of the pixel, and build a cropping range judgment module through combinatorial logic. Use the cropping range judgment module to limit the image range where the three channels overlap. Each of the three channels has a cropping module, so as to obtain the image after the three-channel image registration based on the image range where the three channels overlap.

5. The true color imaging method based on prism spectrometry 3CMOS camera according to claim 1, characterized in that: The automatic white balance module includes a gain coefficient calculation module, an RGB color correction module, and a row and field timing beat module. The specific process of using the automatic white balance module to perform color restoration on the registered RGB image is as follows: The gain coefficient calculation module calculates the gain coefficient based on the registered RGB image. The specific calculation process is implemented using a divider on the FPGA. The specific formula is: Among them, R sum , G sum , B sum are the grayscale values ​​of the three channels of the registered RGB image and K r , K g , K b They are the correction coefficients of the three channels of the registered RGB image; The RGB color correction module is based on the gain coefficient K r , K g , K b Perform RGB color correction on the next frame of image, and multiply the three channels with their own correction coefficients. The correction process is as follows: R * =K r ×R G * =K r ×G B * =K r ×B Among them, R * , G * , B * are the grayscale values ​​of the three-channel pixels after correction, and R, G, and B are the grayscale values ​​of the three-channel pixels before correction; The line and field timing beat module delays the line and field timing on the FPGA, delays the output of synchronization with pixel data, and obtains the final output video image data, which is finally displayed in real time on the host computer through the encoding chip and image acquisition card.

Images