An FPGA-based adaptive image correction method for multispectral cameras

Abstract

This invention belongs to the field of image processing technology, specifically relating to an adaptive image correction method for a multispectral camera based on FPGA. The method includes: shining a laser beam into the lens of a multispectral camera in a relatively dark environment; the multi-channel sensors in the multispectral camera receiving the laser beam and generating channel images; performing preliminary correction on the multi-channel images to obtain preliminarily aligned multi-channel images; performing cropping and padding operations on the preliminarily aligned multi-channel images to obtain fully aligned multi-channel images; and fusing the fully aligned images of each channel and outputting them for display. This invention improves the matching accuracy of multispectral camera images and effectively reduces the overall size of the device.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an adaptive correction method for multispectral camera images based on FPGA. Background Technology

[0002] Since its inception, multispectral imaging technology has attracted considerable attention from researchers. With the miniaturization of its platforms, the demand for multispectral cameras in agriculture, forestry, military, aviation, medicine, and scientific research is increasing. Multispectral cameras can be divided into three categories: multi-lens multispectral cameras, multi-camera multispectral cameras, and beam splitting multispectral cameras. Among these three types, beam splitting cameras have the advantages of simple structure and high image overlap accuracy, therefore, most mainstream multispectral cameras adopt beam splitting technology. Beam splitting multispectral cameras utilize dichroic films in a prism to transmit light of different wavelengths to multiple independent monochrome sensors in space, achieving precise spatial fusion imaging of the multispectral spectrum. However, multispectral cameras need to synthesize images captured under different spectra, which requires spatial alignment of images from different spectra. Due to positional deviations that occur when the sensor and prism are bonded, aberrations and spatial misalignments are caused, resulting in significant errors during fusion imaging. Currently, the solution to eliminate this error is to reduce the error generated when the sensor and prism are bonded using a correction device. However, the correction device can only reduce the error and it is difficult to completely eliminate pixel-level offsets. At the same time, the use of correction devices also increases the design complexity and overall size of the product.

[0003] To address the problems existing in the prior art, this invention proposes an FPGA-based adaptive image correction method for multispectral cameras. It utilizes the advantages of FPGA such as high speed and parallelism, and simultaneously incorporates DDR4 for image storage and correction circuit design to solve the problems of matching accuracy and large size in multispectral cameras. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an FPGA-based adaptive correction method for multispectral camera images, which includes:

[0005] S1: In a relatively dark environment, a laser beam is shone into the lens of a multispectral camera. The multi-channel sensors in the multispectral camera receive the laser beam and generate channel images.

[0006] S2: Perform preliminary correction on the multi-channel image to obtain a preliminary aligned multi-channel image; the multi-channel image includes one main channel and multiple sub-channels.

[0007] S3: Perform cropping and padding operations on the initially aligned multi-channel image to obtain a fully aligned multi-channel image;

[0008] S4: Merge the fully aligned images of each channel and output the result.

[0009] Preferably, the process of preliminary correction of multi-channel images includes:

[0010] S21: Set the pixel threshold and calculate the spot position of each channel image based on the pixel threshold;

[0011] S22: Determine whether the sub-channel image meets the correction conditions based on the spot position of each channel image. If it does, proceed to step S23; otherwise, use the current channel images as the preliminarily aligned multi-channel images.

[0012] S23: Calculate the number of rows and columns of the deviation of the sub-channel based on the position of the light spots in the sub-channel image and the main channel image; determine the image offset direction of the sub-channel based on the number of rows and columns of pixels where the pixel values ​​of the sub-channel and the main channel images are greater than the pixel threshold;

[0013] S24: Adjust the image position of the sub-channel according to the number of rows and columns of the sub-channel deviation and the image offset direction, and return to step S22.

[0014] Furthermore, the correction condition is that the difference between the pixel values ​​at corresponding positions in the sub-channel image and the main channel image is greater than a pixel threshold.

[0015] Furthermore, the process of calculating the deviation row and column number of the sub-channel includes: the deviation row and column number includes the deviation row number and the deviation column number; determining the difference in the number of rows between the sub-channel image and the main channel image based on the spot positions of the sub-channel image and the main channel image, and dividing the difference in the number of rows between the sub-channel image and the main channel image by 2 to obtain the deviation row number of the sub-channel; determining the difference in the number of columns between the sub-channel image and the main channel image based on the spot positions of the sub-channel image and the main channel image, and dividing the difference in the number of columns between the sub-channel image and the main channel image by 2 to obtain the deviation column number of the sub-channel.

[0016] Furthermore, the process of determining the image offset direction of the sub-channel includes:

[0017] If the number of rows of pixels in the sub-channel image when the first pixel exceeds the pixel threshold is less than the number of rows of pixels in the main channel image when the first pixel exceeds the pixel threshold, then the sub-channel image is determined to be shifted upwards compared to the main channel image; otherwise, the sub-channel image is determined to be shifted downwards compared to the main channel image.

[0018] If the number of columns in the first column where the pixel value of a sub-channel image is greater than the pixel threshold is less than the number of columns in the first column where the pixel value of a main channel image is greater than the pixel threshold, then the sub-channel image is determined to be shifted to the left relative to the main channel image; otherwise, the sub-channel image is determined to be shifted to the right relative to the main channel image.

[0019] Preferably, the process of cropping and padding the image includes:

[0020] S31: Remove redundant pixels from the sub-channel image compared to the main channel image;

[0021] S32: Calculate the light intensity ratio coefficient of the sub-channel based on the image pixel values ​​of each channel;

[0022] S33: Multiply the pixel values ​​at the missing parts of the sub-channel image compared to the main channel image by the light intensity ratio coefficient to complete the image, resulting in a fully aligned multi-channel image.

[0023] Furthermore, the process of calculating the final scaling factor of the sub-channel includes: calculating the sub-scaling factor of the sub-channel image under various illumination intensities to obtain multiple sub-scaling factors; and averaging the multiple sub-scaling factors to obtain the light intensity scaling factor.

[0024] Furthermore, the process of calculating the sub-scale factor of the sub-channel image includes:

[0025] The pixel values ​​of the edges of each channel image are obtained based on the positions of the main channel image and the sub-channel image; the average value of the edge pixel values ​​of each channel is calculated to obtain the average edge pixel value of each channel.

[0026] Divide the average edge pixel value of the main channel by the average edge pixel value of each sub-channel to obtain the initial sub-scale coefficient of each sub-channel;

[0027] The average value of the initial sub-proportional coefficients of each sub-channel is used to obtain the sub-proportional coefficients of the sub-channels.

[0028] The beneficial effects of this invention are as follows:

[0029] 1. This invention performs spatial correction on images acquired by multiple sensors by calculating multi-channel image differences, and simultaneously removes redundant terms and fills in missing terms in the corrected image based on the relationship between imaging light intensities across multiple bands. This correction technique solves the problem of poor image quality caused by positional deviations in RGB fusion imaging using multispectral cameras.

[0030] 2. The correction module design implemented by the present invention through pure hardware FPGA can effectively reduce the overall size of the device, and is applicable to sensors and prisms of different sizes, effectively reducing the complexity of the correction device and improving the versatility of the correction. Attached Figure Description

[0031] Figure 1 This is a flowchart of the FPGA-based multispectral camera image adaptive correction method in this invention;

[0032] Figure 2This is a schematic diagram of a laser beam illuminating a lens.

[0033] Figure 3 This is a schematic diagram of the preliminary image correction in this invention;

[0034] Figure 4 This is a schematic diagram of image completion in this invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention proposes an adaptive image correction method for multispectral cameras based on FPGA, such as... Figure 1 As shown, the method includes:

[0037] S1: In a relatively dark environment, a laser beam is shone into the lens of a multispectral camera. The multi-channel sensors in the multispectral camera receive the laser beam and generate channel images.

[0038] like Figure 2 As shown, a multispectral camera lens, a beam splitter prism, and an image sensor are assembled. The assembled device is placed in a relatively dark environment, and a laser beam is shone onto the lens. The light spots formed by the reflection of the prism are collected by each image sensor, so that each channel can obtain a uniform light spot with relatively small light intensity, i.e., a channel image.

[0039] S2: Perform preliminary correction on the multi-channel image to obtain a preliminary aligned multi-channel image.

[0040] A multispectral camera can have multiple channels (greater than or equal to 3). Multiple channels include one master channel and multiple sub-channels. For example, if a multispectral camera has 3 channels, it is specifically one master channel and two sub-sensor channels. The pixel value of the master channel image is represented as V, and the pixel values ​​of the two sub-channels are represented as V1 and V2, respectively.

[0041] like Figure 3 As shown, this invention utilizes the advantages of FPGA, such as parallel computing and small size, to correct images acquired by a sensor. The initial image correction process includes:

[0042] S21: Set the pixel threshold and calculate the position of the light spot in each channel image based on the pixel threshold.

[0043] Set pixel threshold V thIt binds to the master channel while acquiring image data from other channels. This means that the sensor data from each channel is split into pixels based on the pixel width.

[0044] The location of the light spot in each channel image is calculated based on a pixel threshold. Specifically, this involves comparing pixel values ​​in the channel image that are greater than the pixel threshold V. th The number of rows and columns of the pixels is recorded to obtain the position of the light spot formed by these pixels.

[0045] S22: Determine whether the sub-channel image meets the correction conditions based on the spot position of each channel image. If it does, proceed to step S23; otherwise, use the current channel images as the preliminarily aligned multi-channel images.

[0046] The correction condition is that the difference between the pixel values ​​at corresponding positions in the sub-channel image and the main channel image is greater than a pixel threshold. When the correction condition is met, it means that the sub-channel image needs to be corrected and the subsequent steps continue; otherwise, it means that no correction is needed or that it has already been corrected, and the current channel images are used as the initially aligned multi-channel images.

[0047] S23: Calculate the number of rows and columns of the deviation of the sub-channel based on the position of the light spots in the sub-channel image and the main channel image; determine the image offset direction of the sub-channel based on the number of rows and columns of the pixels where the sub-channel and main channel image pixels first exceed the pixel threshold.

[0048] Calculate the number of offset rows and columns for each sub-channel. Specifically, the number of offset rows and columns includes the number of offset rows and columns. Determine the difference in the number of rows between the sub-channel and main channel images based on the spot positions of the sub-channel and main channel images. Divide this difference in row count by 2 to obtain the offset row count for the sub-channel. Determine the difference in the number of columns between the sub-channel and main channel images based on the spot positions of the sub-channel and main channel images. Divide this difference in column count by 2 to obtain the offset column count for the sub-channel.

[0049] The process of determining the row and column differences of the channel images includes:

[0050] For example, in the main channel image, if V>V th The first pixel's row number is X1, and the last row's pixel number is V>V. th The number of rows is X2, and thus the size of the number of pixel rows occupied by the light spot is determined as X = X2 - X1.

[0051] The first calculation will show V>V th Let Y be the starting column number. 11 Then, the maximum number of columns in the same row that meet the conditions is taken as the ending column number and denoted as Y. 12 Then calculate the next row to satisfy V>V thLet Y be the starting column number of the pixel. 21 Let Y be the number of the last column. 22 and take Y 11 With Y 21 The minimum value in the middle is used as the starting column number Y1 of the current spot, and Y is... 12 With Y 22 The larger value is taken as the end column number Y2 of the light spot. The above operation is repeated and continuously updated until the end of a frame or multiple rows when there are no pixels with a number greater than the threshold. The final result is then fed back, and the light spot column difference Y = Y2 - Y1 is obtained.

[0052] Similarly, the difference in the number of rows and columns of the sub-channel image can be obtained.

[0053] The process of determining the image offset direction of a sub-channel includes:

[0054] If the number of rows of pixels in the sub-channel image when the first pixel exceeds the pixel threshold is less than the number of rows of pixels in the main channel image when the first pixel exceeds the pixel threshold, then the sub-channel image is determined to be shifted upwards compared to the main channel image; otherwise, the sub-channel image is determined to be shifted downwards compared to the main channel image.

[0055] If the number of columns in the first column where the pixel value of a sub-channel image is greater than the pixel threshold is less than the number of columns in the first column where the pixel value of a main channel image is greater than the pixel threshold, then the sub-channel image is determined to be shifted to the left relative to the main channel image; otherwise, the sub-channel image is determined to be shifted to the right relative to the main channel image.

[0056] S24: Adjust the image position of the sub-channel according to the number of rows and columns of the sub-channel deviation and the image offset direction, and return to step S22.

[0057] If the image offset direction is determined to be to the left, the sub-channel image is moved to the right according to the number of offset rows and columns, and vice versa. Similarly, if the image offset direction is determined to be to the up, the sub-channel image is moved down according to the number of offset rows and columns, and vice versa.

[0058] S3: Perform cropping and padding operations on the initially aligned multi-channel image to obtain a fully aligned multi-channel image.

[0059] The three-channel image pixels are mapped one-to-one. However, after the initial correction and alignment operation, there are redundant and missing pixels in the three-channel image, such as... Figure 4 As shown, the master image is the upper layer image. Assuming the image is offset to the upper left relative to the master image, the pixels in the black diagonal line portion need to be removed. Simultaneously, to ensure the integrity and accuracy of the image, the missing rows and columns in all channels except the master channel need to be filled in during the alignment process, as shown by the orange diagonal line portion in the image, ensuring that the remaining channels have the same image size as the master channel.

[0060] The process of cropping and padding an image includes:

[0061] S31: Remove redundant pixels from the sub-channel image compared to the main channel image;

[0062] S32: Calculate the light intensity ratio coefficient of the sub-channel based on the image pixel values ​​of each channel;

[0063] Regarding the calculation of pixel completion, to ensure the accuracy of the final fused image, it is necessary to calculate the pixel relationships between the three channel images, thereby making the completed pixels more accurate. Since multi-sensor illumination is obtained through a beam splitter using the same beam, there is a certain correspondence between the output images of each channel captured in the same frame. In this invention, linear relationships between pixels are established between the master channel and other channels. Specifically, images are captured under different lighting conditions, and the sub-scale coefficients of the sub-channel images under various lighting intensities are calculated to obtain multiple sub-scale coefficients. The light intensity proportionality coefficient is obtained by averaging multiple sub-proportional coefficients.

[0064] The process of calculating the sub-scale coefficients of a sub-channel image under a certain light intensity includes:

[0065] The pixel values ​​of the edges of each channel image are obtained based on the positions of the main channel image and the sub-channel images. The pixel values ​​of the main channel are y1...y n The pixel value at the same position in the other channels is x. 11... x 1n x 21... x 2n The average value of the edge pixels in each channel is calculated to obtain the average edge pixel value for each channel.

[0066] Divide the average edge pixel value of the main channel by the average edge pixel value of each sub-channel to obtain the initial sub-scale coefficients k1 and k2 of each sub-channel;

[0067] The average value of the initial sub-scale coefficients of each sub-channel is used to obtain the sub-scale coefficients of the sub-channels under this light intensity. m represents the intensity of the m-th type of light (m>10).

[0068] S33: Multiply the pixel values ​​at the missing parts of the sub-channel image compared to the main channel image by the light intensity ratio coefficient to complete the image, resulting in a fully aligned multi-channel image.

[0069] By calculating the relationship between the image data of each channel under different light intensities in multiple frames, and completing the missing parts of the remaining images except for the master channel, the positions of the multi-channel images can be completely aligned without losing effective information.

[0070] S4: Merge the fully aligned images of each channel and output the result.

[0071] The multi-channel images after position correction are fused together, and a color image imaging technology is achieved using a multi-sensor prism camera to synthesize an RGB image with real color factors.

[0072] In summary, this invention uses an FPGA to calculate the size of the light spot contour, the offset of each sensor image, and the offset direction, and then corrects the multi-channel image based on the calculated correction parameters. This solves the problem of poor image quality caused by positional deviations in RGB fusion imaging using multispectral cameras. The correction module design, implemented entirely in hardware using an FPGA, effectively reduces the overall size of the device and is applicable to sensors and prisms of different sizes, significantly reducing the complexity of the correction device and improving its versatility.

[0073] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive image correction method for multispectral cameras based on FPGA, characterized in that, include: S1: In a relatively dark environment, a laser beam is shone into the lens of a multispectral camera. The multi-channel sensors in the multispectral camera receive the laser beam and generate channel images. S2: Perform preliminary correction on the multi-channel image to obtain a preliminarily aligned multi-channel image; A multi-channel system includes one main channel and multiple sub-channels; The process of preliminary correction for multi-channel images includes: S21: Set the pixel threshold and calculate the spot position of each channel image based on the pixel threshold; S22: Determine whether the sub-channel image meets the correction conditions based on the spot position of each channel image. If it does, proceed to step S23; otherwise, use the current channel images as the preliminarily aligned multi-channel images. S23: Calculate the number of rows and columns of the deviation of the sub-channel based on the position of the light spots in the sub-channel image and the main channel image; determine the image offset direction of the sub-channel based on the number of rows and columns of pixels where the pixel values ​​of the sub-channel and the main channel images are greater than the pixel threshold; The process of calculating the deviation row and column number of a sub-channel includes: the deviation row and column number; determining the difference in the number of rows between the sub-channel image and the main channel image based on the spot positions of the sub-channel image and the main channel image, and dividing the difference in the number of rows between the sub-channel image and the main channel image by 2 to obtain the deviation row number of the sub-channel; determining the difference in the number of columns between the sub-channel image and the main channel image based on the spot positions of the sub-channel image and the main channel image, and dividing the difference in the number of columns between the sub-channel image and the main channel image by 2 to obtain the deviation column number of the sub-channel. The process of determining the image offset direction of a sub-channel includes: If the number of rows of pixels in the sub-channel image when the first pixel exceeds the pixel threshold is less than the number of rows of pixels in the main channel image when the first pixel exceeds the pixel threshold, then the sub-channel image is determined to be shifted upwards compared to the main channel image; otherwise, the sub-channel image is determined to be shifted downwards compared to the main channel image. If the number of columns in the first column where the pixel value of a sub-channel image is greater than the pixel threshold is less than the number of columns in the first column where the pixel value of a main channel image is greater than the pixel threshold, then the sub-channel image is determined to be shifted to the left relative to the main channel image; otherwise, the sub-channel image is determined to be shifted to the right relative to the main channel image. S24: Adjust the image position of the sub-channel according to the number of rows and columns of the sub-channel deviation and the image offset direction, and return to step S22; S3: Perform cropping and padding operations on the initially aligned multi-channel image to obtain a fully aligned multi-channel image; the image cropping and padding process includes: S31: Remove redundant pixels from the sub-channel image compared to the main channel image; S32: Calculate the light intensity scaling factor of the sub-channel based on the image pixel values ​​of each channel; including: The pixel values ​​of the edges of each channel image are obtained based on the positions of the main channel image and the sub-channel image; the average value of the edge pixel values ​​of each channel is calculated to obtain the average edge pixel value of each channel. Divide the average edge pixel value of the main channel by the average edge pixel value of each sub-channel to obtain the initial sub-scale coefficient of each sub-channel; The average value of the initial sub-scaling coefficients for each sub-channel is used to obtain the sub-scaling coefficients for each sub-channel. The light intensity proportionality coefficient is obtained by averaging multiple sub-proportioning coefficients. S33: Multiply the pixel values ​​of the missing parts of the sub-channel image compared to the main channel image by the light intensity ratio coefficient to fill in the missing parts and obtain a fully aligned multi-channel image. S4: Merge the fully aligned images of each channel and output the result.

2. The FPGA-based multispectral camera image adaptive correction method according to claim 1, characterized in that, The correction condition is that the difference between the pixel values ​​at corresponding positions in the sub-channel image and the main channel image is greater than the pixel threshold.

Images