Multispectral camera image adaptive correction method based on FPGA

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a multispectral camera image adaptive correction method based on an FPGA. The method comprises the steps that a laser beam is emitted from a lens of a multispectral camera in a dark environment, and a multi-channel sensor in the multispectral camera receives the laser beam and generates a channel image; performing preliminary correction on the multi-channel image to obtain a preliminarily aligned multi-channel image; cutting and supplementing the preliminarily aligned multi-channel image to obtain a completely aligned multi-channel image; and fusing the completely aligned images of the channels, and outputting and displaying the fused images. According to the invention, the matching precision of the multispectral camera is improved, and the size of the whole machine is effectively reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a multi-spectral camera image adaptive correction method based on FPGA. Background Art

[0002] Since its introduction, multispectral imaging technology has attracted much attention from researchers. With the development of miniaturization of the carrying platform, the demand for multispectral cameras in agriculture, forestry, military, aviation, medicine, scientific research and other fields is increasing. Multispectral cameras can be divided into three categories: the first is a multi-lens multispectral camera; the second is a multi-camera multispectral camera; and the third is a beam separation multispectral camera. Among these three types of multispectral cameras, the advantages of the beam separation camera are simple structure and high image overlap accuracy. Therefore, mainstream multispectral cameras mostly use beam separation technology. The beam separation multispectral camera uses the dichroic film layer in the prism to transmit light of different wavelengths to multiple independent black and white sensors in space for multispectral precise spatial fusion imaging. However, the multispectral camera needs to synthesize images taken under different spectra, which requires spatial alignment of images under different spectra. Because position deviation occurs when the sensor and the prism are bonded, aberration and spatial position misalignment occur, resulting in large errors in fusion imaging. The current solution to eliminate this error is to reduce the error generated when the sensor and the prism are bonded through a correction device. However, the correction device can only reduce the error, and it is difficult to completely eliminate the pixel-level offset. At the same time, the correction device will also increase the design difficulty and overall size of the product.

[0003] In view of the problems existing in the prior art, the present invention proposes a multi-spectral camera image adaptive correction method based on FPGA, which utilizes the advantages of FPGA such as high speed and parallelism, and mounts DDR4 for image storage and correction circuit design to solve the problems of matching accuracy and large size of multi-spectral cameras. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a multi-spectral camera image adaptive correction method based on FPGA, the method comprising:

[0005] S1: In a dark environment, a laser beam is emitted from the lens of a multispectral camera, and a multi-channel sensor in the multispectral camera receives the laser beam and generates a channel image;

[0006] S2: Perform preliminary correction on the multi-channel image to obtain a preliminarily aligned multi-channel image; the multi-channel includes a main channel and multiple sub-channels;

[0007] S3: performing cropping and padding operations on the preliminarily aligned multi-channel image to obtain a fully aligned multi-channel image;

[0008] S4: The fully aligned images of each channel are fused and output for display.

[0009] Preferably, the process of performing preliminary correction on the multi-channel image includes:

[0010] S21: setting a pixel threshold, and calculating the spot position of each channel image according to the pixel threshold;

[0011] S22: judging whether the sub-channel image meets the correction condition according to the spot position of each channel image, if so, executing step S23, otherwise, taking the current channel images as the preliminarily aligned multi-channel images;

[0012] S23: Calculate the number of deviation rows and columns of the sub-channel according to the spot positions of the sub-channel image and the main channel image; determine the image deviation direction of the sub-channel according to the number of rows and columns where the pixel points are located when the pixel values ​​of the sub-channel and the main channel images are greater than the pixel threshold;

[0013] S24: Adjust the image position of the sub-channel according to the deviation row and column number of the sub-channel and the image offset direction, and return to step S22.

[0014] Furthermore, the correction condition is that the difference between the pixel values ​​at the corresponding positions of the sub-channel image and the main channel image is greater than the pixel threshold.

[0015] Furthermore, the process of calculating the deviation row and column number of the sub-channel includes: the deviation row and column number includes the deviation row number and the deviation column number; determining the row number difference between the sub-channel image and the main channel image according to the spot positions of the sub-channel image and the main channel image, and dividing the row number difference between the sub-channel image and the main channel image by 2 as the deviation row number of the sub-channel; determining the column number difference between the sub-channel image and the main channel image according to the spot positions of the sub-channel image and the main channel image, and dividing the column number difference between the sub-channel image and the main channel image by 2 as the deviation column number of the sub-channel.

[0016] Furthermore, the process of determining the image offset direction of the sub-channel includes:

[0017] If the number of rows where the sub-channel image pixel is located when the pixel is greater than the pixel threshold for the first time is less than the number of rows where the pixel is located when the pixel is greater than the pixel threshold for the first time in the main channel image, then it is determined that the sub-channel image is shifted upward compared to the main channel image; otherwise, it is determined that the sub-channel image is shifted downward compared to the main channel image;

[0018] If the number of the first columns whose pixel values ​​of the sub-channel image pixels are greater than the pixel threshold is less than the number of the first columns whose pixel values ​​of the main channel image pixels are greater than the pixel threshold, then it is determined that the sub-channel image is offset to the left compared to the main channel image; otherwise, it is determined that the sub-channel image is offset to the right compared to the main channel image.

[0019] Preferably, the process of cropping and padding the image includes:

[0020] S31: removing redundant pixels of the sub-channel image compared to the main channel image;

[0021] S32: Calculate the light intensity ratio coefficient of the sub-channel according to the image pixel value of each channel;

[0022] S33: Multiply the pixel values ​​of the missing parts of the sub-channel image by the light intensity ratio coefficient to complete the missing parts, so as to obtain a fully aligned multi-channel image.

[0023] Furthermore, the process of calculating the final scale coefficient of the sub-channel includes: calculating the sub-scale coefficient of the sub-channel image under multiple light intensities to obtain multiple sub-scale coefficients; and averaging the multiple sub-scale coefficients to obtain the light intensity scale coefficient.

[0024] Furthermore, the process of calculating the sub-scale coefficient of the sub-channel image includes:

[0025] According to the positions of the main channel image and the sub-channel image, the pixel value of the edge of each channel image is obtained; the edge pixel value of each channel is averaged to obtain the edge pixel average value of each channel;

[0026] Divide the average value of edge pixels of the main channel by the average value of edge pixels of each sub-channel to obtain the initial sub-scale coefficient of each sub-channel;

[0027] The initial sub-proportional coefficients of each sub-channel are averaged to obtain the sub-proportional coefficient of the sub-channel.

[0028] The beneficial effects of the present invention are:

[0029] 1. The present invention completes the spatial correction of images collected by multiple sensors by calculating multi-channel image differences and other schemes, and at the same time, according to the relationship between the imaging light intensities of multiple bands, removes redundant items and completes missing items for the corrected images. This correction technology solves the problem of poor fusion image quality caused by position deviation in the existing multi-spectral camera RGB fusion imaging.

[0030] 2. The correction module design of the present invention is realized by pure hardware FPGA, which can effectively reduce the volume of the whole machine. At the same time, it can be applied to sensors and prisms of different sizes, effectively reducing the complexity of using the correction device and improving the versatility of correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the FPGA-based multi-spectral camera image adaptive correction method in the present invention;

[0032] Figure 2It is a schematic diagram of laser beam irradiating lens;

[0033] Figure 3 This is a schematic diagram of preliminary image correction in the present invention;

[0034] Figure 4 This is a schematic diagram of image completion in the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The present invention proposes a multi-spectral camera image adaptive correction method based on FPGA. Figure 1 As shown, the method includes:

[0037] S1: In a dark environment, a laser beam is emitted from the lens of a multispectral camera, and a multi-channel sensor in the multispectral camera receives the laser beam and generates a channel image.

[0038] like Figure 2 As shown, a multispectral camera lens, a beam splitter prism, and an image sensor are assembled, and the assembled equipment is placed in a darker environment. A beam of laser is irradiated on the lens, and the light spot formed by the prism reflection is collected by each image sensor, so that each channel can obtain a uniform light spot with relatively small light intensity, namely, a channel image.

[0039] S2: Perform preliminary correction on the multi-channel image to obtain a preliminarily aligned multi-channel image.

[0040] A multispectral camera may have multiple channels (greater than or equal to 3), and multiple channels include a main channel and multiple sub-channels. For example, if a multispectral camera has 3 channels, there is a main channel master and two sub-channels sensor. The pixel value of the master channel image is represented as V, and the pixel values ​​of the two sub-channels are represented as V1 and V2 respectively.

[0041] like Figure 3 As shown, the present invention uses the advantages of FPGA parallel computing and small size to correct the image collected by the sensor. The process of performing preliminary correction on the image includes:

[0042] S21: Set a pixel threshold, and calculate the spot position of each channel image according to the pixel threshold.

[0043] Set the pixel threshold V th, bind the master channel and obtain the image data of other channels at the same time. That is, split the sensor data of each channel into pixels according to the pixel width.

[0044] Calculate the spot position of each channel image according to the pixel threshold. Specifically: compare the pixel values ​​in the channel image that are greater than the pixel threshold V th , record the number of rows and columns of these pixels, and obtain the position of the light spot composed of these pixels.

[0045] S22: Determine whether the sub-channel image meets the correction condition according to the spot position of each channel image. If so, execute step S23; otherwise, take the current channel images as the preliminarily aligned multi-channel images.

[0046] The correction condition is that the difference between the pixel values ​​at the corresponding position of the sub-channel image and the main channel image is greater than the pixel threshold. When the correction condition is met, it means that the sub-channel image needs to be corrected and the subsequent steps continue; otherwise, it means that no correction is required or has been corrected, and the current channel images are used as the preliminarily aligned multi-channel images.

[0047] S23: Calculate the number of deviation rows and columns of the sub-channel according to the spot positions of the sub-channel image and the main channel image; determine the image offset direction of the sub-channel according to the number of rows and columns where the pixel points of the sub-channel and the main channel image pixels are located when they are greater than the pixel threshold for the first time.

[0048] Calculate the number of deviation rows and columns of the subchannel, specifically: the number of deviation rows and columns includes the number of deviation rows and the number of deviation columns; determine the difference in the number of rows between the subchannel image and the main channel image according to the spot positions of the subchannel image and the main channel image, and divide the difference in the number of rows between the subchannel image and the main channel image by 2 as the number of deviation rows of the subchannel; determine the difference in the number of columns between the subchannel image and the main channel image according to the spot positions of the subchannel image and the main channel image, and divide the difference in the number of columns between the subchannel image and the main channel image by 2 as the number of deviation columns of the subchannel.

[0049] The process of determining the row number difference and column number difference of the channel image includes:

[0050] For example, in the main channel image, if V>V th The first pixel row number is X1, and the last pixel row number is V>V th The number of rows is X2, thereby determining the number of pixel rows occupied by the light spot X=X2-X1.

[0051] The first calculation V>V th The starting column number is recorded as Y 11 , and then calculate the maximum number of columns that meet the conditions in the same row as the ending number of columns, recorded as Y 12 Then calculate the next row to satisfy V>V thThe starting column number of the pixel is recorded as Y 21 and the end column number is recorded as Y 22 , and take Y 11 With Y 21 The minimum value is taken as the starting column number Y1 of the current spot, and Y 12 With Y 22 The larger value is used as the light spot ending column number Y2. The above operation is repeated and continuously updated until a frame or multiple rows end without a pixel number greater than the threshold, and the final result is fed back to obtain the light spot column difference Y=Y2-Y1.

[0052] Similarly, the row number difference and column number difference of the sub-channel image can be obtained.

[0053] The process of determining the image offset direction of the sub-channel includes:

[0054] If the number of rows where the sub-channel image pixel is located when the pixel is greater than the pixel threshold for the first time is less than the number of rows where the pixel is located when the pixel is greater than the pixel threshold for the first time in the main channel image, then it is determined that the sub-channel image is shifted upward compared to the main channel image; otherwise, it is determined that the sub-channel image is shifted downward compared to the main channel image;

[0055] If the number of the first columns whose pixel values ​​of the sub-channel image pixels are greater than the pixel threshold is less than the number of the first columns whose pixel values ​​of the main channel image pixels are greater than the pixel threshold, then it is determined that the sub-channel image is offset to the left compared to the main channel image; otherwise, it is determined that the sub-channel image is offset to the right compared to the main channel image.

[0056] S24: Adjust the image position of the sub-channel according to the deviation row and column number of the sub-channel and the image offset direction, and return to step S22.

[0057] If the image offset direction is judged to be left, the sub-channel image is moved to the right according to the number of deviation rows and columns, otherwise it is moved to the right; similarly, if the image offset direction is judged to be upward, the sub-channel image is moved downward according to the number of deviation rows and columns, otherwise it is moved upward.

[0058] S3: Perform cropping and padding operations on the preliminarily aligned multi-channel image to obtain a fully aligned multi-channel image.

[0059] The pixels of the three-channel images are matched one by one. However, after the three-channel images are initially corrected and aligned, there are redundant pixels and missing pixels, such as Figure 4 As shown in the figure. The master is the upper image. Assuming that the image is offset to the upper left compared to the master, the pixels in the black diagonal part of the figure need to be removed. At the same time, in order to ensure the completeness and accuracy of the image, the number of missing rows and columns of other channels except the master channel during the alignment process needs to be filled, as shown in the orange diagonal part of the figure, to ensure that the remaining channels are consistent with the master channel image size.

[0060] The process of cropping and padding an image includes:

[0061] S31: removing redundant pixels of the sub-channel image compared to the main channel image;

[0062] S32: Calculate the light intensity ratio coefficient of the sub-channel according to the image pixel value of each channel;

[0063] Regarding the calculation of the padded pixels, in order to ensure the accuracy of the final fused image, it is necessary to calculate the pixel relationship between the three-channel images, so that the padded pixels tend to be accurate. Since the multi-sensor lighting is obtained through the same beam of light through a dichroic prism, there is a certain correspondence between the output images of each channel collected in the same frame. In the present invention, the master channel is used to establish a linear relationship between pixels of other channels. Specifically: the image is collected under different light intensity environments, and the sub-proportional coefficients of the sub-channel images under various light intensities are calculated to obtain multiple sub-proportional coefficients. The light intensity proportionality coefficient is obtained by averaging multiple sub-proportional coefficients.

[0064] Wherein, under a certain light intensity, the process of calculating the sub-scale coefficient of the sub-channel image includes:

[0065] According to the positions of the main channel image and the sub-channel image, the pixel values ​​at the edge of each channel image are obtained. The pixel values ​​of the main channel are y1...y n , the pixel value of the same position in other channels is x 11... x 1n 、x 21... x 2n ; Calculate the average of the edge pixel values ​​of each channel to obtain the average edge pixel value of each channel

[0066] Divide the average value of edge pixels of the main channel by the average value of edge pixels of each sub-channel to obtain initial sub-proportional coefficients k1 and k2 of each sub-channel;

[0067] The initial sub-proportional coefficients of each sub-channel are averaged to obtain the sub-proportional coefficients of the sub-channel under this light intensity. m represents the mth light intensity, (m>10).

[0068] S33: Multiply the pixel values ​​of the missing parts of the sub-channel image by the light intensity ratio coefficient to complete the missing parts, so as to obtain a fully aligned multi-channel image.

[0069] By calculating the relationship between the image data of each channel under multiple frames of different light intensities as described above, the missing parts of the remaining images except the master channel are completed, so that the positions of the multi-channel images can be completely aligned without losing effective information.

[0070] S4: The fully aligned images of each channel are fused and output for display.

[0071] The position-corrected multi-channel images are fused, and the color image imaging technology is realized by using a multi-sensor prism camera to synthesize an RGB image with real color factors.

[0072] In summary, the present invention uses FPGA to complete the calculation of the spot contour size, the offset of each sensor image and the offset direction, and corrects the multi-channel image according to the calculated correction parameters. It solves the problem of poor fusion image quality caused by position deviation in the existing multi-spectral camera RGB fusion imaging. The correction module design implemented by pure hardware FPGA can effectively reduce the volume of the whole machine. At the same time, it can be applied to sensors and prisms of different sizes, effectively reducing the complexity of using the correction device and improving the versatility of correction.

[0073] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-spectral camera image adaptive correction method based on FPGA, characterized in that: include: S1: In a dark environment, a laser beam is emitted from the lens of a multispectral camera, and a multi-channel sensor in the multispectral camera receives the laser beam and generates a channel image; S2: Perform preliminary correction on the multi-channel image to obtain a preliminarily aligned multi-channel image; the multi-channel includes a main channel and multiple sub-channels; S3: performing cropping and padding operations on the preliminarily aligned multi-channel image to obtain a fully aligned multi-channel image; S4: The fully aligned images of each channel are fused and output for display.

2. A method for adaptive correction of multi-spectral camera images based on FPGA according to patent claim 1, characterized in that: The process of preliminary correction of multi-channel images includes: S21: setting a pixel threshold, and calculating the spot position of each channel image according to the pixel threshold; S22: judging whether the sub-channel image meets the correction condition according to the spot position of each channel image, if so, executing step S23, otherwise, taking the current channel images as the preliminarily aligned multi-channel images; S23: Calculate the number of deviation rows and columns of the sub-channel according to the spot positions of the sub-channel image and the main channel image; determine the image deviation direction of the sub-channel according to the number of rows and columns where the pixel points are located when the pixel values ​​of the sub-channel and the main channel images are greater than the pixel threshold; S24: Adjust the image position of the sub-channel according to the deviation row and column number of the sub-channel and the image offset direction, and return to step S22.

3. The method for adaptive correction of multi-spectral camera images based on FPGA according to claim 2, characterized in that: The correction condition is that the difference between the pixel values ​​at the corresponding position of the sub-channel image and the main channel image is greater than the pixel threshold.

4. The method for adaptive correction of multi-spectral camera images based on FPGA according to claim 2, characterized in that: The process of calculating the deviation row and column number of the sub-channel includes: the deviation row and column number includes the deviation row number and the deviation column number; determining the row number difference between the sub-channel image and the main channel image according to the spot positions of the sub-channel image and the main channel image, and dividing the row number difference between the sub-channel image and the main channel image by 2 as the deviation row number of the sub-channel; determining the column number difference between the sub-channel image and the main channel image according to the spot positions of the sub-channel image and the main channel image, and dividing the column number difference between the sub-channel image and the main channel image by 2 as the deviation column number of the sub-channel.

5. The method for adaptive correction of multi-spectral camera images based on FPGA according to claim 2, characterized in that: The process of determining the image offset direction of the sub-channel includes: If the number of rows where the sub-channel image pixel is located when the pixel is greater than the pixel threshold for the first time is less than the number of rows where the pixel is located when the pixel is greater than the pixel threshold for the first time in the main channel image, then it is determined that the sub-channel image is shifted upward compared to the main channel image; otherwise, it is determined that the sub-channel image is shifted downward compared to the main channel image; If the number of the first columns whose pixel values ​​of the sub-channel image pixels are greater than the pixel threshold is less than the number of the first columns whose pixel values ​​of the main channel image pixels are greater than the pixel threshold, then it is determined that the sub-channel image is offset to the left compared to the main channel image; otherwise, it is determined that the sub-channel image is offset to the right compared to the main channel image.

6. The method for adaptive correction of multi-spectral camera images based on FPGA according to claim 1, characterized in that: The process of cropping and padding an image includes: S31: removing redundant pixels of the sub-channel image compared to the main channel image; S32: Calculate the light intensity ratio coefficient of the sub-channel according to the image pixel value of each channel; S33: Multiply the pixel values ​​of the missing parts of the sub-channel image by the light intensity ratio coefficient to complete the missing parts, so as to obtain a fully aligned multi-channel image.

7. The method for adaptive correction of multi-spectral camera images based on FPGA according to claim 6, characterized in that: The process of calculating the final scale coefficient of the sub-channel includes: calculating the sub-scale coefficient of the sub-channel image under multiple light intensities to obtain multiple sub-scale coefficients; averaging the multiple sub-scale coefficients to obtain the light intensity scale coefficient.

8. The method for adaptive correction of multi-spectral camera images based on FPGA according to claim 7, characterized in that: The process of calculating the sub-scale coefficients of a sub-channel image includes: According to the positions of the main channel image and the sub-channel image, the pixel value of the edge of each channel image is obtained; the edge pixel value of each channel is averaged to obtain the edge pixel average value of each channel; Divide the average value of edge pixels of the main channel by the average value of edge pixels of each sub-channel to obtain the initial sub-scale coefficient of each sub-channel; The initial sub-proportional coefficients of each sub-channel are averaged to obtain the sub-proportional coefficient of the sub-channel.

Images