Multi-exposure image acquisition and processing system based on FPGA and prism

Abstract

The invention belongs to the field of image acquisition, transmission and processing, and provides a multi-exposure image acquisition and processing system based on an FPGA (Field Programmable Gate Array) and a prism. According to the technical scheme, the system is composed of five core modules including a prism module, an image acquisition module, an FPGA module, an image transmission module and an upper computer module, the prism module enables four cameras to capture images with different exposure degrees through different light paths, and rich original data is provided for subsequent image fusion and processing. The FPGA module is responsible for temporarily storing image data and providing clock frequency. And the image transmission module adopts an Ethernet technology, so that the stability and the real-time performance in the image data transmission process are ensured. The upper computer module not only can display the collected images in real time, but also is internally provided with an image fusion algorithm to carry out deep fusion and definition optimization on the multi-exposure images, and finally, a clear image with a high dynamic range and rich details is presented.

Description

Technical Field

[0001] The invention belongs to the field of image acquisition, transmission and processing, and in particular relates to a multi-exposure image acquisition and processing system based on FPGA and prism. Background Art

[0002] With the rapid development of digital image processing technology, multi-exposure image acquisition and processing have been widely used in photography, medical imaging, industrial inspection, and autonomous driving. Traditional single-exposure image acquisition usually has difficulty in capturing detailed information that includes both bright and dark areas in high dynamic range (HDR) scenes. However, through multi-exposure image fusion technology, multiple images of the same scene can be captured under different exposure conditions, and then fused to generate a high dynamic range image that retains both bright and dark details. This technology has significant advantages in improving image quality and visual effects.

[0003] Existing multi-exposure image acquisition systems usually acquire multiple images with different exposures by performing multiple exposures at certain intervals, or by performing multiple exposures on the same photo. This method can accomplish the task well in static scenes, but has obvious defects in dynamic scenes. Due to the movement of objects, motion artifacts often appear in the captured images, which cannot meet the requirements of image fusion and therefore has great limitations.

[0004] In addition, existing image acquisition and processing systems are often complex in hardware architecture and have high design costs, making it difficult to meet the market's demand for low cost and integration. This also limits the widespread application of multi-exposure image acquisition technology in portable devices or embedded systems with limited resources.

[0005] To solve the above problems, in recent years, FPGA (field programmable gate array), as a high-performance, low-power, and parallel processing hardware platform, has gradually become a popular choice in the field of image acquisition and processing. FPGA can not only efficiently complete multi-channel data processing, but also accelerate the processing of image data through parallel transmission, significantly improving the real-time processing capability of the system. In addition, combined with optical systems such as prism structures, multi-channel and multi-optical path image acquisition can be effectively realized, greatly improving the efficiency of image data acquisition.

[0006] Based on this, the present invention proposes a multi-exposure image acquisition and processing system based on FPGA and prism. By cleverly combining the efficient parallel processing capability of FPGA and the optical characteristics of prism, a system capable of quickly acquiring multi-exposure images and processing them in real time is constructed. The system can significantly improve the efficiency of image acquisition and processing, and is particularly suitable for image processing applications requiring high dynamic range and high definition. Summary of the invention

[0007] In view of the motion artifact problem existing in the existing multi-exposure image acquisition system in dynamic scenes, as well as the large delay, complex hardware and high cost in the image processing and transmission process, the present invention proposes a multi-exposure image acquisition and processing system based on FPGA and prism. The system aims to improve the efficiency and accuracy of multi-exposure image acquisition, reduce motion artifacts, optimize the hardware architecture to reduce system costs, and realize integrated design.

[0008] In order to achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: a multi-exposure image acquisition and processing system based on FPGA and prism, including a prism module, an image acquisition module, an FPGA module, an image transmission module and a host computer module; the prism module divides the incident light into four paths, each path corresponds to a different exposure level, and is irradiated onto four image sensors of the image acquisition module respectively, and the four image sensors acquire high-speed image data, and the image acquisition module sends the acquired image to the FPGA module, and the FPGA module transmits the image to the host computer module through the image transmission module;

[0009] The host computer module has the function of displaying images in real time, and has a built-in image fusion module, which uses a multi-exposure fusion algorithm to process images to present high dynamic range, clear and delicate images.

[0010] Furthermore, the prism module adopts an integrated cascaded four-beam splitting structure.

[0011] Furthermore, the model of the FPGA module is XC7A35TFGG484I of the Artix-7 series.

[0012] Furthermore, the image transmission module adopts UDP transmission protocol.

[0013] Furthermore, the specific process of the multi-exposure fusion algorithm is as follows:

[0014] Step S1, first aligning images with different exposures;

[0015] Step S2: Use the YCbCr model to extract the values ​​of the three channels R, G, and B of each pixel, and use the formula to convert the color space;

[0016] Step S3, using a weighted fusion method based on the Laplace pyramid to fuse the Y component of the brightness signal;

[0017] Step S4, using the Laplacian pyramid to perform multi-resolution weighted fusion on the Y channel of the multi-exposure images;

[0018] Step S5, chrominance component fusion;

[0019] Step S6: Use a transformation formula to convert the image from the YCbCr color space to the RGB color space, and display the obtained high dynamic range image on a display device.

[0020] Furthermore, the specific steps of image alignment in step S1 are:

[0021] Step S11, feature extraction: extract feature points and descriptors from each image using the ORB algorithm;

[0022] Step S12, feature matching: use BFMatcher to match feature points, and find the best matching point for each feature through a brute force matching algorithm;

[0023] Step S13, removing mismatches: removing mismatches that do not conform to the transformation model, and retaining matching points that conform to the geometric transformation;

[0024] Step S14, image alignment: aligning the images using the calculated transformation matrix.

[0025] Furthermore, the conversion formula of the color space in step S2 is as follows:

[0026] The calculation formula of Y brightness is:

[0027] Y=0.299×R+0.587×G+0.114×B+16

[0028] The calculation formula of Cb chromaticity blue component is:

[0029] U=-0.1687×R-0.3313×G+0.5×B+128

[0030] The calculation formula of Cr chromaticity red component is:

[0031] V=0.5×R-0.4187×G-0.0813×B+128

[0032] In the formula: R, G, B∈[0, 255]; Y channel component Y∈[16, 235]; Cb, Cr channel chrominance components U, V∈[16, 235].

[0033] Furthermore, the specific process of the brightness component fusion is as follows:

[0034] Step S31, weight index selection: determine the weight index used for fusion, and assign different weights to the input images according to the weight index, that is, assign a larger weight to pixels with better quality during the fusion process;

[0035] S311, gradient amplitude calculation

[0036] Use the Sobel operator to calculate the horizontal and vertical gradients of the image:

[0037] G x (i,j)=Y k (i+1,j)-Y k (i-1,j)

[0038] G y (i,j)=Y k (i,j+1)-Y k (i,j-1)

[0039] The gradient magnitude G(i, j) represents the rate of change of brightness at the pixel.

[0040]

[0041] S312, Texture complexity calculation

[0042] The gray level co-occurrence matrix GLCM of the local area is used to measure the complexity of the texture. GLCM can effectively capture the local texture features of the image:

[0043]

[0044] Among them, Pk(p, q) is the element of the gray level co-occurrence matrix, which represents the co-occurrence frequency between gray values ​​p and q, Tk(i, j) represents the texture complexity of the local area, and Tk(i, j) indicates that the details of the local area of ​​the image are rich;

[0045] S313 Adaptive Fusion

[0046] The gradient and texture complexity are combined to generate the final contrast weight through adaptive weighting. The contrast weight calculation formula is:

[0047]

[0048] Among them, α is an adaptive parameter that controls the ratio of gradient and texture weight, max(G) and max(T) are the maximum values ​​of gradient amplitude and texture complexity, respectively, which are used for normalization;

[0049] S314, adaptive parameter α calculation

[0050] The parameter α is dynamically adjusted according to the smoothness of the local area, and α is calculated by the local variance:

[0051]

[0052] Among them, σ2(i, j) is the brightness variance of the local area near pixel (i, j), It is a threshold used to control smoothness. When the variance of the local area is large, it means that there are edge or texture features, and the weight of the gradient is increased; when the variance of the local area is small, it tends to increase the weight of the texture;

[0053] S32, saturation weight index: The calculation formula of saturation is

[0054]

[0055] Among them, m k is the average of the luminance and chrominance channels, Y k (i, j), U k (i, j) and V k (i, j) represent the pixel value of the image at the point (i, j) of the kth color channel. In the YUV color space, Y represents the brightness component, while U and V represent the chrominance components.

[0056]

[0057] S33, exposure weight index, the specific calculation process is as follows:

[0058] S331, Adaptive brightness weight formula: Introduce a local adaptive brightness target value to replace the fixed value p(i, j) to make the weight more adaptable to scene changes

[0059] S332, adaptive adjustment of local brightness target value

[0060] The brightness target value p(i, j) is calculated based on the average value of the brightest and darkest images, and is improved to a local adaptive brightness target value plocal(i, j) that considers the brightness distribution in the local area:

[0061]

[0062] Where plocal(i, j) represents the average brightness value in the n×n window around pixel (i, j);

[0063] S333, local and global brightness target value combination

[0064] Combined with the global brightness target value pglobal(i, j), the weight calculation takes into account the global exposure balance

[0065]

[0066] Where IB(i, j) and ID(i, j) are the brightness values ​​of the brightest and darkest images at (i, j);

[0067] Introduce a balancing factor λ:

[0068] pfinal (i,j)=λ·p local (i,j)+(1-λ)·p global (i,j)

[0069] λ is adaptively adjusted according to the contrast and texture complexity of the image;

[0070] S334. Calculation of final brightness weight

[0071] According to the improved brightness target value, the formula of brightness weight can be changed to:

[0072]

[0073] Among them, σ is the scale parameter of Gaussian filtering.

[0074] Furthermore, the specific formula of multi-resolution weighted fusion in step S4 is:

[0075]

[0076] The fused result Y′ of the brightness channel Y can be reconstructed by performing an inverse Laplace pyramid transform on the fused pyramid.

[0077] Furthermore, the chrominance component fusion in step S5 is to fuse the Cb and Cr channels of the input color multi-exposure sequence image, and the expression of the pixel point (i, j) weight is:

[0078]

[0079] Among them, the parameter explanation U k (i, j) and V k (i, j) represent the pixel values ​​of the Cb and Cr channels in the kth image, respectively. k (i, j) represents the pixel value of the brightness (or YY component) channel in the kth image; M is a constant used to normalize the weights to ensure that the sum of all weights is 1; ∈ is a very small positive number used to prevent the denominator from being zero;

[0080] Progressive exposure control: Use a smooth function to transition the brightness, and use a Sigmoid function or a linear attenuation formula to gradually reduce the brightness when it approaches 200:

[0081]

[0082] Among them, α controls the decay rate after the brightness exceeds 200. In this way, the disappearance of chroma is smoother.

[0083] The final results of the optimized cb and cr channels are:

[0084]

[0085] Among them, T k (i, j) is the brightness adjustment factor of the pixel (i, j) in the kth image, which is used to reduce the brightness of pixels with brightness values ​​close to 200 to achieve a smoother transition; Y k (i, j) represents the pixel value of the brightness channel in the kth image; α is a parameter that controls the slope of the Sigmoid function. A larger α value will make the function steeper, while a smaller value will make the function smoother. Take α = 0.1;

[0086] T k (i, j) is the brightness adjustment factor of pixel (i, j) in the kth image, is the weight of the pixel (i, j) in the Cb channel of the kth image, is the weight of the pixel (i, j) in the Cr channel of the kth image, U k (i, j) and V k (i, j) represent the pixel values ​​of the Cb and Cr channels in the kth image, respectively, and N is the number of input color multi-exposure sequence images.

[0087] Furthermore, the transformation formula in step S6 is:

[0088] R′=Y+1.402×(Cr-128)

[0089] G′=Y-0.344×(Cb-128)-0.714×(Cr-128)

[0090] B′=Y+1.772×(Cb-128)

[0091] After the transformation, the RGB value should be checked. If it is greater than 255, it is assigned to 255. If it is less than 0, it is assigned to 0.

[0092] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention adopts a brand-new structure, innovatively combining the prism with the FPGA. By utilizing the ability of FPGA to process data in parallel and the spectroscopic characteristics of the prism, multiple images with different exposures can be acquired simultaneously, simplifying the acquisition process of multiple exposure images. The image transmission module uses an Ethernet interface and a precise clock synchronization mechanism to ensure that data can be transmitted to the host computer in real time and losslessly. The host computer can interact with the slave computer to control the image transmission process, select the storage and fusion of the corresponding images, and select the transmission frame rate. This design makes the system highly flexible and operable. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 The schematic diagram is a structural diagram of a multi-exposure image acquisition and processing system based on FPGA and prism.

[0094] Figure 2 These are three views of the prism module.

[0095] Figure 3 This is the FPGA working block diagram.

[0096] Figure 4 It is a schematic diagram of the upper computer display interface 1.

[0097] Figure 5 It is a schematic diagram of the upper computer display interface 2.

[0098] Figure 6 It is a schematic diagram of the upper computer display interface 3. DETAILED DESCRIPTION

[0099] The following will describe the implementation methods of the present invention in detail with reference to the accompanying drawings and examples, so as to fully understand and implement the implementation process of how the present invention applies technical means to solve technical problems and achieve technical effects.

[0100] A multi-exposure image acquisition and processing system based on FPGA and prism comprises a prism module, an image acquisition module, an FPGA module, an image transmission module and a host computer module.

[0101] like Figure 2 As shown in the figure, the prism module adopts an integrated cascaded four-splitting structure, which is in an L shape. The prism module divides the incoming light into four light paths, so that the four image sensors can capture images with different exposure conditions in the same scene. This optical design can collect multiple images with different exposures at the same time, effectively avoiding the motion artifact problem caused by the traditional multiple exposure method in dynamic scenes, thereby improving the image fusion effect in dynamic scenes.

[0102] The image acquisition module is composed of four high-performance image sensors and equipped with a 5-megapixel high-definition lens to ensure higher image quality. The image acquisition module uses the photosensitive chip OV5640 and installs a 5-megapixel lens with an aperture adjustment range of F2.0-C. By adjusting the lens aperture and matching the lens with the prism module, four clear images with different exposures can be obtained at the same time.

[0103] The FPGA module is responsible for image data processing and coordination. The module integrates an efficient parallel processing unit that can process multi-channel image data simultaneously and cache data through the DDR3 storage module. The FPGA can quickly complete the transmission of image data to the host computer by processing multiple exposure images in parallel. In addition, the clock module of the FPGA module is used to coordinate the work of various parts of the system to ensure the synchronization and stability of data processing.

[0104] We use Ethernet for image transmission module. The advantages of transmission are high-speed connection, high bandwidth, maturity, standardization, flexibility, cost-effectiveness and reliability. The transmission protocol uses UDP transmission protocol. Compared with TCP transmission protocol, UDP protocol is more practical for real-time video transmission.

[0105] The host computer module has the function of real-time image display and built-in image fusion module, which uses multi-exposure fusion algorithm to process images to present high dynamic range, clear and delicate images. Through these processing algorithms, the system can generate high dynamic range and rich details of the final image. In addition, the host computer also supports user-defined parameter settings and real-time adjustments to meet different image processing requirements.

[0106] The specific working process of the present invention is as follows: first, the FPGA module configures four image sensor modules through the IIC protocol, and then combines the four image sensor modules with the prism module. The prism module adopts an integrated cascade 4-beam splitting structure, which can divide the incident light into four paths, each path corresponding to a different exposure level, and irradiates the four image sensors respectively. The four image sensors can simultaneously output four picture data streams with different exposure values, which are transmitted to the FPGA module through LVDS; after the FPGA module obtains the image data, it caches it in the DDR3 module. The function of the DDR3 module is to cache the image data sent by the image sensor to the FPGA module. The transmission rate of DDR3 can reach 1.6Gbps, and 8 modules can be carried inside, which can well meet the system's requirements for data transmission speed; the image transmission module will read the cached image data in DDR3, package it and transmit it to the host computer through Ethernet; after the host computer receives the data transmitted through Ethernet, it will display it in real time on the display interface, and a fusion algorithm is built in, and the fused image data can be seen on the fusion interface.

[0107] The above describes the structure and processing of the present invention in detail, but does not introduce the built-in fusion algorithm of the upper machine. Based on the above structure, the fusion algorithm of the upper machine is described in detail below.

[0108] The specific process of the multi-exposure fusion algorithm is as follows:

[0109] Step S1: First, align the images with different exposures. Matching and aligning the images can ensure that the pixel positions of multiple images are consistent, avoiding ghosting and blurred details caused by misalignment. The specific process is as follows:

[0110] Step S11, feature extraction: extract feature points and descriptors from each image using the ORB (Oriented FAST and Rotation Invariant BRIEF) algorithm, which can maintain computational efficiency and accuracy.

[0111] Step S12, feature matching: use BFMatcher to match feature points and find the best matching point for each feature through a brute force matching algorithm.

[0112] Step S13, removing mismatches (RANSAC): removing mismatches RANSAC removes mismatches that do not conform to the transformation model, and retains matching points that conform to the geometric transformation.

[0113] Step S14, image alignment: align the images using the calculated transformation matrix. 1.4 This step ensures that the images with different exposures are aligned at the pixel level.

[0114] Step S2, image color space conversion: adopt a simple YCbCr model to implement the algorithm, extract the values ​​of the three channels R, G, and B of each pixel respectively, and use the formula to convert the color space. The conversion formula is as follows:

[0115] The calculation formula of Y brightness is:

[0116] Y=0.299×R+0.587×G+0.114×B+16

[0117] The calculation formula of Cb chromaticity blue component is:

[0118] U=-0.1687×R-0.3313×G+0.5×B+128

[0119] The calculation formula of Cr chromaticity red component is:

[0120] V=0.5×R-0.4187×G-0.0813×B+128

[0121] In the formula: R, G, B∈[0, 255]; Y channel component Y∈[16, 235]; Cb, Cr channel chrominance components U, V∈[16, 235].

[0122] Step S3: The brightness signal Y reflects the brightness information of the image, including the main contour information and most of the details of the object. Processing the Y component separately can effectively avoid the influence of color noise on the final fusion result. Therefore, a weighted fusion method based on the Laplacian pyramid is used to fuse the brightness signal Y component.

[0123] Step S31, weight index selection needs to determine the weight index used for fusion, and assign different weights to the input images according to each weight index, that is, assign a larger weight to pixels with better quality during the fusion process.

[0124] Areas with high contrast have large changes in brightness and often contain edge or contour information of the scene. The traditional method is to use Laplace filtering to take the absolute value of the filter response to calculate the contrast weight. Although it has a certain effect, it also has great disadvantages. It is very sensitive to noise, easily amplifies the noise of the image, and may produce artifacts when processing pictures. The present invention proposes a new method for contrast weight calculation. This method combines local gradient, texture complexity and directional analysis to adaptively assign more accurate contrast weights to each pixel. The specific process is as follows:

[0125] S311, gradient amplitude calculation

[0126] Use the Sobel operator to calculate the horizontal and vertical gradients of the image:

[0127] G x (i,j)=Y k (i+1,j)-Y k (i-1,j)

[0128] G y (i,j)=Y k (i,j+1)-Y k (i,j-1)

[0129] The gradient magnitude G(i, j) represents the rate of change of brightness at the pixel.

[0130]

[0131] S312, Texture complexity calculation

[0132] The gray level co-occurrence matrix GLCM of the local area is used to measure the complexity of the texture. GLCM can effectively capture the local texture features of the image:

[0133]

[0134] Among them, Pk(p, q) is the element of the gray level co-occurrence matrix, which represents the co-occurrence frequency between gray values ​​p and q, Tk(i, j) represents the texture complexity of the local area, and Tk(i, j) indicates that the details of the local area of ​​the image are rich;

[0135] S313 Adaptive Fusion

[0136] The gradient and texture complexity are combined to generate the final contrast weight through adaptive weighting. The contrast weight calculation formula is:

[0137]

[0138] Among them, α is an adaptive parameter that controls the ratio of gradient and texture weight, max(G) and max(T) are the maximum values ​​of gradient amplitude and texture complexity, respectively, which are used for normalization;

[0139] S314, adaptive parameter α calculation

[0140] The parameter α is dynamically adjusted according to the smoothness of the local area. In noisy or smooth areas, it tends to reduce the influence of the gradient and increase the weight of texture complexity. In edge or detail-rich areas, the weight of the gradient is increased. α is calculated by local variance:

[0141]

[0142] Among them, σ2(i, j) is the brightness variance of the local area near pixel (i, j), It is a threshold used to control smoothness. When the variance of the local area is large, it means that there are edge or texture features, and the weight of the gradient is increased; when the variance of the local area is small, it tends to increase the weight of the texture;

[0143] S32, saturation weight index:

[0144] Saturation reflects the purity of color and is an important factor in highlighting the richness of image details. Under moderate exposure, the color expression of the image is the best, so it is believed that the area with the highest color degree has the brightness that is most suitable for the characteristics of the human visual system. The calculation formula for saturation is

[0145]

[0146] Among them, m k is the average of the luminance and chrominance channels, Y k (i, j), U k (i, j) and V k (i, j) represent the pixel value of the image at the point (i, j) of the kth color channel. In the YUV color space, Y represents the brightness component, while U and V represent the chrominance components.

[0147]

[0148] S33, exposure weight index,

[0149] In the traditional algorithm, a Gaussian curve is used to weight each intensity according to the distance between each pixel brightness component Y and 0.5 to obtain an appropriate exposure. However, for the brighter and darker areas in the image, not all expected optimal pixel brightness values ​​are the same, that is, not all pixels with intensities near 0.5 are optimal. This algorithm may eliminate valid brighter and darker pixels and lose detail information. Based on this, the present invention introduces an adaptive brightness target value and adjusts it based on local and global brightness differences to address the dynamic range problem in different scenarios. The specific calculation process is as follows:

[0150] S331, Adaptive brightness weight formula: Introduce a local adaptive brightness target value to replace the fixed value p(i, j) to make the weight more adaptable to scene changes

[0151] S332, adaptive adjustment of local brightness target value

[0152] The brightness target value p(i, j) is calculated based on the average value of the brightest and darkest images, and is improved to a local adaptive brightness target value plocal(i, j) that considers the brightness distribution in the local area:

[0153]

[0154] Where plocal(i, j) represents the average brightness value in the n×n window around pixel (i, j);

[0155] S333, local and global brightness target value combination

[0156] Combined with the global brightness target value pglobal(i, j), the weight calculation takes into account the global exposure balance

[0157]

[0158] Where IB(i, j) and ID(i, j) are the brightness values ​​of the brightest and darkest images at (i, j);

[0159] Introduce a balancing factor λ:

[0160] p final (i,j)=λ·p local (i,j)+(1-λ)·p global (i,j)

[0161] λ is adaptively adjusted according to the contrast and texture complexity of the image;

[0162] S334. Calculation of final brightness weight

[0163] According to the improved brightness target value, the formula of brightness weight can be changed to:

[0164]

[0165] Among them, σ is the scale parameter of Gaussian filtering.

[0166] Step S4: Use the Laplacian pyramid to perform multi-resolution weighted fusion on the Y channel of the multi-exposure images.

[0167] Each image is decomposed into different resolutions. The low-resolution image is in the upper layer, which mainly contains the outline information of the scene. The high-resolution image is in the lower layer, which contains the detailed galaxies of the scene, presenting a pyramid-shaped image sequence. Then the pyramid sequence of all images is fused at the corresponding image layer according to the fusion weight to generate a fusion pyramid. Finally, the fusion pyramid is reconstructed to obtain the fused result image. Weighted fusion based on the Laplacian pyramid can be described as:

[0168]

[0169] The fused result Y′ of the brightness channel Y can be reconstructed by performing an inverse Laplace pyramid transform on the fused pyramid.

[0170] Step S5, chrominance component fusion;

[0171] Chroma component fusion is to fuse the Cb and Cr channels of the input color multi-exposure sequence image. The expression of the pixel (i, j) weight is:

[0172]

[0173] Among them, the parameter explanation U k (i, j) and V k (i, j) represent the pixel values ​​of the Cb and Cr channels in the kth image, respectively. k (i, j) represents the pixel value of the brightness (or YY component) channel in the kth image; M is a constant used to normalize the weights to ensure that the sum of all weights is 1; ∈ is a very small positive number used to prevent the denominator from being zero;

[0174] Progressive exposure control: Use a smooth function to transition the brightness, and use a Sigmoid function or a linear attenuation formula to gradually reduce the brightness when it approaches 200:

[0175]

[0176] Among them, α controls the decay rate after the brightness exceeds 200. In this way, the disappearance of chroma is smoother.

[0177] The final results of the optimized cb and cr channels are:

[0178]

[0179] Among them, T k (i, j) is the brightness adjustment factor of the pixel (i, j) in the kth image, which is used to reduce the brightness of pixels with brightness values ​​close to 200 to achieve a smoother transition; Y k (i, j) represents the pixel value of the brightness channel in the kth image; α is a parameter that controls the slope of the Sigmoid function. A larger α value will make the function steeper, while a smaller value will make the function smoother. Take α = 0.1;

[0180] T k (i, j) is the brightness adjustment factor of pixel (i, j) in the kth image, is the weight of the pixel (i, j) in the Cb channel of the kth image, is the weight of the pixel (i, j) in the Cr channel of the kth image, U k (i, j) and V k (i, j) represent the pixel values ​​of the Cb and Cr channels in the kth image, respectively, and N is the number of input color multi-exposure sequence images.

[0181] Step S6: Use a transformation formula to convert the image from the YCbCr color space to the RGB color space, and display the obtained high dynamic range image on a display device. The transformation formula is:

[0182] R′=Y+1.402×(Cr-128)

[0183] G′=Y-0.344×(Cb-128)-0.714×(Cr-128)

[0184] B′=Y+1.772×(Cb-128)

[0185] After the transformation, the RGB value should be checked. If it is greater than 255, it is assigned to 255. If it is less than 0, it is assigned to 0.

[0186] The present invention performs image processing based on FPGA. The FPGA module has powerful parallel processing capabilities, can efficiently process multi-channel image data, and has a built-in DDR3 storage module for image caching to reduce processing delays. The system also integrates real-time transmission functions and uses Ethernet for data transmission to ensure the stability and real-time performance of image data transmission.

[0187] The present invention not only greatly improves the speed and quality of multi-exposure image acquisition and processing, but also reduces the complexity and cost of the system by optimizing the hardware architecture, and is suitable for the integration requirements of portable devices and embedded systems. This design improves the dynamic range imaging effect while ensuring the practicality and economy of the system.

[0188] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-exposure image acquisition and processing system based on FPGA and prism, characterized in that: It includes a prism module, an image acquisition module, an FPGA module, an image transmission module and a host computer module; the prism module divides the incident light into four paths, each path corresponds to a different exposure level, and irradiates the four image sensors of the image acquisition module respectively. The four image sensors collect high-speed image data. The image acquisition module sends the collected image to the FPGA module, and the FPGA module transmits it to the host computer module through the image transmission module; The host computer module has the function of displaying images in real time, and has a built-in image fusion module, which uses a multi-exposure fusion algorithm to process images to present high dynamic range, clear and delicate images.

2. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 1, characterized in that: The prism module adopts an integrated cascaded four-beam splitting structure.

3. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 2, characterized in that: The model of the FPGA module is XC7A35TFGG484I of the Artix-7 series.

4. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 3, characterized in that: The image transmission module adopts UDP transmission protocol.

5. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 4, characterized in that: The specific process of the multi-exposure fusion algorithm is as follows: Step S1, first aligning images with different exposures; Step S2: Use the YCbCr model to extract the values ​​of the three channels R, G, and B of each pixel, and use the formula to convert the color space; Step S3, using a weighted fusion method based on the Laplace pyramid to fuse the Y component of the brightness signal; Step S4, using the Laplacian pyramid to perform multi-resolution weighted fusion on the Y channel of the multi-exposure images; Step S5, chrominance component fusion; Step S6: Use a transformation formula to convert the image from the YCbCr color space to the RGB color space, and display the obtained high dynamic range image on a display device.

6. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 5, characterized in that: The specific steps of image alignment in step S1 are: Step S11, feature extraction: extract feature points and descriptors from each image using the ORB algorithm; Step S12, feature matching: use BFMatcher to match feature points, and find the best matching point for each feature through a brute force matching algorithm; Step S13, removing mismatches: removing mismatches that do not conform to the transformation model, and retaining matching points that conform to the geometric transformation; Step S14, image alignment: aligning the images using the calculated transformation matrix.

7. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 5, characterized in that: The conversion formula of the color space in step S2 is as follows: The calculation formula of Y brightness is: Y=0.299×R+0.587×G+0.114×B+16 The calculation formula of Cb chromaticity blue component is: U=-0.1687×R-0.3313×G+0.5×B+128 The calculation formula of Cr chromaticity red component is: V=0.5×R-0.4187×G-0.0813×B+128 In the formula: R, G, B∈[0, 255]; Y channel component Y∈[16, 235]; Cb, Cr channel chrominance components U, V∈[16, 235].

8. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 5, characterized in that: The specific process of fusion of the brightness components is as follows: Step S31, weight index selection: determine the weight index used for fusion, and assign different weights to the input images according to the weight index, that is, assign a larger weight to pixels with better quality during the fusion process; S311, gradient amplitude calculation Use the Sobel operator to calculate the horizontal and vertical gradients of the image: G x (i,j)=Y k (i+1,j)-Y k (i-1,j) G y (i,j)=Y k (i,j+1)-Y k (i,j-1) The gradient magnitude G(i, j) represents the rate of change of brightness at the pixel. S312, Texture complexity calculation The gray level co-occurrence matrix GLCM of the local area is used to measure the complexity of the texture. GLCM can effectively capture the local texture features of the image: Among them, Pk(p, q) is the element of the gray level co-occurrence matrix, which represents the co-occurrence frequency between gray values ​​p and q, Tk(i, j) represents the texture complexity of the local area, and Tk(i, j) indicates that the details of the local area of ​​the image are rich; S313 Adaptive Fusion The gradient and texture complexity are combined to generate the final contrast weight through adaptive weighting. The contrast weight calculation formula is: Among them, α is an adaptive parameter that controls the ratio of gradient and texture weight, max(G) and max(T) are the maximum values ​​of gradient amplitude and texture complexity, respectively, which are used for normalization; S314, adaptive parameter α calculation The parameter α is dynamically adjusted according to the smoothness of the local area, and α is calculated by the local variance: Among them, σ2(i, j) is the brightness variance of the local area near pixel (i, j), It is a threshold used to control smoothness. When the variance of the local area is large, it means that there are edge or texture features, and the weight of the gradient is increased; when the variance of the local area is small, it tends to increase the weight of the texture; S32, saturation weight index: The calculation formula of saturation is Among them, m k is the average of the luminance and chrominance channels, Y k (i, j), U k (i, j) and V k (i, j) represent the pixel value of the image at the point (i, j) of the kth color channel. In the YUV color space, Y represents the brightness component, while U and V represent the chrominance components. S33, exposure weight index, the specific calculation process is as follows: S331, Adaptive brightness weight formula: Introduce a local adaptive brightness target value to replace the fixed value p(i, j) to make the weight more adaptable to scene changes S332, adaptive adjustment of local brightness target value The brightness target value p(i, j) is calculated based on the average value of the brightest and darkest images, and is improved to a local adaptive brightness target value plocal(i, j) that considers the brightness distribution in the local area: Where plocal(i, j) represents the average brightness value in the n×n window around pixel (i, j); S333, local and global brightness target value combination Combined with the global brightness target value pglobal(i, j), the weight calculation takes into account the global exposure balance Where IB(i, j) and ID(i, j) are the brightness values ​​of the brightest and darkest images at (i, j); Introduce a balancing factor λ: p final (i,j)=λ·p local (i,j)+(1-λ)·p global (i,j) λ is adaptively adjusted according to the contrast and texture complexity of the image; S334. Calculation of final brightness weight According to the improved brightness target value, the formula of brightness weight can be changed to: Among them, σ is the scale parameter of Gaussian filtering.

9. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 5, characterized in that: The specific formula of multi-resolution weighted fusion in step S4 is: The fused result Y′ of the brightness channel Y can be reconstructed by performing an inverse Laplace pyramid transform on the fused pyramid.

10. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 5, characterized in that: The chrominance component fusion in step S5 is to fuse the Cb and Cr channels of the input color multi-exposure sequence image, and the expression of the pixel point (i, j) weight is: Among them, the parameter explanation U k (i, j) and V k (i, j) represent the pixel values ​​of the Cb and Cr channels in the kth image, respectively. k (i, j) represents the pixel value of the brightness (or YY component) channel in the kth image; M is a constant used to normalize the weights to ensure that the sum of all weights is 1; ∈ is a very small positive number used to prevent the denominator from being zero; Progressive exposure control: Use a smooth function to transition the brightness, and use a Sigmoid function or a linear attenuation formula to gradually reduce the brightness when it approaches 200: Among them, α controls the decay rate after the brightness exceeds 200. In this way, the disappearance of chroma is smoother. The final results of the optimized cb and cr channels are: Among them, T k (i, j) is the brightness adjustment factor of the pixel (i, j) in the kth image, which is used to reduce the brightness of pixels with brightness values ​​close to 200 to achieve a smoother transition; Y k (i, j) represents the pixel value of the brightness channel in the kth image; α is a parameter that controls the slope of the Sigmoid function. A larger α value will make the function steeper, while a smaller value will make the function smoother. Take α = 0.1; T k (i, j) is the brightness adjustment factor of pixel (i, j) in the kth image, is the weight of the pixel (i, j) in the Cb channel of the kth image, is the weight of the pixel (i, j) in the Cr channel of the kth image, U k (i, j) and V k (i, j) represent the pixel values ​​of the Cb and Cr channels in the kth image, respectively, and N is the number of input color multi-exposure sequence images.

11. The multi-exposure image acquisition and processing system based on FPGA and prism according to claim 5, characterized in that: The transformation formula in step S6 is: R′=Y+1.402×(Cr-128) G′=Y-0.344×(Cb-128)-0.714×(Cr-128) B′=Y+1.772×(Cb-128) After the transformation, the RGB value should be checked. If it is greater than 255, it is assigned to 255. If it is less than 0, it is assigned to 0.

Images