A HDR image generation method and system based on multi-sensor imaging device
By adaptively adjusting the exposure parameters and fusion of images in the HSI space through the multi-sensor imaging device, the problem of unclear HDR images and 'ghosting' in dynamic scenes is solved, and HDR images are generated with high definition and high frame rate.
Patent Information
- Application Number
- CN202211719776.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-12-30
AI Technical Summary
In the prior art, the HDR images fused by multi-sensors are not clear, and the exposure parameters are fixed and do not adapt to changes in the brightness range, resulting in a "ghost" phenomenon in dynamic scenes, low frame rate and damaged image clarity.
Using an HDR image generation method based on a multi-sensor imaging device, the external light is divided into N beams through a spectroscope, and the N sensors are exposed respectively. The exposure parameters are adaptively adjusted according to the scene brightness range to generate an HDR image. This method performs image fusion in the HSI space to reduce the color cast problem.
The generated HDR images have the characteristics of high definition, no ghosting in dynamic scenes and high frame rate. They adapt to scenes with different brightness ranges and improve image quality.
Smart Images

Figure CN115988329B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of digital image imaging technology, and in particular to a HDR image generation method and system based on a multi-sensor imaging device. Background Art
[0002] Dynamic range is one of the important indicators for evaluating the quality of digital imaging. High dynamic range (HDR) images or videos can more comprehensively and truly reflect the information of bright and dark fields. However, due to current image sensor technology, the dynamic range of a single exposure still cannot reach the maximum dynamic range that the human eye can perceive.
[0003] High dynamic range is generally achieved by fusing multiple frames of images obtained by multiple exposures of a camera (or multiple cameras) at the same time or in a similar time. There are generally two ways to achieve this: 1) Continuous multiple exposures with a single camera, but this method will cause a loss in frame rate (the frame rate is reduced by half or more), and "ghosting" is prone to occur in dynamic scenes, seriously affecting image quality; 2) Simultaneous exposure of multiple cameras can maintain the frame rate while preventing "ghosting" from occurring in dynamic scenes, but due to the different positions and angles of multiple cameras, the positions of the same target in the images obtained by exposure at the same time are quite different, resulting in serious damage to the clarity of the image after HDR fusion.
[0004] Generally, HDR exposure uses fixed exposure parameters, such as using lower fixed exposure parameters for shooting bright fields and higher fixed exposure parameters for shooting dark fields. This method is more suitable for situations where the scene brightness range is fixed, but it is not the best choice when the scene brightness range is constantly changing.
[0005] In addition, ordinary HDR fusion is fused separately in RGB space. Since the photoelectric response curve of the sensor in the dark field or bright field is not a perfect linear relationship, it is easy to cause color cast in the fused image. Summary of the invention
[0006] The purpose of the present invention is to provide an HDR image generation method and system based on a multi-sensor imaging device, which solves the problem of unclear HDR images fused by each sensor. The exposure parameters are adaptively optimized in real time according to the brightness range of the scene. The generated HDR image has the characteristics of high clarity, no "ghosting" in dynamic scenes, and high frame rate. HDR fusion adopts HSI space fusion to reduce the fusion color cast problem.
[0007] To achieve the above object, the present invention provides the following solutions:
[0008] A HDR image generation method based on a multi-sensor imaging device, the multi-sensor imaging device includes a lens, a beam splitter prism and N sensors, external light enters the beam splitter prism through the lens, the beam splitter prism splits the external light according to a set ratio based on the intensity of the external light to obtain N light beams, the N sensors obtain N RGB images based on the N light beams, N is a positive integer greater than or equal to 2, the HDR image generation method includes:
[0009] Obtain N current frame RGB images based on the multi-sensor imaging device;
[0010] A first current frame light metering value is obtained based on the first current frame RGB image, and an Nth current frame light metering value is obtained based on the Nth current frame RGB image;
[0011] The first current frame total gain is obtained based on the first current frame light metering value and the first set light metering value; the Nth current frame total gain is obtained based on the Nth current frame light metering value and the Nth set light metering value;
[0012] The total gain of the kth current frame is obtained based on the total gain of the 1st current frame and the total gain of the Nth current frame; k∈N;
[0013] Adjusting the exposure parameter of the first sensor based on the first set gain until the absolute value of the first current frame total gain is less than the first set gain, thereby obtaining the first next frame total gain;
[0014] Adjusting the exposure parameter of the Nth sensor based on the Nth set gain until the absolute value of the total gain of the Nth current frame is less than the Nth set gain, thereby obtaining the total gain of the Nth next frame;
[0015] Obtaining a kth next frame total gain based on the 1st next frame total gain and the Nth next frame total gain, and adjusting the exposure parameter of the kth sensor based on the kth next frame total gain until the kth current frame total gain reaches the kth next frame total gain;
[0016] The N sensors adjusted based on the exposure parameters obtain N next-frame RGB images; convert the N next-frame RGB images to obtain N next-frame HSI images;
[0017] Perform image fusion on the N next-frame HSI images to obtain a next-frame HSI fused image; and perform conversion on the next-frame HSI fused image to obtain a next-frame RGB fused image.
[0018] Preferably, obtaining the first current frame light metering value based on the first current frame RGB image, and obtaining the Nth current frame light metering value based on the Nth current frame RGB image, comprises:
[0019] Based on the first current frame RGB image, the first brightness Y image is obtained. The first brightness Y image is divided into blocks according to the set size to obtain M first blocks. The first Y mean of each first block is calculated to obtain M first Y means. N first Y means are selected in ascending order. peak The first Y mean values are calculated and the average value is obtained to obtain the first current frame light metering value; M is a positive integer greater than 1;
[0020] Based on the Nth current frame RGB image, the Nth brightness Y image is obtained. The Nth brightness Y image is divided into blocks according to the set size to obtain M Nth blocks. The NYth mean of each Nth block is calculated to obtain M NYth means. N is selected in descending order. peak The NY-th mean values are calculated and the average value is obtained to obtain the N-th current frame light metering value.
[0021] Preferably, obtaining the first current frame total gain based on the first current frame light metering value and the first set light metering value; obtaining the Nth current frame total gain based on the Nth current frame light metering value and the Nth set light metering value comprises:
[0022] The first set light metering value is divided by the first current frame light metering value to obtain the first current frame light metering ratio; the first current frame total gain is obtained according to the first current frame light metering ratio;
[0023] The Nth set light metering value is divided by the Nth current frame light metering value to obtain the Nth current frame light metering ratio; and the Nth current frame total gain is obtained according to the Nth current frame light metering ratio.
[0024] Preferably, the kth next frame total gain is obtained based on the 1st next frame total gain and the Nth next frame total gain, and the calculation formula is as follows:
[0025]
[0026] Where: is the total gain of the first next frame, is the total gain of the next Nth frame, K1 is the ratio of the intensity of the external light received by the first sensor, K N is the ratio of the intensity of external light received by the Nth sensor, is the total gain of the next k-th frame.
[0027] Preferably, the exposure parameters include aperture size, exposure time, analog gain and digital gain.
[0028] Preferably, performing image fusion on the N next HSI frames to obtain the next HSI fused image includes:
[0029] Fusing the I channels of the N next frame HSI images to obtain an I channel fused image;
[0030] Fusing the S channels of the N next frame HSI images to obtain an S channel fused image;
[0031] Performing vector conversion on the H channels of the N next frame HSI images to obtain N first vectors and N second vectors;
[0032] Performing weighted averaging on the N first vectors to obtain a first average vector; performing weighted averaging on the N second vectors to obtain a second average vector;
[0033] An H channel fused image is obtained based on the first average vector and the second average vector.
[0034] Preferably, the exposure parameters are adjusted in the order of aperture size, exposure time, analog gain and digital gain.
[0035] The present invention also provides an HDR image generation system based on a multi-sensor imaging device, wherein the multi-sensor imaging device includes a lens, a beam splitter prism and N sensors, the external light enters the beam splitter prism through the lens, the beam splitter prism splits the external light according to a set ratio based on the intensity of the external light to obtain N light beams, the N sensors obtain N RGB images based on the N light beams, N is a positive integer greater than or equal to 2, and the HDR image generation system includes: a data acquisition module, a light measurement value module, a total gain module, a parameter adjustment module, and an image fusion conversion module;
[0036] The data acquisition module is used to obtain N current frame RGB images based on the multi-sensor imaging device;
[0037] The light metering value module is used to obtain the first current frame light metering value based on the first current frame RGB image, and obtain the Nth current frame light metering value based on the Nth current frame RGB image;
[0038] The total gain module is used to obtain the first current frame total gain based on the first current frame light metering value and the first set light metering value, and to obtain the Nth current frame total gain based on the Nth current frame light metering value and the Nth set light metering value;
[0039] The total gain module is further used to obtain the kth current frame total gain based on the 1st current frame total gain and the Nth current frame total gain; k∈N;
[0040] The parameter adjustment module is used to adjust the exposure parameter of the first sensor based on the first set gain until the absolute value of the first current frame total gain is less than the first set gain, thereby obtaining the first next frame total gain;
[0041] The parameter adjustment module is further used to adjust the exposure parameter of the Nth sensor based on the Nth set gain until the absolute value of the total gain of the Nth current frame is less than the Nth set gain, thereby obtaining the total gain of the Nth next frame;
[0042] The total gain module is further used to obtain a kth next frame total gain based on the 1st next frame total gain and the Nth next frame total gain, and adjust the exposure parameter of the kth sensor based on the kth next frame total gain until the kth current frame total gain reaches the kth next frame total gain;
[0043] The image fusion conversion module is used to obtain N next-frame RGB images based on the N sensors adjusted with exposure parameters, and convert the N next-frame RGB images to obtain N next-frame HSI images;
[0044] The image fusion conversion module is also used to perform image fusion on the N next frame HSI images to obtain the next frame HSI fused image, and convert the next frame HSI fused image to obtain the next frame RGB fused image.
[0045] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0046] The present invention relates to an HDR image generation method and system based on a multi-sensor imaging device. The multi-sensor imaging device comprises a lens, a beam splitter prism and N sensors. External light enters the beam splitter prism through the lens. The beam splitter prism splits the external light according to a set ratio based on the intensity of the external light to obtain N light beams. The N sensors obtain N RGB images based on the N light beams, where N is a positive integer greater than or equal to 2. The HDR image generation method comprises: obtaining N current frame RGB images based on the multi-sensor imaging device; obtaining a first current frame photometric value based on the first current frame RGB image, and obtaining an Nth current frame photometric value based on the Nth current frame RGB image; obtaining a first current frame total gain based on the first current frame photometric value and a first set photometric value; obtaining an Nth current frame total gain based on the Nth current frame photometric value and the Nth set photometric value; obtaining a kth current frame total gain based on the first current frame total gain and the Nth current frame total gain; k∈N ; Based on the first set gain, the exposure parameter of the first sensor is adjusted until the absolute value of the total gain of the first current frame is less than the first set gain, and the first next frame total gain is obtained; Based on the Nth set gain, the exposure parameter of the Nth sensor is adjusted until the absolute value of the total gain of the Nth current frame is less than the Nth set gain, and the Nth next frame total gain is obtained; Based on the first next frame total gain and the Nth next frame total gain, the kth next frame total gain is obtained, and the exposure parameter of the kth sensor is adjusted based on the kth next frame total gain until the total gain of the kth current frame reaches the kth next frame total gain; Based on the N sensors after the exposure parameters are adjusted, N next frame RGB images are obtained; The N next frame RGB images are converted to obtain N next frame HSI images; The N next frame HSI images are image fused to obtain a next frame HSI fused image; The next frame HSI fused image is converted to obtain a next frame RGB fused image. The present invention solves the problem of unclear HDR images fused by various sensors. Multiple sensors simultaneously perform exposure with different exposure parameters to obtain images of different brightness, and the exposure parameters are adaptively optimized in real time according to the brightness range of the scene. If the brightness range of the scene is large, the exposure parameters of each sensor will be quite different. In this way, HDR fusion is performed again. The generated HDR image has the characteristics of high clarity, no "ghosting" in dynamic scenes, and high frame rate. HDR fusion adopts HSI space fusion to reduce the problem of fusion color cast. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0048] Figure 1 This is a flow chart of a method for generating an HDR image based on a multi-sensor imaging device according to the present invention;
[0049] Figure 2 This is a structural diagram of the HDR image generation system based on a multi-sensor imaging device of the present invention.
[0050] Explanation of symbols: 1. Data acquisition module; 2. Light measurement value module; 3. Total gain module; 4. Parameter adjustment module; 5. Image fusion conversion module. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] The purpose of the present invention is to provide an HDR image generation method and system based on a multi-sensor imaging device, which solves the problem of unclear HDR images fused by each sensor. The exposure parameters are adaptively optimized in real time according to the brightness range of the scene. The generated HDR image has the characteristics of high clarity, no "ghosting" in dynamic scenes, and high frame rate. HDR fusion adopts HSI space fusion to reduce the fusion color cast problem.
[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0054] Figure 1 FIG. 1 is a flow chart of the HDR image generation method based on a multi-sensor imaging device of the present invention. Figure 1 As shown, the present invention provides an HDR image generation method based on a multi-sensor imaging device, wherein the multi-sensor imaging device includes a lens, a beam splitter prism and N sensors, the external light enters the beam splitter prism through the lens, the beam splitter prism splits the external light according to a set ratio based on the intensity of the external light to obtain N light beams, and the N sensors obtain N RGB images based on the N light beams, where N is a positive integer greater than or equal to 2. The set ratio is: K1: K2: …: KN , and K1≥K2≥…≥K N ; Where: K1 is the ratio of the intensity of the external light received by the first sensor, K2 is the ratio of the intensity of the external light received by the second sensor, K N is the ratio of the intensity of the external light received by the Nth sensor.
[0055] The HDR image generation method comprises:
[0056] Step S1, obtaining N current frame RGB images based on the multi-sensor imaging device.
[0057] Step S2, obtaining the first current frame light metering value based on the first current frame RGB image, and obtaining the Nth current frame light metering value based on the Nth current frame RGB image.
[0058] Specifically, step S2 includes:
[0059] Step S21, based on the first current frame RGB image, obtain the first brightness Y image, divide the first brightness Y image into blocks according to the set size, obtain M first blocks, calculate the first Y mean value of each first block, obtain M first Y mean values, and select N first Y mean values in ascending order. peak The first Y mean values are calculated and the average value is obtained to obtain the first current frame light measurement value; M is a positive integer greater than 1. In this embodiment, the size is set to 16*16, N peak Take 500, assuming that the resolution of the first brightness Y image is Q*W, then the number of the first blocks M = (Q / 16)*(W / 16). The calculation formula for the first brightness Y image is as follows:
[0060] Y(i,j)=0.2126R(i,j)+0.7152G(i,j)+0.0722B(i,j);
[0061] Where: R(i,j) represents the R channel value of the pixel in the i-th row and j-th column of the first current frame RGB image, G(i,j) represents the G channel value of the pixel in the i-th row and j-th column of the first current frame RGB image, B(i,j) represents the B channel value of the pixel in the i-th row and j-th column of the first current frame RGB image, and Y(i,j) represents the Y value of the pixel in the i-th row and j-th column of the first current frame RGB image.
[0062] Step S22, based on the Nth current frame RGB image, obtain the Nth brightness Y image, divide the Nth brightness Y image into blocks according to the set size, obtain M Nth blocks, calculate the Nth mean value of each Nth block, obtain M Nth mean values, select N in descending order peakThe N-th mean values are calculated and the average value is obtained to obtain the N-th current frame light measurement value. The calculation method of the N-th brightness Y image refers to the calculation method of the first brightness Y image.
[0063] Step S3, obtaining the first current frame total gain based on the first current frame light metering value and the first set light metering value; obtaining the Nth current frame total gain based on the Nth current frame light metering value and the Nth set light metering value.
[0064] Furthermore, the step S3 comprises:
[0065] Step S31, dividing the first set light metering value by the first current frame light metering value to obtain the first current frame light metering ratio; and obtaining the first current frame total gain according to the first current frame light metering ratio.
[0066] The formula for calculating the metering ratio of the first current frame is as follows:
[0067]
[0068] Where: APM1 is the first current frame metering value, is the first set light metering value, and r1 is the first current frame light metering ratio. In this embodiment, Take 240.
[0069] The calculation formula for the total gain of the first current frame is as follows:
[0070] dBr1=20*log(r1);
[0071] Where: dBr1 represents the total gain of the first current frame.
[0072] Step S32, dividing the Nth set light metering value by the Nth current frame light metering value to obtain the Nth current frame light metering ratio; and obtaining the Nth current frame total gain according to the Nth current frame light metering ratio.
[0073] The calculation formula for the metering ratio of the Nth current frame is as follows:
[0074]
[0075] Where: Set the metering value for the Nth time, APM N is the metering value of the Nth current frame, r N It is the metering ratio value of the Nth current frame.
[0076] The total gain calculation formula of the Nth current frame is as follows:
[0077] dBr N =20*log(r N );
[0078] Where: dBr N Indicates the total gain of the first current frame.
[0079] Step S4, obtaining the kth current frame total gain based on the 1st current frame total gain and the Nth current frame total gain; k∈N. For the specific calculation formula, refer to the calculation formula of the kth next frame total gain.
[0080] Step S5, adjusting the exposure parameters of the first sensor based on the first set gain until the absolute value of the first current frame total gain is less than the first set gain, thereby obtaining the first next frame total gain. In this embodiment, the exposure parameters include aperture size, exposure time, analog gain, and digital gain.
[0081] Preferably, if |dBr1|≤δ1, it means that the light measurement value is reached and there is no need to adjust the exposure parameter. δ1 is the first set gain. In this embodiment, δ1 is 0.25.
[0082] If dBr1<-δ1, it means the metering value is small, indicating that the exposure parameter is small and needs to be increased. The adjustment principle is: (1) According to the total gain increase value of dBr1, adjust the exposure parameters upward: aperture size, exposure time, analog gain, digital gain; (2) The adjustment of each parameter must be within the definition range of each parameter; (3) First adjust the aperture size, then adjust the exposure time, then adjust the analog gain, and then adjust the digital gain; (4) The adjustment can be divided into multiple frames until |dBr1|<δ1.
[0083] If dBr1>δ1, it means the metering value is large, indicating that the exposure parameter is large and needs to be reduced. The adjustment principle is: (1) adjust the exposure parameters downward according to the goal of reducing the total gain by dBr1: aperture size, exposure time, analog gain, digital gain; (2) the adjustment of each parameter must be within the definition range of each parameter; (3) adjust the digital gain first, then adjust the analog gain if it is not enough, then adjust the exposure time if it is not enough, and then adjust the aperture size if it is not enough; (4) the adjustment can be divided into multiple frames until |dBr1|<δ1.
[0084] Step S6, adjusting the exposure parameter of the Nth sensor based on the Nth set gain until the absolute value of the total gain of the Nth current frame is less than the Nth set gain, thereby obtaining the total gain of the Nth next frame.
[0085] Alternatively, if |dBr N |≤δ N Indicates that the metering value has been reached and there is no need to adjust the exposure parameters. N Set the gain for the Nth.
[0086] If dBrN <-δ N It means that the metering value is small, which means that the exposure parameter is small. The exposure parameter needs to be increased. The adjustment principle is: (1) According to the total gain increase value dBr N Adjust the exposure parameters upward within the adjustment range of aperture size, exposure time, analog gain, and digital gain. First adjust the aperture size, then adjust the exposure time, then adjust the analog gain, and then adjust the digital gain in this order until |dBr N |<δ N until.
[0087] If dBr N >δ N It means that the metering value is large, which means that the exposure parameter is large. The exposure parameter needs to be adjusted down. The adjustment principle is: (1) According to the total gain reduction value dBr N Adjust the exposure parameters downward with the goal of: aperture size, exposure time, analog gain, digital gain. (2) The adjustment of each parameter must be within the definition range of each parameter. (3) Adjust the digital gain first, then adjust the analog gain, then adjust the exposure time, and then adjust the aperture size. (4) Adjustment can be divided into multiple frames until |dBr N |<δ N until.
[0088] In this embodiment, because the first sensor is used to shoot dark fields and requires a larger exposure parameter, and the Nth sensor is used to shoot bright fields and requires a smaller exposure parameter, it is necessary to satisfy that the exposure parameter of the Nth sensor is not greater than the first exposure parameter. Therefore, the value when the absolute value of the total gain of the Nth current frame is less than the Nth set gain is recorded as the Nth initial next frame total gain. The Nth next frame total gain is obtained according to the Nth initial next frame total gain and the first next frame total gain. The calculation formula is as follows:
[0089]
[0090] Where: is the total gain of the first next frame, is the total gain of the next Nth frame.
[0091] Step S7, obtaining the kth next frame total gain based on the 1st next frame total gain and the Nth next frame total gain, and adjusting the exposure parameter of the kth sensor based on the kth next frame total gain until the kth current frame total gain reaches the kth next frame total gain.
[0092] The kth next frame total gain is obtained based on the 1st next frame total gain and the Nth next frame total gain, and the calculation formula is as follows:
[0093]
[0094] Where: is the total gain of the next k-th frame.
[0095] Through this calculation, the total gain of each sensor is reduced in the form of an arithmetic progression, ensuring that N sensors can obtain more effective detail information.
[0096] like If the exposure parameters are not adjusted, The exposure needs to be adjusted upward in the order of aperture size, exposure time, analog gain, and digital gain until the total gain of the kth current frame reaches So far, if The exposure needs to be adjusted downward in the order of digital gain, analog gain, exposure time, and aperture size until the total gain of the kth current frame reaches until, is the total gain of the kth current frame.
[0097] Step S8, obtaining N next-frame RGB images based on the N sensors adjusted based on the exposure parameters; converting the N next-frame RGB images to obtain N next-frame HSI images.
[0098] The calculation formula is as follows:
[0099]
[0100]
[0101]
[0102] Wherein: H is the H channel value, S is the S channel value, I is the I channel value, R is the R channel value, G is the G channel value, and B is the B channel value.
[0103] Step S9, performing image fusion on the N next frame HSI images to obtain the next frame HSI fused image; and converting the next frame HSI fused image to obtain the next frame RGB fused image.
[0104] Preferably, the step S9 comprises:
[0105] Step S91, the I channels of the N next frame HSI images are fused to obtain an I channel fused image. The I channel fusion formula is as follows:
[0106]
[0107] Step S92, the S channels of the N next frame HSI images are fused to obtain an S channel fused image. The S channel fusion formula is as follows:
[0108]
[0109] Where: I fusion (i, j) represents the I component fusion value of the i-th row and j-th column of the next frame of HSI image, S fusion (i, j) represents the S component fusion value of the i-th row and j-th column of the next frame of HSI image. k (i, j) represents the I component value of the i-th row and j-th column of the k-th next frame HSI image, S k (i, j) represents the S component value of the i-th row and j-th column of the k-th next frame HSI image, w k (i,j) represents the weight of the pixel in the i-th row and j-th column of the k-th next frame HSI image. This value is the same as I k (i,j) is related to I k (i,j) is too small or too large, the smaller the weight, I k The more centered (i, j) is, the greater the weight is. This setting is because if the brightness is too small or too large, the brightness and saturation are not very accurate, while it is more accurate when the brightness is moderate.
[0110] Step S93, performing vector conversion on the H channels of the N next frame HSI images to obtain N first vectors and N second vectors.
[0111] Step S94, performing weighted averaging on the N first vectors to obtain a first average vector; performing weighted averaging on the N second vectors to obtain a second average vector.
[0112] The first average vector calculation formula is as follows:
[0113]
[0114] The second average vector calculation formula is as follows:
[0115]
[0116] Where: H k (i, j) represents the H component value of the i-th row and j-th column of the k-th next frame HSI image, cosH k (i,j) is the kth first vector, sinH k (i, j) is the kth vector, C represents the first average vector, and S represents the second average vector.
[0117] Step S95, obtaining an H channel fusion image based on the first average vector and the second average vector. The next frame of HSI fusion image includes the I channel fusion image, the S channel fusion image and the H channel fusion image.
[0118] The H channel fusion image calculation formula is as follows:
[0119]
[0120] Where: H fusion (i, j) represents the H component fusion value of the i-th row and j-th column of the next frame of HSI image.
[0121] Step S96, converting the next frame of HSI fused image to obtain the next frame of RGB fused image.
[0122] The conversion formula is as follows:
[0123]
[0124]
[0125]
[0126] Where: R fusion (i, j) is the R channel value of the i-th row and j-th column of the next frame of RGB fusion image, G fusion (i, j) is the G channel value of the i-th row and j-th column of the next frame of RGB fusion image, B fusion (i, j) is the B channel value of the i-th row and j-th column of the next frame of RGB fused image.
[0127] Figure 2 FIG. 1 is a structural diagram of the HDR image generation system based on a multi-sensor imaging device of the present invention. Figure 2 As shown, the present invention provides an HDR image generation system based on a multi-sensor imaging device, wherein the multi-sensor imaging device includes a lens, a beam splitter prism and N sensors, the external light enters the beam splitter prism through the lens, the beam splitter prism splits the external light according to a set ratio based on the intensity of the external light to obtain N light beams, the N sensors obtain N RGB images based on the N light beams, N is a positive integer greater than or equal to 2, and the HDR image generation system includes: a data acquisition module 1, a light measurement value module 2, a total gain module 3, a parameter adjustment module 4, and an image fusion conversion module 5.
[0128] The data acquisition module 1 is used to obtain N current frame RGB images based on the multi-sensor imaging device.
[0129] The light metering value module 2 is used to obtain the first current frame light metering value based on the first current frame RGB image, and to obtain the Nth current frame light metering value based on the Nth current frame RGB image.
[0130] The total gain module 3 is used to obtain the first current frame total gain based on the first current frame light metering value and the first set light metering value, and to obtain the Nth current frame total gain based on the Nth current frame light metering value and the Nth set light metering value.
[0131] The total gain module 3 is further used to obtain the kth current frame total gain based on the 1st current frame total gain and the Nth current frame total gain; k∈N.
[0132] The parameter adjustment module 4 is used to adjust the exposure parameter of the first sensor based on the first set gain until the absolute value of the first current frame total gain is less than the first set gain, thereby obtaining the first next frame total gain.
[0133] The parameter adjustment module 4 is further used to adjust the exposure parameter of the Nth sensor based on the Nth set gain until the absolute value of the total gain of the Nth current frame is less than the Nth set gain, thereby obtaining the total gain of the Nth next frame.
[0134] The total gain module 4 is also used to obtain the kth next frame total gain based on the 1st next frame total gain and the Nth next frame total gain, and adjust the exposure parameters of the kth sensor based on the kth next frame total gain until the kth current frame total gain reaches the kth next frame total gain.
[0135] The image fusion conversion module 5 is used to obtain N next-frame RGB images based on the N sensors adjusted with exposure parameters, and convert the N next-frame RGB images to obtain N next-frame HSI images.
[0136] The image fusion conversion module 5 is further used to perform image fusion on the N next frame HSI images to obtain the next frame HSI fused image, and convert the next frame HSI fused image to obtain the next frame RGB fused image.
[0137] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0138] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A HDR image generation method based on a multi-sensor imaging device, wherein the multi-sensor imaging device comprises a lens, a beam splitter prism and N sensors, wherein external light enters the beam splitter prism through the lens, and the beam splitter prism splits the external light according to a set ratio based on the intensity of the external light to obtain N light beams, and the N sensors obtain N RGB images based on the N light beams, wherein N is a positive integer greater than or equal to 2, wherein: The HDR image generation method comprises: Obtain N current frame RGB images based on the multi-sensor imaging device; A first current frame light metering value is obtained based on the first current frame RGB image, and an Nth current frame light metering value is obtained based on the Nth current frame RGB image; The first current frame total gain is obtained based on the first current frame light metering value and the first set light metering value; the Nth current frame total gain is obtained based on the Nth current frame light metering value and the Nth set light metering value; The total gain of the kth current frame is obtained based on the total gain of the 1st current frame and the total gain of the Nth current frame; k∈N; Adjusting the exposure parameter of the first sensor based on the first set gain until the absolute value of the first current frame total gain is less than the first set gain, thereby obtaining the first next frame total gain; Adjusting the exposure parameter of the Nth sensor based on the Nth set gain until the absolute value of the total gain of the Nth current frame is less than the Nth set gain, thereby obtaining the total gain of the Nth next frame; Obtaining a kth next frame total gain based on the 1st next frame total gain and the Nth next frame total gain, and adjusting the exposure parameter of the kth sensor based on the kth next frame total gain until the kth current frame total gain reaches the kth next frame total gain; The N sensors adjusted based on the exposure parameters obtain N next-frame RGB images; convert the N next-frame RGB images to obtain N next-frame HSI images; Perform image fusion on the N next-frame HSI images to obtain a next-frame HSI fused image; and perform conversion on the next-frame HSI fused image to obtain a next-frame RGB fused image.
2. The HDR image generation method based on a multi-sensor imaging device according to claim 1, characterized in that: The step of obtaining the first current frame light metering value based on the first current frame RGB image and obtaining the Nth current frame light metering value based on the Nth current frame RGB image includes: Based on the first current frame RGB image, the first brightness Y image is obtained. The first brightness Y image is divided into blocks according to the set size to obtain M first blocks. The first Y mean of each first block is calculated to obtain M first Y means. N first Y means are selected in ascending order. peak The first Y mean values are calculated and the average value is obtained to obtain the first current frame light metering value; M is a positive integer greater than 1; Based on the Nth current frame RGB image, the Nth brightness Y image is obtained. The Nth brightness Y image is divided into blocks according to the set size to obtain M Nth blocks. The NYth mean of each Nth block is calculated to obtain M NYth means. N is selected in descending order. peak The NY-th mean values are calculated and the average value is obtained to obtain the N-th current frame light metering value.
3. The HDR image generation method based on a multi-sensor imaging device according to claim 1, characterized in that: The first current frame total gain is obtained based on the first current frame light metering value and the first set light metering value; The Nth current frame total gain is obtained based on the Nth current frame metering value and the Nth set metering value, including: Divide the first set exposure value by the first current frame exposure value to obtain the first current frame exposure ratio; The total gain of the first current frame is obtained according to the metering ratio value of the first current frame; The Nth set light metering value is divided by the Nth current frame light metering value to obtain the Nth current frame light metering ratio; and the Nth current frame total gain is obtained according to the Nth current frame light metering ratio.
4. The HDR image generation method based on a multi-sensor imaging device according to claim 1, characterized in that: The kth next frame total gain is obtained based on the 1st next frame total gain and the Nth next frame total gain, and the calculation formula is as follows: Where: is the total gain of the first next frame, is the total gain of the next Nth frame, K1 is the ratio of the intensity of the external light received by the first sensor, K N is the ratio of the intensity of external light received by the Nth sensor, is the total gain of the kth next frame.
5. The HDR image generation method based on a multi-sensor imaging device according to claim 1, characterized in that: The exposure parameters include aperture size, exposure time, analog gain and digital gain.
6. The HDR image generation method based on a multi-sensor imaging device according to claim 1, characterized in that: The performing image fusion on the N next-frame HSI images to obtain the next-frame HSI fused image includes: Fusing the I channels of the N next frame HSI images to obtain an I channel fused image; Fusing the S channels of the N next frame HSI images to obtain an S channel fused image; Performing vector conversion on the H channels of the N next frame HSI images to obtain N first vectors and N second vectors; Performing weighted averaging on the N first vectors to obtain a first average vector; performing weighted averaging on the N second vectors to obtain a second average vector; An H channel fusion image is obtained based on the first average vector and the second average vector, and the next frame of HSI fusion image includes the I channel fusion image, the S channel fusion image and the H channel fusion image.
7. The HDR image generation method based on a multi-sensor imaging device according to claim 1, characterized in that: Adjust the exposure parameters in the order of aperture size, exposure time, analog gain and digital gain.
8. An HDR image generation system based on a multi-sensor imaging device, the multi-sensor imaging device comprising a lens, a beam splitter prism and N sensors, the external light enters the beam splitter prism through the lens, the beam splitter prism splits the external light according to a set ratio based on the intensity of the external light to obtain N light beams, the N sensors obtain N RGB images based on the N light beams, N is a positive integer greater than or equal to 2, characterized in that: The HDR image generation system includes: a data acquisition module, a light measurement value module, a total gain module, a parameter adjustment module, and an image fusion conversion module; The data acquisition module is used to obtain N current frame RGB images based on the multi-sensor imaging device; The light metering value module is used to obtain the first current frame light metering value based on the first current frame RGB image, and obtain the Nth current frame light metering value based on the Nth current frame RGB image; The total gain module is used to obtain the first current frame total gain based on the first current frame light metering value and the first set light metering value, and to obtain the Nth current frame total gain based on the Nth current frame light metering value and the Nth set light metering value; The total gain module is further used to obtain the kth current frame total gain based on the 1st current frame total gain and the Nth current frame total gain; k∈N; The parameter adjustment module is used to adjust the exposure parameter of the first sensor based on the first set gain until the absolute value of the first current frame total gain is less than the first set gain, thereby obtaining the first next frame total gain; The parameter adjustment module is further used to adjust the exposure parameter of the Nth sensor based on the Nth set gain until the absolute value of the total gain of the Nth current frame is less than the Nth set gain, thereby obtaining the total gain of the Nth next frame; The total gain module is further used to obtain a kth next frame total gain based on the 1st next frame total gain and the Nth next frame total gain, and adjust the exposure parameter of the kth sensor based on the kth next frame total gain until the kth current frame total gain reaches the kth next frame total gain; The image fusion conversion module is used to obtain N next-frame RGB images based on the N sensors adjusted with exposure parameters, and convert the N next-frame RGB images to obtain N next-frame HSI images; The image fusion conversion module is also used to perform image fusion on the N next frame HSI images to obtain the next frame HSI fused image, and convert the next frame HSI fused image to obtain the next frame RGB fused image.
Citation Information
Patent Citations
Automatic exposure control method and device based on FPGA and suitable for space exploration imaging
CN104184958A
Automobile data recorder intelligent camera automatic exposure method
CN107249104A
