Image synthesis method and system

Abstract

The invention provides an image synthesis method and system, and the method comprises the steps: inputting a to-be-supplemented image sequence into an optimal exposure sequence function of a brightness channel constructed based on a least square method, taking the corresponding number of generated images and the exposure time of each generated image as exposure parameters of the to-be-generated image when the pixel value of the generated image sequence in the brightness channel can cover the scene brightness range of the brightness channel and the information entropy is maximum; taking the exposure parameter of each to-be-generated image as the input of an image generation model constructed on the basis of the illumination curve model, obtaining a supplementary image output by the image generation model, and constructing a supplementary image sequence; and carrying out image fusion on the to-be-supplemented image sequence and the supplemented image sequence to obtain a target image. According to the method, the missing illumination information of the image is generated and supplemented through offline dynamic region judgment and illumination curve model construction, and the missing illumination is fused to realize the illumination redistribution of the image, so that an image with wider coverage radiance is obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer image technology, and in particular, to an image synthesis method and system. Background Art

[0002] The clarity of an image not only affects the visual experience but also is directly related to information transmission, analysis accuracy, and detail perception. In various image processing and analysis applications, a blurred image can lead to information loss or misunderstanding, thus affecting the accuracy of decision-making. Therefore, clarity is crucial. However, due to the limitations of existing hardware, it is difficult for an image under a single exposure to fully capture the entire illuminance range of the real scene, especially in high-brightness or low-light environments.

[0003] To solve this problem, multiple exposure techniques are often used, that is, by taking multiple images with different exposures to separately capture the dark and bright details in the scene. Then, through an image fusion algorithm, the information of these images is synthesized into a high-quality high-definition image with complete illuminance information, thereby improving the dynamic range and detail presentation effect of the image, especially suitable for image processing tasks under complex lighting conditions.

[0004] However, due to the hardware limitations of the imaging device, it is difficult to take a sufficient number of exposure images within a specific time. Therefore, it is difficult to obtain an image covering the entire brightness range of the scene and difficult to synthesize a high-definition image with complete illuminance information. Summary of the Invention

[0005] The present invention provides an image synthesis method and system to solve the defect in the prior art that it is difficult to synthesize a high-definition image with complete illuminance information through a limited number of exposure images, and to implement an image synthesis method for high-definition images.

[0006] The present invention provides an image synthesis method, including:

[0007] Inputting the image sequence to be supplemented into the optimal exposure sequence function of the luminance channel constructed based on the least squares method, and taking the number of generated images and the exposure time of each generated image corresponding to the pixel values of the generated image sequence covering the scene luminance range of the luminance channel and having the maximum information entropy as the exposure parameters of the image to be generated;

[0008] Taking the exposure parameters of each of the images to be generated as the input of an image generation model constructed based on an illuminance curve model, obtaining the supplementary images output by the image generation model, and constructing a supplementary image sequence, where the illuminance curve model characterizes the relationship between the exposure time and the pixel value of each pixel point in the image;

[0009] Performing image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain a target image.

[0010] An image synthesis method provided by the present invention, wherein the illuminance curve model includes models for three channels of red, green, and blue, and the step of taking the exposure parameters of each of the images to be generated as the input of an image generation model constructed based on the illuminance curve model to obtain the supplementary images output by the image generation model specifically includes:

[0011] Construct an image generation module for each channel based on the illuminance curve model of each channel, and the image generation model includes image generation modules for all channels;

[0012] Input the exposure parameters of each of the images to be generated into the image generation modules of each channel to obtain the output images of each channel;

[0013] After fusing the output images of all channels of each of the images to be generated, obtain the supplementary image corresponding to each of the images to be generated, and take the supplementary image as the output of the image generation model.

[0014] An image synthesis method provided by the present invention, wherein the step of performing image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain the target image specifically includes:

[0015] Take the image with the lowest brightness in the image sequence to be supplemented as the reference image, and determine the first weight for suppressing the bright area information of the target image based on the reference image;

[0016] Determine the second weight for fusing the saturation and contrast of the target image;

[0017] Construct a comprehensive weight based on the first weight and the second weight, and perform weighted fusion on the image sequence to be supplemented and the supplementary image sequence based on the comprehensive weight.

[0018] Before the step of taking the exposure parameters of each of the images to be generated as the input of an image generation model constructed based on the illuminance curve model according to an image synthesis method provided by the present invention, it further includes:

[0019] Collect image data sets for each period at different times, wherein each image data set for each period includes at least five images obtained in the same manner during the corresponding period, and the exposure duration of each image is different;

[0020] Take any image in each image data set for each period as the current frame image, use the remaining images and the pixels that change in the current frame image as the motion area, and remove the motion area of each image in each image data set for each period to obtain the static area image data set for each period;

[0021] The illuminance curve model corresponding to each time period is fitted on the static area image data set of each time period.

[0022] According to an image synthesis method provided by the present invention, the step of using the pixels that change in the remaining images and the current frame image as the motion area specifically includes:

[0023] Each of the remaining images and the current frame image is sequentially mapped to the intermediate exposure moment of the two images to obtain a set of mapped images corresponding to each of the remaining images;

[0024] Calculate the covariance of each set of mapped images, and determine the pixels with covariance not less than the preset threshold as the motion pixels;

[0025] Integrate all the determined motion pixels to obtain the motion area.

[0026] According to an image synthesis method provided by the present invention, the step of fitting the illuminance curve model corresponding to each time period on the static area image data set of each time period specifically includes:

[0027] Based on polynomial fitting, the illuminance curve model corresponding to each time period is fitted on the static area image data set of each time period.

[0028] The present invention also provides an image synthesis system, including:

[0029] A determination module, configured to input the image sequence to be supplemented into the optimal exposure sequence function of the luminance channel constructed based on the least squares method, and use the number of generated images and the exposure time of each generated image when the luminance of the generated image sequence in the luminance channel covers the scene luminance range of the luminance channel and the information entropy is the largest as the exposure parameters of the image to be generated;

[0030] A generation module, configured to use the exposure parameters of each image to be generated as the input of an image generation model constructed based on the illuminance curve model, obtain the supplementary images output by the image generation model, and construct a supplementary image sequence, where the illuminance curve model represents the relationship between the exposure time and the pixel value of each pixel point in the image;

[0031] A fusion module, configured to perform image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain a target image.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the image synthesis method as described in any one of the above.

[0033] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the image synthesis method described in any one of the above is implemented.

[0034] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the image synthesis method described in any one of the above is implemented.

[0035] The image synthesis method and system provided by the present invention, by constructing an optimal exposure sequence function considering the scene dynamic range and information entropy, solve the exposure parameters of the image to be generated for the existing image sequence to be supplemented, so as to generate a supplementary image corresponding to the exposure parameters, so that the complete image sequence composed of the supplementary image sequence and the image sequence to be supplemented can not only complete the expression of the illuminance information of the scene, but also have as little overlapping of illuminance information as possible. Therefore, by determining appropriate exposure parameters, the generation of redundant images is reduced in the process of generating supplementary images, the efficiency of image synthesis is improved, and the quality of the synthesized image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 is a schematic flowchart of the image synthesis method provided by the present invention;

[0038] Figure 2 is a schematic diagram of the process of generating a supplementary image in the image synthesis method provided by the present invention;

[0039] Figure 3 is a schematic flowchart of image fusion in the image synthesis method provided by the present invention;

[0040] Figure 4 is a schematic flowchart of fitting an illuminance curve model in the image synthesis method provided by the present invention;

[0041] Figure 5 In (a) is the illuminance curve model obtained by fitting under three different exposure ratios in the image synthesis method provided by the present invention;

[0042] Figure 5 In (b) is the image with the lowest brightness in the image sequence to be supplemented in the image synthesis method provided by the present invention;

[0043] Figure 5Among them, (c) is the synthesized image obtained from the illuminance curve model with an exposure ratio of 2 in the image synthesis method provided by the present invention;

[0044] Figure 5 Among them, (d) is the synthesized image obtained from the illuminance curve model with an exposure ratio of 4 in the image synthesis method provided by the present invention;

[0045] Figure 5 Among them, (e) is the synthesized image obtained from the illuminance curve model with an exposure ratio of 8 in the image synthesis method provided by the present invention;

[0046] Figure 6 Among them, (a) is a schematic diagram of three groups of to-be-supplemented image sequences with different exposures input in the to-be-supplemented image sequence in the image synthesis method provided by the present invention;

[0047] Figure 6 Among them, (b) is a schematic diagram of the image directly fused from the input to-be-supplemented image sequence;

[0048] Figure 6 Among them, (c) is a schematic diagram of the image fused using the image synthesis method provided by the present invention:

[0049] Figure 7 Among them, (a) is a schematic diagram of two groups of to-be-supplemented image sequences containing moving regions in the image synthesis method provided by the present invention;

[0050] Figure 7 Among them, (b) is a schematic diagram of the image directly fused from the to-be-supplemented image sequence containing moving regions in the image synthesis method provided by the present invention;

[0051] Figure 7 Among them, (c) is a schematic diagram of the image fused from the to-be-supplemented image sequence containing moving regions after pre-removing the moving regions in the image synthesis method provided by the present invention;

[0052] Figure 8 is a schematic diagram of the structure of the image synthesis system provided by the present invention;

[0053] Figure 9 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed implementation manners

[0054] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0055] The following will introduce the image synthesis method of the present invention in conjunction with Figures 1 to 4 as shown in the following, which includes: Figure 1 Shown as follows:

[0056] Step 101: Input the image sequence to be supplemented into the optimal exposure sequence function of the luminance channel constructed based on the least squares method, and use the number of generated images and the exposure time of each generated image corresponding to when the pixel values of the generated image sequence in the luminance channel cover the scene pixel value range of the luminance channel and the information entropy is the largest as the exposure parameters of the images to be generated;

[0057] The image sequence to be supplemented is several acquired images to be supplemented. Since the dynamic range of the acquired images to be supplemented still cannot fully reflect the dynamic range of the real scene, several supplementary images need to be generated to supplement the missing luminance details.

[0058] On this basis, usually a complete sequence of supplementary images is generated so that the generated image sequence can cover the complete scene dynamic range, and then the generated image sequence and the image sequence to be supplemented are weighted and fused to obtain the target image reflecting the complete illumination range.

[0059] However, this method makes the generated image sequence contain redundant images, and the redundant images will also have an impact during the process of weighted synthesis of the target image from the images. Therefore, in order to generate appropriate supplementary images, so that the supplemented image sequence can cover the complete dynamic range of the scene and reduce the generation of redundant images, an optimal exposure sequence function is constructed based on the least squares method:

[0060]

[0061] In the formula, H(·) represents the least squares function, Δt represents the exposure time, k represents the number of generated images, g(·) represents the pixel value dynamic range, and Q represents the information entropy.

[0062] Specifically, that is, by considering the image sequence with information entropy Q and the constraint of the scene pixel value dynamic range, taking the information entropy and the dynamic range of the scene as the constraint conditions, under the constraint that the pixel value dynamic range of the generated supplementary images and the original image sequence to be supplemented covers the dynamic range of the scene, taking k and Δt when the information entropy is the largest as the obtained exposure parameters, that is, it is obtained that k supplementary images need to be supplemented and the exposure time Δt = {Δt1, Δt2, …, Δt k} corresponding to each supplementary image.

[0063] For a solution process, as shown in Figure 2As shown in the figure, select any image in the image sequence to be supplemented as the current frame, that is, as the comparison benchmark. Preferably, select the image with the lowest brightness in the image sequence to be supplemented as the current frame. On this basis, map the set of dark pixel points r2 in the current frame to R2 of the generated supplementary image, and map the set of bright pixel points R1 of the generated supplementary image to r1 in the current frame. After fusing these two mapping sets, if r1 > r2, it means that there is no overlap in the illumination information between the supplementary image and the current frame; otherwise, continue to iterate to determine new exposure parameters. Through the above method, it is possible to verify and iteratively solve the number of images contained in the supplementary image sequence and the exposure time corresponding to each supplementary image.

[0064] On this basis, it can be considered that by the constraint of information entropy, there is no overlapping information in the generated supplementary image sequence, that is, the generation of redundant images is reduced.

[0065] It should be particularly noted that the optimal exposure sequence function constructed in this embodiment is used to solve for the luminance channel, that is, the current frame image is converted to the YUV space and only the Y luminance channel is operated on, and the obtained exposure parameters are used as the exposure parameters commonly used by the R, G, and B channels to obtain the supplementary image sequences corresponding to each of the three channels.

[0066] Step 102, use the exposure parameter of each of the images to be generated as the input of an image generation model constructed based on an illumination curve model, obtain the supplementary images output by the image generation model, and construct a supplementary image sequence, where the illumination curve model represents the relationship between the exposure time and the pixel value of each pixel point in the image;

[0067] Pre-collect multiple images of any scene, construct an illumination curve model, which represents the relationship between the exposure time and the pixel value of each pixel point in the image. On this basis, determine the exposure time, and then the pixel value of each pixel point in the image corresponding to this exposure time can be determined. Therefore, by constructing an image generation model based on the illumination curve model, it is possible to generate the image corresponding to this exposure time on the basis of clarifying the exposure time.

[0068] It can be understood that the obtained exposure parameters include the number k of images to be generated and the exposure time corresponding to each image to be generated. Correspondingly, the exposure parameter of each image to be generated is the exposure time of this image to be generated.

[0069] On this basis, input the exposure times of k images to be generated into the image generation model in sequence, and then k supplementary images output by the image generation model can be obtained, thus obtaining a supplementary image sequence.

[0070] Step 103, perform image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain a target image.

[0071] After obtaining the supplementary image sequence, integrate it with the image sequence to be supplemented to obtain the complete image sequence of the target image, and then perform image synthesis on all the images in the completed image sequence to obtain the target image.

[0072] At this time, it is considered that the synthesized target image can completely reflect the bright and dark details of the scene, and a high-quality high-definition image with complete illuminance information is obtained, thereby improving the dynamic range and detail presentation effect of the image.

[0073] The present invention constructs an optimal exposure sequence function considering the scene dynamic range and information entropy, solves the exposure parameters of the image to be generated for the existing image sequence to be supplemented, and generates a supplementary image corresponding to the exposure parameters, so that the complete image sequence composed of the supplementary image sequence and the image sequence to be supplemented can not only complete the expression of the illuminance information of the scene, but also have as little overlapping of illuminance information as possible. Therefore, by determining appropriate exposure parameters, the generation of redundant images is reduced in the process of generating supplementary images, the efficiency of image synthesis is improved, and the quality of the synthesized image is improved.

[0074] In the image synthesis method of the present invention, the illuminance curve model includes models of three channels of red, green, and blue. The step of using the exposure parameter of each image to be generated as the input of an image generation model constructed based on the illuminance curve model to obtain the supplementary image output by the image generation model specifically includes:

[0075] Construct an image generation module for each channel based on the illuminance curve model of each channel, and the image generation model includes image generation modules of all channels;

[0076] Input the exposure parameter of each image to be generated into the image generation module of each channel to obtain the output image of each channel;

[0077] After fusing the output images of all channels of each image to be generated, obtain the supplementary image corresponding to each image to be generated, and use the supplementary image as the output of the image generation model.

[0078] It can be understood that for a color image, it has information of three channels of red (R), green (G), and blue. Therefore, an illuminance curve model for each channel is pre-constructed.

[0079] At the same time, corresponding to the illuminance curve model of each channel, an image generation module for each channel is constructed, and the image generation modules of the three channels are used as a complete image generation model.

[0080] Among them, the image generation module can be expressed as:

[0081]

[0082] In the formula, represents the m-th supplementary image of the n-th frame image, and g -1 (·) is the transpose of the illuminance curve model, and I n (p) represents the image with the lowest brightness in the image sequence to be supplemented, and Δt n is the exposure time of the image I n (p), is the exposure time of the image .

[0083] It can be understood that by substituting the illuminance curve model corresponding to each channel into the above formula, the output image of each channel can be generated based on the above formula.

[0084] For the exposure parameters of a group of images to be generated, after inputting them into the image generation model, the image generation model takes their exposure times as the inputs of three image generation modules respectively, obtains the output images of the R, G, and B channels output by the image generation modules, and after fusing the output images of the three channels, the supplementary image corresponding to this group of exposure parameters is obtained.

[0085] By taking the supplementary image as the output of the image generation model, it is possible to output the corresponding supplementary image based on the exposure parameters of each of the input images to be generated.

[0086] In the image synthesis method of the present invention, the step of performing image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain the target image specifically includes:

[0087] Taking the image with the lowest brightness in the image sequence to be supplemented as the reference image, and determining the first weight for suppressing the bright area information of the target image based on the reference image;

[0088] During the process of image synthesis, in order to prevent image overexposure, in this embodiment, the image with the lowest brightness in the image sequence to be supplemented is used as the reference image, and the first weight used in the fusion is determined based on the pixel points of the reference image, so as to assign higher weights to the pixels with larger pixel values in each image to be fused.

[0089] The first weight w1(p) is as follows:

[0090]

[0091] In the formula, a and b are constants, p is the value of the pixel point, and h3(p) is an increasing function to suppress the bright area information in the image after supplementary illumination and avoid overexposure.

[0092] Determining the second weight for fusing the saturation and contrast of the target image;

[0093] Further, in this embodiment, the second weight w2(p) is also obtained using weights such as saturation, contrast, and good exposure as a guiding fusion.

[0094] Based on the first weight and the second weight, a comprehensive weight is constructed, and based on the comprehensive weight, weighted fusion is performed on the image sequence to be supplemented and the supplementary image sequence.

[0095] On this basis, let Y(p) be the luminance component of the image. Therefore, the comprehensive weight of each frame of the image used for fusing to obtain the target image can be expressed as:

[0096] w(I i,n (p)) = w1(Y(p))w2(I i,n (p));

[0097] That is to say, for each frame of the image used for fusing to obtain the target image, its first weight for adjusting the fusion luminance is determined based on the reference image, and its second weight for adjusting saturation, contrast, and good exposure is determined based on itself, and finally the comprehensive weight of each image is obtained.

[0098] Further, in order to obtain a high-definition fused image, in this embodiment, gradient-domain guided image filtering is used to fuse the images. As Figure 3 shown, the weights of each layer of the pyramid are smoothed. At the same time, in order to reduce noise, when guiding the image, the luminance component of the input image is used. The formula for the l-th layer of the fused image is as follows:

[0099]

[0100] In the formula, c is the number of all images used for fusing to obtain the target image, GG is gradient-guided filtering, Y i l is the luminance component of the i-th image, and w i l is the weight of the i-th image. Optionally, if there are moving regions in the images to be fused, the moving regions in the fused images can be removed in advance before performing image fusion.

[0101] In the image synthesis method of the present invention, before the step of using the exposure parameter of each of the to-be-generated images as the input of an image generation model constructed based on an illuminance curve model, the following steps are further included:

[0102] Image data sets of each period are collected at different periods. Among them, each image data set of each period includes at least five images obtained in the same manner at the corresponding period, and the exposure duration of each image is different;

[0103] Taking into account that the dynamic range of scene brightness is different at different time periods, when constructing the illumination curve model, data is collected in different time periods to construct image data sets at different time periods.

[0104] Optionally, a fixed time period may be set, for example, a set of images may be collected every hour or every two hours to construct an image data set for fitting the illumination curve model.

[0105] Optionally, a fixed time point may be selected, for example, to collect a set of images at a certain time point in the early morning, morning, noon, evening, or night to construct an image data set.

[0106] Specifically, for the image data set of each time period, it at least includes five images with different exposure times taken of the same scene at the same position, at the same angle, and using the same device during the time period, that is, the images in the image data set of each time period differ only in exposure time and shooting time nodes.

[0107] In addition, in the image dataset of each time period, the distribution of exposure time of all images must cover the information from the darkest to the brightest part of the image brightness, so that a group of images in the image dataset can fully reflect the dynamic information of the scene, and the exposure time between the previous and next two frames of images needs to be proportional.

[0108] Taking any image in the image data set of each time period as the current frame image, taking the pixels that have changed in the remaining images and the current frame image as the motion area, removing the motion area of each image in the image data set of each time period, and obtaining the static area image data set of each time period;

[0109] During the image fusion process, on the one hand, the brightness information corresponding to the exposure time will affect the quality of the fused image; on the other hand, if the object in the image shooting scene moves or the shooting equipment is shaken, it will also cause the pixels between the exposed images to be misaligned, resulting in image overlap or misalignment. The mismatch of pixels will introduce a "ghosting" effect, which will manifest as some virtual images or blurred boundaries in the fused image, seriously affecting the image clarity and realism.

[0110] Therefore, before constructing the illumination curve model, these motion areas need to be removed so that the fitted illumination curve model can characterize the relationship between the exposure time and the image pixel value.

[0111] Specifically, any image in the image data set of each time period is taken as the current frame image, that is, the pixels in other images that move relative to the current frame image are the pixels in the motion area.

[0112] For each time period image dataset, the current frame image is compared with each of the remaining images in turn to determine the pixels of each image that move relative to the current frame image, and all moving pixels are integrated to obtain the moving area. Furthermore, the pixels in the moving area are removed from each image to obtain image data containing only the static area, thus obtaining a static area image dataset.

[0113] The illumination curve model of the corresponding time period is obtained by fitting the static area image dataset in each time period.

[0114] In this embodiment, the fitting is performed on the static area image data set in each time period, such as Figure 4 As shown, taking the static area image data set including five images as an example, the illumination mapping function curve is obtained by adding a smooth constraint to minimize the following formula:

[0115]

[0116] Where N is the number of static pixels in a single image, I i,n,j is the jth pixel of the nth frame image in the i-th time period, E j represents the radiation value of the jth pixel; Δt n is the exposure time of the nth frame image, B is the mask used to remove the moving area, σ is a constant, I i,n,jmin is the minimum pixel value in the nth frame image in the i-th time period, I i,n,jmax is the maximum pixel value in the nth frame image in the i-th time period.

[0117] By fitting the discrete pixel points obtained by minimizing the above equation, the illumination curve model can be obtained.

[0118] It should be noted that the illumination curve model includes a model for each color channel. For each image in the static area image dataset, its corresponding color channel image is extracted, and the illumination curve model of the corresponding channel can be solved and fitted using the above formula.

[0119] In general, through the above method, the illumination curve model corresponding to each time period can be obtained, and the illumination curve model of each time period includes illumination curve models corresponding to three channels respectively. On this basis, the image generation model of each channel is determined based on the illumination curve model of each channel in each time period, and the image generation model of the corresponding time period can be obtained. When generating the image sequence to be supplemented, it can be input into the image generation model most similar to the time period to realize the generation of supplementary images. For example, if the image sequence to be supplemented is taken at noon, it is input into the image generation model corresponding to the noon time period to obtain the supplementary image sequence.

[0120] In the image synthesis method of the present invention, the step of using the remaining images and the pixels that have changed in the current frame image as the motion area specifically includes:

[0121] Sequentially map each of the remaining images and the current frame image to the intermediate exposure time of the two images to obtain a set of mapped images corresponding to each of the remaining images;

[0122] In order to more accurately determine the motion area, in this embodiment, for each image in the image dataset except the current frame image, map its brightness to the transition image at the intermediate exposure time of the two images respectively, so as to reduce the influence of brightness on the motion area:

[0123]

[0124] In the formula, σ is an empirical constant, and the value of σ is different when the ratio of the exposure times of two consecutive frames of images is different. I i,n (p) represents the current mapped image, I i,n-1 (p) represents the transition image obtained by forward mapping, I i,n+1 (p) represents the transition image obtained by backward mapping; p represents the pixel value, i represents the time period, and n represents the number of frames.

[0125] For example, among the five images in the image dataset, arranged in ascending order of exposure time as P1 to P5, taking P3 as the current frame image, when mapping P2 and P3, map P3 forward to obtain I i,n-1 (p), map P2 backward to obtain I i,n+1 (p), as a set of mapped images; when mapping P3 and P4, map P4 forward to obtain I i,n-1 (p), map P3 backward to obtain I i,n+1 (p), as another set of mapped images.

[0126] Taking five images as an example, four groups of a total of eight mapped images can be obtained.

[0127] Among them, when the ratio of the exposure times of two adjacent frames is 2, the value of σ for two adjacent frames is 0.15.

[0128] Calculate the covariance of each group of mapped images, and determine the pixels with covariance not less than the preset threshold as the motion pixels;

[0129] Integrate all the determined motion pixels to obtain the motion area.

[0130] On this basis, for each group of mapped images, calculate the covariance of the two images. If the covariance is not less than the preset threshold T S , then set this pixel point to 1, otherwise, keep it unchanged, so as to construct a mask B for screening motion pixels:

[0131]

[0132] Wherein, σ is the covariance of two groups of images, and β is a constant used to adjust the confidence constant.

[0133] It should be noted that the above process is also calculated channel by channel. That is, during the calculation, the pixel values of the corresponding channel are used as p to calculate the moving pixels of the corresponding channel.

[0134] It should be noted that the mask B can not only screen moving pixels on the generated illuminance curve, but also screen moving pixels in the fused image.

[0135] By the above method, the moving pixels screened by integrating all the mapped image groups are used to determine the moving area of each image data set in each channel. Thus, based on the moving area of each channel, the static area of each channel is determined, and the illuminance curve model of each channel is fitted.

[0136] In the image synthesis method of the present invention, the step of fitting the illuminance curve model corresponding to each time period on the static area image data set of each time period specifically includes:

[0137] Based on the method of polynomial fitting, the illuminance curve model corresponding to each time period is fitted on the static area image data set of each time period.

[0138] For the sub-data set of each channel of the static area image data set, the pixel points determined by the above method are discrete points. Therefore, the method of polynomial fitting is used to fit the discrete points into a continuous illuminance curve model g(·):

[0139] g(I i,n,j ) = a0 + a1I i,n,j + a2I i,n,j 2 + a3I i,n,j 3 + a4I i,n,j 4 ;

[0140] Wherein, a0 to a4 are constants, and their values are correspondingly adjusted for different shooting devices. In this embodiment, 5 groups of fittings are made and the average values of a0 to a4 are taken.

[0141] In the present invention, by pre-removing the moving area, an illuminance curve model that can accurately reflect the relationship between the image and the exposure time is fitted.

[0142] In a specific embodiment, taking the G channel as an example, the illuminance curve model of the G channel constructed by the above method is as Figure 5 shown. Specifically, Figure 5The red line in [Figure] is the illuminance curve model corresponding to an exposure ratio of 2 between two adjacent frames of images, the blue line is the illuminance curve model corresponding to an exposure ratio of 4 between two adjacent frames of images, and the green line is the illuminance curve model corresponding to an exposure ratio of 8 between two adjacent frames of images.

[0143] On this basis, Figure 5 In [Figure] (b) is the image with the lowest brightness in the image sequence to be supplemented. Based on this, when the exposure ratios are 2, 4, and 8, the synthetic images generated by the image generation model constructed using the corresponding illuminance curve models are respectively as shown in Figure 5 In [Figure] (c), (d), and (e).

[0144] In another specific embodiment, as shown in Figure 6 In [Figure] (a), three groups of images with different exposures are input, and the image sequence to be supplemented is generated using this embodiment, and image fusion is performed. The synthetic image obtained is as shown in Figure 6 In [Figure] (c), and the directly fused image is as shown in Figure 6 In [Figure] (b).

[0145] It should be noted that if multiple images are input, after arranging the exposure times from small to large, the above formula is applied pairwise between adjacent two images to calculate the exposure parameters to be supplemented, and then the image generation model is substituted to generate the supplementary images.

[0146] In another specific embodiment, as shown in Figure 7 In [Figure] (a), if the input images contain a moving area, before image fusion through this embodiment, the two images are projected to the intermediate moment in advance, and the mask B is determined based on the image at the intermediate moment. The result of image synthesis after screening out the moving area is as shown in Figure 7 In [Figure] (c), and the directly synthesized result is as shown in Figure 7 In [Figure] (b).

[0147] Furthermore, as shown in Table 1 below:

[0148] Table 1

[0149]

[0150] As shown in Table 1 above, on 7 different data sets, taking MEF-SSIM (Multi-Exposure Image Fusion Structural Similarity Index) as the evaluation index, the closer its value is to 1, the better the image quality. Comparing the method provided by this embodiment with direct fusion, it can be seen that the fusion effect of this embodiment is better than direct fusion.

[0151] Next, the image synthesis system provided by the present invention will be described. The image synthesis system described below can be correspondingly referred to the image synthesis method described above.

[0152] As Figure 8 shown, the image synthesis system of the present invention includes a determination module 801, a generation module 802, and a fusion module 803;

[0153] The determination module 801 is configured to input the image sequence to be supplemented into the optimal exposure sequence function of the luminance channel constructed based on the least squares method, and when the pixel values of the generated image sequence in the luminance channel cover the scene pixel value range of the luminance channel and the information entropy is the largest, the corresponding number of generated images and the exposure time of each generated image are used as the exposure parameters of the image to be generated;

[0154] The image sequence to be supplemented is several images to be supplemented that have been acquired. Since the dynamic range of the acquired images to be supplemented still cannot fully reflect the dynamic range of the real scene, several supplementary images need to be generated to supplement the missing luminance details.

[0155] On this basis, a complete sequence of supplementary images is usually generated so that the generated image sequence can cover the complete scene dynamic range, and then the generated image sequence and the image sequence to be supplemented are weighted and fused to obtain the target image reflecting the complete illumination range.

[0156] However, this method makes the generated image sequence contain redundant images, and the redundant images also have an impact during the process of weighted synthesis of the images into the target image. Therefore, in order to generate appropriate supplementary images so that the supplemented image sequence can cover the complete dynamic range of the scene and reduce the generation of redundant images, an optimal exposure sequence function is constructed based on the least squares method:

[0157]

[0158] In the formula, H(·) represents the least squares function, Δt represents the exposure time, k represents the number of generated images, g(·) represents the pixel value dynamic range, and Q represents the information entropy.

[0159] Specifically, that is, by considering the image sequence with information entropy Q and the constraint of the scene pixel value dynamic range, taking the information entropy and the dynamic range of the scene as constraint conditions, under the constraint that the pixel value dynamic range of the generated supplementary image and the original image sequence to be supplemented covers the dynamic range of the scene, k and Δt when the information entropy is the largest are used as the solved exposure parameters, that is, it is obtained that k supplementary images need to be supplemented and the exposure time Δt = {Δt1, Δt2,..., Δt k}

[0160] For a solution process, such as Figure 2As shown, any image in the image sequence to be supplemented is selected as the current frame image, that is, as the comparison benchmark. Preferably, the image with the lowest brightness in the image sequence to be supplemented is used as the current frame. On this basis, the set of dark pixel points r2 in the current frame is mapped to R2 of the generated supplementary image, and the set of bright pixel points R1 of the generated supplementary image is mapped to r1 in the current frame. After fusing these two mapping sets, if r1 > r2, it indicates that there is no overlap in illumination information between the supplementary image and the current frame. Through the above method, the number of images contained in the supplementary image sequence and the exposure time corresponding to each supplementary image can be tested and solved iteratively.

[0161] On this basis, it can be considered that through the constraint of information entropy, there is no overlapping information in the generated supplementary image sequence, that is, the generation of redundant images is reduced.

[0162] It should be particularly noted that the optimal exposure sequence function constructed in this embodiment is used to solve in the luminance channel, that is, the current frame image is converted to the YUV space and only the Y luminance channel is operated, and the obtained exposure parameters are used as the exposure parameters commonly used by the R, G, and B channels to obtain the supplementary image sequences corresponding to each of the three channels.

[0163] The generation module 802 is configured to use the exposure parameter of each of the images to be generated as the input of an image generation model constructed based on an illumination curve model, obtain the supplementary image output by the image generation model, and construct a supplementary image sequence, where the illumination curve model represents the relationship between the exposure time and the pixel value of each pixel point in the image;

[0164] Pre-collect multiple images of any scene, construct an illumination curve model, where the illumination curve model represents the relationship between the exposure time and the pixel value of each pixel point in the image. On this basis, by determining the exposure time, the pixel value of each pixel point in the image corresponding to this exposure time can be determined. Therefore, by constructing an image generation model based on the illumination curve model, an image corresponding to this exposure time can be generated on the basis of clarifying the exposure time.

[0165] It can be understood that the obtained exposure parameters include the number k of images to be generated and the exposure time corresponding to each image to be generated. Correspondingly, the exposure parameter of each image to be generated is the exposure time of this image to be generated.

[0166] On this basis, by sequentially inputting the exposure times of k images to be generated into the image generation model, k supplementary images output by the image generation model can be obtained, and a supplementary image sequence can be obtained.

[0167] The fusion module 803 is configured to perform image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain a target image.

[0168] After obtaining the supplementary image sequence, it is integrated with the image sequence to be supplemented to obtain the complete image sequence of the target image, and then all the images in the completed image sequence are synthesized to obtain the target image.

[0169] At this time, it is considered that the synthesized target image can completely reflect the bright and dark details of the scene, and a high-quality high-definition image with complete illuminance information is obtained, thereby improving the dynamic range and detail presentation effect of the image.

[0170] The present invention constructs an optimal exposure sequence function considering the scene dynamic range and information entropy, solves the exposure parameters of the image to be generated from the existing image sequence to be supplemented, and generates a supplementary image corresponding to the exposure parameters, so that the complete image sequence composed of the supplementary image sequence and the image sequence to be supplemented can not only complete the expression of the illuminance information of the scene, but also have as little overlapping of illuminance information as possible, thereby reducing the generation of redundant images in the process of generating the supplementary image by determining appropriate exposure parameters, improving the efficiency of image synthesis and the quality of the synthesized image.

[0171] Figure 9 An example of the physical structure diagram of an electronic device is shown in Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940. Among them, the processor 910, the communication interface 920, and the memory 930 complete communication with each other through the communication bus 940. The processor 910 can call the logical instructions in the memory 930 to execute the image synthesis method, which includes: inputting the image sequence to be supplemented into the optimal exposure sequence function of the luminance channel constructed based on the least squares method, and using the number of generated images and the exposure time of each generated image corresponding to the maximum information entropy when the pixel values of the generated image sequence in the luminance channel cover the scene luminance range of the luminance channel as the exposure parameters of the image to be generated; using the exposure parameters of each of the images to be generated as the input of an image generation model constructed based on an illuminance curve model to obtain the supplementary images output by the image generation model, and constructing a supplementary image sequence, where the illuminance curve model represents the relationship between the exposure time and the pixel value of each pixel point in the image; performing image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain the target image.

[0172] In addition, when the logical instructions in the above-mentioned memory 930 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0173] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image synthesis method provided by the above-mentioned various methods. The method includes: inputting an image sequence to be supplemented into an optimal exposure sequence function of a luminance channel constructed based on the least squares method, and using the number of generated images and the exposure time of each generated image corresponding to the pixel values of the generated image sequence covering the scene luminance range of the luminance channel and having the maximum information entropy in the luminance channel as the exposure parameters of the image to be generated; using the exposure parameters of each image to be generated as the input of an image generation model constructed based on an illuminance curve model, obtaining a supplementary image output by the image generation model, and constructing a supplementary image sequence, where the illuminance curve model represents the relationship between the exposure time and the pixel value of each pixel point in the image; and performing image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain a target image.

[0174] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements an image synthesis method provided by the above-mentioned various methods. The method includes: inputting an image sequence to be supplemented into an optimal exposure sequence function of a luminance channel constructed based on the least squares method, and taking the number of generated images corresponding to the pixel values of the generated image sequence covering the scene luminance range of the luminance channel and having the maximum information entropy and the exposure time of each generated image as the exposure parameters of the image to be generated; taking the exposure parameters of each of the images to be generated as the input of an image generation model constructed based on an illuminance curve model, obtaining a supplementary image output by the image generation model, and constructing a supplementary image sequence, where the illuminance curve model characterizes the relationship between the exposure time and the pixel value of each pixel point in the image; and performing image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain a target image.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0176] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0177] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image synthesis method, characterized in that, Including: Input the image sequence to be supplemented into the optimal exposure sequence function of the luminance channel constructed based on the least squares method, and use the number of generated images and the exposure time of each generated image corresponding to the pixel values of the generated image sequence covering the scene pixel value range of the luminance channel and having the maximum information entropy in the luminance channel as the exposure parameters of the image to be generated; Use the exposure parameters of each image to be generated as the input of the image generation model constructed based on the illuminance curve model, obtain the supplementary image output by the image generation model, and construct a supplementary image sequence, where the illuminance curve model characterizes the relationship between the exposure time and the pixel value of each pixel point in the image; Perform image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain the target image.

2. The image synthesis method according to claim 1, wherein The illuminance curve model includes models for the red, green, and blue channels. The step of using the exposure parameters of each image to be generated as the input of the image generation model constructed based on the illuminance curve model and obtaining the supplementary image output by the image generation model specifically includes: Construct an image generation module for each channel based on the illuminance curve model of each channel, and the image generation model includes image generation modules for all channels; Input the exposure parameters of each image to be generated into the image generation modules of each channel to obtain the output images of each channel; After fusing the output images of all channels of each image to be generated, obtain the supplementary image corresponding to each image to be generated, and use the supplementary image as the output of the image generation model.

3. The image synthesis method according to claim 1, wherein The step of performing image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain the target image specifically includes: Use the image with the lowest luminance in the image sequence to be supplemented as the reference image, and determine the first weight for suppressing the information in the bright area of the target image based on the reference image; Determine the second weight for fusing the saturation and contrast of the target image; Construct a comprehensive weight based on the first weight and the second weight, and perform weighted fusion on the image sequence to be supplemented and the supplementary image sequence based on the comprehensive weight.

4. The image synthesis method according to claim 1, wherein Before the step of using the exposure parameters of each image to be generated as the input of the image generation model constructed based on the illuminance curve model, it further includes: Collect the image data sets for each period at different times, where each image data set for each period includes at least five images obtained in the same manner at the corresponding period and the exposure duration of each image is different; Use any image in the image data set for each period as the current frame image, and use the pixels that change between the remaining images and the current frame image as the moving area, and remove the moving area of each image in the image data set for each period to obtain the stationary area image data set for each period; Fit the illuminance curve model for the corresponding period on the stationary area image data set for each period.

5. The image synthesis method according to claim 4, wherein The step of using the pixels that change between the remaining images and the current frame image as the moving area specifically includes: Map each of the remaining images to the intermediate exposure time of the two images together with the current frame image in sequence, to obtain a set of mapped images corresponding to each of the remaining images; Calculate the covariance of each set of mapped images, and determine the pixels with covariance not less than the preset threshold as moving pixels; Integrate all the determined moving pixels to obtain the moving region.

6. The image synthesis method according to claim 4, wherein The step of fitting the illuminance curve model corresponding to each period on the static region image dataset of each period specifically includes: Based on the method of polynomial fitting, fit the illuminance curve model corresponding to each period on the static region image dataset of each period.

7. An image synthesis system, characterized in that, Include: A determination module, configured to input the image sequence to be supplemented into the optimal exposure sequence function of the luminance channel constructed based on the least squares method, and use the number of generated images and the exposure time of each generated image corresponding to the pixel values of the generated image sequence covering the scene luminance range of the luminance channel and having the maximum information entropy as the exposure parameters of the images to be generated; A generation module, configured to use the exposure parameters of each of the images to be generated as the input of an image generation model constructed based on the illuminance curve model, obtain the supplementary images output by the image generation model, and construct a supplementary image sequence, where the illuminance curve model represents the relationship between the exposure time and the pixel value of each pixel point in the image; A fusion module, configured to perform image fusion on the image sequence to be supplemented and the supplementary image sequence to obtain a target image.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image synthesis method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image synthesis method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image synthesis method according to any one of claims 1 to 6.

Images