Image generation method, device, electronic device, and computer-readable storage medium

Abstract

The present application relates to an image generation method, device, electronic device, and storage medium. The method includes: determining a reference frame from multiple image frames obtained by capturing the same scene; performing block matching on the reference frame and each fused frame other than the reference frame to determine the first block in each fused frame corresponding to the reference block in the reference frame; performing noise estimation on the reference frame and each fused frame to obtain a noise estimation result, and adjusting the first block in each fused frame based on the noise estimation result to obtain a corresponding second block; and merging the reference block in the reference frame with the corresponding second block in each fused frame to generate a target image. This method can remove ghosting in an image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of computer technology, image signal processing technology has emerged. Electronic devices input images into the image signal processing (ISP) process, which processes the images to obtain the images required by users.

[0003] However, in traditional image generation methods, multiple processed images are fused, and the fused image may have a ghosting problem. Summary of the Invention

[0004] The embodiments of the present application provide an image generation method, device, electronic device, and computer-readable storage medium, which can remove ghost images in images.

[0005] A method for generating an image, comprising:

[0006] Determining a reference frame from a plurality of image frames captured of the same scene;

[0007] Performing block matching on the reference frame and each fused frame except the reference frame, respectively, to determine the first block in each fused frame corresponding to the reference block in the reference frame;

[0008] performing noise estimation on the reference frame and each of the fused frames to obtain a noise estimation result, and adjusting the first block in each of the fused frames based on the noise estimation result to obtain a corresponding second block;

[0009] The reference block in the reference frame and the corresponding second block in each of the fused frames are merged to generate a target image.

[0010] An image generating device, comprising:

[0011] A reference frame determination module, configured to determine a reference frame from a plurality of image frames captured for the same scene;

[0012] a block matching module, configured to perform block matching on the reference frame and each fused frame other than the reference frame, and determine a first block in each fused frame corresponding to a reference block in the reference frame;

[0013] a noise estimation module, configured to perform noise estimation on the reference frame and each of the fused frames to obtain a noise estimation result, and adjust the first block in each of the fused frames based on the noise estimation result to obtain a corresponding second block;

[0014] The block merging module is used to merge the reference block in the reference frame and the corresponding second block in each of the fused frames to generate a target image.

[0015] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned image generation method.

[0016] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0017] The above-mentioned image generation method, device, electronic device, and computer-readable storage medium determine a reference frame from multiple image frames obtained by shooting the same scene; perform block matching on the reference frame and each fused frame other than the reference frame to determine the first block in each fused frame corresponding to the reference block in the reference frame; then, perform noise estimation on the reference frame and each fused frame to obtain a noise estimation result. Then, based on the noise estimation result, the first block in each fused frame can be adjusted to obtain a more accurate second block, so that the reference block in the reference frame and the corresponding second block in each fused frame can be more accurately merged to eliminate ghost images in the image and generate a target image with higher clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 is a flowchart of an image generating method in one embodiment;

[0020] Figure 2 is an image frame including a texture in one embodiment;

[0021] Figure 3 is a schematic diagram of calculating the sharpness of an image frame in one embodiment;

[0022] Figure 4 A flowchart of a step in one embodiment of determining a reference frame from a plurality of image frames obtained by capturing the same scene;

[0023] Figure 5 is a schematic diagram of an image pyramid in one embodiment;

[0024] Figure 6is a schematic diagram of block matching in one embodiment;

[0025] Figure 7 1 is a schematic diagram of a noise estimation process in one embodiment;

[0026] Figure 8 is a schematic diagram of block merging in one embodiment;

[0027] Figure 9 A schematic diagram of a process for generating an image in one embodiment;

[0028] Figure 10 A schematic diagram of a process for generating an image in one embodiment;

[0029] Figure 11 is a structural block diagram of an image generating device in one embodiment;

[0030] Figure 12 FIG. 1 is a schematic diagram of the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0032] It will be understood that the terms "first," "second," and the like used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are used solely to distinguish a first element from another element. For example, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client, without departing from the scope of this application. The first client and the second client are both clients, but they are not the same client.

[0033] Figure 1 The following is a flowchart of an image generation method in one embodiment. This embodiment describes the image generation method using an electronic device as an example. The electronic device may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices.

[0034] like Figure 1 As shown, the method for processing image data includes steps 102 to 108 .

[0035] Step 102: Determine a reference frame from a plurality of image frames captured for the same scene.

[0036] The reference frame is the image frame used for calibration. The image format of the reference frame is not limited and can be RGB (Red, Green, Blue), YUV, or HSV (Hue, Saturation, Value). In a YUV image frame, the Y component represents luminance (Luminance), while the U and V components represent chrominance (Chroma).

[0037] Specifically, the electronic device calls the camera and controls the camera to shoot the same scene to obtain multiple image frames, and then determines the reference frame from the multiple image frames. Each image frame contains texture information, such as Figure 2 The image frames shown contain textures, and a reference frame can be determined based on each image frame containing textures. For example, the image frame with the clearest texture information among the image frames is used as the reference frame.

[0038] Furthermore, the electronic device calls the camera, obtains preset shooting parameters, and controls the camera to shoot the same scene with the preset shooting parameters to obtain multiple image frames. The shooting parameters include exposure, aperture factor, or shutter speed.

[0039] In one embodiment, the electronic device may randomly select an image frame from a plurality of image frames as the reference frame. In another embodiment, the electronic device may select the image frame with the highest definition from the plurality of image frames as the reference frame. In other embodiments, the electronic device may also select the first image frame acquired as the reference frame. The method for determining the reference frame from the plurality of image frames is not limited and may be configured according to user needs.

[0040] The electronic device further calculates the sharpness and brightness of each image frame and determines a reference frame based on the sharpness and brightness of each image. The electronic device uses an 8x8 grayscale step size to average the image frames to determine the brightness of the image frame. The electronic device also calculates the x and y gradients of the image frame to determine the sharpness of the image frame.

[0041] Figure 3 This is a schematic diagram of calculating the sharpness of an image frame in one embodiment. The electronic device calculates the sharpness by taking the gradient change of three pixels above, below, and to the left and right of a point in the image frame. The image point is determined in steps of 8 in each direction.

[0042] Step 104 : performing block matching on the reference frame and each fused frame except the reference frame, and determining the first block in each fused frame corresponding to the reference block in the reference frame.

[0043] The fused frame is the image frame other than the reference frame in the plurality of image frames. The reference block is the region in the reference frame. The first block is the region in the fused frame corresponding to the reference block.

[0044] The reference block may be a region of interest (ROI) in the reference frame. Specifically, the electronic device performs ROI detection on the reference frame to determine the ROI as the reference block. Block motion matching is performed on the reference frame and each fused frame other than the reference frame to determine the ROI in each fused frame corresponding to the reference block in the reference frame as the first block.

[0045] Step 106 : performing noise estimation on the reference frame and each fused frame to obtain a noise estimation result, and adjusting the first block in each fused frame based on the noise estimation result to obtain a corresponding second block.

[0046] The noise estimation result may specifically be a noise curve corresponding to the reference frame. The second block is a block obtained by adjusting the first block in the fused frame.

[0047] Specifically, the electronic device calls a preset noise estimation function, inputs the reference frame and each fused frame into the preset noise estimation function, performs noise estimation on the reference frame and each fused frame to obtain a noise estimation result, and adjusts the first block in each fused frame based on the noise estimation result to obtain a corresponding second block.

[0048] Furthermore, before performing noise estimation on the reference frame and each fused frame, the electronic device may also perform preprocessing to obtain the photosensitivity (ISO) and set the maximum and minimum values ​​of the noise corresponding to the sensitivity; obtain the result between the maximum and minimum values ​​from the noise estimation result as a new noise estimation result, and adjust the first block in each fused frame based on the new noise estimation result to obtain the corresponding second block.

[0049] Step 108 : Merge the reference block in the reference frame and the corresponding second block in each fused frame to generate a target image.

[0050] The target image is the merged image. The target image can be in any format, such as jpeg or png.

[0051] Specifically, the electronic device invokes an HDR (High Dynamic Range) synthesis algorithm to combine the reference block in the reference frame with the corresponding second block in each fused frame to generate a high dynamic range image. This high dynamic range image is the target image, and the electronic device can display it.

[0052] The above-mentioned image generation method determines a reference frame from multiple image frames obtained by shooting the same scene; performs block matching on the reference frame and each fused frame other than the reference frame, and determines the first block in each fused frame corresponding to the reference block in the reference frame; then, performs noise estimation on the reference frame and each fused frame to obtain a noise estimation result. Then, based on the noise estimation result, the first block in each fused frame can be adjusted to obtain a more accurate second block, so that the reference block in the reference frame and the corresponding second block in each fused frame can be more accurately merged to eliminate ghost images in the image and generate a target image with higher clarity.

[0053] In another embodiment, the reference block in the reference frame and the corresponding second block in each fused frame are block-merged, i.e., frequency domain fusion and window interpolation fusion. First, the reference frame and each fused frame are fused in the frequency domain, and then the fused image is interpolated to obtain the target image of the original size.

[0054] In another embodiment, the electronic device merges the reference block in the reference frame and the corresponding second block in each fused frame to obtain an intermediate image, and then performs Laplace blending on the intermediate image to obtain a target image.

[0055] In one embodiment, Figure 4 As shown, determining a reference frame from multiple image frames obtained by shooting the same scene includes:

[0056] Step 402 : photographing the same scene with a first exposure duration to obtain a first type of image frame, and photographing the same scene with a second exposure duration to obtain a second type of image frame; the first exposure duration is greater than the second exposure duration.

[0057] The first exposure time and the second exposure time can be set as needed, and the first exposure time is longer than the second exposure time. For example, the first exposure time is 10 milliseconds (ms) and the second exposure time is 5 milliseconds (ms).

[0058] The first type of image frames are obtained by shooting with a first exposure time and are image frames with a longer exposure time. The second type of image frames are obtained by shooting with a second exposure time and are image frames with a shorter exposure time. It is understood that if the first exposure time is greater than the second exposure time, the image frames in the first type are bright frames, and the image frames in the second type are dark frames.

[0059] Step 404 : Determine a first target image frame from the first category of image frames, and determine a second target image frame from the second category of image frames.

[0060] The first target image frame is an image frame determined from the first type of image frames and used to determine the reference frame. The second target image frame is an image frame determined from the second type of image frames and used to determine the reference frame.

[0061] Specifically, the electronic device determines the clarity of each image frame in the first category of image frames, and determines the image frame with the highest clarity in the first category of image frames as the first target image frame; determines the clarity of each image frame in the second category of image frames, and determines the image frame with the highest clarity in the second category of image frames as the second target image frame.

[0062] In another embodiment, the electronic device may randomly determine the first target image frame from the first category of image frames and randomly determine the second target image frame from the second category of image frames. In another embodiment, the electronic device may use the first image frame acquired first in the first category of image frames as the first target image frame and the first image frame acquired first in the second category of image frames as the second target image frame. The specific method for determining the first target image frame and the second target image frame is not limited and can be set as needed.

[0063] Step 406 : Determine a reference frame from the first target image frame and the second target image frame.

[0064] In one embodiment, the electronic device may randomly determine the reference frame from the first target image frame and the second target image frame.

[0065] In another embodiment, the electronic device may directly designate one of the first and second target image frames as the reference frame. For example, the electronic device may directly use the second target image frame as the reference frame. It is understood that dark frames typically contain relatively complete image information and are not affected by the loss of overexposure information in highlight areas. Therefore, using the second target image frame as the reference frame can more accurately generate the target image.

[0066] In another embodiment, the electronic device may determine an image frame with higher definition from the first target image frame and the second target image frame as the reference frame.

[0067] The specific method of determining the reference frame can be set as needed and is not limited here.

[0068] In this embodiment, the same scene is photographed with a first exposure time and a second exposure time to obtain a first type of image frame and a second type of image frame, respectively; the first exposure time is greater than the second exposure time, and then a first target image frame and a second target image frame are determined from the first type of image frame and the second type of image frame, respectively. Then, the reference frame can be determined more accurately based on the first target image frame and the second target image frame.

[0069] In one embodiment, the same scene is photographed with a first exposure time to obtain a first category of image frames, and the same scene is photographed with a second exposure time to obtain a second category of image frames, including: photographing the same scene with the first exposure time to obtain multiple first alternative image frames, and classifying the first alternative image frames with image brightness greater than a first brightness threshold as the first category of image frames; photographing the same scene with the second exposure time to obtain multiple second alternative image frames, and classifying the second alternative image frames with image brightness less than a second brightness threshold as the second category of image frames; the first brightness threshold is greater than or equal to the second brightness threshold.

[0070] The first candidate image frame is an image frame obtained by shooting with a first exposure time and is used as a candidate. The second candidate image frame is an image frame obtained by shooting with a second exposure time and is used as a candidate.

[0071] Both the first brightness threshold and the second brightness threshold can be set as needed, and the first brightness threshold is greater than or equal to the second brightness threshold. Therefore, the first candidate image frame whose image brightness is greater than the first brightness threshold is a bright frame and belongs to the first category of image frames, and the second candidate image frame whose image brightness is less than the second brightness threshold is a dark frame and belongs to the second category of image frames.

[0072] For each image frame, brightness matching is first performed to determine an initial first-category image frame and an initial second-category image frame.

[0073] In this embodiment, the first alternative image frame with an image brightness greater than the first brightness threshold is classified as the first category of image frames, and the second alternative image frame with an image brightness less than the second brightness threshold is classified as the second category of image frames. The first alternative image frame with a brightness less than or equal to the first brightness threshold can be removed from the first alternative image frame, and the second alternative image frame with a brightness greater than or equal to the second brightness threshold can be removed from the second alternative image frame. The first category of image frames and the second category of image frames can be obtained more accurately.

[0074] In one embodiment, after determining the first target image frame from the first category of image frames and determining the second target image frame from the second category of image frames, it also includes: determining the sharpness ratio between the first target image frame and the second target image frame; determining the image frames that do not meet the preset sharpness conditions based on the sharpness of each image frame other than the first target image frame and the second target image frame, and the sharpness ratio; eliminating the image frames that do not meet the preset sharpness conditions, and performing the steps of block matching the base frame with each fused frame other than the base frame based on the image frames after elimination.

[0075] The sharpness ratio refers to the ratio between the sharpness of the first target image frame and the sharpness of the second target image frame. In one embodiment, the electronic device may divide the sharpness of the first target image frame by the sharpness of the second target image frame to obtain the sharpness ratio. In another embodiment, the electronic device may divide the sharpness of the second target image frame by the sharpness of the first target image frame to obtain the sharpness ratio.

[0076] The sharpness ratio can be calculated according to the following formula:

[0077]

[0078] Among them, ratio is the sharpness ratio, brightness i and brightness j are the sharpness of the first target image frame and the sharpness of the second target image frame respectively.

[0079] The preset sharpness condition can be set as needed. For example, the preset sharpness condition can be that the ratio between the sharpness of the image frame and the sharpness ratio is less than a preset ratio threshold, or that the difference between the sharpness of the image frame and the sharpness ratio is not within a threshold range, without limitation thereto.

[0080] Furthermore, the electronic device may also sort the image frames after elimination according to the size of sharpness, which may be sorted from large to small or from small to large.

[0081] It is understandable that, unlike equal exposure fusion which only needs to consider exposure frames of the same brightness, in this embodiment, the EV and exposure time ratio of long and short exposure are not fixed, and there will be differences in different scenes. Therefore, the fusionability of long and short exposure must also be considered. Therefore, it is necessary to calculate the clearest image frames that do not meet the preset sharpness conditions, eliminate inappropriate frames, and thus generate the target image more accurately.

[0082] In one embodiment, after determining a reference frame from a plurality of image frames obtained by shooting the same scene, the method further includes: mapping the brightness of each fused frame except the reference frame to the reference frame, and determining the local motion vectors between each fused frame relative to the reference frame; and based on the local motion vectors between each fused frame relative to the reference frame, executing the step of performing block matching between the reference frame and each fused frame except the reference frame.

[0083] A local motion vector is a vector that describes the position of a block in an image frame relative to the position in a reference frame.

[0084] The electronic device maps the brightness of each fused frame except the reference frame to the reference frame, and can determine the local motion vector between each fused frame relative to the reference frame; based on the local motion vector between each fused frame relative to the reference frame, the reference frame and each fused frame except the reference frame are block matched respectively, and the first block in each fused frame corresponding to the reference block in the reference frame can be determined more accurately.

[0085] Furthermore, the electronic device captures the same scene with a first exposure time to obtain a first type of image frame, and captures the same scene with a second exposure time to obtain a second type of image frame; the first exposure time is greater than the second exposure time; that is, the first type of image frame is a bright frame, and the second type of image frame is a dark frame. When the reference frame is the second target image frame in the second type of image frame, each fused frame is the first type of image frame and the other image frames in the second type of image frame except the reference frame. By mapping the brightness of the first type of image frame to the reference frame, and mapping the brightness of the second type of image frame to the reference frame, local motion vectors between each fused frame and the reference frame can be obtained. Then, based on the local motion vectors corresponding to each fused frame, the first block corresponding to the reference block in the reference frame can be more accurately found.

[0086] Furthermore, in order to obtain a bright frame fused image and a dark frame fused image, the electronic device can also map the brightness of the first type of image frames to the second type of image frames, and map the brightness of the second type of image frames to the first type of image frames.

[0087] In one embodiment, block matching is performed on a reference frame and each fused frame other than the reference frame to determine the first block in each fused frame corresponding to the reference block in the reference frame. The method includes: downsampling the reference frame to obtain an image pyramid with at least two layers; traversing the image pyramid from the bottom layer to the top layer, block matching is performed on each layer image and each fused frame other than the reference frame, and obtaining the first block in each fused frame corresponding to the reference block in the top layer image based on block alignment.

[0088] For a sample value sequence, a sample is taken once every several sample values, so that the new sequence obtained is a downsample of the original sequence. The electronic device downsamples the reference frame to obtain an image pyramid of at least two layers. In one embodiment, Figure 5 As shown, the electronic device downsamples the reference frame to obtain a four-layer image pyramid. In other embodiments, the electronic device may further decompose the image pyramid to obtain a new image pyramid, such as a steerable pyramid.

[0089] Specifically, the electronic device traverses the image pyramid from the bottom layer to the top layer, using multithreading to perform block matching on each layer image with each fused frame other than the reference frame. Based on block alignment, the electronic device obtains the first block in each fused frame corresponding to the reference block in the top layer image. The traversal step size for the bottom layer image is 1 pixel, the search range is 24*24 pixels, and the reference block size is 8*8 pixels. When the image pyramid includes four layers of images, the search radius for the four layers, from the bottom layer to the top layer, is 8, 8, 8, and 2, respectively, and the search range is 24, 24, 24, and 12, respectively.

[0090] In this embodiment, the reference frame is downsampled to obtain an image pyramid of at least two layers. The image pyramid is traversed from the bottom layer to the top layer, and block matching is performed on each layer image with each fused frame other than the reference frame. Based on block alignment, the first block in each fused frame corresponding to the reference block in the top layer image is obtained. This allows for more accurate block matching and, therefore, more accurate determination of the first block corresponding to the reference block.

[0091] Figure 6 Figure 1 is a schematic diagram of block matching in one embodiment. The electronic device begins with the pixel coordinates in the upper left corner and traverses to the pixel coordinates in the lower right corner. To select the optimal block coordinates, the sum of the squared pixel differences of the block is calculated, while the distance from the center point is calculated, all the way to the top-level image. Similarly, the optimal block in the top-level image of the fused frame is selected as the first block in the fused frame.

[0092] In one embodiment, noise estimation is performed on the reference frame and each fused frame to obtain a noise estimation result, including: determining a target root mean square and a target absolute difference value corresponding to each pixel interval of the reference frame based on the reference frame and each fused frame; and constructing a noise curve corresponding to the reference frame based on the target root mean square and the target absolute difference value corresponding to each pixel interval.

[0093] A pixel interval is an interval including a plurality of adjacent pixels. For example, if the pixel value of the reference frame is between 0 and 255 (the image is 8 bits), the pixel value can be divided into 16 equal parts to obtain 16 pixel intervals.

[0094] In data statistical analysis, the root mean square (RMS) is obtained by summing the squares of all values, finding the mean, and then taking the square root. The target RMS is the RMS value used to construct the noise curve for each pixel interval in the reference frame. The target absolute difference is the absolute difference between the noise curves constructed by the magatama for each pixel interval in the reference frame.

[0095] In this embodiment, the electronic device obtains the target root mean square and target difference absolute value corresponding to each pixel interval, and can accurately construct a noise curve corresponding to the reference frame.

[0096] Figure 7 It is a flowchart of noise estimation in one embodiment. The electronic device performs preprocessing, obtains the sensitivity (ISO, photosensibility), and sets the maximum and minimum values ​​of the noise corresponding to the sensitivity. Calculate the target root mean square (RMS, Root mean square), the target difference absolute value sum DIFF (SAD, Sum of absolute differences of the reference frame and the fusion frame) or Inrtensity and other data corresponding to each pixel interval of the reference frame. The electronic device then calculates the intermediate variable, finds the middle value of Diff for each interval, and eliminates the intervals that do not meet the conditions. Among them, the conditions can be set as needed. Among them, the starting point (Search start) is the starting point of the first search range, the end point (Search end) is the starting point of the first search range, and the block size (search window) refers to the block size of the search comparison.

[0097] For each reference block in the reference frame, the fusion frame will find the block in the image that is most similar to the reference block as the first block for fusion.

[0098] In one embodiment, based on the reference frame and each fused frame, a target root mean square and a target difference absolute value corresponding to each pixel interval of the reference frame are determined, including: traversing each pixel of the reference frame, determining the initial root mean square corresponding to each pixel, and determining the pixel interval corresponding to the initial root mean square of each pixel; traversing each pixel of the reference frame, calculating the initial difference absolute value between each pixel in the reference frame and the corresponding pixel of each fused frame, and determining the pixel interval corresponding to the initial root mean square of the same pixel in the reference frame as the pixel interval corresponding to the initial difference absolute value; based on the initial root mean square and initial difference absolute value corresponding to each pixel interval, determining the target root mean square and target difference absolute value of the corresponding pixel interval.

[0099] The initial RMS is the RMS of each pixel in the pixel interval. The initial absolute difference is the absolute difference of each pixel in the pixel interval.

[0100] Specifically, the electronic device traverses each pixel of the reference frame, and for each pixel, calculates the initial root mean square of the pixel and the corresponding pixel in each fused frame, and uses the pixel interval where the value of the initial root mean square is located as the pixel interval corresponding to the initial root mean square of the pixel; calculates the initial mean difference absolute value of the pixel and the corresponding pixel in each fused frame, and determines the pixel interval corresponding to the initial root mean square of the same pixel in the reference frame as the pixel interval corresponding to the initial difference absolute value.

[0101] For example, the electronic device divides the pixels of the reference frame into 16 equal parts, resulting in 16 pixel intervals: 0-15, 16-31, and so on. For each pixel in the reference frame, the electronic device calculates the initial root mean square (RMS) value between the pixel and the corresponding pixel in each fused frame, as well as the initial absolute value of the difference between the pixel and the corresponding pixel in each fused frame. If the fused frame consists of N frames, then N initial RMS values ​​and N initial absolute value differences can be obtained for each pixel.

[0102] For example, if the initial RMS value for a pixel is 23, and the pixel interval for this initial RMS value 23 is 16-31, then the pixel interval corresponding to the initial RMS value 23 for this pixel is 16-31. Simultaneously, the initial absolute value difference between the same pixel in the reference frame and the same pixel in the fused frame is calculated, and the pixel interval corresponding to this initial absolute value difference is also determined to be 16-31, which is the same as the pixel interval corresponding to the initial RMS value 23.

[0103] Each pixel interval has a certain number of initial root mean square and initial absolute value differences, which can be further screened to remove the initial absolute value differences that are greater than a preset threshold.

[0104] In one embodiment, for each pixel interval, the electronic device may calculate the average of the initial RMS values ​​corresponding to the pixel interval as the target RMS value, and calculate the average of the initial absolute value differences corresponding to the pixel interval as the target absolute value difference value.

[0105] In another embodiment, for each pixel interval, the electronic device may determine the median of the initial RMS values ​​corresponding to the pixel interval as the target RMS value, and determine the median of the initial absolute value differences corresponding to the pixel interval as the target absolute value difference.

[0106] It should be noted that the method of determining the target root mean square from each initial root mean square corresponding to the pixel interval, and the method of determining the target difference absolute value from each initial difference absolute value corresponding to the pixel interval, can be set as needed and are not limited here.

[0107] Furthermore, if there is a pixel interval in the reference frame that has no corresponding initial root mean square and initial absolute value difference, the pixel interval is ignored, which can improve the speed of image processing.

[0108] In one embodiment, based on the initial root mean square and initial difference absolute value corresponding to each pixel interval, the target root mean square and target difference absolute value of the corresponding pixel interval are determined, including: for each pixel interval, averaging the initial root mean squares corresponding to the pixel interval, and taking the average value as the target root mean square; determining the median from the initial difference absolute values ​​corresponding to the pixel interval, and taking the median as the target difference absolute value.

[0109] For each pixel interval, there are corresponding initial RMS and initial absolute value differences. The initial RMS corresponding to the pixel interval are averaged and the average value is used as the target RMS. The median is determined from the initial absolute value of the difference corresponding to the pixel interval and the median is used as the target absolute value of the difference. Then, the target RMS and target absolute value can be obtained for each pixel interval, so that the noise curve corresponding to the reference frame can be accurately constructed.

[0110] In one embodiment, adjusting the first block in each fused frame based on the noise estimation result to obtain the corresponding second block includes: obtaining the sum of the absolute values ​​of the differences corresponding to each pixel interval in the reference frame; determining the intensity weight of each fused frame based on the noise estimation result and the sum of the absolute values ​​of the differences; and adjusting the first block in the corresponding fused frame based on each intensity weight to obtain the second block in each fused frame.

[0111] The electronic device obtains the target difference absolute value corresponding to each pixel interval in the reference frame, and adds each target difference absolute value to obtain the sum of the difference absolute values.

[0112] If the noise estimation result is a noise curve, the electronic device inputs the fused frame into the noise estimation result to obtain a noise value for the fused frame. Based on the noise value and the sum of the difference absolute addresses, the electronic device can determine an intensity weight for the fused frame. The intensity weight refers to the weight value adjusted for the first block in the fused frame. The specific manner in which the electronic device determines the fused frame based on the noise value and the sum of the difference absolute addresses can be configured as needed and is not limited herein.

[0113] In this embodiment, the sum of the absolute values ​​of the differences corresponding to each pixel interval in the reference frame is obtained, and the intensity weight of each fused frame is determined based on the noise estimation result and the sum of the absolute values ​​of the differences. Then, the first block in the corresponding fused frame is adjusted based on each intensity weight, so that the second block in each fused frame can be obtained more accurately.

[0114] Figure 8This is a schematic diagram of block merging in one embodiment. The electronic device sets parameters to create variables, obtains the Teansform (ACTransform) transfer parameters, and directly controls the fusion ratio (the larger the value, the more fusion). The electronic device calculates the Y channel RMS (Root Mean Square) of the reference frame and performs a DCT (Discrete Cosine Transform) conversion. The electronic device performs bilinear interpolation on the fused frame to obtain the pixel values ​​corresponding to the decimal coordinates and performs a DCT (Discrete Cosine Transform) conversion. The electronic device fuses the reference frame and each fused frame. Specifically, the electronic device performs a weighted average of the reference block in the reference frame and the corresponding second block in each fused frame to obtain the target image. The intensity weight is determined by the noise estimate and the interpolation in the frequency domain. The reference block and the second block are both small blocks with an 8×8 resolution. The electronic device performs an inverse DCT transform on the target image to generate a large image with a resolution of 8192*6144, which can eliminate ghosting in the large image.

[0115] Figure 9 The figure is a flow chart of image generation in one embodiment. The electronic device obtains a first type of image frame and a second type of image frame, and searches for the best frame from the first type of image frame and the second type of image frame. The first type of image frame is an image frame obtained by photographing the same scene with a first exposure time, and the second type of image frame is an image frame obtained by photographing the same scene with a second exposure time. The first exposure time is greater than the second exposure time, that is, the first type of image frame is a bright frame, and the second type of image frame is a dark frame. The electronic device can search for the image frame with the highest clarity from the first type of image frames as the best frame of the first type of image frames, search for the image frame with the highest clarity from the second type of image frames as the best frame of the second type of image frames, and use the best frame of the first type of image frames as the reference frame.

[0116] The electronic device performs brightness mapping on the reference frame and each fused frame except the reference frame, that is, mapping the brightness of each fused frame except the reference frame to the reference frame, determining the local motion vector between each fused frame relative to the reference frame, and based on the local motion vector between each fused frame relative to the reference frame, performing block matching on the reference frame and each fused frame respectively.

[0117] The electronic device performs noise estimation on the reference frame and each fused frame to obtain a noise estimation result, and adjusts the first block in each fused frame based on the noise estimation result to obtain a corresponding second block; performs block merging on the reference block in the reference frame and the corresponding second block in each fused frame; and performs Laplace blending on the block-merged image to generate a high dynamic range image.

[0118] Figure 10Schematic diagram of image generation in one embodiment. The electronic device obtains 5 image frames in a backlit scene, namely image frame #1, image frame #2, image frame #3, image frame #4 and image frame #5. Image frame #1 and image frame #3 are bright frames (EV0), and image frame #2, image frame #4 and image frame #5 are dark frames (EV-). The above-mentioned image generation method is used to produce a high dynamic range image with the dark frame as the reference frame. It can be seen that the ghosting performance of the high dynamic range image is good in the area where the hand is waving, and the transparency of the entire high dynamic range image is better.

[0119] It should be understood that although Figure 1 、 Figure 4 、 Figures 7 to 9 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 、 Figure 4 、 Figures 7 to 9 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0120] Figure 11 FIG. 1 is a structural block diagram of an image generating device according to an embodiment of the present invention. Figure 11 As shown, an image generation device is provided, including: a reference frame determination module 1102, a block matching module 1104, a noise estimation module 1106 and a block merging module 1108, wherein:

[0121] The reference frame determining module 1102 is configured to determine a reference frame from a plurality of image frames captured for the same scene.

[0122] The block matching module 1104 is configured to perform block matching on the reference frame and each fused frame except the reference frame, and determine the first block in each fused frame corresponding to the reference block in the reference frame.

[0123] The noise estimation module 1106 is configured to perform noise estimation on the reference frame and each fused frame to obtain a noise estimation result, and adjust the first block in each fused frame based on the noise estimation result to obtain a corresponding second block.

[0124] The block merging module 1108 is configured to merge the reference block in the reference frame with the corresponding second blocks in each fused frame to generate a target image.

[0125] The above-mentioned image generation device determines a reference frame from multiple image frames obtained by shooting the same scene; performs block matching on the reference frame and each fused frame other than the reference frame, and determines the first block in each fused frame corresponding to the reference block in the reference frame; then, performs noise estimation on the reference frame and each fused frame to obtain a noise estimation result. Then, based on the noise estimation result, the first block in each fused frame can be adjusted to obtain a more accurate second block, so that the reference block in the reference frame and the corresponding second block in each fused frame can be more accurately merged to eliminate ghost images in the image and generate a target image with higher clarity.

[0126] In one embodiment, the above-mentioned reference frame determination module 1102 is also used to shoot the same scene with a first exposure time to obtain a first type of image frame, and shoot the same scene with a second exposure time to obtain a second type of image frame; the first exposure time is greater than the second exposure time; determine the first target image frame from the first type of image frame, and determine the second target image frame from the second type of image frame; determine the reference frame from the first target image frame and the second target image frame.

[0127] In one embodiment, the above-mentioned reference frame determination module 1102 is also used to shoot the same scene with a first exposure time to obtain multiple first alternative image frames, and classify the first alternative image frames whose image brightness is greater than a first brightness threshold into the first category of image frames; shoot the same scene with a second exposure time to obtain multiple second alternative image frames, and classify the second alternative image frames whose image brightness is less than the second brightness threshold into the second category of image frames; the first brightness threshold is greater than or equal to the second brightness threshold.

[0128] In one embodiment, the above-mentioned reference frame determination module 1102 is also used to determine the sharpness ratio between the first target image frame and the second target image frame; based on the sharpness of each image frame other than the first target image frame and the second target image frame, and the sharpness ratio, determine the image frames that do not meet the preset sharpness conditions; eliminate the image frames that do not meet the preset sharpness conditions, and the above-mentioned block matching module 1104 is also used to perform block matching on the reference frame and each fused frame other than the reference frame based on each image frame after elimination.

[0129] In one embodiment, the above-mentioned device also includes a brightness mapping module, which is used to map the brightness of each fused frame except the reference frame to the reference frame, and determine the local motion vector between each fused frame relative to the reference frame; the above-mentioned block matching module 1104 is also used to perform block matching between the reference frame and each fused frame except the reference frame based on the local motion vector between each fused frame relative to the reference frame.

[0130] In one embodiment, the block matching module 1104 is further configured to downsample the reference frame to obtain an image pyramid of at least two layers; traverse the image pyramid from the bottom layer to the highest layer, perform block matching on each layer image with each fused frame other than the reference frame, and obtain the first block in each fused frame corresponding to the reference block in the highest layer image based on block alignment.

[0131] In one embodiment, the noise estimation module 1106 is further used to determine the target root mean square and target difference absolute value corresponding to each pixel interval of the reference frame based on the reference frame and each fused frame; and construct a noise curve corresponding to the reference frame based on the target root mean square and target difference absolute value corresponding to each pixel interval.

[0132] In one embodiment, the noise estimation module 1106 is further configured to traverse each pixel of the reference frame, determine the initial root mean square corresponding to each pixel, and determine the pixel interval corresponding to the initial root mean square of each pixel; traverse each pixel of the reference frame, calculate the initial absolute value of the difference between each pixel in the reference frame and the corresponding pixel of each fused frame, and determine the pixel interval corresponding to the initial root mean square of the same pixel in the reference frame as the pixel interval corresponding to the initial absolute value of the difference; and determine the target root mean square and target absolute value of the difference for the corresponding pixel interval based on the initial root mean square and initial absolute value of the difference corresponding to each pixel interval.

[0133] In one embodiment, the noise estimation module 1106 is further configured to average the initial RMS values ​​corresponding to each pixel interval, and use the average value as the target RMS value; determine the median value from the initial absolute difference values ​​corresponding to the pixel interval, and use the median value as the target absolute difference value.

[0134] In one embodiment, the noise estimation module 1106 is further used to obtain the sum of the absolute values ​​of the differences corresponding to each pixel interval in the reference frame; determine the intensity weight of each fused frame based on the noise estimation result and the sum of the absolute values ​​of the differences; and adjust the first block in the corresponding fused frame based on each intensity weight to obtain the second block in each fused frame.

[0135] The division of the modules in the above-mentioned image generating device is only for illustration. In other embodiments, the image generating device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned image generating device.

[0136] The specific definition of the image generation device can be found in the definition of the image generation method above and will not be repeated here. Each module in the above-mentioned image generation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0137] Figure 12 Schematic diagram of the internal structure of an electronic device in one embodiment. The electronic device can be any terminal device such as a mobile phone, tablet computer, laptop computer, desktop computer, PDA (Personal Digital Assistant), POS (Point of Sales), vehicle-mounted computer, wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. The processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image generation method provided in each of the following embodiments. The internal memory provides a cached operating environment for the operating system computer program in the non-volatile storage medium.

[0138] The various modules in the image generation device provided in the embodiments of the present application may be implemented in the form of a computer program. The computer program may be executed on a terminal or server. The program modules comprising the computer program may be stored in a memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiments of the present application are implemented.

[0139] The present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of the image generation method.

[0140] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image generation method.

[0141] As used herein, any reference to memory, storage, database, or other medium may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory), or flash memory. Volatile memory may include RAM (Random Access Memory), which serves as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDRSDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).

[0142] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. An image generation method, characterized in that: include: Determining a reference frame from a plurality of image frames captured of the same scene; Performing block matching on the reference frame and each fused frame except the reference frame, respectively, to determine the first block in each fused frame corresponding to the reference block in the reference frame; Traversing each pixel of the reference frame, calculating the initial root mean square of each pixel and the corresponding pixel in each fused frame, and determining the pixel interval in which the value of the initial root mean square lies as the pixel interval corresponding to the initial root mean square of the pixel; traversing each pixel of the reference frame, calculating the initial absolute value of the difference between each pixel in the reference frame and the corresponding pixel in each fused frame, and determining the pixel interval corresponding to the initial root mean square of the same pixel in the reference frame as the pixel interval corresponding to the initial absolute value of the difference; Determining a target root mean square and a target absolute difference value for each pixel interval based on an initial root mean square and an initial absolute difference value corresponding to each pixel interval; constructing a noise curve corresponding to the reference frame based on the target root mean square and the target absolute difference value corresponding to each pixel interval; Adjusting the first block in each of the fused frames based on the noise curve to obtain a corresponding second block; The reference block in the reference frame and the corresponding second block in each of the fused frames are merged to generate a target image.

2. The method according to claim 1, characterized in that Determining a reference frame from a plurality of image frames obtained by shooting the same scene includes: The same scene is photographed with a first exposure time to obtain a first type of image frame, and the same scene is photographed with a second exposure time to obtain a second type of image frame; the first exposure time is greater than the second exposure time; Determine a first target image frame from the first category of image frames, and determine a second target image frame from the second category of image frames; A reference frame is determined from the first target image frame and the second target image frame.

3. The method according to claim 2, characterized in that The method of photographing the same scene with a first exposure time to obtain a first type of image frame and photographing the same scene with a second exposure time to obtain a second type of image frame includes: Shooting the same scene with a first exposure duration to obtain a plurality of first candidate image frames, and classifying the first candidate image frames whose image brightness is greater than a first brightness threshold into a first category of image frames; The same scene is photographed with a second exposure time to obtain multiple second candidate image frames, and the second candidate image frames whose image brightness is less than a second brightness threshold are classified as second category image frames; the first brightness threshold is greater than or equal to the second brightness threshold.

4. The method according to claim 2, characterized in that After determining the first target image frame from the first category of image frames and determining the second target image frame from the second category of image frames, the method further includes: determining a sharpness ratio between the first target image frame and the second target image frame; Determining, based on the sharpness of each image frame other than the first target image frame and the second target image frame, and the sharpness ratio, an image frame that does not meet a preset sharpness condition; the preset sharpness condition includes that the ratio between the sharpness of the image frame and the sharpness ratio is less than a preset ratio threshold, or that the difference between the sharpness of the image frame and the sharpness ratio is not within a threshold range; The image frames that do not meet the preset sharpness condition are eliminated, and the step of performing block matching on the reference frame and each fused frame other than the reference frame is performed based on each image frame after the elimination.

5. The method according to claim 1, characterized in that After determining the reference frame from the multiple image frames obtained by shooting the same scene, the method further includes: Mapping the brightness of each fused frame except the reference frame to the reference frame, and determining a local motion vector between each fused frame and the reference frame; The step of performing block matching on the reference frame and each fused frame except the reference frame is performed based on a local motion vector between each fused frame and the reference frame.

6. The method according to claim 1, characterized in that The performing block matching on the reference frame and each fused frame other than the reference frame to determine the first block in each fused frame corresponding to the reference block in the reference frame includes: Downsampling the reference frame to obtain an image pyramid of at least two layers; The image pyramid is traversed from the bottom layer to the top layer, and each layer image is block matched with each fused frame except the reference frame. Based on the block alignment, the first block in each fused frame corresponding to the reference block in the top layer image is obtained.

7. The method according to claim 1, characterized in that Determining a target root mean square and a target absolute difference value of a corresponding pixel interval based on the initial root mean square and the initial absolute difference value corresponding to each pixel interval includes: For each pixel interval, averaging the initial RMSs corresponding to the pixel interval, and taking the average as the target RMS; A median is determined from the initial absolute difference values ​​corresponding to the pixel intervals, and the median is used as the target absolute difference value.

8. The method according to claim 1, characterized in that The adjusting the first block in each of the fused frames based on the noise curve to obtain a corresponding second block includes: Obtaining a target difference absolute value corresponding to each pixel interval in the reference frame, and adding each target difference absolute value to obtain a sum of the difference absolute values; determining an intensity weight of each fused frame based on the noise curve and the sum of the absolute values ​​of the differences; The first block in the corresponding fused frame is adjusted based on each intensity weight to obtain the second block in each fused frame.

9. An image generating device, characterized in that: include: A reference frame determination module, configured to determine a reference frame from a plurality of image frames captured for the same scene; a block matching module, configured to perform block matching on the reference frame and each fused frame other than the reference frame, and determine a first block in each fused frame corresponding to a reference block in the reference frame; a noise estimation module, configured to traverse each pixel of the reference frame, calculate an initial root mean square value of each pixel and a corresponding pixel in each fused frame, and determine the pixel interval in which the value of the initial root mean square lies as the pixel interval corresponding to the initial root mean square value of the pixel; traverse each pixel of the reference frame, calculate an initial absolute value of the difference between each pixel in the reference frame and a corresponding pixel in each fused frame, and determine the pixel interval corresponding to the initial root mean square value of the same pixel in the reference frame as the pixel interval corresponding to the initial absolute value of the difference; Determining a target root mean square and a target absolute difference value for each pixel interval based on an initial root mean square and an initial absolute difference value corresponding to each pixel interval; constructing a noise curve corresponding to the reference frame based on the target root mean square and the target absolute difference value corresponding to each pixel interval; and adjusting the first block in each fused frame based on the noise curve to obtain a corresponding second block. The block merging module is used to merge the reference block in the reference frame and the corresponding second block in each of the fused frames to generate a target image.

10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image generation method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images