Image processing method, device, electronic device and storage medium
By generating a ghost mask and performing image fusion, combined with color transformation processing, the ghosting problem when the moving target has a large movement amplitude or is blocked in high dynamic range images is solved, and a better image fusion effect is achieved.
Patent Information
- Application Number
- CN202011315377.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-11-20
AI Technical Summary
In the prior art, in the high dynamic range image synthesis, when the moving target has a large movement amplitude or is blocked, it is difficult to effectively remove ghosting.
By obtaining two frames of images with different exposure times, a ghost mask is generated, image fusion is performed, and the pending area in the reference frame is determined according to the ghost mask, and the pending area in the reference frame is performed, and color transformation and processing is performed to remove ghosts.
Effectively remove ghosts in the fused image, solve the problem of color abnormalities, and can also remove ghosts well when the movement amplitude of the moving target is large or blocked, improving the image fusion effect.
Smart Images

Figure CN112581415B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art
[0002] Real-world scenes often contain a lot of image information such as brightness, color, and intensity, but the performance of traditional digital image acquisition technology is limited. Digital images cannot accurately present all the information of real scenes, which has led to the emergence of high dynamic range (HDR) image synthesis technology. However, scenes in real life are often dynamic. Once any changes occur in the scene during the shooting process, such as the intrusion of a moving object, blur or double images will appear in the final synthesized image. This phenomenon is called "ghosting". Since most outdoor scenes are dynamic and moving objects are inevitable, it is crucial to study how to eliminate ghosts in synthesized images.
[0003] In the prior art, when solving the "ghost" phenomenon caused by moving targets, an optical flow method based on the gradient domain or an optical flow method based on energy can be used. The optical flow method based on the gradient domain calculates the motion vector of the moving area and performs consistency processing based on the image of the vector relative to the frame, but it is only applicable to scenes with a small motion amplitude. The energy-based optical flow method can perform image alignment and ghost removal at the same time, but the HDR imaging effect is not good when the moving target has a large motion amplitude or the moving target is occluded. It can be seen that the prior art method cannot solve the ghost phenomenon when the moving target has a large motion amplitude or the moving target is occluded when performing HDR synthesis. Summary of the invention
[0004] In view of the above problems, embodiments of the present invention are proposed to provide an image processing method, device, electronic device and storage medium that overcome the above problems or at least partially solve the above problems.
[0005] According to a first aspect of an embodiment of the present invention, there is provided an image processing method, comprising:
[0006] After acquiring at least two frames of images with different exposure times, generating a ghost mask between the at least two frames of images;
[0007] Fusing the at least two frames of images to obtain a fused image;
[0008] Determine a reference frame from the at least two frames of images, and determine the area to be processed of the fused image and the area to be processed in the reference frame respectively according to the ghost mask;
[0009] According to the to-be-processed area of the fused image, color transformation is performed on the to-be-processed area in the reference frame, and the to-be-processed area after color transformation in the reference frame is used as the color transformation area;
[0010] The area to be processed in the fused image is processed according to the color change area.
[0011] According to a second aspect of an embodiment of the present invention, there is provided an image processing apparatus, comprising:
[0012] A motion detection module, configured to generate a ghost mask between at least two frames of images after acquiring at least two frames of images with different exposure times;
[0013] A fusion module, used for fusing the at least two frames of images to obtain a fused image;
[0014] A processing region determination module, used to determine a reference frame from the at least two frames of images, and to determine a region to be processed in the fused image and a region to be processed in the reference frame respectively according to the fused image and the ghost mask;
[0015] A color transformation module, used for performing color transformation on the area to be processed in the reference frame according to the area to be processed in the fused image, and taking the area to be processed after color transformation in the reference frame as the color transformation area;
[0016] The region processing module is used to process the region to be processed in the fused image according to the color change region.
[0017] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the image processing method described in the first aspect is implemented.
[0018] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in the first aspect is implemented.
[0019] The image processing method, device, electronic device and storage medium provided by the embodiments of the present invention generate a ghost mask between at least two frames of images after acquiring at least two frames of images with different exposure times, fuse the at least two frames of images to obtain a fused image, determine a reference frame from the at least two frames of images, and determine the to-be-processed area of the fused image and the to-be-processed area in the reference frame according to the to-be-processed area of the fused image, perform color transformation on the to-be-processed area in the reference frame according to the to-be-processed area of the fused image, and process the to-be-processed area in the fused image according to the color transformation result. Since the to-be-processed area of the fused image is processed after the color transformation is performed on the to-be-processed area of the reference frame, the ghost in the fused image can be effectively removed, and the problem of color anomaly can be solved. Moreover, since the operation is based on the local area, the generalization ability is strong, and the ghost can be well removed even when the moving target has a large motion amplitude or is blocked, thereby improving the effect of the fused image.
[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.
[0022] Figure 1 is a flowchart of the steps of an image processing method provided by an embodiment of the present invention;
[0023] Figure 2 is a schematic diagram of dividing the area to be processed of the fused image and the area to be processed of the reference frame into blocks in an embodiment of the present invention;
[0024] Figure 3 It is a structural block diagram of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0025] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0026] Figure 1is a flowchart of the steps of an image processing method provided by an embodiment of the present invention. The method can be applied to a scene where high dynamic range (HDR) fusion is performed on multiple frames of images to effectively remove ghost images therein, such as Figure 1 As shown, the method may include:
[0027] Step 101 : after acquiring at least two frames of images with different exposure times, generate a ghost mask between the at least two frames of images.
[0028] When shooting a target scene, at least two frames of images for the target scene can be obtained, and the exposure time of the at least two frames of images is different, so as to facilitate HDR fusion processing of the at least two frames of images, so that the exposure brightness of the fused image is appropriate. In order to obtain a better fusion effect, it is necessary to determine the moving target in the at least two frames of images and determine the ghost area. At this time, a motion detection algorithm can be used to perform motion detection on the at least two frames of images to determine the moving target in the at least two frames of images, thereby determining the moving area formed by the moving target, and generating a ghost mask according to the moving area. The motion detection algorithm can be, for example, a background difference method, an inter-frame difference method, or an optical flow method.
[0029] In one embodiment of the present invention, the generation of a ghost mask between the at least two frames of images includes: aligning the at least two frames of images; performing motion detection on the aligned at least two frames of images, and generating a ghost mask between the at least two frames of images based on the motion detection results.
[0030] Before fusing at least two frames of images, the at least two frames of images are aligned first, so as to facilitate fusing and ghost removal based on the aligned at least two frames of images.
[0031] When aligning at least two frames of images, one frame of image can be selected from the at least two frames of image as a reference image, and then the other frames of image are aligned with the reference image. When aligning one frame of image with the reference image, feature points of the frame of image and the reference image are extracted respectively, and the feature point sets of the frame of image and the reference image are matched to obtain the best match, and then the corresponding relationship between the two images is optimized by using affine transformation or perspective transformation, so as to obtain transformation parameters, and then the frame of image is deformed into the same spatial layout as the reference image by using the transformation parameters, so as to obtain the aligned frame of image and the reference image.
[0032] In one embodiment of the present invention, the alignment processing of the at least two frames of images may optionally include: aligning the at least two frames of images by using an enhanced correlation coefficient algorithm.
[0033] When aligning at least two frames of images, an Enhanced Correlation Coefficient (ECC) algorithm is used to process, one of the at least two frames of images is used as a reference image, and the other frame of images and the reference image are input into an enhanced correlation coefficient alignment module to obtain another aligned frame of images and the reference image, and then the other frame of images and the reference attributes are input into the enhanced correlation coefficient alignment module until at least two frames of images are aligned to obtain at least two aligned frames of images. Aligning by the enhanced correlation coefficient algorithm can ensure that the width distortion of contrast and brightness remains unchanged.
[0034] In one embodiment of the present invention, the motion detection is performed on the at least two aligned image frames, and a ghost mask is generated between the at least two image frames based on the motion detection result, which optionally includes: calculating the optical flow field between the at least two aligned image frames by a fast dense optical flow algorithm; and generating a ghost mask between the at least two image frames based on the optical flow field.
[0035] The optical flow fields of two adjacent frames of the at least two aligned images are calculated respectively by a fast dense optical flow algorithm, and the motion area in the two adjacent frames is determined based on the optical flow fields of the two adjacent frames, and a ghost mask between the at least two aligned images is generated according to the motion area in each of the two adjacent frames of the at least two aligned images. Among them, dense optical flow is an image registration method that performs point-by-point matching on images. Unlike sparse optical flow, which only matches a number of feature points on an image, dense optical flow calculates the offset of all points on the image, thereby forming a dense optical flow field.
[0036] By using a fast dense optical flow algorithm to determine the ghost mask between at least two frames of images, a more accurate ghost mask can be generated to further improve the effect of subsequent fused images and ensure high-quality ghost elimination.
[0037] Step 102: Fusing the at least two frames of images to obtain a fused image.
[0038] The three quality indicators of image contrast, saturation and good exposure are used to perform HDR fusion on at least two frames of images to obtain a fused image.
[0039] In one embodiment of the present invention, fusing the at least two frames of images to obtain a fused image may optionally include: fusing the at least two frames of images to obtain a fused image using an exposure fusion algorithm.
[0040] Exposure fusion is a process that combines multiple images to create a single image while retaining only the correctly exposed elements.
[0041] Using the three quality indicators of image contrast, saturation and good exposure, the exposure fusion algorithm is used to obtain each individual pixel, and a suitable weight is assigned to the individual pixel in at least two frames of images to obtain the optimal quality indicator of the individual pixel, and each individual pixel is processed separately to obtain a fused image. Performing HDR fusion on at least two frames of images through the exposure fusion algorithm can make the fused image obtain a more realistic effect.
[0042] Step 103: determining a reference frame from the at least two frames of images, and determining the area to be processed of the fused image and the area to be processed in the reference frame respectively according to the ghost mask.
[0043] One frame of image can be selected from at least two frames of image as a reference frame, and according to the coordinates of the circumscribed rectangular frame of the ghost mask, the area within the rectangular frame can be determined in the fused image as the area to be processed of the fused image. According to the circumscribed rectangular frame of the ghost mask, the area within the circumscribed rectangular frame in the reference frame is determined as the area to be processed of the reference frame.
[0044] In one embodiment of the present invention, the determining of the reference frame from the at least two frames of images includes: determining one of the at least two frames of images as the reference frame according to the ghost mask and the fused image.
[0045] The region to be processed in the fused image is determined according to the ghost mask, and the region to be processed of each of at least two frames of images is determined, and a frame of image whose brightness of the region to be processed is closest to that of the fused image is determined from the at least two frames of images as a reference frame. By determining a frame of image as a reference frame from at least two frames of images using the ghost mask and the fused image, the amount of calculation for subsequent color transformation can be reduced, and the brightness of the region to be processed of the reference frame is close to that of the region to be processed of the fused image.
[0046] In one embodiment of the present invention, the step of determining a reference frame from the at least two frames of images and respectively determining the area to be processed of the fused image and the area to be processed in the reference frame according to the ghost mask comprises: determining, according to the ghost mask, the area in the fused image corresponding to the ghost mask as the area to be processed of the fused image; determining, according to the ghost mask, the areas in the at least two frames of images corresponding to the ghost mask as the area to be processed of the at least two frames of images; respectively determining the brightness difference value between the area to be processed of the at least two frames of images and the area to be processed of the fused image, determining a frame of image whose brightness difference value meets a preset condition as the reference frame, and determining the area to be processed of the frame of image whose brightness difference value meets the preset condition as the area to be processed of the reference frame.
[0047] According to the ghost mask, the coordinates of the circumscribed rectangular frame of the ghost area are determined, and according to the coordinates of the circumscribed rectangular frame, the area in the fused image located within the circumscribed rectangular frame is determined as the area to be processed in the fused image, and the area in the circumscribed rectangular frame in at least two frames of images is determined as the image to be processed of each frame of image, and the brightness difference values of the area to be processed of at least two frames of image and the area to be processed of the fused image are calculated respectively, and a frame of image whose brightness difference value meets the preset condition is determined from the at least two frames of image, and the frame of image is used as a reference frame, and the area to be processed of the frame of image is used as the area to be processed of the reference frame. By using the frame of image whose brightness difference value meets the preset condition as a reference frame, the accuracy of subsequent color transformation can be improved, and thus the effect of fused image can be further improved. Among them, the preset condition can be that the brightness difference value is the smallest, so that when the area to be processed in the reference frame is processed after the color transformation of the area to be processed in the fused image, the brightness of the two areas can be close.
[0048] Step 104 , according to the to-be-processed area of the fused image, color transformation is performed on the to-be-processed area in the reference frame, and the to-be-processed area after color transformation in the reference frame is used as the color transformation area.
[0049] According to the pixel value of each pixel point in the to-be-processed area of the fused image, the color transformation coefficient corresponding to the pixel value of each pixel point in the to-be-processed area of the reference frame is determined to be transformed into the pixel value of the corresponding pixel point in the to-be-processed area of the fused image, and the color of each pixel point in the to-be-processed area of the reference frame is transformed according to the color transformation coefficient, and the to-be-processed area after the color transformation in the reference frame is used as the color transformation area, so that the color of the color transformation area is consistent with the color in the fused image, which can solve the problem of color abnormality after fusion.
[0050] In one embodiment of the present invention, according to the area to be processed of the fused image, color transformation is performed on the area to be processed in the reference frame, and the area to be processed after color transformation in the reference frame is used as the color transformation area, including: according to a preset block method, the area to be processed of the fused image is divided into multiple blocks as multiple fused blocks, and the area to be processed of the reference frame is divided into multiple blocks as multiple reference blocks; for each fused block, determining the corresponding color transformation coefficient when the corresponding reference block is transformed into the fused block; according to the color transformation coefficient of each reference block, color transformation is performed on each reference block to obtain multiple transformed reference blocks; and the multiple transformed reference blocks are merged to obtain the color transformation area.
[0051] The preset block division method is a preset block division method, such as setting the number of divided blocks and the number of blocks per row and per column. For example, the preset block division method can be a uniform block division method, that is, each block includes the same pixels.
[0052] According to a preset block division method, the area to be processed of the fused image is divided into a plurality of blocks, and the obtained plurality of blocks are used as a plurality of fused blocks. Similarly, according to a preset block division method, the area to be processed of the reference frame is divided into a plurality of blocks, and the obtained plurality of blocks are used as a plurality of reference blocks. The area to be processed of the fused image and the area to be processed of the reference frame are divided according to the same preset block division method, so that one fused block corresponds to one reference block.
[0053] For each fused block, the color transformation coefficient corresponding to the pixel value of each pixel point in the reference block corresponding to the fused block is calculated respectively to transform the pixel value of the corresponding pixel point in the fused block, so as to obtain the color transformation coefficient corresponding to the reference block when it is transformed into the fused block. After obtaining the color transformation coefficient of each reference block, the color transformation of the reference block is performed using the color transformation coefficient of the reference block to obtain a transformed reference block. After color transformation of multiple reference blocks, multiple transformed reference blocks are obtained. Multiple transformed reference blocks are merged according to the positional relationship of the multiple reference blocks to obtain a color transformation area. Block processing can improve the processing speed and obtain more accurate color transformation coefficients.
[0054] In one embodiment of the present invention, for each fused block, determining the color transformation coefficient corresponding to the transformation of the corresponding reference block into the fused block includes: traversing multiple fused blocks, determining the current fused block, and taking the reference block corresponding to the current fused block as the current reference block; sampling the current fused block according to a preset sampling method to obtain multiple fused block sampling points, and sampling the current reference block to obtain multiple reference block sampling points; determining the color transformation coefficient corresponding to the transformation of the pixel value of each reference block sampling point into the pixel value of the corresponding fused block sampling point through color transformation; and taking the color transformation coefficient corresponding to each reference block sampling point as the color transformation coefficient corresponding to the transformation of the current reference block into the current fused block.
[0055] Each fused block in the to-be-processed area of the fused image needs to be processed separately, and each fused block can be processed by traversal, and the currently traversed fused block is used as the current fused block. Since each reference block in the to-be-processed area of the reference frame corresponds to each fused block in the fused image one-to-one, the reference block corresponding to the current fused block in the to-be-processed area of the reference frame is used as the current reference block. The position of the current reference block in the to-be-processed area of the reference frame is the same as the position of the current fused block in the to-be-processed area of the fused image.
[0056] In order to improve the calculation speed of the color transformation coefficient, the current fusion block and the current reference block can be sampled. When sampling, the current fusion block and the current reference block are sampled respectively according to a preset sampling method to obtain multiple fusion block sampling points and multiple reference block sampling points. The preset sampling method pre-sets the sampling step size.
[0057] In some embodiments, when performing color transformation, the correspondence between each reference block sampling point and the corresponding fusion block sampling point is as follows:
[0058] y=a*x 2 +b*x+c (1)
[0059] Among them, y represents the pixel value of the fused block sampling point, x is the pixel value of the reference block sampling point corresponding to the fused block sampling point, and a, b, and c are the color transformation coefficients to be calculated.
[0060] According to the above formula (1), the corresponding relationship between each fusion block sampling point in the current fusion block and each reference block sampling point in the current reference block can be determined as follows:
[0061]
[0062] Among them, n is the number of sampling points in the current fusion block and the current reference block, x1~x n is the pixel value of the current reference block sampling point, y1~y n is the pixel value of the current fusion block sampling point, and a, b, c are the color transformation coefficients to be calculated.
[0063] make Then formula (2) is transformed into:
[0064] A·z=d (3)
[0065] Perform QR decomposition on matrix A and we get:
[0066] A=QR (4)
[0067] Substitute formula (4) into formula (3) and solve to obtain the color transformation coefficient corresponding to each sampling point:
[0068] z=R -1 Q T d (5)
[0069] The color transformation coefficient of each pixel in the current reference block is the same as the color transformation coefficient of the sampling point, that is, the color transformation coefficient of each pixel is the above-mentioned a, b, and c, so that the color transformation coefficient corresponding to each reference block sampling point is used as the corresponding color transformation coefficient when the current reference block is transformed into the current fused block.
[0070] By sampling the current fused block and the current reference block to solve the color transformation coefficient, the calculation speed can be improved compared to the overall calculation of the entire current fused block and the current reference block, thereby improving the overall image processing speed.
[0071] After obtaining the color transformation coefficient corresponding to transforming the current reference block into the current fused block, the color transformation coefficient can be used to perform color transformation on the pixel value of each pixel in the current reference block. At this time, the look-up table (Look-Up-Table) technology can be used to perform color transformation on the pixel value of each pixel to obtain the transformed reference block.
[0072] In an optional implementation, two adjacent blocks in the multiple fused blocks have an overlapping area, and two adjacent blocks in the multiple reference blocks have an overlapping area.
[0073] Figure 2 is a schematic diagram of dividing the fused image to-be-processed area and the reference frame to-be-processed area into blocks in an embodiment of the present invention, such as Figure 2 As shown, two adjacent blocks have an overlap area 1, and the center of four adjacent blocks is a block center 2. The four adjacent blocks overlap, and area 3 is a non-overlap area.
[0074] After color transformation is performed on each reference block to obtain multiple transformed reference blocks, when merging the multiple transformed reference blocks, a weighted color transformation can be used for the overlapping area, that is, a weighted sum is performed for each pixel in the overlapping area of the transformed reference blocks in different blocks, and the transformed pixel value of the pixel is used as the transformed pixel value, so that a high-quality fusion result can be obtained. When weighted color transformation is used for the overlapping area, the same weight can be assigned to different blocks. For example, when color transformation is performed on the overlapping area between two adjacent blocks, the weight of a pixel in each block can be 0.5, and when color transformation is performed on the overlapping area between four adjacent blocks, the weight of a pixel in each block can be 0.25.
[0075] Step 105: Process the area to be processed in the fused image according to the color change area.
[0076] Since the fused image is an image fused by at least two frames of images with different exposure times, and the area to be processed includes ghosts, the color transformation area can be an area obtained by performing color transformation on the area to be processed of the reference frame. Therefore, the area to be processed in the fused image can be processed according to the color transformation area to eliminate the ghosts in the area to be processed in the fused image.
[0077] In one embodiment of the present invention, the processing the area to be processed in the fused image according to the color change area includes: replacing the area to be processed in the fused image with the color change area.
[0078] Since the fused image is an image formed by the fusion of at least two frames of images with different exposure times, and the area to be processed includes ghosts, while the reference frame does not include ghosts, the color-transformed area obtained by color-transforming the area to be processed in the reference frame is used to replace the area to be processed in the fused image, which can effectively remove the ghosts in the fused image.
[0079] The image processing method provided in the present embodiment generates a ghost mask between at least two frames of images after acquiring at least two frames of images with different exposure times, performs R fusion on at least two frames of images to obtain a fused image, determines one of the at least two frames of images as a reference frame, and determines the to-be-processed area of the fused image and the to-be-processed area in the reference frame according to the ghost mask, performs color transformation on the to-be-processed area in the reference frame according to the to-be-processed area of the fused image, uses the to-be-processed area after color transformation in the reference frame as the color transformation area, and processes the to-be-processed area in the fused image according to the color transformation area. Since the to-be-processed area of the fused image is processed after the color transformation is performed on the to-be-processed area of the reference frame, the ghost in the fused image can be effectively removed, and the problem of color anomaly can be solved. Moreover, since the operation is based on the local area, the generalization ability is strong, and the ghost can be well removed even when the moving target has a large motion amplitude or is blocked, thereby improving the effect of the fused image.
[0080] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0081] Figure 3 is a structural block diagram of an image processing device provided by an embodiment of the present invention, such as Figure 3 As shown, the image processing device may include:
[0082] The motion detection module 301 is used to generate a ghost mask between at least two frames of images after acquiring at least two frames of images with different exposure times;
[0083] A fusion module 302 is used to fuse the at least two frames of images to obtain a fused image;
[0084] A processing region determination module 303 is used to determine a reference frame from the at least two frames of images, and to determine the region to be processed of the fused image and the region to be processed in the reference frame according to the ghost mask;
[0085] A color transformation module 304 is used to perform color transformation on the area to be processed in the reference frame according to the area to be processed in the fused image, and use the area to be processed after color transformation in the reference frame as the color transformation area;
[0086] The region processing module 305 is used to process the region to be processed in the fused image according to the color transformation region.
[0087] Optionally, the region processing module is specifically used for:
[0088] The region to be processed in the fused image is replaced by the color-changed region.
[0089] Optionally, the processing area determination module includes:
[0090] A reference frame determining unit is used to determine one of the at least two frames of images as the reference frame according to the ghost mask and the fused image.
[0091] Optionally, the processing area determination module includes:
[0092] a fused image processing region determining unit, configured to determine, according to the ghost mask, a region in the fused image corresponding to the ghost mask as a region to be processed of the fused image;
[0093] At least two frames of image processing region determining unit, used for determining, according to the ghost mask, regions in the at least two frames of image corresponding to the ghost mask as regions to be processed of the at least two frames of image;
[0094] A reference frame determination unit is used to respectively determine the brightness difference values between the to-be-processed areas of the at least two frames of images and the to-be-processed area of the fused image, determine a frame of image whose brightness difference value meets a preset condition as the reference frame, and use the to-be-processed area of the frame of image whose brightness difference value meets the preset condition as the to-be-processed area of the reference frame.
[0095] Optionally, the color conversion module includes:
[0096] A blocking unit, configured to divide the area to be processed of the fused image into a plurality of blocks as a plurality of fusion blocks according to a preset blocking method, and to divide the area to be processed of the reference frame into a plurality of blocks as a plurality of reference blocks;
[0097] A color transformation coefficient determination unit, for determining, for each fused block, a color transformation coefficient corresponding to when transforming the corresponding reference block into the fused block;
[0098] A color transformation unit, used for performing color transformation on each reference block according to a color transformation coefficient of each reference block to obtain a plurality of transformed reference blocks;
[0099] The block merging unit is used to merge the multiple transformed reference blocks to obtain the color transformed area.
[0100] Optionally, the color transformation coefficient determination unit is specifically used for:
[0101] Traverse multiple fused blocks, determine the current fused block, and use the reference block corresponding to the current fused block as the current reference block;
[0102] According to a preset sampling method, the current fusion block is sampled to obtain a plurality of fusion block sampling points, and the current reference block is sampled to obtain a plurality of reference block sampling points;
[0103] Determine a color transformation coefficient corresponding to when the pixel value of each reference block sampling point is transformed into the pixel value of the corresponding fusion block sampling point through color transformation;
[0104] The color transformation coefficient corresponding to each reference block sampling point is used as the color transformation coefficient corresponding to the transformation of the current reference block into the current fused block.
[0105] Optionally, two adjacent blocks in the multiple fused blocks have an overlapping area, and two adjacent blocks in the multiple reference blocks have an overlapping area.
[0106] Optionally, the motion detection module includes:
[0107] An alignment processing unit, used for performing alignment processing on the at least two frames of images;
[0108] The motion detection unit is used to perform motion detection on the at least two aligned image frames, and generate a ghost mask between the at least two image frames according to the motion detection result.
[0109] Optionally, the alignment processing unit is specifically used for:
[0110] The at least two frames of images are aligned using an enhanced correlation coefficient algorithm.
[0111] Optionally, the motion detection unit includes:
[0112] An optical flow field calculation subunit, used for calculating the optical flow field between the at least two aligned image frames by using a fast dense optical flow algorithm;
[0113] The ghost mask generating subunit is used to generate a ghost mask between the at least two frames of images according to the optical flow field.
[0114] Optionally, the fusion module is specifically used for:
[0115] The at least two frames of images are fused by an exposure fusion algorithm to obtain a fused image.
[0116] The image processing device provided by the present embodiment generates a ghost mask between at least two frames of images after acquiring at least two frames of images with different exposure times through the motion detection module, the fusion module fuses at least two frames of images to obtain a fused image, the processing area determination module determines a reference frame from at least two frames of images, and determines the to-be-processed area of the fused image and the to-be-processed area in the reference frame respectively according to the ghost mask, the color transformation module performs color transformation on the to-be-processed area in the reference frame according to the to-be-processed area of the fused image, and uses the to-be-processed area after color transformation in the reference frame as the color transformation area, and the area processing module processes the to-be-processed area in the fused image according to the color transformation area. Since the to-be-processed area of the reference frame is color transformed and then the to-be-processed area of the fused image is processed, the ghost in the fused image can be effectively removed, and the problem of color abnormality can be solved. Moreover, since the operation is based on the local area, the generalization ability is strong, and the ghost can be well removed even when the target motion amplitude is large and the fire moving target is blocked, thereby improving the effect of the fused image.
[0117] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0118] Furthermore, according to an embodiment of the present invention, an electronic device is provided, which may be a computer, a mobile terminal, etc. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, and the computer program implements the image processing method of the aforementioned embodiment when executed by the processor.
[0119] According to one embodiment of the present invention, a computer-readable storage medium is also provided, and the computer-readable storage medium includes but is not limited to a disk storage, a CD-ROM, an optical storage, etc., and a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the image processing method of the aforementioned embodiment is implemented.
[0120] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0121] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0122] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.
[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0125] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0126] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0127] The above is a detailed introduction to an image processing method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. An image processing method, characterized in that: include: After acquiring at least two frames of images with different exposure times, generating a ghost mask between the at least two frames of images; Fusing the at least two frames of images to obtain a fused image; Determine a reference frame from the at least two frames of images, and determine the area to be processed of the fused image and the area to be processed in the reference frame respectively according to the ghost mask; According to the to-be-processed area of the fused image, color transformation is performed on the to-be-processed area in the reference frame, and the to-be-processed area after color transformation in the reference frame is used as the color transformation area; The region to be processed in the fused image is replaced by the color-changed region.
2. The method according to claim 1, characterized in that Determining a reference frame from the at least two frames of images comprises: According to the ghost mask and the fused image, one frame of the at least two frames of images is determined as the reference frame.
3. The method according to any one of claims 1 to 2, characterized in that: The step of determining a reference frame from the at least two frames of images and determining the area to be processed of the fused image and the area to be processed in the reference frame respectively according to the ghost mask comprises: According to the ghost mask, determining an area in the fused image corresponding to the ghost mask as an area to be processed of the fused image; According to the ghost mask, respectively determining the regions corresponding to the ghost mask in the at least two frames of images as the to-be-processed regions of the at least two frames of images; The brightness difference values between the to-be-processed areas of the at least two frames of images and the to-be-processed area of the fused image are respectively determined, a frame of image whose brightness difference value satisfies a preset condition is determined as the reference frame, and the to-be-processed area of the frame of image whose brightness difference value satisfies the preset condition is used as the to-be-processed area of the reference frame.
4. The method according to any one of claims 1 to 2, characterized in that: According to the to-be-processed area of the fused image, color transformation is performed on the to-be-processed area in the reference frame, and the to-be-processed area after color transformation in the reference frame is used as the color transformation area, including: According to a preset block division method, the area to be processed of the fused image is divided into a plurality of blocks as a plurality of fusion blocks, and the area to be processed of the reference frame is divided into a plurality of blocks as a plurality of reference blocks; For each fused block, determining a color transformation coefficient corresponding to when transforming the corresponding reference block into the fused block; According to the color transformation coefficient of each reference block, color transformation is performed on each reference block to obtain a plurality of transformed reference blocks; The multiple transformed reference blocks are merged to obtain the color transformed region.
5. The method according to claim 4, characterized in that The step of determining, for each fused block, a color transformation coefficient corresponding to when transforming a corresponding reference block into a fused block comprises: Traverse multiple fused blocks, determine the current fused block, and use the reference block corresponding to the current fused block as the current reference block; According to a preset sampling method, the current fusion block is sampled to obtain a plurality of fusion block sampling points, and the current reference block is sampled to obtain a plurality of reference block sampling points; Determine a color transformation coefficient corresponding to when the pixel value of each reference block sampling point is transformed into the pixel value of the corresponding fusion block sampling point through color transformation; The color transformation coefficient corresponding to each reference block sampling point is used as the color transformation coefficient corresponding to the transformation of the current reference block into the current fused block.
6. The method according to claim 4, characterized in that Two adjacent blocks among the multiple fused blocks have overlapping areas, and two adjacent blocks among the multiple reference blocks have overlapping areas.
7. The method according to any one of claims 1-2, characterized in that: Generating a ghost mask between the at least two frames of images comprises: Performing alignment processing on the at least two frames of images; Motion detection is performed on the aligned at least two frames of images, and a ghost mask between the at least two frames of images is generated according to the motion detection result.
8. The method according to claim 7, characterized in that Performing alignment processing on the at least two frames of images includes: The at least two frames of images are aligned using an enhanced correlation coefficient algorithm.
9. The method according to claim 7, characterized in that: The step of performing motion detection on the aligned at least two frames of images and generating a ghost mask between the at least two frames of images according to the motion detection result includes: Calculating the optical flow field between the at least two aligned image frames by a fast dense optical flow algorithm; A ghost mask between the at least two frames of images is generated according to the optical flow field.
10. The method according to any one of claims 1-2, characterized in that: Fusing the at least two frames of images to obtain a fused image includes: The at least two frames of images are fused by an exposure fusion algorithm to obtain a fused image.
11. An image processing device, characterized in that: include: A motion detection module, configured to generate a ghost mask between at least two frames of images after acquiring at least two frames of images with different exposure times; A fusion module, used for fusing the at least two frames of images to obtain a fused image; A processing region determination module, used to determine a reference frame from the at least two frames of images, and to determine the region to be processed of the fused image and the region to be processed in the reference frame respectively according to the ghost mask; A color transformation module, used for performing color transformation on the area to be processed in the reference frame according to the area to be processed in the fused image, and taking the area to be processed after color transformation in the reference frame as the color transformation area; The region processing module is used to replace the region to be processed in the fused image with the color conversion region.
12. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image processing method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
Color conversion devices, color conversion methods, and color conversion programs
CN102262776A
System and method for multi-frame temporal de-noising using image alignment
US20150262341A1
