Image processing method and device, electronic equipment and computer storage medium

Abstract

The embodiment of the invention discloses an image processing method, which comprises the following steps: acquiring a first image frame, a second image frame and a mask image, determining a first ghosting image according to the first image frame and the second image frame, and determining a connected domain of a motion area in the mask image and a flag bit of the connected domain, the flag bit being used for indicating an image frame fused by the connected domain, and according to the flag bit of the connected domain, fusing the first image frame and the second image frame to obtain an output image. The embodiment of the invention further provides an image processing device, electronic equipment and a computer storage medium.

Description

Technical Field

[0001] This application relates to the technology of image processing, and in particular, to an image processing method, apparatus, electronic device, and computer storage medium. Background Art

[0002] When taking images, the method of multi-frame fusion can be used to improve the image quality of the output image.

[0003] However, if there is a situation where the object image position changes continuously in multiple frames of images, then in the fusion of multiple frames of images, if the information of a certain frame is completely fused, the image quality of the output image will be poor. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, and computer storage medium, which can improve the image quality of the output image obtained by multi-frame fusion.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an image processing method, including:

[0007] Obtain a first image frame, a second image frame, and a mask image; wherein, the mask image is used to identify a motion area; the motion area is an area where the object image position moves between the first image frame and the second image frame;

[0008] Determine a first ghost image according to the first image frame and the second image frame; wherein, the ghost image is used to represent the value corresponding to the ghost generated during the fusion between the first image frame and the second image frame;

[0009] Determine a flag bit of the connected domain of the motion area in the mask image according to the first ghost image; wherein, the flag bit is used to indicate the image frame fused by the connected domain;

[0010] Fuse the first image frame and the second image frame according to the flag bit of the connected domain to obtain an output image.

[0011] In a second aspect, embodiments of this application provide an image processing apparatus, including:

[0012] An obtaining module, configured to obtain a first image frame, a second image frame, and a mask image; wherein, the mask image is used to identify a motion area; the motion area is an area where the object image position moves between the first image frame and the second image frame;

[0013] A first determination module, configured to determine a first ghost image according to the first image frame and the second image frame; wherein the ghost image is used to characterize a value corresponding to a ghost generated during the fusion between the first image frame and the second image frame.

[0014] A second determination module, configured to determine a flag bit of a connected component of the moving region in the mask image according to the first ghost image; wherein the flag bit is used to indicate the image frame fused by the connected component.

[0015] A processing module, configured to fuse the first image frame and the second image frame according to the flag bit of the connected component to obtain an output image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a storage medium storing processor-executable instructions; the storage medium depends on the processor to execute operations through a communication bus, and when the instructions are executed by the processor, execute the image processing method described in the above one or more embodiments.

[0017] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing executable instructions, and when the executable instructions are executed by one or more processors, the processor executes the image processing method described in the above one or more embodiments.

[0018] An embodiment of the present application provides an image processing method, apparatus, electronic device, and computer storage medium. The method includes obtaining a first image frame, a second image frame, and a mask image. The mask image is used to identify a moving region, and the moving region is a region where the object image position moves between the first image frame and the second image frame. According to the first image frame and the second image frame, a first ghost image is determined. The ghost image is used to characterize a value corresponding to a ghost generated during the fusion between the first image frame and the second image frame. According to the first ghost image, a flag bit of a connected component of the moving region in the mask image is determined. The flag bit is used to indicate the image frame fused by the connected component. According to the flag bit of the connected component, the first image frame and the second image frame are fused to obtain an output image. That is to say, in the embodiment of the present application, by obtaining two image frames and a mask image, a first ghost image is determined according to the two image frames, and then a flag bit of a connected component of the moving region in the mask image is determined based on the first ghost image, so that it is possible to know the image frame fused by the connected component, making the image frame fused by the connected component related to the value corresponding to the ghost during the fusion of the two image frames. Based on this, the two image frames are fused based on different flag bits, so that different image frames can be fused for different connected components, avoiding completely fusing a certain frame in multi-frame fusion, but fusing the corresponding image frames based on the flag bits of different connected components, which can improve the ghost phenomenon in multi-frame fusion, thereby improving the quality of the image obtained by multi-frame fusion. Brief Description of the Drawings

[0019] Figure 1a It is a schematic diagram of a short frame in the related art;

[0020] Figure 1b It is a schematic diagram of a long frame in the related art;

[0021] Figure 2 It is a schematic flowchart of an optional image processing method provided by an embodiment of the present application;

[0022] Figure 3 It is a schematic flowchart of an example of an optional image processing method provided by an embodiment of the present application;

[0023] Figure 4a It is a schematic diagram of an optional loose ghost image provided by an embodiment of the present application;

[0024] Figure 4b It is a schematic diagram of an optional strict ghost image provided by an embodiment of the present application;

[0025] Figure 5 It is a schematic structural diagram of an optional image processing device provided by an embodiment of the present application;

[0026] Figure 6 It is a schematic structural diagram of an optional electronic device provided by an embodiment of the present application. Detailed Description of the Embodiments

[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0028] In the related art, multi-frame fusion can be used to improve the image quality of the output image.

[0029] However, when shooting a moving object image, after collecting multiple frames, Figure 1a It is a schematic diagram of a short frame in the related art, Figure 1b It is a schematic diagram of a long frame in the related art. As shown in Figure 1a and Figure 1b shown, the human body is moving, Figure 1b The exposure time of the long frame in Figure 1a is longer than the exposure time of the short frame in

[0030] In the multi-frame fusion method, ghosts will be generated. At this time, if the long frame or the short frame is completely fused, the image quality of the output image will be poor. Figure 2 It is a schematic flowchart of an optional image processing method provided by an embodiment of the present application, as shown inFigure 2 As shown, the method for processing the image may include:

[0031] S201: Obtain a first image frame, a second image frame, and a mask image;

[0032] In the embodiments of the present application, when the camera of the electronic device takes pictures, a first image frame and a second image frame are obtained from the camera, and the output image is obtained by using the first image frame and the second image frame. Among them, the electronic device may use the main camera to obtain the first image frame and the second image frame, or other cameras may also be used to obtain the first image frame and the second image frame. For example, the other camera may be a wide-angle camera or a telephoto camera.

[0033] After obtaining the first image frame and the second image frame, a mask image may be generated according to the first image frame and the second image frame, so as to obtain the mask image. Among them, the mask image is used to identify the moving area, and the moving area is the area where the object image position moves between the first image frame and the second image frame. That is to say, after obtaining the first image frame and the second image frame, a mask image for identifying the area where the object image position moves between the first image frame and the second image frame can be determined. For example, the pixel value of the moving area of the mask image is 255, and the pixel value of the area other than the moving area in the mask image is 0.

[0034] Here, it should be noted that the trained neural network model can be obtained by training the neural network model. The first image frame and the second image frame are input into the model to obtain the mask image, or the image recognition method can be used to identify the moving area, so as to obtain the mask image. Of course, the mask image can also be obtained by using the model + image recognition method. Here, the embodiments of the present application do not make specific limitations on this.

[0035] S202: Determine a first ghost image according to the first image frame and the second image frame;

[0036] After obtaining the first image frame and the second image frame through the above S201, in S203, a first ghost image can be determined according to the first image frame and the second image frame. Here, it should be noted that the first ghost image can be determined according to the directly obtained first image frame and the second image frame, or the first image frame and the second image frame can be downsampled first and then the first ghost image can be determined. Here, the embodiments of the present application do not make specific limitations on this.

[0037] Among them, the ghost image is used to represent the numerical value corresponding to the ghost generated during the fusion of the first image frame and the second image frame. That is to say, after obtaining the first image frame and the second image frame, based on these two image frames, the numerical value corresponding to the ghost generated during the fusion due to the movement of the object image position between the first image frame and the second image frame can be determined. Here, a trained model can be used to obtain the first ghost image, or a preset algorithm can be used to obtain the first ghost image. Here, the embodiments of the present application do not make specific limitations in this regard.

[0038] In this way, the ghost numerical value in the first ghost image can be used to measure an index data of the ghost generated during the fusion of the first image frame and the second image frame. Moreover, the ghost numerical value can also be used to represent the degree of difference between the first image frame and the second image frame. Among them, the smaller the ghost numerical value, the greater the difference between the two corresponding image blocks of the first image frame and the second image frame; the smaller the ghost numerical value, the smaller the difference between the two corresponding image blocks of the first image frame and the second image frame.

[0039] S203: Determine the flag bit of the connected component of the moving region in the mask image according to the first ghost image;

[0040] After obtaining the mask image through the above S201, the connected component of the moving region in the mask image can be determined. After obtaining the first ghost image through the above S202, the flag bit of the connected component can be determined according to the first ghost image. Here, it should be noted that the determined connected component can be one, or two or more. The embodiments of the present application do not make specific limitations in this regard.

[0041] In the embodiments of the present application, a trained model can be used to determine the connected component of the moving region in the mask image, or the clustering method can be used to determine the connected component of the moving region in the mask image, or other methods can be used to determine the connected component of the moving region in the mask image. Here, the embodiments of the present application do not make specific limitations in this regard.

[0042] Among them, the flag bit is used to indicate the image frame fused by the connected component. That is to say, after determining the connected component, the image frame fused by the connected component can be determined according to the first ghost image. Here, the image frame fused by the connected component can be the first image frame or the second image frame. The embodiments of the present application do not make specific limitations in this regard.

[0043] In determining the flag bits of the connected components based on the first ghost image, the flag bits of the connected components can be directly determined according to the ghost values of the connected components in the first ghost image, or the flag bits of the connected components can be determined according to the ghost values of the connected components in the first ghost image and the pixel values of the connected components in the first image frame, or the flag bits of the connected components can be determined according to the ghost values of the connected components in the first ghost image and the pixel values of the connected components in the second image frame. Here, the embodiments of the present application do not make specific limitations on this.

[0044] S204: According to the flag bits of the connected components, fuse the first image frame and the second image frame to obtain an output image.

[0045] After determining the flag bits of the connected components of the moving regions in the mask image through the above S203, in S204, the first image frame and the second image frame can be fused according to the flag bits of the connected components to obtain an output image.

[0046] Here, first, the first fusion weight of the first image frame and the second fusion weight of the second image frame can be adjusted according to the flag bits of the connected components, and the first image frame and the second image frame are processed based on the adjusted first fusion weight and the adjusted second fusion weight, so as to obtain an output image. It can also be that the image frames fused by the connected components indicated by the flag bits of the connected components are fused with the first image frame and the second image frame to obtain an output image. Here, the embodiments of the present application do not make specific limitations on this.

[0047] Among them, by using the flag bits of the connected components, it is known which image frames the connected components are fused with. Based on the fused image frames, adjustment parameters can be determined, and the adjustment parameters are used to adjust the fusion weights of the regions corresponding to the connected components in the first image frame and the fusion weights of the regions corresponding to the connected components in the second image frame, so that the adjusted fusion weights can be obtained.

[0048] It should be noted that for the regions corresponding to the regions other than the moving regions in the first image frame, the first image frame and the second image frame can be used to obtain an output image by using a multi-frame fusion technology.

[0049] Finally, the output image is displayed on the display screen of the electronic device. The output image obtained by using the above image processing method can improve the image quality of the moving regions in the image after multi-frame fusion and improve the image quality of the image obtained by multi-frame fusion.

[0050] For the above first image frame and second image frame, in an optional embodiment, the exposure duration of the first image frame is greater than the exposure duration of the second image frame.

[0051] Understandably, the exposure duration of the first image frame is greater than that of the second image frame, that is, the exposure times of the first image frames are different. Then, in the case where the imaged object moves, compared with the first and second image frames obtained by shooting, due to the presence of the moving imaged object, the positions of the two image frames with respect to the moving imaged object are different.

[0052] Thus, using two image frames with different exposure durations for multi-frame fusion can further improve the image quality of the image obtained by multi-frame fusion in the case of moving imaged objects.

[0053] In order to determine the first ghost image, in an alternative embodiment, determining the first ghost image based on the first image frame and the second image frame may include:

[0054] Determine a difference image based on the first image frame and the second image frame;

[0055] Determine the image corresponding to the brightness penalty term of the first image frame;

[0056] Determine the image corresponding to the noise penalty term of the first image frame;

[0057] Determine the first ghost image based on the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term.

[0058] Understandably, on the basis of knowing the first image frame and the second image frame, a difference image can be determined based on the first image frame and the second image frame. The difference image can be formed by calculating the difference in pixel values of two corresponding image blocks, or can be formed by calculating the difference after processing the pixel values of two corresponding image blocks. Here, the embodiments of the present application do not make specific limitations in this regard.

[0059] Moreover, based on the first image frame, the image corresponding to the brightness penalty term of the first image frame and the image corresponding to the noise penalty term of the first image frame are determined, and then the first ghost image is determined based on the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term.

[0060] In determining the image corresponding to the brightness penalty term, mainly based on the pixel values of each image block of the first image frame and a preset first constant term, the brightness penalty term of each image block is determined, so as to obtain the image corresponding to the brightness penalty term. Here, the brightness penalty term can be determined using a preset correspondence relationship, or can be determined using a model method. The embodiments of the present application do not make specific limitations in this regard.

[0061] The image corresponding to the brightness penalty term of the first image frame can be obtained using the following formula:

[0062] Coof = exp(-(x_l - 1.0) * (x_l - 1.0) / (2 * sigma * sigma)) (1)

[0063] Among them, Coof represents the brightness penalty term of each channel in each image block, x_l represents the pixel value of each channel in each image block of the first image frame, and sigma represents a preset first constant.

[0064] In the image corresponding to the noise penalty term, mainly after the pixel values of each channel of each image block of the first image frame and the noise parameters are obtained, the noise penalty term can be determined by using a preset corresponding relationship, or the noise penalty term can be determined by using a model. The embodiments of the present application do not make specific limitations on this.

[0065] Finally, after knowing the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term, the ghost value of each image block can be determined based on this, so as to form the first ghost image. Among them, for three mutually corresponding image blocks in the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term, the ghost value of the image block in the first ghost image is obtained by using the values of the three image blocks.

[0066] In this way, through the above difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term, the value corresponding to the ghost generated during the fusion between the first image frame and the second image frame can be determined, which is beneficial to accurately determining the flag bit of the connected domain.

[0067] Further, in order to determine the above difference image, in an optional embodiment, according to the first image frame and the second image frame, determining the difference image may include:

[0068] Subtract the product of the second pixel value and the exposure ratio of the first image frame to the second image frame from the first pixel value to obtain the difference image.

[0069] It can be understood that the first pixel value and the second pixel value are obtained. Among them, the first pixel value is: the pixel value of each channel of each image block of the first image frame; the second pixel value is: the pixel value of each channel of the image block corresponding to each image block of the first image frame in the second image frame. That is to say, the mutually corresponding image blocks in the first image frame and the second image frame are obtained. For each pair of mutually corresponding image blocks, the pixel value of each channel is determined, denoted as the first pixel value and the second pixel value.

[0070] Determine the exposure ratio between the first image frame and the second image frame. Here, the exposure ratio between the first image frame and the second image frame can be pre-stored, or the exposure ratio between the first image frame and the second image frame can be calculated in real time. After obtaining the exposure ratio, multiply the second pixel value by the exposure ratio, and then subtract the product from the first pixel value, so as to obtain the difference of each channel in each image block, and obtain the difference image. Among them, the formula for calculating the difference is specifically as follows:

[0071] diff = x_l - x_s * rate(2)

[0072] Among them, diff represents the difference of each channel in each image block, x_l represents the pixel value of each channel in each image block of the first image frame, and x_s represents the pixel value of each channel in each image block of the second image frame.

[0073] In this way, the above method of finding the difference is used to determine the difference image, so that the determined difference image takes into account the exposure ratio while determining the difference between the first image frame and the second image frame, which helps to determine the value corresponding to the ghosting generated during fusion in the two image frames, that is, the first ghost image, so as to determine the flag bit of the accurate connected domain.

[0074] Further, in order to determine the image corresponding to the noise penalty term of the first image frame, in an optional embodiment, determining the image corresponding to the noise penalty term of the first image frame may include:

[0075] Obtain the noise parameters of the camera that captures the first image frame and the second image frame;

[0076] According to the noise parameters, the pixel value of each image block in the first image frame, and the sensitivity of the obtained camera, determine the image corresponding to the noise penalty term.

[0077] It can be understood that the electronic device first obtains the noise parameters of the camera that captures the first image frame and the second image frame. Here, the noise parameters may include Gaussian noise parameters p_guss = {g0, g1} and / or Poisson noise parameters p_po = {p0, p1, p2}.

[0078] When determining the image corresponding to the noise penalty term according to the noise parameters, the pixel value of each image block in the first image frame, and the sensitivity of the obtained camera, the following formula can be used for calculation:

[0079] G = iso * g0 + g1(3)

[0080] Bias = iso * iso * p0 + iso * p1 + p2(4)

[0081] Warp = min(sqrt(G * x_l + Bias) * 50, 150)(5)

[0082] Among them, Warp represents the noise penalty term for each channel of each image block, iso represents the sensitivity of the camera, and x_l can be the pixel values (x_r, x_g, x_b) of the three channels of each image block in the first image frame.

[0083] In this way, the noise penalty term for each channel of each image block can be determined through the above method, so as to obtain the image corresponding to the noise penalty term, which helps to determine the ghost image that can reflect the difference of the moving object images in the two image frames, and thus determine the flag bit of the accurate connected region.

[0084] Furthermore, in order to determine the first ghost image, in an optional embodiment, determining the first ghost image according to the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term may include:

[0085] According to the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term, call the Wiener filtering algorithm to determine the ghost image corresponding to each channel;

[0086] Select the maximum value of the ghost values in the ghost image corresponding to each channel to obtain the first ghost image.

[0087] It can be understood that when determining the ghost image corresponding to each channel by calling the Wiener filtering algorithm according to the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term, the following formula can be used for calculation:

[0088] V_ghost = 1 - (diff * diff) / (diff * diff + Warp * Coof * Coof * Warp) (6)

[0089] Among them, V_ghost represents the ghost value of each channel of each image block.

[0090] After calculating the ghost values of each channel of each image block by the above method, for each channel in each image block, select the maximum value of the ghost values of each channel and use it to form the first ghost image.

[0091] In this way, the first ghost image is obtained by using the above Wiener filtering algorithm and the maximum value selection method, so that the determined first ghost image can accurately reflect the value corresponding to the ghost generated during the fusion between the first image frame and the second image frame.

[0092] In order to determine the connected region of the moving area in the mask image, in an optional embodiment, the above may include:

[0093] Cluster the moving area in the mask image to obtain the connected region.

[0094] Understandably, in order to obtain connected regions, the moving regions in the mask image can be clustered here. Specifically, the moving regions in the mask image can be clustered based on the position coordinates in the moving regions.

[0095] Here, a clustering algorithm can be used to cluster the moving regions in the mask image to obtain connected regions. Among them, the above clustering algorithm can be the K-means algorithm, the hierarchical clustering algorithm, the spectral density clustering algorithm, etc. Here, the embodiments of the present application do not make specific limitations on this.

[0096] In this way, the connected regions in the moving regions can be obtained through the above clustering method. In this way, different regions in the moving regions can be divided, so that different image frames can be fused for different regions, which helps to improve the image quality of the output image obtained in multi-frame fusion.

[0097] In order to improve the accuracy of the determined connected regions, in an optional embodiment, the above method may further include:

[0098] Perform dilation processing on the mask image to obtain the dilated mask image;

[0099] Use the difference image between the first image frame and the second image frame to perform guided filtering on the dilated mask image to re-obtain the mask image.

[0100] Understandably, before determining the connected regions, dilation processing is performed on the mask image. Here, the dilation processing can be performed based on the convolution kernel method or based on the pixel difference method. Here, the embodiments of the present application do not make specific limitations on this.

[0101] In addition, the difference image between the first image frame and the second image frame can be calculated. Here, the difference image can be obtained by the difference in pixel values of the image blocks at the corresponding positions in the first image frame and the second image frame. Using this difference image, guided filtering can be performed on the dilated mask image, that is, using the difference image as the guidance image and the mask image as the input image, and the output image is re-determined as the mask image.

[0102] In this way, the mask image is re-obtained through the above dilation processing and guided filtering, which can make the white regions in the mask image expand outward, thereby filling holes, enhancing connections or expanding the moving regions, and the edges are smoother, which helps to determine more accurate connected regions.

[0103] Before the first ghost image, in an optional embodiment, the above method may further include:

[0104] Before determining the first ghost image based on the first image frame and the second image frame, downsample the first image frame and the second image frame by a preset first multiple to obtain the first image frame and the second image frame again.

[0105] Understandably, before the first ghost image, downsample the first image frame and the second image frame, and here use the preset first multiple for downsampling to obtain the first image frame and the second image frame again. Among them, the preset first multiple is generally 8.

[0106] Then, based on this, determining the connected component, the first ghost image, and the flag bit of the connected component can reduce the computational complexity while improving the image quality of the image obtained in multi-frame fusion.

[0107] In order to determine the flag bit of the connected component in the moving region of the mask image, in an alternative embodiment, S203 may include:

[0108] Determine the flag bit of the connected component according to whether the first ghost image and the first image frame meet the preset conditions.

[0109] Understandably, after determining the first ghost image, it can be determined whether the first ghost image and the first image frame meet the preset conditions. Here, mainly determine whether the ghost value in the first ghost image and the pixel value in the first image frame meet the preset conditions. The flag bits of the connected component determined when the preset conditions are met and not met are different.

[0110] In this way, by determining the flag bit of the connected component according to whether the first ghost image and the first image frame meet the preset conditions, it is possible to combine the value corresponding to the ghost when the first image frame and the second image frame are fused and the pixel value of the first image frame to determine the image frame fused by the connected component, and a more accurate flag bit can be determined for the connected component, thereby improving the image quality of the output image obtained in multi-frame fusion.

[0111] Further, in order to determine the flag bit of the connected component, in an alternative embodiment, determining the flag bit of the connected component according to whether the first ghost image and the first image frame meet the preset conditions may include:

[0112] Determine the ghost value of the connected component according to the first ghost image;

[0113] Determine the pixel value of the connected component according to the first image frame;

[0114] When the ghost value of the connected component and the pixel value of the connected component meet the preset conditions, determine the flag bit of the connected component as the first value;

[0115] In the case where the ghost value of the connected component and the pixel value of the connected component do not meet the preset conditions, determine that the flag bit of the connected component is the second value.

[0116] Understandably, the ghost value of the connected component in the first ghost image can be determined here, and the pixel value of the connected component in the first image frame can be determined accordingly. Then, it is judged whether the ghost value of the connected component and the pixel value of the connected component meet the preset conditions. Among them, the preset conditions may include: a preset exposure condition and a preset occlusion condition. The preset exposure condition is used to screen out overexposed connected components, and the preset occlusion condition is used to screen out occluded connected components.

[0117] If, after judgment, the ghost value of the connected component and the pixel value of the connected component meet the preset conditions, it indicates that the connected component meets the preset exposure condition and the preset occlusion condition, that is, the connected component belongs to the overexposed and unoccluded area. Therefore, determine that the flag bit of the connected component is the first value; among them, the first value indicates that the image frame fused by the connected component is the first image frame. That is to say, when the connected component belongs to the overexposed and unoccluded area, the image frame fused by the connected component is the long frame.

[0118] The ghost value of the connected component and the pixel value of the connected component do not meet the preset conditions, indicating that the connected component does not meet the preset exposure condition and / or the preset occlusion condition, that is, the connected component does not belong to the overexposed or occluded area. Therefore, determine that the flag bit of the connected component is the second value; among them, the second value indicates that the image frame fused by the connected component is the second image frame. That is to say, when the connected component does not belong to the overexposed or occluded area, the image frame fused by the connected component is the short frame.

[0119] In this way, through the judgment of whether the ghost value of the connected component and the pixel value of the connected component meet the preset conditions, the image frame fused by the connected component can be accurately determined, which helps to adjust the fusion weight of the moving area.

[0120] In order to determine that the ghost value of the connected component and the pixel value of the connected component meet the preset conditions, in an optional embodiment, the above method may further include:

[0121] In the case where the proportion is less than the preset proportion threshold, determine that the ghost value of the connected component and the pixel value of the connected component meet the preset conditions.

[0122] Understandably, the electronic device first calculates the proportion, where the proportion is: the area of the connected region with a ghost value less than the preset ghost threshold and the maximum value of the pixel values in each channel of the connected region greater than the preset pixel threshold occupies the area of the first image frame. That is to say, first select the image blocks of the connected region with a ghost value less than the preset ghost threshold and the maximum value of the pixel values in each channel of the connected region greater than the preset pixel threshold, then calculate the area of the image block, and then determine the proportion of the area occupying the area of the first image frame, so as to obtain the proportion.

[0123] It should be noted that when the ghost value of the above-mentioned connected region is less than the preset ghost threshold, it means that the image block belongs to an unobstructed image block. When the maximum value of the pixel values in each channel of the connected region is greater than the preset pixel threshold, it means that the image block belongs to an overexposed image block. In this way, the overexposed and unobstructed image blocks can be screened out, and thus the proportion of the overexposed and unobstructed image blocks in all the image blocks of the first image frame can be determined.

[0124] It can be seen that when the proportion is less than the preset proportion threshold, the image frame fused by the connected region is a long frame, and the rest are short frames.

[0125] In this way, through the above comparison with the threshold, it can be determined that the ghost value of the connected region and the pixel value of the connected region meet the preset conditions, so as to screen out the overexposed and unobstructed connected regions, and determine the image frame fused by them as the long frame, so that the long frame can be used in the output image, improving the image quality of the output image in multi-frame fusion.

[0126] In order to obtain the output image, in an optional embodiment, S204 may include:

[0127] According to the flag bit of the connected region, adjust the first fusion weight of the first image frame and the second fusion weight of the second image frame to obtain the adjusted first fusion weight and the adjusted second fusion weight;

[0128] According to the adjusted first fusion weight and the adjusted second fusion weight, process the first image frame and the second image frame to obtain the output image.

[0129] Understandably, the first fusion weight and the second fusion weight can be adjusted first according to the flag bit of the connected region, and then the first image frame and the second image frame can be processed according to the adjusted first fusion weight and the adjusted second fusion weight, so as to obtain the output image.

[0130] Among them, the fusion weight refers to the weight assigned to each data source, feature, model, or classifier when fusing the results of multiple data sources, features, models, or classifiers. These weights reflect their importance or influence in the final fusion result. By reasonably allocating weights, a more accurate and robust fusion result can be obtained. In the embodiments of the present application, the fusion weights of the first image frame and the second image frame can be calculated based on the first image frame and the second image frame using some image algorithms, or can be obtained based on the first image frame and the second image frame using some trained models. Moreover, the fusion weight of the first image frame includes the fusion weights of each image block in the first image frame. Similarly, the fusion weight of the second image frame includes the fusion weights of each image block in the second image frame.

[0131] Here, the flag bit of the connected component can be used to know the image frame fused by the connected component. Based on the fused image frame, adjustment parameters can be determined, and the adjustment parameters are used to adjust the fusion weights of the regions corresponding to the connected components in the first image frame and the fusion weights of the regions corresponding to the connected components in the second image frame, so that the adjusted fusion weights can be obtained.

[0132] It should be noted that for the regions corresponding to the regions other than the motion regions in the first image frame, the first fusion weight and the second fusion weight can remain unchanged.

[0133] For the fusion weights corresponding to each image block in the adjusted first fusion weight, determine the fusion weights of the image blocks corresponding to the image blocks in the adjusted first fusion weight in the adjusted second fusion weight. Based on the fusion weights of two corresponding image blocks, perform weighted summation on the pixel values of the image blocks in the first image block corresponding to it and the pixel values of the image blocks in the first image block corresponding to it, so that the pixel values of each image block can be obtained to obtain the output image.

[0134] In this way, when the output image is displayed on the display screen of the electronic device, the output image obtained by using the above image processing method can improve the image quality of the motion regions of the image after multi-frame fusion and improve the image quality of the image obtained by multi-frame fusion.

[0135] In order to implement the adjustment of the first fusion weight and the second fusion weight based on the flag bit of the connected component, in an optional embodiment, according to the flag bit of the connected component, adjusting the first fusion weight of the first image frame and the second fusion weight of the second image frame to obtain the adjusted first fusion weight and the adjusted second fusion weight may include:

[0136] Determine a second ghost image according to the first image frame and the second image frame;

[0137] Determine a third ghost image based on the first image frame, the second image frame, and the image corresponding to the preset penalty term;

[0138] Select a target ghost image from the second ghost image and the third ghost image according to the flag bit of the connected component;

[0139] Adjust the first fusion weight of the first image frame and the second fusion weight of the second image frame by using the ghost values in the target ghost image to obtain the adjusted first fusion weight and the adjusted second fusion weight.

[0140] It can be understood that here the second ghost image is determined first according to the first image frame and the second image frame. Here, the method for determining the second ghost image is the same as the method for determining the first ghost image above, and will not be elaborated here.

[0141] In addition, when determining the third ghost image according to the first image frame, the second image frame, and the image corresponding to the preset penalty term, different from the above determination of the first ghost image, here, the image corresponding to the brightness penalty term and the image corresponding to the noise penalty term are images composed of a fixed value, and moreover, the constant used in determining the third ghost image may be the same as or different from the constant used in determining the first ghost image. Here, the embodiments of the present application do not make specific limitations on this.

[0142] Among them, the ghost image is used to indicate the value corresponding to the ghost generated during the fusion between the first image frame and the second image frame. It can be seen that through the above method, two different ghost images can be determined. Among them, compared with the third ghost image, for the second ghost image, since the image corresponding to the brightness penalty term and the image corresponding to the noise penalty term are a fixed value, the second ghost value is looser and the third ghost value is stricter.

[0143] In order to realize the adjustment of the first fusion weight and the second fusion weight, here, according to the flag bit of the connected component, a target ghost image can be selected from the second ghost image and the third ghost image, and then the ghost values in the target ghost image are used to adjust the first fusion weight and the second fusion weight to obtain the adjusted first fusion weight and the adjusted second fusion weight.

[0144] It should be noted that if the first ghost image and the second ghost image are determined according to the original first image frame, then the first ghost image is the same as the second ghost image. If the first ghost image and the second ghost image are determined by using the first image frame after downsampling, when the downsampling ratio is the same, the determined second ghost image is the same as the first ghost image, and here the first ghost image can be directly used as the second ghost image. If the downsampling ratios are different, the determined first ghost image and second ghost image are different.

[0145] In this way, a relatively loose ghost image and a relatively strict ghost image can be determined through the above method. Based on the flag bits of the connected components, the target ghost image can be selected therefrom to adjust the first fusion weight and the second fusion weight, which is beneficial to improving the image quality of the output image.

[0146] Before determining the second ghost image and the third ghost image, in an optional embodiment, the above method may further include:

[0147] Before determining the second ghost image according to the first image frame and the second image frame, and determining the third ghost image according to the first image frame, the second image frame and the image corresponding to the preset penalty term, downsample the first image frame and the second image frame according to a preset second multiple to obtain the first image frame and the second image frame again.

[0148] It can be understood that the electronic device can downsample the acquired first image frame and second image frame according to a preset second multiple, so as to obtain the first image frame and the second image frame again. Among them, the preset first multiple is generally different from the preset second multiple, and the preset first multiple is greater than the preset second multiple.

[0149] In this way, the method of downsampling can reduce the computational complexity of determining the second ghost image and the third ghost image. Setting the preset second multiple to be less than the preset first multiple can improve the accuracy of the second ghost image and the third ghost image, which helps to effectively adjust the first fusion weight and the second fusion weight.

[0150] In order to determine the target ghost image, in an optional embodiment, according to the flag bits of the connected components, selecting the target ghost image from the second ghost image and the third ghost image may include:

[0151] When the flag bit of the connected component indicates that the image frame fused by the connected component is the first image frame, determine the second ghost image as the target ghost image;

[0152] When the flag bit of the connected component indicates that the image frame fused by the connected component is the second image frame, determine the third ghost image as the target ghost image.

[0153] Understandably, after knowing the flag bit of the connected component, it is possible to know the image frames fused by the connected component. Then, when the flag bit of the connected component indicates that the image frame fused by the connected component is the first image frame, and when the exposure duration of the first image frame is greater than that of the second image frame, it can be seen that when the flag bit of the connected component indicates that the image frame fused by the connected component is a long frame, a loose ghost image can be used to adjust the first fusion weight and the second fusion weight. When the flag bit of the connected component indicates that the image frame fused by the connected component is a short frame, a strict ghost image can be used to adjust the first fusion weight and the second fusion weight.

[0154] Here, it should be noted that in the adjustment of the first fusion weight and the second fusion weight, mainly the first fusion weight and the second fusion weight of the regions corresponding to the motion regions in the first image frame or the second image frame are adjusted.

[0155] In this way, through the image frames fused by the above-mentioned connected components, a loose ghost image or a strict ghost image is determined to adjust the first fusion weight and the second fusion weight of the regions corresponding to the motion regions in the first image frame or the second image frame, realizing the adjustment of the first fusion weight and the second fusion weight by selecting different ghost images for different connected components, and improving the image quality of the output image in multi-frame fusion.

[0156] In order to use the target ghost image to realize the adjustment of the first fusion weight and the second fusion weight, in an optional embodiment, using the ghost value in the target ghost image to adjust the first fusion weight of the first image frame and the second fusion weight of the second image frame, and obtaining the adjusted first fusion weight and the adjusted second fusion weight may include:

[0157] When the flag bit of the connected component indicates that the image frame fused by the connected component is the first image frame, the minimum value of the second fusion weight and the ghost value in the target ghost image is determined as the adjusted second fusion weight, and the difference obtained by subtracting the adjusted second fusion weight from one is determined as the adjusted first fusion weight;

[0158] When the flag bit of the connected component indicates that the image frame fused by the connected component is the second image frame, the minimum value of the first fusion weight and the ghost value in the target ghost image is determined as the adjusted first fusion weight, and the difference obtained by subtracting the adjusted first fusion weight from one is determined as the adjusted second fusion weight.

[0159] Understandably, for the region corresponding to the motion region in the first image frame, when the flag bit of the connected component indicates that the image frame fused by the connected component is a long frame, the second fusion weight of the connected component is compared with the ghost value of the connected component in the second ghost image, and the minimum value is determined as the adjusted second fusion weight. Subtracting the adjusted second fusion weight from one gives the adjusted first fusion weight.

[0160] For the region corresponding to the motion region in the first image frame, when the flag bit of the connected component indicates that the image frame fused by the connected component is a short frame, the first fusion weight of the connected component is compared with the ghost value of the connected component in the third ghost image, and the minimum value is determined as the adjusted first fusion weight. Subtracting the adjusted first fusion weight from one gives the adjusted second fusion weight.

[0161] In this way, different ghost images are determined through the flag bits of different connected components, and different ghost images are used to make different adjustments to the first fusion weight and the second fusion weight, making the adjustment of the fusion weight of the region corresponding to the motion region in the first image frame more targeted, which helps to improve the image quality of the output image obtained in multi-frame fusion.

[0162] When the first image frame and the second image frame are re-obtained by using the downsampling method, in order to implement the adjustment of the first fusion weight and the second fusion weight, in an optional embodiment, the above method may further include:

[0163] When the resolution of the target ghost image is different from the resolution of the first image frame, perform interpolation processing on the target ghost image to re-obtain the target ghost image so that the resolution of the target ghost image is the same as that of the first image frame.

[0164] Understandably, in determining the second ghost image and the third ghost image, for obtaining the second ghost image and the third ghost image based on the first image frame obtained by downsampling according to a preset second multiple, the resolutions of the obtained second ghost image and the third ghost image are different from the resolution of the first image frame acquired by the camera.

[0165] Then, when using the target ghost image to adjust the first fusion weight of the first image frame and the second fusion weight of the second image frame, it is necessary to first perform interpolation processing on the target ghost image to re-obtain the target ghost image so that the resolution of the target ghost image is the same as that of the first image frame.

[0166] In this way, the image blocks between the re-obtained target ghost image and the first image frame acquired by the camera correspond one by one, and the target ghost image can be used to implement the adjustment of the first fusion weight and the second fusion weight of the region corresponding to the motion region in the first image frame.

[0167] Based on the above adjustments, in order to further adjust the edge of the motion area, in an alternative embodiment, the above method may further include:

[0168] When the flag bit of the connected component indicates that the image frame fused by the connected component is the first image frame, determine the maximum value of the pixel values of each channel of each image block in the area corresponding to the connected component in the first image frame;

[0169] When the maximum value is greater than a preset threshold, adjust the adjusted first fusion weight and the adjusted second fusion weight to obtain the adjusted first fusion weight and the adjusted second fusion weight again.

[0170] It can be understood that here it is still determined whether the image frame for which the connected component performs fusion is a long frame. For the case of a long frame, it may be necessary to perform some smoothing processing on the edge. Here, the maximum value of the pixel values of each channel of each image block in the area corresponding to the connected component in the first image frame is determined. For the case where the maximum value is greater than the preset threshold, the adjusted first fusion weight and the adjusted second fusion weight are adjusted. That is to say, for the case where the maximum value is greater than the preset threshold, it indicates that there is a problem of unevenness at the edge of the connected component. Therefore, at this time, the adjusted first fusion weight and the adjusted second fusion weight are adjusted again.

[0171] Here, mainly the first fusion weight and the second fusion weight corresponding to the image block whose maximum value is greater than the preset threshold are adjusted.

[0172] In this way, by screening the image blocks in the case where the image frame fused by the above connected component is a long frame, and then adjusting the adjusted first fusion weight and the adjusted second fusion weight of the screened image blocks again, the smoothness of the boundary of the connected component can be further improved.

[0173] In order to realize the re-adjustment of the adjusted first fusion weight and the adjusted second fusion weight, in an alternative embodiment, when the maximum value is greater than the preset threshold, adjusting the adjusted first fusion weight and the adjusted second fusion weight to obtain the adjusted first fusion weight and the adjusted second fusion weight again may include:

[0174] When the maximum value is greater than the preset threshold, determine an adjustment coefficient according to the maximum value and a preset constant;

[0175] According to the adjustment coefficient, adjust the adjusted second fusion weight to obtain the adjusted second fusion weight again;

[0176] Subtract the adjusted second fusion weight from one to obtain the adjusted first fusion weight again.

[0177] Understandably, when the maximum value is greater than the preset threshold, it indicates that the adjusted first fusion weight and the adjusted second fusion weight of the image block still need to be adjusted again to improve the smoothness of the image block. Here, the adjustment coefficient can be determined according to the maximum value and the preset constant. Here, the adjustment coefficient can be determined from the corresponding relationship according to the preset corresponding relationship between the pixel value, the preset constant, and the adjustment coefficient. The adjustment coefficient can also be obtained according to the preset formula of the adjustment coefficient. Here, the embodiments of the present application do not make specific limitations on this.

[0178] After obtaining the adjustment coefficient, use the adjustment coefficient to adjust the adjusted second fusion weight corresponding to the image block corresponding to the maximum value, and re-obtain the adjusted second fusion weight corresponding to the image block corresponding to the maximum value. Subtract the adjusted second fusion weight corresponding to the image block corresponding to the maximum value from one to re-obtain the adjusted first fusion weight corresponding to the image block corresponding to the maximum value.

[0179] In this way, by determining the adjustment coefficient as described above and then using the adjustment coefficient to adjust the adjusted fusion weight of the image block corresponding to the maximum value again, the smoothness of the edge image of the moving object in the output image can be further improved.

[0180] In addition, in order to determine a more accurate flag bit for the connected domain, in an optional embodiment, the above method may further include:

[0181] Perform image recognition on the first image frame and the second image frame respectively to obtain the recognition result;

[0182] Adjust the flag bit of the connected domain according to the recognition result.

[0183] Understandably, here, image recognition can be performed on the first image frame and the second image frame, and the recognition result can be obtained. The recognition result can indicate the moving object. For example, the moving object is a person or a pet. In this way, the electronic device can adjust the flag bit of the connected domain according to the moving object indicated in the recognition result. For example, adjust the flag bit of the connected domain according to the type of the moving object.

[0184] In this way, by adjusting the flag bit of the connected domain according to the recognition result of the image recognition, the obtained flag bit of the connected domain is more accurate, which helps to improve the image quality of the output image.

[0185] The following gives an example to describe the image processing method described in the above one or more embodiments.

[0186] Figure 3 It is a schematic flowchart of an example of an optional image processing method provided by the embodiments of the present application, as Figure 3As shown, the method may include:

[0187] S301: Process the obtained long-frame image and short-frame image to obtain a mask image;

[0188] Specifically, use a trained neural network to take the long-frame image and short-frame image as inputs and input them into the trained neural network to obtain a mask image marked with the motion area, denoted as motion mask.

[0189] S302: Downsample the long-frame image and short-frame image and then determine the first ghost image;

[0190] Among them, downsample the long-frame image and short-frame image by 8 times respectively, and then calculate the difference between the two images for each channel. Here, taking one channel as an example, the pixel value of the long-frame image at the (i, j) position of a certain channel is x_l, and the pixel value of the short-frame image is x_s. The frame difference between the two can be obtained using the above formula (2); then calculate two penalty terms, namely the brightness penalty term, which can be obtained using the above formula (1), and also calculate the penalty term for noise, which can be obtained using the above formulas (3), (4), and (5).

[0191] Finally, calculate the ghost value of this point for this channel, which can be obtained using the above formula (6). After obtaining the ghost values of each channel, take the maximum value of each channel as the ghost value of this point, denoted as m_gs.

[0192] S303: Process the mask image to obtain connected components;

[0193] Specifically, preprocess the motion mask, first perform dilation, then use the difference diff between the long-frame image and short-frame image as a guidance map for guided filtering, and then cluster the motion mask to cluster the motion areas into individual connected components.

[0194] S304: Determine the image frames fused by each connected component;

[0195] Among them, traverse each connected component, count the occlusion situation of each pixel point in each connected component. If the maximum value of each channel of the long-frame image at this point is greater than th_ovexp (equivalent to the above preset pixel threshold), and the value of m_gs is less than a threshold th_bad (equivalent to the above preset ghost threshold), then this image block is an overexposed and unoccluded area of the long-frame image. Count the size of such areas. If the proportion it occupies is less than a certain threshold, then this area fuses the long-frame image, and the rest fuses the short-frame image. Use a map to record whether each connected component fuses the long-frame image or the short-frame image. If it fuses the long-frame image, the value in this connected component is 1. If it fuses the short-frame image, it is recorded as 0. This map is denoted as baseTag.

[0196] S305: Determine the second ghost image and the third ghost image after downsampling the long-frame image and the short-frame image;

[0197] Among them, perform two-fold downsampling on the long-frame image and the short-frame image, and then calculate the loose second ghost image. The calculation method is the same as S302. Figure 4a This is a schematic diagram of an optional loose ghost image mask provided by an embodiment of the present application. As Figure 4a shown, the obtained second ghost image is denoted as m_loose. At the same time, calculate a strict third ghost image. Figure 4b This is a schematic diagram of an optional strict ghost image mask provided by an embodiment of the present application. As Figure 4b shown, the strict third ghost image is denoted as m_tight, which is also the same as S302, except that both its Warp and Coof are fixed to a constant, and different sigmas are used to calculate the second ghost image and the third ghost image.

[0198] S306: Obtain the original fusion weights of the long-frame image and the original fusion weights of the short-frame image;

[0199] Specifically, obtain the fusion weights of the long-frame image and the short-frame image. Denote the original fusion weight of the long-frame image as w_l and the original fusion weight of the short-frame image as w_s.

[0200] S307: Adjust the fusion weights based on baseTag, m_loose, and m_tight;

[0201] Among them, perform ghost removal on the original fusion weights. The specific operation is as follows:

[0202] a) If this point is not a motion area, keep it unchanged;

[0203] b) If this is a motion area and baseTag selects the short-frame image here, obtain the fusion weights in the following way:

[0204] w_l = min(w_l, g_tight) (7)

[0205] w_s = 1 - w_l (8)

[0206] c) If this is a motion area and baseTag selects the long-frame image here, obtain the fusion weights in the following way:

[0207] w_s = min(w_s, g_loos) (9)

[0208] w_l = 1 - w_s (10)

[0209] S308: Smooth the fusion weights of the long-frame image and the fusion weights of the short-frame image.

[0210] Specifically, since the ghost image calculated on the downsampled image is sometimes not smooth enough, some smoothing processing needs to be done on the edges. Specifically, on the original size: If this is a motion area and the long frame to be fused is selected: The maximum value of each channel at this position in the long frame is denoted as mac_c;

[0211] If max_c > 0.75, then, adjust the fusion weights of the long-frame image and the fusion weights of the short-frame image using the following formula:

[0212] scale = 1 - (max_c - 0.75) * (max_c - 0.75) / ((max_c - 0.75) * (max_c - 0.75) + sigma * sigma) (11)

[0213] w_s = w_s * scale (12)

[0214] w_l = 1 - w_s (13)

[0215] In addition, some semantic information in the long-frame image and the short-frame image can be combined, such as a segmentation information, to further adjust the ghost image or the baseTag.

[0216] This example proposes a multi-scale ghost removal method based on the foreground-background relationship. First, use the segmentation dataset to offset the foreground of the object to construct a ghost dataset. Use the ghost dataset to train a network that can output the motion area between two frames. Calculate the frame difference between the long-frame image and the short-frame image on the downsampled small image. Then, calculate the strictly ghost-removed ghost image and the loosely ghost-removed ghost image through the frame difference on the downsampled large image. Determine the foreground-background relationship through the frame difference, the noise penalty term, and a preset first constant. Select different ghost images for the fusion weights in the connected regions of each motion area according to the foreground-background relationship, and adjust the fusion edges in the weighted fusion stage to eliminate phenomena such as jaggedness.

[0217] This example can more accurately fuse the frame to be fused by selecting the used ghost image for foreground-background judgment at a small scale, ensuring the picture quality while ensuring the dynamics. Finally, only perform edge refinement on the large image, ensuring the smoothness of the edge while ensuring the performance.

[0218] The embodiment of the present application provides an image processing method. By obtaining two image frames and a mask image, a first ghost image is determined according to the two image frames. Furthermore, based on the first ghost image, the flag bits of the connected components in the motion region of the mask image are determined, so that it can be known which image frames the connected components are fused with, making the numerical value of the ghost corresponding to the image frames fused by the connected components related to the two image frames. Based on this, the two image frames are fused based on different flag bits, so that different image frames can be fused for different connected components, avoiding completely fusing a certain frame in multi-frame fusion, but fusing the corresponding image frames based on the flag bits of different connected components, which can improve the ghost phenomenon in multi-frame fusion, thereby improving the quality of the image obtained by multi-frame fusion.

[0219] Based on the same inventive concept as the foregoing embodiment, the embodiment of the present application provides an image processing device. Figure 5 As a schematic structural diagram of an optional image processing device provided by the embodiment of the present application, as Figure 5 shown, the image processing device includes: an acquisition module 51, a first determination module 52, a second determination module 53, and a processing module 54; wherein,

[0220] The acquisition module 51 is configured to acquire a first image frame, a second image frame, and a mask image; wherein, the mask image is used to identify the motion region; the motion region is the region where the object image position moves between the first image frame and the second image frame;

[0221] The first determination module 52 is configured to determine a first ghost image according to the first image frame and the second image frame; wherein, the ghost image is used to characterize the numerical value corresponding to the ghost generated during the fusion between the first image frame and the second image frame;

[0222] The second determination module 53 is configured to determine the flag bits of the connected components in the motion region of the mask image according to the first ghost image; wherein, the flag bits are used to indicate the image frames fused by the connected components;

[0223] The processing module 54 is configured to fuse the first image frame and the second image frame according to the flag bits of the connected components to obtain an output image.

[0224] In an optional embodiment, when the first determination module 52 determines the first ghost image according to the first image frame and the second image frame, it includes: determining a difference image according to the first image frame and the second image frame; determining the image corresponding to the brightness penalty term of the first image frame; determining the image corresponding to the noise penalty term of the first image frame; and determining the first ghost image according to the difference image, the image corresponding to the brightness penalty term, and the image corresponding to the noise penalty term.

[0225] In an alternative embodiment, the first determination module 52 determines an image corresponding to the noise penalty term of the first image frame, including: obtaining the noise parameter of the camera that captures the first image frame and the second image frame; determining the image corresponding to the noise penalty term according to the noise parameter, the pixel values of each image block in the first image frame, and the sensitivity of the obtained camera.

[0226] In an alternative embodiment, the first determination module 52 determines the first ghost image according to the difference image, the image corresponding to the luminance penalty term, and the image corresponding to the noise penalty term, including: calling a Wiener filtering algorithm according to the difference image, the image corresponding to the luminance penalty term, and the image corresponding to the noise penalty term to determine the ghost image corresponding to each channel; selecting the maximum value of the ghost values in the ghost image corresponding to each channel to obtain the first ghost image.

[0227] In an alternative embodiment, the first determination module 52 determines the flag bit of the connected component of the moving region in the mask image according to the first ghost image, including: determining the flag bit of the connected component according to whether the first ghost image and the first image frame meet a preset condition.

[0228] In an alternative embodiment, the device is further configured to: before determining the first ghost image according to the first image frame and the second image frame, downsample the first image frame and the second image frame according to a preset first multiple to obtain the first image frame and the second image frame again.

[0229] In an alternative embodiment, the first determination module 52 determines the flag bit of the connected component according to whether the first ghost image and the first image frame meet a preset condition, including: determining the ghost value of the connected component according to the first ghost image; determining the pixel value of the connected component according to the first image frame; when the ghost value of the connected component and the pixel value of the connected component meet the preset condition, determining that the flag bit of the connected component is a first value; wherein, the first value indicates that the image frame fused by the connected component is the first image frame; when the ghost value of the connected component and the pixel value of the connected component do not meet the preset condition, determining that the flag bit of the connected component is a second value; wherein, the second value indicates that the image frame fused by the connected component is the second image frame.

[0230] In an alternative embodiment, the device is further configured to: when the ratio is less than a preset ratio threshold, determine that the ghost value of the connected component and the pixel value of the connected component meet the preset condition; wherein, the ratio is the ratio of the area where the ghost value of the connected component is less than a preset ghost threshold and the maximum value of the pixel values of each channel of the connected component is greater than a preset pixel threshold to the area of the first image frame.

[0231] In an alternative embodiment, the processing module 54 is specifically configured to: adjust the first fusion weight of the first image frame and the second fusion weight of the second image frame according to the flag bit of the connected component to obtain the adjusted first fusion weight and the adjusted second fusion weight; and process the first image frame and the second image frame according to the adjusted first fusion weight and the adjusted second fusion weight to obtain an output image.

[0232] In an alternative embodiment, when the processing module 54 adjusts the first fusion weight of the first image frame and the second fusion weight of the second image frame according to the flag bit of the connected component to obtain the adjusted first fusion weight and the adjusted second fusion weight, it may include: determining a second ghost image according to the first image frame and the second image frame; determining a third ghost image according to the first image frame, the second image frame, and the image corresponding to the preset penalty term; selecting a target ghost image from the second ghost image and the third ghost image according to the flag bit of the connected component; and adjusting the first fusion weight of the first image frame and the second fusion weight of the second image frame by using the ghost value in the target ghost image to obtain the adjusted first fusion weight and the adjusted second fusion weight.

[0233] In an alternative embodiment, the apparatus is further configured to: downsample the first image frame and the second image frame by a preset second multiple to obtain the first image frame and the second image frame again before determining the second ghost image according to the first image frame and the second image frame and determining the third ghost image according to the first image frame, the second image frame, the image corresponding to the preset penalty term, and the preset constant.

[0234] In an alternative embodiment, when the processing module 54 selects a target ghost image from the second ghost image and the third ghost image according to the flag bit of the connected component, it includes: when the flag bit of the connected component indicates that the image frame fused by the connected component is the first image frame, determining the second ghost image as the target ghost image; and when the flag bit of the connected component indicates that the image frame fused by the connected component is the second image frame, determining the third ghost image as the target ghost image.

[0235] In an alternative embodiment, the processing module 54 adjusts the first fusion weight of the first image frame and the second fusion weight of the second image frame by using the ghost value in the target ghost image to obtain the adjusted first fusion weight and the adjusted second fusion weight, including: when the flag bit of the connected component indicates that the image frame fused by the connected component is the first image frame, determining the minimum value between the second fusion weight and the ghost value in the target ghost image as the adjusted second fusion weight, and determining the difference obtained by subtracting the adjusted second fusion weight from 1 as the adjusted first fusion weight; when the flag bit of the connected component indicates that the image frame fused by the connected component is the second image frame, determining the minimum value between the first fusion weight and the ghost value in the target ghost image as the adjusted first fusion weight, and determining the difference obtained by subtracting the adjusted first fusion weight from 1 as the adjusted second fusion weight.

[0236] In an alternative embodiment, the apparatus is further configured to: when the resolution of the target ghost image is different from that of the first image frame, perform interpolation processing on the target ghost image to obtain a new target ghost image so that the resolution of the target ghost image is the same as that of the first image frame.

[0237] In an alternative embodiment, the apparatus is further configured to: when the flag bit of the connected component indicates that the image frame fused by the connected component is the first image frame, determine the maximum value of the pixel values of each channel of each image block in the region corresponding to the connected component in the first image frame; when the maximum value is greater than a preset threshold, adjust the adjusted first fusion weight and the adjusted second fusion weight to obtain the adjusted first fusion weight and the adjusted second fusion weight again.

[0238] In an alternative embodiment, when the maximum value is greater than the preset threshold, the apparatus adjusts the adjusted first fusion weight and the adjusted second fusion weight to obtain the adjusted first fusion weight and the adjusted second fusion weight again, including: when the maximum value is greater than the preset threshold, determining an adjustment coefficient according to the maximum value and a preset constant; adjusting the adjusted second fusion weight according to the adjustment coefficient to obtain the adjusted second fusion weight again; subtracting the adjusted second fusion weight from 1 to obtain the adjusted first fusion weight again.

[0239] In an alternative embodiment, the apparatus is further configured to: perform image recognition on the first image frame and the second image frame respectively to obtain recognition results; adjust the flag bit of the connected component according to the recognition results.

[0240] In an alternative embodiment, the exposure duration of the first image frame is greater than that of the second image frame.

[0241] In practical applications, the above-mentioned acquisition module 51, first determination module 52, second determination module 53, and processing module 54 can be implemented by a processor located on an image processing device, specifically implemented by a CPU, a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0242] Figure 6 FIG. is a schematic structural diagram of an optional electronic device provided by an embodiment of the present application. As Figure 6 shown, an embodiment of the present application provides an electronic device 600, including:

[0243] A processor 61 and a storage medium 62 storing instructions executable by the processor; the storage medium 62 depends on the processor 61 to execute operations through a communication bus 63. When the instructions are executed by the processor, the image processing method executed on the processor side in the above one or more embodiments is executed.

[0244] It should be noted that in practical applications, each component in the computer device is coupled together through the communication bus 63. It can be understood that the communication bus 63 is used to realize the connection and communication between these components. The communication bus 63 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 6 all kinds of buses are labeled as the communication bus 63.

[0245] An embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the image processing method in the above one or more embodiments.

[0246] Among them, the computer-readable storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.

[0247] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) that contain computer-usable program code.

[0248] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0249] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device realizes the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0250] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or more processes in the flowchart and / or one block or more blocks in the block diagram.

[0251] As described above, it is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for processing an image, characterized in that: include: Acquire a first image frame, a second image frame and a mask image; wherein the mask image is used to identify a motion region; the motion region is a region where an object image position moves between the first image frame and the second image frame; Determine a first ghost image according to the first image frame and the second image frame; wherein the ghost image is used to represent a value corresponding to a ghost generated when the first image frame and the second image frame are fused; Determine, according to the first ghost image, a flag bit of a connected domain of the motion region in the mask image; wherein the flag bit is used to indicate an image frame fused by the connected domain; The first image frame and the second image frame are fused according to the flag bit of the connected domain to obtain an output image.

2. The method according to claim 1, characterized in that: The step of determining the first ghost image according to the first image frame and the second image frame includes: Determine a difference image according to the first image frame and the second image frame; Determining an image corresponding to the brightness penalty item of the first image frame; Determining an image corresponding to the noise penalty item of the first image frame; The first ghost image is determined according to the difference image, the image corresponding to the brightness penalty item, and the image corresponding to the noise penalty item.

3. The method according to claim 2, characterized in that The determining an image corresponding to the noise penalty item of the first image frame includes: Obtaining noise parameters of a camera that captures the first image frame and the second image frame; An image corresponding to the noise penalty item is determined according to the noise parameter, the pixel value of each image block in the first image frame, and the acquired sensitivity of the camera.

4. The method according to claim 2, characterized in that: The determining the first ghost image according to the difference image, the image corresponding to the brightness penalty item, and the image corresponding to the noise penalty item includes: According to the difference image, the image corresponding to the brightness penalty item and the image corresponding to the noise penalty item, calling the Wiener filtering algorithm to determine the ghost image corresponding to each channel; The maximum value of the ghost value in the ghost image corresponding to each channel is selected to obtain the first ghost image.

5. The method according to claim 1, characterized in that: The step of determining, according to the first ghost image, a flag bit of a connected domain of the motion region in the mask image includes: The flag bit of the connected domain is determined according to whether the first ghost image and the first image frame meet a preset condition.

6. The method according to claim 5, characterized in that The method further comprises: Before determining the first ghost image according to the first image frame and the second image frame, the first image frame and the second image frame are downsampled according to a preset first multiple to obtain the first image frame and the second image frame again.

7. The method according to claim 5, characterized in that The step of determining the flag bit of the connected domain according to whether the first ghost image and the first image frame meet a preset condition includes: Determining a ghost value of the connected domain according to the first ghost image; Determining pixel values ​​of the connected domain according to the first image frame; When the ghost value of the connected domain and the pixel value of the connected domain meet the preset condition, determining that the flag bit of the connected domain is a first value; wherein the first value indicates that the image frame fused by the connected domain is the first image frame; When the ghost value of the connected domain and the pixel value of the connected domain do not satisfy the preset condition, the flag bit of the connected domain is determined to be a second value; wherein the second value indicates that the image frame fused by the connected domain is the second image frame.

8. The method according to claim 7, characterized in that The method further comprises: When the proportion is less than a preset proportion threshold, determining that the ghost value of the connected domain and the pixel value of the connected domain meet the preset condition; The proportion is: the proportion of the area of ​​the first image frame occupied by an area in which the ghost value of the connected domain is less than a preset ghost threshold and the maximum value of the pixel values ​​of each channel of the connected domain is greater than the preset pixel threshold.

9. The method according to claim 1, characterized in that: The step of fusing the first image frame and the second image frame according to the flag bit of the connected domain to obtain an output image includes: According to the flag bit of the connected domain, adjusting the first fusion weight of the first image frame and the second fusion weight of the second image frame to obtain an adjusted first fusion weight and an adjusted second fusion weight; The first image frame and the second image frame are processed according to the adjusted first fusion weight and the adjusted second fusion weight to obtain the output image.

10. The method according to claim 9, characterized in that The step of adjusting the first fusion weight of the first image frame and the second fusion weight of the second image frame according to the flag bit of the connected domain to obtain the adjusted first fusion weight and the adjusted second fusion weight includes: determining a second ghost image according to the first image frame and the second image frame; Determining a third ghost image according to the first image frame, the second image frame, and an image corresponding to a preset penalty item; Selecting a target ghost image from the second ghost image and the third ghost image according to the flag bit of the connected domain; The first fusion weight of the first image frame and the second fusion weight of the second image frame are adjusted by using the ghost value in the target ghost image to obtain the adjusted first fusion weight and the adjusted second fusion weight.

11. The method according to claim 10, characterized in that The method further comprises: Before determining the second ghost image based on the first image frame and the second image frame, and determining the third ghost image based on the first image frame, the second image frame, an image corresponding to a preset penalty item and a preset constant, the first image frame and the second image frame are downsampled by a preset second multiple to regain the first image frame and the second image frame.

12. The method according to claim 10, characterized in that The selecting a target ghost image from the second ghost image and the third ghost image according to the flag bit of the connected domain comprises: When the flag bit of the connected domain indicates that the image frame fused by the connected domain is the first image frame, determining the second ghost image as the target ghost image; When the flag bit of the connected domain indicates that the image frame fused by the connected domain is the second image frame, the third ghost image is determined as the target ghost image.

13. The method according to claim 10, characterized in that The step of adjusting the first fusion weight of the first image frame and the second fusion weight of the second image frame by using the ghost value in the target ghost image to obtain the adjusted first fusion weight and the adjusted second fusion weight includes: In the case where the flag bit of the connected domain indicates that the image frame fused by the connected domain is the first image frame, determining the minimum value of the second fusion weight and the ghost value of the target ghost image as the adjusted second fusion weight, and determining the difference between one and the adjusted second fusion weight as the adjusted first fusion weight; When the flag bit of the connected domain indicates that the image frame fused by the connected domain is the second image frame, the minimum value of the first fusion weight and the ghost value of the target ghost image is determined as the adjusted first fusion weight, and the difference between one and the adjusted first fusion weight is determined as the adjusted second fusion weight.

14. The method according to claim 13, characterized in that The method further comprises: When the resolution of the target ghost image is different from the resolution of the first image frame, interpolation processing is performed on the target ghost image to obtain the target ghost image again, so that the resolution of the target ghost image is the same as that of the first image frame.

15. The method according to claim 9, characterized in that The method further comprises: When the flag bit of the connected domain indicates that the image frame fused by the connected domain is the first image frame, determining the maximum value of the pixel values ​​of each channel of each image block in the area corresponding to the connected domain in the first image frame; When the maximum value is greater than a preset threshold, the adjusted first fusion weight and the adjusted second fusion weight are adjusted to obtain the adjusted first fusion weight and the adjusted second fusion weight again.

16. The method according to claim 15, characterized in that When the maximum value is greater than a preset threshold, adjusting the adjusted first fusion weight and the adjusted second fusion weight to obtain the adjusted first fusion weight and the adjusted second fusion weight again includes: When the maximum value is greater than a preset threshold, determining the adjustment coefficient according to the maximum value and a preset constant; Adjusting the adjusted second fusion weight according to the adjustment coefficient to obtain the adjusted second fusion weight again; Subtract the adjusted second fusion weight from one to obtain the adjusted first fusion weight.

17. The method according to any one of claims 1 to 16, characterized in that The method further comprises: Performing image recognition on the first image frame and the second image frame respectively to obtain recognition results; According to the recognition result, the flag bit of the connected domain is adjusted.

18. The method according to any one of claims 1 to 16, characterized in that The exposure time length of the first image frame is greater than the exposure time length of the second image frame.

19. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image frame, a second image frame and a mask image; wherein the mask image is used for identifying a motion region; the motion region is a region where an object image position moves between the first image frame and the second image frame; A first determination module, configured to determine a first ghost image according to the first image frame and the second image frame; wherein the ghost image is used to represent a value corresponding to a ghost generated when the first image frame and the second image frame are fused; A second determination module is used to determine a flag bit of a connected domain of the motion region in the mask image according to the first ghost image; wherein the flag bit is used to indicate an image frame fused by the connected domain; A processing module is used to fuse the first image frame and the second image frame according to the flag bit of the connected domain to obtain an output image.

20. An electronic device, characterized in that: include: A processor and a storage medium storing instructions executable by the processor; The storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the image processing method described in any one of claims 1 to 18 is executed.

21. A computer storage medium, characterized in that Executable instructions are stored, and when the executable instructions are executed by one or more processors, the processors execute the image processing method described in any one of claims 1 to 18.

Images