Image processing method and device and storage medium

Abstract

The embodiment of the invention provides an image processing method and device and a storage medium, and the method comprises the steps: carrying out the local motion estimation of a short frame sequence, and determining pixel points of a motion region in a plurality of first images; comparing the brightness value of each pixel point in the plurality of first images with the brightness value of each pixel point in a reference frame image in the long frame sequence, and determining the pixel points of which the brightness difference values are greater than a brightness threshold value in the plurality of first images as the pixel points of the exposure stripes in the plurality of first images; compensating the pixel values of the pixel points of the exposure stripes in the plurality of first images based on the pixel values of the pixel points in the plurality of second images to obtain a compensated short frame sequence; and carrying out image fusion on the long frame sequence and the compensated short frame sequence to generate a fused image. In the embodiment of the invention, the electronic equipment uses the long frame sequence to compensate the exposure stripes in the short frame sequence, so that the imaging of the moving object in the fused image is clear, the exposure stripes are avoided, and the display effect of the image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method, device and storage medium. BACKGROUND

[0002] With the development of electronic technology, electronic devices can support multiple shooting modes. In the case that a user selects a motion mode, the electronic device can capture a moving object based on a short exposure time to capture a clear image.

[0003] An alternating current driven light source is affected by the frequency of alternating current, and the light emitted by the alternating current driven light source will flicker at a fixed frequency. Since the frequency is high, the human eye is usually difficult to perceive. However, in a shooting environment where there is an alternating current driven light source, when a motion mode is used to shoot at a short exposure time, the frequency flicker of the alternating current driven light source can be captured, so that exposure stripes with different degrees of brightness exist on the finally generated image, and the display effect of the image is reduced. SUMMARY

[0004] The image processing method, device and storage medium provided by the present application aim to remove exposure stripes in an image and improve the display effect of the image.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, an image processing method is provided, including: in response to a shooting instruction, acquiring a short frame sequence and a long frame sequence; the short frame sequence includes a plurality of first images, and the long frame sequence includes a plurality of second images; the exposure amount of the first image is less than that of the second image; performing local motion estimation on the short frame sequence to determine the pixel points of the motion region in the plurality of first images; comparing the brightness values of the pixel points in the plurality of first images with the brightness values of the pixel points in a reference frame image in the long frame sequence, and determining the pixel points in the plurality of first images with a brightness difference greater than a brightness threshold value as the pixel points of exposure stripes in the plurality of first images; based on the overlapping state of the pixel points of the motion region in the plurality of first images and the pixel points of the exposure stripes in the plurality of first images, compensating the pixel values of the pixel points of the exposure stripes in the plurality of first images based on the pixel values of the pixel points in the plurality of second images to obtain a compensated short frame sequence; and performing image fusion on the long frame sequence and the compensated short frame sequence to generate a fused image.

[0007] In this embodiment, the electronic device acquires long frame sequences and short frame sequences with different exposure levels. The first image in the short frame sequence corresponds to a lower exposure level, allowing moving objects to be clearly imaged. The second image in the long frame sequence contains more image detail information. Based on this, after determining the motion area and the pixels in the exposure stripes, the electronic device can use the long frame sequence to compensate for the exposure stripes in the short frame sequence, obtaining a compensated short frame sequence. The long frame sequence and the compensated short frame sequence are then fused, so that the moving objects in the fused image are clearly imaged while avoiding the appearance of exposure stripes, thus improving the image display effect.

[0008] In one possible implementation, based on the overlap between pixels of moving regions in multiple first images and pixels of exposure stripes in multiple first images, the pixel values ​​of the exposure stripes in multiple second images are compensated to obtain a compensated short frame sequence. This includes: determining the type of exposure stripes in the short frame sequence, where the types of exposure stripes include global exposure stripes and local exposure stripes; and compensating the pixel values ​​of the exposure stripes in multiple first images based on the type of exposure stripes and the overlap between pixels of moving regions in multiple first images and pixels of exposure stripes in multiple first images, thereby obtaining the compensated short frame sequence. In this embodiment, compensating for exposure stripes by combining the type of exposure stripes improves the compensation effect.

[0009] In one possible implementation, determining the type of exposure stripes in a short frame sequence includes: determining a first image from the short frame sequence as a reference frame image; accumulating the pixel values ​​of each row of the reference frame image to obtain exposure stripe statistics of the short frame sequence; and determining the type of exposure stripes in the short frame sequence based on the exposure stripe statistics.

[0010] In one possible implementation, the exposure stripes are of the type of local exposure stripes. Based on the type of exposure stripes and the overlap between pixels in the moving regions of multiple first images and pixels in the exposure stripes of multiple first images, the pixel values ​​of the pixels in the exposure stripes of multiple first images are compensated based on the pixel values ​​of each pixel in multiple second images to obtain a compensated short frame sequence. This includes: when pixels in the moving regions of multiple first images do not overlap with pixels in the exposure stripes of multiple first images, determining the intensity of the local exposure stripes based on the exposure stripe statistics of the short frame sequence; when the intensity of the local exposure stripes is less than an intensity threshold, compensating the pixel values ​​of the pixels in the exposure stripes of multiple first images based on the pixel values ​​of each pixel in multiple second images to obtain a compensated short frame sequence. In this embodiment, when the intensity of the local exposure stripes is less than the intensity threshold, it indicates that some image information is retained in the exposure stripes, and compensation can be performed based on the second images to improve the display effect of the fused image.

[0011] In one possible implementation, when the pixels of the moving region in multiple first images do not overlap with the pixels of the exposure stripes in multiple first images, after determining the intensity of the local exposure stripes based on the exposure stripe statistics of the short frame sequence, the method further includes: when the intensity of the local exposure stripes is greater than or equal to an intensity threshold, replacing the pixels of the exposure stripes in multiple first images with pixels at corresponding positions in multiple second images to obtain a compensated short frame sequence.

[0012] In this embodiment of the application, when the intensity of a local exposure stripe is greater than or equal to an intensity threshold, it indicates that the image information of the exposure stripe portion is difficult to recover. In order to improve the display effect of the fused image, the electronic device can replace the pixels of the exposure stripes in multiple first images with the corresponding pixels in multiple second images to obtain a compensated short frame sequence.

[0013] In one possible implementation, the exposure stripe type is a global exposure stripe. Based on the type of exposure stripe and the overlap state between the pixels of the moving region in multiple first images and the pixels of the exposure stripe in multiple first images, the pixel values ​​of the pixels of the exposure stripe in multiple first images are compensated based on the pixel values ​​of each pixel in multiple second images to obtain a compensated short frame sequence. This includes: determining the intensity of the global exposure stripe based on the exposure stripe statistics of the short frame sequence; when the intensity of the global exposure stripe is less than an intensity threshold, compensating the pixel values ​​of the pixels of the exposure stripe in multiple first images based on the pixel values ​​of each pixel in multiple second images to obtain a compensated short frame sequence.

[0014] In one possible implementation, determining the type of exposure stripes in a short frame sequence based on exposure stripe statistics includes: when the entire exposure stripe statistics image presents alternating bright and dark sinusoidal wave patterns, the type of exposure stripes in the short frame sequence is determined to be global exposure stripes; when the exposure stripe statistics image locally presents alternating bright and dark sinusoidal wave patterns, the type of exposure stripes in the short frame sequence is determined to be local exposure stripes. In this embodiment, the type of exposure stripes can be intuitively determined in the spatial domain based on the sinusoidal wave patterns presented in the entire exposure stripe statistics image.

[0015] In one possible implementation, determining the type of exposure stripes in a short frame sequence based on exposure stripe statistics includes: performing a Fourier transform on the exposure stripe statistics to convert them from the spatial domain to the frequency domain, obtaining frequency domain exposure stripe statistics; within a specified frequency band, if there are frequency points in the frequency domain exposure stripe statistics with frequency values ​​higher than a frequency threshold, the type of exposure stripes in the short frame sequence is determined to be local exposure stripes; otherwise, the type of exposure stripes in the short frame sequence is determined to be global exposure stripes. To avoid image content affecting the accuracy of exposure stripe type determination, in this embodiment, the exposure stripe statistics are converted from the spatial domain to the frequency domain, and the type of exposure stripes is determined in the frequency domain, thus improving the accuracy of exposure stripe type determination.

[0016] In one possible implementation, motion estimation is performed on a short frame sequence to determine pixels in moving regions of multiple first images. This includes: performing local motion estimation on the short frame sequence to generate motion vector information; and determining pixels in moving regions of multiple first images based on the relationship between the motion vector information and an offset threshold. In this embodiment, determining pixels in moving regions of multiple first images by performing motion estimation on a short frame sequence facilitates determining whether the moving regions coincide with exposure stripes during subsequent compensation, thereby improving the compensation effect.

[0017] Second aspect: This application provides an electronic device, which includes a processor and a memory;

[0018] The memory is used to store program code and transmit the program code to the processor;

[0019] The processor is used to execute the steps of an image processing method as described above, according to the instructions in the program code.

[0020] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of an image processing method as described above. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of a partial exposure stripe pattern;

[0022] Figure 2 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of exposure stripe statistics provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a global exposure stripe pattern;

[0025] Figure 5 A schematic diagram illustrating the overlap of exposure stripes and a moving area, provided as an embodiment of this application;

[0026] Figure 6 An example diagram illustrating the composition of an electronic device provided in an embodiment of this application;

[0027] Figure 7 This is an example diagram of the software structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0029] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0030] To facilitate understanding, the technical terms used in the embodiments of this application will be introduced first.

[0031] Flicker: The energy transmitted in an AC power grid varies periodically with a fixed frequency, which is called the power frequency. The energy variation caused by the power frequency is called flicker.

[0032] Exposure time: refers to the time interval from when the shutter opens to when it closes. A longer exposure time results in more light reaching the sensor of an electronic device; a shorter exposure time results in less light reaching the sensor. In practical applications, electronic devices can reduce the exposure time to capture moving objects, ensuring a clear image of the moving subject.

[0033] Banding phenomenon: When an electronic device's image sensor uses a line-by-line exposure method, each row of pixels has the same exposure time, but the start and end times of exposure for different rows are different. This results in varying cumulative light energy received by pixels in different rows within the same exposure time. When the cumulative light energy received by pixels in different rows within the same exposure time is different, bright and dark stripes appear on the generated image. This phenomenon is called banding, often shortened to banding, and these bright and dark stripes can be referred to as exposure stripes.

[0034] Currently, electronic devices can eliminate banding by using exposure times that are integer multiples of the strobe period. For example, when the power frequency of the light source is 50Hz and the strobe period is 10ms, the electronic device can avoid banding by adjusting the exposure time to an integer multiple of 10ms when taking the picture.

[0035] When photographing moving objects, electronic devices need to capture the movement with short exposure times to ensure a sharp image. However, when AC-powered light sources are present in the shooting environment, using short exposure times introduces new challenges.

[0036] like Figure 1 As shown, the image captured by the electronic device with a short exposure time contains a screen display device driven by AC power. Because the exposure time does not match the flicker period of the screen display device, exposure stripes will appear in the image 1101 portion of the screen display device in the generated image 1100, which is the banding phenomenon described above.

[0037] The reason for this is that electronic devices usually cannot simultaneously meet the following two conditions: first, to use a low exposure time so that moving objects can be clearly imaged in the image; and second, to adjust the exposure time to an integer multiple of the flicker period of the AC-driven light source in order to eliminate banding caused by the mismatch between the exposure time and the flicker period of the light source.

[0038] Because electronic devices cannot simultaneously meet both conditions, when shooting moving objects in an environment with an AC-driven light source and a short exposure time, exposure stripes appear in the generated images, reducing the overall image display quality.

[0039] Understandable Figure 1 This paper uses a screen display device as an example of an AC-driven light source for illustration. The AC-driven light source is not specifically limited in the embodiments of this application. For example, the AC-driven light source may also include, but is not limited to, indoor fluorescent lamps, incandescent lamps, etc.

[0040] Based on this, embodiments of this application provide an image processing method, which acquires a short frame sequence and a long frame sequence in response to a shooting command; performs motion estimation on the short frame sequence to determine the pixels of the moving region in multiple first images; compares the brightness values ​​of each pixel in the multiple first images with the brightness values ​​of each pixel in the reference frame image in the long frame sequence to determine the pixels of the exposure stripes in the multiple first images; based on this, according to the overlap state between the pixels of the moving region in the multiple first images and the pixels of the exposure stripes in the multiple first images, compensates the pixel values ​​of the pixels of the exposure stripes in the multiple first images based on the pixel values ​​of each pixel in multiple second images to obtain a compensated short frame sequence; and performs image fusion between the long frame sequence and the compensated short frame sequence to generate a fused image.

[0041] In this embodiment, the electronic device acquires long frame sequences and short frame sequences with different exposure levels. The first image in the short frame sequence corresponds to a lower exposure level, allowing moving objects to be clearly imaged. The second image in the long frame sequence contains more image detail information. Based on this, after determining the motion area and the pixels in the exposure stripes, the electronic device can use the long frame sequence to compensate for the exposure stripes in the short frame sequence, obtaining a compensated short frame sequence. The long frame sequence and the compensated short frame sequence are then fused, so that the moving objects in the fused image are clearly imaged while avoiding the appearance of exposure stripes, thus improving the image display effect.

[0042] The following is combined with Figure 2 The flowchart shown illustrates the image processing method provided in the embodiments of this application.

[0043] S101, The electronic device responds to the user's action of taking a picture and acquires an exposure sequence.

[0044] After a user triggers the photo-taking action, the electronic device can take multiple images with different exposure levels based on its own inherent settings. These multiple images with different exposure levels form an exposure sequence, which includes a long frame sequence and a short frame sequence.

[0045] The short frame sequence includes multiple first images, each with an exposure level less than or equal to a first threshold; the long frame sequence includes multiple second images, each with an exposure level greater than the first threshold.

[0046] Exposure is the total amount of light entering the camera, which can be controlled by a combination of the camera's aperture and shutter speed. The first image has less exposure than the second image and is typically used to capture rapidly changing or brightly lit image details. The second image has more exposure than the first image, containing more detail about the light, and can be used to compensate for exposure banding in short frame sequences.

[0047] For example, in a shooting scenario, a user is shooting in an indoor location that includes moving objects and an AC-powered light source, which includes, but is not limited to, lighting sources, screen display devices, etc.

[0048] Users can access the shooting interface by clicking the camera icon in the initial user interface of the electronic device or by waking it up with voice commands. This shooting interface can also be called the camera preview interface.

[0049] In one possible implementation, the shooting interface provides users with multiple shooting modes, among which the motion mode is suitable for capturing moving objects. In motion mode, the electronic device can shoot moving objects based on a shorter exposure time to reduce or eliminate motion blur and obtain a clear image.

[0050] It is understandable that moving objects can include, but are not limited to, people, animals, vehicles, and robots.

[0051] To ensure clear imaging of moving objects, users select sports mode in the shooting interface. After selecting sports mode, users can take photos by clicking the shutter button or triggering the photo via voice commands. In response to the user's photo-taking action, the electronic device captures multiple images with different exposures, resulting in an exposure sequence including both long and short frame sequences.

[0052] S102. The electronic device performs image alignment on the short frame sequence to obtain the aligned short frame sequence.

[0053] Image alignment is a process of precisely aligning two or more images in space, aiming to accurately match common features or content between the images. In this embodiment, considering that during image capture, movement of the electronic device may cause pixel offsets between different first images in a short frame sequence, the electronic device performs image alignment on the short frame sequence to obtain an aligned short frame sequence. The specific process is as follows:

[0054] Step 11: The electronic device performs image alignment on multiple first images in the short frame sequence to obtain the global alignment matrix of the short frame sequence.

[0055] In this embodiment of the application, the electronic device can perform image alignment on multiple first images based on feature extraction and matching, or on optical flow based on feature point tracking, to obtain a global alignment matrix for a short frame sequence.

[0056] The global alignment matrix includes, but is not limited to, affine transformation matrix or homography transformation matrix, which is used to describe the geometric transformation relationship between adjacent frames and can be used to align each first image in a short frame sequence to the same coordinate system.

[0057] In one embodiment, during the process of obtaining the global alignment matrix using a feature extraction and matching method, the electronic device can detect feature points in each first image based on a feature detection algorithm. These feature points are typically key points with uniqueness and stability in the first image, such as corner points and edge points. Feature detection algorithms may include, but are not limited to, Scale Invariant Feature Transform (SIFT), Speed-Up Robust Feature Transform (SURF), and Oriented Fast and Rotated BRIEF (ORB), which combines an oriented fast algorithm with a rotation-based descriptive feature sub-algorithm.

[0058] After the electronic device detects feature points, it can match the feature points of the first image of adjacent frames in the short frame sequence to obtain matching feature point pairs. Then, it can use algorithms such as least squares and iterative nearest point to calculate the global alignment matrix of the short frame sequence.

[0059] In another embodiment, the electronic device can obtain the global alignment matrix using an optical flow approach based on feature point tracking. Compared to feature extraction and matching-based methods, this approach provides pixel-level motion information with higher accuracy. After detecting feature points, the electronic device can compare the similarity between feature points in adjacent frames using optical flow methods, such as the Lucas-Kanade optical flow method, and estimate the motion of the feature points using optical flow equations to obtain the global alignment matrix of the short frame sequence.

[0060] It is understood that the method for obtaining the global alignment matrix by an electronic device in the embodiments of this application is not specifically limited, and the above is only an example.

[0061] Step 12: Based on the global alignment matrix, the electronic device performs image twisting processing on multiple first images in the short frame sequence to obtain the aligned short frame sequence.

[0062] Image warping is the process of changing the position of pixels in an image. Based on this global alignment matrix, electronic devices perform image warping on multiple first images in a short frame sequence, which can make the multiple first images consistent in spatial position, thus obtaining a short frame sequence including multiple aligned first images.

[0063] S103. The electronic device performs self-statistical analysis of exposure stripes on the short frame sequence to obtain exposure stripe statistical information.

[0064] Multiple first images in a short frame sequence correspond to shorter exposure times, which can avoid or reduce the occurrence of blurring or ghosting of moving objects in the first image. However, exposure stripes may exist in the first image due to the influence of ambient light sources or screen display devices in the shooting scene.

[0065] In this embodiment of the application, since the differences between multiple first images in a short frame sequence are small, in order to improve the efficiency of exposure stripe self-statistics, the electronic device can determine a first image in the short frame sequence as a reference frame image, and perform exposure stripe self-statistics on the reference frame image to obtain exposure stripe statistical information.

[0066] In one possible implementation, since the image sensor of the electronic device uses line-by-line exposure for imaging, the exposure stripes are typically horizontal stripes. Therefore, the process of the electronic device performing exposure stripe self-statistics on a short frame sequence involves the electronic device accumulating the pixel values ​​of each row of the reference frame image to obtain exposure stripe statistics. This exposure stripe statistics include the accumulated values ​​of pixels in each row of the reference frame image.

[0067] like Figure 3 As shown, when the exposure stripe statistics 1400 is represented by a one-dimensional image, the pixel value of each pixel in the exposure stripe statistics 1400 corresponds to the accumulated value of a row of pixels in the reference frame image 1401. In this embodiment, the process by which the electronic device accumulates the pixel values ​​of each row in the reference frame image 1401 to obtain the exposure stripe statistics 1400 can amplify the differences between pixel values ​​in adjacent rows, thereby achieving the purpose of detecting exposure stripes.

[0068] S104. The electronic device determines the type of exposure stripes in a short frame sequence based on exposure stripe statistics.

[0069] In this embodiment of the application, exposure stripe statistics are used to determine whether the type of exposure stripes in a short frame sequence is a global exposure stripe or a local exposure stripe.

[0070] Among them, when the type of exposure stripes is global exposure stripes, such as Figure 4 As shown, exposure stripes cover the entire first image; when the type of exposure stripes is localized exposure stripes, such as... Figure 1 As shown, exposure stripes exist in a localized area of ​​the first image.

[0071] For example, when the light source in the environment is an illumination source, if the exposure time does not match the flicker period of the light source in the environment, global exposure stripes may appear in the first image. For example, when an electronic device is photographed in an environment illuminated by an AC light source, the entire captured image may be affected by the AC light source, resulting in global exposure stripes in the first image. When the light source in the environment is the light emitted by the screen display device in the captured image, if the exposure time does not match the flicker period of the screen display device, local exposure stripes may appear in the first image, that is, exposure stripes may exist in the area where the screen display device is located in the first image.

[0072] In one possible implementation, since the exposure stripe statistics exist in the form of a one-dimensional image, where the pixel value of each pixel in the one-dimensional image is the cumulative value of a row of pixels in the reference frame image, when the exposure stripe statistics show alternating bright and dark sine wave patterns across the entire image, it can be determined that the type of exposure stripes in the short frame sequence is global exposure stripes; when the exposure stripe statistics show alternating bright and dark sine wave patterns locally, it can be determined that the type of exposure stripes in the short frame sequence is local exposure stripes.

[0073] In this embodiment of the application, when the type of exposure stripe is local exposure stripe, the electronic device can determine that there are rows of alternating bright and dark sine wave patterns in the exposure stripe statistics, but it cannot determine the specific location of the exposure stripe in the first image.

[0074] In one possible implementation, the alternating bright and dark sinusoidal ripples caused by exposure stripes may be affected by the image content in a short frame sequence, reducing the accuracy of exposure stripe type determination.

[0075] To avoid the influence of image content in a short frame sequence on sinusoidal ripples, electronic devices can convert exposure fringe statistics from the spatial domain to the frequency domain. Based on the frequency domain exposure fringe statistics, the type of exposure fringes in the short frame sequence can be determined. The steps are as follows:

[0076] Step 21: The electronic device performs a Fourier transform on the exposure stripe statistics in the spatial domain to obtain the exposure stripe statistics in the frequency domain.

[0077] As shown in equation (1):

[0078] Banding_Stat_F=Frequency_Domain_Transform(Banding_Stat) (1)

[0079] Wherein, Banding_Stat_F represents the exposure stripe statistics in the frequency domain; Banding_Stat represents the exposure stripe statistics in the spatial domain; and Frequency_Domain_Transform() represents performing a Fourier transform on the data in parentheses.

[0080] Step 22: The electronic device determines whether there are frequency points in the specified frequency band in the exposure stripe statistics information in the frequency domain.

[0081] The specified frequency range can be represented by [frenqucy_low, fequency_high]. Here, frenqucy_low represents the lowest frequency value in the specified frequency range, and fequency_high represents the highest frequency value in the specified frequency range.

[0082] Since the exposure stripes are generated because the flicker period of the light source in the environment does not match the exposure time of the electronic device, the specified frequency band can be determined based on the frequency of the light source in the environment in this embodiment of the application.

[0083] For example, the frequency of non-AC driven light sources in the shooting environment is usually low; in contrast, the frequency of AC driven lighting sources, such as indoor fluorescent lamps and incandescent lamps, is higher, with a frequency of 50Hz-60Hz, while the frequency of screen display devices can be 100Hz-200Hz.

[0084] In this case, if a frequency point of 50Hz-60Hz is detected in the exposure stripe statistics in the frequency domain, it indicates that there is an AC-driven lighting source in the shooting environment; if a frequency point of 100Hz-200Hz is detected in the exposure stripe statistics in the frequency domain, it indicates that there is a screen display device in the shooting environment.

[0085] Therefore, the lowest frequency value of a specified frequency band can be determined as 50Hz, and the highest frequency value of the specified frequency band can be determined as 200Hz. When the exposure stripe statistics in the frequency domain contain a frequency point within the specified frequency band, it can be determined that exposure stripes exist in the short frame sequence.

[0086] It should be noted that the range of values ​​for the specified frequency band is not specifically limited in this embodiment of the application. The value can be set according to the frequency of the light source in the environment. The frequency values ​​of the light source in the environment mentioned above are just examples.

[0087] In one possible implementation, the electronic device can filter the exposure stripe statistics in the frequency domain based on a first filter (Frequency_Domain_Filter). If there are frequency values ​​in the specified frequency range in the exposure stripe statistics in the frequency domain, the electronic device can obtain the frequency points in the specified frequency range in the exposure stripe statistics by filtering the exposure stripe statistics in the frequency domain based on the first filter.

[0088] In this embodiment of the application, after the electronic device determines the type of exposure stripe, it can mark it, for example, mark the global exposure stripe in the frequency domain as 1 and the local exposure stripe in the frequency domain as 0. Based on this mark, the subsequent electronic device can obtain the type of exposure stripe.

[0089] It is understood that the marking of exposure stripes in this application embodiment is not specifically limited, and the above is only an example.

[0090] Step 23: If a frequency point with a frequency value greater than or equal to the frequency threshold is found in the specified frequency band, the electronic device determines that the type of exposure stripe is a local exposure stripe; otherwise, the electronic device determines that the type of exposure stripe is a global exposure stripe.

[0091] When the light source in the environment is an illumination source, if the exposure time does not match the flicker period of the light source in the environment, the entire captured image may be affected by the AC light source, resulting in global exposure stripes in the first image. When the light source in the environment is the light emitted by the screen display device in the captured image, if the exposure time does not match the flicker period of the screen display device, local exposure stripes may appear in the first image, that is, exposure stripes may exist in the area where the screen display device is located in the first image.

[0092] Based on this, since the emission frequency of the screen display device is greater than the frequency of the illumination source, in this embodiment, a frequency threshold can be determined based on the emission frequency of the screen display device. When a frequency value within a specified frequency band exists in the exposure stripe statistics in the frequency domain, it indicates the presence of exposure stripes in the short frame sequence. If a frequency point within the specified frequency band has a frequency value greater than or equal to the frequency threshold, it indicates that the exposure stripe is caused by light emitted from the screen display device. In this case, the electronic device determines the type of exposure stripe as a local exposure stripe. If a frequency point within the specified frequency band has a frequency value less than the frequency threshold, it indicates that the exposure stripe is caused by the illumination source. In this case, the electronic device determines the type of exposure stripe as a global exposure stripe.

[0093] In one possible implementation, the electronic device can filter the exposure stripe statistics in the frequency domain based on a second filter. If there are frequency points in the exposure stripe statistics in the frequency domain with frequency values ​​greater than or equal to a frequency threshold, the electronic device can filter the frequency points in a specified frequency band based on the second filter to obtain frequency points with frequency values ​​greater than or equal to the frequency threshold. These frequency points with frequency values ​​greater than or equal to the frequency threshold constitute local exposure stripes in the frequency domain. If the electronic device does not obtain frequency points with frequency values ​​greater than or equal to the frequency threshold after filtering the frequency points in a specified frequency band based on the second filter, it indicates that the frequency points in the specified frequency band are global exposure stripes in the frequency domain.

[0094] It should be noted that the specific value of the frequency threshold is not specifically limited in the embodiments of this application, and it can be determined according to the light emission frequency of the screen display device.

[0095] For example, the lowest frequency value of the specified frequency band is 50Hz, the highest frequency value is 200Hz, and the frequency threshold is 100Hz. In this case, if there is a frequency point with a frequency value greater than or equal to 100Hz within the specified frequency band, the electronic device determines that the type of exposure stripe is a local exposure stripe; otherwise, the electronic device determines that the type of exposure stripe is a global exposure stripe.

[0096] S105. The electronic device performs image alignment processing on the long frame sequence and the short frame sequence to obtain the aligned long frame sequence.

[0097] Since the long frame sequence and the short frame sequence are not obtained from exposures at the same time, directly fusing them may produce artifacts. To eliminate or reduce the impact of artifacts, the electronic device performs image alignment processing on the long frame sequence and the short frame sequence to obtain an aligned long frame sequence. The specific steps are as follows:

[0098] Step 31: The electronic device aligns the brightness of the short frame sequence with the brightness of the long frame sequence based on the exposure ratio of the long frame sequence to obtain a brightness-aligned short frame sequence.

[0099] The brightness of a brightness-aligned short frame sequence can be expressed by equation (2) as follows:

[0100] Short_Frame_Boost=Short_Frame×Exposure_Ratio (2)

[0101] Among them, Short_Frame_Boost represents the brightness of the brightness-aligned short frame sequence, Short_Frame represents the brightness of the short frame sequence; Exposure_Ratio represents the exposure ratio of the long frame sequence and the short frame sequence.

[0102] Step 32: The electronic device uses the brightness-aligned short frame sequence as a reference to align the long frame sequence to the brightness-aligned short frame sequence, thereby obtaining the aligned long frame sequence.

[0103] In one possible implementation, the electronic device can align a long frame sequence to a brightness-aligned short frame sequence based on feature extraction and matching, or on optical flow based on feature point tracking, to obtain a global alignment matrix for the long frame sequence. Based on the global alignment matrix of the long frame sequence, image twisting processing is performed on the long frame sequence to obtain the aligned long frame sequence.

[0104] S106. The electronic device performs desaturation processing on the exposure stripes in the aligned short frame sequence based on the aligned long frame sequence, according to the type of exposure stripes, to obtain a compensated short frame sequence.

[0105] In one possible implementation, the electronic device performs debanding processing on the exposure stripes in the aligned short frame sequence based on the aligned long frame sequence, according to the type of exposure stripes. Before obtaining the compensated short frame sequence, the aligned long frame sequence and the aligned short frame sequence can be used as inputs to an artificial intelligence debanding (AIDeBanding) network to generate a brightness gain mesh corresponding to each first image, so that the brightness value of the exposure stripes in the short frame sequence can be compensated based on the brightness gain mesh in the subsequent process.

[0106] After inputting the aligned long frame sequence and the aligned short frame sequence into the AIDeBanding network, the AIDeBanding network can determine a second image from the aligned long frame sequence as a reference frame image. Using the brightness value of each pixel in the reference frame image as a benchmark, the network compares the difference between the brightness value of each pixel in each first image in the aligned short frame sequence and the brightness value of each pixel in the reference frame image, and generates a brightness gain grid corresponding to each first image.

[0107] In this embodiment, the value of each grid point in the brightness gain grid corresponding to each first image represents the brightness gain between a pixel in the corresponding frame of the first image and the corresponding pixel in the reference frame image. Specifically, the brightness gain is the brightness difference between the pixel in the first image and the corresponding pixel in the reference frame image.

[0108] The number of grids can be M×N, where M and N are positive integers, and each brightness gain grid can uniformly cover the entire area of ​​the corresponding first image.

[0109] In this embodiment of the application, the electronic device can determine the pixels with a brightness difference greater than a brightness threshold in a plurality of first images as the pixels of exposure stripes in a plurality of first images based on the brightness gain grid, thereby obtaining the position information of exposure stripes in a plurality of first images.

[0110] It is understood that the brightness threshold value is not specifically limited in the embodiments of this application, and can be set according to specific needs.

[0111] Electronic devices are based on a brightness gain grid. On the one hand, the position information of exposure stripes can be determined from the brightness gain grid. On the other hand, the exposure stripes in the aligned short frame sequence can be debanded to obtain a compensated short frame sequence.

[0112] In one possible implementation, when the exposure stripe type is global exposure stripe, the exposure stripe in the first image will overlap with the moving area. To minimize the impact of the exposure stripe on the overall image, the electronic device performs debanding processing on the exposure stripe in the aligned short frame sequence based on the aligned long frame sequence. The process of obtaining the compensated short frame sequence is as follows:

[0113] Step 41: The electronic device determines the intensity of the global exposure stripes based on the exposure stripes in the specified frequency band.

[0114] In one possible implementation, if in S104, the electronic device determines the type of exposure stripes as global exposure stripes based on the exposure stripe statistics in the spatial domain.

[0115] The electronic device determines the intensity of the exposure stripes based on the peak, valley, and average values ​​of the global exposure stripes in the spatial domain, as shown in Equation (4):

[0116] Banding_Srength=

[0117] (max(Global_Banding_Signal_Space_domain)-min(Global_Banding_Signal_Space_domain))

[0118] / mean(Global_Banding_Signal_Space_domain)(3)

[0119] Where max(Global_Banding_Signal_Space_domain) represents the maximum value of the global exposure stripes in the spatial domain, i.e., the peak value of the global exposure stripes; min(Global_Banding_Signal_Space_domain) represents the minimum value of the global exposure stripes in the spatial domain, i.e., the valley value of the global exposure stripes; mean(Global_Banding_Signal_Space_domain) represents the average value of the global exposure stripes in the spatial domain; and Banding_Srength represents the intensity of the exposure stripes.

[0120] In one possible implementation, if in S104 the electronic device determines the type of exposure stripes based on the exposure stripe statistics in the frequency domain, then in order to determine the exposure stripe intensity, the electronic device converts the global exposure stripes in the frequency domain into global exposure stripes in the spatial domain.

[0121] As shown in equation (3):

[0122] Global_Banding_Signal_Space_domain=Frequency_Domain_Transform(Global_Banding_

[0123] Signal)(4)

[0124] Wherein, Global_Banding_Signal_Space_domain represents the global exposure stripes in the spatial domain; Global_Banding_Signal represents the global exposure stripes in the frequency domain; and Frequency_Domain_Transform() represents performing a Fourier transform on the signal in parentheses.

[0125] The electronic device determines the exposure stripe intensity based on the peak, valley and average values ​​of the global exposure stripes in the spatial domain, as shown in the above formula (3). After obtaining the difference between the peak and valley values ​​of the global exposure stripes, the electronic device determines the exposure stripe intensity by the ratio between the difference and the average value of the global exposure stripes.

[0126] Step 42: Based on the relationship between the intensity of the global exposure stripes and the intensity threshold, determine whether the exposure stripes contain image information.

[0127] If the intensity of the global exposure stripes is less than the intensity threshold, the electronic device determines that the exposure stripes contain image information and executes step 43: the electronic device compensates for the global exposure stripes based on the Meshgrid to obtain a compensated short frame sequence.

[0128] In this embodiment, it is considered that some image information may not be lost in the exposure stripes, and some image information is still retained. For the exposure stripes that retain some image information, the electronic device can perform brightness compensation based on the Meshgrid.

[0129] During the process of compensating for the brightness of exposure stripes based on Meshgrid, the electronic device determines the position information of each pixel in the exposure stripes in Meshgrid and obtains the brightness value of the grid points around each pixel in the exposure stripes.

[0130] In one possible implementation, the position information of each pixel in the exposure stripe can be determined based on the exposure stripe mask. The electronic device generates the exposure stripe mask based on the meshgrid, and the position information of each pixel in the exposure stripe can be obtained based on the position information of each pixel in the exposure stripe mask.

[0131] In this embodiment of the application, after the electronic device determines the position information of each pixel in the exposure stripe, it can determine the compensation gain value of each pixel in the exposure stripe based on the brightness value of the grid points around each pixel in the exposure stripe and the brightness value of each pixel in the exposure stripe, and compensate the brightness of each pixel in the exposure stripe based on the compensation gain value.

[0132] If the intensity of the global exposure stripe is greater than or equal to the intensity threshold, the electronic device determines that the exposure stripe does not contain image information and executes step 44, marking it as an uncompensable global exposure stripe.

[0133] In this embodiment, global exposure stripes with an intensity greater than an intensity threshold will result in the complete loss of image information in that area, making it impossible for the electronic device to recover the image information by performing brightness compensation on the exposed stripe area using the Meshgrid. In this case, the electronic device identifies global exposure stripes with an intensity greater than or equal to the intensity threshold as uncompensable global exposure stripes for subsequent processing during image fusion.

[0134] In one possible implementation, when the exposure stripes are of the local exposure stripe type, the electronic device performs debanding on the exposure stripes in the aligned short frame sequence based on the aligned long frame sequence to obtain the compensated short frame sequence, as follows:

[0135] Step 51: The electronic device determines the motion region based on the motion vector information of the short frame sequence.

[0136] In this embodiment of the application, the electronic device can perform local motion estimation on the aligned short frame sequence to obtain the motion vector information of the short frame sequence.

[0137] Considering that when electronic devices capture images of moving objects, the motion of the object may cause pixel differences between different frames, resulting in blurred or ghosted images of the moving object, the electronic device obtains motion vector information for a short frame sequence by performing local motion estimation on multiple aligned first images. This facilitates subsequent determination of motion regions based on the motion vector information of the short frame sequence and allows for processing of the short frame sequence, avoiding blurring or ghosting phenomena caused by object motion in the generated images.

[0138] In one possible implementation, the electronic device divides each first image in the short frame sequence into several blocks, assuming that all pixels within a block have the same displacement. Based on this, the electronic device determines the position of each block in neighboring first images and derives the relative spatial offset between them; this process constitutes local motion estimation of the aligned short frame sequence. The resulting relative offset is also referred to as motion vector information.

[0139] It should be noted that the local motion estimation method in this application embodiment is not specifically limited. For example, the electronic device can perform local motion estimation on the aligned short frame sequence based on motion estimation algorithms based on optical flow estimation, motion estimation algorithms based on block matching, and motion estimation methods based on Bayesian estimation to obtain the motion vector information of the short frame sequence. The above description only uses the motion estimation algorithm based on block matching to obtain the motion vector information of the short frame sequence as an example.

[0140] In one possible implementation, the electronic device can determine the motion region based on whether the motion vector information is greater than an offset threshold. If the motion vector information of a pixel in the grid is greater than the offset threshold, then the pixel is determined to be a pixel in the motion region; otherwise, it is not a pixel in the motion region. After determining the pixels contained in the motion region, the motion region can be defined.

[0141] Step 52: The electronic device determines the overlap state between the motion area and the exposure stripes based on the position information of the exposure stripes and the motion vector information of the short frame sequence.

[0142] In one possible implementation, the electronic device can determine whether there is any overlap between the pixels of the moving area and the exposure stripes by traversing all the pixels in the grid. If it is determined that the position information of the exposure stripes does not overlap with the moving area, step 53 is executed; otherwise, step 54 is executed.

[0143] Step 53: The electronic device performs desaturation processing on the exposure stripes in the aligned short frame sequence based on the aligned long frame sequence to obtain the compensated short frame sequence.

[0144] If the electronic device determines that the position information of the exposure stripe does not overlap with the moving area, it determines the relationship between the intensity of the local exposure stripe and the intensity threshold. If the intensity of the local exposure stripe is less than the intensity threshold, it indicates that the exposure stripe contains image information. The electronic device compensates for the brightness of the local exposure stripe based on the brightness gain grid to obtain a compensated short frame sequence. If the intensity of the local exposure stripe is greater than or equal to the intensity threshold, it indicates that the exposure stripe does not contain image information. The electronic device replaces the corresponding pixels of the exposure stripe with pixels at the corresponding positions in multiple second images to obtain a compensated short frame sequence.

[0145] In one possible implementation, since the long frame sequence and the short frame sequence are not obtained from exposures at the same time, directly fusing the long frame sequence and the short frame sequence may produce artifacts. Aligning the long frame sequence with the short frame sequence can reduce the impact of artifacts, but artifacts may still occur. Therefore, the electronic device calculates an artifact mask based on the brightness difference between the aligned long frame sequence and the brightness-aligned short frame sequence. In this embodiment, the electronic device identifies pixels where the brightness difference between the aligned long frame sequence and the brightness-aligned short frame sequence is greater than a difference threshold as artifact pixels, and an artifact mask can be generated based on these artifact pixels.

[0146] The electronic device can determine whether there is an overlapping region between the artifact mask and the position information of the local exposure stripes. If there is an overlapping region, it means that artifacts will appear after compensating for the exposure stripes based on the aligned long frame sequence. In this case, no processing is performed on the exposure stripes, and step 54 is executed. If there is no overlapping region, the exposure stripes in the aligned short frame sequence are debanded based on the aligned long frame sequence to obtain the compensated short frame sequence.

[0147] Step 54: The electronic device marks the stripe to be exposed as an uncompensated local exposure stripe.

[0148] In one possible implementation, the electronic device determines whether the location information of the exposure stripes overlaps with the moving area by traversing all pixels in the grid. For example... Figure 5 As shown, the position information of the exposure stripe overlaps with the motion area by 1500. The electronic device then identifies the stripe to be exposed as an uncompensated local exposure stripe and does not perform brightness compensation on the local exposure stripe so that it can be processed in the subsequent image fusion process.

[0149] S107. The electronic device fuses the aligned long frame sequence with the compensated short frame sequence to generate a fused image.

[0150] In this embodiment of the application, the electronic device can fuse the aligned long frame sequence and the compensated short frame sequence based on the Exposure Fusion algorithm to generate a fused image.

[0151] In this embodiment, the electronic device, based on the exposure fusion algorithm, can generate weights corresponding to each second image and each first image according to image information such as brightness, saturation, and contrast of each pixel in the aligned long frame sequence and the compensated short frame sequence. The electronic device then fuses the aligned long frame sequence and the compensated short frame sequence according to the calculated weights to generate a fused image.

[0152] In one possible implementation, if the compensated short frame sequence contains globally exposed bands marked as uncompensable, the electronic device removes the first image corresponding to the uncompensable globally exposed bands in the compensated short frame sequence when fusing the aligned long frame sequence with the compensated short frame sequence. The aligned long frame sequence is then fused with the compensated short frame sequence after removing the first image corresponding to the uncompensable globally exposed bands to generate a fused image, or the aligned long frame sequence can be directly fused to generate the fused image.

[0153] In one possible implementation, when a globally exposed exposure stripe, marked as uncompensable, exists in the compensated short-frame sequence, it indicates that the intensity of the uncompensable stripe is greater than or equal to an intensity threshold. In this case, the electronic device can replace the corresponding pixels of the exposure stripe with pixels at corresponding positions in multiple second images to compensate for the uncompensable global exposure stripe. The compensated short-frame sequence, after compensating for the uncompensable global exposure stripe, is then fused with the aligned long-frame sequence to generate a fused image.

[0154] In this process, since the long frame sequence and the short frame sequence are not obtained based on the same exposure time, fusing the long frame sequence and the short frame sequence may produce artifacts. When the position information of the artifact mask and the global exposure stripe overlap, replacing the pixels of the exposure stripe with the corresponding pixels in multiple second images may produce artifacts.

[0155] To address this, the electronic device can determine whether there is an overlapping region between the artifact mask and the global exposure stripe based on their positional information. If an overlapping region exists, it indicates that compensating for the uncompensable global exposure stripe based on the aligned long frame sequence will result in artifacts. In this case, the uncompensable global exposure stripe is not compensated, and the first image corresponding to the uncompensable global exposure stripe in the compensated short frame sequence is removed. The aligned long frame sequence is then fused with the compensated short frame sequence after removing the first image corresponding to the uncompensable global exposure stripe to generate a fused image, or the aligned long frame sequence can be directly fused to generate the fused image.

[0156] If there is no overlapping area between the two, the corresponding exposure stripe pixels are replaced with pixels at the corresponding positions in multiple second images to compensate for the uncompensable global exposure stripes. The compensated short frame sequence, after compensating for the uncompensable global exposure stripes, is then fused with the aligned long frame sequence to generate a fused image.

[0157] In one possible implementation, when there are uncompensable local exposure stripes in the compensated short frame sequence, the electronic device can fuse the aligned long frame sequence with the compensated short frame sequence containing the uncompensable local exposure stripes to generate a fused image.

[0158] In summary, in this embodiment, the electronic device acquires long frame sequences and short frame sequences with different exposure levels. The short frame sequences correspond to lower exposure levels, allowing moving objects to be clearly imaged, while the long frame sequences contain more image detail. Based on this, after determining the motion region and pixels in the exposure stripes, the electronic device can use the long frame sequences to compensate for the exposure stripes in the short frame sequences, obtaining a compensated short frame sequence. The long frame sequences are then fused with the compensated short frame sequences to ensure clear imaging of moving objects in the fused image while avoiding exposure stripes, thus improving the image display effect. Furthermore, this embodiment considers the impact of artifacts on the fused image and uses an artifact mask to guide the fusion process, thereby improving the image display effect.

[0159] This application can be applied to electronic devices. In some embodiments, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, personal digital assistant (PDA), wearable electronic device, smartwatch, etc. This application does not impose any special limitations on the specific form of the above-mentioned electronic devices. In this embodiment, the structure of the electronic device may be as follows: Figure 6 As shown, Figure 6 This is a component example diagram of an electronic device provided in an embodiment of this application.

[0160] like Figure 6 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a sensor module 130, a camera 140, and a display screen 141, etc.

[0161] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0162] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0163] For example, in this embodiment of the application, the processor 110 can respond to a photo-taking command to acquire a short frame sequence and a long frame sequence; perform motion estimation on the short frame sequence to determine the pixels of the moving region in a plurality of first images; compare the brightness value of each pixel in the plurality of first images with the brightness value of each pixel in the reference frame image in the long frame sequence to determine the pixels of the exposure stripes in the plurality of first images; based on this, according to the overlap state between the pixels of the moving region in the plurality of first images and the pixels of the exposure stripes in the plurality of first images, compensate the pixel values ​​of the pixels of the exposure stripes in the plurality of first images based on the pixel values ​​of each pixel in the plurality of second images to obtain a compensated short frame sequence; and perform image fusion between the long frame sequence and the compensated short frame sequence to generate a fused image.

[0164] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are being used repeatedly.

[0165] Electronic devices implement display functions through a GPU, a display screen 141, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 141 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0166] Display screen 141 is used to display images, videos, etc. Display screen 141 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 141, where N is a positive integer greater than 1.

[0167] The display screen 141 of the electronic device can display a series of graphical user interfaces (GUIs). For example, in this embodiment, the image interfaces that can be displayed on the display screen 141 include an initial user interface, a camera interface, and a gallery interface. The user presses the camera button on the camera interface to take a picture to obtain the fused image.

[0168] Electronic devices can achieve shooting functions through ISP, camera 140, video codec, GPU, display 141 and application processor.

[0169] Camera 140 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. In some embodiments, the electronic device may include one or N cameras 140, where N is a positive integer greater than 1.

[0170] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0171] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 121.

[0172] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.

[0173] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0174] Figure 7 This is an example diagram of the software structure of an electronic device provided in an embodiment of this application.

[0175] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0176] The application layer can include a series of application packages. For example... Figure 7 As shown, the application package can include applications such as camera and gallery.

[0177] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 7 As shown, the application framework layer may include an image alignment module, a statistics module, a desaturation module, an image fusion module, etc.

[0178] The system includes: an image alignment module, which executes steps S102 and S105 to align the short frame sequence to obtain an aligned short frame sequence, and performs image alignment processing on the long frame sequence and the short frame sequence to obtain an aligned long frame sequence; a statistics module, which executes steps S103-S104 to perform exposure stripe self-statistics on the short frame sequence to obtain exposure stripe statistical information, and determines the type of exposure stripes in the short frame sequence based on the exposure stripe statistical information; a debanding module, which executes step S106 to perform debanding processing on the exposure stripes in the aligned short frame sequence based on the type of exposure stripes and the aligned long frame sequence to obtain a compensated short frame sequence; and an image fusion module, which executes step S107 to fuse the aligned long frame sequence and the compensated short frame sequence to generate a fused image.

[0179] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0180] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0181] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0182] A system library can include multiple functional modules. For example, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0183] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0184] A 2D graphics engine is a graphics engine for 2D drawing.

[0185] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, and sensor driver.

[0186] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0187] This embodiment also provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the aforementioned method steps to implement the method described in the above embodiment.

[0188] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: In response to a photographing instruction, a short frame sequence and a long frame sequence are acquired; the short frame sequence comprises a plurality of first images, and the long frame sequence comprises a plurality of second images; the exposure amount of the first image is less than that of the second image; Local motion estimation is performed on the short frame sequence to determine the pixel points of the motion region in the plurality of first images; The brightness values of the pixel points in the plurality of first images are compared with the brightness values of the pixel points in the reference frame image in the long frame sequence; the pixel points in the plurality of first images with a brightness difference greater than a brightness threshold value are determined as the pixel points of the exposure streak in the plurality of first images; According to the overlapping state of the pixel points of the motion region in the plurality of first images and the pixel points of the exposure streak in the plurality of first images, the pixel values of the pixel points of the exposure streak in the plurality of first images are compensated based on the pixel values of the pixel points in the plurality of second images to obtain a compensated short frame sequence; The long frame sequence and the compensated short frame sequence are fused to generate a fused image.

2. The method of claim 1, wherein, The compensation of the pixel values of the pixel points of the exposure streak in the plurality of first images based on the pixel values of the pixel points in the plurality of second images according to the overlapping state of the pixel points of the motion region in the plurality of first images and the pixel points of the exposure streak in the plurality of first images to obtain a compensated short frame sequence comprises: The type of the exposure streak in the short frame sequence is determined, and the type of the exposure streak comprises a global exposure streak and a local exposure streak; The pixel values of the pixel points of the exposure streak in the plurality of first images are compensated based on the pixel values of the pixel points in the plurality of second images according to the type of the exposure streak and the overlapping state of the pixel points of the motion region in the plurality of first images and the pixel points of the exposure streak in the plurality of first images to obtain a compensated short frame sequence.

3. The method of claim 2, wherein, The determination of the type of the exposure streak in the short frame sequence comprises: A first image in the short frame sequence is determined as a reference frame image; The pixel values of each row of the reference frame image are accumulated to obtain exposure streak statistical information of the short frame sequence; The type of the exposure streak in the short frame sequence is determined according to the exposure streak statistical information.

4. The method of claim 3, wherein, When the pixel points of the motion region in the plurality of first images and the pixel points of the exposure streak in the plurality of first images are not overlapped, the intensity of the local exposure streak is determined based on the exposure streak statistical information of the short frame sequence; When the intensity of the local exposure streak is less than an intensity threshold value, the pixel values of the pixel points of the exposure streak in the plurality of first images are compensated based on the pixel values of the pixel points in the plurality of second images to obtain a compensated short frame sequence. ​ 5. The method of claim 4, wherein, The method further comprises, when the non-overlapping of the pixel points of the motion region in the plurality of first images and the pixel points of the exposure streaks in the plurality of first images, determining the intensity of the local exposure streaks based on the exposure streak statistical information of the short frame sequence. When the intensity of the local exposure streaks is greater than or equal to the intensity threshold, the pixel points of the exposure streaks in the plurality of first images are replaced by the pixel points of corresponding positions in the plurality of second images to obtain the compensated short frame sequence.

6. The method of claim 3, wherein, The type of the exposure streaks is global exposure streaks, and the compensating of the pixel values of the pixel points of the exposure streaks in the plurality of first images based on the pixel values of the pixel points in the plurality of second images according to the type of the exposure streaks and the overlapping state of the pixel points of the motion region in the plurality of first images and the pixel points of the exposure streaks in the plurality of first images to obtain the compensated short frame sequence comprises: determining the intensity of the global exposure streaks based on the exposure streak statistical information of the short frame sequence; when the intensity of the global exposure streaks is less than the intensity threshold, compensating the pixel values of the pixel points of the exposure streaks in the plurality of first images based on the pixel values of the pixel points in the plurality of second images to obtain the compensated short frame sequence.

7. The method of claim 3, wherein, The determining of the type of the exposure streaks in the short frame sequence according to the exposure streak statistical information comprises: when the exposure streak statistical information presents the sinusoidal wave form ripple in the whole image, it is determined that the type of the exposure streaks in the short frame sequence is global exposure streaks; and when the exposure streak statistical information presents the sinusoidal wave form ripple locally, it is determined that the type of the exposure streaks in the short frame sequence is local exposure streaks.

8. The method of claim 3, wherein, The determining of the type of the exposure streaks in the short frame sequence according to the exposure streak statistical information comprises: performing Fourier transform on the exposure streak statistical information to convert the exposure streak statistical information from the spatial domain to the frequency domain to obtain the exposure streak statistical information in the frequency domain; in a specified frequency band, if there is a frequency point with a frequency value higher than a frequency threshold in the exposure streak statistical information in the frequency domain, it is determined that the type of the exposure streaks in the short frame sequence is local exposure streaks; otherwise, it is determined that the type of the exposure streaks in the short frame sequence is global exposure streaks.

9. The method of claim 1, wherein, The motion estimation of the short frame sequence to determine the pixel points of the motion region in the plurality of first images comprises: performing local motion estimation on the short frame sequence to generate motion vector information; determining the pixel points of the motion region in the plurality of first images based on the size relationship between the motion vector information and the offset threshold.

10. An electronic device, comprising: The electronic device comprises a processor and a memory; the memory is configured to store program code and transmit the program code to the processor; the processor is configured to execute the steps of the image processing method according to the instructions in the program code.

11. A computer readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the image processing method according to any one of claims 1-9. The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the image processing method according to any one of claims 1-9.

Images