Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses an image processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of image processing. According to the image processing method, when water ripples exist in a short-exposure sub-frame, the long-exposure sub-frame can be used as a standard sub-frame for short-exposure sub-frame imaging reference due to the fact that the difference of exposure energy among multiple rows of pixels in a long-exposure sub-frame adjacent to the short-exposure sub-frame is small; therefore, the exposure compensation value for performing exposure compensation on the short exposure sub-frame can be determined based on the difference between the exposure quantity and the brightness characteristic between the short exposure sub-frame and the standard sub-frame, so that water ripples in the short exposure sub-frame are removed by performing exposure compensation on the short exposure sub-frame. Based on the short exposure sub-frame after exposure compensation, the imaging quality of the output image generated by image fusion of the electronic equipment is improved, and the user experience is further improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Stagger high dynamic range (HDR) imaging is a technique used to achieve a greater dynamic range (i.e., greater contrast between light and dark areas) than ordinary digital imaging technology. It involves changing the camera's shutter speed multiple times during the capture of a single frame, generating multiple sub-frames with different shutter speeds. These sub-frames are then fused to obtain a high dynamic range image. Compared to traditional imaging methods, stagger HDR imaging provides more dynamic range and image detail. By using images with the best detail corresponding to each exposure time to synthesize the final displayed image, it better reflects the user's visual experience in the real environment.

[0003] Currently, when using electronic devices with cameras for framing and shooting, imaging methods based on single-frame progressive high dynamic range imaging can improve the dynamic range of the images output by the electronic devices, thereby improving image quality.

[0004] However, in the above methods, if the exposure method of the electronic device is rolling shutter exposure and the electronic device is in an environment with power frequency light source, the power frequency light source will cause significant power frequency interference to the imaging of the electronic device, which will seriously affect the imaging quality. Summary of the Invention

[0005] Embodiments of this application provide an image processing method, apparatus, electronic device, storage medium, and program product for improving the imaging quality of electronic devices.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, this application provides an image processing method, comprising: obtaining target parameters based on a short-exposure subframe in an initial image and a standard subframe adjacent to the short-exposure subframe, the target parameters including the exposure of the standard subframe, the exposure of the short-exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short-exposure subframe, wherein the standard subframe adjacent to the short-exposure subframe is a long-exposure subframe in the initial image or a long-exposure subframe in the previous initial image of the initial image; if the short-exposure subframe has water ripples, determining an exposure compensation value for the short-exposure subframe based on the exposure of the standard subframe, the exposure of the short-exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short-exposure subframe; and performing exposure compensation on the short-exposure subframe based on the exposure compensation value.

[0008] In the image processing method provided in this application embodiment, under the single-frame progressive high dynamic range imaging mode, if there is a power frequency light source in the ambient light and the exposure time of each row of pixels in the short exposure subframe is too short, water ripples are likely to appear in the short exposure subframe. Since the difference in exposure energy between multiple rows of pixels in the long exposure subframe adjacent to the short exposure subframe is small, the long exposure subframe can be used as a standard subframe for imaging the short exposure subframe. Thus, the exposure compensation value for exposure compensation of the short exposure subframe can be determined based on the difference in exposure and brightness characteristics between the short exposure subframe and the standard subframe. This allows the water ripples in the short exposure subframe to be removed through exposure compensation. In addition, the short exposure subframe after exposure compensation improves the imaging quality of the output image generated by image fusion of the electronic device, thereby improving the user experience.

[0009] In one possible implementation, before determining the exposure compensation value of the short exposure subframe based on the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe if water ripples are present in the short exposure subframe, the image processing method further includes: determining the ambient light brightness fluctuation parameters based on the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. The brightness fluctuation parameters include fluctuation depth and fluctuation period. If the change value of the fluctuation depth is less than a first change threshold and the fluctuation depth is greater than a set depth value, and the change value of the fluctuation period is less than a second change threshold and the fluctuation period is greater than the exposure duration of each row of pixels in the short exposure subframe, then it is determined that water ripples exist in the short exposure subframe.

[0010] In one possible implementation, determining the ambient light brightness fluctuation parameters based on the exposure of the standard subframe, the exposure of the short-exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short-exposure subframe includes: obtaining the brightness difference characteristics between the standard subframe and the short-exposure subframe based on the exposure of the standard subframe, the exposure of the short-exposure subframe, the brightness characteristics of the standard subframe, and the brightness difference characteristics of the short-exposure subframe, wherein the exposure of the standard subframe, the exposure of the short-exposure subframe, the brightness characteristics of the standard subframe, the brightness characteristics of the short-exposure subframe, and the brightness difference characteristics satisfy:

[0011]

[0012] Where Delta is the brightness difference feature, L1 is the exposure of the standard subframe, S1 is the exposure of the short exposure subframe, L2 is the brightness feature of the standard subframe, and S2 is the brightness feature of the short exposure subframe; the fluctuation depth is determined based on the second moment of the brightness difference feature and the first moment of the brightness feature of the standard subframe; the fluctuation period is determined based on the brightness difference feature.

[0013] Among the above possible implementations, when using traditional calculation methods to obtain the brightness difference features between standard subframes and short-exposure subframes, there is no need to use complex deep learning algorithms (such as convolutional neural networks) or machine learning algorithms for calculation. That is, while accurately obtaining the brightness fluctuation parameters of ambient light, it saves the computing power of electronic devices and improves the processing speed of electronic devices in the image processing process.

[0014] In one possible implementation, the steps of obtaining the brightness features of the standard subframe and the short exposure subframe based on the short exposure subframe in the initial image and the standard subframe adjacent to the short exposure subframe include: removing pixels at the target position in the standard subframe and the short exposure subframe to obtain preprocessed standard subframe and preprocessed short exposure subframe, where the target position is the position of a pixel whose brightness value does not meet a preset condition in either the standard subframe or the short exposure subframe, and the preset condition includes that the brightness value of the pixel is within a set brightness value range; determining the brightness features of the standard subframe based on the preprocessed standard subframe, and determining the brightness features of the short exposure subframe based on the preprocessed short exposure subframe.

[0015] In the above possible implementations, by preprocessing the standard subframe and the short exposure subframe, the brightness features of the short exposure subframe and the standard subframe are obtained based on the preprocessed image data. The brightness features can be used to obtain more accurate brightness fluctuation parameters of the ambient light, thereby improving the detection accuracy when detecting whether there are water ripples in the short exposure subframe based on the brightness fluctuation parameters.

[0016] In one possible implementation, the exposure compensation value includes the pixel compensation value for each row of pixels in the short exposure subframe. Determining the exposure compensation value of the short exposure subframe based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe includes: determining an exposure ratio, which is the ratio of the exposure of the standard subframe to the exposure of the short exposure subframe; determining the brightness characteristic ratio of each row of pixels between the standard subframe and the short exposure subframe based on the brightness characteristics of the standard subframe and the short exposure subframe; and determining the pixel compensation value for each row of pixels based on the exposure ratio and the brightness characteristic ratio of each row of pixels.

[0017] In one possible implementation, determining the exposure compensation value of a short exposure subframe based on the exposure of a standard subframe, the exposure of a short exposure subframe, the brightness characteristics of a standard subframe, and the brightness characteristics of a short exposure subframe includes: inputting the exposure of a standard subframe, the exposure of a short exposure subframe, the brightness characteristics of a standard subframe, and the brightness characteristics of a short exposure subframe into a first exposure compensation model to obtain the exposure compensation value output by the first exposure compensation model. The first exposure compensation model is obtained by training an initial model using a first training sample set. The first training sample set includes target parameters of adjacent long exposure subframes and short exposure subframes under various lighting sources, as well as the exposure compensation value of the short exposure subframe.

[0018] In the above possible implementations, by using a pre-trained first exposure compensation model to obtain the exposure compensation value of the short exposure subframe, the accurate exposure compensation value output by the first exposure compensation model can be obtained, thereby improving the compensation effect when performing exposure compensation on the short exposure subframe.

[0019] In one possible implementation, the image processing method further includes: if there are no water ripples in the short exposure subframe, performing a fusion process on the long exposure subframe and the short exposure subframe in the initial image to obtain the output image corresponding to the initial image.

[0020] In one possible implementation, the image processing method further includes: fusing the long exposure subframes in the initial image and the short exposure subframes in the initial image after exposure compensation processing to obtain the output image corresponding to the initial image.

[0021] In the above possible implementations, since the water ripples present in the short exposure subframes are eliminated during exposure compensation, the water ripples in the output image obtained by the fusion processing based on the long exposure subframes in the initial image and the short exposure subframes after exposure compensation in the initial image are avoided, thereby improving the imaging quality of electronic devices under the single-frame progressive high dynamic range imaging mode.

[0022] Secondly, this application provides an image processing apparatus, including a target parameter acquisition module, an exposure compensation value determination module, and a first exposure compensation module: the target parameter acquisition module is used to acquire target parameters based on a short exposure subframe in an initial image and a standard subframe adjacent to the short exposure subframe. The target parameters include the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. The standard subframe adjacent to the short exposure subframe is a long exposure subframe in the initial image or a long exposure subframe in the previous initial image of the initial image. The exposure compensation value determination module is used to determine the exposure compensation value of the short exposure subframe based on the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe if the short exposure subframe has water ripples. The first exposure compensation module is used to perform exposure compensation on the short exposure subframe based on the exposure compensation value.

[0023] Thirdly, this application provides an image processing method, comprising: obtaining target parameters based on short exposure subframes in an initial image and standard subframes adjacent to the short exposure subframes, the target parameters including the exposure amount of the standard subframes, the exposure amount of the short exposure subframes, the brightness features of the standard subframes, and the brightness features of the short exposure subframes, wherein the standard subframes adjacent to the short exposure subframes are long exposure subframes in the initial image or long exposure subframes in the previous initial image of the initial image; if the short exposure subframes have water ripples, inputting the short exposure subframes and the standard subframes adjacent to the short exposure subframes into a second exposure compensation model to obtain the short exposure subframes after removing water ripples output by the second exposure compensation model, wherein the second exposure compensation model is obtained by training the initial model using a second training sample set, the second training sample set including adjacent long exposure subframes under various lighting sources and short exposure subframes without water ripples.

[0024] Fourthly, this application provides an image processing apparatus, including a target parameter acquisition module and a second exposure compensation module. The target parameter acquisition module is used to acquire target parameters based on short exposure subframes in an initial image and standard subframes adjacent to the short exposure subframes. The target parameters include the exposure amount of the standard subframes, the exposure amount of the short exposure subframes, the brightness characteristics of the standard subframes, and the brightness characteristics of the short exposure subframes. The standard subframes adjacent to the short exposure subframes are long exposure subframes in the initial image or long exposure subframes in the previous initial image of the initial image. The second exposure compensation module is used to input the short exposure subframes and the standard subframes adjacent to the short exposure subframes into a second exposure compensation model if water ripples exist in the short exposure subframes, to obtain the short exposure subframes after removing water ripples output by the second exposure compensation model. The second exposure compensation model is obtained by training the initial model using a second training sample set, which includes adjacent long exposure subframes under various lighting sources and short exposure subframes without water ripples.

[0025] Fifthly, this application provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the image processing method provided in the first and third aspects and any of the embodiments described above.

[0026] In a sixth aspect, this application provides a computer-readable storage medium in which computer-executable instructions stored in the computer-readable storage medium are executed by a processor of an electronic device, enabling the electronic device to perform the image processing method provided in the first and third aspects and any of the embodiments described above.

[0027] In a seventh aspect, this application provides a computer program product including computer instructions that, when executed on a processor of an electronic device, enable the electronic device to perform the image processing method provided in the first and third aspects and any of the embodiments described above.

[0028] The beneficial effects of aspects two through seven can be found in the description of aspect one and any of its implementations, and will not be repeated here. Based on the implementations provided in the above aspects, this application can be further combined to provide even more implementations. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0030] Figure 2 A comparative diagram showing the exposure methods of four initial images;

[0031] Figure 3 This is a schematic diagram of the imaging of a short-exposure subframe under a power frequency light source;

[0032] Figure 4 A flowchart illustrating an image processing method provided in this application embodiment. Figure 1 ;

[0033] Figure 5 A flowchart illustrating an image processing method provided in this application embodiment. Figure 2 ;

[0034] Figure 6 A flowchart illustrating an image processing method provided in this application embodiment. Figure 3 ;

[0035] Figure 7 A flowchart illustrating an image processing method provided in this application embodiment. Figure 4 ;

[0036] Figure 8 A flowchart illustrating an image processing method provided in this application embodiment. Figure 5 ;

[0037] Figure 9 A flowchart illustrating an image processing method provided in this application embodiment. Figure 6 ;

[0038] Figure 10 A schematic diagram of a process for obtaining a first exposure compensation model provided in an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0040] Figure 12 This is a schematic diagram of another image processing apparatus provided in an embodiment of this application;

[0041] Figure 13 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0042] Figure 14 A schematic diagram of a process for obtaining a second exposure compensation model provided in an embodiment of this application;

[0043] Figure 15 This is a schematic diagram of the structure of another image processing apparatus provided in the embodiments of this application;

[0044] Figure 16 This is a schematic diagram illustrating an application scenario of an image processing device provided in an embodiment of this application;

[0045] Figure 17 This is a schematic diagram illustrating an application scenario of another image processing apparatus provided in an embodiment of this application;

[0046] Figure 18 A schematic diagram illustrating an application scenario of another image processing apparatus provided in this application embodiment;

[0047] Figure 19 This is a schematic diagram of the structure of another image processing apparatus provided in the embodiments of this application;

[0048] Figure 20 This is a schematic diagram of the structure of another image processing apparatus provided in the embodiments of this application;

[0049] Figure 21 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0052] In embodiments of the invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, article, or apparatus that includes that element.

[0053] In embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0054] The image processing method provided in this application can be applied to electronic devices, such as smartphones, tablets, laptops, vehicle reversing cameras, wearable devices, camera modules, and cloud devices. Taking a smartphone as an example... Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 in this embodiment of the application includes at least a processor 110, a communication unit 120, a sensor unit 130, an input unit 140, a display unit 150, a power supply unit 160, an interface unit 170, an audio unit 180, and a camera unit 190.

[0055] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0056] In this embodiment, the camera unit 190 is equipped with multiple image sensors. During image capture, the shutter of the camera unit 190 is opened, allowing the multiple image sensors in the camera unit 190 to be exposed to ambient light. The multiple image sensors are used to convert the received light signals into electrical signals. The camera unit 190 is used to further convert the electrical signals converted by the multiple image sensors into digital signals, forming image data (raw data) corresponding to each subframe in the initial image, which is then transmitted to the processor 110 for image processing. The raw data is the original data after the image sensors convert the acquired light signals into digital signals, and includes the original color information of the image. In this embodiment, the multiple image sensors of the camera unit 190 can adopt a rolling shutter exposure method. Rolling shutter exposure refers to exposing all pixels of the image sensor row by row until all pixels are exposed.

[0057] The display unit 150 includes a plurality of pixels and is used to display an image based on the output image generated after image processing in the processor 110.

[0058] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0059] Electronic devices can use various exposure methods when capturing images or videos, such as... Figure 2 As shown, Figure 2 This is a comparative diagram of four initial image exposure methods. The four exposure methods for electronic devices are explained below.

[0060] Normal exposure mode: Each initial image is a long exposure frame, and there is an exposure gap between adjacent initial images.

[0061] Traditional Digital Overlay (DOL) exposure method: Each initial image frame includes adjacent long exposure subframes, medium exposure subframes, and short exposure subframes, and there is an exposure gap between adjacent initial images.

[0062] Multi-frame digital overlay exposure mode (n DOL): In each pair of adjacent initial images, the previous initial image includes a long exposure subframe and a medium exposure subframe, and the next initial image includes a long exposure subframe and a short exposure subframe, and there is an exposure gap between the two adjacent initial images.

[0063] Full overlapped exposure mode (DOL): Each initial image frame includes adjacent long exposure subframes, medium exposure subframes, and short exposure subframes, and there is no exposure gap between adjacent initial images.

[0064] Among them, digital overlap (DOL) exposure mode refers to an exposure mode of multi-frame high dynamic range imaging of image sensors. Under the digital overlap exposure mode, electronic devices can simultaneously output multiple frames of images with different exposure times. Dynamic range refers to the brightness ratio between the brightest pixel and the darkest pixel in the image. When the dynamic range of the image is large, the output image of the electronic device can fully present the real scene with large differences in brightness levels.

[0065] Among the four exposure methods mentioned above, when electronic devices employ traditional digital overlay exposure, multi-frame digital overlay exposure, and full overlay exposure, they support single-frame progressive high dynamic range imaging. This is achieved by repeatedly changing the camera's shutter speed within a single frame, generating multiple sub-frames with different shutter speeds. These sub-frames are then fused to obtain a high dynamic range image. Furthermore, digital overlay exposure significantly shortens the time interval between two exposed frames, effectively reducing motion blur when the subject moves too fast. Motion blur refers to the phenomenon where the same object appears at different positions in multiple consecutive frames due to relative motion between the camera and the object. When the object moves quickly, noticeable motion blur appears across multiple frames.

[0066] When electronic devices use a single-frame progressive high dynamic range imaging method for recording or image previewing, the initial image captured by the electronic device includes at least long-exposure subframes and short-exposure subframes with different exposure durations. The exposure duration of the long-exposure subframe can be longer than that of the short-exposure subframe, and the exposure duration of each row of pixels in the long-exposure subframe is longer than one times the fluctuation period (i.e., the power frequency period) of the power frequency light source. In some application scenarios, the multiple subframes may also include medium-exposure subframes, which have an exposure duration longer than the short-exposure subframes but shorter than the long-exposure subframes.

[0067] In high dynamic range scenarios, to better capture object details in bright areas, electronic devices need to increase the camera's shutter speed to compress the exposure time of short exposure subframes. In this case, the exposure time of the complementary metal-oxide-semiconductor (CMOS) image sensor corresponding to each row of pixels in the short exposure subframe is easily less than the duration of one oscillation period of the power frequency light source, which can easily cause banding in the short exposure subframe. Power frequency refers to the rated frequency used in power generation, transmission, transformation, and distribution equipment, as well as industrial and civil electrical equipment. The light sources used in household appliances, industrial equipment, and commercial facilities are almost all power frequency power supplies.

[0068] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram of short-exposure subframe imaging under power frequency light source.

[0069] During the capture of short-exposure subframes by electronic devices, the exposure time of each row of pixels in the short-exposure subframe is the same, and the exposure time of each row of pixels is less than one times the fluctuation period of the power frequency light source. However, due to the influence of the power frequency light source, the exposure energy received by different pixel rows from the light source varies. For example, when a pixel row is exposed within the peak range of the power frequency light source, bright stripes appear in the image of the corresponding pixel row area within the short-exposure subframe; when a pixel row is exposed within the trough range of the power frequency light source, dark stripes appear in the image of the corresponding pixel row area within the short-exposure subframe. At this time, because the exposure energy received by a pixel row within the peak range is much greater than that received within the trough range, the difference in exposure energy between different pixel rows in the short-exposure subframe is too large, resulting in alternating bright and dark stripes in the short-exposure subframe, i.e., water ripples.

[0070] If water ripples are present in the short-exposure subframes of the initial image captured under power frequency light source, the output image generated by image fusion of the long-exposure subframes of the initial image and the short-exposure subframes containing water ripples will also contain water ripples, thus affecting the imaging quality and resulting in a poor user experience.

[0071] To improve image quality, it can be first determined whether water ripples exist in the short-exposure subframes, and then processed accordingly to eliminate them. For example, when capturing multiple consecutive initial images, the ambient light brightness fluctuation parameters are determined based on the long-exposure and short-exposure subframes in each initial image, and the presence of water ripples in the short-exposure subframes is determined based on these parameters. In one implementation, the electronic device can also collect ambient light brightness data through a light source sensor in its sensor unit and use a Fourier algorithm to calculate the ambient light brightness fluctuation parameters.

[0072] When water ripples are confirmed to exist in short-exposure subframes, one way to eliminate them is to extend the exposure time of the short-exposure subframes. In this method, the exposure time of the short-exposure subframes is adjusted based on the fluctuation period of ambient light, ensuring that the exposure time of each row of pixels in the short-exposure subframe is greater than twice the fluctuation period of ambient light. This avoids excessive differences in exposure energy between different pixel rows, thus eliminating the water ripples in the short-exposure subframes. However, after extending the exposure time of the short-exposure subframes in each initial image, overexposure is prone to occur in the bright areas of the short-exposure subframes. This makes it difficult to capture details in the bright areas even within the short-exposure subframes, severely impacting the image quality of the output image obtained after image fusion based on short-exposure and long-exposure subframes.

[0073] Another way to eliminate water ripples is to adjust the frame rate of the initial image. The frame rate is adjusted based on the ambient light fluctuation period, ensuring that the frame length of each initial image is an integer multiple of the ambient light fluctuation period. When short-exposure subframes and long-exposure subframes from multiple consecutive initial images are fused, the dynamically rolling water ripples in the resulting multi-frame output image can be converted into static water ripples, effectively reducing the visual impact of dynamic water ripples on the user. Here, the frame length of the initial image is the reciprocal of its frame rate. By adjusting the frame rate of the initial image to convert the dynamically rolling water ripples in multiple consecutive output images into static water ripples, the water ripples in the output image are less noticeable to the user when the ambient light fluctuation depth is small, thus reducing the visual impact of water ripples. However, when the ambient light fluctuation depth is large, noticeable static water ripples still exist in the multi-frame output image. Furthermore, reducing the frame rate of the initial image will cause the electronic device's display frame rate to switch from a high frame rate mode to a low frame rate mode. When there are fast-moving subjects in the shooting scene of the electronic device, blurry images or ghosting are likely to appear in the continuous output images, resulting in a poor user experience. Increasing the frame rate of the initial image will increase the power consumption of the electronic device and greatly increase the operating load of the electronic device.

[0074] Against this background, this application provides an image processing method. In single-frame progressive high dynamic range imaging, if the presence of a power frequency light source in the ambient light causes water ripples in the output image generated after the fusion of multiple subframes, the water ripples in the short-exposure subframes can be removed by performing exposure compensation on the short-exposure subframes among the multiple subframes, thereby removing the water ripples in the output image obtained by image fusion and improving the imaging quality of the electronic device's output image.

[0075] like Figure 4 As shown, Figure 4 A flowchart illustrating an image processing method provided in this application embodiment. Figure 1 The image processing method provided in this application includes the following steps S210 to S270.

[0076] S210: Obtain target parameters based on the short exposure subframes in the initial image and the standard subframes adjacent to the short exposure subframes.

[0077] The target parameters include the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. The standard subframe adjacent to the short exposure subframe is either a long exposure subframe in the initial image or a long exposure subframe in the previous initial image.

[0078] In a long exposure subframe, the exposure duration of each row of pixels is greater than twice the fluctuation period of the power frequency light source. The difference in exposure energy between rows of pixels in a long exposure subframe is relatively small. Therefore, using a long exposure subframe as a standard subframe for imaging reference can determine the impact of the power frequency light source on the imaging of a short exposure subframe. Furthermore, if the initial image's multiple subframes were captured in the order of first capturing a long exposure subframe and then adjusting the camera's shutter speed to obtain a short exposure subframe, then the long exposure subframe adjacent to the short exposure subframe in the initial image can be used as the standard subframe corresponding to the short exposure subframe. If the initial image's multiple subframes were captured in the order of first capturing a short exposure subframe and then adjusting the camera's shutter speed to obtain a long exposure subframe, meaning that the long exposure subframe of the initial image has not yet been obtained when the short exposure subframe is captured, then to shorten image processing time, the long exposure subframe in the preceding initial image can be used as the standard subframe corresponding to the short exposure subframe.

[0079] In one implementation, combining Figure 4 ,like Figure 5 As shown, target parameters are obtained based on short-exposure subframes in the initial image and standard subframes adjacent to the short-exposure subframes, specifically including S210A to S210B:

[0080] S210A: Obtain the exposure of a short exposure subframe based on the short exposure subframe in the initial image; obtain the exposure of a standard subframe based on the standard subframe adjacent to the short exposure subframe.

[0081] Taking the acquisition of exposure for short-exposure subframes as an example, statistical analysis is performed on the image data (raw data) of the short-exposure subframes in the acquired initial image to obtain a histogram of the grayscale distribution of the short-exposure subframes. This histogram, reflecting the brightness distribution of the entire short-exposure subframe, allows the determination of the current shooting environment's brightness. Furthermore, de-mosaicing and color conversion are performed on the image data of the short-exposure subframes to convert the image data of each pixel into a brightness value. Based on the brightness value of each pixel in the short-exposure subframe, the exposure of the corresponding display image for the short-exposure subframe is obtained.

[0082] In one implementation, multiple exposure values ​​can be pre-determined for multiple brightness values ​​that may appear in an image frame, with each brightness value corresponding to one of the multiple exposure values. After determining the brightness value of a short-exposure subframe based on the image data of the short-exposure subframe, the multiple exposure values ​​pre-stored in the electronic device are called to determine the exposure value corresponding to the brightness value of the short-exposure subframe, which is then used as the exposure value of the short-exposure subframe.

[0083] For standard subframes adjacent to short-exposure subframes, the specific steps for obtaining the exposure of the standard subframe based on the image data of the standard subframe can refer to the above-mentioned method for calculating the exposure of short-exposure subframes, and will not be repeated here.

[0084] S210B: Obtain the brightness features of the standard subframe and the brightness features of the short exposure subframe based on the short exposure subframe in the initial image and the standard subframe adjacent to the short exposure subframe.

[0085] Based on image data from short-exposure subframes, brightness features of the short-exposure subframes can be calculated. Based on image data from standard subframes, brightness features of the standard subframes can be calculated. Brightness features can include: brightness fluctuation features, differential brightness features, brightness mean and brightness standard deviation, filtered average value, higher-order brightness features, etc.

[0086] In one implementation, combining Figure 5 ,like Figure 6 As shown, the steps in S210B to obtain the brightness features of the standard subframe and the brightness features of the short-exposure subframe based on the short-exposure subframe in the initial image and the standard subframe adjacent to the short-exposure subframe can specifically include S210B-1 to S210B-2:

[0087] S210B-1: Remove pixels at the target location from the standard subframe and the short exposure subframe to obtain the preprocessed standard subframe and the preprocessed short exposure subframe.

[0088] The target location is the position of a pixel whose brightness value does not meet the preset conditions in either the standard subframe or the short exposure subframe. The preset conditions include that the brightness value of the pixel is within a set brightness value range.

[0089] Specifically, when determining the brightness characteristics of a standard subframe based on image data from a standard subframe, and when determining the brightness characteristics of a short-exposure subframe based on image data from a short-exposure subframe, the presence of overexposed pixels and completely black pixels (pixels whose brightness values ​​are outside the set brightness range) in both the standard and short-exposure subframes will affect the exposure compensation value of the short-exposure subframe determined based on the exposure of the standard subframe, the exposure of the short-exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short-exposure subframe. For example, if the proportion of overexposed pixels is too high, the exposure compensation value for normal pixels may be lower than the optimal brightness compensation value, resulting in a darker image in the normal pixel area after exposure; if the proportion of completely black pixels is too high, the exposure compensation value for normal pixels may be higher than the optimal brightness compensation value, resulting in a brighter image in the normal pixel area after exposure. Specifically, overexposed pixels with brightness values ​​higher than the highest value in the set brightness range result in an overly bright image in the pixel area; completely black pixels with brightness values ​​lower than the lowest value in the set brightness range result in an overly dark image in the pixel area. Both overexposure and completely black pixels can easily cause a loss of detail in the output image.

[0090] In this embodiment, the standard subframe includes multiple first pixels, and the short exposure subframe includes multiple second pixels. The positions of the multiple first pixels and the multiple second pixels correspond one-to-one. When removing overexposed and black pixels in the standard subframe and the short exposure subframe, the pixels in the standard subframe and the short exposure subframe need to be removed symmetrically.

[0091] When preprocessing the standard subframe and the short exposure subframe, pixels at the target position in both the standard and short exposure subframes are removed. Specifically, based on the image data of the standard subframe, the position of the first pixel in the standard subframe that is overexposed and completely black is determined as the first position. Based on the image data of the short exposure subframe, the position of the second pixel in the short exposure subframe that is overexposed and completely black is determined as the second position. The union of the first and second positions is taken as the target position. Pixels at the first and second positions in the standard subframe are removed to obtain the preprocessed standard subframe; pixels at the first and second positions in the short exposure subframe are removed to obtain the preprocessed short exposure subframe, wherein the first and second positions at least partially overlap.

[0092] S210B-2: Determine the brightness characteristics of the standard subframe based on the preprocessed standard subframe, and determine the brightness characteristics of the short exposure subframe based on the preprocessed short exposure subframe.

[0093] This application uses brightness fluctuation features as an example to introduce the image processing process based on brightness fluctuation features. Specifically, the brightness features of the standard subframe are determined based on the preprocessed standard subframe (i.e., removing overexposed pixels and dead black pixels), and the brightness features of the short exposure subframe are determined based on the preprocessed short exposure subframe.

[0094] In one alternative implementation, the brightness fluctuation characteristics of the short exposure subframes are determined based on the preprocessed short exposure subframes, satisfying the following:

[0095]

[0096] Among them, S i Here, H represents the brightness fluctuation characteristics of the short exposure subframe, W represents the image height of the short exposure subframe, and I represents the image width of the short exposure subframe. L Image data for short-exposure subframes. Let be the image data of the pixels in row a and column b of the short exposure subframe, h be the number of pixel rows extracted when calculating the brightness fluctuation feature of the short exposure subframe, and m be the step size (i.e. the unit image area used when calculating the brightness fluctuation feature). When the image size remains unchanged, the smaller the step size, the more brightness fluctuation features are extracted.

[0097] It should be understood that when calculating the brightness fluctuation characteristics of short exposure subframes, the h rows of pixels extracted from the short exposure subframes should satisfy that the total exposure time of the h rows of pixels is greater than or equal to the duration of one fluctuation period of the power frequency light source when there is a power frequency light source in the ambient light, so as to ensure that the power frequency interference information caused by the power frequency light source to the short exposure subframe can be obtained based on the brightness fluctuation characteristics of the calculated short exposure subframes.

[0098] In this embodiment, by preprocessing the standard subframe and the short exposure subframe, overexposed pixels and black pixels in the image are removed. This allows the brightness features of the standard subframe to be obtained based on the image data of the preprocessed standard subframe, and the brightness features of the short exposure subframe to be obtained based on the image data of the preprocessed short exposure subframe. The brightness features can be used to obtain more accurate brightness fluctuation parameters of the ambient light, thereby improving the detection accuracy when detecting water ripples in the short exposure subframe based on the brightness fluctuation parameters.

[0099] In this embodiment, the method for calculating the brightness characteristics of the standard subframe using the image data of the standard subframe can refer to the calculation method of the short exposure subframe described above, and will not be repeated here.

[0100] S220: Determine the ambient light brightness fluctuation parameters based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe.

[0101] Among them, the brightness fluctuation parameters include fluctuation depth and fluctuation period.

[0102] Since the ambient light brightness fluctuation parameter can reflect the presence of a power frequency light source in the ambient light surrounding the captured image, the presence of a power frequency light source can be confirmed when there is a relatively stable fluctuation depth and period in the ambient light. Because the presence of a power frequency light source in the ambient light may cause water ripples in the short exposure subframes, it is necessary to first calculate the ambient light brightness fluctuation parameter when determining whether water ripples will appear in the short exposure subframes. The calculation methods for the brightness fluctuation parameter include, but are not limited to, traditional algorithms and machine learning algorithms.

[0103] In one implementation, after determining the ambient light brightness fluctuation parameters, temporal filtering is applied to these parameters to prevent single-frame detection errors from affecting the detection results of power frequency interference when detecting the presence of power frequency light sources in the ambient light, which could cause water ripples in short-exposure sub-frames of consecutive initial images. For example, when calculating the ambient light brightness fluctuation parameters in the current frame's initial image, the current frame's brightness fluctuation parameters are compared with those of the preceding consecutive initial images. If the difference between the current frame's brightness fluctuation parameters and those of the other frames is too large, a single-frame detection error is identified. The brightness fluctuation parameters of the current frame's initial image are then removed, and the average of the brightness fluctuation parameters of the preceding consecutive initial images is used as the current frame's brightness fluctuation parameters. This prevents water ripple detection errors in the current frame when the current frame's brightness fluctuation parameters are abnormally detected. Temporal filtering includes, but is not limited to, sliding window voting mechanisms, positional algorithms, and neural network temporal algorithms.

[0104] S230: If the change value of the fluctuation depth is less than the first change threshold and the fluctuation depth is greater than the set depth value, and the change value of the fluctuation period is less than the second change threshold and the fluctuation period is greater than the exposure time of each row of pixels in the short exposure subframe, then it is determined that there are water ripples in the short exposure subframe.

[0105] Since the fluctuation depth and fluctuation period in the brightness fluctuation parameters of ambient light can reflect whether there is a power frequency light source in the ambient light, if the change value of the fluctuation depth of the ambient light is less than the first change threshold, that is, the change value of the light intensity of the fluctuation depth of the ambient light in any adjacent period is less than the first change threshold, and the change value of the fluctuation period of the ambient light is less than the second change threshold, that is, the change value of the duration of the fluctuation period of the ambient light in any adjacent period is less than the second change threshold, then it is determined that there is a light source with a stable fluctuation depth and a stable fluctuation period in the ambient light, that is, there is a stable power frequency light source in the ambient light.

[0106] It should be understood that not all power frequency light sources will cause water ripples in the short exposure subframes. For example, if the fluctuation depth of the power frequency light source is less than or equal to the set depth value, that is, when the power frequency light source has only a very small fluctuation depth, the power frequency light source may not cause water ripples in the short exposure subframes.

[0107] For example, based on the ambient light brightness fluctuation parameter, if the detected change value of the ambient light fluctuation depth is less than the first change threshold and the fluctuation depth is greater than the set depth value, and the change value of the ambient light fluctuation period is less than the second change threshold and the fluctuation period is greater than the exposure time of each row of pixels in the short exposure subframe, then it is determined that the ambient light in which the captured image is located will cause significant power frequency interference to the captured short exposure subframe, that is, there are water ripples in the short exposure subframe.

[0108] S240: If water ripples exist in the short exposure subframe, determine the exposure compensation value of the short exposure subframe based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe.

[0109] Specifically, if it is determined that water ripples exist in the short exposure subframe, indicating the presence of a power frequency light source in the ambient light, and that this power frequency light source has a power frequency effect on the short exposure subframe, then based on the obtained exposure values ​​of the standard subframe, the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe, the required exposure compensation value for eliminating the water ripples in the short exposure subframe is determined. Here, the exposure compensation value refers to the increase or decrease in exposure of the short exposure subframe. Positive exposure compensation will brighten the corresponding exposure area of ​​the short exposure subframe, while negative exposure compensation will darken the corresponding exposure area of ​​the short exposure subframe.

[0110] The ambient light in which the image is captured may be mixed light. Mixed light has a complex composition, containing multiple frequency components, which can easily cause signal distortion or noise enhancement when performing spectral analysis on the image data. Under natural light or non-uniform lighting conditions, the brightness and contrast of the image may change due to variations in illumination and the presence of shadows, resulting in a deterioration in the Fourier transform effect. In this embodiment, in calculating the exposure compensation value required to remove water ripples in short-exposure subframes, multi-frame fusion data from a single-frame progressive high dynamic range imaging mode is used. Simultaneously, image data is filtered during the calculation process, improving the Fourier transform effect of the image under mixed light and enhancing image quality.

[0111] S250: Exposure compensation for short exposure subframes based on exposure compensation values.

[0112] When water ripples exist in short-exposure subframes, exposure compensation is applied to the short-exposure subframes in the initial image based on the exposure compensation value of the short-exposure subframes to avoid water ripples appearing in short-exposure subframes obtained by exposing multiple pixels. Furthermore, in this embodiment, a long-exposure subframe adjacent to the short-exposure subframe is used as a standard subframe. The required exposure compensation value for exposing the short-exposure subframe is determined based on the image data of the standard subframe and the short-exposure subframe, thus eliminating the need to additionally adjust the exposure parameters and frame length of the short-exposure subframe when eliminating water ripples. In this way, water ripples in short-exposure subframes can be removed, improving the image quality of the short-exposure subframes, while avoiding the image quality degradation caused by modifying exposure parameters and the loss of smoothness and power consumption caused by modifying the frame length.

[0113] In one implementation, after exposure compensation of the short exposure subframe, brightness smoothing processing is performed on the exposure-compensated short exposure subframe based on the image data of the standard subframe to avoid discontinuous brightness changes of multiple rows of pixels in the exposure-compensated short exposure subframe, thereby further improving the imaging quality after exposure compensation of the short exposure subframe.

[0114] S260: The long exposure subframes in the initial image and the short exposure subframes in the initial image after exposure compensation are fused to obtain the output image corresponding to the initial image.

[0115] Specifically, after exposing the short-exposure subframes with water ripples in the initial image to compensation, the long-exposure subframes in the initial image and the exposing-compensated short-exposure subframes in the initial image are fused together to obtain the output image corresponding to each frame of the initial image in the single-frame progressive high dynamic range imaging mode.

[0116] If the standard subframe adjacent to the short exposure subframe is a long exposure subframe in the previous initial image, exposure compensation is performed on the short exposure subframe in the initial image based on the standard subframe. After removing the water ripples in the short exposure subframe, in order to obtain the output image corresponding to the initial image, it is also necessary to obtain the long exposure subframe in the initial image as the image frame to be fused with the short exposure subframe.

[0117] Furthermore, when removing water ripples and fusing images in each initial frame, image fusion technology can be used to fuse the output images corresponding to multiple initial frames to form multiple consecutive images for output, thereby improving the imaging quality of electronic devices when recording or previewing images.

[0118] S270: If there are no water ripples in the short exposure subframe, fuse the long exposure subframe and the short exposure subframe in the initial image to obtain the output image corresponding to the initial image.

[0119] In this embodiment, if it is determined that there are no water ripples in the short exposure subframe based on the brightness fluctuation parameters of the ambient light, for example, if the fluctuation depth of the ambient light is less than or equal to a set fluctuation depth, then there are no water ripples in the short exposure subframe.

[0120] Specifically, when no water ripples are detected in the short exposure subframe, the long exposure subframe and the short exposure subframe in the initial image are fused to obtain the output image corresponding to the initial image. If the standard subframe adjacent to the short exposure subframe is a long exposure subframe in the preceding initial image, the long exposure subframe in the initial image also needs to be obtained as the image frame to be fused with the short exposure subframe.

[0121] In the solution provided in this application embodiment, when the imaging mode of the electronic device is single-frame progressive high dynamic range imaging, the ambient light brightness fluctuation parameters are obtained based on the exposure and brightness characteristics of the short exposure subframe and the exposure and brightness characteristics of the standard subframe adjacent to the short exposure subframe, thereby determining whether water ripples will appear in the short exposure subframe. Simultaneously, when water ripples exist in the short exposure subframe, the required exposure compensation value for exposure compensation of the short exposure subframe is determined based on the exposure and brightness characteristics of the short exposure subframe and the standard subframe adjacent to the short exposure subframe. Exposure compensation of the short exposure subframe based on the exposure compensation value eliminates the water ripples in the short exposure subframe caused by the presence of power frequency light sources in the ambient light, improving the imaging quality of the electronic device under single-frame progressive high dynamic range imaging. Furthermore, since there is no need to additionally adjust the exposure parameters and frame length of the short exposure subframe when eliminating water ripples, the problems of image quality degradation caused by modifying exposure parameters and loss of smoothness and power consumption caused by modifying frame length are avoided.

[0122] In one implementation, combining Figure 4 Brightness fluctuation parameters include fluctuation depth and fluctuation period, which can be used to determine whether water ripples exist. For example... Figure 7 As shown, the determination of ambient light brightness fluctuation parameters based on the exposure of standard subframes, the exposure of short-exposure subframes, the brightness characteristics of standard subframes, and the brightness characteristics of short-exposure subframes can specifically include S221 to S223:

[0123] S221: Based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe, the brightness difference characteristics between the standard subframe and the short exposure subframe are obtained.

[0124] This application uses brightness features, including brightness fluctuation features, as an example to illustrate the process of image processing based on brightness fluctuation features.

[0125] Among them, the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, the brightness characteristics of the short exposure subframe, and the brightness difference characteristics satisfy the following:

[0126]

[0127] Where Delta is the brightness difference feature, L1 is the exposure of the standard subframe, S1 is the exposure of the short exposure subframe, L2 is the brightness feature of the standard subframe, S2 is the brightness feature of the short exposure subframe, and L2 and S2 are the brightness fluctuation features in the brightness features.

[0128] First, determine the exposure ratio, which is the ratio of the brightness value of the standard subframe to the exposure value of the short exposure subframe. The exposure ratio must satisfy the following:

[0129]

[0130] Where, ratio expo This represents the exposure ratio.

[0131] Furthermore, based on the exposure ratio, the brightness fluctuation curves of the short-exposure subframes are aligned with the brightness fluctuation curves of the standard subframes. After alignment, the brightness characteristics of the short-exposure subframes satisfy the following:

[0132] S2′=S2*ratio expo ;

[0133] S2′ represents the brightness feature of the short-exposure subframe after alignment.

[0134] Finally, based on the brightness characteristics of the standard subframe and the brightness characteristics of the aligned short-exposure subframe, differential processing is performed to determine the brightness difference characteristics between the standard subframe and the short-exposure subframe. The brightness difference characteristics satisfy:

[0135]

[0136] S222: Determine the fluctuation depth based on the second moment of the brightness difference feature and the first moment of the brightness feature of the standard subframe.

[0137] Specifically, the brightness difference feature between the standard subframe and the short exposure subframe is used as the brightness fluctuation feature of the difference frame between the standard subframe and the short exposure subframe. Since the second moment calculation involves the variance of the pixel brightness value in the image and the brightness value of the surrounding area, it can quantify the local brightness fluctuation. In adjacent standard subframes and short exposure subframes, if there is a power frequency fluctuation in the ambient light, the second moment of the brightness difference feature between the standard subframe and the short exposure subframe will reflect a large brightness fluctuation change.

[0138] Furthermore, for the brightness characteristics of the standard subframe, the first moment of the brightness characteristics of the standard subframe is calculated. Since the first moment (i.e. the mean) represents the average brightness of all pixels in the standard subframe, if the light source has relatively stable fluctuations within the standard subframe, the first moment of the brightness characteristics of the standard subframe can be used as a stable brightness fluctuation benchmark.

[0139] Based on this, by comparing the second moment of the brightness difference feature with the first moment of the brightness feature of the standard subframe, the depth of brightness fluctuation when there is brightness fluctuation in the ambient light can be quantified.

[0140] S223: Determine the fluctuation period based on the brightness difference characteristics.

[0141] Specifically, when there is a power frequency light source in the ambient light, the fluctuation of the power frequency light source manifests as line brightness fluctuations in the scanning direction perpendicular to the pixel row in the standard sub-frame and short exposure sub-frame.

[0142] In one implementation, when the brightness difference features of the standard subframe and the short exposure subframe are obtained, the fluctuation period of the ambient light brightness is determined based on the periodic increase or decrease of the brightness value of the pixel row in the brightness difference features.

[0143] In one implementation, the Fourier transform algorithm is used to calculate the brightness difference characteristics, thereby obtaining the fluctuation period of the ambient light brightness.

[0144] In this embodiment, based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness features of the standard subframe, and the brightness features of the short exposure subframe, the brightness difference features between the standard subframe and the short exposure subframe are obtained through traditional calculation methods. Without the need to use complex deep learning algorithms (such as convolutional neural networks) or machine learning algorithms for calculation, the fluctuation depth and fluctuation period in the brightness fluctuation parameters of the current ambient light can be accurately obtained, thereby saving the computing power of electronic devices and improving the processing speed of electronic devices in the image processing process.

[0145] In one implementation, when it is determined that water ripples exist in the short-exposure subframe, the exposure ratio and brightness feature ratio between the standard subframe and the short-exposure subframe can be determined based on the exposure of the standard subframe, the exposure of the short-exposure subframe, the brightness features of the standard subframe, and the brightness features of the short-exposure subframe. The exposure compensation value for the short-exposure subframe is then determined based on these ratios, and exposure compensation is applied to the short-exposure subframe. For example, combining... Figure 4 In one possible implementation, such as Figure 8 As shown, S240 may specifically include S241 to S243:

[0146] S241: Determine the exposure ratio, which is the ratio of the exposure of the standard subframe to the exposure of the short exposure subframe.

[0147] Specifically, the exposure ratio is determined. The exposure ratio is the ratio of the brightness value of the standard subframe to the exposure value of the short exposure subframe. The exposure ratio satisfies the following:

[0148]

[0149] Where, ratio expo The exposure ratio is L1, where L1 is the exposure of the standard subframe and S1 is the exposure of the short exposure subframe.

[0150] S242: Based on the brightness features of the standard subframe and the brightness features of the short exposure subframe, determine the ratio of the brightness features of each row of pixels between the standard subframe and the short exposure subframe.

[0151] Specifically, based on the brightness features of the standard subframe and the short-exposure subframe, the ratio of brightness features of each row of pixels between the standard subframe and the short-exposure subframe is determined, and the brightness feature ratio satisfies:

[0152]

[0153] Where, ratio feature L2 is the brightness feature ratio, and S2 is the brightness feature of the standard subframe and S2 is the brightness feature of the short exposure subframe.

[0154] In one implementation, the brightness feature ratio is smoothed by filtering to remove local glitches and outliers, thereby ensuring the accuracy of the exposure compensation value of the short exposure subframe determined based on the smoothed brightness feature ratio. This improves the exposure compensation effect of the subsequent exposure compensation value obtained based on the smoothed data on the short exposure subframe.

[0155] For example, if the difference between the luminance feature ratio and the mean of the luminance feature ratios of any corresponding pixel row between the standard subframe and the short exposure subframe is greater than a set difference, smoothing the luminance feature ratios can remove the luminance feature ratios corresponding to that pixel row.

[0156] For example, when the exposure ratio between the standard subframe and the short exposure subframe is 3, there exists a row of pixels with normal brightness in the standard subframe but a large number of completely black pixels in the short exposure subframe. The brightness feature ratio of this row between the standard and short exposure subframes is 6. In this case, the brightness feature ratio of the row is much larger than the exposure ratio between the standard and short exposure subframes. If the exposure compensation value of the short exposure subframe is determined based on the filtered brightness feature ratio, it is easy to result in an excessively high exposure compensation value. By processing the brightness feature ratio of each row of pixels using the exposure ratio, the non-linearity in brightness caused by overexposed and completely black areas in multiple pixels in the standard and / or short exposure subframes is removed. This ensures the accuracy of the exposure compensation value of the short exposure subframe obtained based on the exposure ratio and the brightness feature ratio of each row of pixels after removing overexposed and completely black pixel rows.

[0157] S243: Determine the pixel compensation value for each row of pixels based on the exposure ratio and the brightness feature ratio of each row of pixels.

[0158] Specifically, based on the exposure ratio and brightness feature ratio between the standard subframe and the short exposure subframe, the pixel compensation value required for each row of pixels is determined when performing brightness compensation on multiple pixel rows of the short exposure subframe to eliminate water ripples in the short exposure subframe.

[0159] Among them, the brightness compensation algorithms for short exposure subframes include, but are not limited to, traditional brightness compensation algorithms and neural network algorithms: Traditional brightness compensation algorithms are usually based on fixed rules or statistical models, such as histogram equalization or gamma correction; Neural network algorithms utilize deep learning technology, training a large number of image samples to allow the neural network to automatically learn how to adapt to various complex lighting conditions and optimize image brightness. This algorithm can capture more complex lighting change patterns and can perform exposure compensation for images in real time.

[0160] For example, when performing exposure compensation on a short exposure subframe, the pixel compensation range of multiple rows of pixels in the short exposure subframe can be determined first based on the peak and trough values ​​of the brightness feature ratio. Then, within this pixel compensation range, the pixel compensation value of the middle multiple rows of pixels can be determined by linear interpolation. Assuming that the pixel compensation range is [-3, 3] and the preset step size is 1, the pixel compensation value of the multiple rows of pixels can be -3, -2, -1, 0, 1, 2, and 3.

[0161] In one implementation, after determining the pixel compensation value for each row of pixels, the pixel compensation values ​​for multiple rows of pixels in the short exposure subframe can be smoothed by filtering to obtain smooth pixel compensation values ​​for multiple rows of pixels, thereby improving the imaging effect of the output image after exposure compensation of the short exposure subframe.

[0162] In this embodiment, when eliminating water ripples in short exposure subframes, during the process of determining the exposure amount, exposure amount, brightness characteristics of the standard subframe, and brightness characteristics of the short exposure subframe based on the image data of the standard subframe and the short exposure subframe, the exposure compensation value required to remove water ripples in the short exposure subframe is calculated based on the image effect of the short exposure subframe. Compared with the related technologies that use methods to modify exposure parameters and frame length, since the related technologies need to be adjusted according to the parameters of ambient light, the water ripple removal effect under mixed power frequency light sources is not good. The solution of this application is not limited by the type and number of light sources, and can be better applied to the elimination of water ripples in short exposure subframes under mixed light illumination environments, thereby improving the imaging quality of the output image after image processing. The example above illustrates that when water ripples are determined to exist in a short exposure subframe, the exposure ratio and brightness feature ratio between the standard subframe and the short exposure subframe can be determined based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness features of the standard subframe, and the brightness features of the short exposure subframe, in order to determine the exposure compensation value of the short exposure subframe.

[0163] In another implementation, the exposure compensation value for the short exposure subframe can also be determined using the first exposure compensation model. For example, combined with... Figure 4 ,like Figure 9 As shown, S240 may also include:

[0164] S244: Input the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe into the first exposure compensation model to obtain the exposure compensation value output by the first exposure compensation model.

[0165] The first exposure compensation model is obtained by training the initial model using the first training sample set. The first training sample set includes target parameters of adjacent long exposure subframes and short exposure subframes under various lighting sources, as well as exposure compensation values ​​of short exposure subframes.

[0166] In this embodiment, when obtaining the exposure compensation value of the short exposure subframe, a pre-trained first exposure compensation model is used. The exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness features of the standard subframe, and the brightness features of the short exposure subframe are used as input data for the first exposure compensation model. This allows for the acquisition of accurate exposure compensation values ​​output by the exposure compensation model, thereby improving the compensation effect when performing exposure compensation on the short exposure subframe.

[0167] Combination Figure 10 , Figure 10 This is a schematic flowchart illustrating a process for obtaining a first exposure compensation model, provided as an embodiment of this application. The process for obtaining the first exposure compensation model includes:

[0168] S310: Based on multiple initial images, obtain multiple sets of reference image pairs.

[0169] Among them, multiple sets of reference image pairs correspond one-to-one with multiple sets of initial images, and each set of reference image pairs includes adjacent long exposure subframes and short exposure subframes.

[0170] Multiple initial images are obtained by capturing images under various power frequency light source illumination environments. Each initial image corresponds one-to-one with a different power frequency light source, including multiple single power frequency light sources and multiple mixed power frequency light sources. The first model training module obtains multiple reference image pairs based on adjacent long-exposure subframes and short-exposure subframes in each initial image set. Each reference image pair corresponds one-to-one with a different power frequency light source.

[0171] S320: Cleaning and enhancing multiple sets of reference image pairs.

[0172] In this embodiment, the first model training module cleans and enhances multiple sets of reference image pairs in at least the following ways: removing reference image pairs that do not have water ripples from multiple sets of reference image pairs, and removing some reference image pairs with the same power frequency.

[0173] One issue is that the distribution of power frequency for various power frequency light sources is uneven. Most power frequency light sources use frequencies commonly used in real life, such as 50 Hz. This results in a limited number of reference image pairs collected at less common power frequency frequencies. Consequently, after training the exposure compensation model based on multiple unprocessed reference image pairs, the exposure compensation effect of the model on images at less common power frequency frequencies is not ideal. By removing reference image pairs with the same power frequency, the model can achieve balanced processing of multiple reference image pairs, thereby improving the exposure compensation effect of the trained exposure compensation model on images at various power frequency light sources.

[0174] S330: Remove the abnormal image data from each set of reference image pairs to obtain each set of reference image pairs after anomaly processing.

[0175] In this embodiment, removing abnormal data from each set of reference image pairs includes, but is not limited to, removing overexposed pixels and black pixels from each set of reference image pairs.

[0176] S340: Based on each set of reference image pairs after anomaly processing, determine the brightness characteristics of the long exposure subframes and the short exposure subframes in each set of reference image pairs.

[0177] S350: Determine the exposure compensation value for each set of reference image pairs based on the cleaned and enhanced multiple sets of reference image pairs.

[0178] Specifically, multiple sets of reference image pairs after cleaning and enhancement are labeled using artificial intelligence methods or manual annotation, and exposure compensation values ​​for short exposure subframes are set for each set of reference image pairs.

[0179] S360: Determine the first exposure compensation model based on the brightness characteristics of the medium-long exposure subframes and the short exposure subframes of each set of reference image pairs, as well as the exposure compensation value of each set of reference image pairs.

[0180] Specifically, during the training of the first exposure compensation model, the first training sample set includes data from multiple sets of reference image pairs, including the brightness features of long-exposure subframes and short-exposure subframes in each set of reference image pairs, as well as the exposure compensation value for each set of reference image pairs. Based on the brightness features of long-exposure and short-exposure subframes in each set of reference image pairs, and the exposure compensation value for each set of reference image pairs, the brightness features of long-exposure and short-exposure subframes are used as input data, and the exposure compensation value for each set of reference image pairs is used as output data. A first exposure compensation model is then trained using a graphics processing unit (GPU) or a central processing unit (CPU) to obtain the relationship between the brightness features of long-exposure and short-exposure subframes and the exposure compensation value for short-exposure subframes.

[0181] In this embodiment, the training set of the first exposure compensation model is obtained by using image data of multiple sets of reference image pairs after anomaly processing, cleaning and enhancement, so that the first exposure compensation model can be applied to the exposure compensation of images under various power frequency light sources, thereby improving the exposure compensation effect of the trained first exposure compensation model.

[0182] Embodiments of this application also provide an image processing apparatus for implementing the image processing method provided in the embodiments of this application, so as to eliminate water ripples in an image and improve the imaging quality of the image output by the electronic device. Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application.

[0183] In this embodiment of the application, the image processing device 400 includes at least a target parameter acquisition module 410, an exposure compensation value determination module 420, and a first exposure compensation module 430.

[0184] The target parameter acquisition module 410 includes at least a data receiving module 411, a statistics module 412, an automatic exposure module 413, a feature calculation module 414, and an anomaly handling module 415. The data receiving module 411 is communicatively connected to the camera unit 190 in the electronic device. After the camera unit 190 captures images to form image data of the initial image, the data receiving module 411 is used to acquire the image data of the initial image from the camera unit 190.

[0185] The statistics module 412 is used to acquire image data of the initial image from the data receiving module 411, namely, image data of short-exposure subframes and image data of standard subframes. For the short-exposure subframes in the initial image, the statistics module 412 performs statistics based on the image data of the short-exposure subframes to obtain a histogram of the gray-level distribution of the short-exposure subframes, and transmits the histogram to the automatic exposure module 413, so that the automatic exposure module 413 can determine the brightness of the ambient light of the shooting environment based on the received histogram, and output the exposure of the standard subframes and the exposure of the short-exposure subframes.

[0186] The feature calculation module 414 is used to acquire image data of standard subframes and short-exposure subframes from the data receiving module 411, and calculate the brightness features of the standard subframes and the short-exposure subframes based on the image data of the short-exposure subframes. The anomaly processing module 415 is located between the data receiving module 411 and the feature calculation module 414. The anomaly processing module 415 is used to preprocess the image data of the standard subframes and short-exposure subframes, removing image data of pixels in dead black areas and overexposed areas in the standard subframes and short-exposure subframes. This ensures that after the feature calculation module 414 determines the brightness features of the standard subframes and the short-exposure subframes based on the preprocessed image data, the exposure compensation value of the short-exposure subframes determined by the image processing device 400 based on the brightness features of the standard subframes and the short-exposure subframes has a better compensation effect on the short-exposure subframes.

[0187] The water ripple detection module 440 includes a detection submodule 441 and a filtering module 442. The detection submodule 441 can determine the ambient light brightness fluctuation parameters based on the exposure of the standard subframe and the short exposure subframe obtained from the automatic exposure module 413, and the brightness features of the standard subframe and the short exposure subframe obtained from the feature calculation module 414. The filtering module 442 is used to perform temporal filtering on the brightness fluctuation parameters to avoid single-frame detection errors affecting the detection results of power frequency interference when water ripples appear in short exposure subframes of consecutive initial images caused by the presence of power frequency light sources in the ambient light. Furthermore, the filtering module 442 is also used to determine whether water ripples exist in short exposure subframes based on the ambient light brightness fluctuation parameters.

[0188] If the filtering module 442 detects water ripples in the short exposure subframe, the exposure compensation value determination module 420 determines the exposure compensation value required for the short exposure subframe to eliminate the water ripples based on the exposure values ​​of the standard subframe and the short exposure subframe obtained from the automatic exposure module 413, and the brightness features of the standard subframe and the short exposure subframe obtained from the feature calculation module 414. The exposure compensation module 430 performs exposure compensation on the short exposure subframe in the initial image based on the exposure compensation value of the short exposure subframe to remove the water ripples in the short exposure subframe. The feature calculation module 414 and the first exposure compensation module 430 can be hardware modules, such as a neural network processing unit (NPU), a digital signal processor (DSP), or the NEON instruction set supporting single instruction multiple data under an advanced RISC machine (ARM) architecture; no restrictions are imposed here.

[0189] If the filtering module 442 detects that there are no water ripples in the short exposure subframe, or after the first exposure compensation module 430 completes the exposure compensation for the short exposure subframe, the image fusion module 450 performs fusion processing on the long exposure subframe and the short exposure subframe in the initial image to obtain the output image corresponding to each frame of the initial image in the single-frame progressive high dynamic range imaging mode, and sends the output image to the subsequent image signal processing module 460. The image signal processing module 460 performs further image processing on the output image formed by the fusion processing of the long exposure subframe and the short exposure subframe, including but not limited to adjusting the dynamic range of the image, color correction of the image, image sharpening, etc., and outputs the output image after further image processing by the image signal processing module 460 to the display unit 150 which is communicatively connected to the image signal processing module 460, so as to improve the imaging quality of the image at the display unit 150 when the electronic device is recording or previewing the image.

[0190] The image processing apparatus 400 described above can determine the exposure ratio and brightness feature ratio between the standard subframe and the short exposure subframe based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe, thereby determining the exposure compensation value of the short exposure subframe. Embodiments of this application also provide another image processing apparatus that can further determine the exposure compensation value of the short exposure subframe using an exposure compensation model.

[0191] In one implementation, such as Figure 12 As shown, Figure 12This is a schematic diagram of another image processing apparatus provided in an embodiment of this application. The image processing methods of the multiple modules in the image processing apparatus 400 can be referred to the above. Figure 11 The specific details of the corresponding embodiments will not be repeated here. In this embodiment, the exposure compensation value determination module 420 and the first exposure compensation module 430 are deployed in the setting processor.

[0192] In this embodiment, the processor can be a graphics processing unit (GPU) or a central processing unit (CPU), and there is no limitation thereto. The exposure compensation value determination module 420 is also used to communicate with the first model training module 470. The exposure compensation value determination module 420 is used to obtain the first exposure compensation model 471 trained by the first model training module 470 using the training sample set, and store the first exposure compensation model 471 in the register of the exposure compensation value determination module 420.

[0193] In this embodiment, the exposure compensation value determination module 420 calls the pre-stored first exposure compensation model 471 based on the brightness features of the standard subframe and the brightness features of the short exposure subframe obtained from the feature calculation module 414, and uses the brightness features of the standard subframe and the brightness features of the short exposure subframe as input data of the first exposure compensation model 471 to obtain the exposure compensation value output by the first exposure compensation model 471.

[0194] Furthermore, the first exposure compensation module 430 obtains the exposure compensation value of the short exposure subframe from the exposure compensation value determination module 420, and performs exposure compensation on the short exposure subframe based on the exposure compensation value to remove the water ripples in the short exposure subframe. Additionally, the first exposure compensation module 430 transmits the water ripple-removed short exposure subframe to the image fusion module 450 in the image processing device 400 via inter-core communication, so that the image fusion module 450 can perform fusion processing based on the long exposure subframe in the initial image and the exposure-compensated short exposure subframe in the initial image to obtain the output image corresponding to the initial image.

[0195] In one implementation, the first model training module 470 can be used to execute S310 to S360 to train the initial model to obtain the first exposure compensation model.

[0196] This application also provides another image processing method, such as... Figure 13 As shown, Figure 13This is a schematic flowchart of another image processing method provided in an embodiment of this application. This image processing method can obtain short-exposure sub-frames after removing water ripples based on a second exposure compensation model, including steps S510 to S520:

[0197] S510: Obtain target parameters based on the short exposure subframes in the initial image and the standard subframes adjacent to the short exposure subframes.

[0198] The target parameters include the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. The standard subframe adjacent to the short exposure subframe is either a long exposure subframe in the initial image or a long exposure subframe in the previous initial image.

[0199] In this embodiment, the specific implementation steps of S510 can be referred to the content of the above embodiments, and will not be repeated here.

[0200] S520: If the short exposure subframe has water ripples, input the short exposure subframe and the standard subframe adjacent to the short exposure subframe into the second exposure compensation model to obtain the short exposure subframe after removing water ripples output by the second exposure compensation model.

[0201] The second exposure compensation model is obtained by training the initial model using the second training sample set, which includes adjacent long exposure subframes under various lighting sources and short exposure subframes without water ripples.

[0202] In this embodiment, when performing exposure compensation on short exposure subframes, a pre-trained second exposure compensation model is used. The standard subframe and the short exposure subframe are used as input data for the second exposure compensation model, and the short exposure subframe after removing water ripples is output by the second exposure compensation model. This reduces the process of performing exposure compensation on short exposure subframes based on exposure compensation values ​​in the image processing device and improves the speed of image processing.

[0203] Combination Figure 14 , Figure 14 This is a schematic diagram illustrating a process for obtaining a second exposure compensation model, provided as an embodiment of this application. The training process of the second exposure compensation model includes:

[0204] S610: Based on multiple initial images, obtain multiple sets of reference image pairs.

[0205] In this embodiment, the specific implementation of S610 can be referred to the content of S310 in the above embodiment, and will not be repeated here.

[0206] S620: Cleaning and enhancing multiple sets of reference image pairs.

[0207] In this embodiment, the method of cleaning and enhancing multiple sets of reference image pairs includes at least: removing reference image pairs with water ripples from multiple sets of reference image pairs, and removing some reference image pairs with the same power frequency.

[0208] S630: Based on multiple sets of cleaned and enhanced reference image pairs, obtain short-exposure subframes in each set of reference image pairs that do not contain water ripples.

[0209] Specifically, multiple sets of reference image pairs after cleaning and enhancement are labeled using artificial intelligence methods or manual annotation. The exposure compensation value of the short exposure subframes in each set of reference image pairs is set, and the short exposure subframes are exposed to obtain short exposure subframes without water ripples in each set of reference image pairs.

[0210] S640: Determine the second exposure compensation model based on the long exposure subframes and short exposure subframes without water ripples in each set of reference image pairs.

[0211] Specifically, based on the long exposure subframes and short exposure subframes without water ripples in each set of reference image pairs, the image data of the long exposure subframes and short exposure subframes without water ripples in each set of reference image pairs are used as input data, and the image data of the short exposure subframes without water ripples in each set of reference image pairs are used as output data. The second exposure compensation model is trained using a graphics processor or a central processing unit.

[0212] In this embodiment, the training set of the second exposure compensation model is obtained using image data from multiple sets of reference image pairs that have undergone anomaly processing, cleaning, and enhancement. This allows the second exposure compensation model to be applicable to exposure compensation of images under various power frequency light sources, thereby improving the exposure compensation effect of the trained second exposure compensation model. Furthermore, using short exposure subframes after removing water ripples as the output data of the second exposure compensation model reduces the process of performing exposure compensation on short exposure subframes based on exposure compensation values ​​in the image processing device, thus improving the speed of image processing.

[0213] After exposing the short exposure subframes with water ripples in the initial image to perform exposure compensation, the long exposure subframes in the initial image and the exposing-compensated short exposure subframes in the initial image are fused together to obtain the output image corresponding to each frame of the initial image in the single-frame progressive high dynamic range imaging mode. This results in the output of multiple consecutive images, thereby improving the imaging quality of electronic devices when recording or previewing images.

[0214] like Figure 15 As shown, Figure 15This is a schematic diagram of another image processing apparatus provided in an embodiment of this application, used to implement the image processing method provided in this application embodiment to eliminate water ripples in an image and improve the imaging quality of the image output by the electronic device. In this embodiment, the image processing apparatus 400 includes at least a target parameter acquisition module 410 and a second exposure compensation module 480. The image processing methods of the multiple modules in the image processing apparatus 400 can be referred to the above. Figure 11 The specific details of the corresponding embodiments will not be repeated here.

[0215] In this embodiment, the second exposure compensation module 480 is also used to communicate with the second model training module 490. The second exposure compensation module 480 is used to obtain the second exposure compensation model 491 trained by the second model training module 490 using the training sample set, and store the second exposure compensation model 491 in the register of the second exposure compensation module 480.

[0216] The second exposure compensation module 480 is communicatively connected to the data receiving module 411. The second exposure compensation module 480 is used to obtain the standard subframe and the short exposure subframe from the data receiving module 411, and call the pre-stored second exposure compensation model 491. The standard subframe and the short exposure subframe are used as the input data of the second exposure compensation model 491 to obtain the short exposure subframe after removing the water ripples output by the second exposure compensation model 491.

[0217] Furthermore, the second exposure compensation module 480 transmits the short exposure subframe after removing water ripples to the image fusion module 450 in the image processing device 400 via inter-core communication, so that the image fusion module 450 can perform fusion processing based on the long exposure subframe in the initial image and the short exposure subframe after exposure compensation in the initial image to obtain the output image corresponding to the initial image.

[0218] In one implementation, the second model training module 490 can be used to execute S610 to S640 to train the initial model to obtain the second exposure compensation model.

[0219] The image processing apparatus provided in this application embodiment can be integrated into the camera unit of an electronic device. In one possible implementation, such as... Figure 16 As shown, Figure 16 This is a schematic diagram illustrating an application scenario of an image processing device provided in an embodiment of this application. In this embodiment, the image processing device 400 is integrated into the camera unit 190.

[0220] Specifically, the camera unit 190 includes a camera module 191 and an image processor 192. The camera module 191 includes at least a plurality of image sensors 1911, which convert received light signals into electrical signals. The camera module 191 converts the electrical signals into digital signals to form image data of the initial image. The image processor 192 has a built-in image processing device 400 for implementing the image processing method in the above embodiments. That is, the image processing device 400 is an independent hardware structure or software embedded in the image processor 192. The camera module 191 and the image processor 192 are communicatively connected. The image processor 192 acquires the initial image from the camera module 191 and processes the received initial image in real time to form an output image for output, so that the initial image input to the camera module 191 no longer shows water ripples in the output image of the image processor 192 after processing.

[0221] Or, see Figure 17 The image processor can also be integrated into the camera module. Figure 17 This is a schematic diagram illustrating an application scenario of another image processing device provided in an embodiment of this application. In this embodiment, the image processing device 400 is located in the camera module 191.

[0222] Specifically, the camera module 191 includes multiple image sensors 1911 and an image processor 192. The image processor 192 has a built-in image processing device 400 for implementing the image processing method in the above embodiments. That is, the image processing device 400 is either an independent hardware subsystem built into the camera module 191 or a functional module built into the image processor 192, which is not limited here. The image processor 192 is used to process the initial image received from the camera module 191 in real time to avoid water ripples appearing in the output image of the camera module 191.

[0223] Or, see Figure 18 The image processing device, as a third-party independent system, is deployed on cloud devices. Figure 18 This is a schematic diagram illustrating an application scenario of another image processing device provided in an embodiment of this application.

[0224] Specifically, the cloud device 710 includes an image processor 192, which has a built-in image processing device 400 for implementing the image processing method in the above embodiments. That is, the image processing device 400 can be a functional module built into the image processor 192 or an independent hardware subsystem.

[0225] The cloud device 710 is communicatively connected to the local electronic device 720 via a first communication module 711. The local electronic device 720 includes a camera module 191, a sub-image processor 721, and a second communication module 722. The camera module 191 includes multiple image sensors 1911. The camera module 191 in the local electronic device 720 transmits the captured initial image to the sub-image processor 721 for processing. The sub-image processor 721 interacts with the cloud device 710 via the second communication module 722 to send the processed initial image to the cloud device 710.

[0226] In this embodiment, the image processor 192 in the cloud device 710 is used to process the received initial image in real time, that is, to use the computing power of the cloud device 710 to eliminate water ripples that may appear in the initial image, and to return the output image after water ripple elimination to the local electronic device 720 through the second communication module 722, thereby freeing up the computing power of the local electronic device 720 and avoiding the appearance of water ripples in the output image of the local electronic device 720.

[0227] This application embodiment can, according to the above method, exemplarily divide the image processing device 400 into functional modules. For example, the image processing device 400 may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0228] It should be understood that, under the illumination of power frequency light sources, when electronic devices are shooting in single-frame progressive high dynamic range imaging mode, if the displayed image of the electronic device is stable and there are no water ripples, by switching the power frequency light source in the lighting environment (i.e., changing the power frequency of the power frequency light source, but keeping the brightness of the power frequency light source unchanged), obvious water ripples will appear in the displayed image of the electronic device.

[0229] Furthermore, when the water ripples in the electronic device disappear, the exposure parameters and frame lengths of the long and short exposure subframes in the image of the electronic device when the water ripples appeared are recorded, and the exposure parameters and frame lengths of the long and short exposure subframes in the image of the electronic device when the water ripples disappear are also recorded. In addition, the exposure imaging mode of the electronic device is switched to normal exposure mode, and the exposure parameters of the electronic device are switched to the exposure parameters when the water ripples disappear under single-frame progressive high dynamic range imaging mode. If water ripples are detected in the displayed image of the electronic device, and the exposure parameters and frame lengths of the short exposure subframes in the electronic device remain unchanged before and after the water ripples disappear under single-frame progressive high dynamic range imaging mode, then it can be determined that the electronic device is using the image processing method provided in the embodiments of this application under single-frame progressive high dynamic range imaging mode.

[0230] Figure 19 This is a schematic diagram of the structure of another image processing apparatus provided in an embodiment of this application. (Refer to...) Figure 19 The image processing device 400 includes: a target parameter acquisition module 410, an exposure compensation value determination module 420, and a first exposure compensation module 430.

[0231] The target parameter acquisition module 410 is used to acquire target parameters based on the short exposure subframe in the initial image and the standard subframe adjacent to the short exposure subframe. The target parameters include the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. The standard subframe adjacent to the short exposure subframe is a long exposure subframe in the initial image or a long exposure subframe in the previous initial image.

[0232] The exposure compensation value determination module 420 is used to determine the exposure compensation value of the short exposure subframe based on the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe if water ripples exist in the short exposure subframe.

[0233] The first exposure compensation module 430 is used to compensate the exposure of short exposure subframes based on the exposure compensation value.

[0234] Optionally, the image processing device 400 also includes a water ripple determination module.

[0235] The water ripple determination module is used to determine the ambient light brightness fluctuation parameters based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. The brightness fluctuation parameters include the fluctuation depth and the fluctuation period. If the change value of the fluctuation depth is less than the first change threshold and the fluctuation depth is greater than the set depth value, and the change value of the fluctuation period is less than the second change threshold and the fluctuation period is greater than the exposure time of each row of pixels in the short exposure subframe, then it is determined that there are water ripples in the short exposure subframe.

[0236] Optionally, the water ripple determination module is further specifically used to obtain the brightness difference features between the standard subframe and the short exposure subframe based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness features of the standard subframe, and the brightness features of the short exposure subframe. The exposure of the standard subframe, the exposure of the short exposure subframe, the brightness features of the standard subframe, the brightness features of the short exposure subframe, and the brightness difference features satisfy the following:

[0237]

[0238] Where Delta is the brightness difference feature, L1 is the exposure of the standard subframe, S1 is the exposure of the short exposure subframe, L2 is the brightness feature of the standard subframe, and S2 is the brightness feature of the short exposure subframe; the fluctuation depth is determined based on the second moment of the brightness difference feature and the first moment of the brightness feature of the standard subframe; the fluctuation period is determined based on the brightness difference feature.

[0239] The target parameter acquisition module 410 is specifically used to remove pixels at the target position in the standard subframe and the short exposure subframe to obtain the preprocessed standard subframe and the preprocessed short exposure subframe. The target position is the position of the pixel whose brightness value does not meet the preset conditions in either the standard subframe or the short exposure subframe. The preset conditions include that the brightness value of the pixel is within a set brightness value range. The module determines the brightness characteristics of the standard subframe based on the preprocessed standard subframe and the brightness characteristics of the short exposure subframe based on the preprocessed short exposure subframe.

[0240] Optionally, the exposure compensation value includes the pixel compensation value of each row of pixels in the short exposure subframe. The exposure compensation value determination module 420 is specifically used to determine the exposure ratio, which is the ratio of the exposure of the standard subframe to the exposure of the short exposure subframe; based on the brightness characteristics of the standard subframe and the brightness characteristics of the short exposure subframe, determine the brightness feature ratio of each row of pixels corresponding to the standard subframe and the short exposure subframe; and based on the exposure ratio and the brightness feature ratio of each row of pixels, determine the pixel compensation value of each row of pixels.

[0241] The exposure compensation value determination module 420 is also specifically used to input the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe into the exposure compensation model to obtain the exposure compensation value output by the exposure compensation model. The exposure compensation model is obtained by training the initial model using a training sample set. The training sample set includes the target parameters of adjacent long exposure subframes and short exposure subframes under various lighting sources, as well as the exposure compensation value of the short exposure subframe.

[0242] Optionally, the image processing device 400 also includes an image fusion module.

[0243] The image fusion module is specifically used to fuse the long exposure subframes and short exposure subframes in the initial image if there are no water ripples in the short exposure subframes, to obtain the output image corresponding to the initial image.

[0244] The image fusion module is also specifically used to fuse the long exposure subframes in the initial image and the short exposure subframes in the initial image after exposure compensation processing to obtain the output image corresponding to the initial image.

[0245] It should be noted that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0246] Figure 20 This is a schematic diagram of the structure of another image processing apparatus provided in an embodiment of this application. (Refer to...) Figure 20 The image processing device 400 includes a target parameter acquisition module 410 and a second exposure compensation module 480.

[0247] The target parameter acquisition module 410 is used to acquire target parameters based on the short exposure subframe in the initial image and the standard subframe adjacent to the short exposure subframe. The target parameters include the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. The standard subframe adjacent to the short exposure subframe is a long exposure subframe in the initial image or a long exposure subframe in the previous initial image.

[0248] The second exposure compensation module 480 is used to handle short exposure subframes with water ripples. It inputs the short exposure subframe and the standard subframe adjacent to the short exposure subframe into the second exposure compensation model to obtain the short exposure subframe after removing the water ripples. The second exposure compensation model is obtained by training the initial model using a second training sample set. The second training sample set includes adjacent long exposure subframes under various lighting sources and short exposure subframes without water ripples.

[0249] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0250] Figure 21 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application. For example... Figure 21 As shown, the electronic device 100 includes, but is not limited to, a processor 110 and a memory 101.

[0251] The memory 101 described above is used to store the executable instructions of the processor 110. It is understood that the processor 110 is configured to execute instructions to implement the image processing method described in the above embodiments.

[0252] It should be noted that those skilled in the art will understand that Figure 19 The structure of the electronic device 100 shown does not constitute a limitation on the electronic device 100; the electronic device 100 may include, but is not limited to, other electronic devices. Figure 19 This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0253] Processor 110 is the control center of electronic device 100. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data of electronic device 100 by running or executing software programs and / or modules stored in memory 101, and by calling data stored in memory 101. Processor 110 may include one or more processing units. Optionally, processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 110.

[0254] The memory 101 can be used to store software programs and various data. The memory 101 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, application programs required by at least one functional module (such as a determination unit, a processing unit, etc.), etc. In addition, the memory 101 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0255] This application also provides a computer-readable storage medium storing a computer program for implementing the above-described image processing method.

[0256] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0257] This application also provides a computer program product, which includes computer instructions. When the computer instructions are executed on the processor 110 of the electronic device 100, the electronic device 100 performs the image processing method described above. It should be noted that when the instructions in the computer-readable storage medium or the instructions in the computer program product are executed by the processor 110 of the electronic device 100, they can implement the various processes of the above method embodiments and achieve the same technical effects as the above methods. To avoid repetition, they will not be described again here.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0259] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the classified units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0260] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0261] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Target parameters are obtained based on short exposure subframes in the initial image and standard subframes adjacent to the short exposure subframes. The target parameters include the exposure of the standard subframes, the exposure of the short exposure subframes, the brightness characteristics of the standard subframes, and the brightness characteristics of the short exposure subframes. The standard subframes adjacent to the short exposure subframes are long exposure subframes in the initial image or long exposure subframes in the previous initial image of the initial image. If the short exposure subframe has water ripples, the exposure compensation value of the short exposure subframe is determined based on the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe. Exposure compensation is applied to the short exposure subframe based on the exposure compensation value.

2. The method according to claim 1, characterized in that, Before determining the exposure compensation value of the short exposure subframe based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe if water ripples exist in the short exposure subframe, the method further includes: Based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe, the brightness fluctuation parameters of the ambient light are determined, and the brightness fluctuation parameters include the fluctuation depth and the fluctuation period. If the change value of the ripple depth is less than the first change threshold and the ripple depth is greater than the set depth value, the change value of the ripple period is less than the second change threshold and the ripple period is greater than the exposure time of each row of pixels in the short exposure subframe, then it is determined that the short exposure subframe contains the water ripple.

3. The method according to claim 2, characterized in that, The determination of ambient light brightness fluctuation parameters based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe includes: Based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe, a brightness difference feature between the standard subframe and the short exposure subframe is obtained. The exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, the brightness characteristics of the short exposure subframe, and the brightness difference feature satisfy the following: Wherein, Delta is the brightness difference feature, L1 is the exposure of the standard subframe, S1 is the exposure of the short exposure subframe, L2 is the brightness feature of the standard subframe, and S2 is the brightness feature of the short exposure subframe. The fluctuation depth is determined based on the second moment of the brightness difference feature and the first moment of the brightness feature of the standard subframe. The fluctuation period is determined based on the brightness difference characteristics.

4. The method according to claim 1, characterized in that, The steps of obtaining the brightness features of the standard subframe and the brightness features of the short exposure subframe based on the short exposure subframe in the initial image and the standard subframe adjacent to the short exposure subframe include: Remove the pixels at the target position in the standard subframe and the short exposure subframe to obtain the preprocessed standard subframe and the preprocessed short exposure subframe. The target position is the position of the pixel whose brightness value does not meet the preset condition in either the standard subframe or the short exposure subframe. The preset condition includes that the brightness value of the pixel is within a set brightness value range. The brightness characteristics of the standard subframe are determined based on the preprocessed standard subframe, and the brightness characteristics of the short exposure subframe are determined based on the preprocessed short exposure subframe.

5. The method according to any one of claims 1 to 4, characterized in that, The exposure compensation value includes the pixel compensation value for each row of pixels in the short exposure subframe. Determining the exposure compensation value of the short exposure subframe based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe includes: Determine the exposure ratio, which is the ratio of the exposure of the standard subframe to the exposure of the short exposure subframe; Based on the brightness features of the standard subframe and the brightness features of the short exposure subframe, the ratio of the brightness features of each row of pixels between the standard subframe and the short exposure subframe is determined. The pixel compensation value for each row of pixels is determined based on the exposure ratio and the brightness feature ratio of each row of pixels.

6. The method according to any one of claims 1 to 4, characterized in that, Determining the exposure compensation value of the short exposure subframe based on the exposure of the standard subframe, the exposure of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe includes: The exposure values ​​of the standard subframe, the short exposure subframe, the brightness features of the standard subframe, and the brightness features of the short exposure subframe are input into the first exposure compensation model to obtain the exposure compensation value output by the first exposure compensation model. The first exposure compensation model is obtained by training the initial model using a first training sample set. The first training sample set includes the target parameters of adjacent long exposure subframes and short exposure subframes under various lighting sources, as well as the exposure compensation value of the short exposure subframe.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: If the water ripples are not present in the short exposure subframe, the long exposure subframe and the short exposure subframe in the initial image are fused to obtain the output image corresponding to the initial image.

8. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The long exposure subframes in the initial image and the short exposure subframes in the initial image after exposure compensation are fused to obtain the output image corresponding to the initial image.

9. An image processing apparatus, characterized in that, The image processing device includes: The target parameter acquisition module is used to acquire target parameters based on the short exposure subframe in the initial image and the standard subframe adjacent to the short exposure subframe. The target parameters include the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness feature of the standard subframe, and the brightness feature of the short exposure subframe. The standard subframe adjacent to the short exposure subframe is a long exposure subframe in the initial image or a long exposure subframe in the previous initial image. An exposure compensation value determination module is used to determine the exposure compensation value of the short exposure subframe based on the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness characteristics of the standard subframe, and the brightness characteristics of the short exposure subframe if water ripples exist in the short exposure subframe. The first exposure compensation module is used to compensate the exposure of the short exposure subframe based on the exposure compensation value.

10. An image processing method, characterized in that, The method includes: Target parameters are obtained based on short exposure subframes in the initial image and standard subframes adjacent to the short exposure subframes. The target parameters include the exposure of the standard subframes, the exposure of the short exposure subframes, the brightness characteristics of the standard subframes, and the brightness characteristics of the short exposure subframes. The standard subframes adjacent to the short exposure subframes are long exposure subframes in the initial image or long exposure subframes in the previous initial image of the initial image. If the short exposure subframe has water ripples, the short exposure subframe and the standard subframe adjacent to the short exposure subframe are input into the second exposure compensation model to obtain the short exposure subframe after removing the water ripples output by the second exposure compensation model. The second exposure compensation model is obtained by training the initial model using a second training sample set. The second training sample set includes adjacent long exposure subframes under various lighting sources and short exposure subframes without water ripples.

11. An image processing apparatus, characterized in that, The image processing device includes: The target parameter acquisition module is used to acquire target parameters based on the short exposure subframe in the initial image and the standard subframe adjacent to the short exposure subframe. The target parameters include the exposure amount of the standard subframe, the exposure amount of the short exposure subframe, the brightness feature of the standard subframe, and the brightness feature of the short exposure subframe. The standard subframe adjacent to the short exposure subframe is a long exposure subframe in the initial image or a long exposure subframe in the previous initial image. The second exposure compensation module is used to input the short exposure subframe and the standard subframe adjacent to the short exposure subframe into the second exposure compensation model if the short exposure subframe has water ripples, so as to obtain the short exposure subframe after removing the water ripples output by the second exposure compensation model. The second exposure compensation model is obtained by training the initial model using a second training sample set. The second training sample set includes adjacent long exposure subframes under various lighting sources and short exposure subframes without water ripples.

12. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 8 and 10.

13. A computer-readable storage medium, characterized in that, When the computer-executable instructions stored in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device implements the method as described in any one of claims 1 to 8 and 10.

14. A computer program product, the computer program product comprising computer instructions, characterized in that, When the computer instructions are executed on the processor of the electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 8 and 10.

Images