Image processing method and device, electronic equipment, readable storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a readable storage medium and a program product. The method comprises the steps of obtaining a first image sequence in response to a photographing operation; the first image sequence comprises images with exposure parameters exceeding a target exposure parameter range; acquiring at least two frames of images from the first image sequence for fusion to obtain a first image; performing brightness smoothing processing on the first image to obtain a second image; and generating a dynamic image based on the first image sequence and the second image. By adopting the method, the situation of picture jump of the dynamic image can be avoided.

Description

Technical Field

[0001] This application relates to the field of camera technology, and in particular, to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the increasing optimization of the photography function of electronic devices, Live Photo technology has emerged. Live Photo, also known as dynamic image (live photo), is a photography technology that adds live information before and after taking a photo to the photo taken by the user. The user can see the wonderful pictures at the moments before and after taking the photo while viewing the photo, which can improve the user's photography experience.

[0003] However, when an electronic device generates a dynamic image, there is a problem of image jump. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, and computer-readable storage medium, which can avoid the situation of image jump in dynamic images and improve the smoothness of the content and brightness of dynamic images.

[0005] In a first aspect, this application provides an image processing method, including:

[0006] In response to a photographing operation, obtain a first image sequence; the first image sequence includes images whose exposure parameters exceed a target exposure parameter range;

[0007] Obtain at least two frames of images from the first image sequence for fusion to obtain a first image;

[0008] Perform brightness smoothing processing on the first image to obtain a second image;

[0009] Generate a dynamic image based on the first image sequence and the second image.

[0010] In one of the embodiments, the obtaining at least two frames of images from the first image sequence for fusion to obtain a first image includes:

[0011] Obtain at least two frames of first sequence images from the first image sequence;

[0012] For each frame of the first sequence image, perform gain brightness adjustment on the first sequence image to obtain an adjusted first sequence image;

[0013] Fuse at least two frames of the adjusted first sequence images to obtain a first image.

[0014] In one embodiment, for each frame of the first sequence of images, performing gain brightness adjustment on the first sequence of images to obtain the adjusted first sequence of images includes:

[0015] For each frame of the first sequence of images, obtaining the gain parameter of the first sequence of images and obtaining the gain parameter of the preview image before the photographing operation;

[0016] Based on the gain parameter of the first sequence of images and the gain parameter of the preview image, determining a gain ratio;

[0017] Performing gain brightness adjustment on the first sequence of images according to the gain ratio to obtain the adjusted first image.

[0018] In one embodiment, performing brightness smoothing processing on the first image to obtain a second image includes:

[0019] Obtaining a target brightness smoothing processing parameter;

[0020] According to the target brightness smoothing processing parameter, performing brightness smoothing processing on the first image to obtain a second image; the brightness of the second image is within a target brightness range.

[0021] In one embodiment, the target brightness smoothing processing parameter is used for performing brightness smoothing processing on the preview image before the photographing operation, or the target brightness smoothing processing parameter is used for performing brightness smoothing processing on the second sequence of images in the first sequence of images.

[0022] In one embodiment, generating a dynamic image based on the first sequence of images and the second image includes:

[0023] Based on the photographing moment of the photographing operation, obtaining at least two consecutive frames of the second sequence of images from the first sequence of images; the second sequence of images includes at least one of a pre - order frame image and a post - order frame image, the pre - order frame image is an image before the photographing operation in the second sequence of images, and the post - order frame image is an image after the photographing operation in the second sequence of images;

[0024] Generating a dynamic image according to the at least two frames of the second sequence of images and the second image.

[0025] In one embodiment, generating a dynamic image according to the at least two frames of the second sequence of images and the second image includes:

[0026] For each frame of the second sequence of images, performing brightness adjustment on the second sequence of images to obtain a third sequence of images;

[0027] Generate a dynamic image based on at least two frames of third-sequence images and the second image.

[0028] In one embodiment, for each frame of the second-sequence image, performing brightness adjustment on the second-sequence image to obtain a third-sequence image includes:

[0029] For each frame of the second-sequence image, performing gain brightness adjustment on the second-sequence image to obtain an adjusted second-sequence image;

[0030] Performing brightness smoothing processing on the adjusted second-sequence image to obtain a third-sequence image.

[0031] In a second aspect, the present application further provides an image processing apparatus, including:

[0032] An image acquisition module, configured to acquire a first image sequence in response to a photographing operation; the first image sequence includes images with exposure parameters exceeding a target exposure parameter range;

[0033] An image fusion module, configured to acquire at least two frames of images from the first image sequence for fusion to obtain a first image;

[0034] A brightness adjustment processing module, configured to perform brightness smoothing processing on the first image to obtain a second image;

[0035] A dynamic image generation module, configured to generate a dynamic image based on the first image sequence and the second image.

[0036] In a third aspect, the present application further provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] In response to a photographing operation, acquire a first image sequence; the first image sequence includes images with exposure parameters exceeding a target exposure parameter range;

[0038] Acquire at least two frames of images from the first image sequence for fusion to obtain a first image;

[0039] Perform brightness smoothing processing on the first image to obtain a second image;

[0040] Generate a dynamic image based on the first image sequence and the second image.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0042] In response to a photographing operation, obtain a first image sequence; the first image sequence includes images with exposure parameters exceeding a target exposure parameter range;

[0043] Obtain at least two frames of images from the first image sequence for fusion to obtain a first image;

[0044] Perform brightness smoothing processing on the first image to obtain a second image;

[0045] Generate a dynamic image based on the first image sequence and the second image.

[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which when executed by a processor implements the following steps:

[0047] In response to a photographing operation, obtain a first image sequence; the first image sequence includes images with exposure parameters exceeding a target exposure parameter range;

[0048] Obtain at least two frames of images from the first image sequence for fusion to obtain a first image;

[0049] Perform brightness smoothing processing on the first image to obtain a second image;

[0050] Generate a dynamic image based on the first image sequence and the second image.

[0051] For the above image processing method, device, electronic device, computer-readable storage medium, and computer program product, in response to a photographing operation, a first image sequence is obtained, and the first image sequence includes images with exposure parameters exceeding a target exposure parameter range; then, at least two frames of images are obtained from the first image sequence for fusion, and a first image with balanced exposure can be obtained, avoiding the problem of picture jump caused by discarding images with exposure parameters exceeding the target exposure parameter range when generating a dynamic image. Further, performing brightness smoothing processing on the first image can address the problem of uneven regional brightness caused by the first image incorporating images with exposure parameters exceeding the target exposure parameter range, obtaining a second image with smoother picture brightness. Furthermore, a dynamic image is generated based on the first image sequence and the second image, and this dynamic image avoids picture content jump and brightness jump, improving the smoothness of the picture content and the brightness smoothness. Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0053] Figure 1 is a schematic flow chart of an image processing method in an embodiment;

[0054] Figure 2 is a schematic diagram of obtaining a first image sequence in response to a photographing operation in an embodiment;

[0055] Figure 3 is a schematic diagram of obtaining a first image sequence in response to a photographing operation in another embodiment;

[0056] Figure 4 is a schematic diagram of obtaining a first image sequence in response to a photographing operation in another embodiment;

[0057] Figure 5 is a schematic flow chart of an image processing method in another embodiment;

[0058] Figure 6 is a flow chart of the first-stage processing of the ISP module in an embodiment;

[0059] Figure 7 is a flow chart of the second-stage processing of the ISP module in an embodiment;

[0060] Figure 8 is a structural block diagram of an image processing apparatus in an embodiment;

[0061] Figure 9 is an internal structure diagram of an electronic device in an embodiment. Detailed implementation manners

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] In one embodiment, as Figure 1As shown, an image processing method is provided. In this embodiment, the method is exemplified by being applied to an electronic device, which can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server and implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0064] In this embodiment, the image processing method includes the following steps:

[0065] Step S102, in response to a photographing operation, obtain a first image sequence; the first image sequence includes images whose exposure parameters exceed a target exposure parameter range.

[0066] Among them, the photographing operation belongs to an operation instruction detected by the electronic device, which is used to instruct the electronic device to collect a dynamic image. For example, the photographing operation can be a shutter button press operation or a gesture operation detected for indicating photographing.

[0067] The color format of the images in the first image sequence can be RAW, YUV, RGB, etc., and is not limited thereto. Among them, RAW is the original data image file format; in YUV, Y represents luminance, and U and V represent chrominance; RGB represents colors by using different brightness values (gray values) of the three primary colors red (R, Red), green (G, Green), and blue (B).

[0068] The first image sequence is at least two frames of images collected by the electronic device during the process of collecting a dynamic image. The first sequence of images includes at least one of a pre-frame image and a post-frame image. The pre-frame image is an image collected before the photographing operation, and the post-frame image is an image collected after the photographing operation. As Figure 2 shown, the electronic device, in response to the photographing operation, obtains at least two frames of images before the photographing operation as the first image sequence. As Figure 3 shown, the electronic device, in response to the photographing operation, obtains at least two frames of images after the photographing operation as the first image sequence. As Figure 4As shown, in response to a photographing operation, the electronic device acquires at least two frames of images before the photographing operation and at least two frames of images after the photographing operation as a first image sequence.

[0069] The exposure parameter is a parameter that controls the light received by the camera sensor in the electronic device, and may include shutter speed, aperture size, ISO sensitivity, etc. The target exposure parameter range is a preset exposure parameter interval, aiming to ensure that the captured moving image has an ideal exposure effect and avoid overexposure (excessive exposure) or underexposure (insufficient exposure). For example, if the exposure parameter of an image in the first image sequence is greater than the maximum value of the target exposure parameter range, then the image is an overexposed image, and the brightness of the image is greater than the target brightness; if the exposure parameter of an image in the first image sequence is less than the minimum value of the target exposure parameter range, then the image is an underexposed image, and the brightness of the image is less than the target brightness. The first image sequence includes at least one of overexposed images and underexposed images. Exemplarily, the exposure parameters of each image in the first image sequence are EVList (PS: 0, 0, 0, 0, -18, 9, 0), where 0 is EV0, indicating that the exposure parameter is within the target exposure parameter range, and the brightness of the image obtained by EV0 exposure is within the target brightness range; -18 is EV-, indicating that the exposure parameter exceeds the target exposure parameter range, and the image obtained by EV- exposure is an underexposed image with a darker brightness; 9 is EV+, indicating that the exposure parameter exceeds the target exposure parameter range, and the image obtained by EV+ exposure is an overexposed image with a brighter brightness.

[0070] Optionally, in response to a shutter button press operation, the electronic device acquires the images captured before the shutter button press operation and the images captured after the shutter button press operation, and forms the first sequence of images with each image; among them, the press of the shutter button captures a moment near a certain moment in time, rather than just that moment.

[0071] Optionally, before responding to the photographing operation, according to the image data collected by the image sensor, one or more frames of images captured before the photographing operation are updated in real time, and one or more frames of images captured before the photographing operation are updated in real time; when responding to the photographing operation, the brightness of one or more frames of images captured after the photographing operation is adjusted to obtain one or more frames of images captured after the photographing operation; one or more frames of images captured before the photographing operation and one or more frames of images captured after the photographing operation are combined to obtain a first image sequence.

[0072] Optionally, in response to a photographing operation, the electronic device acquires the exposure parameter corresponding to the photographing operation, and performs exposure through this exposure parameter to obtain a first image sequence.

[0073] Step S104: Obtain at least two frames of images from the first image sequence for fusion to obtain a first image.

[0074] The first image is an image obtained by capturing and fusing in the still image mode. Since the first image is obtained by fusing at least two frames of images in the first image sequence, attributes such as the brightness and color of the content represented by the first image can be adaptively adjusted to adaptively adjust the first image according to the corresponding static image requirements.

[0075] The at least two frames of images are the images to be fused obtained from the first image sequence. The at least two frames of images can be all the images in the first image sequence or a preset number of images. Optionally, the at least two frames of images include a first image to be fused and a second image to be fused output by the first photoelectric sensor, and a third image to be fused output by the second photoelectric sensor; the brightness of the first image to be fused is greater than that of the second image to be fused, and the brightness of the first image to be fused is greater than that of the third image to be fused; therefore, the first image to be fused can more clearly represent the information of the static object, and the information in the same area can be represented by different brightness in the first image to be fused, so that the representative image corresponds to a higher dynamic range. And since the brightnesses of the second image to be fused and the third image to be fused are both less than the brightness of the first image to be fused, the second image to be fused and the third image to be fused are not easily blurred due to the moving area, and there are corresponding position differences between the second image to be fused and the third image to be fused, so that more accurate depth information can be obtained, and the phenomena of false virtuality and missed virtuality can be avoided. Finally, according to the depth information for blurring, the depth information can be accurately represented by the representative image.

[0076] Optionally, obtain at least two frames of images from the first image sequence; determine the feature points in each frame of the image to be fused; determine the matching feature points representing the same information in different frames of the image to be fused; adjust the feature points of each frame of the image to be fused according to the feature points and brightness of the adjusted preview image to obtain multiple frames of adjusted images to be fused. The feature points can be each pixel, or pixel points obtained by sampling multiple pixels, or pixel points selected according to a certain step size.

[0077] Optionally, the electronic device obtains at least two frames of images from the first image sequence, performs brightness adjustment on the at least two frames of images to obtain at least two frames of images after brightness adjustment; fuses the at least two frames of images after brightness adjustment to obtain a first image.

[0078] Step S106: Perform brightness smoothing processing on the first image to obtain a second image.

[0079] Among them, the second image is the image obtained by performing brightness smoothing on the first image. Smoothing is an image processing technique aimed at reducing noise and detail variations in the image, making the image smoother and more uniform. In brightness smoothing, the brightness values of the pixels in the first image are adjusted to reduce sudden changes in brightness and make the brightness transition between adjacent pixels more natural.

[0080] Optionally, the electronic device uses the Local Tone Mapping (LTM) algorithm to perform brightness smoothing on the first image to obtain the second image.

[0081] Optionally, the electronic device can also use other algorithms to perform brightness smoothing on the first image, such as the mean filtering algorithm, Gaussian filtering algorithm, etc., and is not limited thereto.

[0082] Step S108, generating a dynamic image based on the first image sequence and the second image.

[0083] Among them, the dynamic image, which can also be referred to as a live image or live photo, is an image set that supports displaying a representative image and a second image sequence. In the case where no dynamic display operation is detected, the dynamic image displays the second image. At this time, the fused second image is equivalent to the cover of the dynamic image; in response to a dynamic display operation, the dynamic image displays at least some of the images in the first image sequence. At this time, a dynamic display effect or a video playback effect can be formed.

[0084] Optionally, the dynamic display operation is determined based on the signal strength triggered by the electronic device. The signal strength includes, but is not limited to, the capacitance of the screen, the capacitance change amount, and / or the capacitance change frequency, the level value generated by the processor based on the acquired data, etc. Optionally, the comparison between the signal strength and one or more intensity thresholds can be used to determine whether to perform one or more operations and can also be used to determine whether to perform a dynamic display operation.

[0085] Taking capacitance as an example, the dynamic display operation is caused by a contact event with a capacitance value greater than a preset capacitance threshold, and the static display operation is caused by a contact event with a capacitance value less than the preset capacitance threshold.

[0086] Optionally, the electronic device uses the second image as the cover image and uses the first image sequence as the display image of the cover image when a dynamic display operation is triggered to obtain a dynamic image. In the case where the cover image triggers a dynamic display operation, the first image sequence can be displayed, making the dynamic image display an effect similar to a video.

[0087] The above image processing method, in response to a photographing operation, obtains a first image sequence, where the first image sequence includes images with exposure parameters exceeding a target exposure parameter range; then, at least two frames of images are obtained from the first image sequence for fusion, and a first image with balanced exposure can be obtained, avoiding the problem of frame jumping caused by discarding images with exposure parameters exceeding the target exposure parameter range when generating a dynamic image. Further, performing brightness smoothing processing on the first image can address the problem of uneven regional brightness caused by the first image fusing images with exposure parameters exceeding the target exposure parameter range, resulting in a second image with smoother picture brightness. Then, a dynamic image is generated based on the first image sequence and the second image. The dynamic image avoids frame content jumping and brightness jumping, improving the smoothness of the frame content and the brightness smoothness.

[0088] In one embodiment, obtaining at least two frames of images from the first image sequence for fusion to obtain a first image includes: obtaining at least two frames of first sequence images from the first image sequence; for each frame of the first sequence images, performing gain brightness adjustment on the first sequence images to obtain adjusted first sequence images; and fusing at least two frames of the adjusted first sequence images to obtain a first image.

[0089] Among them, the first sequence image is a single-frame image in the first image sequence. At least two frames of the first sequence images can be at least two consecutive frames of images in the first image sequence. At least two frames of the first sequence images can include at least one of a pre-order frame image and a post-order frame image. The gain brightness adjustment changes the image brightness by adjusting the gain parameter to achieve the desired brightness effect. At least two frames of the first sequence images include images with exposure parameters exceeding the target exposure parameter range.

[0090] Optionally, for each frame of the first sequence images, the electronic device obtains the gain brightness adjustment parameter of the gain brightness adjustment (Digital Gain) algorithm and uses the gain brightness adjustment parameter to perform gain brightness adjustment on the first sequence images to obtain adjusted first sequence images. Among them, the gain brightness adjustment parameter can be the gain ratio between the gain parameter of the first sequence image and the gain parameter of the preview image.

[0091] Optionally, for each frame of the first sequence images, performing gain brightness adjustment on the first sequence images to obtain adjusted first sequence images includes: for each frame of the first sequence images, obtaining the gain parameter of the first sequence image and the gain parameter of the preview image before the photographing operation; determining the gain ratio based on the gain parameter of the first sequence image and the gain parameter of the preview image; and performing gain brightness adjustment on the first sequence images according to the gain ratio to obtain the adjusted first image.

[0092] Among them, the gain parameter is a value used to control the amplification degree of the image signal during the image processing. It determines the brightness change of the image. A larger gain value will make the image brighter, while a smaller gain value will make the image darker. Different images may correspond to different gain parameters to adapt to different shooting conditions or achieve specific visual effects. The gain parameter of the first sequence of images and the gain parameter of the preview image can be the total gain of the first sequence of images and the total gain of the preview image, or the sensor gain of the first sequence of images and the sensor gain of the preview image, or the digital gain of the first sequence of images and the digital gain of the preview image. Among them, the total gain TotalGain = sensor gain SensorGain * digital gain DigitalGain.

[0093] It can be understood that before the photographing operation, a preview image is displayed on the screen of the electronic device. Since the photographing operation is expected to generate a dynamic image, the exposure parameters corresponding to the dynamic image are different from those of the preview image. Therefore, different gain parameters are determined by different exposure parameters. Then, in order to keep the brightness of the dynamic image and the preview image smoother, it is necessary to perform gain brightness adjustment on each frame of the first sequence of images.

[0094] Optionally, for each frame of the first sequence of images, the electronic device determines the ratio between the gain parameter of the first sequence of images and the gain parameter of the preview image as the gain ratio. During the gain brightness adjustment process, each pixel in the first sequence of images is multiplied by the gain ratio to achieve gain brightness adjustment of the first sequence of images and obtain the adjusted first image.

[0095] In this embodiment, at least two frames of the first sequence of images are obtained from the first image sequence. For each frame of the first sequence of images, gain brightness adjustment is performed on the first sequence of images, and the adjusted first sequence of images with more accurate brightness information can be obtained. Then, at least two frames of the adjusted first sequence of images are fused to obtain the first image with more accurate brightness information.

[0096] Further, for each frame of the first sequence of images, the gain parameter of the first sequence of images is obtained, and the gain parameter of the preview image before the photographing operation is obtained; based on the gain parameter of the first sequence of images and the gain parameter of the preview image, the gain ratio is determined, and the first sequence of images is subjected to gain brightness adjustment according to the gain ratio to obtain the adjusted first image, which can make the brightness of the adjusted first image consistent with that of the preview image. Furthermore, the brightness smoothness of the displayed image can be improved during the preview and display of the dynamic image, and the brightness of the underexposed image EV- and the overexposed image EV+ can be adjusted to be consistent with the brightness of the normal image EV0, maintaining the smoothness of the picture brightness.

[0097] In one embodiment, performing brightness smoothing on a first image to obtain a second image includes: obtaining target brightness smoothing parameters; performing brightness smoothing on the first image according to the target brightness smoothing parameters to obtain the second image; and the brightness of the second image being within a target brightness range.

[0098] Among them, the target brightness smoothing parameters are parameters for performing brightness smoothing on an image. The target brightness smoothing parameters may be the brightness curve of a local brightening algorithm. The target brightness range can be set as needed. The target brightness smoothing parameters are used to adjust the brightness of the first image to within the target brightness range.

[0099] Optionally, the target brightness smoothing parameters are for performing brightness smoothing on a preview image before a photographing operation, or the target brightness smoothing parameters are for performing brightness smoothing on a second sequence of images in the first image sequence.

[0100] It can be understood that the target brightness smoothing parameters used by the electronic device when performing brightness smoothing on the preview image before a photographing operation, or the target brightness smoothing parameters used when performing brightness smoothing on the second sequence of images in the first image sequence, are stored locally in the electronic device or in a cloud server; then, the electronic device obtains the target brightness smoothing parameters from local or the server.

[0101] In this embodiment, the electronic device obtains the target brightness smoothing parameters and performs brightness smoothing on the first image according to the target brightness smoothing parameters, which can suppress the brightness of overexposed areas and underexposed areas in the first image, making the picture of the second image more natural and avoiding the problem of uneven brightness in the image area caused by the fusion of underexposed images or overexposed images.

[0102] In one embodiment, generating a dynamic image based on the first image sequence and the second image includes: obtaining at least two consecutive frames of a second sequence of images from the first image sequence based on the photographing moment of the photographing operation; the second sequence of images including at least one of a pre-frame image and a post-frame image, the pre-frame image being an image before the photographing operation in the second image sequence, and the post-frame image being an image after the photographing operation in the second image sequence; and generating a dynamic image according to the at least two frames of the second sequence of images and the second image.

[0103] Among them, the second sequence of images is a single-frame image in the first image sequence and is an image for dynamic display. At least two frames of the second sequence of images may include images with exposure parameters exceeding the target exposure parameter range.

[0104] Optionally, a dynamic image is generated based on at least two frames of second sequence images and a second image, including: for each frame of the second sequence images, adjusting the brightness of the second sequence images to obtain a third sequence images; generating a dynamic image based on at least two frames of the third sequence images and the second image.

[0105] Optionally, the electronic device uses the second image as a cover image and uses the second image sequence as the display image of the cover image when a dynamic display operation is triggered, to obtain a dynamic image. In the case where the cover image triggers a dynamic display operation, the second image sequence can be displayed, so that the dynamic image shows an effect similar to a video.

[0106] Optionally, for each frame of the second sequence images, adjusting the brightness of the second sequence images to obtain a third sequence images, including: for each frame of the second sequence images, performing gain brightness adjustment on the second sequence images to obtain an adjusted second sequence images; performing brightness smoothing processing on the adjusted second sequence images to obtain a third sequence images.

[0107] For each frame of the second sequence images, performing gain brightness adjustment on the second sequence images to obtain an adjusted second sequence images, including: for each frame of the second sequence images, obtaining the gain parameter of the second sequence images and the gain parameter of the preview image before the photographing operation; based on the gain parameter of the second sequence images and the gain parameter of the preview image, determining the gain ratio corresponding to the second sequence images; performing gain brightness adjustment on the second sequence images according to the gain ratio corresponding to the second sequence images to obtain an adjusted second sequence images.

[0108] Among them, the gain parameter of the second sequence images and the gain parameter of the preview image can be the total gain of the second sequence images and the total gain of the preview image, or the sensor gain of the second sequence images and the sensor gain of the preview image, or the digital gain of the second sequence images and the digital gain of the preview image. Among them, TotalGain = SensorGain * DigitalGain.

[0109] For each frame of the second sequence images, the electronic device determines the ratio between the gain parameter of the second sequence images and the gain parameter of the preview image as the gain ratio corresponding to the second sequence images. During the gain brightness adjustment process, each pixel in the second sequence images is multiplied by the gain ratio corresponding to the second sequence images to implement gain brightness adjustment on the second sequence images to obtain an adjusted second sequence images.

[0110] Perform brightness smoothing on the adjusted second sequence of images to obtain a third sequence of images, including: obtaining target brightness smoothing parameters; performing brightness smoothing on the adjusted second sequence of images according to the target brightness smoothing parameters to obtain a third sequence of images; the brightness of the third sequence of images is within the target brightness range.

[0111] Among them, the target brightness smoothing parameters are used for brightness smoothing of the preview image before the photographing operation, or the target brightness smoothing parameters are used for brightness smoothing of the second sequence of images in the first image sequence.

[0112] In this embodiment, based on the photographing moment of the photographing operation, at least two consecutive frames of the second sequence of images are obtained from the first image sequence. The second sequence of images includes at least one of a pre-frame image and a post-frame image. The pre-frame image is an image before the photographing operation in the second image sequence, and the post-frame image is an image after the photographing operation in the second image sequence. Then, a dynamic image can be accurately generated according to at least two frames of the second sequence of images and the second image.

[0113] Furthermore, for each frame of the second sequence of images, performing brightness adjustment on the second sequence of images can obtain a third sequence of images with more accurate brightness. Then, a dynamic image with smoother brightness can be generated according to at least two frames of the third sequence of images and the second image, which can avoid the situation of frame jumping in the dynamic image.

[0114] Furthermore, for each frame of the second sequence of images, performing gain brightness adjustment on the second sequence of images to obtain the adjusted second sequence of images, and then performing brightness smoothing on the adjusted second sequence of images can obtain a third sequence of images with smoother brightness. Then, a dynamic image is generated based on the third image sequence and the second image. The dynamic image avoids frame content jumping and brightness jumping, and improves the smoothness of the frame content and the brightness smoothness.

[0115] In one embodiment, an image processing method applied to an electronic device is further provided. The image processing method includes the following steps:

[0116] Step A1, in response to a photographing operation, obtain a first image sequence; the first image sequence includes images with exposure parameters exceeding the target exposure parameter range.

[0117] Step A2, obtain at least two frames of the first sequence of images from the first image sequence.

[0118] Step A3: For each frame of the first sequence of images, obtain the gain parameter of the first sequence of images and the gain parameter of the preview image before the capture operation; based on the gain parameter of the first sequence of images and the gain parameter of the preview image, determine the gain ratio; adjust the gain brightness of the first sequence of images according to the gain ratio to obtain the adjusted first image.

[0119] Step A4: Fuse at least two frames of the adjusted first sequence of images to obtain the first image.

[0120] Step A5: Obtain the target brightness smoothing parameter; wherein, the target brightness smoothing parameter is used for performing brightness smoothing processing on the preview image before the capture operation, or the target brightness smoothing parameter is used for performing brightness smoothing processing on the second sequence of images in the first image sequence; perform brightness smoothing processing on the first image according to the target brightness smoothing parameter to obtain the second image; the brightness of the second image is within the target brightness range.

[0121] Step A6: Based on the capture moment of the capture operation, obtain at least two consecutive frames of the second sequence of images from the first image sequence; the second sequence of images includes at least one of the pre-frame image and the post-frame image, the pre-frame image is the image before the capture operation in the second image sequence, and the post-frame image is the image after the capture operation in the second image sequence.

[0122] Step A7: For each frame of the second sequence of images, perform gain brightness adjustment on the second sequence of images to obtain the adjusted second sequence of images; perform brightness smoothing processing on the adjusted second sequence of images to obtain the third sequence of images.

[0123] Step A8: Generate a dynamic image according to at least two frames of the third sequence of images and the second image.

[0124] In one embodiment, as Figure 5 shown, the electronic device opens the camera application, and displays the preview image in the camera application; in response to the capture operation, the camera application sends the exposure parameter corresponding to the capture operation to the Hardware Abstraction Layer (HAL); configures the exposure parameter into the image sensor through the hardware abstraction layer, and controls the image sensor to output the raw image (RAW image) corresponding to each exposure parameter, at least two frames of raw images form the first image sequence, and the first image sequence is sent to the ISP module, and the first image sequence includes images with exposure parameters exceeding the target exposure parameter range.

[0125] Obtain at least two frames of first sequence images from the first image sequence. For each frame of the first sequence images, perform gain brightness adjustment on the first sequence images through the ISP module to obtain the adjusted first sequence images, and send at least two frames of the adjusted first sequence images to the backend processing engine through the hardware abstraction layer and the camera application; fuse at least two frames of the adjusted first sequence images through the backend processing engine to obtain the first image, and return the first image to the ISP module; perform brightness smoothing processing on the first image through the ISP module to obtain the second image, and send the second image to the camera application.

[0126] Obtain at least two consecutive frames of second sequence images from the first image sequence. For each frame of the second sequence images, perform gain brightness adjustment on the second sequence images through the ISP module to obtain the adjusted second sequence images, perform brightness smoothing processing on the adjusted second sequence images to obtain the third sequence images, and send at least two frames of the third sequence images to the camera application.

[0127] Generate a dynamic image by the camera application according to at least two frames of the third sequence images and the second image.

[0128] As Figure 6 and Figure 7 shown, the ISP module includes sub-modules such as Bad Pixel Correction (BPC), Optical Black Correction (OBC), Digital Gain (DGN), LensShading Correction (LSC), White Balance (WB), High-LevelRegister (HLR), Local Tone Mapping (LTM), Demosaic (NM), Color Correct Correction (CCM), gamma (GGM), Yuv422 (C40), Tone and Contrast (TNC), Adaptive KernelSmooting (AKS), and Low-Pass Noise Reduction (LPNR), as Figure 6Shown is the first-stage processing of the ISP module. After the ISP module acquires the first sequence of images or the second sequence of images in RAW color format, it turns on the bad pixel correction, black level correction, and digital gain (gain brightness adjustment) sub-modules, that is, performs bad pixel correction, black level correction, and digital gain on the first sequence of images or the second sequence of images, and obtains the adjusted first sequence of images or the second sequence of images in YUV color format. The adjusted first sequence of images in YUV color format is fused through the back-end processing engine to obtain the first image in YUV color format (P010 color format).

[0129] As Figure 7 Shown is the second-stage processing of the ISP module. After the ISP module acquires the first image in YUV color format or the adjusted second sequence of images, it turns on the bad pixel correction, local brightening (brightness smoothing processing), color calibration matrix, gamma correction, Yuv422 format, brightening and contrast, adaptive kernel smoothing, and low-pass noise reduction sub-modules, that is, performs bad pixel correction, local brightening (brightness smoothing processing), color calibration matrix, gamma correction, Yuv422 format, brightening and contrast, adaptive kernel smoothing, and low-pass noise reduction on the first image in YUV color format or the adjusted second sequence of images, and obtains the adjusted second image in Jpeg color format or the third sequence of images. The camera application generates a dynamic image from at least two frames of the third sequence of images and the second image.

[0130] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiments of the present application also provide an image processing apparatus for implementing the above-mentioned image processing method. The solution provided by the apparatus for solving the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the image processing apparatus provided below can refer to the limitations on the image processing method in the above text, and will not be repeated here.

[0132] In an exemplary embodiment, as Figure 8As shown in the figure, an image processing apparatus is provided, including: an image acquisition module 802, an image fusion module 804, a brightness adjustment processing module 806, and a dynamic image generation module 808, where:

[0133] The image acquisition module 802 is configured to obtain a first image sequence in response to a photographing operation; the first image sequence includes images with exposure parameters exceeding a target exposure parameter range.

[0134] The image fusion module 804 is configured to obtain at least two frames of images from the first image sequence for fusion to obtain a first image.

[0135] The brightness adjustment processing module 806 is configured to perform brightness smoothing processing on the first image to obtain a second image.

[0136] The dynamic image generation module 808 is configured to generate a dynamic image based on the first image sequence and the second image.

[0137] The above image processing apparatus obtains a first image sequence in response to a photographing operation, and the first image sequence includes images with exposure parameters exceeding a target exposure parameter range; then, at least two frames of images are obtained from the first image sequence for fusion, and a first image with balanced exposure can be obtained, avoiding the problem of picture jump caused by discarding images with exposure parameters exceeding the target exposure parameter range when generating a dynamic image. Further, performing brightness smoothing processing on the first image can address the problem of uneven regional brightness caused by the first image fusing images with exposure parameters exceeding the target exposure parameter range, obtaining a second image with smoother picture brightness. Furthermore, a dynamic image is generated based on the first image sequence and the second image, and this dynamic image avoids picture content jump and brightness jump, improving the smoothness of picture content and brightness smoothness.

[0138] In one embodiment, the above image fusion module 804 is further configured to obtain at least two frames of first sequence images from the first image sequence; for each frame of the first sequence images, perform gain brightness adjustment on the first sequence images to obtain adjusted first sequence images; and fuse at least two frames of the adjusted first sequence images to obtain a first image.

[0139] In one embodiment, the above image fusion module 804 is further configured to, for each frame of the first sequence images, obtain the gain parameter of the first sequence images and the gain parameter of the preview image before the photographing operation; determine a gain ratio based on the gain parameter of the first sequence images and the gain parameter of the preview image; and perform gain brightness adjustment on the first sequence images according to the gain ratio to obtain an adjusted first image.

[0140] In one embodiment, the above-mentioned brightness adjustment processing module 806 is further configured to obtain a target brightness smoothing processing parameter; perform brightness smoothing processing on the first image according to the target brightness smoothing processing parameter to obtain a second image; the brightness of the second image is within the target brightness range.

[0141] In one embodiment, the target brightness smoothing processing parameter is used for performing brightness smoothing processing on a preview image before a photographing operation, or the target brightness smoothing processing parameter is used for performing brightness smoothing processing on a second sequence of images in the first image sequence.

[0142] In one embodiment, the above-mentioned dynamic image generation module 808 is further configured to obtain at least two consecutive frames of second sequence images from the first image sequence based on the photographing moment of the photographing operation; the second sequence images include at least one of a pre-frame image and a post-frame image, the pre-frame image is an image before the photographing operation in the second image sequence, and the post-frame image is an image after the photographing operation in the second image sequence; generate a dynamic image according to the at least two frames of second sequence images and the second image.

[0143] In one embodiment, the above-mentioned dynamic image generation module 808 is further configured to perform brightness adjustment on each frame of the second sequence images to obtain a third sequence of images; generate a dynamic image according to the at least two frames of third sequence images and the second image.

[0144] In one embodiment, the above-mentioned dynamic image generation module 808 is further configured to perform gain brightness adjustment on each frame of the second sequence images to obtain an adjusted second sequence of images; perform brightness smoothing processing on the adjusted second sequence of images to obtain a third sequence of images.

[0145] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in the electronic device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0146] In an exemplary embodiment, an electronic device is provided. The electronic device can be a terminal, and its internal structure diagram can be as Figure 9As shown in the figure. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0147] Those skilled in the art can understand that Figure 9 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0148] In one embodiment, an electronic device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0150] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0152] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0153] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0154] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: In response to a photographing operation, acquiring a first image sequence; the first image sequence includes images whose exposure parameters exceed a target exposure parameter range; Acquire at least two frames of images from the first image sequence and fuse them to obtain a first image; Performing brightness smoothing processing on the first image to obtain a second image; A dynamic image is generated based on the first image sequence and the second image.

2. The method according to claim 1, characterized in that The acquiring at least two frames of images from the first image sequence and fusing them to obtain a first image includes: Acquire at least two frames of first sequence images from the first image sequence; For each frame of the first sequence of images, performing gain and brightness adjustment on the first sequence of images to obtain adjusted first sequence of images; At least two frames of the adjusted first sequence images are fused to obtain a first image.

3. The method according to claim 2, characterized in that The step of adjusting the gain and brightness of the first sequence of images for each frame to obtain the adjusted first sequence of images includes: For each frame of the first sequence of images, obtaining a gain parameter of the first sequence of images, and obtaining a gain parameter of a preview image before the photographing operation; Determining a gain ratio based on a gain parameter of the first sequence of images and a gain parameter of the preview image; The gain brightness of the first sequence of images is adjusted according to the gain ratio to obtain an adjusted first image.

4. The method according to claim 1, characterized in that: The step of performing brightness smoothing processing on the first image to obtain a second image includes: Get the target brightness smoothing parameters; According to the target brightness smoothing processing parameter, brightness smoothing processing is performed on the first image to obtain a second image; the brightness of the second image is within the target brightness range.

5. The method according to claim 4, characterized in that The target brightness smoothing processing parameter is used for brightness smoothing processing of the preview image before the photographing operation, or the target brightness smoothing processing parameter is used for brightness smoothing processing of the second sequence image in the first image sequence.

6. The method according to claim 1, characterized in that The generating a dynamic image based on the first image sequence and the second image comprises: Based on the photographing moment of the photographing operation, acquiring at least two consecutive frames of a second sequence of images from the first image sequence; the second sequence of images includes at least one of a preceding frame image and a succeeding frame image, the preceding frame image is an image before the photographing operation in the second image sequence, and the succeeding frame image is an image after the photographing operation in the second image sequence; A dynamic image is generated according to the at least two frames of the second sequence of images and the second image.

7. The method according to claim 6, characterized in that The step of generating a dynamic image according to the at least two frames of the second sequence of images and the second image comprises: For each frame of the second sequence of images, adjusting the brightness of the second sequence of images to obtain a third sequence of images; A dynamic image is generated according to at least two frames of the third sequence of images and the second image.

8. The method according to claim 7, characterized in that The step of adjusting the brightness of the second sequence of images for each frame of the second sequence of images to obtain a third sequence of images includes: For each frame of the second sequence of images, performing gain and brightness adjustment on the second sequence of images to obtain adjusted second sequence of images; The adjusted second sequence of images is subjected to brightness smoothing processing to obtain a third sequence of images.

9. An image processing device, characterized in that: The device comprises: An image acquisition module, configured to acquire a first image sequence in response to a photographing operation; the first image sequence includes images whose exposure parameters exceed a target exposure parameter range; An image fusion module, used for acquiring at least two frames of images from the first image sequence for fusion to obtain a first image; A brightness adjustment processing module, used for performing brightness smoothing processing on the first image to obtain a second image; A dynamic image generation module is used to generate a dynamic image based on the first image sequence and the second image.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images