Image processing method and device, electronic equipment, and storage medium

Abstract

The present disclosure relates to an image processing method and device, electronic equipment and storage medium. The image processing method comprises: in response to an image preview request for a to-be-acquired object in a high dynamic photographing mode, acquiring current exposure information; in a case where the to-be-acquired object is in a target motion state, adjusting the current exposure information to obtain target exposure information; and performing exposure processing based on the target exposure information to obtain a preview image for the to-be-acquired object. In the image processing method proposed in the present disclosure, in a case where it is determined that the to-be-acquired object is in a target motion state, image exposure processing is performed through target exposure information, which can reduce imaging noise introduced by the motion of the to-be-acquired object and improve the imaging effect of a moving object with a dynamic range.

Description

Technical Field

[0001] This disclosure relates to the field of image acquisition and processing, and more particularly to an image processing method and apparatus, electronic device, and storage medium. Background Technology

[0002] High Dynamic Range (HDR) shooting mode is a shooting mode based on High Dynamic Range Imaging (HDR) and is mainly used to solve the problem of unsatisfactory image quality under high dynamic range conditions. HDR shooting mode is a technical solution that fuses multiple overexposed, moderately exposed and underexposed images to obtain the captured image, so that bright areas are not overexposed and dark areas are not underexposed, ultimately presenting users with better image quality.

[0003] However, in high dynamic range (HVR) photography, the automatic exposure control module calculates and determines the current exposure information before transmitting it to the image acquisition sensor for image exposure processing. This still results in significant image noise, which produces artifacts in the image, causing the final displayed image to lose motion information. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, electronic device, and storage medium. In the image processing method proposed in the embodiments of this disclosure, when it is determined that the object to be acquired is in a target motion state, image exposure processing is performed using the target exposure information, which can reduce imaging noise introduced by the motion of the object to be acquired and improve the imaging effect of moving objects with a dynamic range.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising at least:

[0006] In response to an image preview request for the object to be captured in high dynamic range shooting mode, obtain the current exposure information;

[0007] When the object to be collected is in a target motion state, the current exposure information is adjusted to obtain the target exposure information;

[0008] Exposure processing is performed based on the target exposure information to obtain a preview image of the object to be collected.

[0009] In some embodiments, adjusting the current exposure information to obtain target exposure information includes:

[0010] The target exposure information is obtained by adjusting the current exposure information based on a preset exposure coefficient.

[0011] The exposure factor is the exposure ratio when the object to be collected is in the target motion state.

[0012] In some embodiments, adjusting the current exposure information based on a preset exposure coefficient to obtain the target exposure information includes:

[0013] Based on the exposure factor, the current exposure gain in the current exposure information, and the current exposure time, the exposure adjustment gain and the exposure adjustment time are obtained;

[0014] Based on the exposure adjustment gain and the exposure adjustment time, the number of exposed pixel rows and the number of pixels to be exposed in each exposed pixel row are obtained.

[0015] In some embodiments, the method further includes:

[0016] In response to a shooting request for the object to be captured, a first timestamp corresponding to the shooting request is determined;

[0017] Based on the target exposure information, obtain the first exposure image corresponding to the first timestamp;

[0018] A captured image is generated based on the first exposed image.

[0019] In some embodiments, obtaining the first exposure image corresponding to the first timestamp based on the target exposure information includes:

[0020] Multiple frames of cached images are obtained based on the target exposure information;

[0021] Determine the generation timestamp corresponding to the generation of each frame of the cached image;

[0022] Determine a second timestamp that matches the first timestamp from among the multiple generated timestamps;

[0023] The first exposure image is determined based on the cached image corresponding to the second timestamp.

[0024] In some embodiments, determining the first exposed image based on the cached image corresponding to the second timestamp includes:

[0025] Obtain the cached images corresponding to a first preset number of generated timestamps prior to the second timestamp;

[0026] The cached image corresponding to the second timestamp and the cached images corresponding to the first preset number of generated timestamps are subjected to image fusion processing to obtain the first exposure image.

[0027] In some embodiments, determining a second timestamp that matches the first timestamp among a plurality of generated timestamps includes:

[0028] When a timestamp identical to the first timestamp is found among multiple generated timestamps, the generated timestamp identical to the first timestamp is used as the second timestamp;

[0029] When no timestamp identical to the first timestamp is found among the multiple generated timestamps, the multiple differences between the multiple generated timestamps and the first timestamp are calculated.

[0030] The generation timestamp corresponding to the smallest difference among the multiple differences is used as the second timestamp.

[0031] In some embodiments, the method further includes:

[0032] Based on the target exposure information, a second exposure image and a third exposure image are obtained; the brightness of the first exposure image is greater than the brightness of the second exposure image, and the brightness of the first exposure image is less than the brightness of the third exposure image.

[0033] The step of generating a captured image based on the first exposed image includes:

[0034] The first exposure image, the second exposure image, and the third exposure image are subjected to image fusion processing to generate the captured image.

[0035] In some embodiments, the method further includes:

[0036] Based on the current exposure information, at least two frames of original exposure images are obtained;

[0037] A binary image is determined based on the pixel difference for the same pixel point in at least two frames of the original exposure images;

[0038] Based on the binary image and at least two frames of the original exposure image, determine the motion region in at least two frames of the original exposure image;

[0039] Feature points and feature descriptors of a set dimension are determined in the motion regions of at least two frames of the original exposure images, respectively;

[0040] Based on the feature points and the feature descriptors of the defined dimensions of the feature points, it is determined whether the object to be collected is in a target motion state.

[0041] In some embodiments, determining whether the object to be acquired is in a target motion state based on the feature points and the feature descriptors of the feature points with a set dimension includes:

[0042] Based on the defined dimension feature descriptor of the feature points, the first candidate feature matching pair is determined;

[0043] When the pixel distance between two feature points in the first candidate feature matching pair exceeds a preset distance, a second candidate feature matching pair is determined;

[0044] When the number of the second candidate feature matching pairs exceeds the second preset number, it is determined that the object to be collected is in the target motion state.

[0045] In some embodiments, the method further includes:

[0046] When the object to be captured is not in the target motion state, exposure processing is performed based on the current exposure information to obtain the preview image of the object to be captured.

[0047] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising at least:

[0048] The acquisition module is configured to acquire the current exposure information in response to an image preview request for the object to be captured in high dynamic range shooting mode.

[0049] The adjustment module is configured to adjust the current exposure information when the object to be collected is in a target motion state, so as to obtain target exposure information;

[0050] The exposure module is configured to perform exposure processing based on the target exposure information to obtain a preview image of the object to be collected.

[0051] According to a third aspect of the present disclosure, an electronic device is provided, comprising at least:

[0052] processor;

[0053] Memory used to store processor-executable instructions;

[0054] The processor is configured to perform the image processing method as described in the first aspect above.

[0055] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising:

[0056] When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described in the first aspect above.

[0057] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0058] The image processing method proposed in this disclosure is applied in high dynamic range (HVR) photography mode. When it is determined that the object to be captured is in a target motion state, the current exposure information is adjusted to obtain target exposure information. In other words, the target exposure information in this disclosure corresponds to the motion scene of the object to be captured. Thus, by performing image exposure processing using target exposure information, this disclosure can reduce imaging noise introduced by the motion of the object to be captured, improve the imaging effect of moving objects with a dynamic range, and solve the problem of high image noise and inability to adapt to high dynamic range motion scenes in traditional methods that rely on current exposure information for previewing and capturing images. This further improves the image preview effect and the subsequent image display effect.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0060] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0061] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 ;

[0062] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 2 ;

[0063] Figure 3 This is a schematic diagram illustrating the principle of a traditional frame-capturing method based on an image sensor.

[0064] Figure 4 This is a schematic diagram illustrating the principle of a frame-capturing method based on an image sensor, according to an exemplary embodiment.

[0065] Figure 5 This is a schematic diagram illustrating the principle of constructing a Gaussian difference pyramid according to an exemplary embodiment;

[0066] Figure 6 This is a schematic diagram illustrating the principle of determining local extrema in a Gaussian difference pyramid according to an exemplary embodiment;

[0067] Figure 7a This is a feature distribution map of candidate feature points according to an exemplary embodiment;

[0068] Figure 7b This is a schematic diagram illustrating a set-dimensional feature descriptor for candidate feature points according to an exemplary embodiment;

[0069] Figure 8 This is a schematic diagram of the principle of traditional exposure and image output. Figure 1 ;

[0070] Figure 9 A flowchart illustrating a traditional method for generating captured images. Figure 1 ;

[0071] Figure 10 This is a schematic diagram of the principle of traditional exposure and image output. Figure 2 ;

[0072] Figure 11 A flowchart illustrating a traditional method for generating captured images. Figure 2 ;

[0073] Figure 12 This is a schematic diagram illustrating an image preview process according to an exemplary embodiment;

[0074] Figure 13 This is a schematic diagram illustrating the process of image preview according to an exemplary embodiment;

[0075] Figure 14 This is a flowchart illustrating a normal exposure image search method according to an exemplary embodiment;

[0076] Figure 15 This is a schematic diagram illustrating a process for generating a captured image according to an exemplary embodiment;

[0077] Figure 16 This is a diagram illustrating the image quality of a traditional camera app capturing a subject in normal exposure mode.

[0078] Figure 17 This is a schematic diagram illustrating the image effect of a camera application capturing an object in high conversion gain exposure mode, according to an exemplary embodiment.

[0079] Figure 18 This is a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment.

[0080] Figure 19 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0081] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0082] See Figure 1 , Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment; as shown below. Figure 1 As shown, the image processing method proposed in this disclosure includes:

[0083] Step S101: In response to the image preview request for the object to be captured in high dynamic range shooting mode, obtain the current exposure information;

[0084] Step S102: When the object to be collected is in a target motion state, adjust the current exposure information to obtain the target exposure information;

[0085] Step S103: Perform exposure processing based on the target exposure information to obtain a preview image of the object to be collected.

[0086] Here, the image processing method proposed in this disclosure is applied to an electronic device. Here, the electronic device includes mobile terminals and portable electronic devices. The mobile terminal includes, but is not limited to, mobile phones, tablet computers, etc.; the portable electronic device includes, but is not limited to, smartwatches, etc., and the embodiments of this disclosure do not limit this.

[0087] In this disclosure, the electronic device has a camera application, which typically has two functions: preview and capture. When the camera application is operating in HDR shooting mode and the object to be captured is determined to be in motion, the current image capture scene can be determined to be a high dynamic range motion scene. At this time, the current exposure information is adjusted to the target exposure information, and the image sensor performs image exposure processing based on the target exposure information and outputs a frame to obtain a preview image. Furthermore, when the capture function is subsequently activated, the captured image can also be obtained based on the target exposure information. Thus, by performing image exposure processing using the target exposure information, this disclosure can reduce imaging noise introduced by the motion of the object to be captured and improve the imaging effect of moving objects with dynamic range.

[0088] In step S101, the high dynamic range (HDR) mode of the image is generally formed by combining multiple exposed images (overexposed, medium-exposed, and underexposed images) using an HDR algorithm (such as a pyramid-based fusion algorithm). Here, the exposure information can affect the quality of the acquired multiple exposed images, and further affect the display of motion information of the object to be captured in the synthesized HDR image.

[0089] In high dynamic range (HDR) shooting mode, when receiving an image preview request for the object to be captured, responding to the image preview request can obtain the current exposure information. Here, the exposure information includes the exposure level, which can be calculated by the Automatic Exposure Control (AEC) module in the electronic device; here, the AEC module is connected to the image sensor, and the AEC module automatically configures the exposure size to obtain the current exposure level.

[0090] Here, the current exposure information can also be the exposure information manually adjusted by the user; the current exposure information manually adjusted by the user is determined by sensing the touch operation performed by the user on the electronic device.

[0091] In step S102, it is determined whether the object to be collected is in a target motion state. If the object to be collected is in a target motion state, the current preview scene is determined to be a high dynamic range motion scene. At this time, the current exposure information obtained in step S101 is adjusted to obtain the target exposure information.

[0092] In some examples, the object to be captured being in a target motion state includes the object's motion range exceeding a preset range or the object's motion speed exceeding a preset speed. Specifically, in two or three adjacent frames of raw exposure images captured by the image sensor, if the object to be captured is located in the 1 / 3 region of the previous frame and in the 2 / 3 region of the next frame, then the object is determined to be in a target motion state, meaning the current preview scene is a high dynamic range (HMR) scene; or, the object's motion speed is determined by an object detection algorithm, and if the object's motion speed exceeds a preset speed, for example, exceeding 0.5 m / s, then the object is determined to be in a target motion state, meaning the current preview scene is a HMR scene.

[0093] It should be noted that the objects to be collected here include natural persons, animals, plants, sports equipment (such as drones), and natural scenery (such as clouds and raindrops).

[0094] In other examples, the target motion state of the object being collected includes a number of target key points in the object whose position changes exceed a preset number of key points. Here, the object being collected includes a natural person or an animal; for example, when the object being collected includes a natural person, there are 17 target key points, specifically including the left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; when more than 9 of the above target key points are detected to have changed position, the object being collected is determined to be in a target motion state; when the object being collected includes an animal, there may be 11 target key points, specifically including the left eye, right eye, nose, left ear, right ear, left arm, right arm, left hip, right hip, left foot, and right foot; when more than 6 of the above target key points are detected to have changed position, the object being collected is determined to be in a target motion state.

[0095] In this disclosure, when the object to be collected is in a target motion state, the current preview scene is determined to be a high dynamic motion scene. At this time, the above-mentioned current exposure information is adjusted to obtain the target exposure information.

[0096] It should be noted that the target exposure information is adapted to the High Conversion Gain (HCG) mode, while the current exposure information is adapted to the normal exposure mode. In the normal exposure mode, imaging is performed based on the current exposure information calculated by the automatic exposure mode, which cannot adapt to dark or bright environments, has high imaging noise, and causes the final preview and captured images to lose motion information, failing to capture high dynamic range. In contrast, in the HCG mode, the image sensor can perform exposure based on the target exposure parameters, reducing readout noise while utilizing more light for imaging, preserving more image details, and is suitable for imaging high dynamic range motion scenes.

[0097] Here, adjusting the current exposure information can be done by looking up a pre-stored mapping table between the current exposure information and the target exposure information, or by inputting the current exposure information into the adjustment calculation model and obtaining the target exposure information through the calculation of the adjustment calculation model.

[0098] For example, when the current exposure information includes the exposure amount, the target exposure amount is output after data verification and data calculation processing of the current exposure amount.

[0099] In step S103, after obtaining the target exposure information according to the above steps, the image sensor performs exposure processing based on the target exposure information to obtain multi-frame buffered images, and transmits the multi-frame buffered images to the Image Signal Processing (ISP) center; the ISP center preprocesses, crops, and fuses the multi-frame buffered images and outputs a single-frame preview image. Here, the ISP center's preprocessing of the multi-frame buffered images includes: bad pixel correction processing, black level correction processing, white balance processing, de-mosaic processing, color correction processing, gamma processing, etc.

[0100] It should be noted that during the preview of the preview image, if a shooting request for the object to be captured is detected and the shooting stage is entered, the corresponding shooting image can be obtained based on the target exposure information. Here, in the high dynamic range shooting mode, the medium exposure image (i.e., normal exposure image), overexposed image, and underexposed image are all obtained based on the target exposure information. In this way, the shooting image after fusion based on the HDR algorithm can also fully capture the motion information of the object to be captured, and meet the imaging requirements of high dynamic motion scenes.

[0101] The image processing method proposed in this disclosure is applied in high dynamic range (HVR) photography mode. When it is determined that the object to be captured is in a target motion state, the current exposure information is adjusted to obtain target exposure information. In other words, the target exposure information in this disclosure corresponds to the motion scene of the object to be captured. Thus, by performing image exposure processing using target exposure information, this disclosure can reduce imaging noise introduced by the motion of the object to be captured, improve the imaging effect of moving objects with a dynamic range, and solve the problem of high image noise and inability to adapt to high dynamic range motion scenes in traditional methods that rely on current exposure information for previewing and capturing images. This further improves the image preview effect and the subsequent image display effect.

[0102] In some embodiments, adjusting the current exposure information to obtain target exposure information in step S102 includes:

[0103] The current exposure information is adjusted based on the preset exposure coefficient to obtain the target exposure information;

[0104] The exposure factor is the exposure ratio when the object to be collected is in motion.

[0105] Here, the exposure factor is the exposure ratio when the object to be collected is in a target motion state. The current exposure information is adjusted to the target exposure information that is compatible with the HCG mode by using the preset exposure ratio.

[0106] In some examples, the exposure information may include the aperture size, which can be obtained by multiplying a preset exposure factor and the current aperture size. In other words, this disclosure adjusts the amount of light transmitted per unit time by changing the aperture size, thereby obtaining exposure information adapted to the HCG mode.

[0107] Here, the preset exposure factor can be pre-set, for example, set to 0.9.

[0108] In other examples, the exposure information may also include ISO, which can be obtained by multiplying the preset exposure factor and the current ISO.

[0109] Here, the preset exposure factor can be pre-set, for example, set to 0.9; the preset exposure factor can also be determined based on the current ISO value; specifically, the current ISO value and the preset exposure factor have a positive correlation.

[0110] The embodiments of this disclosure adjust the current exposure information by setting a preset exposure coefficient to obtain the target exposure information, so that the details of the final exposed image are richer and the motion information of the object to be collected can be effectively preserved.

[0111] In some embodiments, the method of adjusting the current exposure information based on a preset exposure coefficient to obtain target exposure information, as proposed in the above embodiments of this disclosure, includes:

[0112] Based on the exposure factor, the current exposure gain in the current exposure information, and the current exposure time, the exposure adjustment gain and exposure adjustment time are obtained;

[0113] Based on the exposure adjustment gain and exposure adjustment time, the number of exposed pixel rows and the number of pixels to be exposed in each exposed pixel row are obtained.

[0114] Here, the current exposure information is the current exposure gain and current exposure time calculated by the AEC module for the preview normal exposure. Here, the exposure gain is a parameter used to reflect brightness and noise; the exposure time, also known as the shutter speed, is used to indicate the time it takes for the shutter to open to project light onto the photosensitive surface of the photosensitive material.

[0115] Optionally, the exposure factor is 0.9. Thus, based on the exposure factor, the current exposure gain in the current exposure information, and the current exposure time, the exposure adjustment gain and exposure adjustment time are obtained. This can be implemented as follows: the exposure adjustment gain is obtained by multiplying the current exposure gain (gain) calculated by the AEC module and 0.9, and the exposure adjustment time is obtained by multiplying the current exposure time (exposuretime) calculated by the AEC module and 0.9. Then, the exposure adjustment gain and exposure adjustment time are transmitted to the image sensor to convert the number of exposed pixel rows (FrameLength) and the number of pixels to be exposed in each exposed pixel row (LineCount).

[0116] In this way, the image sensor performs pixel-by-pixel processing based on the number of exposed pixel rows (FrameLength) and the number of pixels to be exposed in each exposed pixel row (LineCount) to obtain multi-frame buffered images; here, the buffered images can be used to synthesize preview images, or to synthesize captured images during subsequent shooting processes.

[0117] This embodiment adjusts the current exposure gain and current exposure time by setting a preset exposure coefficient, thereby changing the normal exposure mode to HCG mode. This further enables the imaging of more light while reducing readout noise, resulting in brighter images and reduced motion blur of moving objects in each exposed frame. This allows scene information to be saved in a shorter time, improving the fidelity of high dynamic range scenes.

[0118] In some embodiments, see Figure 2 , Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 2 ;like Figure 2 As shown, the image processing method proposed in this disclosure further includes the following steps:

[0119] Step S201: In response to a shooting request for the object to be captured, determine the first timestamp corresponding to the shooting request;

[0120] Step S202: Based on the target exposure information, obtain the first exposure image corresponding to the first timestamp;

[0121] Step S203: Generate the captured image based on the first exposure image.

[0122] In traditional HDR photography mode, the camera application sends a shooting request for the subject to be captured. The AEC module calculates the current exposure information and transmits it to the image sensor. The sensor then de-exposes and processes the image to obtain the captured image. Because there is a time difference between the user sending the shooting request and the sensor performing the exposure, the final generated image is not the image seen by the user during the preview. Especially in high dynamic range scenes, when the subject is in motion, the actual captured image will change due to the aforementioned time difference, and the "what you see is what you get" imaging effect cannot be achieved.

[0123] Here, the first timestamp corresponding to the shooting request is the time when the user clicks to take a picture. Specifically, after the user presses the shutter, the time information of pressing the shutter is recorded by the timestamp. The timestamp starts from January 1, 1970, and continues to the current time in milliseconds.

[0124] Therefore, this disclosure obtains the first timestamp corresponding to the shooting request and generates the shooting image by determining the first exposure image that matches the first timestamp, so that the final generated shooting image is the image that the user sees in the preview screen when requesting shooting, thus improving the WYSIWYG capture experience.

[0125] It should be noted that the first exposure image is obtained by the image sensor based on the target exposure information. The first exposure image is a normal exposure image (i.e., a medium exposure image). The brightness range of the first exposure image is within the normal brightness range and can display the image details of the dynamic range normally.

[0126] This disclosure utilizes target exposure information and the first timestamp to obtain the captured image, which can satisfy the exposure processing in HCG mode, reduce motion blur of moving objects in the captured image, save scene information in a shorter time, and improve the reproduction of high dynamic scenes; it can also ensure that the final generated captured image is the image that the user sees in the preview screen when requesting to take the picture, thus improving the WYSIWYG capture experience.

[0127] See Figure 3 , Figure 3 This is a schematic diagram illustrating the principle of a traditional image sensor-based frame-taking method; for example... Figure 3 As shown, when the camera application sends a shooting request, it waits for the request to be transmitted and parsed, the AEC module calculates the current exposure information, and then the image sensor begins exposure processing based on the current exposure information, sequentially determining four normally exposed images (medium exposed images), at least one underexposed image, and at least one overexposed image. Thus, there is a relatively long waiting time between the shooting request and the sensor's exposure execution, resulting in the final generated image not being the image the user saw during the preview.

[0128] In some embodiments, the image processing method proposed in this disclosure further includes:

[0129] Based on the target exposure information, a second exposure image and a third exposure image are obtained; the brightness of the first exposure image is greater than the brightness of the second exposure image, and the brightness of the first exposure image is less than the brightness of the third exposure image.

[0130] Based on the first exposure image, a captured image is generated, including:

[0131] The first exposure image, the second exposure image, and the third exposure image are fused together to generate the captured image.

[0132] Here, the second exposure image is an underexposed image, the third exposure image is an overexposed image, and the first exposure image is a normally exposed image. The brightness of the normally exposed image is greater than that of the underexposed image, and the brightness of the normally exposed image is less than that of the overexposed image.

[0133] See Figure 4 , Figure 4 This is a schematic diagram illustrating the principle of a frame-capturing method based on an image sensor, according to an exemplary embodiment; as shown below. Figure 4 As shown, when the camera application sends a shooting request, it determines the corresponding first timestamp and obtains the normal exposure image (i.e., medium exposure image) corresponding to the first timestamp. Then, the image sensor performs exposure processing based on the target exposure information determined during the preview process to obtain underexposed and overexposed images.

[0134] In this embodiment of the disclosure, obtaining the second exposure image and the third exposure image based on the target exposure information includes:

[0135] Based on the exposure information of the target, multiple frames of underexposed original images and multiple frames of overexposed original images are obtained through exposure processing.

[0136] After image preprocessing and image fusion of multiple underexposed original images, a second exposed image is obtained;

[0137] After image preprocessing and image fusion of multiple overexposed original images, a third exposed image is obtained.

[0138] Here, both the multiple underexposed and overexposed original images are RGGB / BGGR images. The ISP center preprocesses, crops, and performs BSP hardware fusion on the multiple underexposed original images to obtain the second exposure image; the ISP center also preprocesses, crops, and performs BSP hardware fusion on the multiple overexposed original images to obtain the third exposure image. The ISP center's image preprocessing includes converting the RGGB / BGGR images to YUV images, as well as performing bad pixel correction, black level correction, white balance adjustment, de-mosaicing, color correction, and gamma adjustment.

[0139] This disclosure, after obtaining the first exposure image, the second exposure image, and the third exposure image, fuses the first exposure image, the second exposure image, and the third exposure image based on an HDR algorithm to obtain a captured image. Specifically, the HDR algorithm includes image processing algorithms based on a Laplacian pyramid model and a Gaussian pyramid model. Here, the bottom of the pyramid is a high-resolution representation of the images to be processed (the first exposure image, the second exposure image, and the third exposure image), while the top of the pyramid is a low-resolution representation. The bottom layer image in the pyramid is the basic level; if its size is N*N, then the size of the next bottom layer image is N*N, and so on, up to the top layer image. Each layer image is multiplied by its corresponding image weight, and then fused separately to obtain a fused image. The fused image is then remapped back to the original image size to obtain a captured image with high dynamic range.

[0140] In this embodiment, the first exposure image, the second exposure image, and the third exposure image are all obtained based on the target exposure parameters. This enables imaging with more light while reducing readout noise, resulting in brighter images and reduced motion blur in the final captured image. It also allows scene information to be saved in a shorter time, improving the fidelity of high dynamic scenes.

[0141] In some embodiments, step S202, obtaining the first exposure image corresponding to the first timestamp based on the target exposure information, includes:

[0142] Acquire multiple cached images based on target exposure information;

[0143] Determine the generation timestamp corresponding to each frame of cached image;

[0144] Determine the second timestamp that matches the first timestamp from among multiple generated timestamps;

[0145] The first exposure image is determined based on the cached image corresponding to the second timestamp.

[0146] It should be noted that the multi-frame cached images acquired based on exposure information occur during the image preview process; the multi-frame cached images are generated by the image sensor based on the target exposure parameters and are used to synthesize the preview image.

[0147] In this disclosure, the method for determining the second timestamp that matches the first timestamp among multiple generated timestamps can be: searching among multiple generated timestamps using a search algorithm; in this disclosure, the search algorithm can be a binary search algorithm, a sequential search algorithm, a block search algorithm, etc., and this disclosure does not impose further restrictions on it.

[0148] For example, the cached images are arranged in ascending order according to the time sequence of their generation timestamps; for example, the ascending image queue may include 12 of the aforementioned cached images; the first timestamp corresponding to the shooting request is used as the search value, and a binary search algorithm is used to match multiple generation timestamps to locate the cached image in the ascending image queue that matches the timestamp of the shooting request.

[0149] Here, determining the first exposure image by using the cached image corresponding to the second timestamp can be based on the first exposure image being obtained after preprocessing the cached image by the ISP center.

[0150] This disclosure uses a timestamp lookup algorithm to determine the timestamp corresponding to the shooting request in the cached image formed during the preview process. This improves the speed of image synthesis, enhances the user experience, and ensures that the final generated image is the same image that the user saw in the preview screen when requesting the shooting, thus improving the "what you see is what you get" capture experience.

[0151] In some embodiments, determining the first exposure image based on the cached image corresponding to the second timestamp, as proposed in the above embodiments of this disclosure, includes:

[0152] Retrieve the cached images corresponding to the first preset number of generated timestamps prior to the second timestamp;

[0153] The cached image corresponding to the second timestamp and the cached images corresponding to the first preset number of generated timestamps are subjected to image fusion processing to obtain the first exposure image.

[0154] Here, the cached image corresponding to the second timestamp is used as the base frame, and a first preset number of cached images before the second timestamp are determined from the ascending image queue as matching frames; optionally, the first preset number is 3. Combined with... Figure 4 This disclosure selects three matching frames before the reference frame.

[0155] The reference frame and three matching frames are transmitted to the ISP center for image preprocessing and image fusion to obtain the first exposure image. Here, the ISP center performs image preprocessing including converting the RGGB / BGGR image to a YUV image, as well as bad pixel correction, black level correction, white balance, demosaic, color correction, and gamma processing. Image fusion is based on the Bayer Processing Segment (BPS) hardware module of the ISP center.

[0156] In this embodiment, multiple cached images are selected from the cached images and fused to obtain a first exposure image. This reduces the adverse effects of exposure errors in the exposure processing of a single cached image on the final synthesized image, thereby improving the accuracy and quality of the generated image.

[0157] In some embodiments, determining a second timestamp that matches a first timestamp among a plurality of generated timestamps, as proposed in the above embodiments of this disclosure, includes:

[0158] When a timestamp identical to the first timestamp is found among multiple generated timestamps, the generated timestamp identical to the first timestamp is used as the second timestamp;

[0159] When no timestamp matching the first timestamp is found among multiple generated timestamps, the differences between the multiple generated timestamps and the first timestamp are calculated.

[0160] The generation timestamp corresponding to the smallest difference among multiple differences is used as the second timestamp.

[0161] Here, when a timestamp identical to the first timestamp is found among multiple generation timestamps, it can be determined that there is an image directly corresponding to the first timestamp among the cached images of multiple normally exposed frames cached during the preview process; when a timestamp identical to the first timestamp is not found among multiple generation timestamps, it can be determined that there is no image directly corresponding to the first timestamp among the cached images of multiple normally exposed frames cached during the preview process, and at this time, the cached image with the image generation time closest to the first timestamp is selected.

[0162] This disclosure can identify a generation timestamp that is the same as or similar to the first timestamp among multiple generation timestamps, so that the acquired cached image corresponds to the image seen in the preview screen when the user clicks to take a picture, thus improving the "what you see is what you get" capture experience.

[0163] In some embodiments, the image processing method proposed in this disclosure further includes:

[0164] Obtain at least two original exposure images based on the current exposure information;

[0165] A binary image is determined based on the pixel difference for the same pixel point in at least two original exposure images;

[0166] Based on the binary image and at least two original exposure images, determine the motion region in at least two original exposure images;

[0167] Feature points and feature descriptors of defined dimensions for each feature point are determined in the motion regions of at least two original exposed images.

[0168] Based on feature points and feature descriptors with defined dimensions, it is determined whether the object to be collected is in a target motion state.

[0169] Here, when the preview function of the camera application is launched and the preview process begins, an image preview request for the object to be captured is responded to, and it is determined whether the object to be captured is in a target motion state.

[0170] Specifically, the AEC module calculates the current exposure information, which includes the current exposure gain and the current exposure time. The image sensor then generates at least two consecutive frames of raw exposure images based on the current exposure gain and the current exposure time.

[0171] This disclosure can use a combination of background subtraction and SIFT-based feature point matching algorithm to determine whether the object to be acquired is in motion; for example:

[0172] First, the motion region of the original exposed image is determined using background subtraction. Specifically, the pixel difference between two frames of the original exposed image for the same pixel is calculated, and a binary image is determined based on the pixel difference. Image processing, such as masking, is then performed on at least two frames of the original exposed image based on the binary image to determine the motion region within those at least two frames. Here, a binary image refers to an image where each pixel has only two possible values ​​or grayscale levels; typically, a black-and-white image, a grayscale image, or a monochrome image is used to represent the binary image.

[0173] Secondly, based on the SIFT feature point matching algorithm, feature points and feature descriptors of a set dimension are determined in the motion region of at least two frames of original exposure images. Based on the feature points and the feature descriptors of the set dimension, it is determined whether the object to be acquired is in a target motion state.

[0174] It should be noted that SIFT feature points possess spatial scale invariance in images, remaining unchanged by rotation, scaling, brightness changes, etc., making them stable local features in images. Based on SIFT feature points, it is possible to determine the motion changes of moving objects in the motion region of at least two frames of original exposure images, thereby determining whether the object to be acquired is in a target motion state, that is, whether the current preview scene is a high dynamic range motion scene.

[0175] Specifically, determining whether the object to be acquired is in a target motion state based on the SIFT feature point matching algorithm can be implemented through the following steps:

[0176] The first step is to process at least two original exposure images at different sizes to form a Difference of Gaussian (DOG) pyramid, thereby obtaining feature scale spaces at different scales.

[0177] Here, at least two original exposure images are first processed at different sizes to obtain multiple Gaussian pyramids; each Gaussian pyramid has multiple layers of images, and the scales of the multiple layers of images in a Gaussian pyramid are different (that is, the Gaussian parameter σ is different). Here, the feature scale space of the image refers to the degree of blur of the image, that is, the degree of blur of different layers of images in each Gaussian pyramid is different, and the scales of adjacent layers of images differ by a scaling factor.

[0178] See Figure 5 , Figure 5 This is a schematic diagram illustrating the principle of constructing a Gaussian difference pyramid according to an exemplary embodiment; Figure 5 The first Gaussian pyramid a and the second Gaussian pyramid b are shown. The bottom layer image of the second Gaussian pyramid b is obtained by downsampling the image of the first Gaussian pyramid a at a scale of 2σ by a factor of 2.

[0179] After multiple Gaussian pyramids are constructed, the difference between two adjacent layers of images in the Gaussian pyramids is subtracted to obtain the difference between Gaussian pyramids.

[0180] The second step is to detect spatial extrema of the Gaussian difference pyramid to obtain local extrema.

[0181] See Figure 6 , Figure 6 This is a schematic diagram illustrating the principle of determining local extrema in a Gaussian difference pyramid according to an exemplary embodiment; Figure 6 The illustration shows a set of three-layer images in a Gaussian difference pyramid. Here, each layer of the Gaussian pyramid is an image in a two-dimensional image space. For a certain image point R in a layer, the image point R is compared with the 8 points in its 3*3 neighborhood. In the scale space within the same set of Gaussian difference pyramids, the image point R is then compared with the 2*9 points in the two adjacent layers above and below. In this way, it can be determined whether the image point R is a local extremum in both the scale space and the two-dimensional image space.

[0182] It should be noted that for an image point R in the top and bottom layers of a set of Gaussian difference pyramids, the image point R can be compared with the 8 points in its 3*3 neighborhood and the 9 points in the adjacent layer to determine whether the image point R is a local extremum.

[0183] The third step is to map the local extrema points obtained from different layers of the Gaussian difference pyramid onto two original exposure images, and then select the local extrema points as candidate feature points in the motion region.

[0184] Here, after mapping the obtained local extrema points onto the two original exposure images, the local extrema points in the motion region determined by the background subtraction method are used as candidate feature points. Here, the candidate feature points are SIFT feature points with spatial scale invariance.

[0185] The fourth step is to determine the feature distribution of the candidate feature points around the original exposed image, and obtain the 128-dimensional feature descriptor of the candidate feature points.

[0186] Here, the pixels in the scale space of the neighborhood of each candidate feature point are divided into 16*16 small domains, which are further divided into 4*4 blocks, each block consisting of 4*4 small domains; by calculating the gradient direction histogram of the corresponding pixel in each block in 8 directions, a 4*4*8=128-dimensional feature descriptor is finally obtained.

[0187] See Figure 7a , Figure 7a This is a feature distribution map of candidate feature points according to an exemplary embodiment; Figure 7b This is a schematic diagram illustrating a set-dimensional feature descriptor for candidate feature points according to an exemplary embodiment; here, Figure 7a A schematic diagram of the pixel feature distribution in a 2x2 block is shown; here, each block has 16 sub-regions, each sub-region represents a pixel, and each sub-region represents the gradient magnitude and gradient direction of each pixel. The arrow direction represents the gradient direction of the pixel, and the length represents the gradient magnitude. This disclosure uses a Gaussian window to perform weighted calculations on the gradient magnitude and gradient direction of each pixel to obtain the gradient direction histogram of the corresponding pixel in each 2x2 block in 8 directions; here, Figure 7b This shows a 2*2*8=32-dimensional feature description vector, i.e., a feature descriptor, corresponding to a 2*2 block; here, Figure 7b The center point of the grid diagram is Figure 7a The center point of the grid diagram is determined; finally, by combining the four 2*2 block feature descriptors set in the four directions of the candidate feature points, a 4*4*8=128-dimensional feature descriptor is obtained. Here, by utilizing the joint directional information of pixel neighborhoods, the algorithm's noise resistance is enhanced, while providing higher fault tolerance for subsequent feature matching containing positioning errors.

[0188] After determining the feature points and their defined dimensional feature descriptors, it can be determined whether the object to be collected is in a target motion state based on the feature points and their defined dimensional feature descriptors. Specifically:

[0189] In some embodiments, the determination of whether the object to be collected is in a target motion state based on feature points and a set dimension feature descriptor of the feature points proposed in the above embodiments of this disclosure includes:

[0190] Based on the feature points, a dimensional feature descriptor is defined to determine the first candidate feature matching pair;

[0191] When the pixel distance between two feature points in the first candidate feature matching pair exceeds a preset distance, a second candidate feature matching pair is determined.

[0192] When the number of second candidate feature matching pairs exceeds the second preset number, it is determined that the object to be collected is in a target motion state.

[0193] Here, determining the first candidate feature matching pair based on the feature descriptor of the specified dimension of the feature points can be implemented in the following way:

[0194] In some examples, Euclidean distance matching is performed on the 128-dimensional feature descriptors of candidate feature points between two adjacent frames of original exposure images, and the first candidate feature matching pair is determined based on the Euclidean distance matching result.

[0195] In other examples, the first candidate feature matching pair can be determined using the Random Sampling Consensus (RANSAC) algorithm. Here, RANSAC is an algorithm that calculates the mathematical model parameters of a sample dataset containing outliers to obtain valid sample data. The method for determining the first candidate feature matching pair based on RANSAC is as follows: Candidate feature regions are identified between two adjacent frames of raw exposure images. Several feature points are randomly selected from these candidate feature regions. Through calculation and iterative processing, the optimal parameter model is found, in which the most matching candidate feature points are found. In this way, the first candidate feature matching pair is determined from the candidate feature regions of two adjacent frames of raw exposure images.

[0196] Here, the first candidate feature matching pair includes pixels from two original exposure images that describe the same information. In this disclosure, the positional difference between pixels of two feature points in the first candidate feature matching pair is calculated. When the pixel distance between pixels of two feature points in the first candidate feature matching pair exceeds a preset distance, a second candidate feature matching pair is determined. Here, the preset distance can be a distance of 10 pixels.

[0197] When the number of second candidate feature matching pairs obtained after analyzing two original exposure images exceeds the second preset number, it can be determined that the object to be collected is in a target motion state, that is, the current preview scene is a high dynamic motion scene.

[0198] This embodiment of the invention employs a combination of SIFT feature point matching and background subtraction to detect whether the object to be captured is in a state of motion. This effectively improves the accuracy of motion scene detection, facilitates the adjustment of exposure information to achieve HCG mode, and further enhances the experience of capturing moving objects.

[0199] In some embodiments, the image processing method proposed in this disclosure further includes:

[0200] When the object to be captured is not in a state of target motion, exposure processing is performed based on the current exposure information to obtain a preview image of the object to be captured.

[0201] Here, when the object to be acquired is not in a target motion state, exposure processing is performed based on the current exposure information to obtain multiple frames of normally exposed images. These multiple frames of normally exposed images are then transmitted to the ISP center for preprocessing and BPS hardware fusion is completed to obtain a preview image for the image to be acquired.

[0202] Here, when the object to be captured is not in a target motion state, exposure processing is performed based on the current exposure information to achieve image preview in normal exposure mode, simplifying the image processing process and improving the time to generate preview images.

[0203] In some embodiments, the image processing method proposed in this disclosure further includes:

[0204] If the target motion state of the object to be captured is not processed, exposure processing is performed based on the current exposure information to obtain an image of the object to be captured.

[0205] Here, without processing the target motion state of the object to be captured, exposure processing is performed based on the current exposure information to obtain multiple frames of normally exposed, underexposed, and overexposed images. These images are then transmitted to the ISP center for preprocessing and BPS hardware fusion is performed to obtain one normally exposed image, one underexposed image, and one overexposed image. The three images are then further fused using an HDR algorithm (such as a pyramid-based fusion algorithm) to form the captured image.

[0206] The following description, in conjunction with the above embodiments of this disclosure, illustrates an exemplary application of the image processing method proposed in this disclosure.

[0207] The image processing method proposed in this disclosure has high application value in the field of electronic device snapshot capture. Firstly, with the widespread use of electronic devices, users have increasingly higher demands for snapshot capture and an optimal user experience. High Dynamic Range (HDR) technology, as a default shooting function, allows users to restore overexposed or underexposed area information in high-contrast scenes, resulting in a better dynamic range image for the entire picture. This not only significantly improves the image quality but also provides users with a perfect experience.

[0208] Depending on the image sensors used in electronic devices, traditional HDR algorithms mainly employ two implementation methods:

[0209] The first approach is that traditional image sensors produce images based on multiple exposures requested.

[0210] For details, see Figure 8 and Figure 9 , Figure 8 This is a schematic diagram of the principle of traditional exposure and image output. Figure 1 ; Figure 9 A flowchart illustrating a traditional method for generating captured images. Figure 1 ;like Figure 8 and Figure 9 As shown, a traditional sensor processes each request information frame by frame to obtain multiple frames of images. For example, it sequentially generates multiple frames of images for the (N+1)th request information up to the (N+5)th request information. Here, different request information corresponds to different exposure information. After the sensor processes each frame to generate RAW format images with different exposures, the ISP center needs to preprocess them to obtain YUV images. Finally, the multiple frames of YUV images with different brightness are fused using an HDR algorithm to generate a high dynamic range image.

[0211] The second approach is to use a staggered image sensor to output multiple exposure images at once.

[0212] Here, the stagger sensor is a type of outgoing frame hardware that reduces ghosting, effectively mitigating HDR ghosting and compositing issues; see also Figure 10 , Figure 10 This is a schematic diagram of the principle of traditional exposure and image output. Figure 2 ;like Figure 10 As shown, the stagger sensor processes multiple request messages from a single request, outputting three RAW images with different exposure values ​​for each different request message; for example... Figure 10As shown, based on a single request, a medium-exposure image corresponding to the Nth request information is generated; an underexposed image corresponding to the N+1th request information and an overexposed image corresponding to the N+2th request information are generated; then, based on the next request, a medium-exposure image corresponding to the N+3rd request information is generated; an underexposed image corresponding to the N+4th request information and an overexposed image corresponding to the N+5th request information are generated; and so on, to obtain RAW format images with different exposures corresponding to multiple requests.

[0213] See Figure 11 , Figure 11 A flowchart illustrating a traditional method for generating captured images. Figure 2 The image sensor outputs frames using either of the two methods described above, resulting in multiple RAW format images with different exposure conditions. The ISP center performs image signal processing on the RAW format images, outputting three YUV format exposure images representing different brightness levels. Specifically, image signal processing includes bad pixel correction, black level correction, white balance, demosaic, color correction, gamma processing, image cropping, and BSP fusion. Here, the three YUV format exposure images represent a normal exposure image, an overexposed image, and an underexposed image, respectively. The three YUV format exposure images are then fused using Gaussian and Laplacian pyramid algorithms to generate a single captured image with high dynamic range.

[0214] For both of the above image processing methods in HDR mode, whether it is a traditional sensor or a stagger sensor, the multiple frames of images with different exposures are all based on the current exposure information calculated by the AEC module. When the subject being captured is in motion, the motion information of the subject is significantly lost. Furthermore, in the traditional method, the user sends a shooting request through the camera application, and then the image sensor is used to obtain the image. Because the shooting request is not sent to the sensor immediately for execution, the moment when the user presses the shutter cannot be captured, and the problem of motion information loss is even more obvious when the subject being captured is in motion.

[0215] Therefore, this disclosure proposes an image processing method that, when it is determined that the object to be captured is in a target motion state, adjusts the current exposure image to the target exposure image, and then performs image exposure processing using the target exposure information. This method can fully capture the motion information of the object to be captured, reduce image noise, and improve the image preview effect and subsequent photo display effect. Furthermore, this disclosure obtains the first exposure image corresponding to the first timestamp of the shooting request and the second exposure image obtained based on the shooting request and the target exposure information, and performs HDR fusion on the second exposure image, so that the final generated shooting image is the image seen by the user in the preview screen when requesting shooting, thus improving the WYSIWYG capture experience.

[0216] See Figure 12 and Figure 13 , Figure 12 This is a schematic diagram illustrating an image preview process according to an exemplary embodiment; Figure 13 This is a schematic diagram illustrating an image capture process according to an exemplary embodiment. The following is in conjunction with... Figure 12 and Figure 13 The flowchart of the image processing method proposed in this disclosure is explained.

[0217] Here, the camera application on the electronic device has both preview and photo-taking functions;

[0218] For the preview function: after the user opens the camera application (e.g., the user opens the camera), the preview process begins; first, it is detected whether the object to be captured is in a target motion state.

[0219] If the object to be captured is not in motion, preview the image using the normal exposure mode.

[0220] Specifically, the current exposure information is set based on the conventional exposure parameters, and the image sensor outputs frames based on the current exposure information to obtain multiple normally exposed exposure images. These multiple normally exposed exposure images are transmitted to the ISP center for image signal processing and BPS hardware fusion to obtain a preview image for the image to be acquired. The preview image can then be used for preview display on the camera.

[0221] If the object to be captured is in motion, switch from normal exposure mode to high conversion gain HCG exposure mode.

[0222] Specifically, the current exposure information is acquired and adjusted based on a preset exposure factor of 0.9 to obtain target exposure information. This target exposure information includes the number of exposed pixel rows and the number of pixels to be exposed in each exposed pixel row. Based on the number of exposed pixel rows and the number of pixels to be exposed in each exposed pixel row, the image sensor performs frame output processing to obtain multiple normally exposed buffered images. The ISP center then preprocesses and fuses these multiple normally exposed buffered images to obtain a preview image. This preview image can then be used for preview display on the camera.

[0223] During the preview process, the process of determining whether the object to be collected is in a target motion state can be determined by combining the background subtraction method and the SIFT feature point matching algorithm proposed above in this disclosure, which will not be elaborated upon in this disclosure.

[0224] For the shooting function: when the user presses the shutter and the camera generates a shooting request, it first determines whether the object to be captured is in a target motion state; here, whether the object to be captured is in a target motion state can be directly obtained from the aforementioned preview process.

[0225] If the object to be captured is not in motion, the image is captured using the normal exposure mode.

[0226] Specifically, the current exposure information is acquired, and the image sensor performs frame output processing based on the current exposure information to obtain multiple frames of normally exposed, underexposed, and overexposed images. These multiple frames are then transmitted to the ISP center for image signal processing and BPS hardware fusion to obtain one normally exposed image, one underexposed image, and one overexposed image. Finally, the three images are fused using an HDR algorithm (such as a pyramid-based fusion algorithm) to form the captured image.

[0227] If the object to be captured is in motion, the image is captured based on the HCG exposure mode.

[0228] For details, see Figure 14 , Figure 14 This is a flowchart illustrating a normal exposure image search method according to an exemplary embodiment; such as Figure 14 As shown, firstly, the timestamp of the user's click to capture and send the image (first timestamp) is used as the search value to determine an ascending image queue formed by multiple cached images during the preview process in chronological order. Then, using a binary search algorithm, the first timestamp is matched with the generation timestamps of the 12 cached images in the ascending image queue to locate the cached image matching the user's click moment, which is used as the reference frame. If no generation timestamp matching the first timestamp is found, the cached image with the smallest difference from the first timestamp is selected as the reference frame. Combined with... Figure 4 This disclosure identifies a reference frame and three matching frames preceding the reference frame, and reduces the impact of noise through BPS hardware fusion, ultimately synthesizing the first exposure image as the normal exposure image.

[0229] After determining the normal exposure image, the image sensor then performs exposure processing based on the target exposure information to obtain a second exposure image and a third exposure image. Here, the second exposure image is an underexposed image, the third exposure image is an overexposed image, and the first exposure image is a normal exposure image. The brightness of the normal exposure image is greater than that of the underexposed image, and the brightness of the normal exposure image is less than that of the overexposed image. Obtaining the second and third exposure images based on the target exposure information can be achieved using the method described in the embodiments above, and will not be elaborated upon further in this disclosure.

[0230] After obtaining the first exposure image, the second exposure image, and the third exposure image, image fusion is performed based on the high dynamic range imaging algorithm to obtain the captured image.

[0231] For example, see Figure 15 , Figure 15 This is a schematic diagram illustrating a process for generating a captured image according to an exemplary embodiment; such as Figure 15 As shown, the first, second, and third exposure images are output to the Gaussian pyramid model and the Laplacian pyramid model for image fusion processing, and finally synthesized into a captured image with high dynamic range effect.

[0232] See here. Figure 16 and Figure 17 , Figure 16 This is a schematic diagram of an image captured by a traditional camera app in normal exposure mode. Figure 17 This is a schematic diagram illustrating the image effect of a camera application capturing a subject in high conversion gain exposure mode, according to an exemplary embodiment; combined with Figure 16 and Figure 17 As shown, when the image to be captured by the camera application is in motion, based on the normal exposure mode, the motion information of the subject walking is lost, and the position of the subject exceeds the area visible to the user during preview (e.g., ...). Figure 16 The dashed box area in the image); while in high conversion gain exposure mode, the captured image retains the motion information of the subject walking, and the position of the subject is located in the area seen by the user in the preview (e.g., the dashed box area in the image); Figure 17 The dashed box area in the image (in the image) enables a "what you see is what you get" capture experience.

[0233] See Figure 18 , Figure 18 This is a schematic diagram illustrating the structure of an image processing apparatus according to an exemplary embodiment; as shown below. Figure 18 As shown, the image processing apparatus 300 proposed in this embodiment includes:

[0234] The acquisition module 301 is configured to acquire the current exposure information in response to an image preview request for the object to be captured in high dynamic range shooting mode.

[0235] The adjustment module 302 is configured to adjust the current exposure information when the object to be collected is in a target motion state, so as to obtain target exposure information;

[0236] The exposure module 303 is configured to perform exposure processing based on the target exposure information to obtain a preview image of the object to be collected.

[0237] In some embodiments, the adjustment module 302 is further configured to adjust the current exposure information based on a preset exposure coefficient to obtain the target exposure information;

[0238] The exposure factor is the exposure ratio when the object to be collected is in the target motion state.

[0239] In some embodiments, the adjustment module 302 is further configured to obtain the exposure adjustment gain and the exposure adjustment time based on the exposure coefficient, the current exposure gain in the current exposure information, and the current exposure time;

[0240] Based on the exposure adjustment gain and the exposure adjustment time, the number of exposed pixel rows and the number of pixels to be exposed in each exposed pixel row are obtained.

[0241] In some embodiments, the image processing apparatus 300 further includes:

[0242] The timestamp determination module is configured to determine a first timestamp corresponding to the shooting request in response to a shooting request for the object to be collected;

[0243] The exposure image acquisition module is configured to acquire a first exposure image corresponding to the first timestamp based on the target exposure information;

[0244] The image generation module is configured to generate a captured image based on the first exposed image.

[0245] In some embodiments, the exposure image acquisition module is further configured to acquire multiple frames of cached images based on the target exposure information;

[0246] Determine the generation timestamp corresponding to the generation of each frame of the cached image;

[0247] Determine a second timestamp that matches the first timestamp from among the multiple generated timestamps;

[0248] The first exposure image is determined based on the cached image corresponding to the second timestamp.

[0249] In some embodiments, the exposure image acquisition module is further configured to acquire the cached images corresponding to a first preset number of generation timestamps prior to the second timestamp;

[0250] The cached image corresponding to the second timestamp and the cached images corresponding to the first preset number of generated timestamps are subjected to image fusion processing to obtain the first exposure image.

[0251] In some embodiments, the exposure image acquisition module is further configured to, when a timestamp identical to the first timestamp is found among a plurality of the generated timestamps, use the generated timestamp identical to the first timestamp as the second timestamp;

[0252] When no timestamp identical to the first timestamp is found among the multiple generated timestamps, the multiple differences between the multiple generated timestamps and the first timestamp are calculated.

[0253] The generation timestamp corresponding to the smallest difference among the multiple differences is used as the second timestamp.

[0254] In some embodiments, the exposure image acquisition module is further configured to acquire a second exposure image and a third exposure image based on the target exposure information; the brightness of the first exposure image is greater than the brightness of the second exposure image, and the brightness of the first exposure image is less than the brightness of the third exposure image.

[0255] The generation module is further configured to perform image fusion processing on the first exposure image, the second exposure image, and the third exposure image to generate the captured image.

[0256] In some embodiments, the image processing apparatus 300 further includes:

[0257] The motion state determination module is configured to acquire at least two frames of original exposure images based on the current exposure information; determine a binary image based on the pixel difference of the same pixel point in the at least two frames of original exposure images; determine a motion region in the at least two frames of original exposure images based on the binary image and the at least two frames of original exposure images; determine feature points and a set dimension feature descriptor of the feature points in the motion region of the at least two frames of original exposure images respectively; and determine whether the object to be acquired is in a target motion state based on the feature points and the set dimension feature descriptor of the feature points.

[0258] In some embodiments, the motion state determination module is further configured to determine a first candidate feature matching pair based on a set dimension feature descriptor of the feature points; determine a second candidate feature matching pair when the pixel distance between two feature points in the first candidate feature matching pair exceeds a preset distance; and determine that the object to be collected is in the target motion state when the number of the second candidate feature matching pairs exceeds a second preset number.

[0259] In some embodiments, the exposure module 303 is further configured to perform exposure processing based on the current exposure information to obtain the preview image of the object to be collected when the object to be collected is not in the target motion state.

[0260] See Figure 19 , Figure 19 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0261] like Figure 19 As shown, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0262] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0263] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0264] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0265] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0266] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0267] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0268] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0269] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 3G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0270] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0271] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0272] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables an electronic device to perform an image processing method, the method comprising:

[0273] In response to an image preview request for the object to be captured in high dynamic range shooting mode, obtain the current exposure information;

[0274] When the object to be collected is in a target motion state, the current exposure information is adjusted to obtain the target exposure information;

[0275] Exposure processing is performed based on the target exposure information to obtain a preview image of the object to be collected.

[0276] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0277] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: In response to an image preview request for the object to be captured in high dynamic range shooting mode, obtain the current exposure information; When the object to be collected is in a target motion state, the current exposure information is adjusted to obtain the target exposure information; Exposure processing is performed based on the target exposure information to obtain a preview image of the object to be collected. In response to a shooting request for the object to be captured, a first timestamp corresponding to the shooting request is determined; Multiple frames of cached images are obtained based on the target exposure information; The cached image is generated during the image preview process and is used to synthesize the preview image; Determine a second timestamp that matches the first timestamp from among the generation timestamps corresponding to the multiple cached images; Obtain the cached images corresponding to a first preset number of generated timestamps prior to the second timestamp; The cached image corresponding to the second timestamp and the cached images corresponding to the first preset number of generated timestamps are subjected to image fusion processing to obtain the first exposure image; A captured image is generated based on the first exposed image.

2. The image processing method according to claim 1, characterized in that, The step of adjusting the current exposure information to obtain the target exposure information includes: The target exposure information is obtained by adjusting the current exposure information based on a preset exposure coefficient. The exposure factor is the exposure ratio when the object to be collected is in the target motion state.

3. The image processing method according to claim 2, characterized in that, The step of adjusting the current exposure information based on a preset exposure coefficient to obtain the target exposure information includes: Based on the exposure factor, the current exposure gain in the current exposure information, and the current exposure time, the exposure adjustment gain and the exposure adjustment time are obtained; Based on the exposure adjustment gain and the exposure adjustment time, the number of exposed pixel rows and the number of pixels to be exposed in each exposed pixel row are obtained.

4. The method according to any one of claims 1 to 3, characterized in that, Determining a second timestamp that matches the first timestamp from among the plurality of generated timestamps includes: When a timestamp identical to the first timestamp is found among multiple generated timestamps, the generated timestamp identical to the first timestamp is used as the second timestamp; When no timestamp identical to the first timestamp is found among the multiple generated timestamps, the multiple differences between the multiple generated timestamps and the first timestamp are calculated. The generation timestamp corresponding to the smallest difference among the multiple differences is used as the second timestamp.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the target exposure information, a second exposure image and a third exposure image are obtained; the brightness of the first exposure image is greater than the brightness of the second exposure image, and the brightness of the first exposure image is less than the brightness of the third exposure image. The step of generating a captured image based on the first exposed image includes: The first exposure image, the second exposure image, and the third exposure image are subjected to image fusion processing to generate the captured image.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the current exposure information, at least two frames of original exposure images are obtained; A binary image is determined based on the pixel difference for the same pixel point in at least two frames of the original exposure images; Based on the binary image and at least two frames of the original exposure image, determine the motion region in at least two frames of the original exposure image; Feature points and feature descriptors of a set dimension are determined in the motion regions of at least two frames of the original exposure images, respectively; Based on the feature points and the feature descriptors of the defined dimensions of the feature points, it is determined whether the object to be collected is in a target motion state.

7. The method according to claim 6, characterized in that, The step of determining whether the object to be collected is in a target motion state based on the feature points and the defined dimensional feature descriptors of the feature points includes: Based on the defined dimension feature descriptor of the feature points, the first candidate feature matching pair is determined; When the pixel distance between two feature points in the first candidate feature matching pair exceeds a preset distance, a second candidate feature matching pair is determined; When the number of the second candidate feature matching pairs exceeds the second preset number, it is determined that the object to be collected is in the target motion state.

8. The method according to any one of claims 1 to 3, characterized in that, The method further includes: When the object to be captured is not in the target motion state, exposure processing is performed based on the current exposure information to obtain the preview image of the object to be captured.

9. An image processing apparatus, characterized in that, include: The acquisition module is configured to acquire the current exposure information in response to an image preview request for the object to be captured in high dynamic range shooting mode. The adjustment module is configured to adjust the current exposure information when the object to be collected is in a target motion state, so as to obtain target exposure information; The exposure module is configured to perform exposure processing based on the target exposure information to obtain a preview image of the object to be collected. The timestamp determination module is configured to determine a first timestamp corresponding to the shooting request in response to a shooting request for the object to be collected; The exposure image acquisition module is configured to acquire multiple frames of cached images based on the target exposure information; The cached image is generated during the image preview process and is used to synthesize the preview image; a second timestamp that matches the first timestamp is determined from the generation timestamps corresponding to the multiple cached images; Obtain the cached images corresponding to a first preset number of generated timestamps prior to the second timestamp; The cached image corresponding to the second timestamp and the cached images corresponding to the first preset number of generated timestamps are subjected to image fusion processing to obtain the first exposure image; The image generation module is configured to generate a captured image based on the first exposed image.

10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image processing method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described in any one of claims 1 to 8.

Images