Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The image processing method comprises the steps of obtaining current exposure information in response to an image preview request for a to-be-collected object in a high-dynamic photographing mode; under the condition that the to-be-collected object is in a target motion state, adjusting the current exposure information to obtain target exposure information; and performing exposure processing based on the target exposure information to obtain a preview image for the to-be-collected object. According to the image processing method provided by the embodiment of the invention, under the condition that the to-be-collected object is determined to be in the target motion state, the image exposure processing is performed through the target exposure information, so that imaging noise introduced by motion of the to-be-collected object can be reduced, and the imaging effect of the moving object with a dynamic range is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image acquisition and processing, and in particular to an image processing method and device, electronic equipment, and storage medium. Background Art

[0002] The high dynamic range shooting mode is a shooting mode based on high dynamic range imaging (HDR), which is mainly used to solve the problem of unsatisfactory picture quality under high dynamic range. The high dynamic range shooting mode is a technical solution that fuses multiple overexposed images, medium exposed images and underexposed images to obtain a captured image, so that the bright area is not overexposed and the dark area is not too dark, ultimately presenting better picture quality effects to users.

[0003] However, in the related art, in the high-dynamic shooting mode, the automatic exposure control module calculates the current exposure information and then transmits it to the image acquisition sensor for image exposure processing. There is still obvious image noise, which will produce artifacts in the image, so the final displayed image will obviously lose motion information. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method and device, an electronic device, and a storage medium. In the image processing method proposed in the embodiment of the present disclosure, when it is determined that the object to be collected is in a target motion state, image exposure processing is performed through target exposure information, which can reduce the imaging noise introduced by the motion of the object to be collected and improve the imaging effect of a moving object with a dynamic range.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising at least:

[0006] In response to an image preview request for an object to be captured in a high dynamic range shooting mode, obtaining current exposure information;

[0007] When the object to be captured is in a target motion state, adjusting the current exposure information to obtain target exposure information;

[0008] Exposure processing is performed based on the target exposure information to obtain a preview image of the object to be captured.

[0009] In some embodiments, the adjusting the current exposure information to obtain target exposure information includes:

[0010] Adjusting the current exposure information based on a preset exposure coefficient to obtain the target exposure information;

[0011] The exposure coefficient is the exposure ratio of the object to be captured when it is in the target motion state.

[0012] In some embodiments, the adjusting the current exposure information based on a preset exposure coefficient to obtain the target exposure information includes:

[0013] Obtaining an exposure adjustment gain and an exposure adjustment time based on the exposure coefficient, a current exposure gain in the current exposure information, and a current exposure time;

[0014] Based on the exposure adjustment gain and the exposure adjustment time, the number of exposure pixel rows and the number of pixels to be exposed in each exposure pixel row are obtained.

[0015] In some embodiments, the method further comprises:

[0016] In response to a shooting request for the object to be captured, determining a first timestamp corresponding to the shooting request;

[0017] Based on the target exposure information, acquiring a first exposure image corresponding to the first timestamp;

[0018] A captured image is generated based on the first exposure image.

[0019] In some embodiments, acquiring a first exposure image corresponding to the first timestamp based on the target exposure information includes:

[0020] Acquire a multi-frame buffer image based on the target exposure information;

[0021] Determine a generation timestamp corresponding to each frame of the cached image;

[0022] determining a second timestamp matching the first timestamp among the plurality of generated timestamps;

[0023] The first exposure image is determined based on the buffered image corresponding to the second timestamp.

[0024] In some embodiments, determining the first exposure image based on the cached image corresponding to the second timestamp includes:

[0025] Acquire the cached images corresponding to a first preset number of the generation timestamps before the second timestamp;

[0026] Image fusion processing is performed on the cache image corresponding to the second timestamp and the cache images corresponding to the first preset number of generation timestamps to obtain the first exposure image.

[0027] In some embodiments, determining a second timestamp matching the first timestamp among the plurality of generated timestamps comprises:

[0028] When a timestamp identical to the first timestamp is found among the multiple generated timestamps, the generated timestamp identical to the first timestamp is used as the second timestamp;

[0029] When a timestamp identical to the first timestamp is not found among the multiple generated timestamps, counting a plurality of differences between the multiple generated timestamps and the first timestamp respectively;

[0030] The generation timestamp corresponding to the minimum difference value among the multiple differences is used as the second timestamp.

[0031] In some embodiments, the method further comprises:

[0032] Based on the target exposure information, a second exposure image and a third exposure image are acquired; the screen brightness of the first exposure image is greater than the screen brightness of the second exposure image, and the screen brightness of the first exposure image is less than the screen brightness of the third exposure image;

[0033] The step of generating a captured image based on the first exposure image includes:

[0034] The first exposure image, the second exposure image and the third exposure image are subjected to image fusion processing to generate the captured image.

[0035] In some embodiments, the method further comprises:

[0036] Acquire at least two frames of original exposure images based on the current exposure information;

[0037] Determine a binary image based on a pixel difference of at least two frames of the original exposure images for the same pixel point;

[0038] Based on the binary image and the at least two frames of the original exposure images, determining a motion area in the at least two frames of the original exposure images;

[0039] Determine feature points and feature descriptors of set dimensions of the feature points in the motion regions of at least two frames of the original exposure images respectively;

[0040] Based on the feature points and the set dimensional feature descriptors of the feature points, it is determined whether the object to be collected is in a target motion state.

[0041] In some embodiments, the determining whether the object to be collected is in a target motion state based on the feature point and the set dimensional feature descriptor of the feature point includes:

[0042] Determining a first candidate feature matching pair based on a set dimensional feature descriptor of the feature point;

[0043] When the pixel distance between the two feature points in the first candidate feature matching pair exceeds a preset distance, determining a second candidate feature matching pair;

[0044] When the number of the second candidate feature matching pairs exceeds a second preset number, it is determined that the object to be collected is in the target motion state.

[0045] In some embodiments, the method further comprises:

[0046] When the object to be captured is not in the target motion state, exposure processing is performed based on the current exposure information to obtain the preview image for the object to be captured.

[0047] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, comprising at least:

[0048] An acquisition module is configured to acquire current exposure information in response to an image preview request for an object to be captured in a high dynamic shooting mode;

[0049] an adjustment module, configured to adjust the current exposure information to obtain target exposure information when the object to be captured is in a target motion state;

[0050] The exposure module is configured to perform exposure processing based on the target exposure information to obtain a preview image of the object to be captured.

[0051] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising at least:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] Wherein, the processor is configured to execute the image processing method as described in the first aspect above.

[0055] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, including:

[0056] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method in the first aspect described above.

[0057] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0058] The image processing method proposed in the embodiment of the present disclosure is applied in the high-dynamic shooting mode. When it is determined that the object to be captured is in the target motion state, the current exposure information is adjusted to obtain the target exposure information. That is to say, the target exposure information of the embodiment of the present disclosure corresponds to the motion scene of the object to be captured. In this way, the present disclosure performs image exposure processing through the target exposure information, which can reduce the imaging noise introduced by the motion of the object to be captured, improve the imaging effect of the moving object with a dynamic range, solve the problem that the image noise obtained by previewing and taking pictures based on the current exposure information in the traditional method is large and cannot adapt to the high-dynamic motion scene, and further improve the image preview effect and the subsequent photo display effect.

[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0061] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 ;

[0062] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 ;

[0063] Figure 3 It is a schematic diagram of the principle of a traditional frame-taking method based on an image sensor;

[0064] Figure 4 is a schematic diagram showing a principle of a frame taking method based on an image sensor according to an exemplary embodiment;

[0065] Figure 5 is a schematic diagram showing the principle of constructing a Gaussian difference pyramid according to an exemplary embodiment;

[0066] Figure 6 is a schematic diagram showing the principle of determining local extreme value points in a Gaussian difference pyramid according to an exemplary embodiment;

[0067] Figure 7a is a feature distribution diagram of candidate feature points according to an exemplary embodiment;

[0068] Figure 7b is a schematic diagram of a set dimension feature descriptor of a candidate feature point according to an exemplary embodiment;

[0069] Figure 8 This is a schematic diagram of the principle of traditional exposure and image generation. Figure 1 ;

[0070] Fig. 9 This is a traditional method for generating captured images. Figure 1 ;

[0071] Fig.10 This is a schematic diagram of the principle of traditional exposure and image generation. Figure 2 ;

[0072] Fig.11 This is a traditional method for generating captured images. Figure 2 ;

[0073] Fig.12 is a schematic diagram of a process of image preview according to an exemplary embodiment;

[0074] Fig.13 is a schematic diagram of a process of image preview according to an exemplary embodiment;

[0075] Fig.14 is a flow chart of a normal exposure image search method according to an exemplary embodiment;

[0076] Fig.15 is a schematic diagram of a process of generating a captured image according to an exemplary embodiment;

[0077] Fig.16 It is a schematic diagram of the image effect of a traditional camera application in a normal exposure mode for an object to be captured;

[0078] Fig.17 is a schematic diagram showing the effect of photographing an image of an object to be captured by a photographing application in a high conversion gain exposure mode according to an exemplary embodiment;

[0079] Fig.18 is a structural diagram of an image processing device according to an exemplary embodiment

[0080] Fig.19 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0081] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0082] See also Figure 1 , Figure 1 is a flowchart of an image processing method according to an exemplary embodiment; Figure 1 As shown, the image processing method proposed in the present disclosure includes:

[0083] Step S101, in response to an image preview request for an object to be captured in a high dynamic range shooting mode, obtaining current exposure information;

[0084] Step S102, when the object to be captured is in a target motion state, adjusting the current exposure information to obtain target exposure information;

[0085] Step S103: performing exposure processing based on the target exposure information to obtain a preview image of the object to be captured.

[0086] Here, the image processing method proposed in the present disclosure is applied in electronic devices. Here, the electronic devices include mobile terminals and portable electronic devices, the mobile terminals include but are not limited to mobile phones, tablet computers, etc.; the portable electronic devices include but are not limited to smart watches, etc., and the embodiments of the present disclosure do not limit this.

[0087] In the present disclosure, the electronic device has a photographing application, which usually has two functions of previewing and shooting. When the photographing application works in the HDR photographing mode and determines that the object to be captured is in a target motion state, it can be determined that the current image capture scene is a high dynamic motion scene. At this time, the current exposure information is adjusted to the target exposure information, and the image sensor performs image exposure processing and frames based on the target exposure information to obtain a preview image; and when the shooting function is subsequently started, the captured image can also be obtained according to the target exposure information; in this way, the present disclosure performs image exposure processing through the target exposure information, which can reduce the imaging noise introduced by the motion of the object to be captured and improve the imaging effect of moving objects with a dynamic range.

[0088] In step S101, the high dynamic range shooting mode (HDR mode) of the image is generally to form a high-quality HDR image by combining multiple exposure frames (overexposed image, medium exposure image and underexposed image) through an HDR algorithm (for example, a pyramid-based fusion algorithm). Here, the exposure information can affect the quality of the acquired multiple exposure frames, and further affect the display of motion information of the object to be collected in the synthesized HDR image.

[0089] In the high dynamic shooting mode, when receiving an image preview request for the object to be captured, the current exposure information can be obtained in response to the image preview request. Here, the exposure information includes the exposure amount, and the current exposure information can be obtained by calculating the automatic exposure control (Automatic Exposure Control, AEC) module in the electronic device; here, the AEC module is connected to the image sensor, and the AEC module automatically configures the exposure size to obtain the current exposure amount.

[0090] Here, the current exposure information obtained may also be the exposure information manually adjusted by the user; the current exposure information manually adjusted by the user is determined by sensing the touch operation of the user on the electronic device.

[0091] In step S102, it is determined whether the object to be captured is in a target motion state. If the object to be captured is in a target motion state, it is determined that the current preview scene is a high-dynamic motion scene. At this time, the current exposure information obtained in step S101 is adjusted to obtain the target exposure information.

[0092] In some examples, the object to be captured being in the target motion state includes the motion range of the object to be captured exceeding a preset range, or the motion speed of the object to be captured exceeding a preset speed. Specifically, in the original exposure images of two adjacent frames or three adjacent frames captured by the image sensor, if the object to be captured is located in the 1 / 3 area in the original exposure image of the previous frame and is located in the 2 / 3 area in the original exposure image of the next frame, it is determined that the object to be captured is in the target motion state, that is, the current preview scene is a high-dynamic motion scene; or, the motion speed of the object to be captured is determined by the target detection algorithm, and when the motion speed of the object to be captured exceeds the preset speed, for example, exceeds 0.5m / s, it is determined that the object to be captured is in the target motion state, that is, the current preview scene is a high-dynamic motion scene.

[0093] It should be noted that here, the objects to be collected include natural persons, animals, plants, sports equipment (such as drone equipment), natural scenery (such as clouds, raindrops), etc.

[0094] In other examples, the object to be collected is in a target motion state including that the number of target key points whose positions have changed in the object to be collected exceeds the preset number of key points. Here, the object to be collected includes a natural person or an animal; illustratively, when the object to be collected includes a natural person, the target key points include 17, specifically including the left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle; when more than 9 of the above target key points are detected to have position changes, it is determined that the object to be collected is in a target motion state; when the object to be collected includes an animal, the target key points may include 11, specifically including the left eye, right eye, nose, left ear, right ear, left arm, right arm, left hip, right hip, left foot and right foot; when more than 6 of the above target key points are detected to have position changes, it is determined that the object to be collected is in a target motion state.

[0095] In the present disclosure, when the object to be captured is in a target motion state, it is determined that the current preview scene is a high-dynamic motion scene, and at this time, the above-mentioned current exposure information is adjusted to obtain target exposure information.

[0096] It should be noted that the target exposure information is adapted to the High Conversion Gain (HCG) exposure mode; the current exposure information is adapted to the normal exposure mode; in the normal exposure mode, imaging is performed based on the current exposure information calculated based on the automatic exposure model, which cannot adapt to darker or brighter environments, and the imaging noise is large, resulting in the final preview image and captured image losing motion information and failing to capture a high dynamic range; in the HCG mode, the image sensor can perform exposure based on the target exposure parameters, utilize more light for imaging while reducing readout noise, retain more image details, and is suitable for imaging of high dynamic motion scenes.

[0097] Here, the current exposure information may be adjusted by looking up a pre-stored mapping table between the current exposure information and the target exposure information, or the current exposure information may be input into an adjustment calculation model, and the target exposure information may be obtained by calculation of the adjustment calculation model.

[0098] Exemplarily, when the current exposure information includes the exposure amount, the target exposure amount is outputted after data verification and data calculation processing are performed on the current exposure amount.

[0099] In step S103, after obtaining the target exposure information according to the above steps, the image sensor performs exposure processing based on the target exposure information to obtain a multi-frame buffer image, and transmits the multi-frame buffer image to the image signal processing (ISP) center; the ISP center pre-processes, crops, and fuses the multi-frame buffer image and outputs a preview image. Here, the ISP center pre-processes the multi-frame buffer image including: bad pixel correction processing, black level correction processing, white balance processing, de-mosaic processing, color correction processing, gamma processing, etc.

[0100] It should be noted that, in the process of previewing the preview image, if it is detected that the shooting request for the object to be captured enters the shooting stage, the corresponding captured image can be obtained according to the target exposure information; here, in the high-dynamic shooting mode, the obtained medium-exposure image (that is, the normal exposure image), over-exposed image, and under-exposed image are all obtained based on the target exposure information, so that the captured image fused based on the HDR algorithm can also fully capture the motion information of the object to be captured and meet the imaging requirements of high-dynamic motion scenes.

[0101] The image processing method proposed in the embodiment of the present disclosure is applied in the high-dynamic shooting mode. When it is determined that the object to be captured is in the target motion state, the current exposure information is adjusted to obtain the target exposure information. That is to say, the target exposure information of the embodiment of the present disclosure corresponds to the motion scene of the object to be captured. In this way, the present disclosure performs image exposure processing through the target exposure information, which can reduce the imaging noise introduced by the motion of the object to be captured, improve the imaging effect of the moving object with a dynamic range, solve the problem that the image noise obtained by previewing and taking pictures based on the current exposure information in the traditional method is large and cannot adapt to the high-dynamic motion scene, and further improve the image preview effect and the subsequent photo display effect.

[0102] In some embodiments, the current exposure information is adjusted in step S102 to obtain the target exposure information, including:

[0103] Adjusting the current exposure information based on a preset exposure coefficient to obtain target exposure information;

[0104] The exposure coefficient is the exposure ratio when the object to be captured is in the target motion state.

[0105] Here, the exposure coefficient is the exposure ratio when the object to be captured is in a target motion state, and the current exposure information is adjusted to the target exposure information adapted to the HCG mode through the preset exposure ratio.

[0106] In some examples, the exposure information may include the aperture size, and the target aperture size may be obtained by multiplying the preset exposure coefficient and the current aperture size. In other words, the present disclosure adjusts the amount of light per unit time by changing the aperture size, thereby obtaining exposure information adapted to the HCG mode.

[0107] Here, the preset exposure factor may be pre-set, for example, set to 0.9.

[0108] In other examples, the exposure information may also include sensitivity (ISO), and the target sensitivity may be obtained by multiplying a preset exposure factor by the current sensitivity.

[0109] Here, the preset exposure coefficient may be pre-set, for example, set to 0.9; the preset exposure coefficient may also be determined according to the current sensitivity value; specifically, the current sensitivity value has a positive correlation with the preset exposure coefficient.

[0110] The disclosed embodiment adjusts the current exposure information through a preset exposure coefficient to obtain target exposure information, so that the final exposure image has richer details and can effectively retain the motion information of the object to be captured.

[0111] In some embodiments, the above embodiments of the present disclosure provide for adjusting the current exposure information based on a preset exposure coefficient to obtain target exposure information, including:

[0112] Based on the exposure coefficient, the current exposure gain in the current exposure information, and the current exposure time, an exposure adjustment gain and an exposure adjustment time are obtained;

[0113] Based on the exposure adjustment gain and the exposure adjustment time, the number of exposure pixel rows and the number of pixels to be exposed in each exposure pixel row are obtained.

[0114] Here, the current exposure information is the current exposure gain and exposure time of the preview normal exposure calculated by the AEC module. Here, the exposure gain is a parameter used to reflect brightness and noise; the exposure time is also the shutter speed, which is used to indicate the time the shutter needs to be open to project light onto the photosensitive surface of the photosensitive material.

[0115] Optionally, the exposure coefficient is 0.9. In this way, based on the exposure coefficient, the current exposure gain in the current exposure information, and the current exposure time, the exposure adjustment gain and the exposure adjustment time are obtained. This can be implemented as follows: the exposure adjustment gain is obtained based on the product of the current exposure gain (gain) calculated by the AEC module and 0.9, and the exposure adjustment time is obtained based on the product of the current exposure time (exposuretime) calculated by the AEC module and 0.9; in this way, the exposure adjustment gain and the exposure adjustment time are transmitted to the image sensor, and the number of exposure pixel rows (FrameLength) and the number of pixels to be exposed in each exposure pixel row (LineCount) are converted.

[0116] In this way, the image sensor performs pixel-by-pixel processing according to the number of exposed pixel rows (FrameLength) and the number of pixels to be exposed in each exposed pixel row (LineCount) to obtain a multi-frame cache image; here, the cache image can be used to synthesize a preview image, or to synthesize a captured image in a subsequent shooting process.

[0117] The disclosed embodiment adjusts the current exposure gain and the current exposure time through a preset exposure coefficient, thereby adjusting the normal exposure mode to the HCG mode, and further realizing imaging using more light while reducing readout noise, which can not only make the imaging brightness brighter, but also reduce the ghosting of moving objects in each frame of exposure, thereby saving scene information in a shorter time and improving the restoration of high dynamic scenes.

[0118] In some embodiments, see Figure 2 , Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 ;like Figure 2 As shown, the image processing method proposed in the present disclosure also includes the following steps:

[0119] Step S201, in response to a shooting request for an object to be captured, determining a first timestamp corresponding to the shooting request;

[0120] Step S202, acquiring a first exposure image corresponding to a first timestamp based on the target exposure information;

[0121] Step S203: generating a captured image based on the first exposure image.

[0122] Here, in the traditional HDR photography mode, a photography request for the object to be captured is issued through the photography application, and the AEC module calculates the current exposure information and transmits it to the image sensor. The sensor then exposes and processes the captured image. This will cause a time difference between the user sending the shooting request and the sensor performing the exposure, resulting in the final generated image not being the image the user sees during the preview process. Especially in high-dynamic scenes, when the object to be captured is in a target motion state, the actual captured image will change due to the above-mentioned time difference, and the "what you see is what you get" imaging effect cannot be achieved.

[0123] Here, the first timestamp corresponding to the shooting request is the time corresponding to the moment when the user clicks to take a photo. Specifically, after the user presses the shutter button, the time information of pressing the shutter button will be recorded through the timestamp timestamp. Here, the timestamp is based on January 1, 1970 as the starting time and the number of milliseconds that have passed until the current time.

[0124] Therefore, the present invention obtains a first timestamp corresponding to the shooting request and generates a shooting image by determining a first exposure image that matches the first timestamp, so that the finally generated shooting image is the image seen in the preview screen when the user applies for shooting, thereby improving the what-you-see-is-what-you-get snapshot experience.

[0125] It should be noted that the first exposure image is obtained by exposing the image sensor according to the target exposure information. The first exposure image is a normal exposure image (that is, a medium exposure image). The brightness range of the first exposure image is within the normal brightness range and can normally display image details in the dynamic range.

[0126] The present disclosure utilizes the target exposure information and the first timestamp to obtain the captured image, which can satisfy the exposure processing in the HCG mode, so that the captured image reduces the moving object smear, saves the scene information in a shorter time, and improves the restoration degree of the high dynamic scene; it can also make the finally generated captured image the image seen in the preview screen when the user applies for shooting, thereby improving the what-you-see-is-what-you-get snapshot experience.

[0127] See also Figure 3 , Figure 3 It is a schematic diagram of the principle of a traditional frame-taking method based on an image sensor; Figure 3 As shown, when the camera application sends a shooting request, it waits for the shooting request to be transmitted and parsed, and the AEC module calculates the current exposure information. Then the image sensor starts the exposure process based on the current exposure information, and determines four frames of normal exposure images (medium exposure images), at least one frame of underexposure image, and at least one frame of overexposure image in sequence. In this way, there is a long waiting time between the shooting request and the sensor performing the exposure, resulting in the final generated image not being the image the user sees during the preview process.

[0128] In some embodiments, the image processing method proposed in the present disclosure further includes:

[0129] Based on the target exposure information, a second exposure image and a third exposure image are acquired; the screen brightness of the first exposure image is greater than the screen brightness of the second exposure image, and the screen brightness of the first exposure image is less than the screen brightness of the third exposure image;

[0130] Generating a captured image based on the first exposure image, including:

[0131] The first exposure image, the second exposure image and the third exposure image are subjected to image fusion processing to generate a captured image.

[0132] Here, the second exposure image is an underexposed image, the third exposure image is an overexposed image, and the first exposure image is a normal exposure image. The screen brightness of the normal exposure image is greater than the screen brightness of the underexposed image, and the screen brightness of the normal exposure image is less than the screen brightness of the overexposed image.

[0133] See also Figure 4 , Figure 4 is a schematic diagram showing a principle of a frame taking method based on an image sensor according to an exemplary embodiment; Figure 4 As shown, when the photo-taking application sends a shooting request, the first timestamp corresponding to it is determined, and a normally exposed image (i.e., a medium exposure image) corresponding to the first timestamp is obtained, and then the image sensor performs exposure processing according to the target exposure information determined during the preview process to obtain an underexposed image and an overexposed image.

[0134] In the embodiment of the present disclosure, based on the target exposure information, acquiring the second exposure image and the third exposure image includes:

[0135] Based on the target exposure information, exposure processing is performed to obtain multiple frames of underexposed original images and multiple frames of overexposed original images;

[0136] After performing image preprocessing and image fusion on multiple frames of underexposed original images, a second exposure image is obtained;

[0137] After image preprocessing and image fusion of multiple frames of overexposed original images, a third exposure image is obtained.

[0138] Here, the multiple frames of under-exposed original images and the multiple frames of over-exposed original images are both RGGB / BGGR images. The ISP center pre-processes, crops, and fuses the multiple frames of under-exposed original images with BSP hardware to obtain the second exposure image; the ISP center pre-processes, crops, and fuses the multiple frames of over-exposed original images with BSP hardware to obtain the third exposure image. Here, the ISP center pre-processes the images, including: converting the RGGB / BGGR images into YUV images, as well as bad pixel correction processing, black level correction processing, white balance processing, de-mosaic processing, color correction processing, Gamma processing, etc.

[0139] After obtaining the first exposure image, the second exposure image and the third exposure image, the present disclosure fuses the first exposure image, the second exposure image and the third exposure image based on the HDR algorithm to obtain a captured image. Specifically: the HDR algorithm includes an image processing algorithm based on a Laplacian pyramid model and a Gaussian pyramid model, where the bottom of the pyramid is a high-resolution representation of the image to be processed (the first exposure image, the second exposure image and the third exposure image), and the top of the pyramid is a low-resolution representation. The bottom image of the image in the pyramid is the base level. If its size is N*N, the size of the next bottom image is, and so on, until the highest image. Perform a dot multiplication of each layer of image with its corresponding image weight, and then fuse them separately to obtain a fused image; then remap the fused image to the original image size to obtain a captured image with a high dynamic range.

[0140] In the disclosed embodiment, the first exposure image, the second exposure image and the third exposure image are all obtained based on the target exposure parameters, so as to achieve imaging by utilizing more light while reducing the readout noise, which can not only make the imaging brightness brighter, but also reduce the ghosting of moving objects in the final generated image, and can save the scene information in a shorter time, thereby improving the restoration of high dynamic scenes.

[0141] In some embodiments, acquiring a first exposure image corresponding to a first timestamp based on the target exposure information in step S202 includes:

[0142] Acquire a multi-frame buffer image based on target exposure information;

[0143] Determine the generation timestamp corresponding to each frame of cached image;

[0144] determining a second timestamp among the plurality of generated timestamps that matches the first timestamp;

[0145] Based on the buffered image corresponding to the second timestamp, a first exposure image is determined.

[0146] It should be noted that the multi-frame cache image obtained based on the exposure information occurs during the image preview process; the multi-frame cache image is generated by the image sensor based on the target exposure parameters and is used to synthesize the preview image.

[0147] In the present disclosure, the method for determining a second timestamp that matches a first timestamp among multiple generated timestamps may be: searching among multiple generated timestamps through a search algorithm; in the present disclosure, the above-mentioned search algorithm may be a binary search algorithm, a sequential search algorithm, a block search algorithm, etc., and the present disclosure does not impose further restrictions on this.

[0148] Exemplarily, the cached images form an ascending image queue in the time order of the image generation timestamps; for example, the ascending image queue may include 12 frames of the above-mentioned cached images; the first timestamp corresponding to the shooting request is used as the search value, and a binary search algorithm is used to match the multiple generation timestamps to locate the cached image corresponding to the timestamp that matches the shooting request in the ascending image queue.

[0149] Here, determining the first exposure image through the cache image corresponding to the second timestamp may be based on the ISP center preprocessing the cache image to obtain the first exposure image.

[0150] The present invention determines the timestamp corresponding to the shooting request in the cache image formed in the preview process according to the timestamp search algorithm, thereby improving the synthesis speed of the shot image and the user experience, and also makes the finally generated shot image the image seen in the preview screen when the user applies for shooting, thereby improving the "what you see is what you get" snapshot experience.

[0151] In some embodiments, the method of determining the first exposure image based on the cache image corresponding to the second timestamp proposed in the above embodiments of the present disclosure includes:

[0152] Acquire cached images corresponding to a first preset number of generation timestamps before the second timestamp;

[0153] An image fusion process is performed on the cache image corresponding to the second timestamp and the cache images corresponding to the first preset number of generation timestamps to obtain a first exposure image.

[0154] Here, the cached image corresponding to the second timestamp is used as the reference frame, and the cached images of the first preset number of frames before the second timestamp are determined in the ascending image queue as matching frames; optionally, the first preset number is 3. Figure 4 , the present disclosure selects 3 matching frames before the reference frame.

[0155] The reference frame and the three matching frames are transmitted to the ISP center for image preprocessing and image fusion to obtain the first exposure image; here, the ISP center performs image preprocessing including: converting the RGGB / BGGR image into a YUV image, as well as bad pixel correction processing, black level correction processing, white balance processing, de-mosaic processing, color correction processing, Gamma processing, etc.; image fusion is based on the Bayer Processing Segment (BPS) hardware module of the ISP center.

[0156] The disclosed embodiment selects multiple cache images from the cache images for image fusion to obtain a first exposure image, thereby reducing the adverse effect of exposure error in the exposure processing of one frame of cache images on the final synthesized captured image, and improving the accuracy and quality of the generated captured image.

[0157] In some embodiments, the method of determining a second timestamp matching a first timestamp from among a plurality of generated timestamps as proposed in the above embodiments of the present disclosure includes:

[0158] When a timestamp identical to the first timestamp is found among the multiple generated timestamps, the generated timestamp identical to the first timestamp is used as the second timestamp;

[0159] When a timestamp identical to the first timestamp is not found among the multiple generated timestamps, counting a plurality of differences between the multiple generated timestamps and the first timestamp respectively;

[0160] The generation timestamp corresponding to the minimum difference value among the multiple differences is used as the second timestamp.

[0161] Here, when a timestamp identical to the first timestamp is found among multiple generated timestamps, it can be determined that the cached images of multiple frames of normal exposure cached during the preview process have an image directly corresponding to the first timestamp; when a timestamp identical to the first timestamp is not found among multiple generated timestamps, it can be determined that the cached images of multiple frames of normal exposure cached during the preview process do not have an image directly corresponding to the first timestamp, and at this time, the cached image whose image generation time is closest to the first timestamp is selected.

[0162] The present disclosure can determine a generation timestamp that is the same as or close to the first timestamp among multiple generation timestamps, so that the acquired cached image corresponds to the image seen in the preview screen when the user clicks to take a photo, thereby improving the "what you see is what you get" snapshot experience.

[0163] In some embodiments, the image processing method proposed in the present disclosure further includes:

[0164] Acquire at least two frames of original exposure images based on current exposure information;

[0165] Determine a binary image based on a pixel difference of at least two frames of original exposure images for a same pixel point;

[0166] Based on the binary image and the at least two frames of original exposure images, determining a motion area in the at least two frames of original exposure images;

[0167] Determine feature points and feature descriptors of set dimensions of the feature points in motion regions of at least two frames of original exposure images respectively;

[0168] Based on the feature points and the set dimensional feature descriptors of the feature points, it is determined whether the object to be collected is in a target motion state.

[0169] Here, when the preview function of the camera application is started to enter the preview process, a request for previewing an image of the object to be captured is responded to to determine whether the object to be captured is in a target motion state.

[0170] Specifically, the AEC module calculates current exposure information, which includes current exposure gain and current exposure time. The image sensor generates at least two frames of continuous original exposure images based on the current exposure gain and the current exposure time.

[0171] The present disclosure can use a background subtraction method and a Sift feature point matching algorithm to determine whether the object to be collected is in motion; exemplary:

[0172] First, the motion area of ​​the original exposure image is determined by using a background subtraction method; specifically, the pixel difference of the two frames of the original exposure image for the same pixel is calculated, and a binary image is determined according to the pixel difference; based on the binary image, at least two frames of the original exposure image are subjected to image processing such as mask processing, and the motion area is determined in at least two frames of the original exposure image. Here, a binary image means that each pixel on the image has only two possible values ​​or grayscale states. Usually, a black and white image, a grayscale image, or a monochrome image is used to represent the binary image.

[0173] Secondly, based on the Sift feature point matching algorithm, feature points and set dimensional feature descriptors of the feature points are determined in the motion area of ​​at least two frames of original exposure images, and based on the feature points and the set dimensional feature descriptors of the feature points, it is determined whether the object to be captured is in the target motion state.

[0174] It should be noted that Sift feature points have the characteristics of spatial scale invariance in the image, and remain invariant to rotation, scale scaling, brightness changes, etc., and are stable local features in the image. Based on Sift feature points, the motion changes of moving objects in the motion area of ​​at least two frames of original exposure images can be determined, so as to determine whether the object to be collected is in the target motion state, that is, whether the current preview scene is a high-dynamic motion scene.

[0175] Specifically, determining whether the object to be captured is in a target motion state based on the Sift feature point matching algorithm can be implemented by the following steps:

[0176] In the first step, at least two frames of original exposure images are processed at different scales to form a Difference of Gaussian (DOG) pyramid to obtain feature scale spaces at different scales.

[0177] Here, firstly, at least two frames of original exposure images are processed at different sizes to obtain multiple groups of Gaussian pyramids; each group of Gaussian pyramids has multiple layers of images, and the scales of the multiple layers of images in a group of Gaussian pyramids are different (that is, the Gaussian parameters σ are different). Here, the characteristic scale space of the image refers to the blur degree of the image, that is, the blur degrees of different layers of images in each group of Gaussian pyramids are different, and the scales of two adjacent layers of images differ by a proportional factor.

[0178] See also Figure 5 , Figure 5 is a schematic diagram showing the principle of constructing a Gaussian difference pyramid according to an exemplary embodiment; Figure 5 The first group of Gaussian pyramids a and the second group of Gaussian pyramids b are shown. The bottom-level images of the second group of Gaussian pyramids b are obtained by downsampling the images with a scale of 2σ in the first group of Gaussian pyramids a by a factor of 2.

[0179] After multiple groups of Gaussian pyramids are constructed, two adjacent layers of images in the Gaussian pyramid are subtracted to obtain the Gaussian difference pyramid.

[0180] The second step is to detect the spatial extreme points of the Gaussian difference pyramid and obtain the local extreme points.

[0181] See also Figure 6 , Figure 6 is a schematic diagram showing the principle of determining local extreme value points in a Gaussian difference pyramid according to an exemplary embodiment; Figure 6 The diagram shows a set of three-layer images in a Gaussian difference pyramid. Here, each layer of the image in the Gaussian pyramid is an image in a two-dimensional image space. For a certain image point R on a layer of images, the image point R is compared with the 8 points in its 3*3 neighborhood. In the scale space within the same set of Gaussian difference pyramids, the image point R is then compared with the 2*9 points of the two adjacent layers of images. In this way, it can be determined whether the image point R is a local extreme point in the scale space and the two-dimensional image space.

[0182] It should be noted that, for an image point R in the top and bottom images in a set of Gaussian difference pyramids, the image point R can be compared with the 8 points in its 3*3 neighborhood and the 9 points in the adjacent layer of the image to determine whether the image point R is a local extreme point.

[0183] The third step is to map the local extreme value points obtained in different layers of the Gaussian difference pyramid to the two frames of original exposure images, and select the local extreme value points as candidate feature points in the motion area;

[0184] Here, after mapping the obtained local extreme value points to two frames of original exposure images, the local extreme value points in the motion area determined by the background subtraction method are used as candidate feature points. Here, the candidate feature points are Sift feature points with spatial scale invariance.

[0185] The fourth step is to determine the feature distribution around the candidate feature point in the original exposure image, and obtain a 128-dimensional feature descriptor of the candidate feature point.

[0186] Here, the pixels in the scale space of the neighborhood of each candidate feature point are divided into 16*16 small domains, which are further divided into 4*4 blocks, each of which consists of 4*4 small domains; by calculating the gradient direction histogram of the pixels corresponding to each block in 8 directions, a 4*4*8=128-dimensional feature descriptor is finally obtained.

[0187] See also Figure 7a , Figure 7a is a feature distribution diagram of candidate feature points according to an exemplary embodiment; Figure 7b is a schematic diagram of a set dimension feature descriptor of a candidate feature point according to an exemplary embodiment; here, Figure 7a 2*2 block pixel feature distribution schematic diagram is shown; here, each block has 16 small domains, each small domain represents a pixel, each small domain represents the gradient amplitude and gradient direction of each pixel, the arrow direction represents the gradient direction of the pixel, and the length represents the gradient amplitude; the present disclosure uses a Gaussian window to perform weighted operations on the gradient amplitude and gradient direction of each pixel to obtain the gradient direction histogram of the pixels corresponding to each 2*2 block in 8 directions; here, Figure 7b A 2*2*8=32-dimensional feature description vector corresponding to a 2*2 block, i.e., a feature descriptor, is shown; here, Figure 7b The center point of the grid is Figure 7a Finally, the feature descriptors of 4 2*2 blocks set at four positions of the candidate feature points are combined to obtain a 4*4*8=128-dimensional feature descriptor. Here, by utilizing the combination of pixel neighborhood directional information, the algorithm's anti-noise ability is enhanced, and at the same time, it provides higher fault tolerance for subsequent feature matching containing positioning errors.

[0188] After determining the feature points and the feature descriptors of the set dimensions of the feature points, it is possible to determine whether the object to be collected is in a target motion state based on the feature points and the feature descriptors of the set dimensions of the feature points. Specifically:

[0189] In some embodiments, the above embodiments of the present disclosure proposes determining whether the object to be collected is in a target motion state based on feature points and feature descriptors of set dimensions of feature points, including:

[0190] Determine a first candidate feature matching pair based on a set dimension feature descriptor of the feature point;

[0191] When the pixel distance between two feature points in the first candidate feature matching pair exceeds a preset distance, determining a second candidate feature matching pair;

[0192] When the number of the second candidate feature matching pairs exceeds the second preset number, it is determined that the object to be captured is in a target motion state.

[0193] Here, based on the set dimension feature descriptor of the feature point, determining the first candidate feature matching pair can be implemented in the following manner:

[0194] In some examples, a Euclidean distance matching operation is performed on the 128-dimensional feature descriptors of the candidate feature points between two adjacent frames of original exposure images, and a first candidate feature matching pair is determined based on the Euclidean distance matching result.

[0195] In other examples, the first candidate feature matching pair can be determined by a random sampling consensus (RANSAC) algorithm; here, the RANSAC algorithm is an algorithm that calculates the mathematical model parameters of the data based on a set of sample data sets containing abnormal data to obtain valid sample data. The method of determining the first candidate feature matching pair based on the RANSAC algorithm is: determine the candidate feature area of ​​two adjacent frames of original exposure images, randomly select several feature points in the candidate feature area of ​​the two adjacent frames of original exposure images, and find the optimal parameter model through calculation and continuous iteration. In this optimal model, the most candidate feature points can be matched. In this way, the first candidate feature matching pair is determined from the candidate feature area of ​​the two adjacent frames of original exposure images.

[0196] Here, the first candidate feature matching pair includes pixels from two frames of original exposure images that describe the same information. In the present disclosure, the position difference of the pixels between the two feature points in the first candidate feature matching pair is calculated, and when the pixel distance between the pixels between the two feature points in the first candidate feature matching pair exceeds a preset distance, the second candidate feature matching pair is determined. Here, the preset distance can be a distance of 10 pixels.

[0197] When the number of second candidate feature matching pairs obtained after analyzing two frames of original exposure images exceeds a second preset number, it can be determined that the object to be captured is in a target motion state, that is, the current preview scene is a high-dynamic motion scene.

[0198] The disclosed embodiment adopts a method based on combining Sift feature point matching and background subtraction to detect whether the object to be captured is in a target motion state, which can effectively improve the accuracy of motion scene detection, facilitate adjustment of exposure information to achieve HCG mode, and further satisfy the experience of capturing moving objects.

[0199] In some embodiments, the image processing method proposed in the present disclosure further includes:

[0200] When the object to be captured is not in a target motion state, exposure processing is performed based on current exposure information to obtain a preview image of the object to be captured.

[0201] Here, when the object to be captured is not in a target motion state, exposure processing is performed based on the current exposure information to obtain multiple frames of normally exposed exposure images, and the multiple frames of normally exposed exposure images are transmitted to the ISP center for preprocessing, and the BPS hardware fusion is completed to obtain a preview image for the image to be captured.

[0202] Here, when the object to be captured is not in a target motion state, exposure processing is performed based on the current exposure information to implement image preview in a normal exposure mode, simplify the image processing flow, and improve the time for generating a preview image.

[0203] In some embodiments, the image processing method proposed in the present disclosure further includes:

[0204] In the case where the target motion state of the object to be captured is not processed, exposure processing is performed based on the current exposure information to obtain a captured image of the object to be captured.

[0205] Here, in the case where the target motion state of the object to be collected is not processed, exposure processing is performed based on the current exposure information to obtain multiple frames of normally exposed images, underexposed images and overexposed images, and the multiple frames of normally exposed images, underexposed images and overexposed images are transmitted to the ISP center for preprocessing, and BPS hardware fusion is completed to obtain a normally exposed image, an underexposed image and an overexposed image respectively; the above three processed images are further fused through an HDR algorithm (for example, a pyramid-based fusion algorithm) to form a captured image.

[0206] In combination with the above-mentioned embodiments of the present disclosure, an exemplary application of the image processing method proposed in the present disclosure is described below.

[0207] Here, the image processing method proposed in the present disclosure has a high application value in the field of snapshot of electronic devices. First of all, with the popularization of electronic devices, users have an increasing demand for snapshots and ultimate experience. High dynamic range technology (HDR: high-dynamic range) is a user's default shooting function. This function allows users to restore high-exposure or low-exposure area information in high-contrast scenes, so that the entire picture has a better dynamic range image, which not only improves the image to a high level, but also brings a perfect experience effect to users.

[0208] According to the different image sensors in electronic devices, there are two main implementation methods of traditional HDR algorithms:

[0209] The first way is that a traditional image sensor outputs images based on multiple exposure requests.

[0210] For details, see Figure 8 and Fig. 9 , Figure 8 This is a schematic diagram of the principle of traditional exposure and image generation. Figure 1 ; Fig. 9 This is a traditional method for generating captured images. Figure 1 ;like Figure 8 and Fig. 9 As shown, the traditional sensor processes each request information and generates frames one by one to obtain multiple frames of images. For example, it generates multiple frames of images of the N+1th request information to the N+5th request information in sequence. Here, the request information of different requests corresponds to different exposure information. After the sensor processes frame by frame to generate RAW format images with different exposures, the ISP center needs to pre-process them to obtain YUV images. Finally, the multiple frames of YUV images with different brightness are fused based on the HDR algorithm to generate a captured image with a high dynamic range effect.

[0211] The second method is to output multiple exposure frames at one time based on a staggered image sensor.

[0212] Here, stagger sensor is a frame output hardware that can reduce ghosting, which can effectively reduce HDR ghosting and synthesis effect problems; see Fig.10 , Fig.10 This is a schematic diagram of the principle of traditional exposure and image generation. Figure 2 ;like Fig.10 As shown, the stagger sensor processes multiple request information in one request, and outputs three RAW images with different exposure values ​​for different request information; Fig.10As shown, based on one request, a medium-exposed image corresponding to the Nth request information is generated; an underexposed image corresponding to the N+1th request information, and an overexposed image corresponding to the N+2th request information are generated; then based on the next request, a medium-exposed image corresponding to the N+3th request information is generated; an underexposed image corresponding to the N+4th request information, and an overexposed image corresponding to the N+5th request information are generated; and so on, RAW format images of different exposures corresponding to multiple requests are obtained.

[0213] See also Fig.11 , Fig.11 This is a traditional method for generating captured images. Figure 2 ; The image sensor is framed through any of the above two methods to obtain multiple RAW format images with different exposure conditions; the ISP center performs image signal processing on the RAW format image and outputs three YUV format exposure images representing different brightness; specifically: image signal processing includes bad pixel correction, black level correction, white balance, de-mosaicing, color correction, Gamma processing, image cropping, BSP fusion, etc.; here, the three YUV format exposure images are normal exposure image, over-exposure image and under-exposure image respectively; the above three YUV format exposure images are image fused based on Gaussian pyramid and Laplacian pyramid algorithms to generate a captured image with high dynamic range effect.

[0214] For the above two image processing methods in HDR mode, whether it is a traditional sensor or a stagger sensor, the multi-frame images with different exposures are all exposed based on the current exposure information calculated by the AEC module. When the object to be captured is in a target motion state, the motion information of the object to be captured is obviously lost. In addition, in the traditional method, the user sends a shooting request through a camera application, and then uses the image sensor to expose the image. In this way, because the shooting request is not immediately sent to the sensor for execution, the wonderful moment when the user presses the shutter button cannot be obtained, especially when the object to be captured is in a target motion state, the problem of motion information loss is more obvious.

[0215] Therefore, the present disclosure proposes an image processing method. When it is determined that the object to be collected is in a target motion state, the current exposure image is adjusted to the target exposure image, and then image exposure processing is performed according to the target exposure information. This method can fully capture the motion information of the object to be collected, reduce image noise, and improve the image preview effect and subsequent photo display effect. In addition, the present disclosure obtains a first exposure image corresponding to a first timestamp of a shooting request and a second exposure image obtained based on the shooting request and the target exposure information, and performs HDR fusion on the second exposure image, so that the finally generated shooting image is the image seen in the preview screen when the user applies for shooting, thereby improving the what-you-see-is-what-you-get snapshot experience.

[0216] See also Fig.12 and Fig.13 , Fig.12 is a schematic diagram of a process of image preview according to an exemplary embodiment; Fig.13 FIG. 1 is a schematic diagram of a process of image shooting according to an exemplary embodiment. Fig.12 and Fig.13 The process of the image processing method proposed in the present disclosure is described.

[0217] Here, the camera application of the electronic device has two functions: preview and take a photo;

[0218] For the preview function: after the user opens the photo-taking application (for example, the user opens the camera), the preview process begins; first, it is detected whether the object to be captured is in the target motion state.

[0219] If the object to be captured is not in a target motion state, image preview is performed based on the normal exposure mode.

[0220] Specifically, the current exposure information is set based on conventional exposure parameters, the image sensor frames are output based on the current exposure information, and multiple frames of normally exposed exposure images are obtained. The multiple frames of normally exposed exposure images are transmitted to the ISP center for image signal processing, and the BPS hardware fusion is completed to obtain a preview image for the image to be collected; the preview image can be used for preview display on the camera.

[0221] If the object to be captured is in a target motion state, the normal exposure mode is switched to the high conversion gain HCG exposure mode.

[0222] Specifically, the current exposure information is obtained, and the current exposure information is adjusted based on a preset exposure coefficient of 0.9 to obtain the target exposure information, where the target exposure information includes the number of exposed pixel rows and the number of pixels to be exposed in each of the exposed pixel rows. According to the number of exposed pixel rows and the number of pixels to be exposed in each of the exposed pixel rows, the image sensor performs frame processing to obtain multiple frames of normally exposed cached images; based on the ISP center, the multiple frames of normally exposed cached images are preprocessed and image fused to obtain a preview image; according to the preview image, it can be used for preview display on the camera for preview display.

[0223] During the preview process, the process of determining whether the object to be captured is in the target motion state can be judged by combining the background subtraction method and the Sift feature point matching algorithm proposed above in the present disclosure, and the present disclosure will not elaborate on this.

[0224] For the shooting function: when the user presses the shutter button, that is, the camera generates a shooting request, it is first determined whether the object to be captured is in a target motion state; here, the determination of whether the object to be captured is in a target motion state can be directly obtained from the aforementioned preview process.

[0225] If the object to be captured is not in a target motion state, image capture is performed based on the normal exposure mode.

[0226] Specifically, the current exposure information is obtained, and the image sensor performs frame processing based on the current exposure information to obtain multiple frames of normally exposed images, underexposed images, and overexposed images. The multiple frames of normally exposed images, underexposed images, and overexposed images are transmitted to the ISP center for image signal processing, and BPS hardware fusion is completed to obtain a normally exposed image, an underexposed image, and an overexposed image respectively; the three processed images are fused through an HDR algorithm (for example, a pyramid-based fusion algorithm) to form a captured image.

[0227] If the object to be captured is in a target motion state, image capture is performed based on the HCG exposure mode.

[0228] For details, see Fig.14 , Fig.14 is a flow chart of a normal exposure image search method according to an exemplary embodiment; Fig.14 As shown, first, the timestamp (first timestamp) sent when the user clicks to shoot is used as the search value to determine the ascending image queue formed by multiple cached images in chronological order during the preview process; based on the binary search algorithm, the first timestamp is matched with the generation timestamps of the 12 cached images in the ascending image queue to locate the cached image that matches the user's click moment and use it as the reference frame; at this time, if the generation timestamp that is the same as the first timestamp is not matched, the cached image corresponding to the generation timestamp with the smallest difference from the first timestamp is selected as the reference frame. Combined with Figure 4 , the present disclosure determines the reference frame and three matching frames before the reference frame, and reduces the influence of noise through BPS hardware fusion, and finally synthesizes the first exposure image as the normal exposure image.

[0229] After determining the normal exposure image, the image sensor then performs exposure processing according to the target exposure information to obtain the second exposure image and the third exposure image; here, the second exposure image is an underexposed image, the third exposure image is an overexposed image, the first exposure image is a normal exposure image, the screen brightness of the normal exposure image is greater than the screen brightness of the underexposed image, and the screen brightness of the normal exposure image is less than the screen brightness of the overexposed image. Performing exposure processing according to the target exposure information to obtain the second exposure image and the third exposure image can be achieved by the method proposed in the above embodiment of the present disclosure, and the present disclosure will not elaborate on this.

[0230] After the first exposure image, the second exposure image and the third exposure image are obtained, image fusion is performed based on a high dynamic range imaging algorithm to obtain a captured image.

[0231] For example, see Fig.15 , Fig.15 is a schematic diagram of a process of generating a captured image according to an exemplary embodiment; Fig.15 As shown, the acquired first exposure image, second exposure image and third exposure image are output to the Gaussian pyramid model and the Laplacian pyramid model for image fusion processing, and finally a captured image with a high dynamic range effect is synthesized.

[0232] Here, see Fig.16 and Fig.17 , Fig.16 It is a schematic diagram of an image captured by a traditional camera application in a normal exposure mode for an object to be captured; Fig.17 is a schematic diagram showing the effect of photographing an image of an object to be captured in a high conversion gain exposure mode by a photographing application according to an exemplary embodiment; Fig.16 and Fig.17 As shown, when the image to be captured by the camera application is in a target motion state, based on the normal exposure mode, the motion information of the object to be captured walking is lost, and the position of the object to be captured exceeds the position area seen by the user during the preview (such as Fig.16 In the high conversion gain exposure mode, the captured image retains the walking motion information of the object to be captured, and the position of the object to be captured is located in the position area seen by the user during the preview (such as Fig.17 The dotted box area in the figure above realizes the “what you see is what you get” snapshot experience.

[0233] See also Fig.18 , Fig.18 is a schematic diagram of the structure of an image processing device according to an exemplary embodiment; Fig.18 As shown, the image processing device 300 proposed in the embodiment of the present disclosure includes:

[0234] The acquisition module 301 is configured to acquire current exposure information in response to an image preview request for an object to be captured in a high dynamic shooting mode;

[0235] The adjustment module 302 is configured to adjust the current exposure information to obtain target exposure information when the object to be captured is in a target motion state;

[0236] The exposure module 303 is configured to perform exposure processing based on the target exposure information to obtain a preview image of the object to be captured.

[0237] In some embodiments, the adjustment module 302 is further configured to adjust the current exposure information based on a preset exposure coefficient to obtain the target exposure information;

[0238] The exposure coefficient is the exposure ratio of the object to be captured when it is in the target motion state.

[0239] In some embodiments, the adjustment module 302 is further configured to obtain an exposure adjustment gain and an exposure adjustment time based on the exposure coefficient, the current exposure gain in the current exposure information, and the current exposure time;

[0240] Based on the exposure adjustment gain and the exposure adjustment time, the number of exposure pixel rows and the number of pixels to be exposed in each exposure pixel row are obtained.

[0241] In some embodiments, the image processing apparatus 300 further includes:

[0242] A timestamp determination module is configured to, in response to a shooting request for the object to be captured, determine a first timestamp corresponding to the shooting request;

[0243] an exposure image acquisition module, configured to acquire a first exposure image corresponding to the first timestamp based on the target exposure information;

[0244] The image generation module is configured to generate a captured image based on the first exposure image.

[0245] In some embodiments, the exposure image acquisition module is further configured to acquire multiple frames of cached images based on the target exposure information;

[0246] Determine a generation timestamp corresponding to each frame of the cached image;

[0247] determining a second timestamp matching the first timestamp among the plurality of generated timestamps;

[0248] The first exposure image is determined based on the buffered image corresponding to the second timestamp.

[0249] In some embodiments, the exposure image acquisition module is further configured to acquire the cached images corresponding to a first preset number of the generation timestamps before the second timestamp;

[0250] Image fusion processing is performed on the cache image corresponding to the second timestamp and the cache images corresponding to the first preset number of generation timestamps to obtain the first exposure image.

[0251] In some embodiments, the exposure image acquisition module is further configured to use the generated timestamp identical to the first timestamp as the second timestamp when a timestamp identical to the first timestamp is found among the multiple generated timestamps;

[0252] When a timestamp identical to the first timestamp is not found among the multiple generated timestamps, counting a plurality of differences between the multiple generated timestamps and the first timestamp respectively;

[0253] The generation timestamp corresponding to the minimum difference value among the multiple differences is used as the second timestamp.

[0254] In some embodiments, the exposure image acquisition module is further configured to acquire a second exposure image and a third exposure image based on the target exposure information; the screen brightness of the first exposure image is greater than the screen brightness of the second exposure image, and the screen brightness of the first exposure image is less than the screen brightness of the third exposure image;

[0255] The generation module is further configured to perform image fusion processing on the first exposure image, the second exposure image and the third exposure image to generate the captured image.

[0256] In some embodiments, the image processing apparatus 300 further includes:

[0257] The motion state determination module is configured to obtain at least two frames of original exposure images based on the current exposure information; determine a binary image based on the pixel difference of the at least two frames of the original exposure images for the same pixel point; determine the motion area in the at least two frames of the original exposure images based on the binary image and the at least two frames of the original exposure images; determine feature points and set dimensional feature descriptors of the feature points in the motion area of ​​the at least two frames of the original exposure images respectively; determine whether the object to be collected is in the target motion state based on the feature points and the set dimensional feature descriptors of the feature points.

[0258] In some embodiments, the motion state determination module is further configured to determine a first candidate feature matching pair based on a set dimensional feature descriptor of the feature point; determine a second candidate feature matching pair when the pixel distance between two of the feature points in the first candidate feature matching pair exceeds a preset distance; and determine that the object to be collected is in the target motion state when the number of the second candidate feature matching pairs exceeds a second preset number.

[0259] In some embodiments, the exposure module 303 is further configured to perform exposure processing based on the current exposure information to obtain the preview image for the object to be captured when the object to be captured is not in the target motion state.

[0260] See also Fig.19 , Fig.19 8 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0261] like Fig.19 As shown, the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0262] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0263] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0264] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0265] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0266] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0267] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0268] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0269] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 3G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0270] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0271] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0272] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables an electronic device to perform an image processing method, the method comprising:

[0273] In response to an image preview request for an object to be captured in a high dynamic range shooting mode, obtaining current exposure information;

[0274] When the object to be captured is in a target motion state, adjusting the current exposure information to obtain target exposure information;

[0275] Exposure processing is performed based on the target exposure information to obtain a preview image of the object to be captured.

[0276] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0277] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: In response to an image preview request for an object to be captured in a high dynamic range shooting mode, obtaining current exposure information; When the object to be captured is in a target motion state, adjusting the current exposure information to obtain target exposure information; Exposure processing is performed based on the target exposure information to obtain a preview image of the object to be captured.

2. The image processing method according to claim 1, characterized in that: The adjusting the current exposure information to obtain target exposure information includes: Adjusting the current exposure information based on a preset exposure coefficient to obtain the target exposure information; The exposure coefficient is the exposure ratio of the object to be captured when it is in the target motion state.

3. The image processing method according to claim 2, characterized in that: The adjusting the current exposure information based on a preset exposure coefficient to obtain the target exposure information includes: Obtaining an exposure adjustment gain and an exposure adjustment time based on the exposure coefficient, a current exposure gain in the current exposure information, and a current exposure time; Based on the exposure adjustment gain and the exposure adjustment time, the number of exposure pixel rows and the number of pixels to be exposed in each exposure pixel row are obtained.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to a shooting request for the object to be captured, determining a first timestamp corresponding to the shooting request; Based on the target exposure information, acquiring a first exposure image corresponding to the first timestamp; A captured image is generated based on the first exposure image.

5. The method according to claim 4, characterized in that The acquiring, based on the target exposure information, a first exposure image corresponding to the first timestamp, comprises: Acquire a multi-frame buffer image based on the target exposure information; Determine a generation timestamp corresponding to each frame of the cached image; determining a second timestamp matching the first timestamp among the plurality of generated timestamps; The first exposure image is determined based on the buffered image corresponding to the second timestamp.

6. The method according to claim 5, characterized in that The determining the first exposure image based on the cached image corresponding to the second timestamp includes: Acquire the cached images corresponding to a first preset number of the generation timestamps before the second timestamp; Image fusion processing is performed on the cache image corresponding to the second timestamp and the cache images corresponding to the first preset number of generation timestamps to obtain the first exposure image.

7. The method according to claim 5, characterized in that The step of determining a second timestamp matching the first timestamp from among the plurality of generated timestamps comprises: When a timestamp identical to the first timestamp is found among the multiple generated timestamps, the generated timestamp identical to the first timestamp is used as the second timestamp; When a timestamp identical to the first timestamp is not found among the multiple generated timestamps, counting a plurality of differences between the multiple generated timestamps and the first timestamp respectively; The generation timestamp corresponding to the minimum difference value among the multiple differences is used as the second timestamp.

8. The method according to claim 4, characterized in that The method further comprises: Based on the target exposure information, a second exposure image and a third exposure image are acquired; the screen brightness of the first exposure image is greater than the screen brightness of the second exposure image, and the screen brightness of the first exposure image is less than the screen brightness of the third exposure image; The step of generating a captured image based on the first exposure image includes: The first exposure image, the second exposure image and the third exposure image are subjected to image fusion processing to generate the captured image.

9. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire at least two frames of original exposure images based on the current exposure information; Determine a binary image based on a pixel difference of at least two frames of the original exposure images for the same pixel point; Based on the binary image and the at least two frames of the original exposure images, determining a motion area in the at least two frames of the original exposure images; Determine feature points and feature descriptors of set dimensions of the feature points in the motion regions of at least two frames of the original exposure images respectively; Based on the feature points and the set dimensional feature descriptors of the feature points, it is determined whether the object to be collected is in a target motion state.

10. The method according to claim 9, characterized in that The determining whether the object to be collected is in a target motion state based on the feature point and the set dimensional feature descriptor of the feature point includes: Determining a first candidate feature matching pair based on a set dimensional feature descriptor of the feature point; When the pixel distance between the two feature points in the first candidate feature matching pair exceeds a preset distance, determining a second candidate feature matching pair; When the number of the second candidate feature matching pairs exceeds a second preset number, it is determined that the object to be collected is in the target motion state.

11. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the object to be captured is not in the target motion state, exposure processing is performed based on the current exposure information to obtain the preview image for the object to be captured.

12. An image processing device, characterized in that: include: An acquisition module is configured to acquire current exposure information in response to an image preview request for an object to be captured in a high dynamic shooting mode; an adjustment module, configured to adjust the current exposure information to obtain target exposure information when the object to be captured is in a target motion state; The exposure module is configured to perform exposure processing based on the target exposure information to obtain a preview image of the object to be captured.

13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the image processing method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as claimed in any one of claims 1 to 11.

Images