Image processing method and terminal equipment

Abstract

The embodiment of the invention discloses an image processing method and terminal equipment, and is suitable for the technical field of computers, and the image processing method comprises the steps: receiving a photographing instruction which is used for triggering the photographing of a current photographing scene; acquiring N frames of images for the current photographing scene, wherein N is an integer greater than or equal to 2; n glare parameters corresponding to the N frames of images are determined according to the pixel features of each frame of image in the N frames of images, and each glare parameter is used for representing the glare influence degree of the image; according to the N glare parameters, a first image is output, the first image is an image with the glare parameter meeting a first preset condition in the N frames of images, and the glare parameter meeting the first preset condition includes that the glare influence degree represented by the glare parameter is minimum in the N glare parameters. According to the embodiment of the invention, the imaging quality of the terminal equipment can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method and terminal device. Background Art

[0002] Dazzle is a visual condition caused by an unsuitable brightness distribution or extreme brightness contrast in space or time, resulting in visual discomfort and reduced visibility. Typically, glare is caused by high-intensity light sources interfering with multiple reflective surfaces. The essence of glare is the superposition of light caused by refraction within a lens.

[0003] The optical performance of the lenses of mobile phones and other terminal devices cannot reach the optimal level due to limitations in size and material. When shooting scenes with strong light sources, serious glare problems often occur, affecting the final image quality. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an image processing method and a terminal device, which can effectively improve the imaging quality of the terminal device.

[0005] In a first aspect, embodiments of the present application provide an image processing method, comprising: upon receiving a capture instruction, a terminal device may acquire N frames of images of a current capture scene. The terminal device may then determine, based on pixel features of each frame, corresponding glare parameters for each frame, thereby obtaining N corresponding glare parameters. The terminal device may then output a first image based on the glare parameters corresponding to each frame.

[0006] The photo command is used to trigger taking a photo of the current photo scene. The photo command can be a voice photo command or a photo command triggered by a user operation. The user operation can include but is not limited to: pressing a physical key, touching or clicking a photo control, triggering a photo operation through a third-party device (such as headphones), etc.

[0007] The first image is an image whose glare parameter satisfies a first preset condition among the N frames of images. The glare parameter satisfies the first preset condition, including: the glare impact degree represented by the glare parameter is the smallest among the N glare parameters.

[0008] Each frame of image acquired by the terminal device can also be called an original image.

[0009] Optionally, the glare parameter may include at least one of the following: contrast, a first glare degree, a second glare degree, and a probability coefficient, wherein the first glare degree is the overall glare degree of the image, the second glare degree is the glare degree of the target area in the image, and the probability coefficient is the probability of glare existing in the image.

[0010] The target area is the area where the first object in the image is located. The first object may be an object of interest. For example, the first object may be a human face, a human portrait, or an animal.

[0011] The first object can be a default object, such as a face. The first object can also be an object set by the user. For example, the terminal device can respond to the user clicking an object on the photo preview interface, such as a puppy, and set the object as the first object.

[0012] In the embodiment of the present application, since mobile phones and other terminal devices may experience more or less jitter during the process of taking photos, and since in many cases, a slight movement of the terminal device may avoid introducing glare, that is, in the process of the terminal device responding to the user's first operation, the glare degree of each frame image (or called the original image) obtained is often different. Some original images have a large glare degree, some original images have a small glare degree, and some may even have no glare. Therefore, the terminal device filters the first image based on the glare impact degree of each original image, and can filter out the first image that is least affected by glare. The resulting image of the photo obtained based on the first image often has good image quality. In other words, the present application can effectively improve the imaging quality of the terminal device.

[0013] It should be pointed out that since the first glare degree, contrast and probability coefficient can all reflect the degree of glare influence of the original image as a whole, and the second glare degree can reflect the degree of glare influence of the user's area of ​​interest in the original image, the first image can be filtered based on any one of the first glare degree, the second glare degree, the contrast and the probability coefficient, and the first image with less glare influence can be accurately filtered.

[0014] In some optional embodiments, when the glare parameter includes contrast, the terminal device may determine the glare parameter of each frame of the original image in the following manner: determine the contrast of the original image according to the image mean and image standard deviation of the original image.

[0015] In practice, the terminal device may determine the ratio of the image mean to the image standard deviation as the contrast of the original image.

[0016] In the embodiments of the present application, since the ratio of the mean to the standard deviation of a set of data accurately reflects the degree of dispersion of that data, the ratio of the image mean to the image standard deviation accurately reflects the degree of dispersion of each pixel (or pixel point) in the image. Furthermore, since images affected by glare generally exhibit greater dispersion as the glare becomes more severe, the contrast ratio determined by the ratio of the image mean to the image standard deviation accurately describes the degree of glare.

[0017] Optionally, when the glare parameter includes contrast, the first preset condition may include: selecting an original image with the smallest contrast as the first image.

[0018] In some optional embodiments, when the glare parameter includes a first glare intensity, the terminal device may determine the glare parameter for each frame of the original image by determining the first glare intensity of the original image based on the glare index of each pixel in the original image. The glare index of a pixel is the difference between the brightness and saturation corresponding to the pixel. The brightness and saturation are, respectively, brightness and saturation in HSV space.

[0019] In practice, the terminal device may determine the average glare index of all pixels in the original image as the first glare degree of the original image.

[0020] In the embodiment of the present application, since in the HSV space, the greater the brightness of a pixel, the brighter it is, and the smaller the brightness, the darker it is; the greater the saturation of a pixel, the darker the color, and the smaller the saturation, the closer it is to white, and glare usually appears as bright white, therefore, the difference between the brightness and saturation of the pixel can better describe the glare situation at a certain pixel, that is, the glare index can well describe the degree of glare impact of a certain pixel, and the first glare degree determined by the average value of the glare index of each pixel in the original image can better describe the degree of glare impact of the entire original image.

[0021] It is understandable that in some application scenarios, the first glare degree may also be the root mean square value of the glare index of all pixels in the original image.

[0022] Optionally, when the glare parameter includes a first glare degree, the first preset condition may include: selecting an original image with the minimum first glare degree as the first image.

[0023] In some optional embodiments, when the glare parameters include a second glare degree, the terminal device may determine the glare parameters of each frame of the original image in the following manner: determining the second glare degree of the corresponding original image based on the glare index of each pixel point in the target area of ​​the corresponding original image.

[0024] In practice, the terminal device may determine the average of the glare indices of the pixels in the target area as the second glare intensity of the target area. The second glare intensity may better describe the degree of glare influence of the target area in the original image.

[0025] Optionally, when the glare parameter includes the second glare degree, the first preset condition may include: selecting an original image with the minimum second glare degree as the first image.

[0026] It should be noted that the second glare degree can highlight the glare condition in the area of ​​interest to the user. Filtering the first image based on the second glare degree helps to filter out an image that better meets the user's needs, thereby further improving the imaging quality of the terminal device.

[0027] It can be understood that the first glare degree is the glare degree of the entire image range, and the second glare degree is the glare degree of the target area range in the image.

[0028] In some optional embodiments, when the glare parameter includes a probability coefficient, the terminal device may determine the glare parameter of each frame of the original image in the following manner: input the original image into a pre-trained glare detection model to obtain the probability coefficient of the original image.

[0029] In practice, the glare detection model can be a neural network model. The input of the glare detection model can be an image, and the output is the probability of glare in the image. The probability coefficient of the original image can accurately reflect the probability of glare in the original image.

[0030] Optionally, when the glare parameter includes a probability coefficient, the first preset condition may include: selecting an original image with a minimum probability coefficient as the first image.

[0031] In some optional embodiments, the glare parameter may include one of the four elements: contrast, first glare intensity, second glare intensity, and probability coefficient, or may include multiple of these four elements simultaneously. When the glare parameter includes multiple of these four elements, the terminal device may directly use all or part of these elements to filter the first image. For example, when the glare parameter includes all four elements simultaneously, the original image with the lowest second glare intensity may be directly selected as the first image. The terminal device may also add up the weighted values ​​of each element to comprehensively filter the first image. For any element, the weighted value of that element may be the product of the value of that element and the corresponding weighted coefficient.

[0032] In a first possible implementation manner of the first aspect, for any frame of the original image, the terminal device may determine the glare parameter of the original image according to pixel features of the original image by:

[0033] First, the terminal device may determine a first glare degree of the original image based on a glare index of each pixel in the original image, wherein the glare index of a pixel is a difference between brightness and saturation corresponding to the pixel.

[0034] Then, when the terminal device detects that the original image includes the first object, it can determine the second glare degree of the original image based on the glare index of each pixel in the target area, where the target area is the area in the target image where the first object is located.

[0035] Finally, the terminal device may determine a first weighted value of the original image according to the first glare degree, the second glare degree, and a first weight coefficient for the second glare degree.

[0036] At this time, the glare parameter may include a first glare degree, a second glare degree, and a first weighted value.

[0037] Here, the terminal device will continue to determine the second glare degree for the original image only when it detects that the original image includes the first object. Conversely, if the terminal device does not detect that the original image includes the first object, it does not need to determine the second glare degree for the original image.

[0038] In an embodiment of the present application, since the first object is an object of interest to the user, when the terminal device includes the first object in the original image, it comprehensively considers the first glare degree and the second glare degree, and filters the first image based on the first weighted value obtained by comprehensively determining the first glare degree and the second glare degree. This helps to filter the first image with a small degree of glare influence and that meets the user's needs, and helps to further improve the imaging quality of the terminal device.

[0039] In a second possible implementation of the first aspect, the terminal device may select the first image from the multiple frames of original images in the following manner: when the terminal device detects that each original image includes the first object, the terminal device selects the original image with the smallest first weighted value as the first image. When the terminal device detects that each original image does not include the first object, the terminal device selects the original image with the smallest first glare level as the first image.

[0040] In an embodiment of the present application, when the terminal device includes the first object in each original image, that is, when there is a target area, the image is screened with a first weighted value; when the first object is not included in each original image, that is, when there is no target area, the image is screened with a first glare degree. This helps to filter out a first image with a small degree of glare and that meets user needs, and helps to further improve the imaging quality of the terminal device.

[0041] It should be pointed out that since the target may move quickly during the photo-taking process, that is, a part of the original image may include the first object, while another part of the original image may not include the first object. At this time, the terminal device can filter the first image from the part of the original image that includes the first object, which helps to further ensure the imaging quality of the terminal device.

[0042] In a third possible implementation of the first aspect, for each frame of the original image, the terminal device determines the second glare degree of the original image based on the glare index of each pixel point in the target area when detecting that the original image includes the first object. Specifically, it can be: when the terminal device detects that there is glare in the original image and the original image includes the first object, the terminal device determines the second glare degree of the original image based on the glare index of each pixel point in the target area.

[0043] In the embodiment of the present application, the second glare degree of the original image is determined only when there is glare and a target area in the original image. This avoids determining the second glare degree of the original image when there is no glare, helping to save unnecessary computing resources.

[0044] In a fourth possible implementation of the first aspect, for any frame of an original image, the terminal device determines a glare parameter of the original image based on pixel features of the original image. Specifically, when the terminal device detects glare in the original image and the original image does not include the first object, it determines the contrast of the original image based on the image mean and image standard deviation of the original image. Subsequently, the terminal device may determine a second weighted value for the original image based on the first glare level, the contrast, and a second weighting coefficient for the contrast. In this case, the glare parameter may further include the second weighted value.

[0045] In an embodiment of the present application, when an original image contains glare but does not include the first object, the terminal device can further determine a second weighted value for the original image by combining the first glare degree and contrast. This allows the first image to be screened for images less affected by glare, even when the original image does not contain the target area, thereby further improving the imaging quality of the terminal device.

[0046] In a fifth possible implementation of the first aspect, the terminal device may select the first image from the N frames of original images in the following manner: when it is detected that each image includes the first object, the original image with the smallest first weighted value is selected as the first image. When it is detected that each image does not include the first object, the original image with the smallest second weighted value is selected as the first image.

[0047] In an embodiment of the present application, when the first object is included in each original image, that is, when there is a target area, the terminal device can filter the first image with a first weighted value. When the first object is not included in each original image, that is, when there is no target area, the terminal device can filter the first image with a second weighted value, which helps to filter out the first image that is less affected by glare.

[0048] In a sixth possible implementation of the first aspect, before determining the glare parameter of each original image frame based on pixel features of each original image frame, the terminal device may further perform the following operations: first, the terminal device may perform glare detection on each original image frame to obtain a first detection result for each original image frame, where the first detection result indicates whether glare exists in the original image. Thereafter, if glare exists in each original image frame, the terminal device may continue to perform object detection on each original image frame to obtain a second detection result, where the second detection result indicates whether the original image includes the first object and the position of the first object in the original image.

[0049] In an embodiment of the present application, the terminal device may further detect whether each original image includes the first object only when there is glare in each frame of the original image, which can save computing power and improve the overall performance of the terminal device.

[0050] It is understandable that the terminal device can use the method disclosed in the relevant technology to perform glare detection or target detection on the image. The embodiment of the present application does not specifically limit how to perform glare detection or target detection on the image.

[0051] In a seventh possible implementation of the first aspect, the terminal device may further perform the following operation: when there is an image without glare in the N frames of original images, output a second image, where the second image is an image whose frame type satisfies a second preset condition in the original image without glare, wherein the frame types include long frames, short frames, and standard frames, and the frame type satisfies the second preset condition, including: the frame type is a standard frame.

[0052] In an embodiment of the present application, when one or more original images are free of glare, the terminal device can select and output a standard frame type original image from these glare-free original images as the second image. This direct selection method reduces computational complexity and helps improve frame selection efficiency.

[0053] In an eighth possible implementation of the first aspect, the terminal device outputs a first image based on the glare parameters corresponding to each frame of image. Specifically, the terminal device may first obtain a first brightness, where the first brightness is the brightness of light in the current photographic scene. The terminal device may then determine a first coefficient for the first image based on the pixel values ​​of each pixel in the first image. The first coefficient is the proportion of a target pixel in the first image, where the pixel value of the target pixel is the value obtained when the pixel is overexposed. The terminal device may then determine a resulting photographic image based on the N frames of image, the first brightness, and the first coefficient, and output the resulting photographic image.

[0054] In practice, the terminal device can obtain the first brightness through an ambient light sensor. The terminal device can directly calculate the first coefficient of the first image. When the original image is an 8-bit image, the value of the target pixel can be 255. The first coefficient can indicate the overexposure ratio of the original image.

[0055] In an embodiment of the present application, the terminal device determines the first brightness (also referred to as the ambient light brightness) of the current photographic scene and the first coefficient (also referred to as the overexposure ratio or overexposure coefficient) of the first image, and can further determine the resulting photographic image based on the first brightness and the first coefficient. For example, when the ambient light brightness is high and the overexposure coefficient of the first image is low, it generally indicates that the first image obtained by screening is of good quality. The terminal device can directly output the first image as the resulting photographic image. This can effectively reduce unnecessary computational effort while ensuring imaging quality, thereby helping to improve the overall performance of the terminal device.

[0056] In a ninth possible implementation of the first aspect, the terminal device determines and outputs a photographic result image based on N frames of images, the first brightness, and the first coefficient, which may include: when the first brightness is greater than a preset brightness value and the first coefficient is less than or equal to a preset coefficient value, determining the first image as the photographic result image.

[0057] In an embodiment of the present application, when the ambient light brightness is high and the overexposure coefficient of the first image is small, the terminal device directly determines the first image as the photo result image for output, which can effectively reduce unnecessary calculations while ensuring the imaging quality, thereby helping to improve the overall performance of the terminal device.

[0058] In a tenth possible implementation of the first aspect, the terminal device determines and outputs a photographic result image based on N frames of images, the first brightness, and the first coefficient. This may include: when the first brightness is less than or equal to a preset brightness value and the first coefficient is less than or equal to a preset coefficient value, for any other frame, the terminal device may perform image registration processing on the other frame based on the first image, thereby obtaining a first result image of the other frame. The other frames are images other than the first image in the N frames of original images. The terminal device may then perform multi-frame noise reduction processing on the first denoised image set to obtain a third image, wherein the first denoised image set includes the first image and the first result images corresponding to each of the other frames. Finally, the terminal device may determine the third image as the photographic result image.

[0059] In the embodiment of the present application, when the ambient light is low and the overexposure factor of the first image is small, which generally indicates dim lighting, noise is likely to appear in each frame of the original image, including the first image. The terminal device can first perform image registration on each frame of the original image, and then perform multi-frame noise reduction on each of the registered original images to obtain a third image. Finally, this third image is output as the photo result, resulting in a higher-definition photo result image, which can further improve the imaging quality of the terminal device.

[0060] It should be noted that due to slight differences in the acquisition time of each frame of the original image—for example, some original images are acquired slightly earlier and some slightly later—there are spatial deviations between the multiple frames of the original image. Image registration is performed on each frame of the original image, using the first image with the least glare effect as a reference. This can align multiple images at the same spatial position, avoiding image noise caused by spatial position deviations. It can also reduce noise interference introduced by glare and enhance the effective signal in the image. In other words, performing image registration on each frame of the original image helps further improve the imaging quality of the terminal device.

[0061] It is understandable that the embodiments of the present application do not specifically limit the specific operations or algorithms used in the image registration process.

[0062] It is understandable that in some application scenarios, the terminal device may not perform registration processing on the original images, that is, the terminal device may directly perform multi-frame noise reduction processing on N frames of original images.

[0063] In an eleventh possible implementation of the first aspect, the terminal device determines and outputs a resulting photographic image based on N frames of images, the first brightness, and the first coefficient. This may include: when the first coefficient is greater than a preset coefficient value, the terminal device may perform image registration processing on any other frame based on the first image. The terminal device may then perform multi-frame noise reduction processing on each image subset in the first denoised image set to obtain a fourth image corresponding to each image subset, where the image subset is a collection of images with the same frame type in the first denoised image set. Finally, the terminal device may perform high dynamic range imaging processing on the fourth images corresponding to each frame type to obtain a fifth image, and determine the fifth image as the resulting photographic image.

[0064] In an embodiment of the present application, when the overexposure coefficient of the first image is large, it usually indicates that the first image is greatly affected by glare. Taking into account that in the same photographic scene, the information provided by original images of different frame types is different, specifically, the standard frame can provide information of the standard brightness area, the short frame can provide information of the dark area, and the long frame can provide information of the highlight area, the terminal device first performs image registration processing on the original images of each frame with the first image as a reference, and then performs multi-frame noise reduction processing on the original images after the registration processing of each frame type, and finally performs high dynamic range imaging processing on the denoised result images corresponding to the three frame types (that is, the fourth image), so as to achieve effective fusion of the scene information carried by images of different frame types, and obtain photographic result images with complementary information, which helps to further improve the imaging quality of the terminal device.

[0065] It should be pointed out that since there are relatively few scenes where the overexposure coefficient of the first image is high, that is, there are fewer situations where it is necessary to continue to perform high dynamic range imaging processing based on the denoising result images of different frame types. Therefore, in the embodiment of the present application, different processing operations are performed on each original image including the first image in combination with the different values ​​of the ambient light brightness and the overexposure coefficient of the first image, which can effectively reduce the computational complexity of the terminal device, that is, it can improve the overall performance of the terminal device.

[0066] It is understandable that the terminal device performs multi-frame noise reduction on the image subset of each frame type before performing high dynamic range imaging processing in order to reduce the impact of noise and further improve the imaging quality of the terminal device. In some application scenarios, the terminal device may not perform multi-frame noise reduction on the image subset of each frame type. In other words, the terminal device may directly perform high dynamic range imaging processing on the first noise-reduced image set to obtain the resulting image.

[0067] In a twelfth possible implementation manner of the first aspect, the terminal device uses the first image as a reference and performs image registration processing on each of the other frames, specifically:

[0068] First, the terminal device may determine the image gain between the first image and the other frames when the frame type of the other frames is a long frame or a short frame. The image gain is the ratio of the image brightness of the first image to the image brightness of the other frames, where the image brightness is the product of the exposure time and the sensitivity of the corresponding image.

[0069] Then, the terminal device may perform a first alignment process on the other frames according to the image gain to obtain first aligned images corresponding to the other frames. The first alignment process is used to adjust the image brightness of the other frames to be the same as the image brightness of the first image.

[0070] Finally, the terminal device performs image registration processing on the first aligned image with the first image as a reference.

[0071] In an embodiment of the present application, considering that the N frames of original images acquired by the terminal device often include both long-frame type original images and short-frame type original images, and there is usually a large brightness difference between the original images of different frame types in the same shooting scene, the terminal device aligns the brightness of the long-frame type original images and the short-frame type original images with the first image respectively, which can avoid the registration error caused by the obvious brightness difference between the images, thereby ensuring the accuracy of the image registration processing, and helping to further improve the final imaging quality of the terminal device.

[0072] In a thirteenth possible implementation manner of the first aspect, the terminal device performs multi-frame noise reduction processing on the first noise reduction image set, which may be:

[0073] First, the terminal device can determine the pixel gain between each pixel in the first image and the corresponding pixel in the first result image. The pixel gain is the ratio of the brightness of the first pixel in the first image to the brightness of the second pixel in the first result image. The pixel coordinates of the first pixel and the second pixel are the same.

[0074] Then, the terminal device can perform a second alignment process on the first result image according to the pixel gain corresponding to each pixel point to obtain a second aligned image. The second alignment process is used to adjust the pixel brightness of the first result image to be the same as the pixel brightness of the first image.

[0075] Finally, the terminal device may perform multi-frame noise reduction processing on the second noise reduction image set, where the second noise reduction image set includes the first image and second aligned images corresponding to each first result image.

[0076] In an embodiment of the present application, the terminal device may perform a second alignment process on each first result image (that is, other frames after the registration process) before performing multi-frame noise reduction processing on each first result image. After the first result image is subjected to the second alignment process, the overall brightness is closer to the brightness of the first image. Since the glare effect of the first image is relatively small, the second alignment process is performed on the first result image with the first image as a reference, which can further correct the glare in the first result image, that is, the degree of glare effect can be further reduced. In addition, the second alignment process is to perform glare correction on the first result image from the perspective of pixels, with higher adjustment accuracy. The subsequent multi-frame noise reduction process is performed on each first result image after the second alignment process, which has a better noise reduction effect and helps to further improve the imaging quality of the terminal device.

[0077] It can be understood that the operation of performing multi-frame noise reduction processing on the first noise reduction image set by the terminal device is basically the same as the operation of performing multi-frame noise reduction processing on the image subset, and will not be described in detail here.

[0078] In some optional embodiments, the terminal device determining the pixel gain between each pixel in the first image and the corresponding pixel in the first result image may include:

[0079] First, the terminal device determines the gain matrix between the first down-sampling image and the second down-sampling image. The first down-sampling image is an image obtained by down-sampling the first image. The second down-sampling image is an image obtained by down-sampling the first result image. Each element in the gain matrix is ​​the brightness ratio of two pixels at corresponding positions in the first down-sampling image and the second down-sampling image. It can be understood that the embodiment of the present application does not specifically limit the down-sampling ratio. For example, the above-mentioned down-sampling process can be a 4-fold down-sampling. The down-sampling ratios of the first image and the first result image are the same.

[0080] The terminal device can then upsample the gain matrix to obtain the pixel gain of each pixel. Here, the upsampling ratio of the gain matrix is ​​the same as the downsampling ratio of the first image and the first result image. For example, if the downsampling ratio of the first image and the first result image is 4 times, then the upsampling ratio of the gain matrix is ​​also 4 times.

[0081] In the embodiment of the present application, the gain matrix is ​​determined by downsampling, and then upsampled back to obtain the pixel gain of each pixel point, which can effectively reduce the amount of calculation.

[0082] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0083] An input unit, configured to receive a photographing instruction, wherein the photographing instruction is configured to trigger taking a photograph of the current photographing scene;

[0084] A processing unit, configured to obtain N frames of images for a current photographic scene, where N is an integer greater than or equal to 2;

[0085] The processing unit is further configured to determine N glare parameters corresponding to the N frames of image based on pixel features of each frame of image in the N frames of image, wherein each glare parameter is used to characterize the degree of glare impact of the image;

[0086] The processing unit is further configured to output a first image based on the N glare parameters, where the first image is an image in which the glare parameters in the N frames meet a first preset condition, wherein the glare parameters meeting the first preset condition include: the degree of glare impact represented by the glare parameters is the smallest among the N glare parameters.

[0087] Optionally, the display unit is used to display the first image.

[0088] As an embodiment of the present application, the image processing device can implement any method of the first aspect described above.

[0089] In a third aspect, an embodiment of the present application provides a terminal device, which includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, it implements any method as described in the first aspect above.

[0090] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the above-mentioned first aspects.

[0091] In a fifth aspect, embodiments of the present application provide a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement any of the methods described in the first aspect. The chip system may be a single chip or a chip module consisting of multiple chips.

[0092] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when running on a terminal device, enables the terminal device to execute any one of the methods of the first aspect above.

[0093] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1A A schematic diagram of image comparison before and after imaging by a terminal device provided in an embodiment of the present application;

[0095] Figure 1B A flowchart of an image processing method provided in an embodiment of the present application;

[0096] Figure 2 A flowchart for determining glare parameters of a single original image provided in an embodiment of the present application;

[0097] Figure 3 A schematic flow chart of another image processing method provided in an embodiment of the present application;

[0098] Figure 4A A schematic diagram of a flow chart of an imaging output solution for a photographic scene provided in an embodiment of the present application;

[0099] Figure 4BA schematic diagram of a first alignment process operation provided in an embodiment of the present application;

[0100] Figure 4C A schematic diagram of a second alignment process operation provided in an embodiment of the present application;

[0101] Figure 5 A schematic diagram of a flow chart of an imaging output solution for another photographic scenario provided in an embodiment of the present application;

[0102] Figure 6 A schematic diagram of the application architecture of a terminal device provided in an embodiment of the present application;

[0103] Figure 7 A schematic diagram of the hardware structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0105] The following describes some concepts that may be involved in the embodiments of this application:

[0106] (1) Multiple: Unless otherwise specified, in the embodiments of the present application, multiple refers to two or more.

[0107] (2) Long Frames, Short Frames, and Standard Frames: In the embodiments of the present application, in the fields of video processing and computer vision, a frame refers to a still image that constitutes the basic unit of a video sequence.

[0108] Frame rate describes the speed at which the number of frames in a video changes, usually expressed in frames per second (FPS). A higher frame rate can produce a smoother video effect, while a lower frame rate may cause the video to appear less smooth.

[0109] Long frames usually refer to frames with a longer duration in a video sequence. Long frames correspond to lower frame rates and are usually suitable for static scenes or slow motion.

[0110] Short frames refer to frames with shorter duration in a video sequence. Short frames correspond to higher frame rates and are suitable for some fast-moving situations.

[0111] Standard frames are frames that conform to a regular frame rate. Standard frames typically have a relatively short but uniform duration to produce a smooth video effect.

[0112] In an embodiment of the present application, when a terminal device obtains multiple frames of original images of the same photographic scene, the multiple frames of original images often include long frames, short frames and standard frames at the same time.

[0113] (3) Multi-frame noise reduction: In the embodiments of the present application, multi-frame noise reduction is a digital image processing technique used to reduce noise and artifacts in digital images. This technique uses multiple images of the same scene to extract non-noise information from the different images and merge this information to eliminate noise and artifacts, thereby obtaining a clearer image.

[0114] Specifically, multi-frame noise reduction technology combines multiple images of the same scene, calculates the distribution of values ​​for each pixel across the different images, and selects the most reliable value as the final output. Because noise often appears randomly, multi-frame noise reduction can effectively improve noise reduction without excessive loss of image detail.

[0115] In practice, common multi-frame noise reduction algorithms include averaging method, median method, weighted average method, principal component analysis (PCA) noise reduction method, inter-frame difference method, and multi-frame noise reduction algorithm based on deep learning.

[0116] (4) High Dynamic Range Imaging (HDR): In the embodiments of this application, the HDR algorithm is a digital image processing technology used to combine multiple photos with different exposures into a single image with a more balanced distribution of light and shade and richer details. This technology can, to a certain extent, simulate the human eye's ability to perceive high dynamic range environments.

[0117] The implementation principle of the HDR algorithm is to synthesize multiple pictures with different exposure times to obtain light and dark distribution data with a wider dynamic range, and use this data to generate an image with more balanced global illumination.

[0118] (5) Image registration: In the embodiments of the present application, image registration is an important technology in digital image processing, which is used to accurately align multiple images or different parts of images in space or geometry for subsequent processing, such as image stitching, fusion or comparison.

[0119] In image registration, a reference image or reference coordinate system must first be determined. Then, by matching and transforming other images or image features, they are mapped to the reference coordinate system so that they are spatially aligned. Image registration enables quantitative comparison and analysis between different images, improving the usability and accuracy of image data.

[0120] In practice, common image registration algorithms include homography registration and optical flow registration.

[0121] Among them, the homography registration method is to register two images through the homography matrix (Homography Matrix). Among them, the homography matrix is ​​a linear transformation matrix used to describe the perspective relationship or projection relationship between two images. In practice, the terminal device can determine the homography matrix between two images by first determining the feature points of the reference image and the reference image respectively; then matching the feature points in the two images; and then calculating the homography matrix using the feature points with matching relationship between the two images.

[0122] Optical flow registration uses the optical flow field between two images for registration. In practice, a terminal device can use an optical flow algorithm, such as the Lucas-Kanade (LK) optical flow algorithm or the Dense Inverse Search (DIS) algorithm, to determine the optical flow field between the reference image and the referenced image.

[0123] (6) Hue-Saturation-Value (HSV) Model. In the embodiments of the present application, the HSV model is a commonly used color space model that divides color attributes into three elements: hue, saturation, and value. The HSV model is commonly used to describe color.

[0124] (7) Image mean and image standard deviation: In the present embodiment, the image mean refers to the average of all pixel values ​​in the image. In digital image processing, each pixel in the image has a pixel value. The image mean can be calculated by calculating the sum of all pixel values ​​and dividing it by the total number of pixels.

[0125] The image standard deviation is the standard deviation of all pixel values ​​in the image. It's a statistic that measures the distribution of pixel values. A larger standard deviation indicates a more dispersed image, while a smaller standard deviation indicates a closer average.

[0126] Dazzle is a visual condition caused by an unsuitable brightness distribution or extreme brightness contrast in space or time, resulting in visual discomfort and reduced visibility. Typically, glare is caused by high-intensity light sources interfering with multiple reflective surfaces. The essence of glare is the superposition of light caused by refraction within a lens.

[0127] The optical performance of the lenses of mobile phones and other terminal devices cannot reach the optimal level due to limitations in size and material. When shooting scenes with strong light sources, serious glare problems often occur, affecting the final image quality.

[0128] To address the glare issue, one option is to set up two cameras on the terminal device: one for capturing visible light images and the other for capturing non-visible light images. During the capture process, both cameras simultaneously capture visible light and non-visible light images of the same scene. If the visible light image exhibits glare, the glare areas of the visible light image are filled with a non-visible light image to remove the glare.

[0129] However, the above solution may have the following defects: multiple lenses need to be set up, which is costly and requires space to deploy the lenses.

[0130] Taking into account the above-mentioned technical problems existing in the related art, in an embodiment of the present application, when a terminal device such as a mobile phone takes a photo, a camera is used to capture multiple continuous frames of images of the current photo scene, and the frames with the least degree of glare influence in the multiple frames are selected as the first image, thereby obtaining and outputting the resulting photo image. Since the terminal device may experience more or less jitter during the photo-taking process, and since in many cases, a slight movement of the terminal device may avoid introducing glare, that is, the degree of glare in the multiple frames of images acquired by the terminal device often varies. Some images have a greater degree of glare, some have a lesser degree of glare, and some may even have no glare. Therefore, the terminal device selects the first image based on the degree of glare influence of each image, and can select the first image with the least degree of glare influence. The resulting photo image determined based on the first image often has better image quality, that is, the present application can effectively improve the imaging quality of the terminal device.

[0131] It should be pointed out that compared with related technologies, only one lens is needed in the terminal device, and there is no need to install a lens for collecting non-visible light images. This can ensure the imaging quality of the terminal device without increasing the cost of the equipment.

[0132] In order to better illustrate the solution of the present application, the imaging effect of the embodiment of the present application is explained below by taking a mobile phone as an example of a terminal device. Figure 1AA schematic diagram of image comparison before and after imaging of a terminal device provided in an embodiment of the present application.

[0133] Figure 1A In the figure, the left picture is a frame of image before imaging, and the right picture is the photographic result image after imaging. Figure 1A In the current photo scene, there are multiple light tubes, namely, tube 20a, tube 20b and tube 20c. When the mobile phone takes a photo, the light from multiple tubes enters the lens, resulting in multiple glare areas in one frame of the image before imaging, such as glare area 10a, glare area 10b and glare area 10c, resulting in poor image quality. Figure 1A In the picture on the right, the light tubes in the photo are clearer after imaging, and the glare effect is significantly smaller, which means that the imaging quality of the terminal device is better.

[0134] The following describes the usage scenarios of the embodiments of this application:

[0135] The embodiments of the present application can be applied to scenarios where a terminal device is photographing. The photographing scenario can be a scenario where a single image is captured, a scenario where multiple images are captured continuously, or a scenario where a video is captured. It is understood that when the photographing scenario is a video capture scenario, each frame of the video output by the terminal device is an image with the least degree of glare, selected from N frames of images.

[0136] The image processing method provided in the embodiments of the present application can be applied to terminal devices, and the terminal devices can be mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other devices, which are not limited in the embodiments of the present application.

[0137] The following combination Figure 1B The image processing method provided in the embodiments of the present application is described. Figure 1B A flowchart of an image processing method provided in an embodiment of the present application. Figure 1B , the image processing method provided in the embodiment of the present application may include:

[0138] S101: The terminal device receives a photo-taking instruction and obtains N frames of original images of the current photo-taking scene.

[0139] The photo command is used to trigger taking a photo of the current photo scene. The embodiment of the present application does not limit the specific form of the photo command. The user can trigger the photo command in various forms. For example, the photo command can be a voice command, or it can be a photo command triggered by the user through an operation. The user's operation can include but is not limited to: pressing a physical key, touching or clicking a photo control, triggering a photo operation through a third-party device (such as headphones), etc.

[0140] In practice, to ensure the quality of the image presented to the user, the terminal device can capture multiple frames of original images during the capture process. The terminal device can then present a higher-quality image to the user based on these multiple frames. For example, the original image with the least glare effect can be presented to the user as the resulting image.

[0141] S102: The terminal device determines the glare parameter of each frame of the original image according to the pixel features of each frame of the original image.

[0142] The glare parameter indicates the degree of glare in the original image. The glare parameter includes at least one of the following: contrast, a first glare level, a second glare level, and a probability coefficient. The first glare level is the overall glare level of the image, the second glare level is the glare level of the target area in the image, and the probability coefficient is the probability of glare in the original image.

[0143] The target area is the area in the original image where the first object is located. The first object may be an object of interest. For example, the first object may be a human face, a human portrait, or an animal.

[0144] The first object can be a default object, such as a face. The first object can also be an object set by the user. For example, the terminal device can respond to the user clicking an object on the photo preview interface, such as a puppy, and set the object as the first object.

[0145] In practice, for each frame of the original image, the terminal device may calculate the glare parameter of the original image. The glare parameter may include at least one of the following: contrast, first glare degree, second glare degree, and probability coefficient.

[0146] The following describes the various elements that may be included in the glare parameters.

[0147] (1) Contrast

[0148] The smaller the contrast of the image, the less glare the image has, which means the better the image quality.

[0149] Optionally, the contrast of the image may be a ratio of a first difference value to a first sum value, where the first difference value is the difference between a maximum pixel value and a minimum pixel value in the image, and the first sum value is the sum of a maximum pixel value and a minimum pixel value in the image. Specifically, the contrast of the image may be: C = (max - min) ÷ (max + min), where C is the contrast of the image, max is the maximum pixel value in the image, and min is the minimum pixel value in the image.

[0150] Alternatively, the image contrast may be the ratio of the image mean to the image standard deviation. The image mean is the average of the pixels in the image, and the image standard deviation is the standard deviation of the pixels in the image. Specifically, the image contrast may be: C = mean ÷ std, where C is the image contrast, mean is the image mean, and std is the image standard deviation.

[0151] It's important to note that since the ratio of a set of data's mean to its standard deviation accurately reflects the degree of dispersion within that data set, the ratio of an image's mean to its standard deviation accurately reflects the degree of dispersion within each pixel within that image. Furthermore, since images affected by glare generally exhibit greater dispersion the more severe the glare, the contrast ratio, determined by the ratio of the image's mean to its standard deviation, accurately describes the degree of glare.

[0152] (2) First glare level

[0153] The first glare degree may also be referred to as the global glare degree, and may describe the degree of glare influence on the entire image range.

[0154] Optionally, the first glare index can be calculated using the following formula: H = Avg{h(1,1)+...+h(i,j)+...}, where Avg is a mean calculation function and h(i,j) is the difference between the brightness and saturation of the pixel in the i-th row and j-th column of the image. For ease of description, the difference between the brightness and saturation of the pixel can be referred to as the glare index.

[0155] Specifically, h(i,j)=V(i,j)-S(i,j), where V(i,j) is the brightness of the pixel in the i-th row and j-th column in the HSV space, and S(i,j) is the saturation of the pixel in the HSV space.

[0156] Combining the above formula, we can see that the first glare degree can be the average of the glare indices of all pixels in the image. It is understood that in some application scenarios, the first glare degree can also be expressed in other forms, such as the root mean square (RMS) value of the glare indices of all pixels in the original image. It should be noted that calculating the glare degree corresponding to a certain range by taking the average value reduces computational complexity and improves accuracy.

[0157] In HSV space, a pixel's brightness is brighter when its lightness is greater, and darker when its lightness is smaller. A pixel's saturation is darker, and its saturation is closer to white. Glare usually appears as bright white. Therefore, the difference between a pixel's lightness and saturation can better describe the glare situation at a particular pixel. In other words, the glare index can well describe the degree of glare impact at a particular pixel. The first glare degree, determined by the average glare index of each pixel in the original image, can better describe the degree of glare impact of the entire original image.

[0158] (3) Second glare level

[0159] The second glare intensity may also be referred to as a local glare intensity. The local glare intensity is the glare intensity of a region of interest. The region of interest may be a portrait region, a face region, a skin region, etc. in an image.

[0160] The local glare degree may be the average of the glare indices of the pixels in the region of interest in the image. The calculation process of the second glare degree is similar to that of the first glare degree, and will not be described in detail here.

[0161] In practice, the terminal device may first perform target detection on the original image to detect whether the original image includes the first object and the position of the first object in the original image. The terminal device may then calculate the local glare degree using the glare index of each pixel in the target area.

[0162] The target detection is used to detect whether a first object exists in an image. If a first object exists, the area where the first object is located can be obtained, which is the target area. It is understandable that the terminal device can use the methods disclosed in the relevant technology to perform target detection on the original image to determine whether the original image includes the first object. The embodiments of this application do not specifically limit this. For example, the terminal device can use a neural network model to detect whether a face exists in an image.

[0163] It should be noted that the second glare degree can highlight the glare condition in the area of ​​interest to the user. Filtering the first image based on the second glare degree helps to filter out an image that better meets the user's needs, thereby further improving the imaging quality of the terminal device.

[0164] Considering that when the terminal device detects that a target area (such as a face area) exists in a certain image, it may detect one target area or multiple target areas.

[0165] Optionally, for a certain original image, when the terminal device detects a target area, the glare degree corresponding to the target area may be used as the local glare degree of the original image.

[0166] Optionally, for a certain original image, when the terminal device detects multiple target areas, it may use the maximum value of the glare intensities corresponding to the multiple target areas as the local glare intensities of the original image.

[0167] For example, when the terminal device detects multiple target areas, the expression formula of the local glare degree of the original image can be: H l =max(H l1 ,H l2 ,...H lm ), where H lm is the glare intensity of the mth target area, and m is the number of detected target areas. l It is the local glare degree.

[0168] Optionally, for a certain original image, when the terminal device detects multiple target areas, it may determine the average of the glare indices of the pixels in the multiple target areas as the local glare degree of the original image.

[0169] (4) Probability coefficient

[0170] The glare probability described above is used to describe the probability of glare existing in an image. The smaller the probability coefficient, the lower the probability of glare existing. Conversely, the larger the probability coefficient, the greater the probability of glare existing.

[0171] In practice, the terminal device may input each original image into a pre-trained glare detection model to obtain a probability coefficient (or glare probability) corresponding to each original image.

[0172] The glare detection model can be a pre-trained neural network model. In practice, the glare detection model can be a Visual Geometry Group (VGG) model, where the VGG model is a deep convolutional neural network model. In practice, the loss function of the glare detection model can be a cross-entropy loss function. The training samples for the glare detection model can include original images of various photo scenes and the glare probabilities corresponding to the original images. It is understandable that the glare detection model can also be a neural network model other than the VGG model. The terminal device can train the glare detection model using the model training method disclosed in the relevant art. The embodiments of the present application do not specifically limit the glare detection model and the training process of the glare detection model.

[0173] S103: The terminal device selects an original image that meets a first preset condition from N frames of original images as a first image.

[0174] The first preset condition may be a pre-set screening condition.

[0175] As an example, the first preset condition may include one or more of the following:

[0176] Condition 1: When the glare parameter includes global glare, the original image with the smallest global glare is selected as the first image.

[0177] Condition 2: When the glare parameter includes local glare, the original image with the minimum local glare is selected as the first image.

[0178] Condition 3: When the glare parameter includes contrast, the original image with the smallest contrast is selected as the first image.

[0179] Condition 4: when the glare parameter includes the glare probability, the original image with the minimum glare probability is selected as the first image.

[0180] In some optional implementations of this embodiment, the glare parameter may include one of the four elements: contrast, first glare intensity, second glare intensity, and probability coefficient, or may include multiple of these four elements simultaneously. When the glare parameter includes multiple of these four elements, the terminal device may directly use all or part of these elements to filter the first image. For example, when the glare parameter includes all four elements simultaneously, the original image with the lowest second glare intensity may be directly selected as the first image. The terminal device may also add up the weight values ​​of each element to comprehensively filter the first image. For any element, the weight value of that element may be the product of the value of that element and the corresponding weight coefficient.

[0181] In some optional implementations of this embodiment, before executing step S102, the terminal device may further perform the following operations:

[0182] First, the terminal device can perform glare detection on each frame of the original image to obtain a first detection result corresponding to each frame of the original image. The first detection result is used to indicate whether there is glare in the original image.

[0183] Then, when there is glare in each frame of the original image, the terminal device can perform target detection on each original image to obtain a second detection result, which is used to indicate whether the original image includes the first object and the position of the first object in the original image.

[0184] In an embodiment of the present application, the terminal device may further detect whether the original image includes the first object only when there is glare in each frame of the original image, which can save computing power and improve the overall performance of the terminal device.

[0185] It is understandable that in some application scenarios, the terminal device may not detect whether there is glare in each original image, but directly perform target detection on each original image. In other application scenarios, the terminal device may also perform glare detection and target detection on a certain original image at the same time.

[0186] It is understood that the terminal device can use methods disclosed in related technologies to perform glare detection or target detection on images. The embodiments of this application do not specifically limit how to perform glare detection or target detection on images. As an example, the terminal device can use a binary VGG model to detect whether glare exists in the original image.

[0187] S104: The terminal device determines a photographing result image based on the first image.

[0188] Optionally, the terminal device may directly use the first image obtained through screening as the photographing result image.

[0189] Optionally, the terminal device may also process the first image, for example, perform denoising processing, and use the processed image as the photographing result image.

[0190] Optionally, the terminal device may further process other frames based on the first image, and then process all the original images to obtain a resulting image. For example, HDR processing may be performed on all the original images, and the resulting image may be used as the resulting image. The other frames are original images other than the first image.

[0191] S105, the terminal device outputs the photographing result image.

[0192] Here, the terminal device can output the photographing result, for example, it can directly display the photographing result image. For another example, it can store the photographing result image in a gallery.

[0193] In some optional implementations of this embodiment, when each original image includes the first object, the above step S102 can be implemented as the following steps S1021 to S1023. Figure 2 This is a flowchart for determining the glare parameters of a single original image provided in an embodiment of the present application.

[0194] S1021: The terminal device determines a first glare degree of the original image according to a glare index of each pixel in the original image.

[0195] The glare index of a pixel is the difference between the brightness and saturation of the pixel.

[0196] S1022: When the terminal device detects that the original image includes the first object, the terminal device determines a second glare degree of the original image according to the glare index of each pixel in the target area.

[0197] The target area is the area where the first object is located in the original image.

[0198] Here, the operation of determining the first glare degree and the second glare degree of the original image can be found in the description of the aforementioned step S102 , and will not be described in detail here.

[0199] S1023: The terminal device determines a first weighted value of the original image according to the first glare degree, the second glare degree, and a first weight coefficient for the second glare degree.

[0200] At this time, the glare parameter may include a first glare degree, a second glare degree, and a first weighted value.

[0201] The first weighted value is the weighted sum of the global glare and the local glare.

[0202] Specifically, the calculation formula of the first weighted value can be: H w =(1-w)H g +wH l Among them, H w is the first weighted value, w is the first weight coefficient, H g is the global glare, H l It is the local glare degree.

[0203] In an embodiment of the present application, since the first object is an object of interest to the user, when the terminal device includes the first object in the original image, it comprehensively considers the first glare degree and the second glare degree, and filters the first image based on the first weighted value obtained by comprehensively determining the first glare degree and the second glare degree. This helps to filter the first image with a small degree of glare influence and that meets the user's needs, and helps to further improve the imaging quality of the terminal device.

[0204] In some application scenarios, the first weight coefficient can be a preset value, such as 0.5. The terminal device can also adjust the value of the first weight coefficient based on user needs. As an example, the terminal device can respond to a user click operation on a photo preview image displayed on the terminal device, and determine the first weight coefficient based on the distance between the area clicked by the user and the center of the target area. The first weight coefficient can be inversely correlated with the first distance, where the first distance is the distance between the location point clicked by the user and the center of the target area.

[0205] Taking into account that there may be multiple target areas in the image at the same time, when there are multiple target areas, the above-mentioned first distance can be the distance between the position point clicked by the user and the center of the nearest target area, that is, the first distance can be the minimum value of the distance between the position point clicked by the user and the center of each target area.

[0206] It should be noted that the terminal device adjusts the first weight coefficient according to user needs, which helps to screen out a first image that better meets the user needs, thereby further improving the user experience.

[0207] For any original image, if the original image does not have a target area, that is, the first object is not detected, then the original image does not have a local glare degree, and therefore does not have a first weighted value.

[0208] Optionally, when the original images have a target area, the first image may be selected from the N frames of original images based on the first weighted value. When the original images do not have a target area, the terminal device may select the first image from the N frames of original images based on the first glare level.

[0209] Since the target may move quickly during the photographing process, that is, the target area may exist in a part of the original image, while the target area may not exist in another part of the original image.

[0210] Optionally, the terminal device may select the first image from the portion of the original image containing the target area. For example, the original image with the smallest first weighted value may be selected from the portion of the original image containing the target area as the first image. This can ensure the imaging quality of the terminal device.

[0211] In some optional implementations of this embodiment, step S103 may be replaced by: when each original image includes the first object, the terminal device selects the original image with the smallest first weighted value from the N frames of original images as the first image. In this case, the first preset condition may be: selecting the original image with the smallest first weighted value as the first image.

[0212] In some optional implementations of this embodiment, step S103 may be replaced by: when the first object is not included in each original image, the terminal device selects the original image with the smallest first glare degree from the N frames of original images as the first image. In this case, the first preset condition may be: selecting the original image with the smallest first weighted value as the first image.

[0213] In an embodiment of the present application, when there is a target area in each original image, the first image is filtered with a first weighted value; when there is no target area in each original image, the first image is filtered with a first glare degree. This helps to filter the first image with a small degree of glare and meets user needs, and helps to further improve the imaging quality of the terminal device.

[0214] Considering that the photographing scenes may be diverse, for example, some photographing scenes have glare, some photographing scenes do not have glare, some photographing scenes have a first object, and some photographing scenes do not have a first object.

[0215] The following takes a human face as an example as the first object to further illustrate the image processing method provided in the embodiment of the present application. Figure 3 This is a flow chart of another image processing method provided in an embodiment of the present application. Figure 3 The image processing method provided in the embodiment of the present application may include the following steps S301 to S310.

[0216] S301: The terminal device performs glare detection on each frame of the original image.

[0217] The glare detection is used to detect whether the original image contains glare. As an example, the terminal device can use a glare detection model to detect whether the original image contains glare. The glare detection model can be a binary classification neural network model, such as a binary classification VGG model.

[0218] It should be noted that the terminal device can adopt the model and model training method disclosed in the relevant technology to obtain the above-mentioned glare detection model. The embodiment of the present application does not specifically limit the model and model training method used for glare detection.

[0219] In practice, the terminal device may perform glare detection on each frame of the original image, thereby determining whether each original image has glare.

[0220] S302: The terminal device determines whether there is glare in each frame of the original image.

[0221] Here, there is no glare in the original image, indicating that the image quality is relatively good. When there is an original image without glare, S303 can be continued to be executed. If there is no original image without glare, S304 can be continued to be executed.

[0222] S303: The terminal device selects a second image.

[0223] The second image is an image without glare.

[0224] In practice, considering that N frames of original images often contain long frames, short frames, and standard frames, if there is an original image without glare, the terminal device can select a standard frame from the original image without glare as the second image. The long frames, short frames, and standard frames correspond to different acquisition frame rates: long frames correspond to a lower frame rate, short frames correspond to a higher frame rate, and standard frames correspond to a standard frame rate.

[0225] S304: The terminal device determines the global glare of the original image.

[0226] The global glare intensity is the overall glare intensity of the original image and can also be referred to as the first glare intensity.

[0227] Here, the operation of determining the global glare degree of the original image can be found in the description of the aforementioned step S102 , and will not be described in detail here.

[0228] S305: The terminal device performs face detection on the original image.

[0229] Here, the terminal device can use the face detection method disclosed in the relevant technology to perform face detection on each frame of the original image, thereby detecting the face area (that is, the target area) in each frame of the original image. As an example, the terminal device can detect the face area in the image through a neural network model. It is understandable that the embodiment of the present application does not specifically limit the face detection method.

[0230] S306: The terminal device determines whether a face is detected.

[0231] Here, for each frame of the original image, the terminal device can combine the face detection results to determine whether a face is detected. Specifically, if one or more face regions are detected from the frame of the original image, it is considered that a face is detected. If no face region is detected from the frame of the image, it is considered that no face is detected.

[0232] S307: When a face is detected, the terminal device may determine the local glare degree of the original image.

[0233] The local glare intensity may also be referred to as the second glare intensity.

[0234] Here, the operation of determining the local glare degree of the original image can be found in the description of the aforementioned step S102 , and will not be described in detail here.

[0235] In practice, for any original image, when the terminal device detects a face area from the original image, the glare degree of the face area can be used as the local glare degree of the original image; when the terminal device detects multiple face areas from the original image, the glare degree of each face area can be calculated separately, and then the maximum value of the glare degrees corresponding to the multiple face areas can be used as the local glare degree of the original image.

[0236] It should be pointed out that using the maximum glare degree corresponding to each facial area as the local glare degree can highlight the glare condition of the face, which helps to screen out a first image that better meets user needs, thereby further improving the imaging quality of the terminal device.

[0237] In the embodiment of the present application, the terminal device will further determine the second glare level of the original image only when there is glare and a target area in the original image. This can avoid determining the second glare level of the original image when there is no glare, helping to save unnecessary computing resources.

[0238] S308: The terminal device selects a first image based on the local glare and global glare corresponding to each original image.

[0239] Optionally, the terminal device may select the original image corresponding to the minimum local glare as the first image, so as to highlight the glare condition of the face.

[0240] Optionally, the terminal device may also select the original image corresponding to the minimum global glare as the first image, so as to highlight the glare condition of the entire image.

[0241] Optionally, the terminal device may select the original image corresponding to the smallest first weighted value as the first image, wherein the first weighted value is the weighted sum of the local glare and the global glare.

[0242] Here, the operation of determining the first weighted value of the original image can be found in the description of the aforementioned step S1023, which will not be described in detail here.

[0243] S309: The terminal device determines the contrast of the original image.

[0244] Here, for each frame of the original image, the terminal device may determine the contrast of the original image.

[0245] The operation of determining the contrast of the original image can be found in the description of the aforementioned step S102 and will not be described in detail here.

[0246] S310: The terminal device selects a first image based on the contrast and global glare of each original image.

[0247] Optionally, the terminal device may select the original image with the smallest corresponding contrast as the first image.

[0248] Optionally, the terminal device may also select the original image corresponding to the minimum global glare as the first image.

[0249] Optionally, the terminal device may select the original image corresponding to the smallest second weighted value as the first image, wherein the second weighted value is a weighted sum of contrast and global glare.

[0250] Specifically, the calculation formula of the second weighted value can be: H λ =(1-λ)H g +λC. Among them, H λ is the second weighted value, λ is the second weight coefficient, or called the contrast weight coefficient, H g is the global glare, and C is the contrast.

[0251] When there is glare in the original image but no target area, the terminal device can further determine a second weighted value for the original image by combining the first glare degree and contrast. This allows the first image to be screened for images less affected by glare when there is no target area, helping to further improve the imaging quality of the terminal device.

[0252] It should be pointed out that combining multiple dimensions to filter the first image helps to ensure the image quality of the filtered first image, thereby further improving the imaging quality of the terminal device.

[0253] In some optional implementations of this embodiment, step S103 may be replaced by: when the target area exists in each original image, the terminal device selects the original image with the smallest first weighted value from the N frames of original images as the first image. In this case, the first preset condition may be: selecting the original image with the smallest first weighted value as the first image.

[0254] In some optional implementations of this embodiment, step S103 may be replaced by: when the target area does not exist in any of the original images, the terminal device selects the original image with the smallest second glare degree from the N frames of original images as the first image. In this case, the first preset condition may be: selecting the original image with the smallest second weighted value as the first image.

[0255] Taking into account different shooting scenarios, the ambient light brightness (or first brightness) and shooting angle, etc., have a great impact on the imaging quality of the terminal device. For example, when the first brightness is high and the shooting is done at a forward-lighting angle, the imaging of the terminal device is generally clear, that is, the imaging quality is good; when the first brightness is high and the shooting is done at a backlighting angle, a large exposure area (or glare area) is likely to appear in the imaging of the terminal device.

[0256] In order to ensure the imaging quality of the terminal device in various shooting scenes, the terminal device can process the original image obtained accordingly based on the ambient light brightness and shooting angle in the shooting scene to ensure the quality of the image output to the user.

[0257] Because the terminal device directly determines the shooting angle, that is, the relative angle between the lens and the light source or reflected light source, it is relatively difficult. Considering that the size of the glare area in images captured at different shooting angles is directly related to the shooting angle, the terminal device can process the original image accordingly based on the size of the glare area in the original image and the ambient light intensity to ensure the quality of the image output to the user.

[0258] Among them, the terminal device can detect the first brightness of the photo scene through the ambient light sensor. The size of the glare area can be measured by the proportion of the glare area to the entire image. The proportion of the glare area to the entire image can be called the glare area ratio, or the first coefficient, or the overexposure ratio. When the original image is an 8-bit image, the pixel value of each pixel (or pixel point) in the glare area is 255.

[0259] In practice, for any frame of the original image, the terminal device can count the number of pixels with a pixel value of 255 in the original image, record it as the first number, and then determine the proportion of the glare area by the ratio of the first number to the total number of pixels in the original image.

[0260] In practice, the terminal device can determine the specific shooting scene by analyzing the first brightness captured by the ambient light sensor and the overexposure ratio (or glare area ratio) of the filtered first image. The terminal device can then process the acquired original image accordingly based on different shooting scenes to ensure the quality of the image output to the user.

[0261] In some optional implementations of this embodiment, step S104 may be replaced by: the terminal device determines a resulting photographic image based on the first image, the other frames, and the photographic scene. The other frames are original images other than the first image. It will be appreciated that, if N original image frames are acquired, there are a total of N-1 other frames.

[0262] The following describes the imaging output solutions corresponding to various photography scenarios.

[0263] (1) Photographing scene 1: the first brightness is greater than a preset brightness value, and the proportion of the glare area is less than or equal to a preset coefficient value.

[0264] The preset brightness value is a pre-set brightness threshold. In a photo shooting scene corresponding to the preset brightness value, the picture can be clearly captured. As an example, the preset brightness value can be 150 lux.

[0265] The preset coefficient value is usually a preset ratio and is usually relatively small, for example, 5%.

[0266] In photo scene 1, since the first brightness is high and the glare area accounts for a small proportion, that is, the overall image clarity is good, the terminal device can directly output the first image selected from N frames of original images as an imaging image (or called a photo result image).

[0267] (2) Photographing scene 2: the first brightness is less than or equal to a preset brightness value, and the proportion of the glare area is less than or equal to a preset coefficient value.

[0268] In the photo shooting scene 2, the first brightness is low, and noise is more likely to exist in the original image obtained. In order to ensure the quality of the image output by the terminal device, the terminal device can perform denoising on each original image before outputting it.

[0269] In some embodiments, the terminal device may use a multi-frame noise reduction algorithm to perform denoising on N frames of original images, thereby obtaining a frame of clear denoised result image, and then the terminal device may output the denoised result image as an imaging image.

[0270] In some embodiments, the terminal device may first perform image registration processing on each of the other frames based on the first image. Then, a multi-frame noise reduction algorithm is used to denoise the N registered original images, thereby obtaining a clear denoised image. The terminal device may then output the denoised image as the image.

[0271] The following combination Figure 4A The imaging output scheme corresponding to the photo scene 2 is described. Figure 4AA flowchart of an imaging output solution for photographing scene 2 provided in an embodiment of the present application. It is understandable that the imaging output solution for photographing scene 2 may include all steps from step S401 to step S405, or may include only some of the steps. It is understandable that the steps may be combined arbitrarily without conflict. As an optional embodiment of the present application, the imaging output solution for photographing scene 2 may include only step S402, step S404, and step S405. As another optional embodiment of the present application, the imaging output solution for photographing scene 2 may also include only step S401, step S402, step S404, and step S405.

[0272] S401: The terminal device performs a first alignment process on other frames with the first image as a reference to obtain a first aligned image.

[0273] The first alignment process is used to adjust the image brightness of other frames to be the same as the image brightness of the first image. The image brightness of the original image is the product of the exposure time and the sensitivity of the original image.

[0274] In practice, parameters such as exposure time and sensitivity of the original image are usually derived from the original image's image metadata. Image metadata is a metadata standard embedded in digital image files that contains information about the shooting equipment, shooting parameters, and shooting environment.

[0275] Combine Figure 4B The first alignment processing operation may include the following steps S4011 to S4013. Figure 4B This is a schematic diagram of the first alignment processing operation provided by an embodiment of the present application. The terminal device can perform the operations of steps S4011 to S4013 on each other frame to obtain a first aligned image corresponding to each other frame.

[0276] S4011: The terminal device determines the frame type of other frames.

[0277] Here, the terminal device may determine the types of other frames in combination with the image metadata of other frames.

[0278] S4012: When the other frames are long frames or short frames, the terminal device determines an image gain between the first image and the other frames.

[0279] The image gain is the ratio of the image brightness of the first image to the image brightness of other frames.

[0280] S4013: The terminal device performs a first alignment process on the other frames according to the image gain to obtain first aligned images corresponding to the other frames.

[0281] It can be understood that, since the first image is a standard frame, when other frames are also standard frames, the terminal device does not need to perform the first alignment process on the other frames.

[0282] For example, the first alignment process may include: the terminal device first calculates the image gain corresponding to the other frames, and then multiplies the image gain by the other frames to obtain the other frames after the first alignment process. The image gain is the ratio of the image brightness of the first image to the image brightness of the other frames.

[0283] Optionally, the image gain of any other frame may be:

[0284]

[0285] Among them, gain is the image gain of other frames, iso ref is the sensitivity of the first image, expo ref is the exposure time of the first image, iso ref ×expo ref is the brightness of the first image. iso cur The sensitivity of other frames, expo cur is the exposure time of other frames, iso cur ×expo cur It should be noted that for any frame of image, the image brightness is described by the product of the sensitivity of the image and the exposure time, which has high accuracy and low computational complexity.

[0286] In some application scenarios, the image gain of any other frame may also be a ratio of the average pixel value of the first image to the average pixel value of the other frames.

[0287] In some optional implementations, the first alignment process may also include: the terminal device performs histogram equalization on the other frames and the first image respectively to obtain the other frames and the first image after the histogram equalization process, wherein the brightness of the other frames and the first image after the histogram equalization process are aligned.

[0288] It should be pointed out that, considering that the multiple frames of original images acquired by the terminal device often include both long-frame type original images and short-frame type original images, and there is usually a large brightness difference between the original images of different frame types in the same shooting scene, the terminal device aligns the brightness of the long-frame type original images and the short-frame type original images with the first image respectively, which can avoid the registration error caused by the obvious brightness difference between the images, thereby ensuring the accuracy of the image registration processing, and helping to further improve the final imaging quality of the terminal device.

[0289] S402: The terminal device uses the first image as a reference to perform image registration processing on other frames to obtain a first result image.

[0290] In the embodiment of the present application, for the convenience of description, the image obtained after the image registration process is performed on other frames may be referred to as the first result image.

[0291] Here, for any other frame, if the first alignment processing operation in S401 is performed on the other frame, the terminal device performs image registration processing on the first aligned image corresponding to the other frame with the first image as a reference. If the first alignment processing operation in S401 is not performed on the other frame, the terminal device directly performs image registration processing on the other frame with the first image as a reference.

[0292] It should be noted that the terminal device can use the image registration processing solution disclosed in the relevant technology to perform image registration on other frames, which will not be described in detail here.

[0293] It should be noted that due to slight differences in the acquisition time of each frame of the original image—for example, some original images are acquired slightly earlier and some slightly later—there are spatial deviations between the multiple frames of the original image. Image registration is performed on each frame of the original image, using the first image with the least glare effect as a reference. This can align multiple images at the same spatial position, avoiding image noise caused by spatial position deviations. It can also reduce noise interference introduced by glare and enhance the effective signal in the image. In other words, performing image registration on each frame of the original image helps further improve the imaging quality of the terminal device.

[0294] It is understandable that in some application scenarios, the terminal device may not perform registration processing on the original images, that is, the terminal device may directly perform multi-frame noise reduction processing on N frames of original images.

[0295] S403: The terminal device performs a second alignment process on each first result image with the first image as a reference to obtain a second aligned image.

[0296] The second alignment process is used to adjust the pixel brightness of the first result image to be the same as the pixel brightness of the first image. The pixel brightness is usually a pixel value.

[0297] Combine Figure 4C The second alignment processing operation may include the following steps S4031 to S4034. Figure 4C This is a schematic diagram of the second alignment process provided by an embodiment of the present application. The terminal device may perform the operations of steps S4031 to S4034 on each first result image.

[0298] S4031: The terminal device downsamples the first image and the first result image respectively.

[0299] Here, the terminal device downsamples the first image and the first result image respectively, which can reduce the dimension of subsequent image processing, thereby reducing the amount of computation. For ease of description, the first image after the downsampling process can be referred to as the first downsampled image, and the first result image after the downsampling process can be referred to as the second downsampled image.

[0300] In practice, the first image and the first result image may be downsampled by a factor of 4.

[0301] S4032: The terminal device determines a first gain map between the first image and the first result image.

[0302] The first gain map is used to describe the brightness difference between each matching pixel in two down-sampled images (specifically, the first down-sampled image and the second down-sampled image).

[0303] The first gain map can also be called a gain matrix. Each element in the gain matrix is ​​the brightness ratio of two pixels at corresponding positions in the first downsampled image and the second downsampled image. The first downsampled image is the image obtained by downsampling the first image. The second downsampled image is the image obtained by downsampling the first result image (i.e., the image obtained after image registration processing is performed on the other frames).

[0304] Optionally, the gain map may be determined as follows:

[0305]

[0306] Where gainmap is the gain map between the first downsampled image and the second downsampled image. The gainmap can describe the brightness difference between the two images at the pixel level. F1 is the first downsampled image, F2 is the second downsampled image, and gain is the image gain of the other frames corresponding to the second downsampled image.

[0307] S4033: The terminal device upsamples the gain map to the target resolution to obtain a second gain map.

[0308] Here, the target resolution is the resolution corresponding to the first image.

[0309] The second gain map is used to describe the brightness difference between the first image and other registered frames (ie, the first result image).

[0310] Each element in the second gain map is a pixel gain between each pixel in the first image and a corresponding pixel in the first result image. The pixel gain is a ratio of the brightness of a first pixel in the first image to the brightness of a second pixel in the first result image. The first pixel and the second pixel have the same pixel coordinates.

[0311] It is understandable that the upsampling ratio of the first gain map is the same as the downsampling ratio in step S4031. For example, if the first image and the first result image are downsampled by a factor of 4 in step S4031, the upsampling ratio of the first gain map is upsampled by a factor of 4 in step S4033.

[0312] It should be noted that determining the gain matrix by downsampling and then upsampling to obtain the pixel gain of each pixel can effectively reduce the amount of calculation. It is understandable that in some application scenarios, the terminal device can also directly use the first image and the first result image to calculate the pixel gain of each pixel.

[0313] S4034: The terminal device performs a second alignment process on the first result image using the second gain map to obtain a second aligned image.

[0314] Here, for the convenience of description, the image obtained after the second alignment process is performed may be referred to as a second aligned image.

[0315] The first result image is the other frame after the registration process.

[0316] In practice, the terminal device may multiply the second gain map by the first result image to obtain a second aligned image corresponding to the first result image.

[0317] It should be pointed out that since the second gain map can describe the brightness difference between each matching pixel in the two images, the brightness adjustment of the first result image based on the second gain map can achieve finer-grained brightness adjustment. After the first result image is subjected to the second alignment processing, the overall brightness is closer to the brightness of the first image. Since the glare effect of the first image is relatively small, the second alignment processing of the first result image with the first image as a reference can further correct the glare in the first result image, that is, the degree of glare effect can be further reduced. In addition, the second alignment processing is to correct the glare of the first result image from the perspective of pixels, and the adjustment accuracy is higher. The subsequent multi-frame noise reduction processing of each first result image after the second alignment processing has a better noise reduction effect, which helps to further improve the imaging quality of the terminal device.

[0318] It is understandable that in some application scenarios, the terminal device may not perform the above-mentioned operation of S403 on other frames, that is, it may directly perform the operation of S404.

[0319] S404: The terminal device performs multi-frame noise reduction processing on the first image and each second aligned image to obtain a third image.

[0320] Here, the terminal device may perform multi-frame noise reduction processing on the first image and the second aligned images corresponding to other frames, thereby obtaining a processing result image after the multi-frame noise reduction processing (ie, the third image).

[0321] It should be pointed out that the terminal device can adopt the multi-frame noise reduction processing solution disclosed in the relevant technology to perform multi-frame noise reduction processing on multiple images of the same scene, thereby obtaining a frame of processing result image, which will not be elaborated here.

[0322] It is understandable that when the terminal device does not perform the second alignment process in step S403 on other frames, the terminal device can directly use the first image and other frames after the alignment process (that is, the first result image) to perform multi-frame noise reduction processing together.

[0323] It is understandable that when the terminal device does not perform the image registration process in step S402 on other frames, the terminal device can use the first image and other frames after the first alignment process to perform multi-frame noise reduction processing together.

[0324] It is understandable that, in the case that the terminal device does not perform the first alignment process in step S401 on other frames, the terminal device can directly perform multi-frame noise reduction processing on the N frames of original images.

[0325] S405: The terminal device outputs the third image after the multi-frame noise reduction processing as a photographing result image.

[0326] Here, the terminal device may use the processed result image obtained by multi-frame noise reduction processing as the photographing result image of the terminal device.

[0327] (3) Photographing scene 3, the proportion of the glare area is greater than the preset coefficient value.

[0328] It should be noted that Photo Scenario 3 actually includes two sub-scenarios. One of the sub-scenarios is: the first brightness is less than or equal to the preset brightness value, and the proportion of the glare area is greater than the preset coefficient value. The other sub-scenario is: the first brightness is greater than the preset brightness value, and the proportion of the glare area is greater than the preset coefficient value.

[0329] In practice, a large overexposure factor in the first image typically indicates that the first image is significantly affected by glare. Considering that in the same shooting scene, original images of different frame types provide different information. Specifically, standard frames provide information about standard brightness areas, short frames provide information about dark areas, and long frames provide information about highlight areas, to reduce the impact of glare on the image, the terminal device can fuse the scene information carried by images of different frame types. This produces a resulting image with complementary information, improving the terminal device's imaging quality.

[0330] The following combination Figure 5 The imaging output scheme corresponding to the photo scene 3 is described. Figure 5 A flowchart of an imaging output solution for photographing scene 3 provided in an embodiment of the present application. It is understandable that the imaging output solution for photographing scene 3 may include all steps from step S501 to step S505, or may include only some of the steps. It is understandable that the steps may be combined arbitrarily without conflict. As an optional embodiment of the present application, the imaging output solution for photographing scene 3 may include only step S502 and step S505. As another optional embodiment of the present application, the imaging output solution for photographing scene 3 may also include only step S502, step S504, and step S505.

[0331] S501: The terminal device performs a first alignment process on other frames with the first image as a reference to obtain a first aligned image.

[0332] S502: The terminal device performs image registration processing on other frames respectively with the first image as a reference to obtain a first result image.

[0333] S503: The terminal device performs a second alignment process on each first result image with the first image as a reference to obtain a second aligned image.

[0334] It should be noted that the operations of step S501 to step S503 are basically the same as the operations of the aforementioned step S401 to step S403, and are not described in detail here.

[0335] S504: The terminal device groups the first image and each second aligned image based on the frame type, and performs multi-frame noise reduction processing on each group of images to obtain a fourth image corresponding to each frame type.

[0336] The second aligned image is an image obtained after the second alignment process is performed on the other frames.

[0337] Here, the terminal device can group the first image and each second aligned image based on the frame type corresponding to each image, and obtain three groups of images, one group of which is long frame type images, another group of which is short frame type images, and another group of which is standard frame type images.

[0338] The terminal device can perform multi-frame denoising processing on each group of images respectively, so that a denoised result image (that is, the fourth image) can be obtained for each group of images, that is, a denoised result image can be obtained for each frame type.

[0339] It should be noted that the terminal device can adopt the multi-frame noise reduction processing scheme disclosed in the relevant technology to perform multi-frame noise reduction processing on each group of images respectively, so as to obtain the denoised result image corresponding to the group of images, which will not be elaborated here.

[0340] It can be understood that when the terminal device does not perform the second alignment processing in step S503 on other frames, the terminal device can directly group the first image and the images after the registration processing (that is, the first result image), and perform multi-frame noise reduction processing on each group of images respectively.

[0341] It is understood that, if the terminal device does not perform the image registration processing in step S502 on other frames, the terminal device can directly group the first image and the images after the first alignment processing, and perform multi-frame noise reduction processing on each group of images. The images after the first alignment processing include: other frames of the standard frame type, first aligned images corresponding to other frames of the long frame type, and first aligned images corresponding to other frames of the short frame type.

[0342] It is understandable that, when the terminal device does not perform the first alignment process of step S501 on other frames, the terminal device can directly group the N frames of original images and perform multi-frame noise reduction processing on each group of images respectively.

[0343] S505: The terminal device performs HDR processing on each of the fourth images after the noise reduction processing of the multiple frames to obtain a fifth image, and outputs the fifth image as a photographing result image.

[0344] Here, the terminal device may use the processing result image obtained by HDR processing as the photographing result image of the terminal device.

[0345] It can be understood that in the embodiments of the present application, the terminal device can use the HDR technology disclosed in the relevant technology to perform HDR processing on images of various frame types, which will not be described in detail here.

[0346] It should be pointed out that the terminal device performs HDR processing on the denoised result images (that is, the fourth image) corresponding to the three frame types, which can effectively fuse the scene information carried by images of different frame types, and obtain a photographic result image with complementary information, which helps to further improve the imaging quality of the terminal device.

[0347] It is understandable that the terminal device performs multi-frame noise reduction processing on each frame type of image before performing HDR processing in order to reduce the impact of noise, thereby further improving the imaging quality of the terminal device. In some application scenarios, the terminal device may not perform multi-frame noise reduction processing on each frame type of image. That is, the terminal device may directly perform HDR processing on the first image and each second aligned image to obtain the resulting image.

[0348] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0349] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0350] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0351] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0352] In addition, in the description of this application and the appended claims, the terms "first," "second," "third," etc. are used only to distinguish and describe, and should not be understood as indicating or implying relative importance. It should also be understood that although the terms "first," "second," etc. are used in the text to describe various elements in some embodiments of the present application, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0353] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0354] The image processing method provided in the embodiments of the present application can be applied to a terminal device, which can be a tablet computer, a mobile phone, a wearable device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not limit the specific type of the terminal device.

[0355] Figure 6 This is a schematic diagram of the architecture (including software system and some hardware) of the embodiment of this application. Figure 6 As shown, the application architecture is divided into several layers, each with clear roles and divisions of labor. Layers communicate with each other via software interfaces. In some embodiments, the application architecture can be divided into five layers: from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer.

[0356] like Figure 6 As shown, the application layer includes the camera and the gallery. Figure 6 The application layer may also include other applications, which is not limited in this application. For example, the application layer may also include information, alarm clock, weather, stopwatch, compass, timer, flashlight, calendar, Alipay and other applications.

[0357] like Figure 6 As shown, the application framework layer includes a camera access interface. The camera access interface includes camera management and camera devices. The hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer includes multiple camera devices. The camera algorithm library includes an algorithm processing module. In some embodiments, the algorithm processing module is used to select an image with minimal glare from multiple frames of images captured by the camera device and pass the selected image to the graphics processor driver for processing.

[0358] The driver layer is used to drive hardware resources. The driver layer can include multiple driver modules. Figure 6 As shown, the driver layer includes camera device driver, digital signal processor driver and graphics processor driver, etc.

[0359] The hardware layer includes sensors, image signal processors, digital signal processors, and graphics processors. The sensors include multiple sensors, TOF cameras, and multispectral sensors.

[0360] For example, a user may click on a camera application. When the user clicks on the camera to take a photo, the photo taking instruction may be sent to the camera hardware abstraction layer through the camera access interface. The camera hardware abstraction layer calls the camera device driver and the camera algorithm library. The algorithm processing module in the camera algorithm library is used to select the image with the least glare effect from the multiple frames of images captured by the camera device and pass the selected image to the graphics processor driver for processing. The camera algorithm library is also used to send digital signals to the digital signal processor driver in the driver layer so that the digital signal processor driver calls the digital signal processor in the hardware layer for digital signal processing. The digital signal processor can return the processed digital signal to the camera algorithm library through the digital signal processor driver. The camera algorithm library is also used to send digital signals to the graphics signal processor driver in the driver layer so that the graphics signal processor driver calls the graphics processor in the hardware layer for digital signal processing. The graphics processor can return the processed graphics data to the camera algorithm library through the graphics processor driver.

[0361] Additionally, the image signal processor output can be sent to the camera device driver. The camera device driver can send the image signal processor output to the camera hardware abstraction layer. The camera hardware abstraction layer can then feed the image into a post-processing algorithm module for further processing, or it can feed the image into the camera access interface. The camera access interface can then send the image returned by the camera hardware abstraction layer to the camera.

[0362] The above describes in detail the software system used in the embodiments of the present application.

[0363] Below, combined with the attached Figure 7 Provide a detailed introduction to the hardware system of the terminal equipment.

[0364] Figure 7 This is a schematic diagram of the hardware structure of a terminal device provided in one embodiment of the present application.

[0365] The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a camera 193, a touch screen 194, and a SIM card interface 195, etc.

[0366] The audio module 170 may include a speaker and a microphone, etc.

[0367] Among them, the sensor module 180 may include a pressure sensor, an inertial sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, and an ambient light sensor (of course, the terminal device may also include other sensors, such as a temperature sensor, a bone conduction sensor, etc., which are not shown in the figure).

[0368] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. Among them, the controller can be the nerve center and command center of the terminal device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0369] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0370] The processor 110 can execute the image processing method provided in the embodiments of the present application. The processor 110 can include different devices. For example, when the processor 110 integrates a CPU and a GPU, the CPU and the GPU can cooperate to execute the image processing method provided in the embodiments of the present application. For example, part of the algorithm in the image processing method is executed by the CPU, and another part of the algorithm is executed by the GPU to achieve faster processing efficiency.

[0371] It should be understood that the terminal device shown in the figure is only one example, and the terminal device may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components.

[0372] In addition, those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0373] An embodiment of the present application also provides a terminal device, which includes at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, the terminal device implements the steps of any of the above-mentioned method embodiments.

[0374] In addition, those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0375] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0376] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0377] An embodiment of the present application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the steps in the above-mentioned method embodiments.

[0378] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. Computer-readable storage media may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0379] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0380] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0381] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0382] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Receiving a photographing instruction, wherein the photographing instruction is used to trigger taking a photograph of the current photographing scene; Acquire N frames of images for the current photographing scene, where N is an integer greater than or equal to 2; determining N glare parameters corresponding to the N frames of image according to pixel features of each frame of image in the N frames of image, each glare parameter being used to characterize a degree of glare impact of the image; A first image is outputted based on the N glare parameters, where the first image is an image in which the glare parameters of the N frames of images satisfy a first preset condition, wherein the glare parameters satisfying the first preset condition include: a glare impact degree represented by the glare parameters being the smallest among the N glare parameters.

2. The image processing method according to claim 1, wherein: The N frames of images include a target image, and the target image is any frame of image in the N frames of images; The determining of N glare parameters corresponding to the N frames of images according to the pixel features of each frame of the N frames of images includes: determining a first glare degree according to the glare index of each pixel in the target image, wherein the first glare degree is used to represent the overall glare degree of the target image, and the glare index of each pixel is used to represent the difference between the brightness and saturation corresponding to the pixel; In a case where the target image includes a first object, determining a second glare degree according to a glare index of each pixel point in a target area, wherein the second glare degree is used to characterize the glare degree of the target area, where the target area is the area in the target image where the first object is located; A first weighted value of the target image is determined according to the first glare degree, the second glare degree, and a first weight coefficient for the second glare degree, wherein the glare parameters of the target image include the first glare degree, the second glare degree, and the first weighted value.

3. The image processing method according to claim 2, wherein: Outputting a first image according to the N glare parameters includes: In a case where each image includes the first object, outputting an image having the smallest first weighted value among the N frames of images; In a case where the first object is not included in each image, an image with the minimum first glare level among the N frames of images is output.

4. The image processing method according to any one of claims 2 to 3, characterized in that: When the target image includes the first object, determining the second glare degree according to the glare index of each pixel point in the target area includes: In a case where there is glare in the target image and the target image includes a first object, a second glare degree is determined according to a glare index of each pixel point in the target area.

5. The image processing method according to claim 4, characterized in that The determining, based on pixel features of each of the N frames of images, N glare parameters corresponding to the N frames of images further includes: In a case where there is glare in the target image and the target image does not include the first object, determining the contrast of the target image according to an image mean and an image standard deviation of the target image; A second weighted value of the target image is determined according to the first glare degree, the contrast, and a second weight coefficient for the contrast, wherein the glare parameter also includes the second weighted value.

6. The image processing method according to claim 5, characterized in that: Outputting a first image according to the N glare parameters includes: In a case where each image includes the first object, outputting an image having the smallest first weighted value among the N frames of images; In a case where each image does not include the first object, an image having the smallest second weighted value among the N frames of images is output.

7. The image processing method according to claim 1, wherein: Before determining N glare parameters corresponding to the N frames of images based on pixel features of each frame of the N frames of images, the method further includes: Performing glare detection on each frame of image to obtain a first detection result for each frame of image, wherein the first detection result is used to indicate whether there is glare in the image; In the case where there is glare in each frame of image, target detection is performed on each image to obtain a second detection result, where the second detection result is used to indicate whether the corresponding image includes the first object and the position of the first object in the image.

8. The image processing method according to any one of claims 1 to 7, characterized in that: The method further comprises: In the case that there is an image without glare among the N frames of images, a second image is output, where the second image is an image whose frame type satisfies a second preset condition among the images without glare, wherein the frame types include long frames, short frames, and standard frames, and satisfying the second preset condition includes: the frame type is a standard frame.

9. The image processing method according to claim 1, wherein: Outputting a first image according to the N glare parameters includes: Obtaining a first brightness, where the first brightness is the brightness of light in the current photographing scene; determining a first coefficient of the first image according to the pixel value of each pixel in the first image, where the first coefficient is a proportion of a target pixel in the first image, and the pixel value of the target pixel is a value when the pixel is overexposed; Determine a photographing result image according to the N frames of images, the first brightness, and the first coefficient; Output the photographing result image.

10. The image processing method according to claim 9, wherein: The determining of a photographing result image according to the N frames of images, the first brightness, and the first coefficient includes: When the first brightness is greater than a preset brightness value and the first coefficient is less than or equal to a preset coefficient value, the first image is determined as the photographing result image.

11. The image processing method according to claim 9, wherein: The determining of a photographing result image according to the N frames of images, the first brightness, and the first coefficient includes: When the first brightness is less than or equal to a preset brightness value and the first coefficient is less than or equal to a preset coefficient value, performing image registration processing on each of the other frames with reference to the first image to obtain first result images corresponding to the other frames, wherein the other frames are images other than the first image in the N frames of images; performing multi-frame denoising processing on a first denoised image set to obtain a third image, wherein the first denoised image set includes the first image and first result images corresponding to other frames respectively; The third image is determined as the photographing result image.

12. The image processing method according to claim 9, wherein: The determining of a photographing result image according to the N frames of images, the first brightness, and the first coefficient includes: When the first coefficient is greater than a preset coefficient value, performing image registration processing on other frames respectively with reference to the first image to obtain first result images corresponding to other frames; performing multi-frame denoising processing on each image subset in the first denoised image set to obtain a fourth image corresponding to each image subset, wherein the image subset is a set of images having the same frame type in the first denoised image set; Performing high dynamic range imaging processing on the fourth images corresponding to the respective frame types to obtain a fifth image; The fifth image is determined as the photographing result image.

13. The image processing method according to claim 11 or 12, characterized in that: The step of performing image registration processing on each of the other frames with the first image as a reference includes: When the frame type of the other frames is a long frame or a short frame, determining an image gain between the first image and the other frames, where the image gain is a ratio of image brightness of the first image to image brightness of the other frames, wherein image brightness is a product of image exposure time and sensitivity; performing a first alignment process on the other frames according to the image gain to obtain a first aligned image corresponding to the other frames, wherein the first alignment process is used to adjust the image brightness of the other frames to be the same as the image brightness of the first image; Perform image registration processing on the first aligned image with the first image as a reference.

14. The image processing method according to any one of claims 11 to 13, characterized in that: The performing multi-frame noise reduction processing on the first noise reduction image set includes: determining a pixel gain between each pixel in the first image and a corresponding pixel in the first result image, where the pixel gain is a ratio of the brightness of a first pixel in the first image to the brightness of a second pixel in the first result image, the first pixel and the second pixel having the same pixel coordinates; performing a second alignment process on the first result image according to the pixel gain corresponding to each pixel point to obtain a second aligned image, wherein the second alignment process is used to adjust the pixel brightness of the first result image to be the same as the pixel brightness of the first image; Multi-frame noise reduction processing is performed on a second denoised image set, where the second denoised image set includes the first image and second aligned images corresponding to each first result image.

15. A terminal device, characterized in that: The terminal device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the image processing method according to any one of claims 1 to 14 is implemented.

Images