Image processing method, device, storage medium and electronic device

Abstract

An image processing method, an image processing device, a storage medium and an electronic device. The image processing method comprises: obtaining an image to be processed from a plurality of consecutive frames of images (S310); performing brightness mapping processing on the image to be processed to generate an intermediate image (S320); obtaining at least one reference image using the plurality of consecutive frames of images (S330); performing temporal filtering processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed (S340). The method improves the brightness problem existing in the image, realizes image denoising and repair of local information loss, and obtains a high-quality image optimization result.

Description

Background Art

[0002] Image brightness refers to the brightness of an image, and is an important factor that affects people's visual experience when viewing an image. When the image brightness is inappropriate, such as too high or too low, the image content cannot be fully presented, for example, faces and text in the image are difficult to recognize, thus affecting the image quality. Summary of the invention

[0003] The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, thereby improving the brightness problem in an image at least to a certain extent.

[0004] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: obtaining an image to be processed from a plurality of consecutive image frames; performing brightness mapping processing on the image to be processed to generate an intermediate image; obtaining at least one reference image using the plurality of consecutive image frames; and performing fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.

[0005] According to a second aspect of the present disclosure, there is provided an image processing device, comprising a processor; wherein the processor is used to execute the following program modules stored in a memory: a module for acquiring an image to be processed, for acquiring an image to be processed from a plurality of consecutive image frames; an intermediate image generation module, for performing brightness mapping processing on the image to be processed to generate an intermediate image; a reference image acquisition module, for acquiring at least one reference image using the plurality of consecutive image frames; and an image fusion processing module, for performing fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.

[0006] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method of the first aspect and possible implementation methods thereof are implemented.

[0007] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image processing method of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0008] The technical solution disclosed in this disclosure has the following beneficial effects:

[0009] On the one hand, brightness mapping of the image to be processed can improve brightness problems in the image, such as too low brightness, too high brightness, uneven local brightness, etc. Fusion processing of the reference image and the intermediate image can achieve noise reduction and repair the local information loss in the image, thereby obtaining an optimized image with clear and visible image content. On the other hand, this solution can optimize any frame of the image based on the continuous multi-frame images collected during video shooting or image preview, without the need for external information, and has low implementation cost and high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A Shows a face image captured in a low-light environment;

[0011] Figure 1B Shows a face image captured in a backlit environment;

[0012] Figure 2 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0013] Figure 3 A flowchart showing an image processing method in this exemplary embodiment is shown;

[0014] Figure 4 A flowchart of a method for obtaining an image to be processed in this exemplary embodiment is shown;

[0015] Figure 5 A schematic diagram showing conversion of a RAW image into a single-channel image in this exemplary embodiment;

[0016] Figure 6 A schematic diagram showing a mapping curve in this exemplary embodiment;

[0017] Figure 7 A flowchart showing a brightness mapping processing method in this exemplary embodiment is shown;

[0018] Figure 8 A flowchart of a method for obtaining a reference image in this exemplary embodiment is shown;

[0019] Fig. 9 A diagram showing an example of image processing in this exemplary embodiment;

[0020] Fig.10 An example diagram showing image processing and face recognition in this exemplary embodiment;

[0021] Fig.11 A schematic diagram showing the structure of an image processing device in this exemplary embodiment is shown;

[0022] Fig.12 A schematic diagram of an architecture in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0024] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0025] The ambient lighting conditions when shooting images and the exposure parameters of the shooting equipment will affect the image brightness. The ambient lighting conditions have a greater impact. In extreme environments such as low light, backlight, and extremely strong light sources, the images shot may lack important information. For example, Figure 1A As shown in Figure 1, when taking a portrait, if the ambient light is very weak, that is, in a low-light environment, the overall brightness of the captured image is very low, making it difficult to recognize the face; Figure 1B As shown in FIG. 1 , if the light source is behind the person being photographed, that is, in a backlit environment, the brightness of the face is low and it is also difficult to identify. Therefore, it is necessary to improve the brightness of images taken in extreme environments.

[0026] To this end, the exemplary embodiment of the present disclosure first provides an image processing method, and its application scenarios include but are not limited to: in the intelligent interaction scenario of the mobile terminal, the AON (Always ON) camera is used to collect, monitor and detect face and gesture information to achieve specific interactive functions, such as automatically activating the display screen when a face is detected, and automatically turning the page of the user interface when a page turning gesture is detected; however, in environments such as weak lighting, backlight, and extremely strong light sources, the face and gesture images collected by the AON camera may have too low or too high brightness, affecting the accuracy of the above detection; through the image processing method of this exemplary embodiment, the image brightness can be improved to improve the accuracy of face, gesture, etc. detection. In the target tracking scenario, it is necessary to collect images in real time to identify the target in the image; if it is in the above extreme lighting environment, the brightness of the target part in the image is too low or too high, making it difficult to identify; it can also be improved through the image processing method of this exemplary embodiment to improve the recognition accuracy.

[0027] The exemplary embodiment of the present disclosure also provides an electronic device for executing the above-mentioned image processing method. The electronic device includes but is not limited to a computer, a smart phone, a tablet computer, a game console, a wearable device, etc. In general, the electronic device includes a processor and a memory. The memory is used to store executable instructions of the processor, and can also store application data, such as image data, game data, etc.; the processor is configured to execute the image processing method in this exemplary embodiment by executing the executable instructions.

[0028] Below Figure 2 Taking the mobile terminal 200 in the example, the structure of the above electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The construction in can also be applied to fixed type equipment.

[0029] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal storage area 221, an external memory interface 222, a USB (Universal Serial Bus) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294 and a SIM (Subscriber Identification Module) card interface 295, etc.

[0030] The processor 210 may include one or more processing units, for example, the processor 210 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0031] In some implementations, the processor 210 may include one or more interfaces, and may be connected to other components of the mobile terminal 200 via different interfaces.

[0032] The internal storage area 221 may be used to store computer executable program codes, which include instructions. The internal storage area 221 may include volatile memory, such as DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), and may also include non-volatile memory, such as at least one disk storage device, flash memory device, UFS (Universal Flash Storage), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal storage area 221 and / or instructions stored in a memory provided in the processor.

[0033] The external memory interface 222 can be used to connect an external memory, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory communicates with the processor 210 through the external memory interface 222 to implement a data storage function, such as storing music, video and other files.

[0034] The USB interface 230 is an interface that complies with USB standard specifications and can be used to connect a charger to charge the mobile terminal 200, or to connect headphones or other electronic devices.

[0035] The charging management module 240 is used to receive charging input from a charger. While the charging management module 240 is charging the battery 242, it can also power the device through the power management module 241; the power management module 241 can also monitor the status of the battery.

[0036] The wireless communication function of the mobile terminal 200 can be implemented by antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the mobile terminal 200. The wireless communication module 260 can provide wireless communication solutions including WLAN (Wireless Local Area Networks) (such as Wi-Fi (Wireless Fidelity) network), BT (Bluetooth), GNSS (Global Navigation Satellite System), FM (Frequency Modulation), NFC (Near Field Communication), IR (Infrared) and the like applied to the mobile terminal 200.

[0037] The mobile terminal 200 can realize the display function through the GPU, the display screen 290 and the AP.

[0038] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the encoder, the decoder, the GPU, the display screen 290 and the AP, etc. The camera module 291 may include various types of cameras, such as an AON camera, a wide-angle camera, a high-definition camera, etc. The camera can be set at any position of the mobile terminal 200, for example, on a side with the display screen 290 to form a front camera, or on a side opposite to the display screen 290 to form a rear camera.

[0039] The mobile terminal 200 can implement audio functions through the audio module 270, the speaker 271, the receiver 272, the microphone 273, the earphone interface 274 and the AP.

[0040] The sensor module 280 may include a depth sensor 2801 , a pressure sensor 2802 , a gyroscope sensor 2803 , an air pressure sensor 2804 , etc., to implement different sensing detection functions.

[0041] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc. Motor 293 may generate vibration prompts, and may also be used for touch vibration feedback, etc. Buttons 294 include a power button, a volume button, etc.

[0042] The mobile terminal 200 may support one or more SIM card interfaces 295 for connecting a SIM card to implement functions such as calls and data communications.

[0043] Figure 3 The schematic flow of the image processing method in this exemplary embodiment is shown, which may include:

[0044] Step S310, obtaining an image to be processed from a plurality of consecutive frames of images;

[0045] Step S320, performing brightness mapping processing on the image to be processed to generate an intermediate image;

[0046] Step S330, obtaining at least one reference image frame by using the above-mentioned continuous multiple frames of images;

[0047] Step S340: perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.

[0048] Through the above method, on the one hand, brightness mapping processing is performed on the image to be processed, which can improve the brightness problems existing in the image, such as too low brightness, too high brightness, uneven local brightness, etc., and the fusion processing of the reference image and the intermediate image can achieve noise reduction and repair the local information loss in the image, so as to obtain an optimized image with clear and visible image content, which is conducive to further realizing applications such as face recognition, gesture recognition, and target detection; further, this solution improves the robustness of image capture and processing in extreme lighting environments, and reduces the dependence on the performance of hardware such as cameras and image sensors. For example, for image sensors with low photosensitivity, this solution can be used to optimize the captured image to obtain high-quality images, which is conducive to reducing hardware costs. On the other hand, this solution can optimize any frame of the image based on the continuous multi-frame images collected during video capture or image preview, without the need for external information, and has low implementation costs and high practicality.

[0049] The following are Figure 3 Each step in the procedure is described in detail.

[0050] In step S310, an image to be processed is obtained from a plurality of consecutive frames of images.

[0051] The above-mentioned continuous multiple frames of images can be images continuously captured by a camera, such as a video captured by a camera or a preview image continuously captured. The image to be processed can be any frame of the image. Generally, when the camera captures an image, the currently captured frame of the image is used as the image to be processed in real time. Figure 3 method to realize the processing of each frame of image.

[0052] In an optional embodiment, reference Figure 4 As shown, step S310 may include

[0053] Step S410, obtaining a current RAW image from a plurality of consecutive RAW image frames;

[0054] Step S420, performing channel conversion processing on the current RAW image to obtain an image to be processed.

[0055] RAW images refer to images stored in RAW format, generally referring to the original images captured by the image sensor in the camera. The image sensor collects light signals through the Bayer filter and converts them into digital signals to obtain RAW images. Figure 5 As shown in the left figure, each pixel in the RAW image has only one color in RGB and is arranged in a Bayer array. In an optional implementation, an AON camera can be set on the terminal device to collect the above-mentioned continuous multiple frames of RAW images.

[0056] In this exemplary embodiment, a frame of RAW image currently captured is obtained. Since the color channels of each pixel are different, channel conversion processing is performed on the current RAW image to obtain an image to be processed. Channel conversion processing is to unify the color channels of each pixel, including but not limited to the following two methods:

[0057] 1. Convert each pixel of the current RAW image to any channel in RGB, for example, convert them to the R channel. Generally, since the human eye is more sensitive to green, refer to Figure 5 As shown, the R channel and B channel of the current RAW image can be converted to the G channel to obtain an image consisting entirely of G channel pixels. Specifically, the pixel points of the R channel and the B channel can be mapped to be converted to the pixel values ​​of the G channel, for example:

[0058] P R *a 1 =P G , P B *a 2 =P G (1)

[0059] Among them, P R , P G , P B Represents the pixel values ​​of R channel, G channel, and B channel respectively; a 1 and a 2 They are the conversion coefficients of the R channel and the B channel to the G channel, which can be empirical coefficients, such as a 1 =0.299 / 0.587=0.509 (0.299 and 0.587 are the coefficients of grayscale conversion for R channel and G channel respectively), a 2=0.114 / 0.587=0.194 (0.114 is the coefficient of the B channel when grayscale conversion is performed) It can be seen that when converting the R and B channels to the G channel, the consistency of brightness is taken into consideration, that is, the brightness level of the same pixel before and after the conversion is basically the same.

[0060] 2. Convert the current RAW image to a grayscale image. For example, the R, G, and B channels can be converted to grayscale values ​​according to a certain coefficient ratio, or the RAW image can be demosaiced to obtain an RGB image, and then the grayscale value (Gray) of each pixel can be calculated by the following formula (2) to obtain a grayscale image.

[0061] Gray=P R *0.299+P G *0.587+P B *0.114 (2)

[0062] In the current RAW image, each pixel requires 10 bits, of which 8 bits record the pixel value and 2 bits record the channel information. Through channel conversion processing, a single-channel image (grayscale image can be regarded as a special single-channel image) is obtained, and there is no need to record channel information, so each pixel only requires 8 bits. Compared with RGB images, the amount of data in a single-channel image is greatly reduced, and the bit width used to record channel information is saved, thereby reducing the amount of data for subsequent processing, which is conducive to real-time optimization processing. In addition, a single-channel image can characterize the brightness of the image to be processed, and the information is relatively sufficient.

[0063] Continue to refer Figure 3 In step S320, brightness mapping is performed on the image to be processed to generate an intermediate image.

[0064] Brightness mapping refers to mapping the brightness value of each pixel in the image to be processed to a new value. The brightness values ​​of all pixels can be adjusted in the same direction, such as uniformly increasing or uniformly decreasing, and the brightness change values ​​of different pixels can be the same or different; the brightness values ​​of different pixels can also be adjusted in different directions, such as reducing the brightness of the overly bright part, increasing the brightness of the overly dark part, adjusting the grayscale of the image, etc.

[0065] In this exemplary embodiment, a mapping relationship between brightness values ​​before and after mapping may be preconfigured, and the mapping relationship may be related to the brightness level or brightness distribution of the image to be processed. Brightness mapping processing is then performed based on the mapping relationship to obtain an intermediate image after processing.

[0066] In an optional implementation, step S320 may include:

[0067] The image to be processed is subjected to brightness mapping processing according to the illumination information of the image to be processed.

[0068] The illumination information is information indicating the ambient illumination conditions of the image to be processed, such as the illumination value and the backlight ratio. The illumination value is a measure of the amount of illumination received per unit area of ​​the image to be processed. The backlight ratio is the proportion of the backlit area in the image to be processed. The illumination information can reflect whether there are illumination defects in the image to be processed, such as overall over-illumination, insufficient illumination, imbalanced local illumination distribution, etc., and thus, brightness mapping processing can be performed on the image to be processed in a targeted manner.

[0069] Exemplarily, the illuminance value of the image to be processed can be determined according to the exposure parameters of the image to be processed. The exposure parameters include at least one of exposure time (Exptime), sensitivity (ISO), and aperture value (F value). Generally, the exposure parameters are recorded when shooting the image to be processed. For the camera on a smartphone, the aperture value is usually a fixed value. The illuminance value is estimated by the exposure parameters. The more comprehensive the exposure parameters obtained, the more accurate the estimated illuminance value. For example, the following formula (3) can be referred to:

[0070] Lumi=a 3 *F 2 / (Exptime*ISO) (3)

[0071] Lumi represents the illumination value, the unit is lux; a 3 It represents the empirical coefficient, such as 12.4, which can be adjusted according to the actual shooting scene or camera performance; F is the aperture value, which is the ratio of the focal length of the lens to the effective diameter of the lens; Exptime is the exposure time in seconds; ISO is the sensitivity.

[0072] The illumination value of the image to be processed reflects the global illumination of the image to be processed. When the illumination value is too low, it means that the image to be processed is underexposed as a whole. When the illumination value is too high, it means that the image to be processed is overexposed as a whole. Therefore, in an optional implementation, step S320 may include:

[0073] When the illumination value of the image to be processed meets the first preset condition, a global brightness mapping process is performed on the image to be processed.

[0074] The first preset condition may include: less than the first threshold, or greater than the second threshold. The first threshold is a threshold for measuring whether it is underexposed, and the second threshold is a threshold for measuring whether it is overexposed. The two thresholds can be determined based on experience, or can be adjusted based on actual scenarios, such as appropriately increasing the first threshold and the second threshold during the day, and appropriately reducing the first threshold and the second threshold at night. When the illumination value of the image to be processed meets the first preset condition, there is underexposure or overexposure, so it is subjected to global luminance mapping. Specifically, the following steps may be included:

[0075] When the illumination value of the image to be processed is less than the first threshold, the image to be processed is subjected to global brightness upward mapping processing, that is, global brightening.

[0076] When the illumination value of the image to be processed is greater than the second threshold, the image to be processed is subjected to global brightness downward mapping processing, that is, global brightening.

[0077] Taking global brightening as an example, the brightness values ​​of different pixels in the image to be processed can be mapped to higher brightness values ​​according to a pre-set mapping curve. The mapping curve can be a linear curve, reflecting a linear mapping relationship, or a quadratic curve or other nonlinear curve, reflecting a nonlinear mapping relationship. It should be noted that the exemplary embodiment can be configured with multiple mapping curves, corresponding to different mapping intensities, and the higher the mapping intensity, the more significant the brightening. During specific processing, the mapping intensity and which mapping curve to use can be determined according to the illumination value of the image to be processed or the actual scene requirements. For example, the lower the illumination value, the higher the mapping intensity, and the mapping curve with a larger overall slope is used.

[0078] In an optional implementation, step S320 may further include:

[0079] When the illumination value of the image to be processed does not meet the first preset condition, determining the backlight ratio of the image to be processed;

[0080] When the backlight ratio of the image to be processed meets the second preset condition, tone mapping processing is performed on the image to be processed.

[0081] Among them, if the illumination value of the image to be processed does not meet the first preset condition, it means that the global illumination condition of the image to be processed is relatively appropriate, and then the backlight ratio is calculated to determine whether there is any local inappropriate situation.

[0082] In an optional embodiment, the backlight ratio of the image to be processed can be determined based on the brightness histogram of the image to be processed, which can specifically include: setting multiple brightness levels, counting the ratio of pixels in each brightness level, and forming a brightness histogram of the image to be processed; if there are at least two brightness levels, and the brightness difference between them reaches a set value (the set value can be an empirical value, and can also be determined according to a specific scene, for example, when there is a significant light source in the scene, the set value is generally larger, and the set value is generally smaller under natural light or no significant light source), then the lower brightness level is determined as the backlight part; counting the brightness histogram of all backlight parts, and obtaining the ratio of the backlight part to the image to be processed, that is, the backlight ratio.

[0083] In an optional implementation, the highest brightness value in the image to be processed can also be obtained, and the brightness threshold can be determined based on the highest brightness value. For example, the highest brightness value can be multiplied by a fixed coefficient less than 1 to obtain the brightness threshold; pixels below the brightness threshold are screened out, and connected areas therein are extracted, that is, isolated pixels therein are filtered out; and the proportion of the connected area in the image to be processed is taken as the backlight proportion.

[0084] The second preset condition may include: the backlight ratio is greater than the third threshold. It should be noted that the above calculation of the backlight ratio is to estimate the backlight phenomenon that may exist in the image to be processed. When the backlight ratio is greater than the third threshold, it can be considered that the probability of the backlight phenomenon is high. The third threshold is a threshold for measuring whether there is a backlight phenomenon, which can be determined based on experience or actual scenes. In this case, the image to be processed is subjected to tone mapping. The tone mapping process is essentially still a mapping of brightness. Unlike the above global brightness mapping process, the tone mapping process can change the brightness range or brightness level distribution of the image, and the brightness adjustment direction of each pixel can be different. For example, in the global brightness mapping process, for all pixels in the entire image, the brightness is adjusted in the same direction, such as increasing the brightness or decreasing the brightness as a whole, so as to adjust the overall brightness level; in the tone mapping process, for different pixels in the entire image, the brightness can be adjusted in different directions, such as mapping the brightness downward for the part with higher brightness and mapping the brightness upward for the part with lower brightness, so as to adjust the brightness distribution.

[0085] In this exemplary embodiment, the tone mapping process may be implemented through a mapping curve. Figure 6The mapping curve used in the tone mapping process is shown, where the horizontal axis is the brightness value before mapping, the vertical axis is the brightness value after mapping, and curves A, B, and C are mapping curves under different mapping intensities, the mapping intensity of curve C is the highest, and the mapping intensity of curve A is the lowest. It can be seen that after mapping the brightness value through the mapping curve, the brightness distribution of the image to be processed can be mapped to a smaller range. The higher the mapping intensity, the smaller the brightness range after mapping, thereby improving the problem of local invisibility caused by extremely uneven brightness distribution of different local areas. Generally, the higher the backlight ratio, or the greater the difference between the backlight part and the high brightness part, the higher the mapping intensity is used. It can be seen that the mapping curve used in the tone mapping process is different from the mapping curve used in the above-mentioned global brightness mapping process: the former is generally a nonlinear curve, one section of which has a slope greater than 45 degrees (i.e., the part of the brightness mapping process upward, generally a section with lower brightness), and the other section has a slope less than 45 degrees (i.e., the part of the brightness mapping process downward, generally a section with higher brightness); the latter can be a linear curve or a nonlinear curve, and the slope of the entire curve is greater than 45 degrees or less than 45 degrees.

[0086] In an optional implementation, when the backlight ratio of the image to be processed does not meet the second preset condition, it indicates that the global illumination and local illumination conditions of the image to be processed are both relatively good, and brightness mapping processing may not be performed, and the image to be processed may be used as an intermediate image.

[0087] Figure 7 The schematic flow of brightness mapping processing is shown, including:

[0088] Step S710, determining the illuminance value Lumi of the image to be processed;

[0089] Step S720, comparing the illuminance value with the first threshold value T1 and the second threshold value T2; when Lumi<T1 or Lumi>T2, determining that the first preset condition is met, executing step S730; otherwise, executing step S740;

[0090] Step S730, performing global brightness mapping processing on the image to be processed;

[0091] Step S740, determining a backlight ratio (BL ratio) of the image to be processed;

[0092] Step S750, comparing the backlight ratio with the third threshold value T3; when BL ratio>T3, it is determined that the second preset condition is met, and step S760 is executed; otherwise, the image to be processed is not processed, and the process jumps to step S770;

[0093] Step S760, performing tone mapping processing on the image to be processed;

[0094] Step S770, obtaining an intermediate image.

[0095] Through the above methods, the brightness of the image to be processed is improved from both global and local levels, and the obtained intermediate image can restore the image information lost due to the brightness problem to a certain extent.

[0096] Continue to refer Figure 3 In step S330, at least one reference image is acquired using the above-mentioned continuous multiple frames of images.

[0097] The reference image is one or more frames of images that are temporally continuous with the image to be processed, and can supplement the image information of the image to be processed.

[0098] In an optional embodiment, reference Figure 8 As shown, step S330 may include:

[0099] Step S810, obtaining at least one frame of image other than the image to be processed from the above-mentioned continuous multiple frames of images;

[0100] Step S820, performing brightness mapping processing on the at least one frame of image to obtain a reference image;

[0101] Step S830: Acquire an optimized image corresponding to the at least one frame of image as a reference image.

[0102] Among them, any one or more frames other than the image to be processed can be selected from the above-mentioned continuous multiple frames. Generally, the more images selected and the closer they are to the time of the image to be processed, the more conducive to improving the optimization effect. For example, let the image to be processed be the i-th frame image in the above-mentioned continuous multiple frames, i is a positive integer not less than 2, then the im-th frame image to the i-1-th frame image, that is, the continuous m-frame images located before the image to be processed, can be selected for subsequent optimization, and m is an arbitrary positive integer. The value of m can be determined by combining experience, actual needs and computing power performance.

[0103] In step S820 and step S830, one of them can be selectively performed. In step S820, the brightness mapping process of the at least one frame of image can refer to step S320 and step S830. Figure 7 In step S830, the optimized image corresponding to the at least one frame of image may be a brightness mapping process performed on the image to be processed. Figure 3 The image is obtained by optimizing the image by the method flow. For example, for each frame of image collected by the camera in real time, Figure 3 The method process is optimized to obtain an optimized image corresponding to each frame image, and the optimized image can be used as a reference image when optimizing the next frame image.

[0104] In an optional embodiment, the exposure parameters of the image to be processed may be different from the exposure parameters of the at least one frame of image, for example, any one of the exposure time, sensitivity, and aperture value may be different. Thus, the image to be processed or the intermediate image may form information complementarity in terms of exposure and brightness with the at least one frame of image or the reference image. Furthermore, the device may control the camera to capture the above-mentioned continuous multiple frames of images with different exposure parameters, for example, gradually increasing the exposure time or gradually increasing the sensitivity when capturing images, so that the exposure parameters between different frames of images are different, so as to maximize the information in the continuous multiple frames of images to form more effective information complementarity.

[0105] Continue to refer Figure 3 In step S340, the reference image and the intermediate image are fused to obtain an optimized image corresponding to the image to be processed.

[0106] The reference image and the intermediate image have a slight time difference, so they have different image information. Fusion of the two images can repair the image information of the details. For example, the image frequencies in the two images can be scanned and compared, and the area in the reference image with a higher image frequency than the intermediate image can be selected and fused with the intermediate image.

[0107] During fusion, pixel points at the same position may be weighted, for example, weights may be determined according to image frequencies of pixel points in two images, and weighted fusion may be performed.

[0108] In an optional implementation, step S340 may include:

[0109] The reference image and the intermediate image are subjected to temporal filtering.

[0110] Specifically, the reference image and the intermediate image can be arranged into an image sequence according to their timestamps, and then the image information in the image sequence can be converted into a time series signal, and the time series signal can be filtered. For example, various methods such as Gaussian filtering and mean filtering can be used to achieve further optimization effects such as noise reduction.

[0111] In an optional embodiment, after the reference image and the intermediate image are fused, the brightness of the fused image can be adjusted based on the above-mentioned continuous multiple frame images to obtain an optimized image corresponding to the image to be processed. Since the intermediate image has undergone brightness mapping processing, its brightness may be significantly different from that of other frame images, resulting in brightness jumps in the video screen. Therefore, the brightness of the fused image can be adjusted according to the brightness of other frame images. For example, the same area is selected in the other frame images and the fused image, and the overall brightness of the fused image is adjusted based on the brightness difference of the area in the two images, thereby ensuring the brightness consistency between continuous frame images. It should be noted that, unlike the brightness mapping process in step S320, the brightness adjustment performed here is generally brightness fine-tuning or brightness smoothing.

[0112] Fig. 9 The figure shows the effect of optimizing the image to be processed, where the upper row is the image to be processed and the lower row is the corresponding optimized image. It can be seen that the brightness of the face or gesture in the image to be processed is low, which makes it difficult to see clearly. After optimization, clearer face or gesture information can be obtained.

[0113] In an optional implementation, after obtaining the optimized image corresponding to the image to be processed, at least one of target detection, face recognition, and gesture recognition may be performed on the optimized image, thereby obtaining a more accurate detection or recognition result. Fig.10 An example diagram of face recognition on an optimized image is shown. It can be seen that after optimization, the face part is clearly visible, so that the face area and facial feature points can be accurately detected.

[0114] The image processing method of this exemplary embodiment was used to perform an optimization test on a data set of 845 face and gesture images, most of which had an illumination value of 3 to 15 lux, and the distance between the face or gesture and the camera was 15 to 40 cm, which belongs to a weak light environment and a backlit environment.

[0115] Without optimizing the above dataset, the face and gesture recognition tests were conducted, with an accuracy of 95.2%, a recall of 16.7%, and an F1 value (F1 value is an evaluation index for algorithm models in the field of machine learning, calculated by

[0116] After the above dataset is optimized, it is tested for face and gesture recognition, with an accuracy of 99.1%, a recall rate of 82.3%, and an F1 value of 90%.

[0117] It can be seen that the image processing method of this exemplary embodiment has a significant improvement effect on the recognition algorithm in weak lighting environment and backlight environment. Therefore, even in extreme lighting environment, it is possible to obtain results similar to those in normal lighting environment, thereby increasing the robustness of the recognition algorithm and broadening its application scenarios.

[0118] The exemplary embodiment of the present disclosure also provides an image processing device. Fig.11 As shown, the image processing device 1100 may include a processor 1110 and a memory 1120. The memory 1120 stores the following program modules:

[0119] The to-be-processed image acquisition module 1121 is used to acquire the to-be-processed image from a plurality of consecutive frames of images;

[0120] An intermediate image generation module 1122 is used to perform brightness mapping processing on the image to be processed to generate an intermediate image;

[0121] A reference image acquisition module 1123 is used to acquire at least one reference image frame using the above-mentioned continuous multiple frames of images;

[0122] The image fusion processing module 1124 is used to fuse the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed;

[0123] The processor 1110 may be configured to execute the above program modules.

[0124] In an optional implementation, the intermediate image generation module 1122 is configured to:

[0125] The image to be processed is subjected to brightness mapping processing according to the illumination information of the image to be processed.

[0126] In an optional implementation, the illumination information includes an illumination value; the intermediate image generation module 1122 is configured to:

[0127] When the illumination value of the image to be processed meets the first preset condition, a global brightness mapping process is performed on the image to be processed.

[0128] In an optional implementation, the intermediate image generation module 1122 is configured to:

[0129] The illumination value of the image to be processed is determined according to the exposure parameters of the image to be processed; the exposure parameters include at least one of exposure time, sensitivity, and aperture value.

[0130] In an optional implementation, the illumination information further includes a backlight ratio; the intermediate image generation module 1122 is configured to:

[0131] When the illumination value of the image to be processed does not meet the first preset condition, determining the backlight ratio of the image to be processed;

[0132] When the backlight ratio of the image to be processed meets the second preset condition, tone mapping processing is performed on the image to be processed.

[0133] In an optional implementation, the intermediate image generation module 1122 is configured to:

[0134] The backlight ratio of the image to be processed is determined according to the brightness histogram of the image to be processed.

[0135] In an optional implementation, the intermediate image generation module 1122 is configured to:

[0136] When the backlight ratio of the image to be processed does not meet the second preset condition, the image to be processed is used as an intermediate image.

[0137] In an optional implementation, the reference image acquisition module 1123 is configured to:

[0138] Acquire at least one frame of image other than the image to be processed from the continuous multiple frames of image;

[0139] Performing brightness mapping processing on the at least one frame of image to obtain a reference image; or

[0140] An optimized image corresponding to the at least one frame of image is obtained as a reference image.

[0141] In an optional implementation, the exposure parameter of the image to be processed is different from the exposure parameter of the at least one frame of image.

[0142] In an optional embodiment, the image to be processed is the i-th frame image in a continuous plurality of frame images; the above-mentioned at least one frame image includes the im-th frame image to the i-1-th frame image in the continuous plurality of frame images; i is a positive integer not less than 2, and m is any positive integer.

[0143] In an optional implementation, the image fusion processing module 1124 is configured to:

[0144] The reference image and the intermediate image are subjected to temporal filtering.

[0145] In an optional implementation, the image fusion processing module 1124 is configured to:

[0146] After the reference image and the intermediate image are fused, the brightness of the fused image is adjusted according to the above-mentioned continuous multiple-frame images to obtain an optimized image corresponding to the image to be processed.

[0147] In an optional implementation, the to-be-processed image acquisition module 1121 is configured as follows:

[0148] Get the current RAW image from multiple consecutive RAW images;

[0149] Perform channel conversion processing on the current RAW image to obtain the image to be processed.

[0150] In an optional implementation, the to-be-processed image acquisition module 1121 is configured as follows:

[0151] Convert the R and B channels of the current RAW image to the G channel; or

[0152] Convert the current RAW image to grayscale.

[0153] In an optional implementation, the image processing apparatus 1100 is configured in a terminal device, and the terminal device includes an AON camera for acquiring the above-mentioned continuous multiple-frame RAW images.

[0154] In an optional implementation, the memory 1120 further includes the following program modules:

[0155] The image recognition application module is used to perform at least one of target detection, face recognition, and gesture recognition on the above-mentioned optimized image.

[0156] The specific details of each part of the above-mentioned device 1100 have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0157] Fig.12 FIG. 1 shows a schematic diagram of the architecture of this exemplary embodiment. Fig.12 As shown, the electronic device is equipped with an AON camera, which runs the AON camera service, and can implement the underlying processing of the image through the image signal processor, such as executing the image processing method of this exemplary embodiment, processing the collected original image, obtaining the corresponding optimized image, and providing the optimized image to the AON software service. The AON software service can perform monitoring, recognition and other services through a digital signal processor, such as performing face recognition and gesture recognition on the optimized image, obtaining corresponding recognition results, and providing the recognition results to the application service. The application service can run related applications through the main processor, and use the above-mentioned face and gesture recognition results to implement specific interactive instructions, such as locking and unlocking the screen, turning pages in the user interface, etc.

[0158] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In an optional embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0159] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0161] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0162] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0163] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0164] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0165] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0166] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0167] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed from multiple consecutive frames of images; Performing brightness mapping processing on the image to be processed to generate an intermediate image; Acquire at least one reference image frame using the continuous multiple image frames; and The reference image and the intermediate image are fused to obtain an optimized image corresponding to the image to be processed. Wherein, the step of acquiring at least one reference image frame by using the continuous multiple image frames includes: Acquire at least one frame of image other than the image to be processed from the continuous multiple frames of image; Performing brightness mapping processing on the at least one frame of image to obtain the reference image; or obtaining an optimized image corresponding to the at least one frame of image as the reference image, The exposure parameter of the image to be processed is different from the exposure parameter of the at least one frame of image.

2. The method according to claim 1, characterized in that The performing brightness mapping processing on the image to be processed includes: Perform brightness mapping processing on the image to be processed according to the illumination information of the image to be processed.

3. The method according to claim 2, characterized in that The illumination information includes an illumination value; and performing brightness mapping processing on the image to be processed according to the illumination information of the image to be processed includes: When the illumination value of the image to be processed meets a first preset condition, a global brightness mapping process is performed on the image to be processed.

4. The method according to claim 3, characterized in that The method further comprises: Determining an illumination value of the image to be processed according to an exposure parameter of the image to be processed; The exposure parameters include at least one of exposure time, sensitivity, and aperture value.

5. The method according to claim 3, characterized in that: The illumination information further includes a backlight ratio; and the performing brightness mapping processing on the image to be processed according to the illumination information of the image to be processed further includes: When the illumination value of the image to be processed does not meet the first preset condition, determining the backlight ratio of the image to be processed; When the backlight ratio of the image to be processed meets the second preset condition, tone mapping processing is performed on the image to be processed.

6. The method according to claim 5, characterized in that The determining the backlight ratio of the image to be processed includes: The backlight ratio of the image to be processed is determined according to the brightness histogram of the image to be processed.

7. The method according to claim 5, characterized in that The method further comprises: When the backlight ratio of the image to be processed does not meet the second preset condition, the image to be processed is used as the intermediate image.

8. The method according to claim 1, characterized in that The image to be processed is the i-th frame image in the continuous multiple frame images; the at least one frame image includes the im-th frame image to the i-1-th frame image in the continuous multiple frame images; i is a positive integer not less than 2, and m is an arbitrary positive integer.

9. The method according to claim 1, characterized in that: The fusing process of the reference image and the intermediate image comprises: Temporal filtering is performed on the reference image and the intermediate image.

10. The method according to claim 1, characterized in that After fusing the reference image and the intermediate image, the method further includes: According to the continuous multiple frames of images, the brightness of the fused image is adjusted to obtain an optimized image corresponding to the image to be processed.

11. The method according to claim 1, characterized in that: The step of acquiring the image to be processed from the continuous multiple frames of images comprises: Get the current RAW image from multiple consecutive RAW images; Channel conversion processing is performed on the current RAW image to obtain the image to be processed.

12. The method according to claim 11, characterized in that The performing channel conversion processing on the current RAW image includes: Convert the R channel and the B channel of the current RAW image into a G channel; or The current RAW image is converted into a grayscale image.

13. The method according to claim 11, characterized in that The method is applied to a terminal device, wherein the terminal device comprises a normally-on camera for collecting the continuous multiple frames of RAW images.

14. The method according to claim 1, characterized in that After obtaining the optimized image corresponding to the image to be processed, the method further includes: The optimized image is subjected to at least one of target detection, face recognition, and gesture recognition.

15. An image processing device, characterized in that: Including processors; The processor is used to execute the following program modules stored in the memory: The module for acquiring the image to be processed is used to acquire the image to be processed from a plurality of consecutive frames of images; An intermediate image generation module, used for performing brightness mapping processing on the image to be processed to generate an intermediate image; A reference image acquisition module, used to acquire at least one reference image frame using the continuous multiple frames of images; An image fusion processing module is used to fuse the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed. Wherein, the reference image acquisition module is also used for: Acquire at least one frame of image other than the image to be processed from the continuous multiple frames of image; Performing brightness mapping processing on the at least one frame of image to obtain the reference image; or obtaining an optimized image corresponding to the at least one frame of image as the reference image, The exposure parameter of the image to be processed is different from the exposure parameter of the at least one frame of image.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

17. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 14 by executing the executable instructions.

Images