Image processing method and device and readable storage medium

Abstract

The embodiment of the invention provides an image processing method and device and a readable storage medium. The method is applied to an image acquisition device, the image acquisition device comprises an image sensor and an image processing unit, and the method comprises the following steps: acquiring a long frame image and a short frame image output by the image sensor; the long frame image and the short frame image are RAW images; inputting the short-frame image into a trained flicker recognition model, and outputting a flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model obtained by training based on training data of the RAW image; and adjusting parameters of the image processing unit according to the flicker identification result. According to the embodiment of the invention, the parameter adjustment result can take effect in real time in the current frame, and the flicker phenomenon of the image is eliminated in real time, so that the quality of the HDR video image output by the image acquisition equipment can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device and readable storage medium. Background Art

[0002] Image sensors often use a rolling shutter to control exposure, and the exposure of the entire image is completed by exposing the image line by line. The exposure time of different lines on the image is the same, but the start and end time of the exposure are inconsistent. When the energy of light received by different lines of the image is different, flicker will appear on the captured image, usually manifested as periodically changing light and dark stripes on the image. In some complex light source environments, such as when there are multiple light sources in the picture or there are plane occlusions, reflections, etc., the mutual interference of multiple light sources will cause the flicker to be more varied and difficult to detect accurately. In addition, when performing high dynamic range (HDR) image acquisition, the exposure time of short frames is usually shorter, which is more prone to flicker.

[0003] The presence of flicker causes a significant degradation in the quality of video imaging, and the processing of flicker depends on accurate flicker detection. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method, an apparatus and a readable storage medium, which can make the parameter adjustment result take effect in real time in the current frame, so as to eliminate the image flickering phenomenon in real time, thereby improving the quality of the HDR video image output by the image acquisition device.

[0005] In a first aspect, an embodiment of the present invention discloses an image processing method, which is applied to an image acquisition device, wherein the image acquisition device includes an image sensor and an image processing unit, and the method includes:

[0006] Acquire a long frame image and a short frame image output by the image sensor; the long frame image and the short frame image are RAW images;

[0007] Inputting the short frame image into a trained flicker recognition model, and outputting a flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on training data of the RAW image;

[0008] The parameters of the image processing unit are adjusted according to the flicker recognition result.

[0009] In a second aspect, an embodiment of the present invention discloses an image processing device, which is applied to an image acquisition device, wherein the image acquisition device includes an image sensor and an image processing unit, and the device includes:

[0010] An image acquisition module, used to acquire a long-frame image and a short-frame image output by the image sensor; the long-frame image and the short-frame image are RAW images;

[0011] A flicker recognition module, used for inputting the short frame image into a trained flicker recognition model, and outputting a flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on the training data of the RAW image;

[0012] A parameter adjustment module is used to adjust the parameters of the image processing unit according to the flicker recognition result.

[0013] In a third aspect, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors of a device, enables the device to execute one or more of the image processing methods described above.

[0014] The embodiments of the present invention include the following advantages:

[0015] The image processing method provided by the embodiment of the present invention can input the short frame image in the RAW image output by the image sensor into the trained flicker recognition model when the image acquisition device acquires a high dynamic (HDR) video image, and output the flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on the training data of the RAW image, which can effectively identify the flicker phenomenon in the RAW image; and then the parameters of the image processing unit can be adjusted according to the flicker recognition result to eliminate the flicker phenomenon in the HDR image. Since the embodiment of the present invention detects whether there is a flicker phenomenon based on the RAW image output by the image sensor, and adjusts the ISP parameters according to the flicker recognition result, the parameter adjustment result can take effect in real time in the current frame, thereby improving the quality of the HDR video image output by the image acquisition device. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1 It is a flow chart of flicker recognition and ISP parameter adjustment based on RGB images of the present invention;

[0018] Figure 2 is a flowchart of steps of an image processing method embodiment of the present invention;

[0019] Figure 3 It is a schematic diagram of the present invention for processing a clean RGB image in sequence through preset modules to output a clean RAW image;

[0020] Figure 4 It is a schematic diagram of a process of performing high dynamic synthesis according to flicker recognition results of the present invention;

[0021] Figure 5 It is a structural block diagram of an image processing device embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, the term "and / or" in the specification and claims is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments of the present invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0024] To facilitate understanding of the embodiments of the present invention, some terms used in the embodiments of the present invention are explained below to facilitate understanding by those skilled in the art.

[0025] Image sensor, also known as photosensitive element, is used to convert light signals into electrical signals, obtain raw data, and pass it to ISP for processing.

[0026] RAW images are the original data that the image sensor converts the captured light signals into digital signals. They are lossless and contain the original color information of the object.

[0027] The image processing unit, also known as the image signal processor (ISP), is used to perform post-processing on the raw data output by the image sensor, mainly including linear correction, noise removal, bad pixel removal, interpolation, white balance, automatic exposure control, etc., and convert the raw data output by the image sensor into a format that can be displayed and stored on various devices, such as converting it into an image signal in a standard RGB format.

[0028] The parameters of the image processing unit include, but are not limited to, exposure parameters. The exposure parameters include aperture, shutter speed, and sensitivity. In actual implementation, the appropriate exposure can be obtained by controlling the three parameters of aperture, shutter speed, and sensitivity.

[0029] Exposure refers to the total amount of light received by the photosensitive element during the exposure time. Exposure determines the brightness of the image. The greater the exposure, the brighter the image; the smaller the exposure, the darker the image. The exposure can be changed by changing the exposure time, thereby changing the brightness of the captured image. For example, the exposure time can be reduced by reducing the length of time that light is incident on the image sensor (for example, by increasing the speed at which the image sensor is scanned in a "rolling shutter" capture), thereby reducing the exposure and reducing the brightness of the image.

[0030] High Dynamic Range (HDR), the image acquisition device may be provided with an HDR mode or function. After the HDR mode of the image acquisition device is turned on, the image acquisition device uses the AE (Auto Exposure) algorithm to adjust the exposure parameters, quickly and continuously capture multiple frames of images, such as underexposed images and overexposed images, and synthesize these images. The synthesized HDR image has no overexposure in the bright parts and clear details in the dark parts.

[0031] AE algorithm adjusts the exposure by automatically controlling exposure parameters such as aperture (lens aperture size), ISO sensitivity value and shutter speed.

[0032] Long frame images and short frame images, in the embodiments of the present invention, for the convenience of description, images with long exposure time are called long frame images, and images with short exposure time are called short frame images. It should be noted that the length of the exposure time here is relative, and the specific exposure time value can be determined according to actual use requirements, and the embodiments of the present invention do not limit it.

[0033] In an embodiment of the present invention, the image acquisition device also includes a camera and a video codec. The image acquisition device can implement a shooting function through a camera, an image sensor, an ISP, a video codec, etc. to shoot HDR images (including shooting static HDR images or HDR videos).

[0034] Specifically, during the shooting process, the image acquisition device usually uses an image sensor to output two frames of images each time, including a long frame image and a short frame image, and synthesizes the two frames of images into one frame of image to improve the dynamic range of the image and obtain an HDR image.

[0035] In order to reduce the flicker phenomenon in the HDR image output by the image acquisition device, a neural network model can be used to identify flicker in the RGB image, and then the parameters of the image processing unit (hereinafter referred to as ISP parameters) can be adjusted according to the flicker identification results to eliminate the flicker phenomenon. However, flicker identification based on RGB images and then adjusting the ISP parameters according to the flicker identification results will have the following problems:

[0036] Reference Figure 1 , shows a schematic diagram of a process of flicker recognition based on RGB images and adjusting ISP parameters in the present invention. Figure 1 As shown, the RGB image is generated by the RAW image after a series of ISP processes are processed. Exemplarily, the ISP process includes the following operations in sequence: tone mapping, gamma compression, color correction, white balance, and demosaicing. Specifically, the image sensor acquires and outputs the RAW image of the Tth frame, and the image processing unit performs a series of ISP processes on the RAW image of the Tth frame and outputs the RGB image. After flicker recognition is performed on the RGB image, the ISP parameters are adjusted according to the flicker recognition results to obtain the adjusted ISP parameters. Assume that there is flicker in the RGB image of the Tth frame. Next, the image sensor acquires and outputs the RAW image of the T+1th frame, and the image processing unit performs a series of ISP processes on the RAW image of the T+1th frame using the adjusted ISP parameters and outputs the RGB image. At this time, the RGB image of the T+1th frame can eliminate the flicker phenomenon. It can be seen that flicker recognition based on the RGB image and then adjusting the ISP parameters will result in the effect of the parameter adjustment only taking effect in the next frame (such as the T+1 frame), and the effect of the ISP parameter adjustment will be delayed by one frame, thereby affecting the quality of the output HDR image.

[0037] In view of this, an embodiment of the present invention proposes an image processing method, which performs flicker recognition based on the RAW image output by the image sensor, thereby making it possible to implement the flicker recognition result of the Tth frame to take effect on the ISP parameter adjustment of the Tth frame. Specifically, the image sensor acquires and outputs the RAW image of the Tth frame, and after flicker recognition is performed based on the RAW image of the Tth frame, the ISP parameters are adjusted according to the flicker recognition result to obtain the adjusted ISP parameters, and the RAW image of the Tth frame is subjected to a series of ISP processing using the adjusted ISP parameters to output an RGB image, so that the RGB image of the Tth frame can eliminate the flicker phenomenon. In this way, it is possible to avoid the situation where the effect of the ISP parameter adjustment is delayed by one frame, and the timeliness of the flicker recognition processing can be improved, thereby improving the quality of the output HDR video image.

[0038] Reference Figure 2 , shows a flowchart of the steps of an image processing method embodiment of the present invention, the method can be applied to an image acquisition device, the image acquisition device includes an image sensor and an image processing unit, the method may include the following steps:

[0039] Step 101, acquiring a long frame image and a short frame image output by the image sensor; the long frame image and the short frame image are RAW images;

[0040] Step 102: input the short frame image into a trained flicker recognition model, and output a flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on the training data of the RAW image;

[0041] Step 103: Adjust parameters of the image processing unit according to the flicker recognition result.

[0042] The image processing method provided in the embodiment of the present invention can be applied to an image acquisition device, which can be an electronic device with an image acquisition function. The electronic device includes, but is not limited to, earphones, voice recorders, home smart terminals (including air conditioners, refrigerators, rice cookers, water heaters, etc.), business smart terminals (including videophones, conference desktop smart terminals, etc.), wearable devices (including smart watches, smart glasses, etc.), financial smart terminals, as well as smart phones, tablet computers, personal digital assistants (PDAs), vehicle-mounted devices, servers, etc.

[0043] The image processing method provided by the embodiment of the present invention can identify whether a flicker phenomenon exists based on a RAW image output by an image sensor when an image acquisition device acquires a high dynamic range (HDR) image, and adjust ISP parameters according to the flicker identification result, so that the parameter adjustment result can take effect in real time, thereby improving the quality of the HDR video image output by the image acquisition device.

[0044] The process of capturing HDR video by an image acquisition device can be as follows: a single camera is used to expose the same scene multiple times with different exposure amounts to cover the brightness range of the entire scene, and then these images with different exposure amounts are combined into an HDR image, and then multiple HDR images are combined into an HDR video.

[0045] In the embodiment of the present invention, the image sensor outputs two frames of images each time, including a long frame image and a short frame image, and the long frame image and the short frame image are RAW images. In a specific implementation, the image acquisition device can acquire images using a DOL-HDR (Digital Overlap HDR, line interleaved HDR) method, and a long exposure frame image (referred to as a long frame image) and a short exposure frame image (referred to as a short frame image) can be obtained by adjusting the exposure time.

[0046] Among them, a long exposure frame image refers to an image captured with a longer exposure time, which is more sensitive to details in the dark area of ​​the image, but may be overexposed in the bright area of ​​the image; a short exposure frame image refers to an image captured with a shorter exposure time, which is more sensitive to details in the bright area of ​​the image, but shows a loss of details in the dark area of ​​the image. Therefore, the long exposure frame image and the short exposure frame image can be combined to form an HDR image, so that the bright area is not overexposed and the dark area has details, so as to improve the image quality.

[0047] For the two frames of images (long frame image and short frame image) currently output by the image sensor, the embodiment of the present invention inputs the short frame image into the trained flicker recognition model, and outputs the flicker recognition result through the flicker recognition model.

[0048] Flicker usually occurs in indoor shooting scenes. If the exposure time is not an integer multiple of the energy cycle of the light source, the accumulated signal strength at different positions of the image is different and changes periodically, which is the case of a single-frame image. Among the two frames of images output by the image sensor, only the exposure time of the short-frame image may meet the conditions for flicker generation. Therefore, it is only necessary to input the short-frame image into the trained flicker recognition model to determine whether the flicker phenomenon occurs through the model.

[0049] The flicker recognition model is a neural network model trained based on the training data of the RAW image, which can effectively identify the flicker phenomenon in the RAW image. The flicker recognition model can be obtained by supervised training of the existing neural network based on a large amount of training data and machine learning methods. It should be noted that the embodiment of the present invention does not limit the model structure and training method of the flicker recognition model. The flicker recognition model can integrate multiple neural networks. The neural network can include but is not limited to at least one or at least two of the following combinations, superpositions, and nestings: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory) network, RNN (Simple Recurrent Neural Network), attention neural network, etc.

[0050] The parameters of the image processing unit can be adjusted according to the flicker recognition result to eliminate the flicker phenomenon. Since the embodiment of the present invention adjusts the ISP parameters after flicker recognition on the RAW image output by the image sensor, the parameter adjustment results can take effect in real time on the current RAW image, thereby improving the quality of the video image output by the image acquisition device.

[0051] In an optional embodiment of the present invention, the flicker recognition model may be trained by the following steps:

[0052] Step S11, constructing training data; the training data includes positive samples of RAW images and negative samples of RAW images;

[0053] Step S12, inputting the training data into the initial flicker recognition model in sequence, and outputting the model prediction result through the initial flicker recognition model;

[0054] Step S13, calculating the model loss according to the difference between the model prediction result and the label of the input training data;

[0055] Step S14: adjusting the model parameters of the initial flicker recognition model according to the model loss, and entering the next round of iteration, and obtaining a trained flicker recognition model when the iteration stop condition is reached.

[0056] Specifically, a large amount of training data is first constructed; the training data includes positive samples of RAW images and negative samples of RAW images. The positive samples of RAW images refer to RAW images with flickering. The negative samples of RAW images refer to RAW images without flickering. Then, the flicker recognition model is iteratively trained using the constructed large amount of training data.

[0057] In the current round of iterative training, the current round of training data is selected from the constructed training data and input into the initial flicker recognition model. The current round of training data is flicker recognized by the initial flicker recognition model, and the model prediction result is output. The model prediction result may include the presence of flicker phenomenon or the absence of flicker phenomenon. If the current round of training data is a positive sample, the corresponding label is the presence of flicker phenomenon; if the current round of training data is a negative sample, the corresponding label is the absence of flicker phenomenon. According to the difference between the model prediction result and the label of the current round of training data, the model loss is calculated. According to the model loss, the model parameters of the initial flicker recognition model can be adjusted, and the next round of iteration is entered. When the iteration stop condition is reached, a trained flicker recognition model is obtained.

[0058] The iteration stopping condition may include that the model loss is less than a preset threshold, or the number of iterative training times reaches a preset number.

[0059] It should be noted that the embodiment of the present invention does not limit the method for calculating the model loss. For example, a general loss function can be used to calculate the model loss. General loss functions include cross entropy loss function, cosine similarity loss function, etc.

[0060] In an optional embodiment of the present invention, constructing training data may include:

[0061] Step S21, obtaining a clean RGB image;

[0062] Step S22, performing a conversion operation on the clean RGB image to obtain a clean RAW image;

[0063] Step S23: divide the clean RAW image into two parts, perform a flicker simulation operation on one part of the clean RAW image to obtain a positive sample of the RAW image, and use the other part of the clean RAW image as a negative sample of the RAW image.

[0064] The training data for training the flicker recognition model can be obtained by collecting a large number of RAW images of real scenes and annotating them. However, the acquisition cost of RAW images is relatively high. At present, RAW images can be obtained by using devices that store RAW image data, including professional photographic and video equipment and development boards of sensor manufacturers, etc., while the equipment cost of the former is relatively high, and the latter requires a certain technical threshold. In addition, it is difficult to annotate RAW images. RAW images are different from RGB images. They are not conducive to the perception of the human eye in terms of contrast, color, etc., so there is a certain difficulty in annotating RAW images. As a result, it is difficult to obtain large-scale real training data when performing flicker recognition based on RAW images, and the amount of training data is very important for obtaining a deep neural network with high accuracy and good generalization. Therefore, in order to solve this problem, the embodiment of the present invention adopts a data degradation method to degrade the RGB image through an inverse ISP operation to obtain RAW image data. Therefore, the RGB image can be used to construct training data for model training, and the final model can realize flicker recognition on the RAW image.

[0065] Furthermore, due to the characteristics of the flicker phenomenon itself, it is difficult to collect data where the flicker phenomenon occurs in reality. Therefore, the embodiment of the present invention obtains a clean RGB image, and the clean RGB image refers to an RGB image without the flicker phenomenon. The clean RGB image is subjected to an inverse ISP degradation operation to convert the clean RGB image into a clean RAW image. A clean RAW image refers to a RAW image without the flicker phenomenon. The clean RAW image is divided into two parts, and a flicker simulation operation is performed on one part of the clean RAW image to obtain a positive sample of the RAW image, and the other part of the clean RAW image is used as a negative sample of the RAW image, thereby obtaining a large amount of training data of RAW images containing positive samples and negative samples. The embodiment of the present invention generates a RAW image (positive sample) with flicker by simulating the flicker phenomenon on a clean RAW image. Compared with collecting data where flicker actually occurs, the operation difficulty and operation cost can be greatly reduced, so that a large amount of training data can be constructed more efficiently.

[0066] In a specific implementation, the flicker recognition model obtained by training the training data constructed by simulation of the present invention can achieve a better recognition effect in a real scene. In order to further improve the accuracy of the flicker recognition model, in the process of training the model, a strategy of mixed training of simulated training data and real collected training data can be adopted. That is, adding a small part of real collected training data to a large amount of simulated training data for joint training can further improve the accuracy of the flicker recognition model obtained by training for flicker recognition of RAW images in real scenes.

[0067] In an optional embodiment of the present invention, the converting operation on the clean RGB image to obtain a clean RAW image may include:

[0068] The clean RGB image is input into the first preset module among the plurality of preset modules connected in sequence, each preset module among the plurality of preset modules connected in sequence performs inverse processing on the received input information, and the last preset module among the plurality of preset modules connected in sequence outputs a clean RAW image; wherein the input information of the first preset module is the clean RGB image, and the input information of the other preset modules except the first preset module is the output information of the previous preset module.

[0069] The embodiment of the present invention implements the inverse process of the ISP process by connecting a plurality of preset modules in sequence. The first preset module receives a clean RGB image, and after the clean RGB image is processed by each preset module in sequence, a clean RAW image is outputted by the last preset module.

[0070] In an optional embodiment of the present invention, the plurality of preset modules connected in sequence may include in sequence: a tone mapping inverse processing module, a gamma compression inverse processing module, a color correction inverse processing module, a white balance inverse processing module, and a demosaicing inverse processing module.

[0071] Reference Figure 3 , which shows a schematic diagram of the present invention processing a clean RGB image through preset modules in sequence to output a clean RAW image. Figure 3 Described, the clean RGB image is input into the first preset module (tone mapping inverse processing module), the clean RGB image is processed by the tone mapping inverse processing module, the processing result of the tone mapping inverse processing module is output to the gamma compression inverse processing module for processing, the processing result of the gamma compression inverse processing module is output to the color correction inverse processing module for processing, the processing result of the color correction inverse processing module is output to the white balance inverse processing module for processing, the processing result of the white balance inverse processing module is output to the demosaicing inverse processing module for processing, and the processing result output by the demosaicing inverse processing module is the clean RAW image.

[0072] The tone mapping inverse processing module is used to simulate the inverse process of tone mapping. The tone mapping process refers to the process of mapping the luminous intensity of the display device to the signal intensity output by the camera sensor. In the embodiment of the present invention, it is assumed that the tone mapping process used in the ISP process is as follows:

[0073] S(x)=3x 2 -2x 3 (1)

[0074] Where x is the input of tone mapping and S(x) is the output of tone mapping; the inverse process of tone mapping can be expressed as:

[0075]

[0076] Among them, y is the input of the tone mapping inverse processing module, S -1 (y) is the output of the tone mapping inverse processing module.

[0077] The gamma compression inverse processing module is used to simulate the inverse process of gamma compression. Because human vision is sensitive to contrast changes in dark areas, the output of tone mapping generally undergoes a process of gamma compression to enhance dark area pixels. In the embodiment of the present invention, it is assumed that the gamma compression process used in the ISP process is as follows:

[0078]

[0079] Among them, x is the input of gamma compression, ∈ is to prevent abnormal minimum values ​​such as 1e-8, and γ(x) is the output of gamma compression; the inverse process of gamma compression can be expressed as:

[0080] γ -1 (y)=max(y,∈) 2.2 (4)

[0081] Among them, y is the input of the gamma compression inverse processing module, γ -1 (y) is the output of the gamma compression inverse processing module.

[0082] The color correction inverse processing module is used to simulate the inverse process of color correction. The color correction process generally converts the image from the raw-RGB space to the sRGB color space through a 3×3 color correction matrix. The inverse process of color correction adopted by the embodiment of the present invention can be: inverting the color correction matrix and acting on the sRGB space image, so as to return the sRGB space image to the raw-RGB space.

[0083] The white balance inverse processing module is used to simulate the inverse process of white balance. White balance is to restore white objects in different ambient color temperatures to true white, that is, to multiply the three channels of R, G, and B by different gain values. The inverse process of white balance adopted by the embodiment of the present invention can be to divide the three channels of R, G, and B by different gain values.

[0084] The de-mosaicing inverse processing module is used to simulate the inverse process of de-mosaicing. Since most image sensors are Bayer (Bayer arrangement) mode, the light must pass through the Bayer filter for Bayer arrangement before entering the image sensor to obtain a RAW domain image, and then the Bayer arrangement image is converted into an RGB image through an interpolation algorithm. The inverse process of the Bayer arrangement adopted in the embodiment of the present invention refers to converting the RGB image into a RAW domain image of the Bayer arrangement through inverse interpolation.

[0085] In an optional embodiment of the present invention, a flicker simulation operation can be performed on a clean RAW image by the following steps: a linear transformation is performed on the clean RAW image using a brightness adjustment weight to obtain a positive sample of the RAW image; wherein the brightness adjustment weight changes periodically according to the number of rows of the clean RAW image. Based on the principle of the Flicker phenomenon, the Flicker phenomenon can be understood as a periodic change in the brightness of different rows in an image, and this process usually conforms to the change law of the sin function.

[0086] Exemplarily, the clean RAW image can be linearly transformed as follows using the brightness adjustment weight: y=αx, wherein x is a clean RAW image without the Flicker phenomenon, for example, x can be obtained by performing inverse ISP processing on a clean RGB image using the method of an embodiment of the present invention, α is a brightness adjustment weight, and the brightness adjustment weight changes periodically according to the number of rows of the clean RAW image; y is a simulated RAW image with the Flicker phenomenon.

[0087] Among them, α changes periodically according to the number of rows of the clean RAW image, that is, α of each row is consistent, and α of different rows changes periodically. In an example, α can be calculated by the following formula:

[0088]

[0089] Among them, fps represents the random image frame rate, freq represents the random light source frequency, h_index represents the current number of rows of the current clean RAW image, and h_max represents the maximum number of rows of the current clean RAW image.

[0090] The image frame rate means that a video file is composed of multiple consecutive frames of images. The image frame rate can be understood as the number of frames of images in the video screen recorded or played per second. The unit of the image frame rate is fps (frames per second), that is, the number of frames output per second. The higher the image frame rate, the smoother the video playback. The light source frequency refers to the frequency of general alternating current, such as 50Hz or 60Hz.

[0091] In order to make the flicker recognition model obtained by training applicable to RAW images collected at different image frame rates and different light source frequencies, the embodiment of the present invention adds two random parameters, namely, random image frame rate and random light source frequency, when generating positive samples of RAW images. The present invention does not limit the values ​​of the random image frame rate and the random light source frequency. For example, the random image frame rate fps can be randomly selected from 10, 25, 30, 60, etc., and the random light source frequency freq can be randomly selected from 50 and 60.

[0092] The embodiment of the present invention generates a RAW image with flicker (positive sample) by simulating the flicker phenomenon on a clean RAW image. Compared with collecting data of actual flicker, the operation difficulty and cost can be greatly reduced, thereby constructing a large amount of training data more efficiently.

[0093] In an optional embodiment of the present invention, the method may further include: performing high-dynamic synthesis on the long-frame image and the short-frame image based on the adjusted parameters of the image processing unit according to the flicker recognition result, and outputting a high-dynamic image.

[0094] Reference Figure 4 , shows a schematic diagram of a process of performing high dynamic synthesis based on flicker recognition results of the present invention. Figure 4 As shown, in HDR mode, light is transmitted to the image sensor through the camera, and the image sensor outputs a long frame image and a short frame image; the short frame image is input into the trained flicker recognition model, and the flicker recognition result is output through the flicker recognition model; the ISP parameters are adjusted according to the flicker recognition result to obtain the adjusted ISP parameters; and according to the flicker recognition result, the long frame image and the short frame image are synthesized in high dynamic according to the adjusted ISP parameters to output a high dynamic image. Specifically, if the flicker recognition result is that flicker phenomenon exists, the flicker elimination processing operation is performed in the high dynamic synthesis process; if the flicker recognition result is that flicker phenomenon does not exist, the flicker elimination processing operation is not performed in the high dynamic synthesis process.

[0095] The embodiment of the present invention can realize flicker recognition based on RAW images, and further can assist in adjusting the ISP parameters of the current frame to improve the real-time performance of image quality. The adjustment of the ISP parameters is such as adjusting the relevant parameters in the HDR synthesis algorithm.

[0096] In an embodiment of the present invention, the image acquisition device may include a high dynamic range synthesis module (HDR module for short). Figure 4 As shown, the flicker recognition result output by the flicker recognition model as well as the long frame image and the short frame image can be input into the HDR module together. The HDR module determines the HDR synthesis strategy based on the flicker recognition result and the current ISP parameters. The HDR synthesis strategy includes an adjustment method for relevant parameters in the HDR synthesis algorithm. According to the determined HDR synthesis strategy, an HDR synthesis operation is performed on the received long frame image and short frame image to obtain an HDR image. The embodiment of the present invention does not limit the HDR synthesis strategy adopted by the HDR module. By way of example, the proportion of long frame images and short frame images in HDR synthesis can be adjusted.

[0097] It should be noted that each step of the method of the embodiment of the present invention can be executed on the image acquisition device. Furthermore, the embodiment of the present invention performs flicker recognition based on the RAW image, and then performs HDR synthesis based on the flicker recognition result to output the HDR image (i.e., RGB image). In order to avoid taking too long to perform flicker recognition based on the RAW image, which affects the speed of outputting RGB and thus affects the image frame rate, the embodiment of the present invention can take optimization measures to optimize the flicker recognition process.

[0098] In order to enable the image acquisition device to still meet the preset image frame rate when executing the method of the present invention, the embodiment of the present invention may adopt one or more optimization means. Exemplarily, one optimization means may be to use low-resolution short-frame images for flicker recognition. For example, a downsampled short-frame image may be obtained, and the downsampled short-frame image may be input into a trained flicker recognition model for flicker recognition. Since the downsampled short-frame image has a lower resolution, the speed of flicker recognition can be greatly improved to reduce the processing time of flicker recognition. For example, another optimization means may be to optimize and streamline the flicker recognition model, reduce the model parameters while ensuring the accuracy of flicker recognition, thereby reducing the processing time of the flicker recognition model. It can be understood that the above-mentioned optimization means are only used as an example, and a variety of other optimization means may be used in the specific implementation to improve the efficiency of flicker recognition based on RAW images in the embodiment of the present invention, so that the speed at which the image acquisition device outputs HDR images can meet the preset image frame rate. For example, assuming the preset image frame rate is 25fps, the processing time for flicker recognition based on RAW images can be reduced through optimization means to ensure that the total processing time for each frame is around 40ms to ensure the preset image frame rate, that is, to ensure that 25 frames of HDR images are output per second, and ultimately 25fps HDR video is output, thereby achieving the purpose of not reducing the image frame rate.

[0099] It should be noted that the image processing method of the embodiment of the present invention can be applied to the scene of two-frame HDR, and the two-frame HDR refers to a frame of HDR image synthesized from two frames of non-HDR images. The two frames of non-HDR images include a frame of long-frame image and a frame of short-frame image. Furthermore, the image processing method of the embodiment of the present invention can also be applied to the scene of multi-frame HDR (here, multi-frame refers to more than two frames), and the multi-frame HDR refers to a frame of HDR image synthesized from multiple frames of non-HDR images. The multiple frames of non-HDR images include at least one frame of long-frame image and at least one frame of short-frame image.

[0100] The two-frame HDR scenario means that the image sensor (Sensor) completes one frame of exposure, and the sensor generates two frames of non-HDR images. The two frames of non-HDR images are transmitted to the image processing unit (ISP), and the ISP then synthesizes the two frames of non-HDR images into one frame of HDR image.

[0101] The three-frame HDR scenario means that the Sensor completes one frame of exposure, the Sensor generates three frames of non-HDR images, the three frames of non-HDR images are transmitted to the ISP, and the ISP then synthesizes the three frames of non-HDR images into one frame of HDR image.

[0102] It should be noted that in the multi-frame HDR scenario, any short-frame image can be selected from the multi-frame non-HDR images currently generated by the Sensor for flicker recognition, and HDR synthesis can be performed on the multi-frame non-HDR images currently generated by the Sensor based on the flicker recognition results.

[0103] It should be noted that the embodiment of the present invention is mainly described by taking a two-frame HDR scene as an example. The detection method of the embodiment of the present invention can be applied to any multi-frame HDR scene. The operation process is similar and can be referenced to each other.

[0104] In summary, the image processing method provided by the embodiment of the present invention can input the short frame image in the RAW image output by the image sensor into the trained flicker recognition model when the image acquisition device acquires a high dynamic (HDR) video image, and output the flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on the training data of the RAW image, which can effectively identify the flicker phenomenon in the RAW image; and then the parameters of the image processing unit can be adjusted according to the flicker recognition result to eliminate the flicker phenomenon in the HDR image. Since the embodiment of the present invention detects whether there is a flicker phenomenon based on the RAW image output by the image sensor, and adjusts the ISP parameters according to the flicker recognition result, the parameter adjustment result can take effect in real time in the current frame, thereby improving the quality of the HDR video image output by the image acquisition device.

[0105] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0106] Reference Figure 5 , shows a structural block diagram of an embodiment of an image processing device of the present invention, the device can be applied to an image acquisition device, the image acquisition device includes an image sensor and an image processing unit, the device includes:

[0107] An image acquisition module 201 is used to acquire a long frame image and a short frame image output by the image sensor; the long frame image and the short frame image are RAW images;

[0108] A flicker recognition module 202 is used to input the short frame image into a trained flicker recognition model, and output a flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on the training data of the RAW image;

[0109] The parameter adjustment module 203 is used to adjust the parameters of the image processing unit according to the flicker recognition result.

[0110] Optionally, the device further comprises:

[0111] A training data construction module, used to construct training data; the training data includes positive samples of RAW images and negative samples of RAW images;

[0112] A model training module is used to input the training data into an initial flicker recognition model in sequence, and output a model prediction result through the initial flicker recognition model; calculate the model loss according to the difference between the model prediction result and the label of the input training data; adjust the model parameters of the initial flicker recognition model according to the model loss, and enter the next round of iteration, and obtain a trained flicker recognition model when the iteration stop condition is reached.

[0113] Optionally, the training data construction module includes:

[0114] Image acquisition submodule, used to obtain clean RGB images;

[0115] An image conversion submodule, used for performing a conversion operation on the clean RGB image to obtain a clean RAW image;

[0116] The sample construction submodule is used to divide the clean RAW image into two parts, perform a flicker simulation operation on one part of the clean RAW image to obtain a positive sample of the RAW image, and use the other part of the clean RAW image as a negative sample of the RAW image.

[0117] Optionally, the image conversion submodule is specifically used to:

[0118] The clean RGB image is input into the first preset module among the plurality of preset modules connected in sequence, each preset module among the plurality of preset modules connected in sequence performs inverse processing on the received input information, and the last preset module among the plurality of preset modules connected in sequence outputs a clean RAW image; wherein the input information of the first preset module is the clean RGB image, and the input information of the other preset modules except the first preset module is the output information of the previous preset module.

[0119] Optionally, the plurality of preset modules connected in sequence include in sequence: a tone mapping inverse processing module, a gamma compression inverse processing module, a color correction inverse processing module, a white balance inverse processing module, and a demosaicing inverse processing module.

[0120] Optionally, the sample construction submodule is specifically used to:

[0121] The clean RAW image is linearly transformed by using a brightness adjustment weight to obtain a positive sample of the RAW image; wherein the brightness adjustment weight changes periodically according to the number of rows of the clean RAW image.

[0122] Optionally, the device further comprises:

[0123] A dynamic synthesis module is used to perform high-dynamic synthesis on the long-frame image and the short-frame image based on the adjusted parameters of the image processing unit according to the flicker recognition result, and output a high-dynamic image.

[0124] The image processing device provided by the embodiment of the present invention can input the short frame image in the RAW image output by the image sensor into the trained flicker recognition model when the image acquisition device acquires a high dynamic (HDR) video image, and output the flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on the training data of the RAW image, which can effectively identify the flicker phenomenon in the RAW image; and then the parameters of the image processing unit can be adjusted according to the flicker recognition result to eliminate the flicker phenomenon in the HDR image. Since the embodiment of the present invention detects whether there is a flicker phenomenon based on the RAW image output by the image sensor, and adjusts the ISP parameters according to the flicker recognition result, the parameter adjustment result can take effect in real time in the current frame, thereby improving the quality of the HDR video image output by the image acquisition device.

[0125] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0127] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0128] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), the device can execute the above Figure 1The description of the image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment of the present invention, please refer to the description of the method embodiment of the present invention.

[0129] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed by the present invention. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.

[0130] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0132] The above is a detailed introduction to an image processing method, device and machine-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image processing method, characterized in that: Applied to an image acquisition device, the image acquisition device includes an image sensor and an image processing unit, and the method includes: Acquire a long frame image and a short frame image output by the image sensor; the long frame image and the short frame image are RAW images; Inputting the short frame image into a trained flicker recognition model, and outputting a flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on training data of the RAW image; The parameters of the image processing unit are adjusted according to the flicker recognition result.

2. The method according to claim 1, characterized in that The method further comprises: Constructing training data; the training data includes positive samples of RAW images and negative samples of RAW images; Inputting the training data into an initial flicker recognition model in sequence, and outputting a model prediction result through the initial flicker recognition model; Calculating the model loss based on the difference between the model prediction result and the label of the input training data; The model parameters of the initial flicker recognition model are adjusted according to the model loss, and the next round of iteration is entered, and a trained flicker recognition model is obtained when an iteration stop condition is reached.

3. The method according to claim 2, characterized in that The constructing of training data comprises: Get a clean RGB image; Performing a conversion operation on the clean RGB image to obtain a clean RAW image; The clean RAW image is divided into two parts, a flicker simulation operation is performed on one part of the clean RAW image to obtain a positive sample of the RAW image, and the other part of the clean RAW image is used as a negative sample of the RAW image.

4. The method according to claim 3, characterized in that The converting operation is performed on the clean RGB image to obtain a clean RAW image, including: The clean RGB image is input into the first preset module among the plurality of preset modules connected in sequence, each preset module among the plurality of preset modules connected in sequence performs inverse processing on the received input information, and the last preset module among the plurality of preset modules connected in sequence outputs a clean RAW image; wherein the input information of the first preset module is the clean RGB image, and the input information of the other preset modules except the first preset module is the output information of the previous preset module.

5. The method according to claim 4, characterized in that The plurality of preset modules connected in sequence include: a tone mapping inverse processing module, a gamma compression inverse processing module, a color correction inverse processing module, a white balance inverse processing module, and a demosaicing inverse processing module.

6. The method according to claim 3, characterized in that To simulate flicker on a clean RAW image, follow these steps: The clean RAW image is linearly transformed by using a brightness adjustment weight to obtain a positive sample of the RAW image; wherein the brightness adjustment weight changes periodically according to the number of rows of the clean RAW image.

7. The method according to claim 1, characterized in that The method further comprises: According to the flicker recognition result, the long frame image and the short frame image are synthesized with high dynamic range based on the adjusted parameters of the image processing unit to output a high dynamic range image.

8. An image processing device, characterized in that: Applied to an image acquisition device, the image acquisition device includes an image sensor and an image processing unit, and the device includes: An image acquisition module, used to acquire a long-frame image and a short-frame image output by the image sensor; the long-frame image and the short-frame image are RAW images; A flicker recognition module, used for inputting the short frame image into a trained flicker recognition model, and outputting a flicker recognition result through the flicker recognition model; the flicker recognition model is a neural network model trained based on the training data of the RAW image; A parameter adjustment module is used to adjust the parameters of the image processing unit according to the flicker recognition result.

9. The device according to claim 8, characterized in that The device also includes: A training data construction module, used to construct training data; the training data includes positive samples of RAW images and negative samples of RAW images; A model training module is used to input the training data into an initial flicker recognition model in sequence, and output a model prediction result through the initial flicker recognition model; calculate the model loss according to the difference between the model prediction result and the label of the input training data; adjust the model parameters of the initial flicker recognition model according to the model loss, and enter the next round of iteration, and obtain a trained flicker recognition model when the iteration stop condition is reached.

10. A machine-readable storage medium having instructions stored thereon, which, when executed by one or more processors of a device, causes the device to execute the image processing method according to any one of claims 1 to 7.

Images