Image processing method and electronic equipment

Abstract

The invention relates to the field of image processing, and provides an image processing method and electronic equipment, the method is applied to the electronic equipment, a light source of a shooting environment where the electronic equipment is located is a stroboscopic light source, and the method comprises the following steps: running a camera application program of the electronic equipment; a first image is displayed, the first exposure time is shorter than the first duration, and the first image comprises stripes and / or color cast image areas; detecting a first operation; in response to the first operation, obtaining a second image, the second exposure time being an integral multiple of the first duration; obtaining a color conversion matrix and / or a brightness parameter based on the first image and the second image; performing first image processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain a third image, the third image being an image in which stripes and / or color cast image areas are removed; and displaying or storing the third image. Based on the scheme of the invention, stripes and / or local color cast in the image can be removed when the moving object is shot, and the image quality is improved.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 15, 2022, with application number 202211123861.X and application name “Image Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of images, and in particular, to an image processing method and electronic device. Background Art

[0003] With the development of imaging technology in electronic devices, users have increasingly higher requirements for shooting functions. For example, when shooting a moving object, users generally expect the electronic device to capture an image of the moving object at the moment of motion. Because the subject is in motion, the electronic device generally needs to reduce motion blur in the image. Currently, to reduce motion blur in the image, electronic devices can usually shorten the exposure time and increase the sensitivity value. However, in shooting scenes with an electric light source (for example, 50Hz AC), shortening the exposure time will result in the exposure time not being an integer multiple of 10ms. If the exposure time is not an integer multiple of 10ms, the image captured by the electronic device will show image banding, such as alternating light and dark banding stripes or color cast in local image areas, thus affecting image quality.

[0004] Therefore, how to remove image banding and / or local color cast problems in images and improve image quality has become an urgent problem to be solved.

[0005] Summary of the Invention

[0006] The present application provides an image processing method and electronic device, which can remove image striping problems and / or local color cast problems in an image and improve image quality.

[0007] In a first aspect, an image processing method is provided, which is applied to an electronic device, wherein the light source of a shooting environment in which the electronic device is located is a stroboscopic light source, and the image processing method includes:

[0008] running a camera application of the electronic device;

[0009] displaying a first image, where the first image is an image of a photographic object captured based on a first exposure time, the photographic object is a moving object, the first exposure time is less than a first duration, and the first image includes stripes and / or color cast image areas;

[0010] detecting a first operation, wherein the first operation instructs the electronic device to take a photo or record a video;

[0011] In response to the first operation, acquiring a second image, where the second image is an image of the subject captured based on a second exposure time, where the second exposure time is an integer multiple of the first exposure time;

[0012] obtaining a color conversion matrix and / or a brightness parameter based on the first image and the second image, wherein the color conversion matrix is ​​used to perform color adjustment on the first image, and the brightness parameter is used to perform brightness adjustment on the first image;

[0013] performing a first image processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain a third image, wherein the third image is an image in which the stripes and / or the color cast image area are removed;

[0014] The third image is displayed or saved.

[0015] In an embodiment of the present application, when the light source of the shooting environment in which the electronic device is located is a stroboscopic light source, and the electronic device needs to shoot a moving object, thereby reducing the exposure time, resulting in banding stripes in the image; in an embodiment of the present application, the first image (exposure image) is subjected to color migration and brightness migration based on the second image (for example, a normally exposed image), to obtain a color conversion matrix and / or brightness parameters; based on the color conversion matrix and / or brightness parameters, the stripes and / or color cast image areas in the first image are removed; and when capturing images of the moving moment of a moving object, the stripes and / or color cast image areas in the image are removed to improve the image quality.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0017] The first image and the second image are input into an image processing model to obtain bilateral grid data; wherein the image processing model is used to perform color migration processing and brightness migration processing on the first image based on the second image, the bilateral grid data includes the color conversion matrix and / or the brightness parameter, and the size of the first image is the same as the size of the second image.

[0018] In an embodiment of the present application, a bilateral grid for migrating the color and brightness of an image frame obtained with a normal exposure time or a long exposure time to an image frame with a short exposure time can be obtained through an image processing model; in a shooting scene with a stroboscopic light source having an alternating current of 50 Hz, since the normal exposure time or the long exposure time is an integer multiple of 10 ms, there are usually no image stripes or color cast image areas in the long exposure image and the normal exposure image; a color conversion matrix and / or brightness parameters are obtained based on the bilateral grid output by the image processing model; when the color and brightness migration are performed on the first image through the image processing model, the difference in image content between the first image and the second image can be identified; therefore, the color conversion matrix and / or brightness parameters obtained through the image processing model will not introduce ghost areas when the color and / or brightness adjustment is performed on the first image, thereby improving image quality.

[0019] In combination with the first aspect, in some implementations of the first aspect, performing first image processing on the first image based on the color conversion matrix and / or brightness parameter to obtain a third image includes:

[0020] The first image is interpolated based on the color conversion matrix and / or the brightness parameter to obtain the third image.

[0021] In a possible implementation, the color conversion matrix and / or the brightness parameter may be multiplied by the matrix of the first image to obtain the third image.

[0022] In combination with the first aspect, in some implementations of the first aspect, performing interpolation processing on the first image based on the color conversion matrix and / or brightness parameter to obtain the third image includes:

[0023] performing the first image processing on the first image based on the color conversion matrix and / or the brightness parameter in a first color space to obtain a processed image;

[0024] In the first color space, a second image processing is performed on the processed image to obtain the third image, where the second image processing is a color processing algorithm in the first color space.

[0025] In an embodiment of the present application, in the process of the image signal processor processing the Raw image captured by the electronic device, when executing the algorithm of the first color space, the first image processing based on the color conversion matrix and / or brightness parameter can be performed on the first image first, and then other algorithms of the first color space can be executed; because the first image processing can obtain an image with banding stripes and / or color cast image areas removed; in other words, the first image processing can correct the color and brightness of the image; when the accuracy of color and brightness is high, executing other algorithms of the first color space can ensure that the image with banding stripes and / or color cast image areas removed is processed by other algorithms of the first color space on the basis of high accuracy of color and brightness of the image, thereby improving image quality.

[0026] In combination with the first aspect, in certain implementations of the first aspect, before acquiring the second image, it also includes: detecting the shooting scene in which the electronic device is located, detecting the moving object; and detecting the presence of the stripes and / or color cast image areas in the first image.

[0027] In combination with the first aspect, in certain implementations of the first aspect, the first duration is obtained based on the number of times the stroboscopic light source flashes on and off per second.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the first duration=1000 / number of times the stroboscopic light source switches on and off per second.

[0029] In combination with the first aspect, in certain implementations of the first aspect, the number of times the stroboscopic light source switches on and off per second is associated with the frequency of an operating voltage of the stroboscopic light source.

[0030] In one possible implementation, the frequency of the operating voltage of the stroboscopic light source is 50 Hz, that is, the stroboscopic light source switches on and off 100 times per second, and the first duration is 10 ms; at this time, the exposure time of the first image is less than 10 ms, and the exposure time of the second image is an integer multiple of 10 ms.

[0031] In one possible implementation, the frequency of the operating voltage of the stroboscopic light source is 60 Hz, that is, the stroboscopic light source switches on and off 120 times per second, and the first duration is 8.3 ms; at this time, the exposure time of the first image is less than 8.3 ms, and the exposure time of the second image is an integer multiple of 8.3 ms.

[0032] In combination with the first aspect, in some implementations of the first aspect, the image processing model is a convolutional neural network.

[0033] In one possible implementation, the image processing model is HDRnet.

[0034] In conjunction with the first aspect, in certain implementations of the first aspect, the image processing model is trained by the following method:

[0035] Acquiring sample data, the sample data including a first sample image, a second sample image, and a third sample image, wherein the second sample image includes image content, stripes, and / or color cast image areas of the first sample image, the third sample image has the same image content as the first sample image, and image quality of the third sample image is higher than that of the first sample image;

[0036] Inputting the first sample image and the second sample image into an image processing model to be trained to obtain predicted bilateral grid data;

[0037] performing interpolation processing on the second sample image based on the predicted bilateral grid data to obtain a predicted image;

[0038] The image processing model to be trained is trained based on the difference between the predicted image and the third sample image to obtain the image processing model.

[0039] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an image signal processor, and the first image is an image output by the image signal processor.

[0040] In an embodiment of the present application, the first image may be an image output by an image signal processor; since the first image is an image output by the signal processor, the image signal processor may perform denoising on the short-exposure Raw image, and therefore, although the first image includes banding stripes, the image detail information in the first image is relatively rich.

[0041] In combination with the first aspect, in some implementations of the first aspect, the second image is an image obtained by performing a third image processing on the Raw image captured by the electronic device, and the third image processing includes a color space conversion process.

[0042] It should be understood that in the embodiments of the present application, the second image is a normally exposed image; for example, the second image can be an image obtained by downsampling and color space conversion of the normally exposed Raw image; since in the embodiments of the present application, the color information and brightness information in the second image are required to perform migration processing on the first image, the requirements for the detail information in the second image are relatively low, that is, the second image does not need to be processed by the image signal processor.

[0043] In a second aspect, an electronic device is provided. The electronic device includes one or more processors and a memory. The memory is coupled to the one or more processors, and the memory is used to store computer program code. The computer program code includes computer instructions. The one or more processors invoke the computer instructions to cause the electronic device to execute:

[0044] running a camera application of the electronic device;

[0045] displaying a first image, where the first image is an image of a photographic object captured based on a first exposure time, the photographic object is a moving object, the first exposure time is less than a first duration, and the first image includes stripes and / or color cast image areas;

[0046] detecting a first operation, wherein the first operation instructs the electronic device to take a photo or record a video;

[0047] In response to the first operation, acquiring a second image, where the second image is an image of the subject captured based on a second exposure time, where the second exposure time is an integer multiple of the first exposure time;

[0048] obtaining a color conversion matrix and / or a brightness parameter based on the first image and the second image, wherein the color conversion matrix is ​​used to perform color adjustment on the first image, and the brightness parameter is used to perform brightness adjustment on the first image;

[0049] performing a first image processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain a third image, wherein the third image is an image in which the stripes and / or the color cast image area are removed;

[0050] The third image is displayed or saved.

[0051] In conjunction with the second aspect, in some implementations of the second aspect, one or more processors invoke computer instructions to cause the electronic device to execute:

[0052] The first image and the second image are input into an image processing model to obtain bilateral grid data; wherein the image processing model is used to perform color migration processing and brightness migration processing on the first image based on the second image, the bilateral grid data includes the color conversion matrix and / or the brightness parameter, and the size of the first image is the same as the size of the second image.

[0053] In conjunction with the second aspect, in some implementations of the second aspect, one or more processors invoke computer instructions to cause the electronic device to execute:

[0054] The first image is interpolated based on the color conversion matrix and / or the brightness parameter to obtain the third image.

[0055] In conjunction with the second aspect, in some implementations of the second aspect, one or more processors invoke computer instructions to cause the electronic device to execute:

[0056] performing the first image processing on the first image based on the color conversion matrix and / or the brightness parameter in a first color space to obtain a processed image;

[0057] In the first color space, a second image processing is performed on the processed image to obtain the third image, where the second image processing is a color processing algorithm in the first color space.

[0058] In combination with the second aspect, in certain implementations of the second aspect, one or more processors call computer instructions to cause the electronic device to execute: detecting the shooting scene in which the electronic device is located, detecting the moving object; and detecting the presence of the stripes and / or color cast image areas in the first image.

[0059] In combination with the second aspect, in certain implementations of the second aspect, the first duration is obtained based on the number of times the stroboscopic light source flashes on and off per second.

[0060] In combination with the second aspect, in certain implementations of the second aspect, the first duration=1000 / number of times the stroboscopic light source switches on and off per second.

[0061] In combination with the second aspect, in certain implementations of the second aspect, the number of times the stroboscopic light source switches on and off per second is associated with the frequency of an operating voltage of the stroboscopic light source.

[0062] In combination with the second aspect, in some implementations of the second aspect, the image processing model is a convolutional neural network.

[0063] In conjunction with the second aspect, in certain implementations of the second aspect, the image processing model is trained by the following method:

[0064] Acquiring sample data, the sample data including a first sample image, a second sample image, and a third sample image, wherein the second sample image includes image content, stripes, and / or color cast image areas of the first sample image, the third sample image has the same image content as the first sample image, and image quality of the third sample image is higher than that of the first sample image;

[0065] Inputting the first sample image and the second sample image into an image processing model to be trained to obtain predicted bilateral grid data;

[0066] performing interpolation processing on the second sample image based on the predicted bilateral grid data to obtain a predicted image;

[0067] The image processing model to be trained is trained based on the difference between the predicted image and the third sample image to obtain the image processing model.

[0068] In combination with the second aspect, in some implementations of the second aspect, the electronic device includes an image signal processor, and the first image is an image output by the image signal processor.

[0069] In combination with the second aspect, in some implementations of the second aspect, the second image is an image obtained by performing third image processing on the Raw image captured by the electronic device, and the third image processing includes color space conversion processing.

[0070] In a third aspect, an electronic device is provided, comprising a module / unit for executing the image processing method in the first aspect or any one of the implementations of the first aspect.

[0071] In a fourth aspect, an electronic device is provided, comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, and the one or more processors calling the computer instructions to enable the electronic device to execute the image processing method in the first aspect or any one of the implementations of the first aspect.

[0072] In a fifth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the first aspect or any one of the image processing methods in the first aspect.

[0073] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes the image processing method in the first aspect or any one of the implementations of the first aspect.

[0074] In a seventh aspect, a computer program product is provided, comprising: a computer program code, which, when executed by an electronic device, enables the electronic device to execute the image processing method in the first aspect or any one of the implementations of the first aspect.

[0075] In the case where the light source of the shooting environment in which the electronic device is located is a stroboscopic light source, and the electronic device needs to shoot a moving object, thereby reducing the exposure time, resulting in banding (for example, banding stripes and / or color cast image areas) in the image; in an embodiment of the present application, by performing color migration and brightness migration on the short exposure image based on the normal exposure image, a color conversion matrix and / or brightness parameters are obtained; based on the color conversion matrix and / or brightness parameters, the banding stripes in the short exposure image are removed; and when capturing images of the moving moment of the moving object, the banding stripes in the image are removed to improve the image quality.

[0076] In addition, in an embodiment of the present application, a color conversion matrix and / or brightness parameters are obtained based on a bilateral grid output by an image processing model; when the first image is subjected to color migration and brightness migration through the image processing model, the deviation area where the image content difference between the first image and the second image is relatively large, that is, the image processing model can identify the ghost area between the second image and the first image; therefore, the color conversion matrix and / or brightness parameters obtained through the image processing model will not introduce ghost areas when the color adjustment and / or brightness adjustment are performed on the first image, thereby improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] FIG1 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

[0078] FIG2 is a schematic diagram of a software system of an electronic device applicable to the present application;

[0079] FIG3 is a schematic diagram of an application scenario applicable to an embodiment of the present application;

[0080] FIG4 is a schematic diagram of a graphical user interface applicable to an embodiment of the present application;

[0081] FIG5 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

[0082] FIG6 is a schematic flowchart of an image signal processor processing method provided by an embodiment of the present application;

[0083] FIG7 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

[0084] FIG8 is a schematic flow chart of a training method for an image processing model provided in an embodiment of the present application;

[0085] FIG9 is a schematic diagram of a graphical user interface applicable to an embodiment of the present application;

[0086] FIG10 is a schematic diagram of a graphical user interface applicable to an embodiment of the present application;

[0087] FIG11 is a schematic diagram of a graphical user interface applicable to an embodiment of the present application;

[0088] FIG12 is a schematic diagram of a graphical user interface applicable to an embodiment of the present application;

[0089] FIG13 is a schematic diagram of a graphical user interface applicable to an embodiment of the present application;

[0090] FIG14 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0091] FIG15 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] In the embodiments of this application, the terms "first," "second," and the like are used for descriptive purposes only and should not be understood to indicate or imply relative importance or to implicitly indicate the number of the technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0093] In order to facilitate the understanding of the embodiments of the present application, the relevant concepts involved in the embodiments of the present application are first briefly described.

[0094] 1. Flicker

[0095] The energy transmitted in the AC power grid is not stable, but changes with a fixed frequency, which is generally called the power frequency; the energy change caused by the power frequency is called flicker.

[0096] 2. Image banding

[0097] In a shooting environment with a stroboscopic light source, the phenomenon of stripes formed on the image when the image sensor in the electronic device captures the flicker is called image banding, which can be generally referred to as banding.

[0098] For example, for 50HZ alternating current, that is, a charged light source that turns on and off 100 times per second; if the exposure time of the electronic device meets the integer multiples of 10ms, the exposure integration period can offset the banding; if the exposure time of the electronic device does not meet the integer multiples of 10ms, the amount of light entering when collecting the image will fluctuate according to the law of the alternating current sine wave, resulting in regular stripes in the image.

[0099] 3. Bilateral grid

[0100] A bilateral grid is essentially a data structure; for example, for a single-channel grayscale value, the bilateral grid can be a three-dimensional array obtained by combining the two-dimensional spatial domain information of the image and the one-dimensional grayscale information.

[0101] 4. Exposure time

[0102] Exposure time refers to the time from when the camera shutter opens to when it closes that light strikes the film or sensor.

[0103] 5. Neural Networks

[0104] A neural network refers to a network formed by connecting multiple single neural units together, that is, the output of one neural unit can be the input of another neural unit; the input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be an area composed of several neural units.

[0105] 6. Convolutional neural network (CNN)

[0106] A convolutional neural network is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution processing on the input signal. In a convolutional layer of a convolutional neural network, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which is composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. The convolution kernel can be initialized as a matrix of random size, and during the training process of the convolutional neural network, the convolution kernel can be learned to acquire appropriate weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the convolutional neural network, thereby reducing the risk of overfitting.

[0107] 7. HDRnet

[0108] HDRNet is a typical 3D interpolation grid; for example, the spatial domain can be divided into 16x16, and the value domain can be divided into 8 intervals.

[0109] 8. Backpropagation algorithm

[0110] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0111] The image processing method and electronic device provided in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0112] FIG1 shows a hardware system of an electronic device suitable for the present application.

[0113] The electronic device 100 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 100.

[0114] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0115] It should be noted that the structure shown in FIG1 does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in FIG1, or the electronic device 100 may include a combination of some of the components shown in FIG1, or the electronic device 100 may include sub-components of some of the components shown in FIG1. ​​The components shown in FIG1 may be implemented in hardware, software, or a combination of software and hardware.

[0116] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices. The controller may generate an operation control signal based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0117] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0118] In some embodiments, the processor 110 may include one or more interfaces. For example, the processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and a USB interface.

[0119] Exemplarily, in an embodiment of the present application, the processor 110 can be used to execute the image processing method provided by the embodiment of the present application; for example, a camera application of an electronic device; displaying a first image, the first image is an image of a photographic object captured based on a first exposure time, the photographic object is a moving object, the first exposure time is less than a first duration, and the first image includes stripes and / or color cast image areas; detecting a first operation, the first operation instructs the electronic device to shoot or record; in response to the first operation, acquiring a second image, the second image is an image of the photographic object captured based on a second exposure time, and the second exposure time is an integer multiple of the first duration; based on the first image and the second image, obtaining a color conversion matrix and / or brightness parameters, the color conversion matrix is ​​used to adjust the color of the first image, and the brightness parameters are used to adjust the brightness of the first image; performing first image processing on the first image based on the color conversion matrix and / or brightness parameters to obtain a third image, the third image is an image from which stripes and / or color cast image areas are removed; displaying or saving the third image.

[0120] 1 is merely a schematic illustration and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0121] The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0122] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0123] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0124] Display screen 194 may be used to display images or videos.

[0125] Optionally, the display screen 194 can be used to display images or videos. The display screen 194 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0126] Exemplarily, the electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0127] Exemplarily, the ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the camera to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0128] Exemplarily, the camera 193 (also referred to as a lens) is used to capture still images or videos. It can be triggered to turn on through application instructions to implement the photo function, such as capturing images of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and is finally converged on the image sensor. The imaging lens is mainly used to converge the light emitted or reflected by all objects in the photographic field of view (also referred to as the scene to be photographed, the target scene, or the scene image that the user expects to capture) to form an image; the filter is mainly used to filter out excess light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV.

[0129] Exemplarily, the digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0130] For example, a video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. This allows the electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0131] Exemplarily, the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse motion to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0132] For example, the accelerometer 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (typically the x-axis, y-axis, and z-axis). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The accelerometer 180E can also be used to identify the posture of the electronic device 100, which can be used as an input parameter for applications such as landscape / portrait switching and pedometers.

[0133] Exemplarily, the distance sensor 180F is used to measure distance. The electronic device 100 can measure distance by infrared or laser. In some embodiments, for example, in a shooting scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0134] For example, ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0135] For example, the fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0136] For example, the touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, which is also referred to as a touch screen. The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different location from the display screen 194.

[0137] The hardware system of the electronic device 100 is described in detail above. The software system of the electronic device 100 is introduced below.

[0138] FIG2 is a schematic diagram of a software system of an electronic device provided in an embodiment of the present application.

[0139] As shown in FIG. 2 , the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 , and a hardware layer 250 .

[0140] The application layer 210 may include a camera application.

[0141] Optionally, the application layer 210 may also include applications such as gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0142] The application framework layer 220 provides an application programming interface (API) and a programming framework for the applications in the application layer; the application framework layer may include some predefined functions.

[0143] For example, the application framework layer 220 may include a camera access interface, which may include camera management and camera devices. The camera management may be used to provide an access interface for managing the camera, and the camera device may be used to provide an interface for accessing the camera.

[0144] The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer can include a camera abstraction layer and other hardware device abstraction layers; the camera abstraction layer can include camera device 1, camera device 2, etc.; the camera hardware abstraction layer can be connected to the camera algorithm library, and the camera hardware abstraction layer can call the algorithms in the camera algorithm library.

[0145] Exemplarily, the camera algorithm library may include an image processing algorithm, which is used to execute the image processing method provided in the embodiment of the present application when the image processing algorithm is run.

[0146] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver.

[0147] The hardware layer 250 may include an image sensor, an image signal processor, and other hardware devices.

[0148] At present, when users shoot moving objects, they usually expect electronic devices to capture images of moving objects at the moment of movement; since the object is moving, electronic devices usually need to reduce motion blur in the image; currently, in order to reduce motion blur in the image, electronic devices can usually shorten the exposure time and increase the sensitivity value; however, in the shooting scene of a stroboscopic light source (for example, 50HZ alternating current), the exposure time will not meet the integer multiple of 10ms due to shortening the exposure time; if the exposure time does not meet the integer multiple of 10ms, image banding problems will appear in the image captured by the electronic device, that is, light and dark alternating banding stripes will appear in the image; wherein, banding can include Luma banding and Chroma banding; Luma banding refers to the banding stripes generated by shortening the exposure time; wherein, Luma banding only has light and dark changes, no color changes; Chroma banding refers to the image color cast in the local area of ​​the image, the local area in the image does not match the overall color of the image, and red, green or blue color cast appears; Luma banding The reason for banding is that for shooting scenes with a 50Hz AC stroboscopic light source, if the exposure time of the electronic device is not an integer multiple of 10ms, the AC sinusoidal wave band generated by the stroboscopic light source in the shooting environment cannot offset the exposure integration period, resulting in regular brightness stripes in the captured image. The reason for chroma banding is that when the voltage of the stroboscopic light source changes by 10%, the color temperature of the light source will change significantly (for example, around 1000K). The imaging of the image is related to the color temperature. Since the color temperature will change significantly when the voltage of the charged light source changes slightly, the color cast problem will appear in the image. Therefore, if banding occurs in the image, it will seriously affect the image quality.

[0149] In view of this, an embodiment of the present application provides an image processing method and an electronic device; in the case where the light source of the shooting environment in which the electronic device is located is a stroboscopic light source, and the electronic device needs to shoot a moving object, thereby reducing the exposure time, resulting in banding stripes and / or color cast image areas in the image; in an embodiment of the present application, the short exposure image is subjected to color migration and brightness migration based on the normal exposure image, thereby removing the banding stripes and color cast image areas in the short exposure image; ensuring that when capturing the moving moment image of the moving object, the banding stripes and color cast image areas in the image are removed, thereby improving the image quality.

[0150] FIG3 is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application.

[0151] Exemplarily, the image processing method in the embodiments of the present application can be applied to the field of photography; for example, the image processing method of the present application can be applied to photographing moving objects in a shooting environment with a stroboscopic light source (for example, an indoor shooting environment); wherein the moving object can refer to a moving user, a moving object, or an image played in a video (for example, a movie), etc.

[0152] For example, as shown in FIG3 , a subject 270 is moving in an indoor location with a stroboscopic light source, and indoor lighting equipment (e.g., lamp 260) provides illumination. The electronic device 100 runs a camera application. During the process of the electronic device 100 capturing an image including the subject 270, a preview image including the subject 270 can be displayed in real time on the display screen. When the user views the preview image on the display screen of the electronic device 100, if they want to capture an image of the subject 270 in motion, they can click a capture control on the capture interface. When the capture control on the electronic device 100 is triggered, the electronic device 100 can capture an image of the subject 270 in motion.

[0153] Optionally, FIG3 illustrates an example of a user as subject 270; the subject may also be an animal, a vehicle, a robot, etc. For example, a subject exercising in a stroboscopic light environment may be a user playing badminton, table tennis, or practicing yoga indoors. The subject in the electronic device's shooting perspective may be a single user or multiple users.

[0154] Indoor spaces also require lighting equipment, which can operate under alternating current (AC). For example, a fluorescent lamp operates by discharging the inert gas inside the tube under the action of a high-voltage current. The generated electrons reach the tube, causing the lamp to emit light. For example, if the operating voltage of a fluorescent lamp is 50Hz AC, the 50Hz AC current will cause the lamp to flicker 100 times per second.

[0155] Optionally, taking the working voltage of the fluorescent lamp as 60 Hz alternating current as an example, when the fluorescent lamp is driven by the alternating current, the 60 Hz alternating current causes the fluorescent lamp to flash 120 times in 1 second.

[0156] For example, in an embodiment of the present application, the electronic device is in a camera mode, the display screen of the electronic device may display a preview image, and when the camera control of the electronic device is triggered, the electronic device may generate a captured image (also referred to as a captured image). When the electronic device captures the motion of the subject, the stroboscopic light source in the captured scene may affect the image captured by the image sensor, which may cause banding stripes to appear in the image in the preview image stream and / or cause banding stripes to appear in the image captured by the electronic device; for example, as shown in FIG4 .

[0157] The image processing method of the embodiment of the present application can be applied to the photo mode of a camera application; through the image processing method provided by the embodiment of the present application, when an electronic device is in a shooting environment with a stroboscopic light source, when shooting a moving object, the banding stripes and color cast image areas in the image can be removed, thereby improving the color accuracy and brightness accuracy of the image and improving the image quality.

[0158] Exemplarily, when a camera application is in a preview state (e.g., photo preview), the preview image displayed by the electronic device includes banding stripes. After the shooting control of the electronic device is triggered, the image processing method provided in the embodiment of the present application can be executed to obtain an image with banding removed. The image with banding removed can refer to removing local color cast problems in the image (e.g., Chroma banding), and / or removing light and dark stripes in the image (e.g., Luma banding).

[0159] Optionally, if the electronic device has sufficient computing power, the image processing method in the embodiment of the present application can also be applied to the field of video recording, video calling or other image processing fields.

[0160] For example, video call scenarios may include but are not limited to the following scenarios:

[0161] Video calls, video conferencing applications, long and short video applications, live video applications, online video courses, portrait intelligent mirroring application scenarios, system camera recording function video recording, video surveillance, or smart cat-eye and other portrait shooting scenarios, etc.

[0162] It should be understood that the above is an example of an application scenario and does not limit the application scenario of this application.

[0163] The image processing method provided in the embodiment of the present application is described in detail below with reference to FIG. 6 to FIG. 12 .

[0164] FIG5 is a schematic flow chart of an image processing method provided in an embodiment of the present application. The method 300 may be executed by the electronic device shown in FIG1 ; the method 300 includes steps S310 to S350 , each of which is described in detail below.

[0165] It should be understood that the method shown in Figure 5 is applied to the shooting environment where the light source of the electronic device is a stroboscopic light source; since the light source of the solution of the present application is a stroboscopic light source, when the exposure time when the electronic device acquires an image does not meet the integer multiple of (1000 / number of times the light source is on and off per second) milliseconds, banding stripes will appear in the image.

[0166] Step S310: Run a camera application in the electronic device.

[0167] For example, a user can instruct the electronic device to run the camera application by clicking the icon of the "Camera" application; or, when the electronic device is in the lock screen state, the user can instruct the electronic device to run the camera application by sliding to the right on the display screen of the electronic device. Alternatively, the electronic device is in the lock screen state, and the lock screen interface includes the icon of the camera application. The user instructs the electronic device to run the camera application by clicking the icon of the camera application. Alternatively, when the electronic device is running other applications, the application has the permission to call the camera application; the user can instruct the electronic device to run the camera application by clicking the corresponding control. For example, when the electronic device is running an instant messaging application, the user can instruct the electronic device to run the camera application by selecting the control of the camera function.

[0168] It should be understood that the above is an example of the operation of running a camera application; the electronic device can also run a camera application through voice instructions or other operations; this application does not impose any limitation on this.

[0169] It should also be understood that running the camera application may refer to launching the camera application.

[0170] Step S320: Display a first image, where the first image includes stripes and / or color cast image areas.

[0171] The first image is an image of a shooting object captured based on a first exposure time, the shooting object is a moving object, and the first exposure time is less than the first duration.

[0172] Optionally, the first image may include stripes, that is, the first image includes Luma banding.

[0173] Optionally, the first image may include stripes and a color cast image area, that is, the first image includes Luma banding and Chroma banding.

[0174] It should be understood that Luma banding refers to the banding stripes caused by shortening the exposure time. Among them, Luma banding only changes in brightness and darkness, without any color change. The reason for Luma banding is that the AC sinusoidal wave band generated by the stroboscopic light source in the shooting environment cannot offset the exposure integration period, resulting in regular brightness stripes in the captured image.

[0175] Chroma banding refers to the color cast in localized areas of an image, where the localized area does not match the overall color of the image, resulting in a red, green, or blue cast. Chroma banding occurs because energy changes are accompanied by changes in color temperature, leading to color casts in the image. For example, Chroma banding occurs when the light and dark stripes in Luma banding appear different colors.

[0176] It should also be understood that stripes can refer to global banding, that is, light and dark stripes appear in the overall area of ​​the image; color cast image area can refer to local banding, that is, local color cast areas appear in the image, and the color cast areas do not match the overall color of the image; for example, red, green or blue color casts may appear.

[0177] It should also be understood that since the exposure time of the first image is less than the first duration, the exposure integration period cannot offset the banding; therefore, banding stripes may appear in the first image; in addition, when the voltage of the stroboscopic light source changes by 10%, the color temperature of the light source will change significantly (for example, around 1000K); the imaging of the image is related to the color temperature; since the color temperature will change significantly when the voltage of the charged light source changes slightly, a color cast problem will appear in the image.

[0178] For example, the first image may be the image shown in (a) of FIG10 , which includes light and dark stripes and a color-shifted image region 704 ; the image region 704 may be a color-shifted image region of red, green, blue or other colors.

[0179] Optionally, the first image may refer to a preview image in a preview interface of an electronic device; wherein the preview interface may refer to a photo taking preview interface or a video recording preview interface.

[0180] Step S330: A first operation is detected, where the first operation instructs the electronic device to take a photo or record a video.

[0181] For example, when the camera application is in a photo taking mode, the first operation may refer to an operation of clicking a photo taking control, as shown in (b) in FIG11 .

[0182] For example, when the camera application is in recording mode, the first operation may be clicking the recording control. It should be understood that the above is an example of the first operation; the electronic device may also be instructed to take a photo or record a video through voice commands or other operations; this application does not impose any limitations on this.

[0183] Optionally, the motion object may refer to a moving user, a moving object, or an image played in a video (eg, a movie).

[0184] Optionally, the electronic device includes an image signal processor, and the first image may be an image output by the image signal processor.

[0185] In an embodiment of the present application, the first image may be an image output by an image signal processor; since the first image is an image output by the signal processor, the image signal processor may perform denoising on the short-exposure Raw image, and therefore, although the first image includes banding stripes, the image detail information in the first image is relatively rich.

[0186] Step S340: In response to the first operation, acquire a second image.

[0187] The second image is an image of the object captured based on a second exposure time, and the second exposure time is an integer multiple of the first time length.

[0188] It should be understood that, since the exposure time of the second image is an integer multiple of the first time length, the exposure integration period can offset the banding; therefore, no banding stripes appear in the second image.

[0189] For example, for a 50 Hz alternating current stroboscopic light source, the exposure time of the second image may be an integer multiple of 10 ms.

[0190] It should be understood that since the exposure time of the second image is an integer multiple of the first time length, each row of pixels in the second image receives the same energy, so there will be no energy fluctuation and no color cast problem in the second image; therefore, no stripes and color cast image areas will appear in the second image, that is, there is no Luma banding and Chroma banding in the second image.

[0191] Optionally, the first duration is obtained based on the number of times the stroboscopic light source switches on and off per second.

[0192] Optionally, the number of times the stroboscopic light source switches on and off per second is associated with the frequency of an operating voltage of the stroboscopic light source.

[0193] For example, the first duration=1000 / number of times the stroboscopic light source switches on and off per second.

[0194] In one example, the frequency of the operating voltage of the stroboscopic light source is 50 Hz, that is, the stroboscopic light source switches on and off 100 times per second, and the first duration is 10 ms; at this time, the exposure time of the first image is less than 10 ms, and the exposure time of the second image is an integer multiple of 10 ms.

[0195] In one example, the frequency of the operating voltage of the stroboscopic light source is 60 Hz, that is, the stroboscopic light source lights up and darkens 120 times per second, and the first duration is 8.3 ms; at this time, the exposure time of the first image is less than 8.3 ms, and the exposure time of the second image is an integer multiple of 8.3 ms.

[0196] It should be understood that the above is an example of the first duration; this application does not impose any limitation on the size of the first duration.

[0197] Optionally, the second image is an image obtained by performing a third image processing on a Raw image captured by the electronic device, and the third image processing includes a color space conversion process.

[0198] Illustratively, the electronic device may capture a normally exposed Raw image, and perform color space conversion processing on the normally exposed Raw image to obtain a second image.

[0199] For example, the second image may be an image in an RGB color space, or an image in a YUV color space.

[0200] It should be understood that in the embodiments of the present application, the second image is a normally exposed image; for example, the second image can be an image obtained by downsampling and color space conversion of the normally exposed Raw image; since in the embodiments of the present application, the color information and brightness information in the second image are required to perform migration processing on the first image, the requirements for the detail information in the second image are relatively low, that is, the second image does not need to be processed by the image signal processor.

[0201] Optionally, before acquiring the second image, the method further includes: detecting a shooting scene in which the electronic device is located, detecting a moving object; and detecting the presence of stripes and / or color cast image areas in the first image.

[0202] Exemplarily, the electronic device may include a detection module, which may detect the photographed object; when the photographed object includes a moving object, the detection module outputs an identifier, which may indicate that the photographed scene includes a moving object.

[0203] For example, an anti-flicker sensor (Flicker Sensor) in an electronic device may be used to detect whether stripes exist; wherein the anti-flicker sensor (Flicker Sensor) may be a sensor that samples ambient light.

[0204] In an embodiment of the present application, when there is a moving object in the shooting scene and the captured image includes banding, the electronic device can be triggered to execute the image processing method provided in the embodiment of the present application, that is, the method for removing banding in the image.

[0205] It should be understood that if the shooting scene includes a moving object, the electronic device will capture an image of the moving object at the moment of movement; since the shooting object is moving, the electronic device usually needs to reduce motion blur in the image; in order to reduce motion blur in the image, the electronic device can usually shorten the exposure time and increase the sensitivity value; however, for the shooting environment of a stroboscopic light source, reducing the exposure time will cause banding to appear in the image, so the banding in the image needs to be processed.

[0206] Step S350: Obtain a color conversion matrix and / or brightness parameters based on the first image and the second image.

[0207] The color conversion matrix is ​​used to adjust the color of the first image, and the brightness parameter is used to adjust the brightness of the first image.

[0208] In the embodiment of the present application, since the second image is an image with a normal exposure time, there is no banding in the second image; the color conversion matrix and brightness parameters can be obtained by migrating the color and brightness of the second image to the first image.

[0209] Optionally, the above method further includes:

[0210] The first image and the second image are input into an image processing model to obtain bilateral grid data; wherein the image processing model is used to perform color migration processing and brightness migration processing on the first image based on the second image, the bilateral grid data includes a color conversion matrix and / or brightness parameters, and the size of the first image is the same as the size of the second image.

[0211] Exemplarily, the image processing model is a convolutional neural network; for example, the image processing model may be HDRnet. Optionally, the training method of the image processing model can refer to the relevant description shown in the subsequent FIG8.

[0212] In an embodiment of the present application, a color conversion matrix and / or brightness parameters are obtained based on a bilateral grid output by an image processing model; when the first image is subjected to color migration and brightness migration through the image processing model, the deviation area where the image content difference between the first image and the second image is relatively large, that is, the image processing model can identify the ghost area between the second image and the first image; therefore, the color conversion matrix and / or brightness parameters obtained through the image processing model will not introduce ghost areas when the color adjustment and / or brightness adjustment is performed on the first image, thereby improving the image quality.

[0213] Optionally, the first image and the second image may be firstly subjected to registration processing and smoothing processing; and pixel-by-pixel difference may be performed on the first image and the second image after the registration processing to obtain a color conversion matrix and / or brightness parameters.

[0214] Step S360: Perform first image processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain a third image.

[0215] The third image is an image with stripes and / or color cast image areas removed.

[0216] Exemplarily, the color of the first image can be adjusted based on the color conversion matrix to remove the color cast image area in the first image; the brightness of the first image can be adjusted based on the brightness parameter to remove the banding stripes in the first image; for example, as shown in (c) in Figure 11, the image area 704 in the image can be removed based on the color conversion matrix; and the light and dark stripes in the image can be removed based on the brightness parameter.

[0217] For example, the first image may be as shown in (d) of FIG. 11 , and the third image may be as shown in FIG. 12 .

[0218] Optionally, performing first image processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain a third image includes:

[0219] The first image is interpolated based on the color conversion matrix and / or the brightness parameter to obtain a third image.

[0220] Exemplarily, the color conversion matrix and / or the brightness parameter may be multiplied by the matrix of the first image to obtain the third image.

[0221] Optionally, performing interpolation processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain the third image includes:

[0222] performing first image processing on the first image based on a color conversion matrix and / or a brightness parameter in a first color space to obtain a processed image;

[0223] In the first color space, a second image processing is performed on the processed image to obtain a third image, where the second image processing is a color processing algorithm in the first color space.

[0224] Exemplarily, the first color space may be an RGB color space; that is, the first image processing may be performed in the RGB color space.

[0225] For example, in the RGB color space, the first image processing may be performed first, and then other algorithms related to color processing in the RGB color space may be performed.

[0226] In an embodiment of the present application, in the process of the image signal processor processing the Raw image captured by the electronic device, when executing the RGB domain algorithm, the first image processing based on the color conversion matrix and / or brightness parameter can be performed on the first image before executing other RGB domain algorithms; because the first image processing can obtain an image with banding stripes and / or color cast image areas removed; in other words, the first image processing can correct the color and brightness of the image; when the accuracy of color and brightness is high, executing other RGB domain algorithms can ensure that the image with banding removed is processed by RGB domain algorithms on the basis of high accuracy of color and brightness of the image, thereby improving image quality.

[0227] For example, in the RGB color space, the first image processing may be performed first, and then other algorithms in other RGB color spaces may be performed.

[0228] Exemplarily, the first color space may be a YUV color space; that is, the first image processing may be performed in the YUV color space.

[0229] Optionally, the image processing model is trained by:

[0230] Acquiring sample data, where the sample data includes a first sample image, a second sample image, and a third sample image, wherein the second sample image includes image content, stripes, and / or color cast image areas of the first sample image, the third sample image has the same image content as the first sample image, and image quality of the third sample image is higher than that of the first sample image;

[0231] Inputting the first sample image and the second sample image into the image processing model to be trained to obtain predicted bilateral grid data;

[0232] interpolating the second sample image based on the predicted bilateral grid data to obtain a predicted image;

[0233] The image processing model to be trained is trained based on the difference between the predicted image and the third sample image to obtain an image processing model.

[0234] Step S370: Display or save the third image.

[0235] For example, when the electronic device is running another application, the application has permission to call the camera application; when the other application calls the camera application to acquire an image, the third image can be displayed. For example, when the electronic device is running the camera application in shooting mode (or video recording mode), after the electronic device detects an operation indicating a photo capture (or an operation indicating a video capture), the third image can be saved.

[0236] In an embodiment of the present application, when the light source of the shooting environment in which the electronic device is located is a stroboscopic light source, and the electronic device needs to shoot a moving object, thereby reducing the exposure time, resulting in banding stripes and / or color cast image areas in the image; in an embodiment of the present application, by performing color migration and brightness migration on the short exposure image based on the normal exposure image, a color conversion matrix and / or brightness parameters are obtained; based on the color conversion matrix, the color cast problem of the local image area in the short exposure image can be removed; based on the brightness parameters, the banding stripes in the short exposure image can be removed; ensure that when capturing the moving moment image of the moving object, the banding in the image is removed (for example, the banding stripes and color cast image areas are removed), thereby improving the image quality.

[0237] In addition, in an embodiment of the present application, a color conversion matrix and / or brightness parameters are obtained based on the bilateral grid output by the image processing model; when the color migration and brightness migration are performed on the first image through the image processing model, the difference in image content between the first image and the second image can be identified; therefore, the color conversion matrix and / or brightness parameters obtained by the image processing model will not introduce ghost areas when the color adjustment and / or brightness adjustment are performed on the first image, thereby improving the image quality.

[0238] Figure 6 is a schematic flow chart of a processing flow of an image signal processor provided by an embodiment of the present application. The method 400 includes steps S410 to S460, which are described in detail below.

[0239] Step S410: The image sensor collects a Raw image.

[0240] It should be understood that a Raw image refers to an image in a Raw color space; a Raw image may refer to original image data output from an image sensor without undergoing interpolation or color mixing.

[0241] Step S420: Perform a first color space conversion process on the Raw image.

[0242] Exemplarily, the Raw image may be converted into an RGB color space to obtain an RGB image.

[0243] Step S430: RGB domain algorithm processing.

[0244] Optionally, the image processing method provided in the embodiment of the present application, i.e., image strip removal processing (for example, banding removal processing in an image), can be the first algorithm executed in the RGB domain algorithm processing; for example, the RGB domain algorithm processing of the RGB image in step S430 includes: first performing image strip removal processing on the RGB image to obtain an RGB image with banding removed; and then performing other RGB domain algorithm processing on the RGB image with banding removed.

[0245] Optionally, the image processing method provided in the embodiment of the present application, that is, the image stripe removal process, can be the first algorithm executed among the color-related algorithms in the RGB domain algorithm processing.

[0246] In an embodiment of the present application, during the process of the image signal processor processing the Raw image captured by the electronic device, when executing the RGB domain algorithm, image banding removal processing can be performed first, so that the color and brightness accuracy of the image is higher; when the color and brightness accuracy is high, executing other RGB domain algorithms can ensure that the image with image banding removed is processed by the RGB domain algorithm on the basis of the high accuracy of the color and brightness of the image, thereby improving the image quality.

[0247] For example, the RGB domain algorithm processing performed on the RGB image in step S430 includes: color-related algorithm processing and other RGB domain algorithm processing; when performing the color-related algorithm processing, first performing image striping processing to obtain an RGB image without banding; and then performing other color-related algorithm processing on the RGB image without banding.

[0248] In an embodiment of the present application, before executing color-related algorithm processing, the image processing method provided in the embodiment of the present application can be executed first, that is, image stripping removal processing can be executed; by removing image stripping processing, an image with banding removed can be obtained, so that the accuracy of the color and brightness of the image is higher; when the accuracy of color and brightness is high, color-related algorithm processing is executed again, so as to ensure that other color-related algorithm processing is performed on the image with banding removed on the basis of the high accuracy of the color and brightness of the image, thereby improving the image quality.

[0249] Optionally, the image processing method provided in the embodiment of the present application, i.e., image strip removal processing, may not be limited to the execution time in the RGB domain algorithm.

[0250] Optionally, the implementation of the image strip removal process may refer to steps S510 to S570 shown in FIG. 7 .

[0251] Step S440: Perform a second color space conversion process on the processed RGB image.

[0252] Exemplarily, the second color space may be a YUV color space; the processed RGB image may be converted to a YUV color space; and YUV domain arithmetic processing may be performed.

[0253] Step S450: YUV domain algorithm processing.

[0254] Exemplarily, YUV domain algorithm processing includes but is not limited to:

[0255] Brightness noise reduction processing, edge enhancement processing, contrast processing, etc.

[0256] Step S460: output the processed image.

[0257] It should be understood that the above steps S410 to S460 are illustrated by taking the RGB domain algorithm processing including image strip removal processing as an example; optionally, image strip removal processing can also be performed in the YUV domain; in other words, image strip removal processing can also be performed in step S450.

[0258] In an embodiment of the present application, by adding image striping removal processing to the RGB domain algorithm processing; it is possible to remove banding in a short-exposure image when the light source of the shooting environment in which the electronic device is located is a stroboscopic light source, and the electronic device needs to shoot a moving object, thereby reducing the exposure time and causing banding in the image; ensure that when capturing the moving moment image of the moving object, the banding stripes and color cast image areas in the image are removed, thereby improving the image quality.

[0259] FIG7 is a schematic flow chart of an image processing method provided in an embodiment of the present application. The method 500 may be executed by the electronic device shown in FIG1 ; the method 500 includes steps S501 to S508 , which are described in detail below.

[0260] It should be noted that, in the embodiment of the present application, the light source of the shooting environment where the electronic device is located is a stroboscopic light source.

[0261] It should be understood that the method 500 shown in FIG. 7 may refer to a related algorithm for image strip removal processing shown in FIG. 6 .

[0262] Optionally, the method 500 shown in FIG. 7 may be the first algorithm executed in the RGB domain algorithm processing shown in FIG. 6 .

[0263] In an embodiment of the present application, in the process of the image signal processor processing the Raw image captured by the electronic device, when executing the RGB domain algorithm, the method shown in Figure 7 can be executed first and then other RGB domain algorithms can be executed; because the image processing method shown in Figure 7 can obtain an image with banding removed; in other words, the method shown in Figure 7 can correct the color and brightness of the image; when the accuracy of color and brightness is high, executing other RGB domain algorithms can ensure that the image with banding removed is processed by RGB domain algorithms on the basis of high accuracy of color and brightness of the image, thereby improving image quality.

[0264] Optionally, the method 500 shown in FIG. 7 may be executed before all color-related algorithms in the RGB domain algorithm processing shown in FIG. 6 .

[0265] In an embodiment of the present application, before executing a color-related algorithm, the method 500 shown in FIG. 7 may be executed first; through the image processing method shown in FIG. 7 , an image with banding removed may be output; in other words, the method shown in FIG. 7 may correct the color and brightness of an image; when the color and brightness accuracy are high, executing other color-related algorithms may ensure that other color processing is performed on the image with banding removed on the basis of the high accuracy of the color and brightness of the image, thereby improving the image quality.

[0266] Optionally, the method 500 shown in FIG. 7 may not be limited to being executed in the RGB domain algorithm shown in FIG. 6 .

[0267] Optionally, the method 500 shown in FIG. 7 may be executed in the YUV domain algorithm shown in FIG. 6 ; this application does not impose any limitation on this.

[0268] Step S501: Run the camera application.

[0269] For example, a user can instruct the electronic device to run the camera application by clicking the icon of the "Camera" application; or, when the electronic device is in the lock screen state, the user can instruct the electronic device to run the camera application by sliding to the right on the display screen of the electronic device. Alternatively, the electronic device is in the lock screen state, and the lock screen interface includes the icon of the camera application. The user instructs the electronic device to run the camera application by clicking the icon of the camera application. Alternatively, when the electronic device is running other applications, the application has the permission to call the camera application; the user can instruct the electronic device to run the camera application by clicking the corresponding control. For example, when the electronic device is running an instant messaging application, the user can instruct the electronic device to run the camera application by selecting the control of the camera function.

[0270] It should be understood that the above is an example of the operation of running a camera application; the electronic device can also run a camera application through voice instructions or other operations; this application does not impose any limitation on this.

[0271] It should also be understood that running the camera application may refer to launching the camera application.

[0272] Step S502: Detect that the photographed objects include moving objects.

[0273] Exemplarily, the electronic device may include a detection module, which may detect the photographed object; when the photographed object includes a moving object, the detection module outputs an identifier, which may indicate that the photographed scene includes a moving object.

[0274] Optionally, the motion object may refer to a moving user, a moving object, or an image played in a video.

[0275] Step S503: Detect the presence of image banding.

[0276] It should be understood that image banding may include Luma banding and Chroma banding; wherein Luma banding refers to banding stripes generated by shortening the exposure time; wherein Luma banding only has light and dark changes, no color changes; for example, the stripes shown in (a) in Figure 10; Chroma banding refers to the image color cast in a local area in the image, and the local area in the image does not match the overall color of the image; for example, the image color cast may include the appearance of red, green or blue color cast, etc.; for example, the image area 704 shown in (a) in Figure 10.

[0277] For example, an anti-flicker sensor (Flicker Sensor) in an electronic device may be used to detect whether image striping exists; wherein the anti-flicker sensor (Flicker Sensor) may be a sensor that samples ambient light.

[0278] Step S504: Acquire a short exposure image (an example of the first image) after ISP processing.

[0279] Optionally, the short-exposure image output after ISP processing may refer to an RGB image obtained with reference to the short-exposure Raw image in the multiple frames of Raw images.

[0280] It should be understood that due to the short exposure time of short-exposure images, the exposure time of short-exposure images is generally not an integer multiple of 10ms; therefore, banding may be present in short-exposure images. Furthermore, since a 10% change in the stroboscopic light source voltage can significantly change the color temperature of the light source (for example, by approximately 1000K), image formation is related to color temperature. Since a small change in the stroboscopic light source voltage can significantly change the color temperature, this can lead to color casts in the image. Therefore, short-exposure images may also contain areas of color casts.

[0281] Optionally, step S503 may be performed first and then step S504; or, step S504 may be performed first and then step S503; or, step S503 and step S504 may be performed simultaneously; this application does not impose any limitation on the order of step S503 and step S504.

[0282] Optionally, a full-size short-exposure Raw image may be acquired, and an ISP process may be performed on the short-exposure Raw image to obtain a short-exposure image (eg, a 512*512 RGB image).

[0283] For example, if the maximum resolution supported by the camera in the camera module is 4096*2160, the resolution of the acquired full-size short-exposure Raw image may be 4096*2160.

[0284] Step S505: Detect a first operation of instructing to take a photo.

[0285] Optionally, as shown in (b) of FIG11 , the first operation may be an operation of clicking the photo control 705 .

[0286] It should be understood that the above is an example of the first operation of instructing to take a photo; it is also possible to instruct the electronic device to take a photo through voice instructions or other operations; this application does not impose any limitation on this.

[0287] Step S506 : In response to the first operation, acquire a normally exposed image (an example of the second image).

[0288] It should be understood that for a shooting scenario with a stroboscopic light source of 50 Hz AC power, a normal exposure image may refer to a Raw image with an exposure time that is an integer multiple of 10 ms; that is, when capturing a normal exposure image, the exposure time of the electronic device is an integer multiple of 10 ms.

[0289] It should be understood that since the exposure time of a normally exposed image is an integer multiple of 10ms, there are no banding stripes in the normally exposed image, that is, there is no Luma banding in the image. In addition, Chroma banding is caused by the fact that the energy changes, accompanied by the change in color temperature, resulting in color cast problems in the image. For example, Chroma banding appears in the image when the light and dark stripes in the Luma banding appear different colors. For example, the position of the dark stripes is reddish, and the position of the light stripes is bluish, etc. For a normally exposed image, each row of pixels receives the same energy, so there will be no energy fluctuation or color cast problems in the normally exposed image. Therefore, Luma banding and Chroma banding will not appear in the normally exposed image.

[0290] Optionally, in an embodiment of the present application, the electronic device may collect multiple frames of Raw images, including S A N B ; Among them, S A Represents A frame short exposure image, N B Represents a B-frame normally exposed image; where A is an integer greater than or equal to 1, and B is an integer greater than or equal to 1; the exposure time of a short-exposure image is less than 10 ms; the exposure time of a normally exposed image is an integer multiple of 10 ms.

[0291] Optionally, in an embodiment of the present application, the electronic device may collect multiple frames of Raw images, including S A N B L C ; Among them, S A Represents A frame short exposure image, N B Indicates the B frame normal exposure image, L C Represents C frames of long-exposure images; where A is an integer greater than or equal to 1, B is an integer greater than or equal to 1, and C is an integer greater than or equal to 0. The exposure time of a short-exposure image is less than 10 ms; the exposure time of a normal-exposure image is an integer multiple of 10 ms; and the exposure time of a long-exposure image is greater than that of a normal-exposure image.

[0292] For example, multiple frames of Raw images can refer to 7 frames of images of S4N2L; that is, Raw images of SSSSNNL, where the 4 frames S are short-exposure Raw images in the preview frames; the 2 frames N represent normal exposure images with an exposure value greater than or equal to 4 frames of short-exposure images and an exposure time that is an integer multiple of 10ms; and the 1 frame L represents a long-exposure image.

[0293] Optionally, a color space conversion process (eg, demosaicing process) may be performed on the first frame of normally exposed Raw image among the one or more frames of normally exposed Raw images collected to obtain an RGB image.

[0294] In an embodiment of the present application, a normally exposed image can be obtained based on the first normally exposed Raw image in multiple frames of images; since the time difference between the first normally exposed Raw image and the short-exposed Raw image is short; therefore, the normally exposed image is obtained based on the first normally exposed Raw image, and color migration processing and brightness migration processing are performed on the short-exposure image, which can avoid the introduction of motion ghosting to a certain extent.

[0295] Step S507: Input the short exposure image and the normal exposure image into the image processing model for processing to obtain a bilateral grid.

[0296] It should be understood that the bilateral grid refers to a data structure; in the embodiments of the present application, the bilateral grid can be a grid matrix; the grid matrix includes a color correction conversion matrix and brightness parameters (e.g., brightness level), wherein the color correction conversion matrix includes red pixel gain (R gain), green pixel gain (G gain), and blue pixel gain (B gain); based on the data in the bilateral grid, the color and brightness of the normal exposure image can be transferred to the short exposure image, thereby removing banding stripes and color cast image areas in the short exposure image.

[0297] Exemplarily, the bilateral grid can be a 32*32*8*9 grid matrix; where 32*32 can represent the width and height, 8 can represent the brightness level; and 9 can represent the color correction conversion matrix, that is, a 3*3 matrix acting on each RGB value.

[0298] Optionally, the network structure of the image processing model may be a convolutional neural network; for example, the image processing model may be HDR Net; the training method of the image processing model may refer to the relevant description shown in the subsequent FIG8 .

[0299] Step S508: Process the short-exposure image based on the bilateral grid to obtain an image (an example of a third image) from which stripes and / or local color cast areas are removed.

[0300] Optionally, color migration processing can be performed on the short exposure image based on the color conversion matrix in the bilateral grid to remove the color cast image area in the short exposure image; and / or brightness migration processing can be performed on the short exposure image based on the brightness parameters in the bilateral grid to remove the banding stripes in the short exposure image.

[0301] Optionally, an image with banding removed may be obtained by performing interpolation processing based on the data in the bilateral grid and the short exposure image.

[0302] For example, the data in the bilateral grid is multiplied with the pixel matrix of the short exposure image to obtain the image without banding.

[0303] It should be noted that the above step S506 is to obtain a normally exposed image after detecting the first operation indicating taking a photo; optionally, in one implementation, when the electronic device is in a preview state, that is, before the electronic device detects the first operation, the electronic device can use single-frame progressive HDR (Stagger HDR) technology to obtain a short exposure image and a normally exposed image.

[0304] It should be understood that stagger HDR refers to a technique that uses "long and short frames" as the output unit. That is, two exposures are taken in sequence to obtain a normal exposure image and a short exposure image.

[0305] It should be noted that, in the method 500 shown in FIG7 , the exposure time of the short exposure image is less than 10 ms, and the exposure time of the normal exposure image is an integer multiple of 10 ms. This application does not impose any limitation on this.

[0306] Optionally, in an embodiment of the present application, the exposure time of the short exposure image and the exposure time of the normal exposure image are related to the number of times the stroboscopic light source of the shooting environment in which the electronic device is located brightens and darkens per second; for example, the exposure time of the normal exposure image = an integer multiple of (1000 / the number of times the stroboscopic light source brightens and darkens per second) milliseconds.

[0307] For example, if the operating voltage frequency of the stroboscopic light source in the shooting environment is 50HZ, that is, the stroboscopic light source lights up and dims 100 times per second, then the short exposure image refers to an image with an exposure time less than 10ms; the normal exposure image refers to an image with an exposure time that is an integer multiple of 10ms.

[0308] For example, the operating voltage frequency of the stroboscopic light source in the shooting environment is 60HZ, that is, the stroboscopic light source turns on and off 120 times per second. Then the short exposure image refers to an image with an exposure time less than 8.3ms; the normal exposure image refers to an image with an exposure time that is an integer multiple of 8.3ms.

[0309] Optionally, steps S501 to S508 shown in FIG7 are illustrated by taking the removal of banding stripes and color cast image areas performed in the RGB color space as an example; the above steps of removing banding stripes and color cast image areas may also be performed in the YUV color space; if performed in the YUV color space, the short exposure image and the normal exposure image may be YUV images.

[0310] Optionally, steps S501 to S508 shown in FIG7 are described by taking a scenario where an electronic device takes a photo as an example; the method shown in FIG7 can also be applied to a scenario where an electronic device records a video.

[0311] Optionally, the above is an example of obtaining a bilateral grid based on an image processing model; in an embodiment of the present application, the first image and the second image can be first registered and smoothed; the first image and the second image after the registration processing are subtracted pixel by pixel to obtain a color conversion matrix and / or brightness parameters.

[0312] In an embodiment of the present application, when the light source of the shooting environment in which the electronic device is located is a stroboscopic light source, and the electronic device needs to shoot a moving object, thereby reducing the exposure time, resulting in banding stripes and / or color cast image areas in the image; in an embodiment of the present application, by performing color migration and brightness migration on the short exposure image based on the normal exposure image, a color conversion matrix and / or brightness parameters are obtained; based on the color conversion matrix, the color cast problem of the local image area in the short exposure image can be removed; based on the brightness parameters, the banding stripes in the short exposure image can be removed; ensure that when capturing the moving moment image of the moving object, the banding in the image is removed (for example, the banding stripes and color cast image areas are removed), thereby improving the image quality.

[0313] In addition, in an embodiment of the present application, a color conversion matrix and / or brightness parameters are obtained based on the bilateral grid output by the image processing model; when the color migration and brightness migration are performed on the first image through the image processing model, the difference in image content between the first image and the second image can be identified; therefore, the color conversion matrix and / or brightness parameters obtained by the image processing model will not introduce ghost areas when the color adjustment and / or brightness adjustment are performed on the first image, thereby improving the image quality.

[0314] FIG8 is a schematic flowchart of a method for training an image processing model provided in an embodiment of the present application. The method 600 may be performed by the electronic device shown in FIG1 ; the method 600 includes steps S610 to S650 , each of which is described in detail below.

[0315] Step S610: Obtain training data.

[0316] The training data includes a first sample image, a second sample image, and a third sample image; the first sample image is an image without banding; the second sample image is an image with banding added to the first image; the third sample image is an image without banding and the image quality of the third sample image is higher than that of the first sample image.

[0317] For example, the third sample image may be an RGB image without banding after ISP processing; the first sample image may be an RGB image obtained by performing color space conversion based on a normally exposed sample Raw image; therefore, the image quality of the third sample image is higher than that of the first sample image.

[0318] Step S620: Input the first sample image and the second sample image into the image processing model to be trained to obtain a predicted bilateral grid.

[0319] It should be noted that the image processing model is used to learn the color and brightness differences between images with and without banding in the same shooting scene; the purpose is to enable the color and brightness of the banding image to be transferred to the non-banding image through the output predicted bilateral grid.

[0320] Optionally, the network structure of the image processing model can be a VGG network, or an HDR Net.

[0321] Step S630: Perform interpolation processing on the first sample image based on the predicted bilateral grid to obtain a predicted image.

[0322] Exemplarily, the data in the bilateral grid may be multiplied with the data of the first sample image to obtain a predicted image.

[0323] Step S640: Based on the difference between the predicted image and the third sample image, update the parameters of the image processing model to obtain a trained image processing model.

[0324] Exemplarily, the difference between each pixel in the predicted image and the third sample image can be calculated, and the image processing model to be trained can be trained by the back propagation algorithm so that the loss function of the image processing model to be trained converges to obtain the trained image processing model.

[0325] Optionally, in an embodiment of the present application, it is considered that the Chroma Banding image processing model may include 8 parameters; the 8 parameters are A*3, a*1, b*1, c*3; where A represents the amplitude; a represents the frequency; b represents the initial phase; c represents the offset term; A*3 represents the amplitude corresponding to the RGB pixels respectively; c*3 represents the offset term corresponding to the RGB pixels respectively.

[0326] In an embodiment of the present application, a bilateral grid for migrating the color and brightness of an image frame obtained with a normal exposure time or a long exposure time to an image frame with a short exposure time can be obtained through an image processing model; in a shooting scene with a stroboscopic light source having an alternating current of 50 Hz, since the normal exposure time or the long exposure time is an integer multiple of 10 ms, there are usually no image stripes or color cast image areas in the long exposure image and the normal exposure image; a color conversion matrix and / or brightness parameters are obtained based on the bilateral grid output by the image processing model; when the color and brightness migration are performed on the first image through the image processing model, the difference in image content between the first image and the second image can be identified; therefore, the color conversion matrix and / or brightness parameters obtained through the image processing model will not introduce ghost areas when the color and / or brightness adjustment are performed on the first image, thereby improving image quality.

[0327] The following describes the interface schematic diagrams in the electronic device with reference to FIG9 to FIG13 .

[0328] FIG9 is a schematic diagram of an interface of an electronic device provided in an embodiment of the present application.

[0329] In an embodiment of the present application, after the electronic device runs a camera application, the preview image displayed in the preview interface displayed in the electronic device includes banding stripes and / or color cast areas; after the electronic device detects that the user clicks the control, it can execute the image processing method provided in the embodiment of the present application, that is, perform image stripe removal processing, and when the user clicks the photo control, the electronic device captures an image; the image is an image with the banding stripes and / or color cast areas removed, that is, the image output after the image stripe removal processing.

[0330] In one example, as shown in (b) of FIG9 , after the electronic device detects the operation of clicking the control 703 , it executes the image processing method provided in the embodiment of the present application.

[0331] For example, as shown in FIG9 , the graphical user interface (GUI) shown in FIG9 (a) is a desktop 701 of an electronic device; the electronic device detects that a user clicks a control 702 of a camera application on the desktop 701, as shown in FIG9 (b); after the electronic device detects that the user clicks the control 702 of the camera application on the desktop 701, the electronic device runs the camera application; for example, as shown in FIG10 (a), the electronic device may display a photo preview interface; the photo preview interface includes a preview image and a control 703, wherein the preview image includes light and dark stripes and an image area 704; the image area 704 may be red, green, blue, or other colors; the electronic device detects that the user clicks the control 703, as shown in FIG10 (b); After the electronic device detects that the user clicks on the control 703, the electronic device can execute the image processing method provided in the embodiment of the present application and display a preview interface as shown in (a) of Figure 11; the preview interface includes a photo control 705; the electronic device detects that the user clicks on the photo control 705, as shown in (b) of Figure 11; after the electronic device detects that the user clicks on the photo control 705, it displays a display interface as shown in (c) of Figure 11, and the display interface includes an album control 706; the electronic device detects that the user clicks on the album control 706, as shown in (d) of Figure 11; after the electronic device detects that the user clicks on the album control 706, it displays an interface as shown in Figure 12.

[0332] In one example, as shown in (d) in FIG13 , after the electronic device detects the operation of clicking the control 709 , it executes the image processing method provided in the embodiment of the present application.

[0333] Exemplarily, after a camera application is run in an electronic device, a preview interface as shown in (a) in FIG13 may be displayed; the preview interface includes a preview image and a control 707, wherein the preview image includes light and dark stripes and an image area 704, and the image area 704 may be red, green, blue or other colors; the electronic device detects that a user clicks on the control 707, as shown in (b) in FIG13; after the electronic device detects that the user clicks on the control 707, a settings interface is displayed, as shown in (c) in FIG13; the settings interface includes a control 709 for removing image strips; the electronic device detects that a user clicks on the control 709 for removing image strips, as shown in (d) in FIG13; after the electronic device detects that the user clicks on the control 709 for removing image strips, the image processing method provided in an embodiment of the present application is executed.

[0334] It should be noted that the above is an example description of the display interface in an electronic device, and this application does not impose any limitation on this.

[0335] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or variations based on the above examples, and such modifications or variations also fall within the scope of the embodiments of the present application.

[0336] The image processing method provided by the embodiments of the present application is described in detail above with reference to Figures 1 to 13 ; the device embodiment of the present application will be described in detail below with reference to Figures 14 and 15 . It should be understood that the device in the embodiments of the present application can perform the various methods of the aforementioned embodiments of the present application. That is, the specific working processes of the various products below can refer to the corresponding processes in the aforementioned method embodiments.

[0337] FIG14 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 800 includes a processing module 810 and a display module 820 .

[0338] It should be noted that the light source of the shooting environment where the electronic device is located is a stroboscopic light source.

[0339] Among them, the processing module 810 is used to run the camera application of the electronic device; the display module 820 is used to display a first image, which is an image of the shooting object captured based on a first exposure time, the shooting object is a moving object, the first exposure time is less than a first duration, and the first image includes stripes and / or color cast image areas; the processing module 810 is also used to detect a first operation, the first operation instructs the electronic device to shoot or record; in response to the first operation, a second image is acquired, the second image is an image of the shooting object captured based on a second exposure time, and the second exposure time is an integer multiple of the first duration; based on the first image and the second image, a color conversion matrix and / or brightness parameters are obtained, the color conversion matrix is ​​used to adjust the color of the first image, and the brightness parameters are used to adjust the brightness of the first image; based on the color conversion matrix and / or brightness parameters, the first image is subjected to first image processing to obtain a third image, the third image is an image from which the stripes and / or color cast image areas are removed; and the third image is displayed or saved.

[0340] Optionally, as an embodiment, the processing module 810 is further configured to:

[0341] The first image and the second image are input into an image processing model to obtain bilateral grid data; wherein the image processing model is used to perform color migration processing and brightness migration processing on the first image based on the second image, the bilateral grid data includes the color conversion matrix and / or the brightness parameter, and the size of the first image is the same as the size of the second image.

[0342] Optionally, as an embodiment, the processing module 810 is specifically configured to:

[0343] The first image is interpolated based on the color conversion matrix and / or the brightness parameter to obtain the third image.

[0344] Optionally, as an embodiment, the processing module 810 is specifically configured to:

[0345] performing the first image processing on the first image based on the color conversion matrix and / or the brightness parameter in a first color space to obtain a processed image;

[0346] In the first color space, a second image processing is performed on the processed image to obtain the third image, where the second image processing is a color processing algorithm in the first color space.

[0347] Optionally, as an embodiment, the processing module 810 is further configured to: detect a shooting scene in which the electronic device is located, detect the moving object; and detect the presence of the stripes and / or color cast image area in the first image.

[0348] Optionally, as an embodiment, the first duration is obtained based on the number of times the stroboscopic light source switches on and off per second.

[0349] Optionally, as an embodiment, the first duration=1000 / number of times the stroboscopic light source switches on and off per second.

[0350] Optionally, as an embodiment, the number of times the stroboscopic light source switches on and off per second is associated with the frequency of the operating voltage of the stroboscopic light source.

[0351] Optionally, as an embodiment, the image processing model is a convolutional neural network.

[0352] Optionally, as an embodiment, the image processing model is trained by the following method:

[0353] Acquiring sample data, the sample data including a first sample image, a second sample image, and a third sample image, wherein the second sample image includes image content, stripes, and / or color cast image areas of the first sample image, the third sample image has the same image content as the first sample image, and image quality of the third sample image is higher than that of the first sample image;

[0354] Inputting the first sample image and the second sample image into an image processing model to be trained to obtain predicted bilateral grid data;

[0355] performing interpolation processing on the second sample image based on the predicted bilateral grid data to obtain a predicted image;

[0356] The image processing model to be trained is trained based on the difference between the predicted image and the third sample image to obtain the image processing model.

[0357] Optionally, as an embodiment, the electronic device includes an image signal processor, and the first image is an image output by the image signal processor.

[0358] Optionally, as an embodiment, the second image is an image obtained by performing a third image processing on a Raw image captured by the electronic device, and the third image processing includes a color space conversion process.

[0359] It should be noted that the electronic device 800 is implemented in the form of a functional module. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0360] For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0361] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0362] Figure 15 shows a schematic diagram of the structure of an electronic device provided by the present application. The dotted lines in Figure 15 indicate that the unit or module is optional; the electronic device 900 can be used to implement the image processing method described in the above method embodiment.

[0363] The electronic device 900 includes one or more processors 901, which can support the electronic device 900 to implement the image processing method in the method embodiment. The processor 901 can be a general-purpose processor or a special-purpose processor. For example, the processor 901 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0364] Optionally, the processor 901 may be used to control the electronic device 900, execute software programs, and process data of the software programs. The electronic device 900 may further include a communication unit 905 for implementing signal input (reception) and output (transmission).

[0365] For example, the electronic device 900 may be a chip, the communication unit 905 may be an input and / or output circuit of the chip, or the communication unit 905 may be a communication interface of the chip, and the chip may be a component of a terminal device or other electronic device.

[0366] For another example, the electronic device 900 may be a terminal device, and the communication unit 905 may be a transceiver of the terminal device, or the communication unit 905 may include one or more memories 902 in 900, on which a program 904 is stored. The program 904 can be executed by the processor 901 to generate instructions 903, so that the processor 901 executes the image processing method described in the above method embodiment according to the instructions 903.

[0367] Optionally, data may also be stored in the memory 902 .

[0368] Optionally, the processor 901 may also read data stored in the memory 902 . The data may be stored at the same storage address as the program 904 , or may be stored at a different storage address from the program 904 .

[0369] Optionally, the processor 901 and the memory 902 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0370] Exemplarily, the memory 902 can be used to store a related program 904 of the image processing method provided in the embodiment of the present application, and the processor 901 can be used to call the related program 904 of the image processing method stored in the memory 902 when executing the image processing method, and execute the image processing method of the embodiment of the present application; for example, running a camera application of the electronic device; displaying a first image, the first image is an image of the photographed object captured based on a first exposure time, the photographed object is a moving object, the first exposure time is less than the first duration, and the first image includes stripes and / or color cast image areas; detecting a first operation, the first operation instructs the electronic device to shoot or record; in response to the first operation, obtaining a second image, the second image is an image of the photographed object captured based on a second exposure time, and the second exposure time is an integer multiple of the first duration; based on the first image and the second image, obtaining a color conversion matrix and / or brightness parameter, the color conversion matrix is ​​used to adjust the color of the first image, and the brightness parameter is used to adjust the brightness of the first image; performing first image processing on the first image based on the color conversion matrix and / or brightness parameter to obtain a third image, the third image is an image with stripes and / or color cast image areas removed; displaying or saving the third image.

[0371] Optionally, the present application further provides a computer program product, which, when executed by the processor 901, implements the image processing method in any method embodiment of the present application.

[0372] For example, the computer program product may be stored in the memory 902 , such as a program 904 , which is converted into an executable target file that can be executed by the processor 901 after undergoing preprocessing, compilation, assembly, and linking.

[0373] Optionally, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the image processing method described in any method embodiment of the present application. The computer program may be a high-level language program or an executable target program.

[0374] For example, the computer-readable storage medium is memory 902. Memory 902 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0375] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0376] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0377] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the embodiments of the electronic device described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0378] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0379] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0380] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0381] In addition, the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0382] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0383] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims. In short, the above is only a preferred embodiment of the technical solution of the present application, and is not used to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, where the light source of the shooting environment of the electronic device is a stroboscopic light source, the image processing method includes: running a camera application of the electronic device; displaying a first image, where the first image is an image of a photographic object captured based on a first exposure time, the photographic object is a moving object, the first exposure time is less than a first duration, and the first image includes stripes and / or color cast image areas; detecting a first operation, wherein the first operation instructs the electronic device to take a photo or record a video; In response to the first operation, acquiring a second image, where the second image is an image of the subject captured based on a second exposure time, where the second exposure time is an integer multiple of the first exposure time; obtaining a color conversion matrix and / or a brightness parameter based on the first image and the second image, wherein the color conversion matrix is ​​used to perform color adjustment on the first image, and the brightness parameter is used to perform brightness adjustment on the first image; performing a first image processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain a third image, wherein the third image is an image in which the stripes and / or the color cast image area are removed; The third image is displayed or saved.

2. The image processing method according to claim 1, wherein: Also includes: The first image and the second image are input into an image processing model to obtain bilateral grid data; wherein the image processing model is used to perform color migration processing and brightness migration processing on the first image based on the second image, the bilateral grid data includes the color conversion matrix and / or the brightness parameter, and the size of the first image is the same as the size of the second image.

3. The image processing method according to claim 1 or 2, wherein: The performing first image processing on the first image based on the color conversion matrix and / or the brightness parameter to obtain a third image includes: The first image is interpolated based on the color conversion matrix and / or the brightness parameter to obtain the third image.

4. The image processing method according to any one of claims 1 to 3, wherein: The interpolation processing is performed on the first image based on the color conversion matrix and / or the brightness parameter to obtain the third image, comprising: performing the first image processing on the first image based on the color conversion matrix and / or the brightness parameter in a first color space to obtain a processed image; In the first color space, a second image processing is performed on the processed image to obtain the third image, where the second image processing is a color processing algorithm in the first color space.

5. The image processing method according to any one of claims 1 to 4, characterized in that: Before acquiring the second image, the method further includes: Detecting the shooting scene in which the electronic device is located and detecting the moving object; and The presence of the stripes and / or color cast image areas is detected in the first image.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The first duration is obtained based on the number of times the stroboscopic light source turns on and off per second.

7. The image processing method according to claim 6, wherein: The first duration=1000 / number of times the stroboscopic light source turns on and off per second.

8. The image processing method according to claim 7, wherein: The number of times the stroboscopic light source turns on and off per second is associated with the frequency of the operating voltage of the stroboscopic light source.

9. The image processing method according to claim 2, wherein: The image processing model is a convolutional neural network.

10. The image processing method according to claim 2 or 9, wherein: The image processing model is trained by the following method: Acquiring sample data, the sample data including a first sample image, a second sample image, and a third sample image, wherein the second sample image includes image content, stripes, and / or color cast image areas of the first sample image, the third sample image has the same image content as the first sample image, and image quality of the third sample image is higher than that of the first sample image; Inputting the first sample image and the second sample image into an image processing model to be trained to obtain predicted bilateral grid data; performing interpolation processing on the second sample image based on the predicted bilateral grid data to obtain a predicted image; The image processing model to be trained is trained based on the difference between the predicted image and the third sample image to obtain the image processing model.

11. The image processing method according to any one of claims 1 to 10, wherein: The electronic device includes an image signal processor, and the first image is an image output by the image signal processor.

12. The image processing method according to any one of claims 1 to 11, wherein: The second image is an image obtained by performing a third image processing on the Raw image collected by the electronic device, and the third image processing includes a color space conversion process.

13. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 12.

14. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is executed by an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 12.

Images