Downsampling model training method, image processing method and device

Abstract

A downsampling model training method, image processing method, and device, the training method comprising: obtaining training data, the training data comprising a first sample image and a second sample image; inputting the first sample image into the downsampling model to obtain a first predicted image, the first predicted image being an image in a first color space; training the downsampling model based on a third sample image and the second predicted image to obtain a trained downsampling model, wherein the third sample image is an image obtained by downsampling the second sample image, and the second prediction is an image in a second color space obtained by demosaicing the first predicted image. Based on the technical solution of the present application, when downsampling an image in the first color space using the downsampling model, the clarity of the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and specifically to a downsampling model training method, an image processing method, and a device. Background Art

[0002] When taking photos (or videos) in low-light environments, such as night scenes, the image noise is relatively high due to the low amount of light entering the electronic device. To reduce the noise in the image and improve image quality, Raw domain noise reduction is usually performed on the image. However, due to the limitations of the computing performance and power consumption of electronic devices, the size of the Raw domain image needs to be reduced when performing Raw domain noise reduction. The current method of reducing the size of the Raw domain image will significantly reduce the image clarity.

[0003] Therefore, when downsampling the Raw domain image, how to improve the image clarity becomes an urgent problem to be solved. Summary of the Invention

[0004] The present application provides a downsampling model training method, an image processing method, and an apparatus, which can avoid loss of clarity and improve image clarity when downsampling a Raw domain image.

[0005] In a first aspect, a downsampling model training method is provided, comprising:

[0006] Acquire training data, wherein the training data includes a first sample image and a second sample image, the first sample image is an image in a first color space, and the second sample image is an image in a second color space obtained by demosaicing the first sample image;

[0007] Inputting the first sample image into a downsampling model to obtain a first predicted image, where the first predicted image is an image in the first color space, and the downsampling model is used to perform downsampling processing on the image in the first color space;

[0008] The downsampling model is trained based on the third sample image and the second predicted image to obtain a trained downsampling model, wherein the third sample image refers to an image obtained after the downsampling process is performed on the second sample image, and the second prediction refers to an image in the second color space obtained by demosaicing the first predicted image.

[0009] In an embodiment of the present application, when training the downsampling model, a low-resolution RGB domain image is used as the target value, and the low-resolution RGB domain image is obtained by downsampling the high-resolution RGB domain image; therefore, the clarity of the low-resolution RGB domain image and the high-resolution RGB domain image is the same; therefore, the Raw domain image after downsampling processing obtained by the downsampling model will not have the problem of clarity loss; in other words, when training the downsampling model, a low-resolution RGB domain image with the same clarity as the high-resolution RGB domain image is used as the target value; therefore, when the Raw domain image is downsampled by the downsampling model in the embodiment of the present application, the loss of clarity can be avoided and the clarity of the image can be improved.

[0010] It should be understood that a Raw domain image may refer to an image in a Raw color space; a Raw domain image refers to an image in a Raw color space; similarly, an RGB domain image refers to an image in an RGB color space, that is, an image in an RGB color space.

[0011] In a possible implementation, the first color space refers to a Raw color space; the second color space refers to an RGB color space, or other color spaces.

[0012] In one possible implementation, the downsampling model may be a deep neural network, a convolutional neural network, or other neural network structures, and this application does not impose any limitation on this.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0014] A demosaicing algorithm is obtained, where the demosaicing algorithm is used to perform the demosaicing process.

[0015] In an embodiment of the present application, when a demosaicing algorithm is known, the demosaicing algorithm can be obtained; a downsampling model is obtained based on the training data and the demosaicing algorithm, and therefore, the downsampling model can be a neural network adapted to the known demosaicing algorithm; different pre-trained downsampling neural networks can be obtained for different demosaicing algorithms.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0017] Inputting the first sample image into a demosaicing model to obtain a third predicted image, where the third predicted image is an image in the second color space;

[0018] The demosaicing model is trained based on the third predicted image and the second sample image to obtain a trained demosaicing model.

[0019] In an embodiment of the present application, when the demosaicing algorithm is unknown, a demosaicing model corresponding to the learning sample data can be obtained by acquiring a large amount of sample data for training, that is, a pre-trained demosaicing neural network is obtained; in an embodiment of the present application, the downsampling model can be adapted to the demosaicing model; different pre-trained downsampling models can be obtained for different demosaicing models.

[0020] In one possible implementation, the demosaicing model may be a deep neural network, a convolutional neural network, or other neural network structures, and this application does not impose any limitation on this.

[0021] In combination with the first aspect, in some implementations of the first aspect, the parameters of the downsampling model are obtained by iterative back propagation according to the difference between the third sample image and the second predicted image.

[0022] In combination with the first aspect, in some implementations of the first aspect, the parameters of the demosaicing model are obtained by iteratively performing a back-propagation algorithm based on the difference between the third predicted image and the first sample image.

[0023] In a second aspect, an image processing method is provided, characterized in that it is applied to an electronic device and includes:

[0024] Displaying a first interface, wherein the first interface includes a first control;

[0025] detecting a first operation on the first control;

[0026] In response to the first operation, a first image is acquired, where the first image is an image in a first color space;

[0027] Processing the first image according to a downsampling model to obtain a second image, wherein the downsampling model is used to perform downsampling processing on the image in the first color space, the downsampling model is trained based on training data and target data, the training data includes a first sample image and a second sample image, the first sample image refers to an image in the first color space, the second sample image refers to an image in the second color space obtained by performing demosaicing processing on the first sample image, and the target data refers to an image after the downsampling processing is performed on the second sample image;

[0028] Perform image processing on the second image to obtain a third image.

[0029] In an embodiment of the present application, when training the downsampling model, a low-resolution RGB domain image is used as the target value, and the low-resolution RGB domain image is obtained by downsampling the high-resolution RGB domain image; therefore, the clarity of the low-resolution RGB domain image and the high-resolution RGB domain image is the same; therefore, the Raw domain image after downsampling processing obtained by the downsampling model will not have the problem of clarity loss; in other words, when training the downsampling model, a low-resolution RGB domain image with the same clarity as the high-resolution RGB domain image is used as the target value; therefore, when the Raw domain image is downsampled by the downsampling model in the embodiment of the present application, the loss of clarity can be avoided and the clarity of the image can be improved.

[0030] It should be understood that a Raw domain image may refer to an image in a Raw color space; a Raw domain image refers to an image in a Raw color space; similarly, an RGB domain image refers to an image in an RGB color space, that is, an image in an RGB color space.

[0031] In a possible implementation, the first color space refers to a Raw color space; the second color space refers to an RGB color space, or other color spaces.

[0032] In one possible implementation, the downsampling model may be a deep neural network, a convolutional neural network, or other neural network structures, and this application does not impose any limitation on this.

[0033] In combination with the second aspect, in certain implementations of the second aspect, the first interface refers to the main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to a control corresponding to the camera application.

[0034] In a possible implementation, the first operation refers to an operation of clicking the camera application.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the first interface refers to a photo-taking interface, and the first control refers to a control for instructing photo-taking.

[0036] In a possible implementation, the first operation refers to an operation of clicking the control for instructing to record a video.

[0037] In combination with the second aspect, in certain implementations of the second aspect, the first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

[0038] In a possible implementation, the first operation refers to an operation of clicking the control for indicating a video call.

[0039] The above description uses the example of a click operation as the first operation; the first operation may also include a voice instruction operation, or other operations that instruct the electronic device to take a photo or make a video call; the above description is for illustration only and does not impose any limitation on this application.

[0040] In a possible implementation, the image processing includes at least one of the following processes:

[0041] Noise reduction, super-resolution or deblurring.

[0042] Optionally, the above-mentioned image processing may include algorithmic processing of the first color space; or the image processing may include algorithmic processing of the first color space and the second color space.

[0043] In combination with the second aspect, in certain implementations of the second aspect, the parameters of the downsampling model are obtained by iterative backpropagation based on the difference between the second predicted image and the target data, the second prediction refers to the image of the second color space obtained by performing the de-mosaicing process on the first predicted image, and the first predicted image refers to the image obtained by inputting the first sample image into the downsampling model.

[0044] In combination with the second aspect, in some implementations of the second aspect, the demosaicing process is performed by a demosaicing algorithm.

[0045] In an embodiment of the present application, when a demosaicing algorithm is known, the demosaicing algorithm can be obtained; a downsampling model is obtained based on the training data and the demosaicing algorithm, and therefore, the downsampling model can be a neural network adapted to the known demosaicing algorithm; different pre-trained downsampling neural networks can be obtained for different demosaicing algorithms.

[0046] In combination with the second aspect, in some implementations of the second aspect, the demosaicing process is performed by a demosaicing model, and the demosaicing model is obtained by training with the first sample image as input data and the second sample image as a target value.

[0047] In an embodiment of the present application, when the demosaicing algorithm is unknown, a demosaicing model corresponding to the learning sample data can be obtained by acquiring a large amount of sample data for training, that is, a pre-trained demosaicing neural network is obtained; in an embodiment of the present application, the downsampling model can be adapted to the demosaicing model; different pre-trained downsampling models can be obtained for different demosaicing models.

[0048] In combination with the second aspect, in certain implementations of the second aspect, the parameters of the demosaicing model are obtained by iterating through backpropagation based on the difference between a third predicted image and the first sample image, and the third predicted image refers to the image obtained by inputting the first sample image into the demosaicing model.

[0049] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0050] A second operation is detected, where the second operation is used to instruct to turn on a night scene mode of the electronic device, where the night scene mode refers to a shooting mode in which the amount of light entering the electronic device is less than a preset threshold.

[0051] In an embodiment of the present application, when an electronic device records a video in a dark scene, the user can turn on the night scene mode of the electronic device, thereby running the image processing method provided in the embodiment of the present application to enhance the clarity of the image and improve the image quality.

[0052] According to a third aspect, a training device for a downsampling model is provided, the training device comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory being configured to store computer program code, the computer program code comprising computer instructions, the one or more processors invoking the computer instructions to cause the training device to execute: obtaining training data, wherein the training data comprises a first sample image and a second sample image, the first sample image being an image in a first color space, and the second sample image being an image in a second color space obtained by demosaicing the first sample image;

[0053] Inputting the first sample image into a downsampling model to obtain a first predicted image, where the first predicted image is an image in the first color space, and the downsampling model is used to perform downsampling processing on the image in the first color space;

[0054] The downsampling model is trained based on the third sample image and the second predicted image to obtain a trained downsampling model, wherein the third sample image refers to an image obtained after the downsampling process is performed on the second sample image, and the second prediction refers to an image in the second color space obtained by demosaicing the first predicted image.

[0055] In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors call the computer instructions to cause the training device to execute:

[0056] A demosaicing algorithm is obtained, where the demosaicing algorithm is used to perform the demosaicing process.

[0057] In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors call the computer instructions to cause the training device to execute:

[0058] Inputting the first sample image into a demosaicing model to obtain a third predicted image, where the third predicted image is an image in the second color space;

[0059] The demosaicing model is trained based on the third predicted image and the second sample image to obtain a trained demosaicing model.

[0060] In combination with the third aspect, in some implementations of the third aspect, the parameters of the downsampling model are obtained by iteratively performing a back propagation operation based on the difference between the third sample image and the second predicted image.

[0061] In combination with the third aspect, in some implementations of the third aspect, the parameters of the demosaicing model are obtained by iteratively performing a back-propagation algorithm based on the difference between the third predicted image and the first sample image.

[0062] According to a fourth aspect, an image processing apparatus is provided, comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory being configured to store computer program code, the computer program code comprising computer instructions, the one or more processors invoking the computer instructions to cause the image processing apparatus to: display a first interface, the first interface comprising a first control;

[0063] detecting a first operation on the first control;

[0064] In response to the first operation, a first image is acquired, where the first image is an image in a first color space;

[0065] Processing the first image according to a downsampling model to obtain a second image, wherein the downsampling model is used to perform downsampling processing on the image in the first color space, the downsampling model is trained based on training data and target data, the training data includes a first sample image and a second sample image, the first sample image refers to an image in the first color space, the second sample image refers to an image in the second color space obtained by performing demosaicing processing on the first sample image, and the target data refers to an image after the downsampling processing is performed on the second sample image;

[0066] Perform image processing on the second image to obtain a third image.

[0067] In combination with the fourth aspect, in certain implementations of the fourth aspect, the parameters of the downsampling model are obtained by iterative backpropagation based on the difference between the second predicted image and the target data, the second prediction refers to the image of the second color space obtained by performing the de-mosaicing processing on the first predicted image, and the first predicted image refers to the image obtained by inputting the first sample image into the downsampling model.

[0068] In combination with the fourth aspect, in some implementations of the fourth aspect, the demosaicing process is performed by a demosaicing algorithm.

[0069] In combination with the fourth aspect, in certain implementations of the fourth aspect, the demosaicing process is performed by a demosaicing model, and the demosaicing model is obtained by training with the first sample image as input data and the second sample image as a target value.

[0070] In combination with the fourth aspect, in certain implementations of the fourth aspect, the parameters of the demosaicing model are obtained by iterating through backpropagation based on the difference between a third predicted image and the first sample image, and the third predicted image refers to the image obtained by inputting the first sample image into the demosaicing model.

[0071] In conjunction with the fourth aspect, in certain implementations of the fourth aspect, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the image processing apparatus to execute:

[0072] A second operation is detected, where the second operation is used to instruct to turn on a night scene mode of the electronic device, where the night scene mode refers to a shooting mode in which the amount of light entering the electronic device is less than a preset threshold.

[0073] In combination with the fourth aspect, in certain implementations of the fourth aspect, the first interface refers to the main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to a control corresponding to the camera application.

[0074] In combination with the fourth aspect, in certain implementations of the fourth aspect, the first interface refers to a photo-taking interface, and the first control refers to a control for instructing photo-taking.

[0075] In combination with the fourth aspect, in certain implementations of the fourth aspect, the first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

[0076] In a fifth aspect, a training device for a downsampling model is provided, comprising a module / unit for executing the first aspect or any one of the methods in the first aspect.

[0077] In a sixth aspect, an image processing device is provided, comprising a module / unit for executing the second aspect or any one of the methods in the second aspect.

[0078] In the seventh aspect, a training device for a downsampling model is provided, the training device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the training device to perform any one of the training methods in the first aspect.

[0079] In an eighth aspect, an image processing device is provided, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the image processing device to perform any one of the image processing methods in the second aspect.

[0080] In the ninth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute any one of the methods in the first aspect or the second aspect.

[0081] In a tenth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes any one of the methods in the first aspect or the second aspect.

[0082] In an eleventh aspect, a computer program product is provided, comprising: a computer program code, which, when executed by an electronic device, enables the electronic device to execute any one of the methods in the first aspect or the second aspect.

[0083] In an embodiment of the present application, when training the downsampling model, a low-resolution RGB domain image is used as the target value, and the low-resolution RGB domain image is obtained by downsampling the high-resolution RGB domain image; therefore, the clarity of the low-resolution RGB domain image and the high-resolution RGB domain image is the same; therefore, the Raw domain image after downsampling processing obtained by the downsampling model will not have the problem of clarity loss; in other words, when training the downsampling model, a low-resolution RGB domain image with the same clarity as the high-resolution RGB domain image is used as the target value; therefore, when the Raw domain image is downsampled by the downsampling model in the embodiment of the present application, the loss of clarity can be avoided and the clarity of the image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

[0085] Figure 2 is a schematic diagram of a software system for an electronic device applicable to the present application;

[0086] Figure 3 is a schematic diagram of an application scenario applicable to an embodiment of the present application;

[0087] Figure 4 is a schematic diagram of an application scenario applicable to an embodiment of the present application;

[0088] Figure 5 is a schematic diagram of a training method for a downsampling model suitable for this application;

[0089] Figure 6 is a schematic diagram of a training method for a downsampling model suitable for this application;

[0090] Figure 7 is a schematic diagram of an image processing method applicable to the present application;

[0091] Figure 8 This is a schematic diagram of the effect of the image processing method provided by the present application;

[0092] Figure 9 is a schematic diagram of a display interface of an electronic device provided in an embodiment of the present application;

[0093] Figure 10 is a schematic diagram of a display interface of an electronic device provided in an embodiment of the present application;

[0094] Figure 11 is a schematic diagram of a display interface of an electronic device provided in an embodiment of the present application;

[0095] Figure 12 Schematic diagram of a downsampling model training device provided in an embodiment of the present application;

[0096] Figure 13 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0097] Figure 14 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] Since the embodiments of the present application involve the application of a large number of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0099] 1. Neural Networks

[0100] A neural network refers to a network formed by connecting multiple single neural units together, that is, the output of one neural unit can be the input of another neural unit; the input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be an area composed of several neural units.

[0101] 2. Deep Neural Networks

[0102] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers can be fully connected, meaning that any neuron in layer i can be connected to any neuron in layer i+1.

[0103] 3. Convolutional Neural Networks

[0104] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. A CNN consists of a feature extractor consisting of a convolutional layer and a subsampling layer. This feature extractor can be considered a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to some of the neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neural units.

[0105] 4. Backpropagation Algorithm

[0106] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backpropagation movement dominated by error loss, used to obtain the optimal parameters of the neural network model; for example, the weight matrix.

[0107] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0108] Figure 1 A hardware system of an electronic device suitable for the present application is shown.

[0109] The electronic device 100 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 100.

[0110] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0111] It should be noted that Figure 1 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Figure 1 More or fewer components than those shown, or the electronic device 100 may include Figure 1 Combinations of some of the components shown, or alternatively, the electronic device 100 may include Figure 1 Subassemblies of some of the components shown. Figure 1The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0112] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices.

[0113] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0114] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0115] Figure 1 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0116] The wireless communication function of the electronic device 100 can be implemented through components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0117] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0118] Electronic device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0119] Display screen 194 may be used to display images or videos.

[0120] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0121] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0122] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red, green, blue (RGB), YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0123] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0124] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0125] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0126] For example, in an embodiment of the present application, the gyro sensor 180B may be used to collect jitter information, and the jitter information may be used to indicate changes in the posture of the electronic device during shooting.

[0127] Accelerometer 180E can detect the magnitude of the acceleration of electronic device 100 in various directions (generally the x-axis, y-axis, and z-axis). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. Accelerometer 180E can also be used to identify the posture of electronic device 100, which can serve as an input parameter for applications such as landscape / portrait switching and pedometers.

[0128] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, for example, in a photography scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0129] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0130] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0131] The touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a touch screen. The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100 and at a different location from the display screen 194.

[0132] The hardware system of the electronic device 100 is described in detail above. Now, the software system of the imaging electronic device 100 will be introduced.

[0133] Figure 2 Schematic diagram of the software system of the electronic device provided in an embodiment of the present application.

[0134] like Figure 2 As shown, the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 and a hardware layer 250 .

[0135] The application layer 210 may include applications such as camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0136] The application framework layer 220 provides an application programming interface (API) and a programming framework for the application programs of the application layer; the application framework layer may include some predefined functions.

[0137] For example, the application framework layer 220 may include a camera access interface, which may include camera management and camera devices. The camera management may be used to provide an access interface for managing the camera, and the camera device may be used to provide an interface for accessing the camera.

[0138] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer can include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call algorithms in the camera algorithm library.

[0139] For example, a camera algorithm library may include software algorithms for image processing.

[0140] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver, a digital signal processor driver, a graphics processor driver, or a central processing unit driver.

[0141] The hardware layer 250 may include camera devices and other hardware devices.

[0142] For example, the hardware layer 250 includes a camera device, a digital signal processor, a graphics processor, or a central processing unit; illustratively, the camera device may include an image signal processor, which may be used for image processing.

[0143] Exemplarily, the training method of the downsampling model provided in the embodiments of the present application can be executed in a digital signal processor, a graphics processor, or a central processing unit; or in other computing hardware of an electronic device.

[0144] Currently, when shooting in low-light environments, for example, when taking images or recording videos in night scenes, the image noise is relatively high due to the low amount of light entering the electronic device. In order to reduce the noise in the image and improve the image quality, the image is usually subjected to Raw domain noise reduction processing. Due to the limitations of the computing performance and power consumption of electronic devices, the image size needs to be reduced when performing Raw domain noise reduction. However, after downsampling the Raw domain, the image clarity will be impaired.

[0145] In view of this, an embodiment of the present application provides a downsampling model training method and an image processing method. When the Raw domain image is downsampled by the downsampling model in the embodiment of the present application, the loss of clarity can be avoided and the clarity of the image can be improved.

[0146] The following combination Figure 3 and Figure 4 The application scenarios of the image processing method provided in the embodiments of the present application are illustrated with examples.

[0147] Application scenario 1: photography

[0148] The image processing method of the present application can be applied to the field of photography; for example, it can be applied to capturing images or recording videos in dark scenes; Figure 3 As shown, when taking pictures in a dark scene (e.g., at night), the amount of light entering the electronic device is small, resulting in a large amount of noise in the acquired image; Figure 3 (a) is to perform image processing by sampling an existing solution to obtain a preview image of the photographed object 261; Figure 3 (b) is a preview image of the photographed object 261 obtained by the image processing method provided by the embodiment of the present application; Figure 3 The preview image shown in (b) is the same as Figure 3 Compared with the preview image shown in (a), the clarity is improved; therefore, the image processing method of the embodiment of the present application can improve the clarity of the image and improve the image quality.

[0149] Application scenario 2: Video call

[0150] The image processing method of the present application can be applied to the field of video calls; for example, when making a video call in a dark scene, the image noise of the video call is large due to the small amount of light entering the electronic device; Figure 4 As shown, Figure 4 (a) is to perform image processing by sampling an existing solution to obtain a preview image of the photographed object 262; Figure 4 (b) is a preview image of the photographed object 262 obtained by the image processing method provided by the embodiment of the present application; Figure 4 The preview image shown in (b) is the same as Figure 4 Compared with the preview image shown in (a), the clarity is improved; therefore, the image processing method of the embodiment of the present application can improve the clarity of the image and improve the image quality.

[0151] It should be understood that the above is an example of an application scenario and does not limit the application scenario of this application.

[0152] The following combination Figure 5 and Figure 11 The training method and image processing method of the downsampling model provided in the embodiments of the present application are described in detail.

[0153] First, the training method of the downsampling model provided in the embodiment of the present application is described in detail; the downsampling model obtained by the training method of the embodiment of the present application can be used to downsample the Raw domain image; thereby, the clarity of the image can be improved while meeting the operating performance of the electronic device.

[0154] Implementation method 1

[0155] In one example, when the demosaicing algorithm is available, that is, when the demosaicing algorithm is known, the demosaicing algorithm can be obtained by Figure 5 The training method shown results in a downsampled model.

[0156] Figure 5 This is a schematic diagram of a downsampling model training method provided in an embodiment of the present application.

[0157] It should be understood that Figure 5 The training method of the downsampling model shown can be performed by a training device, which can be Figure 1 The electronic equipment in Figure 2 The digital signal processor, graphics processor or central processing unit shown is executed; Figure 5 The training method 300 shown includes steps S310 to S360, and steps S310 to S360 are described in detail below.

[0158] Step S310: Acquire a sample Raw domain image (an example of a first sample image) and a demosaicing algorithm.

[0159] It should be understood that the sample Raw domain image refers to the sample image in the Raw color space; the sample Raw domain image refers to the sample image in the Raw color space.

[0160] Step S320 : Process the sample Raw domain image according to a demosaicing algorithm to obtain a first sample RGB domain image (an example of the second sample image).

[0161] It should be understood that the first sample RGB domain image refers to a sample image in the RGB color space; the first sample RGB domain image refers to a sample image in the RGB color space.

[0162] It should also be understood that the first sample RGB domain image and the sample Raw domain image have the same resolution size.

[0163] Step S330: downsample the first sample RGB domain image to obtain a second sample RGB domain image (an example of the third sample image).

[0164] Exemplarily, the first sample RGB domain image may be an RGB domain image of a first resolution, and the second sample RGB domain image may be an RGB domain image of a second resolution, where the first resolution is greater than the second resolution; for example, the first resolution may refer to a high resolution, and the second resolution may refer to a low resolution.

[0165] In an example, the sample Raw domain image may be a 4K image; the first sample RGB domain image may be a 4K image; and the second sample RGB domain image may be a 2K image.

[0166] Step S340: The sample Raw domain image is input as input data into a downsampling model to obtain a predicted Raw domain image (an example of a first predicted image).

[0167] For example, the resolution of the predicted Raw domain image and the resolution of the sample Raw domain image may satisfy a certain proportional relationship; for example, if the resolution of the sample Raw domain image is A, the resolution of the predicted Raw domain image may be A / 2, or A / 3, etc.

[0168] Step S350: Process the predicted Raw domain image according to a demosaicing algorithm to obtain a predicted RGB domain image (an example of a second predicted image).

[0169] It should be noted that the resolution of the predicted Raw domain image and the predicted RGB domain image may be the same.

[0170] Step S360: Using the second sample RGB domain image as a target value, a downsampling model is trained according to the difference between the predicted RGB domain image and the second sample RGB domain image.

[0171] For example, the predicted RGB domain image is compared with the second sample RGB domain image, and the parameters of the downsampling model are iterated through the back propagation algorithm until the downsampling model to be trained converges to obtain the trained downsampling model, that is, the pre-trained downsampling neural network.

[0172] For example, in the embodiments of the present application, the downsampling model may be a deep neural network, a convolutional neural network or other neural network structures, and the present application does not impose any limitation on this.

[0173] It should be understood that the above description uses sample Raw domain images and sample RGB domain images as training data for example; the sample RGB domain images may also be images in other color spaces, and this application does not impose any limitation on this.

[0174] It should also be understood that the downsampling model is obtained by training the training data with a known demosaicing algorithm; therefore, the downsampling model can be a neural network adapted to the known demosaicing algorithm; different pre-trained downsampling neural networks can be obtained for different demosaicing algorithms.

[0175] In an embodiment of the present application, when training the downsampling model, a second sample RGB domain image (for example, a low-resolution RGB domain image) is used as the target value, and the second sample RGB domain image is obtained by downsampling the first sample RGB domain image (for example, a high-resolution RGB domain image); therefore, the clarity of the second sample RGB domain image is the same as that of the first sample RGB domain image; therefore, the Raw domain image after downsampling obtained by the downsampling model will not have the problem of clarity loss; in other words, when training the downsampling model, a low-resolution RGB domain image with the same clarity as the high-resolution RGB domain image is used as the target value; therefore, when the Raw domain image is downsampled by the downsampling model in the embodiment of the present application, the loss of clarity can be avoided and the clarity of the image can be improved.

[0176] Implementation method 2

[0177] In one example, when the demosaicing algorithm cannot be obtained, that is, when the demosaicing algorithm is unknown, a demosaicing model can be obtained by training first; according to the demosaicing model and sample data, a downsampling model is obtained, such as Figure 6 shown.

[0178] It should be noted that the difference between implementation method 2 and implementation method 1 is that in implementation method 2, the demosaicing algorithm cannot be obtained directly. It is necessary to obtain a large amount of sample data for training to obtain a demosaicing model of the demosaicing algorithm corresponding to the learning sample data, that is, to obtain a pre-trained demosaicing neural network.

[0179] Figure 6 This is a schematic diagram of a downsampling model training method provided in an embodiment of the present application.

[0180] It should be understood that Figure 6 The training method of the downsampling model shown can be performed by a training device, which can be Figure 1 The electronic equipment in Figure 2 The digital signal processor, graphics processor or central processing unit shown is executed; Figure 6 The training method 400 shown includes steps S410 to S460, and steps S410 to S460 are described in detail below.

[0181] Step S410: Acquire a sample Raw domain image (an example of a first sample image) and a sample RGB domain image (an example of a second sample image).

[0182] It should be understood that the sample Raw domain image refers to the sample image in the Raw color space; the sample Raw domain image refers to the sample image in the Raw color space; similarly, the sample RGB domain image refers to the sample image in the RGB color space; the sample RGB domain image refers to the sample image in the RGB color space.

[0183] It should be noted that the sample RGB domain image refers to an image obtained by processing the sample Raw domain image through an unknown demosaicing algorithm.

[0184] Step S420 : Using the sample Raw domain image as input data and the sample RGB domain image as a target value, a demosaicing model is trained to obtain a trained demosaicing model.

[0185] Exemplarily, the sample Raw domain image is input into the demosaicing model to obtain a predicted RGB domain image (an example of the third predicted image); the predicted RGB domain image is compared with the sample RGB domain image; the parameters of the demosaicing model to be trained are iterated through the back propagation algorithm until the demosaicing model to be trained converges, thereby obtaining a trained demosaicing model, that is, a pre-trained demosaicing neural network.

[0186] For example, in the embodiments of the present application, the demosaicing model may be a deep neural network, a convolutional neural network or other neural network structures, and the present application does not impose any limitation on this.

[0187] Step S430: down-sample the sample RGB domain image to obtain a third sample RGB domain image (an example of the third sample image).

[0188] Exemplarily, the sample RGB domain image may be an RGB domain image of a first resolution, and the third sample RGB domain image may be an RGB domain image of a second resolution, where the first resolution is greater than the second resolution; for example, the first resolution may refer to a high resolution, and the second resolution may refer to a low resolution.

[0189] In an example, the sample Raw domain image may be a 4K image, the sample RGB domain image may be a 4K image, and the third sample RGB domain image may be a 2K image.

[0190] Step S440: The sample Raw domain image is input as input data into the downsampling model to be trained to obtain a predicted Raw domain image (an example of a first predicted image).

[0191] For example, the resolution of the predicted Raw domain image and the resolution of the sample Raw domain image may satisfy a certain proportional relationship; for example, if the resolution of the sample Raw domain image is A, the resolution of the predicted Raw domain image may be A / 2, or A / 3, etc.

[0192] Step S450: Input the predicted Raw domain image (an example of the first predicted image) into the demosaicing model to obtain a predicted RGB domain image (an example of the second predicted image).

[0193] It should be noted that the resolution of the predicted Raw domain image and the predicted RGB domain image may be the same.

[0194] Step S460: Using the second sample RGB domain image as a target value, a downsampling model is trained according to the difference between the predicted RGB domain image and the second sample RGB domain image.

[0195] For example, the predicted RGB domain image is compared with the third sample RGB domain image, and the parameters of the downsampling model are iterated through the back propagation algorithm until the downsampling model to be trained converges to obtain the trained downsampling model, that is, the pre-trained downsampling neural network.

[0196] For example, in the embodiments of the present application, the downsampling model may be a deep neural network, a convolutional neural network or other neural network structures, and the present application does not impose any limitation on this.

[0197] It should be understood that the above description uses sample Raw domain images and sample RGB domain images as training data for example; the sample RGB domain images may also be images in other color spaces, and this application does not impose any limitation on this.

[0198] It should also be understood that the downsampling model is obtained by training the training data and the demosaicing model; therefore, the downsampling model can be a neural network adapted to the demosaicing model; different pre-trained downsampling neural networks can be obtained for different demosaicing models.

[0199] In an embodiment of the present application, when training the downsampling model, a third sample RGB domain image (for example, a low-resolution RGB domain image) is used as the target value, and the third sample RGB domain image is obtained by downsampling the sample RGB domain image (for example, a high-resolution RGB domain image); therefore, the clarity of the third sample RGB domain image is the same as that of the sample RGB domain image; therefore, the Raw domain image after downsampling obtained by the downsampling model will not have the problem of clarity loss; in other words, when training the downsampling model, a low-resolution RGB domain image with the same clarity as the high-resolution RGB domain image is used as the target value; therefore, when the Raw domain image is downsampled by the downsampling model in the embodiment of the present application, the loss of clarity can be avoided and the clarity of the image can be improved.

[0200] pass Figure 5 and Figure 6 The downsampling model obtained by the training method shown can be applied to image processing methods; the following set Figure 7 The image processing method provided in the embodiment of the present application is described.

[0201] Figure 7 It is a schematic diagram of an image processing method provided in an embodiment of the present application. Figure 7 The image processing method shown can be Figure 1 The electronic device shown in , or, configured in Figure 1 The chip in the electronic device shown performs; Figure 7 The image processing method 500 shown includes steps S510 to S530 , and steps S510 to S530 are described in detail below.

[0202] Step S510: Acquire a Raw domain image (an example of a first image).

[0203] Exemplarily, a Raw domain image captured by a sensor of the electronic device may be acquired.

[0204] It should be understood that the Raw domain image may refer to an image acquired in the Raw color space; that is, the Raw domain image may refer to an image in the Raw color space.

[0205] Step S520: Input the Raw domain image into a downsampling model to obtain a downsampled Raw domain image (an example of a second image).

[0206] Exemplarily, the acquired Raw domain image has a first resolution, and the down-sampled Raw domain image has a second resolution, wherein the first resolution is greater than the second resolution.

[0207] For example, the acquired Raw domain image may refer to a 4K full-size image captured by the sensor, and the down-sampled Raw domain image may refer to a 2K Raw domain image.

[0208] It should be understood that the downsampling model can be based on Figure 5 or Figure 6 The training method shown is obtained and will not be described here.

[0209] Step S530: Perform image processing on the down-sampled Raw domain image to obtain a processed image (an example of a third image).

[0210] Exemplarily, image processing may include but is not limited to:

[0211] Noise reduction, super-resolution or deblurring processing, etc.

[0212] Optionally, Figure 7 The image processing method shown can be an image processing method independent of ISP; for example, Figure 7 The image processing method shown can be used in Figure 2 The camera algorithms shown are implemented in the library.

[0213] Optionally, Figure 7 The image processing method shown can be executed in an ISP; for example, the ISP may include a first module, the first module being configured to execute the following Figure 7 The image processing method shown; for example, the training method of the downsampling model can be executed in the NPU to obtain the parameters of the downsampling model; the first module in the ISP can obtain the parameters of the downsampling model to obtain the downsampling model, and downsample the first image through the downsampling model.

[0214] In an embodiment of the present application, when training the downsampling model, a low-resolution RGB domain image is used as the target value, and the low-resolution RGB domain image is obtained by downsampling the high-resolution RGB domain image; therefore, the clarity of the low-resolution RGB domain image and the high-resolution RGB domain image is the same; therefore, the Raw domain image after downsampling processing obtained by the downsampling model will not have the problem of clarity loss; in other words, when training the downsampling model, a low-resolution RGB domain image with the same clarity as the high-resolution RGB domain image is used as the target value; therefore, when the Raw domain image is downsampled by the downsampling model in the embodiment of the present application, the loss of clarity can be avoided and the clarity of the image can be improved.

[0215] Figure 8 It is a schematic diagram of the effect of the image processing method provided in an embodiment of the present application.

[0216] like Figure 8 As shown, Figure 8 (a) is an output image obtained by the image processing method of an embodiment of the present application; Figure 8 (b) is the output image obtained by performing noise reduction on the existing Raw domain image and then using the V2H2 binning method; Figure 8 Compared to the output image shown in (a), Figure 8 The output image shown in (b) has a low clarity; therefore, the image processing method provided by the embodiment of the present application can improve the clarity of the image when performing noise reduction processing on the image compared with the existing solution.

[0217] In one example, the night scene mode can be turned on in the camera application of the electronic device. When the electronic device recognizes that the current shooting environment is in a scene with less light input, the image processing method provided in the embodiment of the present application is used to perform noise reduction processing on the Raw domain image collected by the sensor, thereby outputting the processed image or video.

[0218] Figure 9 A graphical user interface (GUI) of an electronic device is shown.

[0219] like Figure 9 The GUI shown in (a) of FIG. 1 may be a display interface of a camera application in a photographing mode, and the display interface may include a photographing interface 610; the photographing interface 610 may include a viewfinder 611 and controls; for example, the viewfinder 611 may include a control 612 for indicating photographing and a control 613 for indicating settings; when a user clicks on the control 613, a settings interface is displayed in response to the user operation, such as Figure 9As shown in (b); the setting interface includes a night scene mode 614, and it is detected that the user turns on the night scene mode; after the electronic device turns on the night scene mode, the image captured by the electronic device can be processed by the image processing method provided in the embodiment of the present application.

[0220] In one example, if Figure 10 The shooting interface 610 shown in (a) may include a control 613 for indicating settings; detecting the operation of the user clicking the control 613, and displaying the setting interface in response to the user operation, such as Figure 10 As shown in (b); the setting interface includes an artificial intelligence (AI) night scene mode 615, and it is detected that the user turns on the AI ​​night scene mode; after the user turns on the AI ​​night scene mode, the user does not need to manually select to turn on the night scene mode; that is, the electronic device can automatically identify the dark light scene according to the amount of light entering the shooting scene, and automatically turn on the night scene mode in the dark light scene; after the night scene mode is turned on, the image processing method provided in the embodiment of the present application can be used to process the acquired image to obtain a processed image.

[0221] In one example, if Figure 11 As shown in the photo mode, the shooting interface 610 may also include a control 616, which is used to indicate whether to turn the night scene mode on / off; after the electronic device detects that the user clicks the control 616, the electronic device can turn on the night scene mode and perform image processing on the image captured by the electronic device through the image processing method provided in the embodiment of the present application.

[0222] In an embodiment of the present application, a low-resolution RGB domain image can be obtained by combining a low-resolution Raw domain image output by a pre-trained downsampling neural network with a demosaicing algorithm (or a pre-trained demosaicing neural network); when training the downsampling neural network, the low-resolution sample RGB domain image is used as the target value, and the low-resolution sample RGB domain image is obtained by downsampling the RGB domain image of the high-resolution sample; therefore, the clarity of the low-resolution sample RGB domain image and the high-resolution sample RGB domain image is the same; therefore, the low-resolution Raw domain image obtained by the downsampling model will not have the problem of clarity loss, therefore, the loss of image clarity can be avoided and the clarity of the image can be improved.

[0223] Combined with the above Figures 1 to 11 The training method and image processing method of the downsampling model provided by the embodiment of the present application are described in detail; Figure 12 and Figure 14The device embodiments of the present application are described in detail. It should be understood that the devices in the embodiments of the present application can execute the various methods of the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0224] Figure 12 7 is a schematic diagram of a structure of a downsampling model training device provided in an embodiment of the present application. The training device 700 includes an acquisition module 710 and a processing module 720.

[0225] In which, the acquisition module 710 is used to acquire training data, wherein the training data includes a first sample image and a second sample image, the first sample image refers to an image in a first color space, and the second sample image refers to an image in a second color space obtained by demosaicing the first sample image; the processing module 720 is used to input the first sample image into a downsampling model to obtain a first prediction image, the first prediction image is an image in the first color space, and the downsampling model is used to downsample the image in the first color space; the downsampling model is trained based on a third sample image and a second prediction image to obtain a trained downsampling model, wherein the third sample image refers to an image obtained after the downsampling process is performed on the second sample image, and the second prediction refers to an image in the second color space obtained by demosaicing the first prediction image.

[0226] Optionally, as an embodiment, the processing module 720 is further configured to:

[0227] A demosaicing algorithm is obtained, where the demosaicing algorithm is used to perform the demosaicing process.

[0228] Optionally, as an embodiment, the processing module 720 is further configured to:

[0229] Inputting the first sample image into a demosaicing model to obtain a third predicted image, where the third predicted image is an image in the second color space;

[0230] The demosaicing model is trained based on the third predicted image and the second sample image to obtain a trained demosaicing model.

[0231] Optionally, as an embodiment, the parameters of the downsampling model are obtained by iterative back propagation according to the difference between the third sample image and the second predicted image.

[0232] Optionally, as an embodiment, the parameters of the demosaicing model are obtained by iteratively performing a back-propagation algorithm according to the difference between the third predicted image and the first sample image.

[0233] It should be noted that the training device 700 is implemented in the form of functional modules. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0234] Figure 13 FIG. 8 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device 800 includes a display module 810 and a processing module 820 .

[0235] Among them, the display module 810 is used to display a first interface, which includes a first control; the processing module 820 is used to detect a first operation on the first control; in response to the first operation, a first image is acquired, and the first image is an image in a first color space; the first image is processed according to a downsampling model to obtain a second image, wherein the downsampling model is used to downsample the image in the first color space, and the downsampling model is trained based on training data and target data, and the training data includes a first sample image and a second sample image, the first sample image refers to the image in the first color space, the second sample image refers to the image in the second color space obtained by de-mosaicing the first sample image, and the target data refers to the image after the downsampling processing is performed on the second sample image; image processing is performed on the second image to obtain a third image.

[0236] Optionally, as an embodiment, the parameters of the downsampling model are obtained by iterative back propagation according to the difference between the second predicted image and the target data, the second prediction refers to the image of the second color space obtained by performing the de-mosaicing process on the first predicted image, and the first predicted image refers to the image obtained by inputting the first sample image into the downsampling model.

[0237] Optionally, as an embodiment, the demosaicing process is performed by a demosaicing algorithm.

[0238] Optionally, as an embodiment, the demosaicing process is performed by a demosaicing model, and the demosaicing model is obtained by training with the first sample image as input data and the second sample image as a target value.

[0239] Optionally, as an embodiment, the parameters of the demosaicing model are obtained by iterating through back propagation according to the difference between a third predicted image and the first sample image, and the third predicted image refers to an image obtained by inputting the first sample image into the demosaicing model.

[0240] Optionally, as an embodiment, the processing module 820 is further configured to:

[0241] A second operation is detected, where the second operation is used to instruct to turn on a night scene mode of the image processing device, where the night scene mode refers to a shooting mode in which an amount of light entering the image processing device is less than a preset threshold.

[0242] Optionally, as an embodiment, the first interface refers to a main screen interface of the image processing device, the main screen interface includes a camera application, and the first control refers to a control corresponding to the camera application.

[0243] Optionally, as an embodiment, the first interface refers to a photographing interface, and the first control refers to a control for instructing photographing.

[0244] Optionally, as an embodiment, the first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

[0245] It should be noted that the image processing device 800 is implemented in the form of a functional module. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0246] For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0247] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0248] Figure 14 A schematic structural diagram of an electronic device provided in this application is shown. Figure 14 The dotted line in the figure indicates that the unit or module is optional; the electronic device 900 can be used to implement the method described in the above method embodiment.

[0249] The electronic device 900 includes one or more processors 901, which can support the training method of the downsampling model or the image processing method in the embodiment of the electronic device 900 implementation method. The processor 901 can be a general-purpose processor or a special-purpose processor. For example, the processor 901 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0250] The processor 901 can be used to control the electronic device 900, execute software programs, and process data of the software programs. The electronic device 900 can also include a communication unit 905 to implement signal input (reception) and output (transmission).

[0251] For example, the electronic device 900 may be a chip, the communication unit 905 may be an input and / or output circuit of the chip, or the communication unit 905 may be a communication interface of the chip, and the chip may be a component of a terminal device or other electronic device.

[0252] For another example, the electronic device 900 may be a terminal device, and the communication unit 905 may be a transceiver of the terminal device, or the communication unit 905 may be a transceiver circuit of the terminal device.

[0253] The electronic device 900 may include one or more memories 902 on which a program 904 is stored. The program 904 can be executed by the processor 901 to generate instructions 903, so that the processor 901 executes the training method or image processing method described in the above method embodiment according to the instructions 903.

[0254] Optionally, data may be stored in the memory 902. Optionally, the processor 901 may read data stored in the memory 902. The data may be stored at the same storage address as the program 904, or may be stored at a different storage address from the program 904.

[0255] The processor 901 and the memory 902 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of a terminal device.

[0256] Exemplarily, the memory 902 can be used to store a related program 904 of the training method for the downsampling model provided in the embodiment of the present application, and the processor 901 can be used to call the related program 904 of the training method for the downsampling model stored in the memory 902 when executing the training of the downsampling model, and execute the training method of the downsampling model of the embodiment of the present application; for example, obtaining training data, wherein the training data includes a first sample image and a second sample image, the first sample image refers to an image in a first color space, and the second sample image refers to an image in a second color space obtained by demosaicing the first sample image; inputting the first sample image into the downsampling model to obtain a first predicted image, the first predicted image is an image in the first color space, and the downsampling model is used to downsample the image in the first color space; training the downsampling model based on the third sample image and the second predicted image to obtain a trained downsampling model, wherein the third sample image refers to an image obtained after the downsampling process is performed on the second sample image, and the second prediction refers to an image in the second color space obtained by demosaicing the first predicted image.

[0257] Exemplarily, the memory 902 can be used to store a related program 904 of the image processing method provided in the embodiment of the present application, and the processor 901 can be used to call the related program 904 of the image processing method stored in the memory 902 when performing image processing, and execute the image processing method of the embodiment of the present application; for example, displaying a first interface, the first interface includes a first control; detecting a first operation on the first control; in response to the first operation, acquiring a first image, the first image is an image in a first color space; processing the first image according to a downsampling model to obtain a second image, wherein the downsampling model is used to downsample the image in the first color space, the downsampling model is trained based on training data and target data, the training data includes a first sample image and a second sample image, the first sample image refers to the image in the first color space, the second sample image refers to the image in the second color space obtained by de-mosaicing the first sample image, and the target data refers to the image after the downsampling processing is performed on the second sample image; image processing is performed on the second image to obtain a third image.

[0258] The present application also provides a computer program product, which, when executed by the processor 901, implements the training method or image processing method described in any method embodiment of the present application.

[0259] The computer program product may be stored in the memory 902 , for example, a program 904 , which is converted into an executable target file that can be executed by the processor 901 after undergoing processes such as preprocessing, compilation, assembly, and linking.

[0260] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the image processing method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0261] The computer-readable storage medium is, for example, memory 902. Memory 902 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0262] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0263] In the several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0264] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0265] Additionally, the terms "system" and "network" are often used interchangeably. The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects.

[0266] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. An image processing method, characterized in that: Used in electronic equipment, including: Displaying a first interface, wherein the first interface includes a first control; detecting a first operation on the first control; In response to the first operation, a first image is acquired, where the first image is an image in a first color space; Processing the first image according to a trained downsampling model to obtain a second image, wherein the trained downsampling model is trained based on the training data and the third sample image; performing image processing on the second image to obtain a third image; The trained downsampling model is obtained by training based on the training data and the third sample image, and includes: Acquire the training data, where the training data includes a first sample image and a second sample image, where the first sample image refers to an image in the first color space, and the second sample image refers to an image in the second color space obtained by demosaicing the first sample image; Inputting the first sample image into a downsampling model to obtain a first predicted image, where the first predicted image is an image in the first color space, and the downsampling model is used to perform downsampling processing on the image in the first color space; The downsampling model is trained based on the third sample image and the second predicted image to obtain the trained downsampling model, wherein the third sample image refers to the image obtained after the downsampling processing is performed on the second sample image, and the second predicted image refers to the image in the second color space obtained by demosaicing the first predicted image.

2. The image processing method according to claim 1, wherein: The parameters of the downsampling model are obtained by iteratively performing a back propagation algorithm according to the difference between the second predicted image and the third sample image.

3. The image processing method according to claim 1 or 2, wherein: The demosaicing process is performed by a demosaicing algorithm.

4. The image processing method according to claim 1 or 2, wherein: The demosaicing process is performed by a demosaicing model, where the demosaicing model is obtained by training with the first sample image as input data and the second sample image as a target value.

5. The image processing method according to claim 4, wherein: The parameters of the demosaicing model are obtained by iterating a back propagation algorithm according to the difference between a third predicted image and the first sample image. The third predicted image refers to an image obtained by inputting the first sample image into the demosaicing model.

6. The image processing method according to claim 1 or 2, wherein: Also includes: A second operation is detected, where the second operation is used to instruct to turn on a night scene mode of the electronic device, where the night scene mode refers to a shooting mode in which the amount of light entering the electronic device is less than a preset threshold.

7. The image processing method according to claim 1 or 2, wherein: The first interface refers to a main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to a control corresponding to the camera application.

8. The image processing method according to claim 1 or 2, wherein: The first interface refers to a photo taking interface, and the first control refers to a control for instructing photo taking.

9. The image processing method according to claim 1 or 2, wherein: The first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

10. The image processing method according to claim 1, wherein: Also includes: A demosaicing algorithm is obtained, where the demosaicing algorithm is used to perform the demosaicing process.

11. The image processing method according to claim 1, wherein: Also includes: Inputting the first sample image into a demosaicing model to obtain a third predicted image, where the third predicted image is an image in the second color space; The demosaicing model is trained based on the third predicted image and the second sample image to obtain a trained demosaicing model.

12. An image processing device, characterized in that: The image processing device comprises: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the image processing method according to any one of claims 1 to 11.

13. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program code which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 11.

Images