Image processing method, device and electronic device

Abstract

The present application provides an image processing method, device and electronic device. In the technical solution of the present application, an image to be processed is obtained; the image to be processed is processed by the first branch of a pre-trained neural network model to obtain a first type of parameter, and the first type of parameter is used to perform global color processing on the image; the image to be processed is processed by the second branch of the neural network model to obtain a second type of parameter, and the second type of parameter is used to perform local color processing on the image; the image to be processed is color processed according to the first type of parameter and the second type of parameter to obtain a color-processed image. In the above technical solution, the input of the neural network model is the image to be processed, and different branches output different types of parameters, which can avoid the error accumulation problem in the parameter calculation process and obtain more accurate parameters. Furthermore, the image to be processed is processed according to the obtained parameters, which can improve the image color correction effect.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to an image processing method, device and electronic device. Background Art

[0002] The main function of image signal processing (ISP) is to post-process the image signal output by the front-end image sensor. Relying on ISP, the images obtained under different optical conditions can better restore the details of the scene.

[0003] ISP processing flow is as follows Figure 1 As shown, a natural scene 101 obtains a Bayer image through a lens 102, and then obtains an analog electrical signal 105 through a photoelectric conversion 104, and further obtains a digital image signal (i.e., a raw image) 107 through noise reduction and analog-to-digital processing 106, and then enters a digital signal processing chip 100. The steps in the digital signal processing chip 100 are the core steps of ISP processing. The digital signal processing chip 100 generally includes black level correction (BLC) 108, lens shading correction 109, bad pixel correction (BPC) 110, demosaic 111, Bayer domain noise reduction (denoise) 112, auto white balance (AWB) 113, Y gamma 114, auto exposure (AE) 115, auto focus (AF) ( Figure 1 ), color correction (CC) 116, gamma correction 117, color gamut conversion 118, color denoising / detail enhancement 119, color enhancement (CE) 120, formatter 121, input / output (I / O) control 122 and other modules.

[0004] The color-related modules in ISP processing mainly include AWB113, CC116, CE120 and other modules. Among them, AWB and CC modules are global color processing modules, and CE is a local color processing module. The color-related modules in ISP processing can be implemented by a neural network model, but since ISP processing is serial processing, that is, the output of the previous module is used as the input of the next module, there is a problem of error accumulation. Summary of the invention

[0005] The present application provides an image processing method and device, which can avoid the error accumulation problem in the serial ISP image color processing flow and improve the image color processing effect.

[0006] In a first aspect, the present application provides an image processing method, the method comprising: obtaining an image to be processed; processing the image to be processed through a first branch of a pre-trained neural network model to obtain a first type of parameters, the first type of parameters being used to perform global color processing on the image; processing the image to be processed through a second branch of the neural network model to obtain a second type of parameters, the second type of parameters being used to perform local color processing on the image; performing color processing on the image to be processed according to the first type of parameters and the second type of parameters to obtain a color-processed image.

[0007] In the above technical solution, the input of the neural network model is the image to be processed, and different branches output different types of parameters, which can avoid the error accumulation problem in the parameter calculation process and obtain more accurate parameters. Furthermore, the image to be processed is processed according to the obtained parameters, which can improve the image color correction effect.

[0008] In a possible implementation manner, the image to be processed is a raw image.

[0009] In the above technical solution, the input of the neural network model is a raw image, which retains the image information to the greatest extent. The first type of parameters and the second type of parameters obtained in this way are more accurate, and thus the image color correction effect is better.

[0010] In a possible implementation, the global color processing includes automatic self-balancing and / or color correction, and the local color processing includes color rendering and / or color enhancement.

[0011] In the above technical solution, the first and second parameters can correspond to the traditional ISP module, so that when the image color correction effect is not ideal, the first and second parameters can be adjusted according to the experience of traditional ISP debugging, thereby solving the inherent problem of the subjective effect of the neural network that cannot be adjusted.

[0012] In a possible implementation, the first branch and the second branch share a shared parameter layer of the neural network model.

[0013] It can be understood that, in the case where both the first type of parameters and the second type of parameters require intermediate feature layer data, the layer of the first branch that obtains the intermediate feature layer data and the layer of the second branch that obtains the intermediate feature layer data can share structural parameters. For example, the image to be processed is processed through the shared parameter layer of the pre-trained neural network model to obtain the intermediate feature layer data; the intermediate feature layer data is processed through the first branch of the pre-trained neural network model (except the shared parameter layer) to obtain the first type of parameters; the intermediate feature layer data is processed through the second branch of the neural network model (except the shared parameter layer) to obtain the second type of parameters.

[0014] The second branch of the neural network model in the above technical solution can directly use the shared parameter layer to obtain the intermediate feature layer data. That is to say, the first branch and the second branch can reuse part of the calculation process, thereby reducing the complexity of the calculation and reducing the storage space occupied.

[0015] In a possible implementation, the first type of parameters are in matrix form; the color processing of the image to be processed according to the first type of parameters and the second type of parameters includes: performing matrix multiplication of the image to be processed and the first type of parameters to obtain a first image; and performing local color processing on the first image according to the second type of parameters to obtain a second image.

[0016] In a possible implementation, the local color processing of the first image according to the second type of parameters includes: calculating the difference between the numerical value of the color channel and the numerical value of the brightness channel of the first image; adjusting the difference according to the second type of parameters; and adding the adjusted difference to the numerical value of the brightness channel of the first image.

[0017] In a possible implementation, the image format of the first image is a color RGB format, the second type of parameters includes a color processing coefficient beta1 of a color channel R, a color processing coefficient beta2 of a color channel G, and a color processing coefficient beta3 of a color channel B; performing local color processing on the first image according to the second type of parameters includes: performing local color processing according to the formula Perform local color adjustment on the first image, where Y″ is the value of the brightness channel of the first image, R″′, G″′, and B″′ are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the second image, and R″, G″, and B″ are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the first image.

[0018] In a possible implementation manner, before performing color processing on the image to be processed according to the first type of parameters and the second type of parameters, the method further includes: performing demosaicing and denoising processing on the image to be processed.

[0019] In a second aspect, the present application provides an image processing device, comprising a module for executing the first aspect or any one of the implementations of the first aspect.

[0020] In a third aspect, the present application provides an image processing device, including a memory and a processor, for executing the method described in the first aspect or any one of the implementations of the first aspect.

[0021] In a fourth aspect, the present application provides a chip, which is connected to a memory and is used to read and execute a software program stored in the memory to implement the method described in the first aspect or any one of the implementation methods of the first aspect.

[0022] In a fifth aspect, the present application provides an electronic device, comprising a processor and a memory, for executing the method described in the first aspect or any one of the implementations of the first aspect.

[0023] In a sixth aspect, the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0024] In a seventh aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect or any one of the implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic flow chart of ISP processing.

[0026] Figure 2 It is a schematic flowchart of the image processing method of an embodiment of the present application.

[0027] Figure 3 This is a specific example of the image processing method of the embodiment of the present application.

[0028] Figure 4 It is the network framework of the neural network model of the embodiment of the present application.

[0029] Figure 5 It is a network framework of a neural network model of another embodiment of the present application.

[0030] Figure 6 It is a network framework of a neural network model of another embodiment of the present application.

[0031] Figure 7 It is a schematic structural diagram of the image processing device provided in an embodiment of the present application.

[0032] Figure 8 It is a schematic structural diagram of an image processing device provided in another embodiment of the present application.

[0033] Fig. 9 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0035] The technical solution of the present application can be applied to any scenario that requires color processing of images, such as safe cities, remote driving, human-computer interaction, and other scenarios that require taking photos, recording videos, or displaying images.

[0036] It should be understood that the terms "system" and "network" are often used interchangeably in this application. The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0037] Figure 2 200 is a schematic flow chart of an image processing method according to an embodiment of the present application. The method 200 can be executed by any device having an image processing function. Figure 2 As shown, method 200 may include at least part of the following contents.

[0038] In 210, an image to be processed is obtained.

[0039] In 220, the image to be processed is processed by a first branch of a pre-trained neural network model to obtain a first type of parameters, where the first type of parameters are used to perform global color processing on the image.

[0040] In 230, the image to be processed is processed by the second branch of the neural network model to obtain second type parameters, where the second type parameters are used to perform local color processing on the image.

[0041] In 240, color processing is performed on the image to be processed according to the first type of parameters and the second type of parameters to obtain a color-processed image.

[0042] In method 200, the inputs of different branches of the neural network model are all images to be processed. When they have the same input, different types of parameters are output. That is to say, the first branch and the second branch of the neural network model have the same input, so that the processing process of the image to be processed by the first branch and the second branch is parallel processing. Compared with the serial processing process, the error accumulation problem in the parameter calculation process can be avoided, and more accurate parameters can be obtained. Furthermore, the image to be processed is processed according to the obtained parameters to improve the image color correction effect. And the output of the neural network model is the color processing parameters, and the above parameters can correspond to the parameters corresponding to the traditional ISP processing module. When the image color processing effect is not ideal, the parameters can be fine-tuned according to the traditional ISP parameter debugging experience, so as to further improve the image color correction effect and solve the problem that the parameters of the neural network cannot be debugged, which is the biggest criticism.

[0043] The image to be processed in the embodiment of the present application may be acquired image data, image signal, etc. In practical applications, the image to be processed may be acquired through an image acquisition device (e.g., a lens and a sensor), or may be received from other devices, which is not specifically limited in the embodiment of the present application.

[0044] The image to be processed may be a raw image. Since the raw image is the raw data obtained by converting the captured light source signal into a digital signal by a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, it is lossless, so it contains the original information of the object. The input of the neural network model is a raw image, which retains the image information to the greatest extent. The first and second parameters obtained in this way can remind the information of the real scene, and thus the image color correction effect is better. Of course, the image to be processed may also be an image after other image processing processes other than color processing. Other image processing processes include any one or any combination of black level correction, lens shading correction, bad pixel correction, de-mosaicing, Bayer domain noise reduction, automatic exposure, automatic focus, etc.

[0045] The pre-trained neural network model of the embodiment of the present application can be stored on an image processing device, for example, stored on a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) AR / virtual reality (VR), a vehicle-mounted terminal, etc. The pre-trained neural network model of the embodiment of the present application can also be stored on a server or cloud, etc. The pre-trained neural network model can be a corresponding target model / rule generated based on different training data for different goals (or different tasks), and the corresponding target model / rule can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. For example, in an embodiment of the present application, it can be the output of the first type of parameters and the second type of parameters for image color processing.

[0046] Different branches of the neural network model can be understood as different processing processes of the neural network model. In the embodiment of the present application, the first branch of the neural network model corresponds to generating the first type of parameters (i.e., parameters for global color processing of the image), and the second branch of the neural network model corresponds to generating the second type of parameters (i.e., parameters for local color processing of the image).

[0047] The first branch and the second branch can be two independent neural network models. That is to say, the neural network model of the embodiment of the present application is a set consisting of a first neural network model corresponding to the first branch and a second neural network model corresponding to the second branch, and the inputs of the first neural network model and the second neural network model are both images to be processed; the first branch and the second branch can also be two parts of the same neural network model, which is not specifically limited in the embodiment of the present application.

[0048] Optionally, the first branch and the second branch can reuse or share part of the processing process. For example, the first branch and the second branch share the shared parameter layer of the neural network model. It can be understood that in the case where both the first type of parameters and the second type of parameters need to obtain the intermediate feature layer data, the layer of the first branch that obtains the intermediate feature layer data and the layer of the second branch that obtains the intermediate feature layer data can share structural parameters. For example, the image to be processed is processed by the shared parameter layer of the pre-trained neural network model to obtain the intermediate feature layer data; the intermediate feature layer data is processed by the first branch of the pre-trained neural network model (except the shared parameter layer) to obtain the first type of parameters; the intermediate feature layer data is processed by the second branch of the neural network model (except the shared parameter layer) to obtain the second type of parameters. In this way, the second branch of the neural network model can directly use the shared parameter layer to obtain the intermediate feature layer data, that is, the first branch and the second branch can reuse part of the calculation process, thereby reducing the complexity of the calculation and reducing the storage space occupied.

[0049] The first type of parameters is used to perform global color processing on the image, wherein the global color processing includes at least one global color processing. For example, the first type of parameters is used to perform automatic self-balancing processing and / or color correction processing on the image. Global color processing is processing the entire image. Optionally, the first type of parameters may include M parameters, and the M parameters correspond to N types of global color processing, and M and N are both integers greater than or equal to 1. The relationship between the M parameters and the global color processing in N can be one-to-one, one-to-many, or many-to-one, and the embodiment of the present application is not specifically limited. Taking the first type of parameters for performing automatic white balance processing and / or color correction processing on the image as an example, the first type of parameters may include a first parameter corresponding to the automatic white balance processing and / or a second type of parameters corresponding to the color correction processing; the first type of parameters may also include a first parameter corresponding to the automatic white balance processing and a third parameter corresponding to the color correction processing; the first type of parameters may also include a fourth parameter and a fifth parameter corresponding to the automatic white balance processing, and a sixth parameter corresponding to the color correction processing, etc. When the above-mentioned at least one global color processing corresponds to multiple parameters, the embodiment of the present application does not limit the execution order of the at least one global color processing. Optionally, the first type of parameters may be in matrix form. Taking the first type of parameters used for automatic white balance processing and color correction processing of an image as an example, the first type of parameters includes a matrix for automatic white balance processing and a matrix for color correction processing.

[0050] When the first type of parameters are in matrix form, color processing is performed on the image to be processed according to the first type of parameters, which may be performed by performing matrix multiplication on the first type of parameters and the image to be processed.

[0051] For example, automatic white balance processing of the image to be processed can be processed according to the following formula:

[0052]

[0053] Wherein a, b, and c can be determined by a neural network model, R′, G′, and B′ are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the image after the automatic white balance processing, and R, G, and B are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the image before the automatic white balance processing. In the embodiment of the present application, R′, G′, and B′ are uniformly used to represent the values ​​of the color channel R, the color channel G, and the color channel B of the image after one or more global color processings, and R, G, and B represent the values ​​of the color channel R, the color channel G, and the color channel B of the image before the one or more color processings, which will not be explained below.

[0054] For example, the color correction of the image to be processed can be processed according to the following formula:

[0055]

[0056] Among them, R′, G′, and B′ are the values ​​of the color channel R, the value of the color channel G, and the value of the color channel B of the image after color correction, respectively, and R, G, and B are the values ​​of the color channel R, the value of the color channel G, and the value of the color channel B of the image before color correction, respectively.

[0057] The embodiment of the present application can also use the following quadratic terms, cubic terms, square root terms, etc. to perform color correction processing:

[0058] ρ 2，3 =(R, G, B, R 2 , G 2 , B 2 , RG, GB, RB) T

[0059]

[0060]

[0061] Where R, G, and B are the values ​​of the color channel R, color channel G, and color channel B of the image before color correction, respectively, and T represents transposition. Corresponding to the above format, the matrix used for color correction will also have different formats. Taking the color correction matrix with quadratic terms as an example, the color correction of the image to be processed can be processed according to the following formula. At this time, the matrix M used for color correction can be a 3*10 matrix:

[0062]

[0063] Among them, R′, G′, and B′ are the values ​​of the color channel R, the color channel G, and the channel B of the image after color correction, respectively, and R, G, and B are the values ​​of the color channel R, the color channel G, and the color channel B of the image before color correction, respectively.

[0064] It can be understood that the at least one global color processing mentioned above may also correspond to only one matrix, which is not specifically limited in the embodiment of the present application.

[0065] The second type of parameters is used to perform local color processing on the image, wherein the local color processing includes at least one local color processing. Local color processing is processing a part or portion of an image. For example, the second type of parameters is used to perform color enhancement or color rendering processing on a part of an image. Similarly, the relationship between the second type of parameters and at least one local color processing can be one-to-one, one-to-many, or many-to-one, and the embodiments of the present application do not specifically limit it. When the above-mentioned at least one local color processing corresponds to multiple parameters, the embodiments of the present application do not limit the execution order of the at least one local color processing. Optionally, the second type of parameters can be filter function parameters, color adjustment coefficients, etc.

[0066] The embodiment of the present application does not specifically limit the order of performing global color processing and local color processing. Taking the case where global color processing is performed first and the first type of parameters are in matrix form as an example, the image to be processed is matrix multiplied by the first type of parameters to obtain a first image; and the first image is locally color processed according to the second type of parameters to obtain a second image.

[0067] There are many ways to perform local color processing on the image to be processed according to the second type of parameters, which are not specifically limited in the embodiments of the present application. As an example, the difference between the value of the color channel and the value of the brightness channel of the first image is calculated, the difference is adjusted according to the second parameter, and the adjusted difference is added to the value of the brightness channel of the first image to obtain the value of the color channel of the second image.

[0068] There are many ways to adjust the difference value according to the second parameter, which are not specifically limited in the embodiments of the present application. For example, the second type parameter can be multiplied by the difference value, the second type parameter and the difference value can be input into a preconfigured function, the second type parameter can be added to the difference value, etc.

[0069] The embodiment of the present application does not specifically limit the image format of the first image, for example, it can be a color RGB format, a YUV format, etc. Taking the first image in color RGB format as an example, the second type of parameters can include a color processing coefficient of a color channel R, a color processing coefficient of a color channel G, and a color processing coefficient of a color channel B, and the first image can be locally adjusted in color according to the following formula:

[0070]

[0071] Wherein, Y″ is the value of the brightness channel of the first image, R″, G″, and B″ are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the second image, R, G, and B are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the first image, and beta1, beta2, and beta3 are respectively the color processing coefficients of the color channel R, the color processing coefficients of the color channel G, and the color processing coefficients of the color channel B.

[0072] Y″ is generally obtained by Y″=a*R″+b*G″+c*B″, where R″, G″, and B″ are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the image before local color processing.

[0073] In some embodiments, before color processing is performed on the image to be processed according to the first type of parameters and the second type of parameters, some basic processing may be performed on the image to be processed, such as denoising, demosaicing, etc.

[0074] Figure 3 is a specific example of the image processing method of the embodiment of the present application. It should be understood that Figure 3 This is only for illustrative purposes and is intended to help those skilled in the art understand the embodiments of the present application, rather than to limit the embodiments of the present application to the specific scenarios illustrated. Figure 3 As shown, the original image (raw image) is processed by demosaicing and denoising to obtain a linear RGB image. At the same time, the original image will enter the pre-trained neural network model. After being processed by the neural network model, a global color correction matrix M and a local color processing coefficient beta are obtained, where beta can be divided into three channels of R, G, and B, and the size is the same as the original image. The linear RGB image is processed by the global color matrix M to obtain an R″G″B″ image, which is the result of global color correction (for example, automatic white balance and / or color correction), and then after local color processing, an R″′G″′B″′ image after local color processing (for example, color rendering and / or color enhancement) is obtained.

[0075] The above image color processing method focuses on global color processing and local color processing at the same time, and uses the same neural network model to complete all color processing from the raw image of the sensor to the final image. Since the adjustment parameters used in each color processing are obtained using the raw image, firstly, the problem of error accumulation can be avoided, and secondly, since the raw image retains the image information to the greatest extent, the obtained adjustment parameters are more accurate.

[0076] Figures 4 to 6 Three structural forms of the neural network model of the embodiment of the present application are given. It should be understood that: Figures 4 to 6 This is only for the purpose of helping those skilled in the art understand the embodiments of the present application, and is not intended to limit the embodiments of the present application to the specific scenarios illustrated. The neural network model of the embodiments of the present application may also be in other structural forms, as long as it can be a method for implementing the embodiments of the present application.

[0077] Figure 4 The neural network model 400 includes processing such as max pooling 401, convolution 402, deconvolution 403, connection 404, global pooling 405, full connection 406, and reshape 407.

[0078] The role of convolution 402 in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution layer includes multiple convolution operators, which are also called kernels. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels, or three pixels after three pixels, and so on, the number of interval pixels depends on the value of the step length stride) along the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. Deconvolution 403 is also called transposed convolution, which is the inverse process of convolution 402.

[0079] After convolution 402, it is often necessary to periodically introduce pooling processing, which can be a layer of convolution processing followed by a layer of pooling processing, or multiple layers of convolution processing followed by one or more layers of pooling processing. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. Pooling processing may include average pooling using an average pooling operator and / or maximum pooling 401 using a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range within a specific range as the result of maximum pooling. The embodiment of the present application adopts maximum pooling 401.

[0080] The first branch of the neural network model 400 starts with 4 layers of image data of size M*N, and is processed by convolution 402 and maximum pooling 401 to obtain 512 layers of image data of M / 16*N / 16 (i.e., intermediate feature layer data), and is further processed by global pooling 405, full connection 406, and reshaping 407, and finally obtains the first type of parameters. The second branch starts with 4 layers of image data of size M*N, and is processed by convolution 402 and maximum pooling 401 to obtain 512 layers of image data of M / 16*N / 16 (i.e., intermediate feature layer data), and is further processed by deconvolution 403, convolution 402, and connection 404, and finally obtains the second type of parameters. In this way, the inputs of the first branch and the second branch are both 4 layers of image data of size M*N.

[0081] Optionally, the first branch and the second branch of the neural network model 400 can reuse part of the image data starting from 4 layers of size M*N, and obtaining 512 layers of M / 16*N / 16 image data (i.e., intermediate feature layer data) after processing by convolution 402 and maximum pooling 401.

[0082] Figure 5 The neural network model 500 in FIG. 5 includes processing such as max pooling 501, convolution 502, tiling 503, connection 504, global pooling 505, full connection 506, and reshape 507. The functions of each processing can be referred to in Figure 4 The relevant description in will not be repeated here.

[0083] The first branch of the neural network model 500 starts with 4 layers of image data of size M*N, and obtains 512 layers of 1*1 image data (i.e., intermediate feature layer data) after convolution 502, maximum pooling 501, and global pooling 505, and further obtains the first type of parameters after full connection 506, reshaping 507, etc. The second branch starts with 4 layers of image data of size M*N, and obtains 512 layers of 1*1 image data (i.e., intermediate feature layer data) after convolution 502, maximum pooling 501, and global pooling 505, and further obtains 512 layers of M*N image data after tiling 503. At the same time, the second branch also performs convolution 502 on the 4 layers of M*N image data without changing the image size to obtain 512 layers of M*N image data, and connects the two parts of 512 layers of M*N image data through connection 504 to obtain 1024 layers of M*N image data, and further performs convolution 402 to obtain the second type of parameters. In this way, the inputs of the first branch and the second branch are both 4-layer image data of size M*N.

[0084] Optionally, the first branch and the second branch of the neural network model 500 can reuse part of the image data starting from 4 layers of size M*N, and obtaining 512 layers of 1*1 image data (i.e., intermediate feature layer data) after being processed by convolution 502, maximum pooling 501 and global pooling 505.

[0085] Figure 6 The neural network model 600 in FIG. 6 includes processing such as max pooling 601, convolution 602, global pooling 605, full connection 606, and reshape 607. The functions of each processing can be referred to in Figure 4 The relevant description in will not be repeated here.

[0086] The first branch of the neural network model 600 starts with 4 layers of image data of size M*N, and is processed by convolution 602 to obtain 32 layers of M*N image data (i.e., intermediate feature layer data), and is further processed by convolution 602, maximum pooling 601, global pooling 605, full connection 606, reshaping 607, etc., to finally obtain the first type of parameters. The second branch starts with 4 layers of image data of size M*N, and is processed by convolution 602 to obtain 32 layers of M*N image data (i.e., intermediate feature layer data), and is further processed by convolution 602 to obtain the second type of parameters. In this way, the inputs of the first branch and the second branch are both 4 layers of image data of size M*N.

[0087] Optionally, the first branch and the second branch of the neural network model 600 can reuse the part starting from 4 layers of image data of size M*N, and obtaining 32 layers of M*N image data (i.e., intermediate feature layer data) after convolution 602 processing.

[0088] Combine the following Figures 7 to 9 The device or apparatus embodiment of the present application is described.

[0089] Figure 7 is a schematic structural diagram of an image processing device provided in an embodiment of the present application. Figure 7 As shown, the device 700 includes an acquisition module 710 and a processing module 720 .

[0090] The acquisition module 710 is used to acquire the image to be processed.

[0091] The processing module 720 is used to process the image to be processed through the first branch of the pre-trained neural network model to obtain the first type of parameters, and the first type of parameters are used to perform global color processing on the image.

[0092] The processing module 720 is further used to process the image to be processed through the second branch of the neural network model to obtain second type parameters, and the second type parameters are used to perform local color processing on the image.

[0093] The processing module 720 is further configured to perform color processing on the image to be processed according to the first type of parameters and the second type of parameters to obtain a color-processed image.

[0094] Optionally, the image to be processed is a raw image.

[0095] Optionally, the global color processing includes automatic white balance and / or color correction, and the local color processing includes color rendering and / or color enhancement.

[0096] Optionally, the first branch and the second branch share a shared parameter layer of the neural network model.

[0097] It can be understood that, in the case where both the first type of parameters and the second type of parameters require intermediate feature layer data, the layer of the first branch that obtains the intermediate feature layer data and the layer of the second branch that obtains the intermediate feature layer data can share structural parameters. For example, the image to be processed is processed through the shared parameter layer of the pre-trained neural network model to obtain the intermediate feature layer data; the intermediate feature layer data is processed through the first branch of the pre-trained neural network model (excluding the shared parameter layer part) to obtain the first type of parameters; the intermediate feature layer data is processed through the second branch of the neural network model (excluding the shared parameter layer part) to obtain the second type of parameters.

[0098] Optionally, the first type of parameters are in matrix form; the processing module 720 is specifically used to perform matrix multiplication on the image to be processed and the first type of parameters to obtain a first image; and perform local color processing on the first image according to the second type of parameters to obtain a second image.

[0099] Optionally, the processing module 720 is specifically used to calculate the difference between the numerical value of the color channel and the numerical value of the brightness channel of the first image; adjust the difference according to the second type of parameters; and add the adjusted difference to the numerical value of the brightness channel of the first image.

[0100] Optionally, the image format of the first image is a color RGB format, and the second type of parameters includes a color processing coefficient beta1 of a color channel R, a color processing coefficient beta2 of a color channel G, and a color processing coefficient beta3 of a color channel B; the processing module 720 is specifically configured to process the color channel R according to the formula Perform local color adjustment on the first image, where Y″ is the value of the brightness channel of the first image, R″′, G″′, and B″′ are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the second image, and R″, G″, and B″ are respectively the values ​​of the color channel R, the color channel G, and the color channel B of the first image.

[0101] The acquisition module 710 may be implemented by a transceiver or a processor. The processing module 720 may be implemented by a processor. The specific functions and beneficial effects of the acquisition module 710 and the processing module 720 may be found in Figure 2 The method shown will not be described in detail here.

[0102] Figure 8 FIG. 1 is a schematic structural diagram of an image processing device provided by another embodiment of the present application. Figure 8 As shown, the apparatus 800 may include a processor 820 and a memory 830 .

[0103] Figure 8 Only one memory and processor are shown. In an actual image processing device product, there may be one or more processors and one or more memories. The memory may also be referred to as a storage medium or a storage device, etc. The memory may be set independently of the processor or may be integrated with the processor, and the embodiments of the present application do not limit this.

[0104] The processor 820 and the memory 830 communicate with each other through internal connection paths to transmit control and / or data signals.

[0105] Specifically, the processor 820 is used to obtain an image to be processed; to process the image to be processed through a first branch of a pre-trained neural network model to obtain a first type of parameters, wherein the first type of parameters are used to perform global color processing on the image; and to process the image to be processed through a second branch of the neural network model to obtain a second type of parameters, wherein the second type of parameters are used to perform local color processing on the image; and to perform color processing on the image to be processed according to the first type of parameters and the second type of parameters to obtain a color-processed image.

[0106] The memory 830 described in each embodiment of the present application is used to store computer instructions and parameters required for the operation of the processor.

[0107] The specific working process and beneficial effects of the device 800 can be found in Figure 2 The description in the illustrated embodiment will not be repeated here.

[0108] The embodiment of the present application also provides an electronic device, which may be a terminal device. The device may be used to execute the functions / steps in the above method embodiment.

[0109] Fig. 9 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 900 includes a processor 910 and a transceiver 920. Optionally, the electronic device 900 may also include a memory 930. The processor 910, the transceiver 920 and the memory 930 may communicate with each other through an internal connection path to transmit control and / or data signals, and the memory 930 is used to store a computer program, and the processor 910 is used to call and run the computer program from the memory 930.

[0110] Optionally, the electronic device 900 may further include an antenna 940 for transmitting the wireless signal output by the transceiver 920 .

[0111] The processor 910 and the memory 930 may be combined into a processing device, or more commonly, they are independent components. The processor 910 is used to execute the program code stored in the memory 930 to implement the above functions. In specific implementation, the memory 930 may also be integrated into the processor 910, or may be independent of the processor 910. Figure 8 Corresponding to the processing module 820 in the device 800.

[0112] In addition, in order to make the functions of the electronic device 900 more complete, the electronic device 900 may further include one or more of an input unit 960, a display unit 970, an audio circuit 980, a camera 990, and a sensor 901, and the audio circuit may further include a speaker 982, a microphone 984, etc. The display unit 970 may include a display screen.

[0113] Optionally, the electronic device 900 may further include a power supply 950 for providing power to various devices or circuits in the terminal device.

[0114] It should be understood that Fig. 9 The electronic device 900 shown can implement Figures 2 to 6 The various processes of the method embodiment shown. The operations and / or functions of the various modules in the electronic device 900 are respectively to implement the corresponding processes in the above method embodiment. For details, please refer to the description in the above method embodiment. To avoid repetition, the detailed description is appropriately omitted here.

[0115] The processor described in each embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The processor described in each embodiment of the present application may be a general-purpose processor, a digital signal processor (digitalsignal processor, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory (random access memory, RAM), a flash memory, a read-only memory (read-only memory, ROM), a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the instructions in the memory and completes the steps of the above method in combination with its hardware.

[0116] In various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0117] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0123] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0124] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Processing the image to be processed by a first branch of a pre-trained neural network model to obtain first-type parameters, where the first-type parameters are used to perform global color processing on the image; Processing the image to be processed by the second branch of the neural network model to obtain second type parameters, wherein the second type parameters are used to perform local color processing on the image; Color processing is performed on the image to be processed according to the first type of parameters and the second type of parameters to obtain a color-processed image.

2. The method according to claim 1, characterized in that The image to be processed is a raw image.

3. The method according to claim 1 or 2, characterized in that: The global color processing includes automatic white balance and / or color correction, and the local color processing includes color rendering and / or color enhancement.

4. The method according to any one of claims 1 to 3, characterized in that The first branch and the second branch share a shared parameter layer of the neural network model.

5. The method according to any one of claims 1 to 4, characterized in that The first type of parameters are in matrix form; The color processing of the image to be processed according to the first type of parameters and the second type of parameters includes: Performing matrix multiplication on the image to be processed and the first type of parameters to obtain a first image; According to the second type of parameters, local color processing is performed on the first image to obtain a second image.

6. The method according to claim 5, characterized in that The performing local color processing on the first image according to the second type of parameters includes: Calculating the difference between the value of the color channel and the value of the brightness channel of the first image; adjusting the difference according to the second type of parameters; The adjusted difference value is added to the value of the brightness channel of the first image.

7. The method according to claim 6, characterized in that The image format of the first image is a color RGB format, and the second type of parameters includes a color processing coefficient of a color channel R, a color processing coefficient of a color channel G, and a color processing coefficient of a color channel B; The performing local color processing on the first image according to the second type of parameters includes: According to the formula performing local color adjustment on the first image, in is the value of the brightness channel of the first image, , , are respectively the value of the color channel R, the value of the color channel G and the value of the color channel B of the second image, , , They are respectively the value of the color channel R, the value of the color channel G, and the value of the color channel B of the first image.

8. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A processing module, used for processing the image to be processed by a first branch of a pre-trained neural network model to obtain a first type of parameter, wherein the first type of parameter is used for performing global color processing on the image; The processing module is further used to process the image to be processed through the second branch of the neural network model to obtain second type parameters, and the second type parameters are used to perform local color processing on the image; The processing module is further used to perform color processing on the image to be processed according to the first type of parameters and the second type of parameters.

9. The device according to claim 8, characterized in that The image to be processed is an original raw image.

10. The device according to claim 8 or 9, characterized in that The global color processing includes automatic white balance and / or color correction, and the local color processing includes color rendering and / or color enhancement.

11. The device according to any one of claims 8 to 10, characterized in that The first branch and the second branch share a shared parameter layer of the neural network model.

12. The device according to any one of claims 8 to 11, characterized in that The first type of parameters are in matrix form; The processing module is specifically used to perform matrix multiplication on the image to be processed and the first type of parameters to obtain a first image; and perform local color processing on the first image according to the second type of parameters to obtain a second image.

13. The device according to claim 12, characterized in that The processing module is specifically used to calculate the difference between the value of the color channel and the value of the brightness channel of the first image; adjust the difference according to the second type of parameters; and add the adjusted difference to the value of the brightness channel of the first image.

14. The device according to claim 13, characterized in that The image format of the first image is a color RGB format, and the second type of parameters includes a color processing coefficient of a color channel R. beta 1. Color processing coefficient of color channel G beta 2. Color processing coefficient of color channel B beta 3; The processing module is specifically used to The first image is processed Local color adjustment, including is the value of the brightness channel of the first image, , , are respectively the value of the color channel R, the value of the color channel G and the value of the color channel B of the second image, , , They are respectively the value of the color channel R, the value of the color channel G, and the value of the color channel B of the first image.

15. An electronic device, characterized in that: The method comprises a processor and a memory, and is configured to execute the method as claimed in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 7.

Images