Image processing method and device, computer readable storage medium, and terminal

Abstract

An image processing method and device, a computer readable storage medium and a terminal, the method comprising: obtaining a white balance matrix, the white balance matrix comprising a color information dimension and a scene information dimension, elements in the white balance matrix being light source estimation values; obtaining a to-be-processed image and acquisition information of the to-be-processed image; mapping pixel information of the to-be-processed image to the color information dimension to obtain a color vector of the to-be-processed image, and mapping the acquisition information to the scene information dimension to obtain a scene vector of the to-be-processed image; and determining a light source estimation value corresponding to the to-be-processed image in the white balance matrix according to the color vector and the scene vector. The scheme provided in the application is conducive to taking into account the accuracy and adjustability of the white balance algorithm.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus, a computer-readable storage medium, and a terminal. Background Technology

[0002] Automatic White Balance (AWB) technology is commonly used in imaging systems to mimic the human ability to perceive light sources, ensuring that the same color information acquired by the sensor remains consistent under different light sources. For example, the human eye perceives a white sheet of paper as white under different light sources. However, sensors, unlike the human eye, cannot perceive light sources, and therefore acquire different color information for the same white paper under different lighting conditions. Therefore, introducing AWB technology into imaging systems estimates the color of the light source, thereby removing the light source color from the color information acquired by the sensor, resulting in a white image, achieving a function similar to the human eye. Thus, the application of the AWB algorithm is a crucial step in obtaining high-quality images. Summary of the Invention

[0003] The technical problem addressed in this application is how to balance the accuracy and adjustability of the white balance algorithm.

[0004] To address the aforementioned technical problems, this application provides an image processing method, comprising: acquiring a white balance matrix, the white balance matrix including a color information dimension and a scene information dimension, the elements of the white balance matrix being light source estimates; acquiring an image to be processed and its acquisition information; mapping the pixel information of the image to be processed to the color information dimension to obtain a color vector of the image to be processed, and mapping the acquisition information to the scene information dimension to obtain a scene vector of the image to be processed; and determining the light source estimate corresponding to the image to be processed in the white balance matrix based on the color vector and the scene vector.

[0005] Optionally, determining the light source estimate corresponding to the image to be processed in the white balance matrix based on the color vector and the scene vector includes: rounding the mapping vector to obtain an index vector, wherein the mapping vector includes the color vector and the scene vector; searching in the white balance matrix based on the index vector to obtain a reference light source estimate; and interpolating the reference light source estimate based on the distance between the mapping vector and the index vector to obtain the light source estimate.

[0006] Optionally, the white balance matrix includes D sub-matrices, where different sub-matrices represent different scene information, and D is a positive integer greater than 1. Mapping the acquired information to the scene information dimension to obtain the scene vector of the image to be processed includes: extracting features from the acquired information to obtain a scene feature vector; and mapping the scene feature vector to [0, D-1] to obtain the scene vector.

[0007] Optionally, obtaining the white balance matrix includes: Step 1: Inputting the sample image and its acquisition information into a white balance model, the white balance model including a first feature extraction module, a fusion module, a mapping module, an estimation module, and an update module; Step 2: Using the first feature extraction module to extract features from the acquisition information of the sample image to obtain a first sample feature vector; Step 3: Using the fusion module to fuse the first sample feature vector and the sample image feature vector to obtain the current estimation matrix, wherein the sample image feature vector is obtained by the first feature extraction module from the currently input sample image or from the first input sample image; Step 4: Using the mapping module to... Step 5: The pixel information of the sample image is mapped to the color information dimension to obtain a sample color vector, and the acquisition information of the sample image is mapped to the scene information dimension to obtain a sample scene vector; Step 6: The estimation module determines the light source estimation value corresponding to the sample image in the current estimation matrix based on the sample color vector and the sample scene vector; Step 7: The update module determines whether the white balance model has converged based on the change in the loss value between the light source estimation value corresponding to the sample image and the light source label value of the sample image. If the determination result is yes, the current estimation matrix is ​​used as the white balance matrix. If the determination result is no, the white balance model is updated, and the process returns to Step 1.

[0008] Optionally, the light source label value of the sample image includes the light source label values ​​of multiple pixels, and step five includes: determining the light source estimation value corresponding to each pixel in the current estimation matrix according to the sample scene vector and the sample color vector of multiple pixels; calculating the loss value corresponding to each pixel according to the light source estimation value and the light source label value.

[0009] Optionally, the method further includes: calculating the light source estimate corresponding to the verification image based on the white balance matrix; determining whether to correct the white balance matrix based on the error between the light source estimate corresponding to the verification image and the light source label value corresponding to the verification image; and if the determination result is yes, obtaining the light source adjustment value of the verification region in the white balance matrix.

[0010] Optionally, the method further includes: correcting the image to be processed using the light source estimation value corresponding to the image to be processed, to obtain a corrected image.

[0011] This application embodiment also provides an image processing apparatus, the apparatus comprising: a first acquisition module, configured to acquire a white balance matrix, the white balance matrix including a color information dimension and a scene information dimension, the elements in the white balance matrix being light source estimates; a second acquisition module, configured to acquire an image to be processed and acquisition information of the image to be processed; a mapping module, configured to map the pixel information of the image to be processed to the color information dimension to obtain a color vector of the image to be processed, and to map the acquisition information to the scene information dimension to obtain a scene vector of the image to be processed; and an estimation module, configured to determine the light source estimate corresponding to the image to be processed in the white balance matrix based on the color vector and the scene vector.

[0012] This application also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, performs the steps of the image processing method described above.

[0013] This application embodiment also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of the above-described image processing method when running the computer program.

[0014] Compared with the prior art, the technical solution of this application embodiment has the following beneficial effects: In the scheme of this application embodiment, a white balance matrix including color information dimension and scene information dimension is set. On the one hand, a color vector is determined based on the pixel information of the image to be processed; on the other hand, a scene vector is determined based on the scene information of the image to be processed. Then, a light source estimate is determined from the white balance matrix based on the color vector and the scene vector. Compared with the scheme that only determines the light source estimate based on the color information of the image, the scheme provided by this application embodiment takes into account the influence of the image acquisition scene on the AWB algorithm, which can make the obtained light source estimate more accurate. In addition, in the above scheme, the white balance matrix distinguishes between the color information dimension and the scene information dimension. If the accuracy of the white balance matrix in a certain scene is not high, the light source estimate corresponding to that scene can be adjusted by adjusting the scene information dimension of the white balance matrix. Therefore, the scheme provided by this application embodiment has the advantages of being adjustable and easy to adjust, reducing the difficulty of parameter tuning. As a result, the scheme provided by this application embodiment can take into account both the accuracy and adjustability of the white balance algorithm.

[0015] Furthermore, in this embodiment, the index vector is obtained by rounding, and then a lookup is performed in the white balance matrix based on the index vector to obtain the reference light source estimate. Then, based on the distance between the mapping vector and the index vector, the reference light source estimate is interpolated to obtain the light source estimate. This approach helps improve the accuracy of the light source estimate determined based on the white balance matrix. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an image processing method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a white balance model in an embodiment of this application; Figure 3 This is a flowchart illustrating a method for generating a white balance matrix according to an embodiment of this application; Figure 4 This is a partial flowchart of another image processing method in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an image processing device according to an embodiment of this application. Detailed Implementation

[0017] As described in the background section, the accuracy of the current AWB algorithm still needs to be improved.

[0018] Specifically, in some scenarios, the solution to AWB (i.e., the obtained light source value) is not unique. For example, white light shining on a yellow wall and yellow light shining on a white wall will result in very similar color information captured by the sensor when photographing the wall. Assuming the wall color is x, the light source color is y, and the color information obtained by the sensor after the light reaches the wall is z, then the AWB algorithm will yield two solutions: white light and yellow light. To improve the accuracy of the AWB algorithm, the scene in which the image is captured must also be considered. For example, during the design phase of the AWB algorithm, real light source values ​​in multiple scenes are collected. Then, during the application phase, the estimated light source value is inferred based on the experience gained from the pre-collected real light source values ​​in multiple scenes. However, among the pre-collected scenes, there may be multiple possible light sources, requiring manual parameter tuning of the AWB algorithm. When the number of scenes is large, manual parameter tuning becomes very labor-intensive.

[0019] To reduce the workload of manual parameter tuning, machine learning methods can be introduced into the AWB algorithm. Specifically, deep learning is first performed using real light sources from sample images in multiple scenes as labels to obtain a trained model. Then, the trained model is applied to calculate the light source values ​​of the images. However, if the light source values ​​obtained by the model for a certain scene are found to be inaccurate, the model needs to be retrained and optimized using image data from that scene, which is a lengthy process. Therefore, this approach has poor adjustability.

[0020] In view of this, this application provides an image processing method. In this embodiment, a white balance matrix containing color information and scene information dimensions is set. On one hand, a color vector is determined based on the pixel information of the image to be processed; on the other hand, a scene vector is determined based on the scene information of the image to be processed. Then, a light source estimate is determined from the white balance matrix based on the color vector and the scene vector. Compared to a scheme that determines the light source estimate only based on the image's color information, the scheme provided in this application considers the impact of the image acquisition scene on the AWB algorithm, making the obtained light source estimate more accurate. Furthermore, in the above scheme, the white balance matrix distinguishes between color information and scene information dimensions. If the accuracy of the white balance matrix in a certain scene is not high, the light source estimate corresponding to that scene can be adjusted in the scene information dimension of the white balance matrix. Therefore, the scheme provided in this application further has the advantages of being adjustable and easy to adjust, reducing the difficulty of parameter tuning. Thus, the scheme provided in this application can balance the accuracy and adjustability of the white balance algorithm.

[0021] To make the above-mentioned objectives, features and beneficial effects of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0022] Reference Figure 1 , Figure 1 This is a flowchart illustrating an image processing method according to an embodiment of this application. The method can be executed by a terminal, which can be any existing terminal device with data receiving and processing capabilities, such as a mobile phone, computer, IoT device, or server, but is not limited thereto. For example, the terminal can be configured with an image sensor, or the terminal can be coupled to an image sensor to acquire images collected by the image sensor. Figure 1 The image processing method shown may include the following steps: Step S11: Obtain the white balance matrix, which includes color information dimension and scene information dimension, and the elements in the white balance matrix are light source estimates; Step S12: Obtain the image to be processed and the acquisition information of the image to be processed; Step S13: Map the pixel information of the image to be processed to the color information dimension to obtain the color vector of the image to be processed, and map the acquired information to the scene information dimension to obtain the scene vector of the image to be processed; Step S14: Determine the light source estimate corresponding to the image to be processed in the white balance matrix based on the color vector and the scene vector.

[0023] In step S11, the terminal can obtain a white balance matrix. The white balance matrix may include color information dimension and scene information dimension, and the elements in the white balance matrix are all light source estimates.

[0024] Specifically, the white balance matrix in this embodiment can be a multi-dimensional matrix. The white balance matrix includes Q dimensions, where Q is a positive integer greater than 1. Among these Q dimensions, some are scene-related dimensions, and others are color-related dimensions. All elements in the white balance matrix are estimated light source values.

[0025] Those skilled in the art will understand that a light source estimate can be uniquely determined in the white balance matrix based on a set of indices across Q dimensions. The indexes in each dimension can range from 0 to L-1, where L is the number of elements in the dimension. Different indices in the scene information dimension can represent different scenes.

[0026] In one embodiment of this application, the white balance matrix may include D sub-matrices, each representing different scene information. In other words, different sub-matrices correspond to different scene information. That is, different sub-matrices are used to determine the light source estimation values ​​for different scenes. The scene information can refer to the image acquisition scene. For example, the acquisition scene can be a low-light scene, a backlight scene, a bright-light scene, etc., but is not limited to these.

[0027] Furthermore, each submatrix may include at least one dimension, and all dimensions included in the submatrix are color information dimensions.

[0028] In one embodiment of this application, the white balance matrix can be a three-dimensional matrix. Specifically, the white balance matrix can be an M×N×D matrix, where M, N, and D are all positive integers greater than 1. Here, D represents the scene information dimension, and M×N represents the color information dimension. The index range of the D dimension is [0, D-1], the index range of the M dimension is [0, M-1], and the index range of the N dimension is [0, N-1].

[0029] For example, the M×N color information dimensions can represent U chromaticity and V chromaticity respectively. Specifically, U chromaticity can correspond to the U component in the two-dimensional UV color space, and V chromaticity can correspond to the V component in the two-dimensional UV color space.

[0030] In another embodiment of this application, the dimension of the white balance matrix can be greater than 3, wherein the scene information dimension can include at least two dimensions, and the color information dimension can include at least two dimensions.

[0031] For example, a white balance matrix can be a four-dimensional matrix. Specifically, the scene information dimensions can include a first dimension and a second dimension, where the first dimension can be used to represent brightness and the second dimension can be used to represent exposure time; the color information dimensions can include a third dimension and a fourth dimension, where the third dimension can be used to represent U chromaticity and the fourth dimension can be used to represent V chromaticity. Taking a white balance matrix of M×N×D1×D2 as an example, where D1 represents brightness and D2 represents exposure time, different indices on the D1 dimension can represent different brightness levels, and different indices on the D1 dimension can represent different exposure times.

[0032] In practice, the white balance matrix can be pre-stored in a memory accessible to the terminal. Whenever the terminal requires white balance correction, it can read the white balance matrix from the memory to perform the correction.

[0033] For more information on the generation or calculation process of the white balance matrix, please refer to the following section. Figure 2 and Figure 3 Related descriptions.

[0034] In step S12, the terminal acquires the image to be processed and the acquisition information of the image to be processed.

[0035] In a specific implementation, the image to be processed can be an image acquired by an image sensor. This embodiment does not limit the size and format of the image. For example, the image to be processed can be a three-channel image of red (R), green (G), and blue (B).

[0036] Furthermore, the acquired information can be parameters set by the image sensor when acquiring the image to be processed. Non-limitingly, the acquired information may include any one or more of the following: exposure time, sensitivity, and brightness. It should be noted that the brightness in this embodiment may refer to the ambient brightness during image acquisition; in practical applications, the ambient brightness can be calculated based on the pixel information of the image.

[0037] It should be noted that the execution order of steps S11 and S12 is not limited in this embodiment. In practical applications, step S12 can be executed first and then S11, or step S11 can be executed first and then S12, or S11 and S12 can be executed simultaneously.

[0038] In step S13, the terminal maps the pixel information of the image to be processed to the color information dimension to obtain the color vector of the image. The color vector of the image can be used to find the estimated light source value in the color information dimension of the white balance matrix. It should be noted that the color vector may include color feature values, and the number of color feature values ​​is the same as the number of color information dimensions.

[0039] In step S13, the color vectors of multiple pixels in the image to be processed can be determined. For example, it can include the color vectors of all pixels in the image to be processed. Or, for example, the color vectors of the image to be processed can include the color vectors of a subset of pixels (such as valid pixels).

[0040] In one embodiment of this application, the color vector of a pixel can be represented as: .

[0041] Specifically, the U and V chromaticity values ​​of a pixel can be calculated using the following formula: u=log(g / r); v=log(g / b); Where u is the U chromaticity value of the pixel, v is the V chromaticity value of the pixel, r is the pixel value in the R channel, g is the pixel value in the G channel, and b is the pixel value in the B channel.

[0042] Furthermore, the U and V chromaticity values ​​of a pixel can be mapped to the index range of the color information dimension to obtain a color vector.

[0043] For example, as mentioned above, the white balance matrix can be an M×N×D matrix, where M×N represents the color information dimension. The u-value of a pixel can be mapped to [0, M-1] to obtain u_mapped. Similarly, the v-value of a pixel can be mapped to [0, N-1] to obtain v_mapped. Thus, the color vector of the pixel can be obtained. Where 0 ≤ u_mapped ≤ M-1, 0 ≤ v_mapped ≤ N-1. That is, the color feature values ​​include u_mapped and v_mapped respectively.

[0044] In another embodiment of this application, the color vector of a pixel can be obtained based on the color feature vector obtained by feature extraction of the image to be processed. For example, the color feature vector can be calculated using the feature extraction module in the white balance model described below, and then the color feature vector can be mapped to the index range of the color information dimension. The dimension or size of the color feature vector and the color vector can be the same.

[0045] As will be understood by those skilled in the art, the color vector of a pixel can also be a vector representation of the pixel in other color spaces.

[0046] On the other hand, the terminal maps the acquired information of the image to be processed to the scene information dimension to obtain the scene vector of the image. This scene vector can be used to find the estimated light source value in the scene information dimension of the white balance matrix. It should be noted that the scene vector can include scene feature values, and the number of scene feature values ​​is the same as the number of scene information dimensions.

[0047] Specifically, feature extraction can be performed on the acquired information of the image to be processed to obtain a scene feature vector. In a practical implementation, the second feature extraction unit of the white balance model described below can be used to extract features from the acquired information of the image to be processed.

[0048] Furthermore, the scene feature vector can be mapped to the index range of the scene information dimension to obtain the scene vector.

[0049] If the scene information dimension has only one dimension, the mapped scene vector is a single scene feature value. For example, as described above, the white balance matrix can include D submatrices, where D is the scene information dimension. The scene feature vector can be mapped to [0, D-1] to obtain a single scene feature value, which can be denoted as d_mapped, where 0 ≤ d_mapped ≤ D-1.

[0050] If the scene information has multiple dimensions, the scene feature vector can be mapped to the index range of each dimension to obtain the scene vector. For example, if the scene information has the first and second dimensions mentioned above, the scene vector can be represented as: , where 0≤d1_mapped≤D1 -1, 0≤d2_mapped≤D2 -1.

[0051] As described above, step S13 yields the color vector and scene vector of the image to be processed. The color vector may include multiple color feature values, and the scene vector may include at least one scene feature value. The number of color feature values ​​in the color vector depends on the number of color information dimensions of the white balance matrix, and the number of color feature values ​​in the scene vector depends on the number of scene information dimensions of the white balance matrix. The feature values ​​obtained in step S13 and the dimensions of the white balance matrix can be in one-to-one correspondence, and the feature values ​​can be used to find the light source estimate in the corresponding dimension.

[0052] In step S14, the terminal determines the estimated light source value corresponding to the image to be processed in the white balance matrix based on the color vector and scene vector of the image to be processed. For ease of description, the color vector and scene vector will be collectively referred to as the mapping vector in the following sections. That is, the mapping vector includes the color vector and the scene vector, and the mapping vector includes the eigenvalues ​​in the color vector and the eigenvalues ​​in the scene vector.

[0053] In one embodiment of this application, in step S13, mapping vectors corresponding to multiple pixels can be obtained, and then in step S14, the light source estimation value corresponding to each pixel can be determined according to the mapping vectors corresponding to each pixel.

[0054] In another embodiment of this application, in step S13, the mapping vector (i.e., one mapping vector) corresponding to the image to be processed can be obtained, and then in step S14, the light source estimation value corresponding to the image to be processed can be determined according to the mapping vector corresponding to the image to be processed.

[0055] In practical implementation, the mapping vector can be used as an index to determine the estimated light source value in the white balance matrix. In real-world applications, the eigenvalues ​​in the color vector and scene vector obtained in step S13 may not be integers. In this case, the solution in this embodiment can round the mapping vector to obtain the index vector.

[0056] For example, the eigenvalues ​​in the mapping vector can be rounded up or down to obtain an index vector. Then, the corresponding light source estimate can be found in the white balance matrix based on the index vector. For instance, the light source estimate found using the index vector can be directly used as the light source estimate for the image to be processed.

[0057] For another example, the mapping vector can be rounded up and down to obtain multiple index vectors. Then, multiple reference light source estimates can be obtained by searching within the white balance matrix using these index vectors. Finally, interpolation is performed based on the distance between the mapping vector and the index vectors, along with the multiple reference light source estimates, to obtain the light source estimate, thereby improving the accuracy of the light source estimate. Here, the reference light source estimate refers to the light source estimate obtained by searching within the white balance matrix using the index vectors.

[0058] For example, the mapping vector can be rounded up to obtain a first index vector, and the estimated value of the first reference light source can be determined by searching the white balance matrix using the first index vector. Similarly, the mapping vector can be rounded down to obtain a second index vector, and the estimated value of the second reference light source can be determined by searching the white balance matrix using the second index vector. Furthermore, based on the ratio of the distance between the mapping vector and the first index vector to the distance between the first and second index vectors, interpolation can be performed between the estimated values ​​of the first and second reference light sources to obtain the light source estimated value corresponding to the mapping vector.

[0059] In practice, each eigenvalue in the mapping vector can be rounded up and down to obtain... There are index vectors. Where Q is the number of eigenvalues ​​in the mapping vector, based on... A single index vector can be used to determine... One reference light source estimate. Further, based on... The estimated values ​​of the reference light source are interpolated in multiple directions to obtain the light source estimated value corresponding to the mapping vector.

[0060] The following example uses a white balance matrix of M×N×D to illustrate the specific process of step S14 in a non-restrictive manner.

[0061] Specifically, taking an M×N×D white balance matrix as an example, the mapping vector can be [u_mapped, v_mapped, d_mapped]. Rounding up and down on u_mapped, v_mapped, and d_mapped respectively yields u_mapped_up, u_mapped_down, v_mapped_up, u_mapped_down, para_mapped_up, and para_mapped_down. This results in eight index vectors. These index vectors can then be used to search within the white balance matrix to obtain a set of reference light source estimates (i.e., eight reference light source estimates).

[0062] Referring to Table 1, Table 1 contains a set of index vectors obtained by rounding and a set of estimated reference light sources obtained by searching.

[0063]

[0064] Furthermore, based on the distances between the mapping vectors [u_mapped, v_mapped, d_mapped] and each index vector, multi-directional interpolation can be performed on the set of reference light source estimates to obtain the light source estimates corresponding to the mapping vectors [u_mapped, v_mapped, d_mapped]. After step S14, the light source estimates obtained in step S14 can be used to correct the image to be processed, resulting in a corrected image.

[0065] In one embodiment of this application, step S14 obtains the light source estimation values ​​corresponding to each of the multiple pixels. Further, the terminal can perform a weighted calculation based on the light source estimation values ​​corresponding to the multiple pixels to obtain the light source estimation value corresponding to the image to be processed. Therefore, the terminal can remove the light source estimation value corresponding to the image to be processed from the color information of each pixel in the image to be processed, thereby obtaining the corrected image.

[0066] In another embodiment of this application, step S13 calculates the light source estimation value corresponding to each of the multiple pixels. In step S14, the terminal can remove the light source estimation value corresponding to each pixel from the color information of the multiple pixels to obtain the corrected color information, thereby obtaining the corrected image.

[0067] Reference Figure 2 , Figure 2 This is a schematic diagram of the structure of a white balance model according to an embodiment of this application. Figure 3 This is a flowchart illustrating a method for generating a white balance matrix according to an embodiment of this application. The following is a summary... Figure 2 and Figure 3 A method for generating the white balance matrix in one embodiment of this application is described. In the scheme of this embodiment, the white balance matrix can be obtained by training the white balance model using supervised training.

[0068] like Figure 2 As shown, the white balance model may include a first feature extraction module 21, a fusion module 22, a mapping module 23, an estimation module 24, and an update module 25. The first feature extraction module 21 includes a first feature extraction unit 211 and a second feature extraction unit 212.

[0069] Figure 3 The method shown may include the following steps: Step S31: Input the sample image and the acquisition information of the sample image into the white balance model.

[0070] Specifically, the type of the sample image and the type of the image to be processed can be the same. For example, the sample image can be a three-channel image, but this embodiment is not limited to this.

[0071] Furthermore, the sample images can be preprocessed. For example, de-mosaic color interpolation and downsampling can be performed on the sample images. This approach helps reduce the size of the input data for the white balance model and improves the efficiency of training the white balance model. Specific details regarding the acquisition information of the sample images can be found in the description of the acquisition information of the images to be processed above, and will not be repeated here.

[0072] In addition, the sample image may include a color chart, which can be used to calculate the light source label value of the sample image. The light source label value can be considered as the ground truth for supervision.

[0073] Step S32: The first feature extraction unit 211 is used to extract features from the collected information of the sample image to obtain the first sample feature vector, and the second feature extraction unit 212 is used to extract features from the sample image to obtain the sample image feature vector.

[0074] Specifically, the first feature extraction unit 211 may include multiple convolutional layers, and the second feature extraction unit 212 may also include multiple convolutional layers. This application embodiment does not limit the specific structure of the first feature extraction unit 211 and the second feature extraction unit 212. For ease of description, the first sample feature vector is denoted as Fpara, and the sample image feature vector is denoted as Frgb. For example, Fpara can be a one-dimensional vector of length k, where k is a positive integer greater than 1, and Frgb is a multi-dimensional vector, which can be an n×n×c vector. The specific values ​​of n, c, and k depend on the structure of the first feature extraction unit 211 and the second feature extraction unit 212.

[0075] Step S33: The fusion module 22 is used to fuse the feature vector of the first sample and the feature vector of the sample image to obtain the current estimation matrix.

[0076] The dimensions of the current estimation matrix and the white balance matrix can be the same. That is, the current estimation matrix can include color information dimensions and scene information dimensions. It should be noted that the dimensions and size of the white balance matrix in this embodiment can be predefined. For example, the white balance matrix can be M×N×D.

[0077] Specifically, the fusion module 22 can first stretch Frgb into a one-dimensional vector, and then concatenate the one-dimensional Frgb with Fpara to obtain a concatenated one-dimensional vector. Further, the concatenated one-dimensional vector can be transformed into an M×N×D matrix. In specific implementations, the fusion module 22 may include fully connected layers. The input to the fully connected layer can be the concatenated one-dimensional vector, and the output of the fully connected layer can be the current estimated matrix. This embodiment does not limit the number of fully connected layers included in the fusion module 22.

[0078] Step S34: The mapping module 23 maps the pixel information of the sample image to the color information dimension to obtain the sample color vector, and maps the acquisition information of the sample image to the scene information dimension to obtain the sample scene vector.

[0079] Specifically, the first sample feature vector can be mapped to the scene information dimension to obtain the sample scene vector.

[0080] In addition, the u and v chromaticity values ​​can be calculated based on the RGB values ​​of pixels in the sample image, and then the u and v chromaticity values ​​can be mapped to the index range of the color information dimension to obtain the sample color vector.

[0081] In other embodiments, a sample color feature vector can be obtained by feature extraction, and then the sample color feature vector can be mapped to the index range of the color information dimension to obtain a sample color vector.

[0082] For example, feature extraction can be performed on the feature vector of the sample image to obtain the sample color feature vector. The feature extraction module may also include a third feature extraction unit ( Figure 2 (Not shown), the input of the third feature extraction unit is connected to the output of the first feature extraction unit, and the output of the third feature extraction unit is connected to the input of the mapping module 23. The third feature extraction unit can extract features from the feature vector of the sample image to obtain the sample color feature vector. In this case, the sample image does not need to be directly input to the mapping module 23.

[0083] For example, features can be directly extracted from the sample image to obtain the sample color feature vector. For instance, the feature extraction module may also include a fourth feature extraction unit (…). Figure 2 (Not shown), the input of the fourth feature extraction unit is the sample image, and the output of the fourth feature extraction unit is connected to the input of the mapping module 23. The fourth feature extraction unit can extract features from the sample image to obtain a sample color feature vector. In this case, the sample image does not need to be directly input to the mapping module 23. Unlike the second feature extraction unit, the dimension of the sample color feature vector output by the fourth feature extraction unit is the same as the dimension of the sample color vector.

[0084] For more information on step S34, please refer to [link / reference]. Figure 1 The relevant description of step S13.

[0085] Step S35: The estimation module 24 determines the light source estimation value corresponding to the sample image in the current estimation matrix based on the sample color vector and the sample scene vector.

[0086] For details regarding step S35, please refer to the description of step S14 above, which will not be repeated here.

[0087] Therefore, the light source estimate corresponding to the sample image can be obtained. This light source estimate can be calculated based on the light source estimates corresponding to multiple pixels in the sample image, or it can include the light source estimates corresponding to multiple pixels in the sample image.

[0088] Step S36: The update module 25 determines whether the white balance model has converged based on the change in the loss value between the light source estimation value corresponding to the sample image and the light source label value of the sample image. If the determination result is yes, the current estimation matrix is ​​used as the white balance matrix. If the determination result is no, the white balance model is updated and the process returns to step S31.

[0089] In the embodiments of this application, a loss function is used to supervise the training process of the white balance model. It should be noted that the embodiments of this application do not limit the type of loss function used; for example, it can be an L1 loss function, an L2 loss function, etc.

[0090] The solution in this embodiment uses gradient descent to train the white balance model. Specifically, in step S36, a loss value can be calculated based on the loss function, the light source label value, and the light source estimate obtained in step S35. Then, the white balance model is judged to have converged based on the change in the loss value. For example, if the calculated loss value is still decreasing, it can be determined that the model has not yet converged; if the calculated loss value remains stable and no longer decreases, it can be determined that the model has converged.

[0091] In one embodiment of this application, a light source estimation value corresponding to a sample image can be calculated based on the light source estimation values ​​of multiple pixels in the sample image, and then a loss value used to determine whether convergence has been calculated based on the light source estimation value corresponding to the sample image and the light source label value.

[0092] In another embodiment of this application, the light source label value of the sample image may include the light source label values ​​of multiple pixels. In step S36, the loss value corresponding to each pixel can be calculated based on the light source estimation value and the light source label value corresponding to each pixel. Then, the loss value used to determine whether convergence has been achieved is calculated based on the loss value of each pixel. This approach helps to ensure the accuracy of the trained white balance matrix in mixed lighting scenes.

[0093] Furthermore, if the white balance model is determined to have converged based on the change in the calculated loss value, the current estimated matrix can be used as the white balance matrix. If the white balance model has not yet converged based on the change in the loss value, the white balance model can be updated, and then the process returns to step S31. Updating the white balance model may include updating the first feature extraction module 21 and the fusion module 22 in the white balance model.

[0094] Specifically, the current batch of sample data can be used to return to step S31 to continue training the white balance model, or the next batch of sample images can be obtained and returned to step S31 for training.

[0095] From the above, through Figure 3 The method shown is for Figure 2 The white balance model shown can be trained under supervision to obtain the white balance matrix.

[0096] Reference Figure 4 , Figure 4 This is a flowchart illustrating another image processing method in an embodiment of this application. Figure 4 The illustrated scheme may include steps S41 to S43.

[0097] Step S41: Calculate the light source estimate corresponding to the verification image based on the white balance matrix.

[0098] For details regarding step S41, please refer to the above text. Figure 1 The relevant descriptions will not be repeated here.

[0099] Step S42: Based on the error between the estimated light source value corresponding to the verification image and the light source label value corresponding to the verification image, determine whether to correct the white balance matrix.

[0100] Specifically, if the error between the light source label value and the actual light source label value of the verified image is greater than or equal to the error threshold, then the white balance matrix needs to be corrected. If the error is less than the error threshold, then the white balance matrix does not need to be corrected.

[0101] Step S43: If the judgment result is yes, then obtain the light source adjustment value of the verification area in the white balance matrix.

[0102] Specifically, in step S41, a color vector can be determined based on the pixel information of the verification image, and a scene vector can be determined based on the acquisition information of the verification image. Then, an index vector is determined based on the color vector and the scene vector. Further, a reference light source estimate is determined in the white balance matrix based on the index vector, and the verification region can include the determined reference light source estimate. For example, the size of the verification region can be predetermined. Alternatively, the verification region can be determined by the developers or maintainers of the white balance model.

[0103] Furthermore, the estimated light source values ​​for the verification area can be adjusted based on the obtained light source adjustment values ​​to obtain an optimized white balance matrix. In a specific embodiment, the light source adjustment values ​​can be input by the developers or maintainers of the white balance model. Alternatively, the light source adjustment values ​​can be determined based on the error obtained in step S42.

[0104] Therefore, in the solution of this application embodiment, a verification image can be used to verify the light source estimation value in the white balance matrix. If the error of the light source estimation value obtained from the white balance matrix is ​​large, the area to be adjusted (i.e., the verification area) can be determined in the white balance matrix, and then the light source estimation value in that area can be corrected. Therefore, the solution of this application embodiment has the advantages of being adjustable and easy to adjust.

[0105] It is understandable that, in specific implementation, the above method can be implemented using a software program that runs in the processor integrated inside the chip or chip module; or, the method can be implemented using hardware or a combination of hardware and software, such as using a dedicated chip or chip module, or using a dedicated chip or chip module combined with a software program.

[0106] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of an image processing device according to an embodiment of this application. Figure 5 The apparatus shown may include: The first acquisition module 51 is used to acquire a white balance matrix, which includes a color information dimension and a scene information dimension, and the elements in the white balance matrix are light source estimates. The second acquisition module 52 is used to acquire the image to be processed and the acquisition information of the image to be processed; The mapping module 53 is used to map the pixel information of the image to be processed to the color information dimension to obtain the color vector of the image to be processed, and to map the acquired information to the scene information dimension to obtain the scene vector of the image to be processed. The estimation module 54 is used to determine the light source estimation value corresponding to the image to be processed in the white balance matrix based on the color vector and the scene vector.

[0107] For more information on the working principle, working method, and beneficial effects of the image processing device in the embodiments of this application, please refer to the relevant description of the image processing method above, which will not be repeated here.

[0108] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it performs the steps of the image processing method described above. The storage medium may include ROM, RAM, a magnetic disk, or an optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0109] This application also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the image processing method described above. The terminal includes, but is not limited to, terminal devices such as mobile phones, computers, and tablet computers.

[0110] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0111] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0112] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0115] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0116] In this application's embodiments, "multiple" refers to two or more. The descriptions of "first," "second," etc., appearing in this application's embodiments are merely illustrative and for distinguishing the described objects; they do not indicate any order and do not imply a specific limitation on the number of devices in this application's embodiments, nor do they constitute any limitation on the embodiments of this application. Although this application discloses the above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

[0117] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. An image processing method, characterized by, The method includes: Obtain a white balance matrix, which includes color information dimension and scene information dimension, and the elements in the white balance matrix are light source estimates; Acquire the image to be processed and the acquisition information of the image to be processed; The pixel information of the image to be processed is mapped to the color information dimension to obtain the color vector of the image to be processed, and the acquired information is mapped to the scene information dimension to obtain the scene vector of the image to be processed. Based on the color vector and the scene vector, the estimated light source value corresponding to the image to be processed is determined in the white balance matrix.

2. The image processing method of claim 1, wherein, Determining the estimated light source value corresponding to the image to be processed in the white balance matrix based on the color vector and the scene vector includes: The mapping vector is rounded down to obtain the index vector, wherein the mapping vector includes the color vector and the scene vector; The reference light source estimate is obtained by searching the white balance matrix according to the index vector; the light source estimate is then interpolated according to the distance between the mapping vector and the index vector to obtain the light source estimate.

3. The image processing method of claim 1, wherein, The white balance matrix comprises D sub-matrices, where different sub-matrices represent different scene information, and D is a positive integer greater than 1. Mapping the acquired information to the scene information dimension yields the scene vector of the image to be processed, which includes: The collected information is used to extract features to obtain a scene feature vector; The scene feature vector is mapped to [0, D-1] to obtain the scene vector.

4. The image processing method of claim 1, wherein, Obtaining the white balance matrix includes: Step 1: Input the sample image and the acquisition information of the sample image into the white balance model. The white balance model includes a first feature extraction module, a fusion module, a mapping module, an estimation module, and an update module. Step 2: Use the first feature extraction module to extract features from the collected information of the sample image to obtain the first sample feature vector; Step 3: The fusion module is used to fuse the first sample feature vector and the sample image feature vector to obtain the current estimation matrix, wherein the sample image feature vector is obtained by the first feature extraction module from the current input sample image or the first input sample image; Step 4: The mapping module is used to map the pixel information of the sample image to the color information dimension to obtain a sample color vector, and the acquisition information of the sample image is mapped to the scene information dimension to obtain a sample scene vector; Step 5: The estimation module determines the light source estimate value corresponding to the sample image in the current estimation matrix based on the sample color vector and the sample scene vector; Step Six: The update module determines whether the white balance model has converged based on the change in the loss value between the estimated light source value corresponding to the sample image and the light source label value of the sample image. If the determination result is yes, the current estimation matrix is ​​used as the white balance matrix. If the determination result is no, the white balance model is updated and the process returns to Step One.

5. The image processing method of claim 4, wherein, The light source label value of the sample image includes the light source label values ​​of multiple pixels, and step five includes: Based on the sample scene vector and the sample color vectors of multiple pixels, the light source estimation value corresponding to each pixel is determined in the current estimation matrix; The loss value for each pixel is calculated based on the estimated light source value and the light source label value for each pixel.

6. The image processing method of claim 1, wherein, The method further includes: Calculate the light source estimate corresponding to the verification image based on the white balance matrix; Based on the error between the estimated light source value corresponding to the verification image and the light source label value corresponding to the verification image, determine whether to correct the white balance matrix; If the judgment result is yes, then obtain the light source adjustment value of the verification area in the white balance matrix.

7. The image processing method of claim 1, wherein, The method further includes: The image to be processed is corrected using the light source estimation value corresponding to the image to be processed, and the corrected image is obtained.

8. An image processing apparatus characterized by comprising: The device includes: The first acquisition module is used to acquire a white balance matrix, which includes a color information dimension and a scene information dimension, and the elements in the white balance matrix are light source estimates. The second acquisition module is used to acquire the image to be processed and the acquisition information of the image to be processed; The mapping module is used to map the pixel information of the image to be processed to the color information dimension to obtain the color vector of the image to be processed, and to map the acquired information to the scene information dimension to obtain the scene vector of the image to be processed. An estimation module is used to determine the light source estimation value corresponding to the image to be processed in the white balance matrix based on the color vector and the scene vector.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is run by the processor, it performs the steps of the image processing method according to any one of claims 1 to 7.

10. A terminal comprising a memory and a processor, said memory having stored thereon a computer program that is operable on said processor, characterized in that, When the processor runs the computer program, it performs the steps of the image processing method according to any one of claims 1 to 7.

Images