Method for determining color correction matrix, color correction method and electronic device

Abstract

The invention discloses a method for determining a color correction matrix, a color correction method and an electronic device. The method of determining a color correction matrix may include: receiving a first reference image generated by photographing a first object under a first illumination condition using a reference camera; receiving a first image generated by photographing a first object under a first illumination condition using a first camera; determining a first color correction matrix of the first camera using the first reference image, the first image, and a first reference color correction matrix of the reference camera, where the first color correction matrix and the first reference color correction matrix correspond to a first illumination condition; the actual image is corrected using the first color correction matrix.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more particularly, to a method for determining a color correction matrix, a color correction method and an electronic device. Background Art

[0002] The color correction matrix (CCM) can be used for color correction in image processing. The existing method for determining the color correction matrix is ​​that the debugging personnel use the debugging tool to visually adjust the color correction matrix so that the actual image captured by the camera corrected by the color correction matrix is ​​close to the standard image. However, the process of adjusting the color correction matrix using the debugging tool is very time-consuming (for example, it may take several hours to determine the color correction matrix corresponding to one lighting condition).

[0003] Therefore, there is a need for a method that can improve the efficiency of determining a color correction matrix. Summary of the invention

[0004] This Summary is provided to introduce in simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features and / or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] The present application provides a method for determining a color correction matrix, a color correction method, and an electronic device, which can at least improve the efficiency of determining the color correction matrix.

[0006] According to one or more example embodiments, a method for determining a color correction matrix may include: receiving a first reference image generated by photographing a first object under a first lighting condition using a reference camera; receiving a first image generated by photographing the first object under the first lighting condition using a first camera; determining a first color correction matrix of the first camera using the first reference image, the first image, and a first reference color correction matrix of the reference camera, wherein the first color correction matrix and the first reference color correction matrix correspond to the first lighting condition; and correcting an actual image using the first color correction matrix.

[0007] Compared with the solution of using a debugging tool to determine the color correction matrix, in the present application, a first reference image of a reference camera and a first reference color correction matrix are used to calculate a first color correction matrix of a first camera, thereby improving the efficiency of determining the color correction matrix.

[0008] Determining a first color correction matrix of the first camera corresponding to the first lighting condition may include determining the first color correction matrix to minimize a difference between a first image corrected using the first color correction matrix and a first reference image corrected using a first reference color correction matrix.

[0009] The operation of determining the first color correction matrix may include determining the first color correction matrix by correcting the first reference color correction matrix using a first correction term representing a first RGB matrix of the first image and a first reference RGB matrix of the first reference image.

[0010] The method may also include determining a second color correction matrix of the first camera corresponding to the second lighting condition by correcting a second reference color correction matrix of the reference camera corresponding to the second lighting condition using a first correction term representing the first RGB matrix of the first image and the first reference RGB matrix of the first reference image. The second lighting condition may be different from the first lighting condition.

[0011] The first correction term may represent an inverse of a product of a first transposed matrix of the first RGB matrix and the first RGB matrix, a product of the first transposed matrix and a first reference RGB matrix.

[0012] The method may also include: determining a reference spectral response function of a reference camera: determining a first spectral response function of a first camera; predicting a second reference image of the reference camera using the reference spectral response function, wherein the second reference image corresponds to a second lighting condition; predicting a second image of the first camera using the first spectral response function, wherein the second image corresponds to the second lighting condition; determining a second color correction matrix of the first camera using the second reference image, the second image, and a second reference color correction matrix of the reference camera, wherein the second color correction matrix and the second reference color correction matrix correspond to the second lighting condition. The second lighting condition may be different from the first lighting condition.

[0013] Compared with determining the color correction matrix corresponding to each lighting condition through a debugging tool, in the present application, the reference image and image corresponding to the remaining lighting conditions of the reference camera and the first camera are predicted by using the spectral response functions of the reference camera and the first camera and the remaining lighting conditions except the first lighting condition, and the predicted reference image and image are used to calculate the color correction matrix corresponding to the remaining lighting conditions of the first camera, which can reduce the time for photographing the first object under the remaining lighting conditions, thereby further improving the efficiency of determining the color correction matrix.

[0014] The reference spectral response function may be a polynomial matrix using a first reference RGB matrix of the first reference image as a variable.

[0015] The 1st-order coefficient matrix to the nth-order coefficient matrix of the polynomial matrix can be determined in the order from the 1st-order coefficient matrix to the nth-order coefficient matrix. The i-th-order coefficient matrix of the polynomial matrix can be determined in the following manner: obtaining an updated polynomial matrix corresponding to the i-th-order coefficient matrix; determining the i-th-order coefficient matrix by using the updated polynomial matrix corresponding to the i-th-order coefficient matrix and the camera response function. n can be an integer greater than 1, and i is an integer greater than or equal to 1 and less than or equal to n. The updated polynomial matrix corresponding to the 1st-order coefficient matrix can be obtained by setting the 2nd-order coefficient matrix to the nth-order coefficient matrix of the original polynomial matrix to the zero matrix. The updated polynomial matrix corresponding to the m-th-order coefficient matrix can be obtained by substituting the previously determined 1st-order coefficient matrix to the m-1th-order coefficient matrix into the original polynomial matrix and determining the m+1th-order coefficient matrix to the nth-order coefficient matrix of the original polynomial matrix as the zero matrix, where m is an integer greater than 1 and less than n. An updated polynomial matrix corresponding to the n-th order coefficient matrix may be obtained by substituting the previously determined 1st order coefficient matrix to the n-1th order coefficient matrix into the original polynomial matrix. In the original polynomial matrix, the 1st order coefficient matrix to the n-th order coefficient matrix may not be determined.

[0016] The camera response function may be based on a relationship between a first reference RGB matrix and a spectral reflectance matrix of the first object, a first spectral power distribution corresponding to the first lighting condition, and a reference spectral response function.

[0017] The operation of determining the second color correction matrix may include determining the second color correction matrix so as to minimize a difference between the second image corrected using the second color correction matrix and the second reference image corrected using the second reference color correction matrix.

[0018] The first lighting condition may represent a standard lighting condition that makes illumination uniform across each region of the first object.

[0019] The first object may be a standard color card.

[0020] According to one or more example embodiments, a color correction method includes: receiving a first reference image generated by photographing a first object under a first lighting condition using a reference camera; receiving a first image generated by photographing the first object under the first lighting condition using a first camera; determining a first color correction matrix of the first camera using the first reference image, the first image, and a first reference color correction matrix of the reference camera, wherein the first color correction matrix and the first reference color correction matrix correspond to the first lighting condition; receiving a first actual image generated by photographing a second object under the first lighting condition using the first camera; and correcting the first actual image using the first color correction matrix. The first object may be the same as or different from the second object.

[0021] An electronic device may include: at least one processor configured to execute the method.

[0022] A non-transitory computer-readable storage medium may store a computer program, wherein the method is implemented when the computer program is executed by at least one processor.

[0023] According to one or more example embodiments, a color correction method may include: receiving a first reference image generated by photographing a first object under a first lighting condition using a reference camera; receiving a first image generated by photographing the first object under the first lighting condition using a first camera; determining a first color correction matrix of the first camera by determining the first color correction matrix so that a difference between the first image corrected using the first color correction matrix and the first reference image corrected using the first reference color correction matrix is ​​minimized using the first reference image, the first image, and a first reference color correction matrix of the reference camera, wherein the first color correction matrix and the first reference color correction matrix correspond to the first lighting condition; receiving a first actual image generated by photographing a second object under the first lighting condition using the first camera; and correcting the first actual image using the first color correction matrix. The first object may be the same as or different from the second object.

[0024] Additional aspects and / or advantages of the inventive concepts will be set forth in part in the description which follows and, in part, will be apparent from the description, and / or may be learned by practice of various example embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings;

[0026] Figure 1 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments;

[0027] Figure 2 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments;

[0028] Figure 3 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments;

[0029] Figure 4 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments;

[0030] Figure 5 is a flow chart illustrating a method of determining a spectral response function according to one or more embodiments;

[0031] Figure 6 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments;

[0032] Figure 7 is a flow chart illustrating a color correction method according to one or more embodiments;

[0033] Figure 8 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments; and

[0034] Fig. 9 A block diagram of an electronic device according to one or more embodiments is shown. DETAILED DESCRIPTION

[0035] The following detailed description is provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, except for the operations that must be sent in a specific order, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0036] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0037] The structural or functional description of the examples disclosed herein below is intended only for the purpose of describing the examples, and the examples may be implemented in various forms. The examples are not intended to be limiting, but various modifications, equivalents, and substitutes are also intended to be included within the scope of the claims.

[0038] Although the terms "first" or "second" are used to explain various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another component. For example, within the scope of the rights according to the concept of the present disclosure, a "first" component may be referred to as a "second" component, or similarly, a "second" component may be referred to as a "first" component.

[0039] It will be understood that when a component is referred to as being “connected to” another component, the component may be directly connected or coupled to the other component or intervening components may be present.

[0040] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. It should also be understood that when the terms "include" and / or "comprising" are used in this specification, it indicates the presence of the stated features, integers, operations, operations, elements, components or combinations thereof, but does not preclude the presence or addition of one or more other features, integers, operations, operations, elements, components and / or groups thereof.

[0041] Unless otherwise defined, all terms (including technical or scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the examples belong. It will also be understood that, unless expressly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense.

[0042] Hereinafter, examples will be described in detail with reference to the accompanying drawings. Regarding reference numerals assigned to elements in the drawings, it should be noted that the same elements will be denoted by the same reference numerals and redundant descriptions thereof will be omitted.

[0043] Figure 1 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments.

[0044] like Figure 1 As shown in FIG. 1 , in operation S110 , a lighting condition may be set.

[0045] In operation S120, a specific object (eg, a standard color chart) may be photographed under a set lighting condition using the first camera to obtain an actual image, and a standard image corresponding to the set lighting condition may be obtained.

[0046] In operation S130, the actual image may be compared with the standard image, and a color correction matrix (CCM) of the first camera corresponding to the set lighting condition may be determined based on the comparison result. Generally, it may take several hours to perform operation S130.

[0047] In operation S140, it may be determined whether the first camera has photographed the specific object under all lighting conditions.

[0048] When it is determined that the first camera does not photograph the specific object under all lighting conditions (No in operation S140), in operation S150, the lighting condition may be changed to select an unused lighting condition from among all lighting conditions, and after operation S150, the process may proceed to operation S120.

[0049] When it is determined that the first camera has photographed the specific object under all lighting conditions (Yes in operation S140 ), the process of determining the color correction matrix may be ended.

[0050] Figure 2 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments. Figure 2 yes Figure 1 Detailed description of operation S130.

[0051] like Figure 2 As shown in , in operation S210, a color space transformation may be performed on the actual image to obtain a transformed actual image, and a color space transformation may be performed on the standard image to obtain a transformed standard image.

[0052] In operation S220, a color correction matrix may be determined based on the transformed actual image and the transformed standard image using a debugging tool.

[0053] In operation S230, the transformed actual image may be corrected using a color correction matrix to obtain a corrected actual image.

[0054] In operation S240, it may be determined whether a difference between the corrected actual image and the transformed standard image satisfies a predetermined condition.

[0055] When it is determined that the difference between the corrected actual image and the transformed standard image does not satisfy the predetermined condition (NO in operation S240 ), the process may proceed to operation S220 .

[0056] When it is determined that the difference between the corrected actual image and the transformed standard image satisfies the predetermined condition (Yes in operation S240 ), the process may proceed to operation S140 .

[0057] Figure 3 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments. Figure 3 is a flow chart illustrating a method of determining a color correction matrix corresponding to a first lighting condition.

[0058] like Figure 3 As shown in FIG. 1 , in operation S310 , a first reference image generated by photographing a first object under a first lighting condition by a reference camera may be received.

[0059] In the present application, the first camera may refer to a camera to be calibrated or corrected, and the reference camera may refer to a camera as a target for which the first camera is calibrated.

[0060] According to one or more embodiments, the first lighting condition may represent a standard lighting condition that makes illumination uniform across each region of the first object.

[0061] According to one or more embodiments, the first object may be a standard color chart (eg, a 24-color chart, a 140-color chart, etc.) for calibrating a camera.

[0062] In one example, the first reference image may also be determined in advance (eg, determined before determining a color correction matrix for the first camera to adjust color consistency between the first camera and a reference camera).

[0063] In operation S320, a first image generated by photographing a first object under a first lighting condition by a first camera may be received.

[0064] In operation S330, a first color correction matrix of the first camera corresponding to the first lighting condition may be calculated using the first reference image, the first image, and a first reference color correction matrix of the reference camera corresponding to the first lighting condition.

[0065] Hereinafter, the i-th reference color correction matrix may denote a color correction matrix usable for correcting an image captured by a reference camera under an i-th lighting condition, and the i-th color correction matrix may denote a color correction matrix usable for correcting an image captured by a first camera under an i-th lighting condition.

[0066] According to one or more embodiments, a reference color correction matrix of a reference camera corresponding to each lighting condition may be determined in advance and stored.

[0067] According to one or more embodiments, a first color correction matrix may be determined such that a difference between a first image corrected by the first color correction matrix and a first reference image corrected by a first reference color correction matrix is ​​minimized.

[0068] In one example, the first color correction matrix may be determined in an iterative manner (e.g., using a gradient descent method, a quasi-Newton method, etc.). For example, the first color correction matrix may be determined so that the difference between the first calculation matrix and the first reference matrix is ​​minimized. For example, the first calculation matrix may represent a matrix determined by applying a first intermediate matrix obtained by multiplying the first color correction matrix by the first RGB matrix of the first image to a predetermined conversion function. For example, the first reference matrix may represent a matrix determined by applying a first reference intermediate matrix obtained by multiplying the first reference color correction matrix by the first reference RGB matrix of the first reference image to a predetermined conversion function.

[0069] For example, the first color correction matrix may be determined by Equation 1 below.

[0070]

[0071] In Equation 1, T s The first color correction matrix Cs A first RGB matrix (in one example, C s may be an N×3 matrix, where N may be the number of color blocks of the first object); T t It can represent the first reference color correction matrix; C t Can represent the first reference RGB matrix; ‖*‖ 2 represents a 2-norm; f() may represent a predetermined conversion function for converting a perception space of a camera into a perception space of a human eye.

[0072] For example, T s and T t Each of the correction matrices includes a plurality of correction matrices corresponding to a plurality of colors, respectively (in one example, each of the plurality of correction matrices may be a 3×3 matrix).

[0073] For example, f() can be a linear function or a nonlinear function. In one example, when f() is implemented as a linear function, the perception space of the camera can be maintained but the computational complexity can be reduced; when f() is implemented as a nonlinear function, the perception space of the human eye can be approached but the computational complexity can be increased.

[0074] In another example, the first color correction matrix may be determined by correcting the first reference color correction matrix using a first correction term representing the first RGB matrix of the first image and the first reference RGB matrix of the first reference image.

[0075] For example, the first color correction matrix can be determined by multiplying the first correction term by the first reference color correction matrix, and the first color correction matrix determined in this way can minimize the difference between the first image corrected by the first color correction matrix and the first reference image corrected by the first reference color correction matrix.

[0076] For example, the first correction term may represent the inverse of the product of the first transposed matrix of the first RGB matrix and the first RGB matrix, and the product of the first transposed matrix and the first reference RGB matrix.

[0077] According to one or more embodiments, the predetermined conversion function f() of Equation 1 may be implemented by using an identity function, in which case the first correction term and the first color correction matrix corresponding to the first lighting condition may be calculated using the following Equation 2.

[0078]

[0079] In equation 2, (C s T C s ) -1 C s T Ct may represent the first correction term.

[0080] Compared with determining the color correction matrix through a debugging tool, in the present application, it can take less time to calculate or determine the color correction matrix of the first camera corresponding to the standard lighting condition (for example, because the calculation steps can be performed by a computing device), thereby improving the efficiency of determining the color correction matrix.

[0081] Figure 4 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments. Figure 4 is a flow chart illustrating a method of determining color correction matrices corresponding to remaining lighting conditions other than a first lighting condition.

[0082] like Figure 4 As shown in , after receiving the first reference image and the first image (eg, after operation S320 or operation S330), in operation S410, a reference spectral response function of the reference camera may be determined.

[0083] Hereinafter, the reference spectral response function may refer to a spectral response function, a spectral sensitivity, or a spectral response matrix of a reference camera. In one example, the reference spectral response function may be determined in advance.

[0084] In operation S420, a first spectral response function of the first camera may be determined.

[0085] Hereinafter, the first spectral response function may represent a spectral response function, a spectral sensitivity, or a spectral response matrix of the first camera.

[0086] According to one or more embodiments, the first spectral response function may be determined in the same manner as the reference spectral response function. Figure 5 A process of determining a spectral response function (eg, a reference spectral response function or a first spectral response function) is described in detail.

[0087] In operation S430, a second reference image of the reference camera corresponding to the second lighting condition may be predicted using the reference spectral response function.

[0088] According to one or more embodiments, the second lighting condition may be different from the first lighting condition, and the second lighting condition may be one of the lighting conditions other than the first lighting condition.

[0089] According to one or more embodiments, the processing of the predicted second reference image may mean predicting an image that may be generated when a reference camera photographs the first object under a second lighting condition, rather than actually photographing the first object under the second lighting condition using a reference camera, thereby reducing time and labor costs. In one example, the second reference image may be determined in advance.

[0090] According to one or more embodiments, a second reference image may be calculated by applying a spectral reflectance matrix of the first object, a second spectral power distribution corresponding to a second lighting condition, and a reference spectral response function to a camera response function (eg, Equation 3 below).

[0091] For example, the camera response function may reflect (based on) the relationship between the RGB matrix of the camera's image corresponding to the lighting conditions and the spectral reflectance matrix of the object photographed by the camera, the spectral power distribution corresponding to the lighting conditions, and the spectral response function of the camera.

[0092] In one example, the camera response function may be expressed as Equation 3 below.

[0093] C = RLS (Equation 3)

[0094] In Equation 3, C may represent an image of the camera corresponding to the lighting condition, R may represent a spectral reflectance matrix of an object photographed by the camera, L may represent a diagonal matrix of spectral power distribution corresponding to the lighting condition, and S may represent a spectral response function of the camera.

[0095] In this case, the second reference image may be calculated or determined by determining R of Equation 3 as the spectral reflectance matrix of the first object, determining L as the diagonal matrix of the second spectral power distribution corresponding to the second lighting condition, and determining S as the reference spectral response function. For example, the second reference image may be calculated or determined as shown in the following Equation 4.

[0096] C t ′=RL′S t (Equation 4)

[0097] In Equation 4, C t ' may represent a second reference image, L' may represent a diagonal matrix of a second spectral power distribution corresponding to a second illumination condition, S t It can represent the reference spectral response function.

[0098] That is, the second reference image may be calculated by multiplying the spectral reflectance matrix of the first object, the diagonal matrix of the second spectral power distribution corresponding to the second lighting condition, and the reference spectral response function.

[0099] In addition, examples of camera response functions are not limited to the above equation 3, and any camera response function that can reflect the relationship between the RGB matrix of the camera's image corresponding to the lighting conditions and the spectral reflectance matrix of the object photographed by the camera, the spectral power distribution corresponding to the lighting conditions, and the spectral response function of the camera can be used.

[0100] In operation S440, a second image of the first camera corresponding to the second lighting condition may be predicted using the first spectral response function.

[0101] Similarly, the processing of the predicted second image may mean predicting an image that may be generated by the first camera photographing the first object under the second lighting condition, rather than actually using the first camera to photograph the first object under the second lighting condition, thereby reducing time and labor costs.

[0102] According to one or more embodiments, the second image may be calculated or determined by determining R of Equation 3 as the spectral reflectance matrix of the first object, determining L as the diagonal matrix of the second spectral power distribution corresponding to the second lighting condition, and determining S as the first spectral response function. For example, the second image may be calculated as shown in the following Equation 5.

[0103] C s ′=RL′S s (Equation 5)

[0104] In Equation 5, C s ' can represent the second image, S s A first spectral response function may be represented.

[0105] That is, the second image may be calculated by multiplying the spectral reflectance matrix of the first object, the diagonal matrix of the second spectral power distribution corresponding to the second lighting condition, and the first spectral response function.

[0106] In operation S450, a second color correction matrix of the first camera corresponding to the second lighting condition may be calculated using the second reference image, the second image, and a second reference color correction matrix of the reference camera corresponding to the second lighting condition.

[0107] According to one or more embodiments, the second color correction matrix may be determined such that a difference between a second image corrected by the second color correction matrix and a second reference image corrected by a second reference color correction matrix is ​​minimized.

[0108] In one example, the second color correction matrix may be determined in an iterative manner (e.g., using a gradient descent method, a quasi-Newton method, etc.). For example, the second color correction matrix may be determined so that the difference between the second calculation matrix and the second reference matrix is ​​minimized. For example, the second calculation matrix may represent a matrix determined by applying a second intermediate matrix obtained by multiplying the second color correction matrix by the second RGB matrix of the second image to a predetermined conversion function. For example, the second reference matrix may represent a matrix determined by applying a second reference intermediate matrix obtained by multiplying the second reference color correction matrix by the second reference RGB matrix of the second reference image to a predetermined conversion function.

[0109] Compared with determining the color correction matrix corresponding to each lighting condition through a debugging tool, in the present application, the spectral response functions of the reference camera and the first camera are used to predict the images of the reference camera and the first camera corresponding to the remaining lighting conditions, and the predicted images are used to calculate the color correction matrix of the first camera corresponding to the remaining lighting conditions. This can reduce the time for photographing the first object under the remaining lighting conditions, thereby further improving the efficiency of determining the color correction matrix.

[0110] Although the above describes the process of predicting the second reference image and the second image, the example is not limited to this, and when calculating the second color correction matrix, a second reference image generated by photographing the first object under second lighting conditions with a reference camera and a second image generated by photographing the first object under second lighting conditions with a first camera may be received.

[0111] According to another embodiment, in operation S460, the second reference color correction matrix may be corrected using the first correction item to determine the second color correction matrix. That is, when determining the second color correction matrix, operations S410 to S450 or operation S460 may be performed.

[0112] For example, the second color correction matrix can be determined by multiplying the first correction term by the second reference color correction matrix, and the second color correction matrix determined in this way can minimize the difference between the second image corrected by the second color correction matrix and the second reference image corrected by the second reference color correction matrix.

[0113] Compared with determining the color correction matrix corresponding to each lighting condition through a debugging tool, in the present application, a simple first correction item corresponding to the first lighting condition is used to calculate the color correction matrix corresponding to the remaining lighting conditions of the first camera. This can reduce the complexity of calculating the color correction matrix and reduce the time for photographing the first object under the remaining lighting conditions, thereby further improving the efficiency of determining the color correction matrix.

[0114] Figure 5is a flow chart illustrating a method of determining a spectral response function according to one or more embodiments.

[0115] The process of determining the reference spectral response function of the reference camera is similar to the process of determining the first spectral response function of the first camera, and therefore, the process of determining the reference spectral response function of the reference camera described below can be equally applied to the process of determining the first spectral response function of the first camera.

[0116] According to one or more embodiments, the reference spectral response function may be a polynomial matrix using the first reference RGB matrix of the first reference image as a variable. For example, the polynomial matrix may be expressed as the following Equation 6.

[0117] S t =A 1 C t 1 +A 2 C t 2 +…+A n C t n +O(C t n )(Equation 6)

[0118] In Equation 6, S t The reference spectral response function (in one example, S t It can be an M×3 matrix, where M represents the number of sampling points in the spectral wavelength dimension); C t i A can represent the i-th order response matrix of the reference image; i It can represent the i-th order coefficient matrix. The following will describe in detail how to determine the coefficient matrix. In one example, when determining the coefficient matrix, the remainder O(C t n ) can be ignored.

[0119] According to one or more embodiments, C t It can be an N×3 matrix, where N represents the number of color blocks of the first object (e.g., a standard color card). t A plurality of elements (eg, 3 elements (eg, a red component R, a green component G, and a blue component B)) corresponding to each of the plurality of color blocks may be included. t The multiple elements corresponding to a color block may include an average value of multiple red components R of multiple pixel values ​​corresponding to the color block of the first reference image, an average value of multiple green components G of multiple pixel values, and an average value of multiple blue components of multiple pixel values.

[0120] In one example, C ti Each of the multiple elements corresponding to a color block can represent C t The i-th power of the corresponding element in the multiple elements corresponding to the one color block. That is, C t i It can be expressed by C t A matrix consisting of the i-th power of each element of .

[0121] In another example, C t i Each of the multiple elements corresponding to a color block can be represented by C t At least one of the multiple elements corresponding to the color block constitutes a monomial of degree i.

[0122] like Figure 5 As shown in , in operation S510 , a variable i may be set to 1.

[0123] In operation S520, an updated polynomial matrix corresponding to an i-th order coefficient matrix may be obtained.

[0124] For example, the second-order coefficient matrix A of the original polynomial matrix can be 2 To the nth order coefficient matrix A n Set to zero matrix to obtain the first-order coefficient matrix A 1 The corresponding updated polynomial matrix.

[0125] For example, the previously determined first-order coefficient matrix A 1 To the m-1th order coefficient matrix A m-1 Substitute the original polynomial matrix and replace the m+1th order coefficient matrix A of the original polynomial matrix m+1 To the nth order coefficient matrix A n Determine the zero matrix to obtain the m-th order coefficient matrix A m The corresponding updated polynomial matrix, m is an integer greater than 1 and less than n.

[0126] For example, the previously determined first-order coefficient matrix A 1 To the n-1th order coefficient matrix A n-1 Substitute the original polynomial matrix to obtain the n-th order coefficient matrix A n The corresponding updated polynomial matrix.

[0127] In the original polynomial matrix, the 1st order coefficient matrix to the nth order coefficient matrix are not determined.

[0128] In operation S530, an i-th order coefficient matrix A may be determined by using an updated polynomial matrix corresponding to the i-th order coefficient matrix and a camera response function. i .

[0129] For example, the updated polynomial matrix S t =A 1 C t 1 +A 2 C t 2 +…+A i C t i Substituting a camera response function (e.g., C ) that reflects the relationship between the first reference RGB matrix and the spectral reflectance matrix of the first object, the first spectral power distribution corresponding to the first lighting condition, and the reference spectral response function t =RLS t , where L may represent a diagonal matrix of the first spectral power distribution corresponding to the first illumination condition), to calculate the i-th order coefficient matrix A i .

[0130] In operation S540 , the variable i may be increased by 1.

[0131] In operation S550, it may be determined whether the variable i is greater than n.

[0132] When the variable i is less than or equal to n (No in operation S550 ), the process may proceed to operation S520 .

[0133] When the variable i is greater than n (yes in operation S550 ), the process may end.

[0134] According to one or more embodiments, the first term A of the polynomial matrix 1 C t 1 The first-order coefficient matrix A 1 To the nth item A n C t n The n-th order coefficient matrix A n According to the first-order coefficient matrix A 1 To the nth order coefficient matrix A n The order is determined.

[0135] In one example, when determining the first-order coefficient matrix A 1 When the second-order coefficient matrix A of the original polynomial matrix is 2 To the nth order coefficient matrix A n Set to zero matrix to obtain the updated polynomial matrix S t =A 1 C t 1 .

[0136] The updated polynomial matrix St =A 1 C t 1 Substituting the above camera response function (C t =RLS t ), thus obtaining the following equation 7.

[0137] C t =RLA 1 C t 1 (Equation 7)

[0138] According to Equation 7, the first-order coefficient matrix A 1 It can be calculated as shown in Equation 8 below.

[0139] A 1 =[(RL) T (RL)+λ(D T D)] -1 (RL) T (Equation 8)

[0140] In Equation 8, λ may represent an adjustable regularization coefficient; and D may represent a predetermined discrete second-order derivative operator matrix (eg, a discrete Laplacian operator matrix).

[0141] In determining the second-order coefficient matrix A 2 When the first-order coefficient matrix A determined previously can be 1 Substitute the original polynomial matrix and replace the third-order coefficient matrix A 3 To the nth order coefficient matrix A n Set to zero matrix to obtain the updated polynomial matrix S t =A 1 C t 1 +A 2 C t 2 .

[0142] The updated polynomial matrix S t =A 1 C t 1 +A 2 C t 2 Substituting the above camera response function (C t =RLS t ), and the second-order coefficient matrix A 2 can be calculated as shown in Equations 9 and 10 below.

[0143]

[0144] Δ1 =C t -RLA 1 C t 1 (Equation 10)

[0145] In Equation 9, λ 2 It can represent an adjustable regularization coefficient.

[0146] Furthermore, the coefficient matrix A can be determined in a similar manner 3 To A n .

[0147] In the present application, by using a polynomial matrix to represent the spectral response functions of the reference camera and the first camera, it is possible to avoid using a complex instrument such as a monochromator to determine the spectral response function, which can more conveniently determine the spectral response functions of the reference camera and the first camera and reduce the time for photographing the first object under the residual lighting conditions, thereby further improving the efficiency of determining the color correction matrix. In addition, by using a polynomial matrix to approximate the spectral response functions of the reference camera and the first camera, the error of the estimated spectral response function can be reduced and the accuracy of the estimated spectral response function can be improved.

[0148] Although the above describes a method of determining the spectral response function using a polynomial matrix, the present disclosure is not limited thereto, and other existing methods of determining the spectral response function may also be used.

[0149] Figure 6 is a flow chart illustrating a method of determining a color correction matrix according to one or more embodiments.

[0150] like Figure 6 As shown in FIG. 5 , in operation S610 , a first reference image generated by photographing a first object under a first lighting condition by a reference camera may be received.

[0151] In operation S620, a first image generated by photographing a first object under a first lighting condition by a first camera may be received.

[0152] In operation S630, a first color correction matrix corresponding to a first lighting condition may be determined.

[0153] In operation S640, it may be determined whether a color correction matrix of the first camera corresponding to all lighting conditions is determined.

[0154] When the color correction matrix corrections of the first camera corresponding to all lighting conditions are determined (Yes in operation S640 ), the process may end.

[0155] When the color correction matrix correction corresponding to all lighting conditions of the first camera has not been determined (No in operation S640), in operation S650, one of the lighting conditions corresponding to the uncalculated color correction matrix among all lighting conditions may be selected.

[0156] In operation S660, a color correction matrix corresponding to the selected lighting condition may be determined.

[0157] For example, when the predetermined conversion function is implemented as an identity function, the color correction matrix in operation S630 and operation S660 may be determined (for example, using the above-mentioned first correction term). For another example, when the predetermined conversion function is implemented as a function other than the identity function, the color correction matrix in operation S630 and operation S660 may be determined (for example, using an iterative manner). Thereafter, the process may proceed to operation S640.

[0158] Figure 7 is a flow chart illustrating a method of color correction according to one or more embodiments.

[0159] Figure 7 Operation S710, operation S720 and operation S730 may be performed with Figure 3 Operation S310 , operation S320 , and operation S330 are similar, and thus, detailed descriptions thereof will be omitted to avoid redundancy.

[0160] In operation S740, a first actual image generated by photographing a second object under a first lighting condition by a first camera may be received.

[0161] According to one or more embodiments, the second object may be the same as or different from the first object. In one example, the second object may be an object photographed by the first camera when it is actually working.

[0162] In operation S750, the first actual image may be corrected using the first color correction matrix. For example, the first actual image may be corrected by multiplying the first color correction matrix by a first actual RGB matrix of the first actual image.

[0163] In addition, you can also use the above reference Figure 4 and Figure 5 The second color correction matrix described is used to correct the actual image generated by the first camera under the second lighting condition.

[0164] Figure 8 is a flow chart illustrating a method of determining a color correction matrix according to another embodiment.

[0165] like Figure 8As shown in, in operation S810, a first type camera (e.g., a golden module of a reference camera) may be used as a reference camera and a second type camera (e.g., a golden module of a camera to be calibrated) may be used as a first camera to determine a color correction matrix of the second type camera corresponding to each lighting condition.

[0166] According to one or more embodiments, reference may be used Figures 3 to 6 The method for determining the color correction matrix described above is used to determine the color correction matrix corresponding to each lighting condition of the second type camera. In one example, the color correction matrix corresponding to each lighting condition can be determined by using a reference Figures 3 to 6 After the method for determining the color correction matrix determines the color correction matrix corresponding to each lighting condition of the second type camera, the color correction matrix corresponding to each lighting condition of the second type camera is adjusted again according to the customer's requirements (eg, inclination and preference, etc.).

[0167] In operation S820, a color correction matrix and an image corresponding to each lighting condition of the second type camera (including an image obtained by photographing a first object under a first lighting condition by the second type camera and predicted images of the second type camera corresponding to remaining lighting conditions) may be stored.

[0168] For example, the color correction matrix and the image of the second type camera corresponding to each lighting condition may be stored in a non-volatile memory device (eg, an electrically erasable programmable read-only memory (EEPROM), but not limited thereto).

[0169] In operation S830, the second type camera may be used as a reference camera and the third type camera (eg, a non-golden module of the camera to be calibrated) may be used as a first camera to determine a color correction matrix of the third type camera corresponding to each lighting condition.

[0170] According to one or more embodiments, reference may be used Figures 3 to 6 The method for determining a color correction matrix is ​​described to determine a color correction matrix corresponding to each lighting condition of a third type camera.

[0171] In one example, the color correction matrix of the second type of camera may be calculated before it operates (e.g., by a debugger's computer), and the color correction matrix of the third type of camera may be calculated before it operates (e.g., by a debugger's computer) or while it operates (e.g., by a processor of the electronic device in which it is located).

[0172] Compared with only determining the color correction matrix of the second type of camera and using the determined color correction matrix as the color correction matrix of the third type of camera, in the present application, the color correction matrix of the third type of camera is determined by taking the second type of camera as the reference camera and the third type of camera as the first camera, so that the color correction matrix suitable for the third type of camera can be determined, thereby improving the quality of camera calibration.

[0173] Fig. 9 A block diagram of an electronic device according to one or more embodiments is shown.

[0174] like Fig. 9 As shown, the electronic device 200 in one or more embodiments includes a sensor unit 210 , at least one processor 220 , a communication unit 230 , an input unit 240 , a storage unit 250 , and a display unit 260 .

[0175] The sensor unit 210 is connected to the processor 220. The processor 220 may execute the above-mentioned Figures 3 to 8 A method for determining a color correction matrix and a color correction method are described.

[0176] The communication unit 230 may perform a communication operation of the electronic device. The communication unit 230 may establish a communication channel to a communication network and / or may perform communication associated with, for example, image processing.

[0177] The input unit 240 is configured to receive various input information and various control signals and transmit the input information and control signals to the processor 220. The input unit 240 may be implemented by various input devices such as a touch screen; however, one or more embodiments are not limited thereto.

[0178] The storage unit 250 may include a volatile memory and / or a non-volatile memory. The storage unit 250 may store various data generated and used by the electronic device. For example, the storage unit 250 may store an operating system, an application program (e.g., an application program associated with the method of the present disclosure) for controlling the operation of the electronic device. The processor 220 may control the overall operation of the electronic device and may control part or all of the internal elements of the electronic device. The processor 220 may be implemented as a general-purpose processor, an application processor (AP), an application-specific integrated circuit, a field programmable gate array, etc., but one or more embodiments are not limited thereto.

[0179] The devices, units, modules and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, cameras, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware component can also access, manipulate, process, create and store data pages in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. The hardware component can have any one or more of different processing configurations, and examples of different processing configurations include: a single processor, an independent processor, a parallel processor, a single instruction single data page (SISD) multiprocessing, a single instruction multiple data page (SIMD) multiprocessing, a multiple instruction single data page (MISD) multiprocessing, and a multiple instruction multiple data page (MIMD) multiprocessing.

[0180] The method for performing the operations described in this application is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as executing instructions or software as described above to perform the operations performed by the method described in this application. For example, a single operation, or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation, or two or more operations.

[0181] The instructions or software for controlling a processor or computer to implement a hardware component and perform the method as described above may be written as a computer program, a code segment, an instruction, or any combination thereof, to individually or collectively instruct or configure a processor or computer to operate as a machine or a special-purpose computer to perform operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by a processor or computer. In another example, the instructions or software include high-level code executed by a processor or computer using an interpreter. An ordinary programmer in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, and the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing operations performed by the hardware components and methods as described above.

[0182] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above, and any associated data pages, data page files, and data page structures are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card memory (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data page storage, optical data page storage, hard disk, solid state disk, and any other device, any other device is configured to store instructions or software and any associated data pages, data page files and data page structures in a non-temporary manner and provide the instructions or software and any associated data pages, data page files and data page structures to a processor or computer so that the processor or computer can execute the instructions.

[0183] While the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims.

Claims

1. A method for determining a color correction matrix, comprising: receiving a first reference image generated by photographing a first object under first lighting conditions using a reference camera; receiving a first image generated by photographing a first object under first lighting conditions using a first camera; calculating a first color correction matrix for the first camera using the first reference image, the first image, and a first reference color correction matrix for the reference camera, wherein the first color correction matrix and the first reference color correction matrix correspond to a first lighting condition; The actual image is corrected using the first color correction matrix.

2. The method according to claim 1, wherein: The operation of calculating a first color correction matrix of the first camera corresponding to the first lighting condition includes determining the first color correction matrix so as to minimize the difference between the first image corrected using the first color correction matrix and the first reference image corrected using the first reference color correction matrix.

3. The method according to claim 2, wherein: The operation of determining the first color correction matrix includes: correcting the first reference color correction matrix using a first correction item representing a first RGB matrix of the first image and a first reference RGB matrix of the first reference image to determine the first color correction matrix.

4. The method according to claim 1, further comprising: determining a second color correction matrix of the first camera corresponding to the second lighting condition by correcting a second reference color correction matrix of the reference camera corresponding to the second lighting condition using a first correction term representing the first RGB matrix of the first image and the first reference RGB matrix of the first reference image, The second lighting condition is different from the first lighting condition.

5. The method according to claim 3, wherein: The first correction term represents the inverse of the product of the first transposed matrix of the first RGB matrix and the first RGB matrix, and the product of the first transposed matrix and the first reference RGB matrix.

6. The method according to claim 1, further comprising Determine the reference spectral response function of the reference camera: determining a first spectral response function of a first camera; Using the reference spectral response function, predict a second reference image of the reference camera, where, The second reference image corresponds to a second lighting condition; predicting a second image from the first camera using the first spectral response function, wherein the second image corresponds to a second lighting condition; calculating a second color correction matrix for the first camera using the second reference image, the second image, and a second reference color correction matrix for the reference camera, wherein the second color correction matrix and the second reference color correction matrix correspond to a second lighting condition, The second lighting condition is different from the first lighting condition.

7. The method according to claim 6, wherein: The reference spectral response function is a polynomial matrix using the first reference RGB matrix of the first reference image as a variable.

8. The method according to claim 7, wherein: The first-order coefficient matrix to the n-th-order coefficient matrix of the polynomial matrix are determined in the order from the first-order coefficient matrix to the n-th-order coefficient matrix, The i-th order coefficient matrix of the polynomial matrix is ​​determined in the following manner: Obtaining an updated polynomial matrix corresponding to the i-th order coefficient matrix; The i-th order coefficient matrix is ​​determined by using the updated polynomial matrix corresponding to the i-th order coefficient matrix and the camera response function, Wherein, n is an integer greater than 1, i is an integer greater than or equal to 1 and less than or equal to n, Wherein, an updated polynomial matrix corresponding to the first-order coefficient matrix is ​​obtained by setting the second-order coefficient matrix to the n-th-order coefficient matrix of the original polynomial matrix to a zero matrix, wherein an updated polynomial matrix corresponding to the m-th order coefficient matrix is ​​obtained by substituting the previously determined 1st order coefficient matrix to the m-1th order coefficient matrix into the original polynomial matrix and determining the m+1th order coefficient matrix to the nth order coefficient matrix of the original polynomial matrix as zero matrices, where m is an integer greater than 1 and less than n, Wherein, an updated polynomial matrix corresponding to the n-th order coefficient matrix is ​​obtained by substituting the previously determined 1st order coefficient matrix to the n-1th order coefficient matrix into the original polynomial matrix, Among them, in the original polynomial matrix, the 1st order coefficient matrix to the nth order coefficient matrix are not determined.

9. A color correction method, comprising: receiving a first reference image generated by photographing a first object under first lighting conditions using a reference camera; receiving a first image generated by photographing a first object under first lighting conditions using a first camera; calculating a first color correction matrix for the first camera using the first reference image, the first image, and a first reference color correction matrix for the reference camera, wherein the first color correction matrix and the first reference color correction matrix correspond to a first lighting condition; receiving a first actual image generated by photographing a second object under a first lighting condition using a first camera; Correcting the first real image using a first color correction matrix, The first object is the same as or different from the second object.

10. A color correction method, comprising: receiving a first reference image generated by photographing a first object under first lighting conditions using a reference camera; receiving a first image generated by photographing a first object under first lighting conditions using a first camera; using the first reference image, the first image, and a first reference color correction matrix for the reference camera, calculating a first color correction matrix for the first camera by determining the first color correction matrix to minimize a difference between the first image corrected using the first color correction matrix and the first reference image corrected using the first reference color correction matrix, wherein the first color correction matrix and the first reference color correction matrix correspond to a first lighting condition; receiving a first actual image generated by photographing a second object under a first lighting condition using a first camera; Correcting the first real image using a first color correction matrix, The first object is the same as or different from the second object.

Images