Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, which are used for processing image contents. The method comprises the following steps: acquiring a first color card area contained in a first image, and generating a first color matrix corresponding to the first color card area; obtaining a second color card area contained in a second image, and generating a second color matrix corresponding to the second color card area; determining a mapping matrix according to the first color matrix and the second color matrix, wherein the mapping matrix is used for realizing conversion between the first color matrix and the second color matrix; and performing conversion processing on the first image or the second image according to the mapping matrix to obtain a target image. According to the method, the target image is obtained by performing conversion processing on the first image or the second image through the mapping matrix, the image data of the image can be processed by means of the conversion operation of the mapping matrix, and a more flexible image processing mode is provided.

Description

Technical Field

[0001] The present disclosure relates to the field of image technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Image processing refers to the technology of using computers to analyze images to achieve the desired results. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values. Image processing technology generally includes image compression, enhancement and restoration, matching, description and recognition.

[0003] In the related art, in order to realize the augmentation processing of the image data set, it can be realized through various processing methods such as mirror flipping, random cropping, rotation and scaling. For example, by performing mirror flipping, random cropping, rotation, scaling and other processing on the initial image, a new image after the change can be obtained, and then the new image is added to the image data set to realize the augmentation of the image data set. The original image processing method is relatively simple. Summary of the invention

[0004] The present disclosure provides an image processing method, device, electronic device and storage medium, which can at least solve the technical problem of how to process images.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including: obtaining a first color card area contained in a first image, and generating a first color matrix corresponding to the first color card area; obtaining a second color card area contained in a second image, and generating a second color matrix corresponding to the second color card area; determining a mapping matrix based on the first color matrix and the second color matrix, the mapping matrix being used for conversion between the first color matrix and the second color matrix; and performing conversion processing on the first image or the second image according to the mapping matrix to obtain a target image.

[0006] In a second aspect, an embodiment of the present disclosure provides an electronic device, which includes one or more memories and a processor; the one or more memories store a computer program that can be executed by one or more processors, and when the computer program is executed by the processor, the image processing method provided in any of the above aspects is implemented.

[0007] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method provided in any of the above aspects is implemented.

[0008] According to an image processing method provided by an embodiment of the present disclosure, a first color matrix corresponding to a first color card area can be generated according to a first color card area included in a first image, and a second color matrix corresponding to a second color card area can be generated according to a second color card area included in a second image. Accordingly, a mapping matrix for realizing the conversion between the first color matrix and the second color matrix is ​​obtained, and the first image or the second image is converted by the mapping matrix to obtain a target image. It can be seen that the target image in this method is obtained by converting the first image or the second image by the mapping matrix. With the help of the conversion operation of the mapping matrix, the image data of the target image can be processed, thereby providing a more flexible and diverse image processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the accompanying drawings of the embodiments of the present disclosure:

[0010] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0011] Figure 2 A flowchart of an image processing method provided in Example 1 of the present disclosure;

[0012] Figure 3 A flowchart of an image processing method provided in Example 2 of the present disclosure;

[0013] Figure 4a A schematic diagram showing the light source replacement process before and after the first image and the second image are performed;

[0014] Figure 4b A schematic diagram showing before and after light source synthesis processing is performed on a first image and a second image;

[0015] Figure 5 A schematic block diagram of an image processing device provided by an embodiment of the present disclosure;

[0016] Figure 6 A schematic block diagram of a composition of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, an image processing method, device, electronic device and storage medium provided by an embodiment of the present disclosure are described in detail below with reference to the accompanying drawings.

[0018] The present disclosure will be described more fully below with reference to the accompanying drawings, but the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as being limited to the embodiments set forth below. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete, and will enable those skilled in the art to fully understand the scope of the present disclosure.

[0019] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification, and are used together with the detailed embodiments to explain the present disclosure, and do not constitute a limitation of the present disclosure. By describing the detailed embodiments with reference to the accompanying drawings, the above and other features and advantages will become more apparent to those skilled in the art.

[0020] The present disclosure may be described with reference to plan views and / or cross-sectional views by means of ideal schematic views of the present disclosure. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0021] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0022] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. The term "and / or" as used in the present disclosure includes any and all combinations of one or more related enumerated items. The singular forms "one" and "the" as used in the present disclosure are also intended to include plural forms, unless the context clearly indicates otherwise. The terms "including", "made of..." as used in the present disclosure specify the presence of the features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0023] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0024] The present disclosure is not limited to the embodiments shown in the drawings, but includes modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the drawings have schematic properties, and the shapes of the regions shown in the drawings illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0025] First, Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 1 The present disclosure provides an image processing method, which may include the following steps:

[0026] Step S110: Acquire a first color card area contained in the first image, and generate a first color matrix corresponding to the first color card area.

[0027] Wherein, the first image may be any image including a first color card area. The first color card area is an area in the image corresponding to the first physical color card. The first physical color card may be composed of color blocks of multiple standard colors, and each color block in the first physical color card may correspond to a standard color. Accordingly, according to the actual pixel values ​​of the multiple color blocks in the first color card area in the first image, the first color matrix corresponding to the first color card area may be calculated. Wherein, the first color matrix is ​​used to characterize the actual pixel values ​​of the multiple color blocks in the first color card area, and the number of matrix elements in the first color matrix may be consistent with the number of color blocks in the first color card area. It can be seen that the multiple matrix elements in the first color matrix may correspond one-to-one to the multiple color blocks in the first color card area, and each matrix element in the first color matrix is ​​used to characterize the actual pixel value presented by the color block at the corresponding position in the first image.

[0028] In an optional implementation, the first color card area can be obtained according to the first physical color card. For example, the first color card area can be obtained by photographing the first physical color card: in the process of photographing the first image, the first physical color card is placed in the photographing range, and accordingly, the photographed first image contains the first color card area corresponding to the first physical color card. The first physical color card is also called the first color card, which is a real color card that exists physically. Different from the virtual color card, the first physical color card contains multiple color blocks, each of which corresponds to a color. In order to improve the calculation accuracy, each color card in the first physical color card corresponds to a standard color. The so-called standard color refers to a color with a standard value and a known color. Since the standard color of each color block is known, and the actual pixel value presented by each color block in the first image is also measurable, the color cast of the corresponding color block can be determined according to the difference between the standard color value of each color block and the actual pixel value presented. Accordingly, the color cast of the first image can be characterized by the first color matrix.

[0029] Step S120: Acquire a second color card area contained in the second image, and generate a second color matrix corresponding to the second color card area.

[0030] Wherein, the second image can be any image containing a second color card area. The second color card area is an area in the image corresponding to the second physical color card. The second physical color card can be composed of color blocks of multiple standard colors, and each color block in the second physical color card can correspond to a standard color. Correspondingly, according to the actual pixel values ​​of the multiple color blocks in the second color card area in the second image, the second color matrix corresponding to the second color card area can be calculated. Wherein, the second color matrix is ​​used to characterize the actual pixel values ​​of the multiple color blocks in the second color card area, and the number of matrix elements in the second color matrix is ​​consistent with the number of color blocks in the second color card area. It can be seen that the multiple matrix elements in the second color matrix can correspond one-to-one to the multiple color blocks in the second color card area, and each matrix element in the second color matrix is ​​used to characterize the actual pixel value presented by the color block at the corresponding position in the second image.

[0031] In an optional implementation, the second color card area can be obtained based on the second physical color card. For example, the second color card area can be obtained by photographing the second physical color card: in the process of photographing the second image, the second physical color card is placed in the shooting range, and accordingly, the second image obtained by photographing contains the second color card area corresponding to the second physical color card. Among them, the second physical color card is also called the second color card, which is a real color card that exists physically. Unlike the virtual color card, the second physical color card contains multiple color blocks, each of which corresponds to a color. Among them, in order to improve the calculation accuracy, each color card in the second physical color card corresponds to a standard color. Since the standard color of each color block is known, and the actual pixel value presented by each color block in the second image is also measurable, therefore, according to the difference between the standard color value of each color block and the actual pixel value presented, the color cast of the corresponding color block can be determined. Accordingly, the color cast of the second image can be characterized by the second color matrix.

[0032] It can be seen that the color matrix corresponding to the color card area is used to characterize the pixel value of each color block in the color card area, and the multiple matrix elements in the color matrix correspond to the multiple color blocks in the color card area one by one. Accordingly, when generating the color matrix corresponding to the color card area, it can be achieved in the following manner: obtaining the actual pixel value of each color block in the color card area; generating the color matrix corresponding to the color card area according to the actual pixel value of each color block in the color card area and the correspondence between the multiple color blocks in the color card area and the multiple matrix elements in the color matrix. For example, generating the first color matrix corresponding to the first color card area includes: obtaining the actual pixel value of each color block in the first color card area; generating the first color matrix corresponding to the first color card area according to the actual pixel value of each color block in the first color card area and the correspondence between the multiple color blocks in the first color card area and the multiple matrix elements in the first color matrix. Correspondingly, generating a second color matrix corresponding to the second color card area includes: obtaining the actual pixel value of each color block in the second color card area; generating the second color matrix corresponding to the second color card area according to the actual pixel value of each color block in the second color card area and the correspondence between the multiple color blocks in the second color card area and the multiple matrix elements in the second color matrix.

[0033] It should be noted that the multiple color blocks included in the first physical color card and the multiple color blocks included in the second physical color card correspond one to one, and the color of any color block in the first physical color card is the same as the color of the color block at the corresponding position in the second physical color card. It can be seen that the first physical color card and the second physical color card are exactly the same, and the standard color values ​​of each color block contained therein are also exactly the same. In actual situations, the first physical color card and the second physical color card can be the same color card. Correspondingly, if there is no color cast in the first image and the second image, the matrix values ​​of the first color matrix and the second color matrix should be the same. However, in actual situations, the first image and the second image are affected by the brightness of the light source, the shooting angle, etc. during the shooting process, and are bound to produce a certain degree of color cast. Therefore, in actual situations, the values ​​of the second color matrix and the first color matrix are different.

[0034] Step S130: determining a mapping matrix according to the first color matrix and the second color matrix, where the mapping matrix is ​​used for conversion between the first color matrix and the second color matrix.

[0035] Among them, since the mapping matrix can realize the conversion between the first color matrix and the second color matrix, the second color matrix can be obtained by performing a preset operation on the first color matrix and the mapping matrix; or, the first color matrix can be obtained by performing a preset operation on the second color matrix and the mapping matrix.

[0036] In an optional implementation, the mapping matrix includes: a first mapping matrix for converting a first color matrix into a second color matrix, or a second mapping matrix for converting a second color matrix into the first color matrix.

[0037] For example, the first mapping matrix is ​​used to convert the first color matrix into the second color matrix. When the first color matrix and the second color matrix are known, the first mapping matrix can be obtained by matrix operation. Accordingly, the first color matrix is ​​multiplied by the first mapping matrix, and the result obtained is the second color matrix.

[0038] Step S140: converting the first image or the second image according to the mapping matrix to obtain a target image.

[0039] Wherein, the mapping matrix can be used to convert the first image or the second image to obtain the target image. When the mapping matrix includes the above-mentioned first mapping matrix or second mapping matrix, the target image includes the first target image or the second target image. Accordingly, this step can be implemented in the following manner: performing matrix operations on the image data of the first image according to the first mapping matrix to obtain the first target image; or, performing matrix operations on the image data of the second image according to the second mapping matrix to obtain the second target image. Wherein, matrix operations include various types of matrix addition, subtraction, multiplication, transposition, etc., wherein the multiplication of two matrices is usually limited to the case where the number of columns of the first matrix is ​​equal to the number of rows of the other matrix. In the present application, matrix operations mainly refer to matrix multiplication operations, and accordingly, the image data is also converted into matrix form to realize the multiplication operation between the two matrices. Of course, matrix operations can also be other implementation forms such as number multiplication operations, which are not limited in the present application.

[0040] It can be seen that the number of target images can be one or two. When the number of target images is one, the target image can be the first target image obtained by performing a matrix operation on the image data of the first image according to the first mapping matrix, or the second target image obtained by performing a matrix operation on the image data of the second image according to the second mapping matrix. In addition, when the number of target images is two, the target images include both the first target image and the second target image.

[0041] Since the mapping matrix can realize the mutual conversion between the two color matrices, the mapping matrix can be used to realize the conversion processing of the first image or the second image. The conversion processing can be used to change the internal properties of the image, that is, to convert the internal data of the image.

[0042] In an optional implementation, the conversion process includes: light source conversion processing. Accordingly, by mapping the matrix, the light source information in the image can be changed, so that the brightness and brightness of the image are changed. For example, according to the first mapping matrix, the first image can be subjected to light source conversion processing to obtain a first target image. The first mapping matrix can convert the first color matrix into the second color matrix. Accordingly, with the help of the first mapping matrix, the first image can be subjected to light source conversion processing to obtain the first target image after the light source conversion processing.

[0043] Among them, since the first target image is obtained according to the first mapping matrix, and the function of the first mapping matrix is ​​to realize the conversion between two color matrices, and the numerical value of the color matrix can indirectly represent the light source situation of the image, accordingly, the first target image obtained according to the first mapping matrix is ​​obtained by performing light source conversion processing on the first image.

[0044] In some exemplary embodiments, the light source conversion process may include various types such as light source replacement process and light source synthesis process. The light source replacement process refers to replacing the first light source in the first image with the second light source in the second image, or replacing the second light source in the second image with the first light source in the first image. For example, directly multiplying the first image by the first mapping matrix can realize a form of light source replacement process.

[0045] Light source synthesis processing refers to: synthesizing a light source different from the first light source and the second light source according to the first light source in the first image and the second light source in the second image, and the light source different from the first light source and the second light source can also be called a synthesized light source or a new light source, and applying the light source different from the first light source and the second light source to the first image or the second image to obtain a target image. Since the mapping matrix is ​​obtained according to the first color matrix corresponding to the first light source and the second color matrix corresponding to the second light source, correspondingly, according to the mapping matrix, a new light source associated with the first light source and the second light source can be obtained through a preset operation, thereby generating a target image corresponding to the first image or the second image according to the new light source.

[0046] No matter which light source conversion process is adopted, the obtained target image is substantially the same as the image scene in the original first image or the second image, and the main difference lies in the different light sources corresponding to the two images.

[0047] It can be seen that the target image in this method is obtained by performing light source conversion processing on the first image or the second image through the mapping matrix. Since the mapping matrix can realize the conversion between the first color matrix and the second color matrix, the information in one image can be used to process the image content of another image by performing the conversion process, thereby obtaining the target image after the conversion process. Since the target image is obtained by the conversion method, it can be adjusted according to the internal properties of the image, that is, it can be processed according to the image data contained in the image.

[0048] The present application does not limit the specific implementation details of the above embodiments, and those skilled in the art may also make various changes and modifications to the above embodiments:

[0049] In an optional implementation, the image data contained in the first image may be multiplied by the first mapping matrix to obtain the first target image; and / or, the image data contained in the second image may be multiplied by the second mapping matrix to obtain the second target image. Since the mapping matrix is ​​used to realize the conversion between the first color matrix and the second color matrix, and the specific value of the color matrix is ​​associated with the light source value of the image, the first light source in the first image can be replaced by the second light source in the second image, or the second light source in the second image can be replaced by the first light source in the first image, thereby realizing the light source replacement process and obtaining the target image after the light source is replaced.

[0050] In yet another optional implementation, before converting the first image or the second image by using the mapping matrix to obtain the target image, the following operations may be further performed:

[0051] The first light source value corresponding to the first image is determined according to the first color card area, and the second light source value corresponding to the second image is determined according to the second color card area. Since the first color matrix is ​​used to represent the actual pixel values ​​presented by each standard color block in the first color card area in the first image, the first light source value corresponding to the first image can be calculated using the first color matrix. Similarly, the second light source value corresponding to the second image can be calculated using the second color matrix.

[0052] According to the preset interpolation weight, the first light source value and the second light source value are adjusted to obtain the interpolated light source value. The specific value of the interpolation weight can be set according to actual needs. Usually, the value of the interpolation weight is greater than 0 and less than 1. Through the adjustment of the interpolation weight, the interpolated light source value can be obtained. It can be seen that the interpolated light source value is obtained by interpolating the first light source value and the second light source value, and is a new light source value different from the first light source value and the second light source value.

[0053] According to the interpolated light source value, the mapping matrix is ​​adjusted to obtain a color mapping matrix. Accordingly, the target image is obtained in the following manner: by using the color mapping matrix, a matrix operation is performed on the image data of the first image or the image data of the second image to obtain the target image. The matrix operation may be a matrix multiplication operation, and accordingly, the image data is also converted into a matrix form, thereby realizing a multiplication operation between two matrices.

[0054] In an optional implementation, the first mapping matrix is ​​adjusted according to the interpolated light source value to obtain a first color mapping matrix. Accordingly, the target image is obtained by: performing a matrix operation on the image data of the first image through the first color mapping matrix to obtain the first target image; and / or performing a matrix operation on the image data of the second image through the second color mapping matrix to obtain the second target image.

[0055] Among them, since the interpolated light source value corresponds to the synthesized light source, the color mapping matrix obtained by adjusting the mapping matrix through the interpolated light source value can replace the light source in the first image or the second image with the newly synthesized interpolated light source. It can be seen that in this method, the target image is realized based on the newly synthesized interpolated light source, so that the light source value of the first image or the second image can be flexibly changed, and the first target image with multiple light source forms can be generated according to actual business needs.

[0056] In a specific implementation, before adjusting the mapping matrix according to the interpolation light source value to obtain the color mapping matrix, the following operation is further performed: the mapping matrix is ​​weight-transformed by the interpolation weight to obtain the interpolation mapping matrix. Since the interpolation mapping matrix is ​​obtained by adjusting the interpolation weight, and the interpolation mapping matrix is ​​associated with the light source in the image, the form of the light source contained in the final target image can be made more flexible by flexibly setting the value of the interpolation weight, and target images of various light source forms can be flexibly generated. Accordingly, the color mapping matrix can be generated in the following manner: generating a gain adjustment matrix according to the interpolation light source value, and performing gain adjustment on the interpolation mapping matrix by the gain adjustment matrix to obtain the color mapping matrix. Optionally, in order to avoid the residual error of gray pixels on the newly generated target image, the specific value of the interpolation mapping matrix is ​​adjusted with the help of the gain adjustment matrix to adjust the color gain of the interpolation light source, thereby obtaining the final color mapping matrix.

[0057] In an optional implementation, when generating a gain adjustment matrix according to the interpolated light source value, it is specifically implemented in the following manner: First, a mapping light source value is generated according to the interpolated mapping matrix, and the first light source value or the second light source value. In an optional implementation, the first light source value is adjusted with the help of the interpolated mapping matrix to obtain the mapping light source value. Then, the above-mentioned gain adjustment matrix is ​​generated according to the interpolated light source value and the mapping light source value. Among them, the specific numerical value of the light source value can be represented by various forms such as a matrix or a vector. For example, in one example, the interpolated mapping matrix is ​​a 3*3 matrix, the first light source value or the second light source value is a 1*3 vector, and correspondingly, the mapping light source value is a 1*3 vector. Similarly, the interpolated light source value can also be a 1*3 vector. The gain adjustment matrix is ​​generated according to the interpolated light source value and the mapped light source value, and the specific form can be various forms of matrices or vectors, which are not limited in this application.

[0058] Optionally, the gain adjustment matrix is ​​a diagonal matrix, and the matrix element values ​​located on the diagonal of the diagonal matrix are determined by: calculating a first summation result between the interpolated light source value and the first preset adjustment value. Calculating a second summation result between the mapped light source value and the second preset adjustment value. According to the ratio between the first summation result and the second summation result, the matrix element values ​​located on the diagonal of the diagonal matrix are determined. Among them, the values ​​of the first preset adjustment value and the second preset adjustment value can be the same or different. Optionally, the values ​​of the first preset adjustment value and the second preset adjustment value are the same and both are less than the preset value. The role of the first preset adjustment value and the second preset adjustment value is to prevent the denominator from being 0. In the case where the interpolated light source value and the mapped light source value are both 1*3 vectors, the gain adjustment matrix can be a 3*3 diagonal matrix, and the values ​​of the elements in the diagonal matrix except the three element values ​​on the diagonal are all 0. Among them, the first matrix element value on the diagonal is determined by: calculating the first interpolation summation result between the first value in the interpolated light source value in the form of a 1*3 vector and the first preset adjustment value. Calculate the first mapping summation result between the first numerical value in the mapped light source value in the form of a 1*3 vector and the second preset adjustment value. Determine the element value of the first matrix element located on the diagonal of the diagonal matrix according to the ratio between the first interpolation summation result and the first mapping summation result. Similarly, the second matrix element value on the diagonal is determined by the following method: calculate the second interpolation summation result between the second numerical value in the interpolation light source value in the form of a 1*3 vector and the first preset adjustment value. Calculate the second mapping summation result between the second numerical value in the mapped light source value in the form of a 1*3 vector and the second preset adjustment value. Determine the element value of the second matrix element located on the diagonal of the diagonal matrix according to the ratio between the second interpolation summation result and the second mapping summation result. The method for determining the value of the third matrix element on the diagonal is similar and will not be repeated here.

[0059] In an optional implementation, before obtaining the first color card area contained in the first image and generating the first color matrix corresponding to the first color card area, the following operations are further performed: obtaining an original image sample set, determining any image in the original image sample set as the first image; determining any image in the original image sample set that is different from the first image and satisfies a preset condition as the second image. Wherein, the preset condition includes: the error between the light source value of the second image and the light source value of the first image is greater than a preset error threshold. Wherein, the error between the two light source values ​​is used to characterize the degree of difference between the two light sources. If the degree of difference between the two light sources is greater, the error between the two light source values ​​is also greater. Wherein, the error between the light source values ​​can be in various forms such as angle error, distribution error, pixel error, etc., and this application does not limit this, as long as it can characterize the difference between the two light sources. For example, in an implementation, the preset condition includes: the angle error between the light source value of the second image and the light source value of the first image is greater than a preset angle error threshold. Alternatively, the preset condition may also be: the similarity between the light source value of the second image and the light source value of the first image is less than a preset similarity threshold. Among them, the original image sample set contains multiple images, each image can be used as the first image in turn, and multiple images that meet the preset conditions with the current first image can be processed as second images. It can be seen that in the above process, any image may be used as the first image, and any image may also be used as the second image. Therefore, the first image and the second image can be interchangeable, that is, the processing method for the first image is also applicable to the second image. In addition, the purpose of setting the preset conditions is to avoid performing light source conversion processing on two images with the same or similar light source information. In other words, there should be a certain difference in the light source information corresponding to the two images undergoing light source conversion processing to ensure that the effect of the light source conversion processing is more obvious.

[0060] In an optional implementation, the above-mentioned conversion processing includes: a conversion processing of a light source replacement type. The so-called conversion processing of a light source replacement type refers to: replacing the light source information in two images. For example, the first light source value in the first image is replaced by the second light source value in the second image, or the second light source value in the second image is replaced by the first light source value in the first image. Correspondingly, the original image sample set further includes: a first sample set of the light source to be replaced, and a second sample set corresponding to the standard light source; then the first image comes from the first sample set, and the second image comes from the second sample set. Correspondingly, when the first image or the second image is converted, the conversion processing of the light source replacement type can be performed on the first image according to the second image, so as to obtain the target image. This method aims to perform a light source replacement process on the non-standard light source in the first image according to the second image corresponding to the standard light source in the second sample set, so as to replace the light source in the first image with the light source in the second image.

[0061] In another optional implementation, the above-mentioned conversion processing includes: a conversion processing of a light source synthesis type. The so-called conversion processing of the light source synthesis type refers to: performing a light source synthesis operation according to the light source information in at least two images to obtain a new light source. Correspondingly, determining any image in the original image sample set as the first image specifically includes: determining each image in the original image sample set as the first image in turn. In this method, any two images in the original image sample set are respectively used as the first image and the second image, so that a new interpolation light source is synthesized according to the first light source of the first image and the second light source of the second image, and the newly synthesized interpolation light source is applied to the first image and / or the second image, so as to obtain a processed target image.

[0062] In an optional implementation, the original image sample set is a white balance sample set. Accordingly, after obtaining the target image, the target image is further annotated according to the white balance gain of the target image. The annotated target image is added to the original image sample set to expand the original image sample set. In the above manner, an expanded white balance sample set can be obtained, thereby obtaining a large number of sample images annotated with white balance gains, which can be applied to the machine learning task of white balance calculation.

[0063] For ease of understanding, the following two specific examples are used to introduce examples of implementation methods of the image processing method in this application:

[0064] White balance is an important preprocessing step in the field of image processing and computer vision. It is used to eliminate color deviations in images caused by different lighting conditions, making the colors in the image more realistic and accurate. With the advancement of computer vision and image processing technology, automatic white balance algorithms are constantly improving. Traditional statistical-based methods (such as gray world algorithms and white point algorithms) are still widely used, but in recent years, machine learning-based methods (such as deep learning) have begun to be applied to automatic white balance. These algorithms can estimate white balance gain more accurately by learning a large amount of image data and have better generalization capabilities under different light source conditions.

[0065] With the development of artificial intelligence, the current mainstream algorithms for estimating white balance gain are mainly based on neural network training. These methods aim to train network models to more accurately estimate the white balance gain of the current scene from the original images taken by the camera. However, in order to improve the effect of model training, there are high requirements for the training data set: in order to meet the needs of the training model, the training data set needs to cover a rich light source scene and ensure that the training data set can cover the original photos taken by a variety of lenses to improve the generalization ability of the model. However, the number of existing white balance training data sets is limited and difficult to produce. Therefore, there is an urgent need for a technical solution that can expand the existing data set.

[0066] White balance dataset augmentation technology is a technology used to increase the number and diversity of white balance dataset samples. In the research and application of white balance algorithms, a large amount of image data with real lighting conditions is required for training and evaluation. However, due to the limitations and costs of different lighting conditions, it is very difficult and expensive to obtain large-scale real lighting datasets. In addition, the number of commonly used automatic white balance public datasets is relatively small. When training your own automatic white balance (AWB) model, you need to shoot more new datasets according to development needs to improve the training effect. To solve this problem, this example proposes a white balance dataset augmentation technology.

[0067] Most of the existing public automatic white balance datasets are made by placing color cards in various scenes, shooting the original pictures with color cards, and using the color card information to calculate the white balance gain in the scene; to ensure that the white balance gain value calculated by the color card is as correct as possible, when taking photos, the color card is required to be in a uniform light position in the current scene as much as possible, and the photos taken should cover as many color temperatures as possible, which has certain requirements for the shooting process, is time-consuming and laborious, so the number of photos in various commonly used white balance datasets is relatively small. This also leads to insufficient training data for the white balance network and poor results. In addition, the commonly used methods of image dataset expansion, such as mirror flipping, random cropping, rotation and scaling, are very effective in general image processing tasks, but are not applicable in the expansion of AWB datasets, because the purpose of the AWB algorithm is to adapt to real scenes with different light sources to obtain more accurate white balance effects, and the above-mentioned methods of expanding common image datasets cannot simulate the changes in light sources in real scenes. AWB datasets usually annotate images with white balance gain as an indicator for training and evaluation; however, the general image dataset augmentation operation may lead to inconsistency in annotation, making the annotated white balance gain mismatched with the image information. This may have a negative impact on the accuracy of algorithm training and evaluation.

[0068] It can be seen that the existing image data augmentation technology is not applicable to the white balance data set. In order to solve the above problems, this application proposes a data set augmentation technology related to light source, which achieves the purpose of data set augmentation by operating light source information to synthesize richer data samples, so that the model can be trained on more diverse data sets to obtain better performance. This application aims to process images in the original image sample set, so as to achieve the purpose of sample set augmentation based on the processed images.

[0069] In the present application, two image processing methods are proposed, one is a light source replacement type of image conversion processing, and the other is a light source synthesis type of image conversion processing.

[0070] Example 1: Image conversion processing of light source replacement type:

[0071] This example aims to replace the first light source in the first image with the second light source in the second image. Similarly, the second light source in the second image can also be replaced with the first light source in the first image. Accordingly, the light source replacement process in this example can also be called a light source exchange process.

[0072] First, the method of obtaining the original image sample set is introduced. The original image sample set further includes: a first sample set of the light source to be replaced, and a second sample set corresponding to the standard light source. This example aims to replace the light source of the image in the first sample set with the light source of the image in the second sample set.

[0073] The second sample set corresponding to the standard light source is obtained in the following manner: In order to obtain reliable data samples of the exchanged light source and avoid the light source values ​​of the two image samples of the exchanged light source being the same or too close, photos taken under different light sources (i.e., different color temperatures) using a 24-color physical color card in a light box can be used as standard light source samples for the exchanged light source, or photos with a physical color card taken at different times of the day can be selected to simulate different color temperatures to obtain standard light source samples (i.e., the second sample set corresponding to the standard light source). In short, this example aims to ensure that the standard light source sample can contain more light source values ​​under different color temperatures.

[0074] For the first sample set of the light source to be replaced, photos in a data set of different scenes shot with a 24-color physical color card can be used, or photos in a data set shot with a 24-color physical color card can be selected from a public data set. For the convenience of description below, the second sample set corresponding to the standard light source is named the standard sample set, and the first sample set of the light source to be replaced is named the sample sample set.

[0075] In addition, in order to improve the quality of image data, each image in the two sample sets can be obtained in the following manner: Defective Pixel Correction (DPC), black level compensation (BLC), lens correction (LSC) and the like are performed on any original image (i.e., raw image) obtained by shooting based on a 24-color physical color card, and then the original data (i.e., Raw data) is converted into RGB data to generate a bitmap image (such as a bitmap in PNG format), and the final bitmap image is used as the image in the sample set. Alternatively, the Raw image can be directly demosaiced to obtain RGB data, or the Raw image can be demosaiced by simply downsampling to obtain RGB data. In short, this application does not limit the preprocessing method of the images in the sample set. In addition, if there are images in the photos of the public data set that have undergone the above operations, they can also be directly used to generate the sample set.

[0076] Then, a dataset expansion method based on light source replacement operation is introduced. Figure 2 FIG. 4 is a flow chart showing the image conversion process of the light source replacement type in Example 1. Figure 2 As shown, Example 1 specifically includes the following steps:

[0077] Step S201: determine any image in the sample sample set as a first image; and determine any image in the standard sample set that is different from the first image and meets a preset condition as a second image.

[0078] Among them, the preset condition includes: the error between the light source value of the second image and the light source value of the first image is greater than the preset error threshold. The purpose of setting this preset condition is to prevent the light source replacement processing of two images with similar light sources. In the case where the light sources of the two images are similar, if the light source exchange processing is performed on the two images, the final processed image effect will not be significant. Therefore, in order to improve the distinguishability of each sample in the amplified sample set, the above preset condition is set.

[0079] It should be noted that the above preset condition is an optional condition. In other implementations, the preset condition may be omitted, and any image in the standard sample set that is different from the first image may be directly determined as the second image.

[0080] In a specific implementation, one image may be selected from the sample set as the first image without repetition each time, and then each image in the standard sample set may be selected in turn as the second image corresponding to the first image.

[0081] Step S202: generating a first color matrix corresponding to the first color card area according to the first color card area included in the first image; generating a second color matrix corresponding to the second color card area according to the second color card area included in the second image.

[0082] Step S203: Calculate a first mapping matrix for converting the first color matrix into the second color matrix.

[0083] Step S204: performing light source conversion processing on the first image according to the first mapping matrix to obtain a first target image.

[0084] In the specific implementation, for the first image and the second image, the color matrix and light source value corresponding to each image are calculated according to the color card area corresponding to the physical color card contained in the image. Using the color matrix, the mapping matrix between the first image and the second image is calculated using the linear least squares method. For example, assuming that the two samples are (P XA ,L A ) and (P YB ,L B ), where the sample (P XA ,L A ) belongs to the standard sample set, and sample (P YB ,L B ) belongs to the sample set, where the subscripts X and Y represent different pictures, A and B represent different light sources corresponding to different pictures, and P XA and P YB Represents image data, L A and L BRepresents the light source value of the image, and solves the linear mapping matrix M between the two color matrices by the least squares method AtoB and M BtoA , as shown in the following formula:

[0085] CM B =CM A ×M AtoB

[0086] CM A =CM B ×M BtoA

[0087] Among them, CM A is the first color matrix of the first image, CM B is the second color matrix of the second image, the linear mapping matrix M AtoB is a first mapping matrix for converting a first color matrix into a second color matrix, and the linear mapping matrix M BtoA is a second mapping matrix used to convert the second color matrix into the first color matrix.

[0088] After obtaining the two mapping matrices, the photo of the exchanged light source can be obtained by the following formula:

[0089] P YA =P YB ×M BtoA

[0090] P XB =P XA ×M AtoB

[0091] Among them, P XA is the image data contained in the first image, M AtoB is the first mapping matrix, and the image data contained in the first image is subjected to matrix multiplication operation by the first mapping matrix to obtain the first target image. It can be seen that the first target image is obtained by replacing the light source in the first image with the light source in the second image.

[0092] Similarly, P YB is the image data contained in the second image, M BtoA is the second mapping matrix, and the image data contained in the second image is subjected to matrix multiplication operation by the second mapping matrix to obtain the second target image. It can be seen that the second target image is obtained by replacing the light source in the second image with the light source in the first image.

[0093] Through the above method, two new images P with the light source exchanged can be obtained. YA and P XBThe test found that after the light source replacement process, the original sample and the generated new sample were white balanced using the corresponding light source. The effect of the image is basically only the difference in brightness, and there is no abnormal color cast. It can be seen that the result of the white balance sample obtained by the above light source replacement process is more accurate, and the problem of inaccurate sample labeling due to abnormal color cast will not occur. Figure 4a FIG. 2 shows a schematic diagram of a first image and a second image before and after performing a light source replacement process. Figure 4a As shown, the light source in the second image is replaced by the light source in the first image to obtain the target image. Therefore, the scene in the target image is consistent with the scene in the second image, but the light source information is determined based on the first image. Figure 4a Also shown is an image after white balancing the second image and an image after white balancing the target image.

[0094] Example 2: Image conversion processing of light source synthesis type:

[0095] This example aims to synthesize a new interpolation light source according to the first light source in the first image and the second light source in the second image, and generate a new image according to the newly synthesized interpolation light source.

[0096] First, we will introduce how to obtain the original image sample set: you can use photos from a dataset of different scenes shot with a 24-color physical color card, or you can select photos from a dataset shot with a 24-color physical color card from a public dataset. Perform DPC, BLC, LSC and other processing on the original image (i.e., raw image) in the sample set, and then convert the Raw data into RGB data to generate a PNG image. Alternatively, you can also demosaic the Raw image to obtain RGB data, or you can simply downsample the Raw image to demosaic it to obtain RGB data. In addition, if there are PNG images in the public dataset that have undergone the above operations, they can also be used directly to generate the sample set.

[0097] Then, the specific implementation details of the light source synthesis type image conversion processing in this example are introduced. Figure 3 The flowchart of Example 2 is shown in FIG. Figure 3 As shown, Example 2 specifically includes the following steps:

[0098] Step S301: sequentially determine each image in the original image sample set as a first image, and determine any image in the original image sample set that is different from the first image and meets a preset condition as a second image.

[0099] Among them, the preset condition includes: the error between the light source value of the second image and the light source value of the first image is greater than the preset error threshold. The purpose of setting the preset condition is to prevent the light source synthesis processing of images with two light sources close in height. In the case where the light sources of the two images are close in height, if the light source synthesis processing is performed on the two images, the final processed image effect will not be significant. Therefore, in order to improve the distinguishability of each sample in the amplified sample set, the above preset condition is set. For example, for any first image in the original image sample set, first, the color matrix and light source value corresponding to the first image are calculated according to the color card area contained in the first image. Then, the color matrix and light source value corresponding to any other image in the original image sample set that is different from the first image are calculated in the same way. Finally, according to the light source value of the first image and the light source value of any other image, the light source angle error between the first image and any other image is calculated, and according to the preset angle error threshold, the image whose angle error with the first image is less than the threshold is eliminated from the original image sample set, and the second image is selected from the remaining images to ensure that the angle error between the first image and the second image is large.

[0100] It should be noted that the above preset condition is an optional condition. In other implementations, the preset condition may be omitted, and any image in the original image sample set that is different from the first image may be directly determined as the second image.

[0101] Step S302: generating a first color matrix corresponding to the first color card area according to the first color card area included in the first image; generating a second color matrix corresponding to the second color card area according to the second color card area included in the second image.

[0102] Step S303: Calculate a first mapping matrix for converting the first color matrix into the second color matrix.

[0103] The specific generation method of the color matrix and the mapping matrix can be referred to above and will not be repeated here.

[0104] Step S304: calculating a first light source value corresponding to the first image according to the first color card area, and calculating a second light source value corresponding to the second image according to the second color card area.

[0105] The light source value is determined based on the difference between the standard color of each color block in the color card area and the actual pixel value of each color block in the image. Specifically, the light source value can be calculated based on a preset column or preset row in the color matrix (such as the last row in the color matrix). Therefore, in this step, the first light source value corresponding to the first image and the second light source value corresponding to the second image can be calculated respectively.

[0106] Step S305: adjusting the first light source value and the second light source value according to the interpolation weight to obtain an interpolated light source value.

[0107] Specifically, by interpolating and adjusting the light source values ​​of the two images using the specified interpolation weights, a new interpolated light source value can be obtained. The interpolated light source value is different from both the first light source value and the second light source value, and is a light source value newly synthesized through the interpolation operation.

[0108] Step S306: adjusting the weight of the first mapping matrix by using the interpolation weight to obtain a first interpolation mapping matrix.

[0109] The first mapping matrix can be weighted by adjusting the interpolation weight and the unit matrix to obtain a first interpolation mapping matrix. The first interpolation mapping matrix is ​​different from the first mapping matrix, thereby ensuring that the light source in the processed image is different from both the first light source value and the second light source value.

[0110] Step S307: generating a first mapping light source value according to the first interpolation mapping matrix and the first light source value; and generating a gain adjustment matrix according to the interpolation light source value and the first mapping light source value.

[0111] The function of the gain adjustment matrix is ​​to further perform gain adjustment processing on the first interpolation mapping matrix to avoid residual errors of gray pixels in the newly generated image. The gain adjustment matrix can be a diagonal matrix, and the matrix element values ​​located on the diagonal of the diagonal matrix can be determined in the following manner: first, a first summation result between the interpolation light source value and the first preset adjustment value is calculated; then, a second summation result between the first mapping light source value and the second preset adjustment value is calculated; finally, according to the ratio between the first summation result and the second summation result, the matrix element values ​​located on the diagonal of the diagonal matrix are determined.

[0112] Step S308: adjusting the first interpolation mapping matrix by using the gain adjustment matrix to obtain a first color mapping matrix.

[0113] Specifically, a matrix multiplication operation may be performed on the gain adjustment matrix and the first interpolation mapping matrix to obtain the first color mapping matrix.

[0114] Step S309: performing a matrix operation on the image data of the first image through the first color mapping matrix to obtain a first target image.

[0115] Specifically, a matrix multiplication operation may be performed on the first color mapping matrix and the image data of the first image, thereby obtaining the first target image.

[0116] In this way, a new interpolated light source value can be synthesized according to the first light source value of the first image and the second light source value of the second image, and the mapping matrix can be adjusted according to the interpolated light source value, so as to achieve the purpose of applying the new interpolated light source value to the first target image.

[0117] In a specific implementation, two images can be selected from the original image sample set as the first image and the second image, and the first color matrix of the first image and the second color matrix of the second image can be calculated by the method in Example 1. In addition, based on the first color matrix and the second color matrix, a first mapping matrix M for converting the first color matrix into the second color matrix is ​​calculated. AtoB , and a second mapping matrix M for converting the second color matrix into the first color matrix BtoA The specific calculation details can be referred to the description of the corresponding part in Schematic 1, which will not be repeated here.

[0118] Next, according to the specified interpolation weight, the mapping matrix is ​​linearly interpolated to obtain a new mapping matrix (i.e., interpolation mapping matrix), as shown in the following formula:

[0119] M AtoC =(1-w)×I+w×M AtoB

[0120] M BtoC =(1-w)×I+w×M BtoA

[0121] Among them, w is the weight value of the interpolation weight, M AtoC is the first interpolation mapping matrix obtained according to the interpolation weight and the first mapping matrix, M BtoC is the second interpolation mapping matrix obtained according to the interpolation weight and the second mapping matrix, and I is the unit matrix.

[0122] Then, according to the interpolation weight, the first light source L of the first image is A and the second light source L of the second image B Perform interpolation processing to obtain a new interpolation light source L C , as shown in the following formula:

[0123] L C =(1-w)×L A +w×L B

[0124] Then use the first interpolation mapping matrix M AtoC and the second interpolation mapping matrix M BtoC , for the first light source L A and the second light source L B Transform and obtain the mapping light source L under light source A and light source B respectivelyC_A and L C_B , as shown in the following formula:

[0125] L C_A =L A ×M AtoC

[0126] L C_B =L B ×M BtoC

[0127] Among them, L C_A The value of is the first mapping light source value generated according to the first interpolation mapping matrix and the first light source value mentioned above, L C_B The value of is the second mapped light source value generated according to the second interpolation mapping matrix and the second light source value mentioned above.

[0128] Then, in order to avoid the residual error of gray pixels in the newly generated light source photo (i.e., the target image), the first gain adjustment matrix M is used. C_A and the second gain adjustment matrix M C_B , respectively adjust the color gain under light source A and light source B, where M C_A and M C_B They are all diagonal matrices, and the element values ​​on the diagonal of the diagonal matrix are shown in the following formula:

[0129] Val C_A =(L C +EPS) / (L C_A +EPS)

[0130] Val C_B =(L C +EPS) / (L C_B +EPS)

[0131] Among them, Val C_A is the first gain adjustment matrix M C_A The element values ​​on the diagonal of the matrix, Val C_B is the second gain adjustment matrix M C_B The element values ​​on the diagonal of the matrix, where EPS is a preset very small value to prevent the denominator from being 0; after adjustment by the gain adjustment matrix, the final color mapping matrix is ​​shown in the following formula:

[0132] M AtoC_new =M AtoC ×M C_A

[0133] M BtoC_new =M BtoC ×M C_B

[0134] It can be seen that the first color mapping matrix is ​​obtained according to the product between the first interpolation mapping matrix and the first gain adjustment matrix, and the second color mapping matrix is ​​obtained according to the product between the second interpolation mapping matrix and the second gain adjustment matrix.

[0135] Then, the new light source photo can be synthesized by the following formula:

[0136] P XC =P XA ×M AtoC_new

[0137] P YC =P YB ×M BtoC_new

[0138] It can be seen that the first target image is obtained according to the product between the image data of the first image and the first color mapping matrix, and the second target image is obtained according to the product between the image data of the second image and the second color mapping matrix.

[0139] By processing any two images in the original image sample set in the above manner, two new images can be obtained after processing, so as to obtain an enlarged sample set according to the new images.

[0140] Optionally, for any two new images in the expanded sample set, the light source value of the new image can be calculated according to the color card area, and the angular error between the two new images can be calculated according to the new light source value. Then, through a set error threshold, new images with an angle error less than the threshold are eliminated, and only new images with an angle error greater than the threshold are retained, thereby avoiding the light sources in the new images being too close.

[0141] Figure 4b FIG. 2 shows a schematic diagram of a first image and a second image before and after performing light source synthesis processing. Figure 4b As shown, a new light source is synthesized according to the first light source value in the first image and the second light source value in the second image, and the new light source is applied to the second image to obtain a target image. Therefore, the scene in the target image is consistent with the scene in the second image, but the light source information is determined based on the first image and the second image. Correspondingly, Figure 4bAlso shown is an image after white balancing for the second image, and an image after white balancing for the target image. In addition, the above two examples can also be used in combination. For example, the two examples share the same original image sample set, and in the above two examples, by determining whether the preset interpolation weight is 1, the replacement light source processing method of Example 1 is selected when the interpolation weight is 1, and the synthetic light source processing method of Example 2 is selected when the interpolation weight is less than 1. The overall process is: different data augmentation methods are flexibly selected through the value of the pre-set interpolation weight w. Specifically, when the value of the interpolation weight w is set to 1, it is equivalent to not interpolating the light sources of the two samples, and what is performed at this time is the replacement light source augmentation method. In other words, Schematic 1 is essentially a special case in Example 2 where the interpolation weight is equal to 1.

[0142] In summary, the present application provides a method for expanding a white balance dataset, aiming to solve the problem that the existing white balance datasets are limited in number and difficult to produce. The data expansion method of the present application mainly processes the light sources of the photos in the existing dataset, and expands the dataset by exchanging the light sources of any two photos or using the light sources of any two photos to interpolate to synthesize more diverse light source photos. This method can generate a large number of new photos based on the original dataset, greatly enriching the training dataset.

[0143] Through the data augmentation method of the present application, the sample diversity and quantity of the white balance data set can be effectively increased, and the performance and generalization ability of the training model can be improved, thereby providing richer data support for the development and improvement of the white balance algorithm, and further promoting the development of the field of camera white balance technology.

[0144] The present application discloses a method for augmenting a white balance data set, which is mainly augmented by original photos taken with a 24-color color card, so it is suitable for data sets shot based on 24-color color cards. Of course, the 24-color color card can also be replaced with a 12-color or 48-color color card, and the present application does not limit this. The present application can exchange existing light sources to synthesize new data samples and use existing light sources to interpolate and synthesize data samples with new light sources. Compared with traditional data augmentation operations such as random cropping, flipping and rotation that are not suitable for white balance data sets, the present application is to obtain the color card color matrix corresponding to the photo and the white balance gain by detecting the color card information in the data set photos to exchange and synthesize the light source, so that each scene shot in the data set can be augmented to more different light sources, so as to obtain a richer augmented data set based on the existing data set. This method can make up for the small number of existing public data sets, and the time-consuming and labor-intensive problems of shooting and producing large quantities of data sets by yourself. The present application only needs to select or shoot a certain number of original photos of different color temperatures in the public data set, and then use the data augmentation method to obtain a large number of augmented photos with good effects, effectively reducing the cost of data set acquisition and production.

[0145] Due to the limitations and costs of different lighting conditions, it is very difficult and expensive to obtain a large-scale real lighting dataset, and the number of public white balance datasets is relatively small. In addition, the existing commonly used image dataset augmentation technology is not applicable to white balance datasets, because the purpose of the AWB algorithm is to adapt to real scenes with different light sources to obtain a more accurate white balance effect, and the above-mentioned commonly used image dataset augmentation methods cannot simulate the changes in light sources in real scenes. In comparison, this white balance dataset augmentation technology can be augmented on the basis of existing self-shot or public datasets to obtain a large number of new light source photos and enrich the original dataset. For example, for the method of generating new data samples by replacing the light source in Example 1, if the standard sample set contains n standard photos of different light sources, and the sample sample set has a total of m photos of different scenes, then for each scene in the sample sample set, n new photos of different light sources can be generated, and the entire sample set can generate a total of n×m new sample photos. As for the method of generating new data samples by interpolating and synthesizing light sources in Example 2, if the sample set has h photos, new light sources are synthesized by interpolating the images in the sample set, and a new light source can act on the scene of the two sample images involved in the interpolation, then h×(h-1) photos can be generated. It can be seen that the amount of data in the augmented sample set obtained by the above method can be increased by at least dozens of times.

[0146] In addition, there are very few existing amplification technologies for white balance data sets. The light source exchange and new light source synthesis methods proposed in this white balance data set amplification technology can obtain amplified data with good quality to help train and improve the AWB algorithm. According to actual needs, this application can be combined with a certain scene classification method to screen and classify photo scenes in public data sets or self-shot data sets, and then cooperate with the method of this application to specifically amplify sample data of the selected scene category. In short, the white balance data set amplification technology of this application can be applied to the training of the AWB algorithm model based on deep learning to improve the effect of the AWB algorithm.

[0147] Figure 5 FIG. 4 is a structural diagram of an image processing device 40 provided in an embodiment of the present disclosure. Figure 5 , the present disclosure provides an image processing device 40, which may include:

[0148] A first acquisition module 41 is adapted to acquire a first color card area included in a first image, and generate a first color matrix corresponding to the first color card area;

[0149] A second acquisition module 42, adapted to acquire a second color card area included in the second image, and generate a second color matrix corresponding to the second color card area;

[0150] A determination module 43, adapted to determine a mapping matrix according to the first color matrix and the second color matrix, the mapping matrix being used for conversion between the first color matrix and the second color matrix;

[0151] The conversion module 44 is adapted to perform conversion processing on the first image or the second image according to the mapping matrix to obtain a target image.

[0152] The specific implementation details of each of the above modules can refer to the description of the corresponding part in the method embodiment, which will not be repeated here. The image processing device in this embodiment can execute the steps of any of the above embodiments or a combination thereof.

[0153] Reference Figure 6 The embodiment of the present disclosure also provides an electronic device 50, which includes: at least one processor 501; at least one memory 502; and one or more I / O interfaces 503, connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by at least one processor 501, and the one or more computer programs are executed by at least one processor 501 so that the at least one processor 501 can execute the image processing method provided in any of the above aspects.

[0154] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image processing method provided in any of the above aspects is implemented.

[0155] Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules / units in the apparatus disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0156] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0157] Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-temporary medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0158] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first color card area included in the first image, and generate a first color matrix corresponding to the first color card area; Acquire a second color card area included in the second image, and generate a second color matrix corresponding to the second color card area; Determine a mapping matrix according to the first color matrix and the second color matrix, wherein the mapping matrix is ​​used for conversion between the first color matrix and the second color matrix; According to the mapping matrix, the first image or the second image is converted to obtain a target image.

2. The method according to claim 1, characterized in that The mapping matrix includes: a first mapping matrix for converting the first color matrix into the second color matrix, or a second mapping matrix for converting the second color matrix into the first color matrix; And the converting process is performed on the first image or the second image according to the mapping matrix to obtain a target image, comprising: Performing matrix operation on the image data of the first image according to the first mapping matrix to obtain a first target image; or, A matrix operation is performed on the image data of the second image according to the second mapping matrix to obtain a second target image.

3. The method according to claim 1, characterized in that The conversion process includes: light source conversion process; Before converting the first image or the second image according to the mapping matrix to obtain a target image, the method further includes: Determine a first light source value corresponding to the first image according to the first color card area, and determine a second light source value corresponding to the second image according to the second color card area; According to a preset interpolation weight, adjusting the first light source value and the second light source value to obtain an interpolated light source value; According to the interpolated light source value, the mapping matrix is ​​adjusted to obtain a color mapping matrix; The target image is obtained by performing a matrix operation on the image data of the first image or the image data of the second image through the color mapping matrix.

4. The method according to claim 3, characterized in that Before the mapping matrix is ​​adjusted according to the interpolated light source value to obtain the color mapping matrix, the method further includes: performing weight transformation processing on the mapping matrix by using the interpolation weight to obtain the interpolation mapping matrix; and The adjusting the mapping matrix according to the interpolated light source values ​​to obtain a color mapping matrix includes: generating a gain adjustment matrix according to the interpolated light source values, and performing gain adjustment on the interpolation mapping matrix by using the gain adjustment matrix to obtain the color mapping matrix.

5. The method according to claim 4, characterized in that Generating a gain adjustment matrix according to the interpolated light source value comprises: Generate a mapping light source value according to the interpolation mapping matrix and the first light source value or the second light source value; The gain adjustment matrix is ​​generated according to the interpolated light source values ​​and the mapped light source values.

6. The method according to claim 5, characterized in that The gain adjustment matrix is ​​a diagonal matrix, and the matrix element values ​​located on the diagonal of the diagonal matrix are determined in the following manner: Calculating a first summation result between the interpolated light source value and a first preset adjustment value; Calculating a second summation result between the mapped light source value and a second preset adjustment value; According to the ratio between the first summation result and the second summation result, the values ​​of the matrix elements located on the diagonal line of the diagonal matrix are determined.

7. The method according to any one of claims 1 to 6, characterized in that: The first color card area is obtained according to the first physical color card, and the second color card area is obtained according to the second physical color card; the multiple color blocks included in the first physical color card correspond to the multiple color blocks included in the second physical color card one by one, and the color of any color block in the first physical color card is the same as the color of the color block at the corresponding position in the second physical color card; Generating the first color matrix corresponding to the first color card area includes: acquiring the actual pixel value of each color block in the first color card area; generating the first color matrix corresponding to the first color card area according to the actual pixel value of each color block in the first color card area and the correspondence between the plurality of color blocks in the first color card area and the plurality of matrix elements in the first color matrix; The generating of the second color matrix corresponding to the second color card area includes: obtaining the actual pixel value of each color block in the second color card area; generating the second color matrix corresponding to the second color card area according to the actual pixel value of each color block in the second color card area and the correspondence between the multiple color blocks in the second color card area and the multiple matrix elements in the second color matrix.

8. The method according to any one of claims 2 to 6, characterized in that: Before acquiring the first color card area included in the first image and generating the first color matrix corresponding to the first color card area, the method further includes: Acquire an original image sample set, and determine any image in the original image sample set as a first image; Determine any image in the original image sample set that is different from the first image and meets a preset condition as a second image; The preset condition includes: an error between the light source value of the second image and the light source value of the first image is greater than a preset error threshold.

9. The method according to claim 8, characterized in that The original image sample set further includes: a first sample set of the light source to be replaced, and a second sample set corresponding to the standard light source; the first image comes from the first sample set, and the second image comes from the second sample set; The converting the first image or the second image to obtain a target image comprises: According to the second image, a light source replacement type conversion process is performed on the first image to obtain the target image.

10. The method according to claim 9, characterized in that The original image sample set is a white balance sample set, and after obtaining the target image, the following steps are further included: Annotating the target image according to the white balance gain of the target image; The labeled target image is added to the original image sample set to expand the original image sample set.

11. An electronic device, characterized in that: The device comprises one or more memories and processors; the one or more memories store computer programs that can be executed by the one or more processors, and the computer programs implement the image processing method described in any one of claims 1 to 10 when executed by the processors.

12. A computer readable medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

Images