Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an original image which is not subjected to image signal processing; performing color detection based on the original image to obtain color information corresponding to the original image; and performing image processing on the original image according to the color information. The color gamut where the original image is located does not change, and the original image can well retain the information input to the terminal, so that when color detection is performed based on the original image, the information adjusted by each processing mode of the image signal mode can be used for color detection; therefore, the color information corresponding to the original image can be obtained more accurately for image processing. Besides, in the color detection process of the original image, the color detection can be directly carried out in the color gamut where the original image is located, the color gamut does not need to be converted to the YUV color gamut, the detected color information can be used in the whole image signal processing process, and the problem of frame delay does not exist.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] In the field of computer vision, with the gradual development of image processing technology, the scenarios and elements of image color restoration are gradually increasing, and among the basic technologies of image color restoration, there is image color detection technology.

[0003] In the conventional technology, the image color detection is mainly performed in the YUV color gamut, and the input data of the color detection belongs to the data of the YUV color gamut. However, in some scenarios, when the image color detection is performed based on the YUV color gamut, the detection result has a poor accuracy. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can accurately detect the color of an image.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Acquire an original image without image signal processing;

[0007] Performing color detection based on the original image to obtain color information corresponding to the original image;

[0008] Image processing is performed on the original image according to the color information.

[0009] In a second aspect, the present application also provides an image processing device. The device comprises:

[0010] An acquisition module, used for acquiring an original image without image signal processing;

[0011] A detection module, used to perform color detection based on the original image to obtain color information corresponding to the original image;

[0012] The image processing module is used to perform image processing on the original image according to the color information.

[0013] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0014] Acquire an original image without image signal processing;

[0015] Performing color detection based on the original image to obtain color information corresponding to the original image;

[0016] Image processing is performed on the original image according to the color information.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0018] Acquire an original image without image signal processing;

[0019] Performing color detection based on the original image to obtain color information corresponding to the original image;

[0020] Image processing is performed on the original image according to the color information.

[0021] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0022] Acquire an original image without image signal processing;

[0023] Performing color detection based on the original image to obtain color information corresponding to the original image;

[0024] Image processing is performed on the original image according to the color information.

[0025] The above-mentioned image processing method obtains the original image without image signal processing; since the signal processing methods such as white balance and color correction do not act on the original image, the color domain of the original image does not change, and the original image can better retain the information when it is input to the terminal. Therefore, when color detection is performed based on the original image, the information to be adjusted by each processing method of the image signal method is still retained in the original image, and this information can be used for color detection, so as to more accurately obtain the color information corresponding to the original image, and then use this color information for image processing. In addition, in the process of color detection of the original image, it can be directly performed in the color domain where the original image is located, and the detected color information can be used in the entire image signal processing process, and there is no problem of frame delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0028] Figure 2 is a flowchart of an image processing method in one embodiment;

[0029] Figure 3 A flowchart of obtaining color information corresponding to an original image in one embodiment;

[0030] Figure 4 is a schematic diagram of a spectral sensitivity function curve in one embodiment;

[0031] Figure 5 A flowchart of a process for generating a preset color space conversion relationship in one embodiment;

[0032] Figure 6 A flowchart of a process of applying a preset color space conversion relationship in one embodiment;

[0033] Figure 7 Schematic diagram of the effect before and after the enhancement of the orange-yellow color in one embodiment;

[0034] Figure 8 is a schematic diagram of the effect of white balance on color detection in one embodiment;

[0035] Fig. 9 It is a schematic diagram of the effects under different light sources in one embodiment;

[0036] Fig.10 A schematic diagram of the principle of displaying an original image in an embodiment;

[0037] Fig.11 It is a flowchart of traditional image signal processing;

[0038] Fig.12 is a schematic diagram of a flow chart of image signal processing in one embodiment;

[0039] Fig.13 is a schematic diagram of an original image and a comparison image in one embodiment;

[0040] Fig.14 is a schematic diagram of energy distribution of spectral channels in one embodiment;

[0041] Fig.15 is a flowchart of an image processing method in one embodiment;

[0042] Fig.16 A schematic diagram of the effect of color detection on image restoration in one embodiment;

[0043] Fig.17 A schematic diagram of the effect of color detection on accurate viewing angle judgment in one embodiment;

[0044] Fig.18 is a structural block diagram of an image processing device in one embodiment;

[0045] Fig.19 A computer device for image processing in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0048] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0049] In one embodiment, Figure 2 As shown, an image processing method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0050] Step 202: Acquire an original image without image signal processing.

[0051] Image Signal Processing (ISP) is the image quality enhancement or color gamut conversion processing performed on the original image. Optionally, image signal processing includes white balance (Auto White Balance, Abw), color correction (Color Correction Matrix, CCM), gamma (Gamma), color space transformation (Color Space Transformation, CST) and image processing under the Yuv color gamut. Optionally, the original image is the image before white balance processing to avoid the color influence caused by white balance processing, so as to more accurately detect the color of the original image; optionally, the scene light source information of the original image is obtained through white balance processing to ensure the accuracy of color detection while saving multi-spectral color sensors.

[0052] The original image is an image that has not been processed by the terminal. Optionally, the original image is an unprocessed image collected by the sensor of the terminal, and the image has a large amount of data and can reflect more detailed information. Optionally, the detailed information reflected by the original image is represented by the channel value of the image, which includes the color channel value and may also contain the brightness channel value; the original image can be obtained by exposing the pixel array in the image sensor.

[0053] Optionally, the original image is image data that has not been processed and compressed. The compression process of image data often causes image loss, and the uncompressed image data represents richer information. Optionally, the original image can be a raw image, which is an image stored in a raw format. The raw image has a relatively large amount of data and is an image that has not been processed or compressed.

[0054] In one embodiment, obtaining an original image without image signal processing includes: obtaining at least a portion of a raw image obtained by an image sensor through exposure. Thus, the original image can be not only a complete raw image, but also a portion of a raw image obtained by cropping. When the original image is a portion of a raw image, the amount of data processed in this case is reduced, which can improve the calculation efficiency. Moreover, since cropping does not change the channel value of the image, the accuracy of color detection of the image can be guaranteed to remain unchanged.

[0055] Step 204: Perform color detection based on the original image to obtain color information corresponding to the original image.

[0056] The color information is the color detection result of the original image. Optionally, the color information is used to perform image enhancement processing or color change processing on the original image. Optionally, the color information can be converted into a CIE color system, and then the color space distribution statistics of the color information in the CIE color system are performed to obtain the color distribution; and then the original image is enhanced or changed according to the color information and the color distribution.

[0057] Optionally, the color information is color information in the raw domain or in the CIE color system, and the color information in the raw domain or in the CIE color system has less loss and higher accuracy. Optionally, when the color information belongs to the raw domain, the speed of color detection is faster, which can make the frame delay smaller; the frame delay refers to the speed of the continuous display process of multiple frames of images; the smaller the frame delay, the higher the image coherence of the continuous display process, and when the frame delay is too large and exceeds a certain threshold, the image may be refreshed untimely or freeze.

[0058] In one embodiment, color detection is performed based on the original image to obtain color information corresponding to the original image, including: detecting the color channel value of each pixel in the original image. By detecting the color channel value of the original image, the color information belongs to the raw domain.

[0059] In one embodiment, color detection is performed based on the original image to obtain color information corresponding to the original image, including: converting the color channel values ​​of the original image into color channel values ​​under a preset light source; detecting the color channel values ​​under the preset light source; wherein the preset light source is a pre-set light source, including but not limited to 5000K, 6500k or 7500k light sources. Thus, the influence of different light sources on color information detection is weakened by the preset light source, so that the accuracy of color information is increased. In addition, this process is not performed according to the linear processing framework of image signal processing, the frame delay is relatively low, and the original image loses relatively less information, and the accuracy of the detected color information is relatively high.

[0060] In one embodiment, the color information is the result of the combined effect of the sensitivity (SSF) of the sensor response and the light source (spd), and the reflectivity of the object. The color information is used to express that the effect of the light source on the color can be weakened under the preset light source. The color information is the result of the superposition of the above sensitivity, light source and reflectivity. For images collected by the same sensor, the color information is determined based on the lighting conditions and reflectivity generated by the light source, so the light source is converted to a preset light source to reduce the interference caused by the light source on the reflectivity, so that the reflectivity is close to the true color of the object. Among them, the preset light source can be a standard light source, which is a light source specified for unified color measurement, including but not limited to international standard artificial daylight source; illustratively, the standard light source can be a D65 or D50 light source.

[0061] In one embodiment, converting the color channel values ​​of the original image into the color channel values ​​under the preset light source includes: determining a preset weight coefficient according to the image information of the original image; performing weighted processing on the color channel values ​​of the original image according to the preset weight coefficient to obtain the color channel values ​​under the preset light source. Thus, according to the image information of the original image, the preset weight coefficient is adaptively screened, and then the color channel values ​​are weighted by the preset weight coefficient, so that the color channel values ​​of the original image are more accurately converted into the color channel values ​​under the preset light source, so that the accuracy of the color information is further increased.

[0062] In a feasible implementation, determining the preset weight coefficient according to the image information of the original image includes: determining the reference light source of the original image according to the scene light source information of the original image; and selecting the preset weight coefficient according to the reference light source. Thus, the reference light source of the original image is characterized by the scene light source information, and the preset weight coefficient is selected according to the reference light source, so that the color channel value under the preset light source is more accurate.

[0063] In a specific implementation, before determining the reference light source of the original image based on the scene light source information of the original image, the method further includes: acquiring the scene light source information through a spectral sensor, or generating the scene light source information through a target step in image signal processing. Thus, the scene light source information within a specific wavelength range is collected through components such as a photosensitive element or a filter of the spectral sensor, or the scene light source information is generated more accurately through certain processing methods of image signal processing.

[0064] Step 206: performing image processing on the original image according to the color information.

[0065] In an optional embodiment, performing image processing on the original image according to the color information includes: performing image enhancement processing or color change processing on the original image according to the color information.

[0066] In an optional embodiment, performing image processing on the original image includes: any image signal processing in the above step 202 .

[0067] The above-mentioned image processing method obtains the original image without image signal processing; since the signal processing methods such as white balance and color correction do not act on the original image, the color domain of the original image has not changed, and the original image can better retain the information when it is input to the terminal. Therefore, when color detection is performed based on the original image, the information to be adjusted by each processing method of the image signal method is still retained in the original image. This information can be used for color detection, so as to more accurately obtain the color information corresponding to the original image, and then use this color information for image processing. In addition, in the process of color detection of the original image, it can be directly performed in the color domain where the original image is located, and the detected color information can be used in the entire image signal processing process, and there is no problem of frame delay. In this way, while increasing accuracy, the accuracy and stability are improved by more than 50%.

[0068] In one embodiment, Figure 3 As shown, color detection is performed based on the original image to obtain color information corresponding to the original image, including:

[0069] Step 302: Perform chromatic adaptation processing on the original image according to a preset light source conversion relationship to obtain a chromatic adapted image.

[0070] The preset light source conversion relationship is an image color conversion relationship based on different light source settings. Optionally, the preset light source conversion relationship for color adaptation can be set based on CIECAM02 or Bradford. The preset light source conversion relationship is used to convert the light source in the original image to obtain a color-adapted image suitable for color detection by changing the lighting conditions in the original image. Optionally, the preset light source conversion relationship is selected based on the information of the original image so that the original image can be more accurately adjusted to a color-adapted image.

[0071] The chromatic adaptation image is an image in which the light source is converted to a preset light source; the color detection accuracy under the preset light source is relatively high. Optionally, the original image is subjected to chromatic adaptation processing according to the preset light source conversion relationship to obtain a chromatic adaptation image; the purpose of this process is to adaptively adjust the light source in the original image according to the dynamic mechanism of the human visual system to eliminate the influence of different light sources in color detection.

[0072] In one embodiment, the preset light source conversion relationship includes a preset chromatic adaptation conversion weight. The preset chromatic adaptation conversion weight is a conversion parameter set based on a certain algorithm; optionally, it is a preset parameter in the Bradford conversion method, including a reference light source to be detected, and a preset chromatic adaptation conversion weight matrix [MA] and its inverse matrix [MA] -1 , specifically as follows;

[0073] [MA]=[0.8951000 0.2664000 -0.1614000

[0074] -0.7502000 1.7135000 0.0367000

[0075] 0.0389000 -0.0685000 1.0296000];

[0076] [MA] -1 =[0.9869929 -0.1470543 0.1599627 0.4323053 0.5183603 0.0492912

[0078] -0.0085287 0.0400428 0.9684867].

[0079] According to a preset light source conversion relationship, the original image is subjected to chromatic adaptation processing to obtain a chromatic adaptation image, including: detecting a reference light source in the original image; according to target chromatic adaptation conversion weights between the reference light source and the standard light source, the color channel values ​​of the original image are subjected to weighted processing to obtain a chromatic adaptation image under the standard light source; wherein the target chromatic adaptation conversion weights are a set of preset chromatic adaptation conversion weights among the preset chromatic adaptation conversion weights.

[0080] The reference light source is the light source in the original image. The reference light source may be the light source during the shooting process of the original image. The reference light source is determined and characterized based on the scene light source information. The scene light source information may include the color information of the original image, and may also include transparency, identification data, or other information used to determine the light source.

[0081] The standard light source is a light source whose color detection accuracy meets a certain regulation. Optionally, the standard light source is a certain international standard artificial daylight source. When color detection is performed on the color adaptation image under the standard light source, the color channel value obtained is more accurate. Exemplarily, the standard light source is a D65 light source, which is a light source with a color temperature of 6500 Kelvin for outdoor use.

[0082] The preset chromatic adaptation conversion weight is a candidate light source conversion weight. Each preset chromatic adaptation conversion weight is used to convert the color channel value of the light source corresponding to it. Optionally, the preset chromatic adaptation conversion weight is set for chromaticity, which refers to the saturation or purity of the color. A higher chromaticity value indicates that the color is more saturated or brighter, while a lower chromaticity value indicates that the color is darker or lighter. Optionally, in the case where the original image is a RAW RGB image, the color domain to which the RGB image belongs only contains red channel values, green channel values, and blue channel values, and there is no chromaticity channel value. At this time, the preset chromatic adaptation conversion weight is used to adjust the relative intensity between different color channel values ​​of the RGB image, thereby changing the color saturation of the original image, and realizing chromaticity adjustment with color saturation. Optionally, the preset chromatic adaptation conversion weight is a conversion matrix. Optionally, the preset chromatic adaptation conversion weight can be a Bradford conversion matrix; in the Bradford conversion matrix, it is used to determine the change ratio of each color channel value to itself during the change process between the reference light source and the standard light source. The target chromatic adaptation conversion weight is a preset chromatic adaptation conversion weight selected based on the reference light source.

[0083] In an optional embodiment, detecting a reference light source in an original image includes: detecting scene light source information in the original image by an optical sensor, or performing image information processing based on the original image to obtain scene light source information; and determining a reference light source according to the scene light source information.

[0084] In a feasible implementation, the color channel values ​​of the original image are weighted according to the target chromatic adaptation conversion weights between the reference light source and the standard light source, including: determining the target chromatic adaptation conversion weights for converting the reference light source to the standard light source from the preset chromatic adaptation conversion weights; and weighting the color channel values ​​of the original image according to the target chromatic adaptation conversion weights. Thus, for different reference light sources, the target chromatic adaptation conversion weights are adaptively selected so that after the color channel values ​​of the original image are converted into the color channel values ​​of the chromatic adaptation image, the color channel values ​​of the chromatic adaptation image can still more accurately represent the information contained in the original image.

[0085] In an optional embodiment, determining a target chromatic adaptation conversion weight for converting a reference light source into a standard light source from preset chromatic adaptation conversion weights includes: according to the reference light source, searching for the target chromatic adaptation conversion weight for converting the reference light source into the standard light source in a mapping table, blockchain or database of preset chromatic adaptation conversion weights.

[0086] By detecting the reference light source in the original image, the information of the reference light source is adaptively adjusted; then the color channel values ​​of the original image are weighted according to the target color adaptation conversion weights between the reference light source and the standard light source to obtain the color adaptation image under the standard light source. Since the target color adaptation conversion weights are adaptively adjusted for each reference light source and are directly used to process the color channel values ​​of the original image, they can be made closer to the color channel values ​​under the standard light source to improve the accuracy of color detection.

[0087] In an optional implementation, detecting a reference light source in the original image includes: detecting scene light source information in the original image by a multi-spectral color sensor; and determining a reference light source according to the scene light source information.

[0088] A multispectral color sensor is a sensor that analyzes scene light source information based on a certain range of wavelengths. Through the multispectral color sensor, the scene light source information of the original image in a certain color space can be obtained. Optionally, the scene light source information detected by the multispectral color sensor is the color space of the human eye color perception model; optionally, the scene light source information detected by the multispectral color sensor is in the CIE-XYZ color space, and the specific light source can be characterized by brightness and chromaticity.

[0089] Optionally, the multispectral color sensor is composed of an array or multiple photosensitive elements, each of which is used to filter light in a certain wavelength range to separate the light entering the sensor according to different wavelengths. Each photosensitive element corresponds to a filter in a specific wavelength range, which only allows light in the corresponding wavelength range to pass through, and blocks light of other wavelengths. In this way, each photosensitive element only receives light signals in a specific wavelength range, and then converts the light signals into electrical signals to obtain scene light source information.

[0090] In a specific embodiment, the scene light source color information in the original image is detected by a multispectral color sensor, including: during the process of shooting the original image, the scene light source is detected on the environment in which the original image is shot by the multispectral color sensor to obtain the scene light source color information in the original image.

[0091] In a feasible implementation, determining the reference light source according to the scene light source information includes: in a spectrum containing multiple preset light sources, using a preset light source that matches the scene light source information as a reference light source.

[0092] The spectral distribution energy of the original image is detected by a multi-spectral color sensor, which detects the original image from an angle other than the color channel value. This process does not perform calculations based on the color channel value, so as to more accurately determine the reference light source.

[0093] In an optional implementation, detecting a reference light source in an original image includes: performing white balance processing on the original image to obtain scene light source information in the original image; and determining a reference light source according to the scene light source information.

[0094] White balance processing is a processing method for correcting the degree of color shift. In one embodiment, white balance processing is performed on the original image to obtain scene light source information in the original image, including: based on the color channel values ​​of the original image, white balance processing is performed on the color channel values ​​of the original image to obtain the color channel values ​​of the white balance image; the color channel values ​​of the white balance image belong to the scene light source information in the original image.

[0095] White balance processing is a type of image signal processing based on the color space of the original image. The scene light source information obtained only through white balance processing still has high accuracy in most scenes, and there is no need to configure a multispectral color sensor hardware device.

[0096] In one embodiment, the preset light source conversion relationship includes a preset color space conversion relationship. The preset color space conversion relationship is used to convert the original image into the color space of the human eye color perception model to eliminate the influence caused by different light sources from the perspective of human eye color perception. The color space of the human eye color perception model is a color space set to simulate the color perception rules of the human eye, and can be an XYZ color space set by the International Commission on Illumination (CIE); wherein X represents the weighted sum of the red component and the brightness, Y represents the weighted sum of the green component and the brightness, and Z represents the weighted sum of the blue component and the brightness.

[0097] According to a preset light source conversion relationship, the original image is subjected to color adaptation processing to obtain a color adapted image, including: according to a preset color space conversion relationship, the color channel value of the original image is converted to the color space of the human eye color perception model to obtain the original image in the human eye perception color space; based on the original image in the human eye perception color space, the color adaptation processing is performed to obtain the color adapted image.

[0098] In an optional embodiment, the color channel values ​​of the original image are converted to the color space of the human eye color perception model according to a preset color space conversion relationship, including: determining a color space conversion matrix between the original image and CIE-XYZ; and converting the color channel values ​​of the original image according to the color space conversion matrix.

[0099] Therefore, by converting the color space, the color channel values ​​of the converted image are made consistent with human eye perception, so as to better eliminate the influence of the light source when performing color adaptation processing.

[0100] In a feasible embodiment, based on the original image in the color space perceived by the human eye, color adaptation processing is performed to obtain a color adaptation image, including: detecting a reference light source in the original image in the color space perceived by the human eye; performing weighted processing on the color channel values ​​of the original image in the color space perceived by the human eye according to the target color adaptation conversion weight between the reference light source and the standard light source, to obtain a color adaptation image under the standard light source. Therefore, the preset light source conversion relationship includes a preset color space conversion relationship and a preset color adaptation conversion weight, and the preset color space conversion relationship is first executed to convert the original image to conform to the human eye perception rules, and then the color channel value is weighted based on the preset color adaptation conversion weight to make the color adaptation processing effect better.

[0101] Step 304: Perform color detection on the chromatic adaptation image to obtain color information corresponding to the original image.

[0102] The color information corresponding to the original image is the color detection result of the chromatic adaptation image. For images collected by the same sensor, the color information is determined based on the lighting conditions generated by the light source and the reflectivity of the objects contained in the image, while the lighting conditions in the chromatic adaptation image are generated based on the preset light source, which can more accurately reflect the color information corresponding to the original image.

[0103] In one embodiment, color detection is performed on a chromatic adaptation image to obtain color information corresponding to an original image, including: obtaining color channel values ​​of the chromatic adaptation image; determining the color channel values ​​of the chromatic adaptation image as color channel values ​​corresponding to the original image according to corresponding pixels in the chromatic adaptation image and the original image; wherein corresponding pixels refer to pixels in the original image and the chromatic adaptation image that are at the same position or in the neighborhood. Corresponding pixels are statistics of color channel values ​​determined from the perspective of pixels, and their accuracy is relatively high.

[0104] According to the preset light source conversion relationship, the original image is color-adapted to obtain a color-adapted image, and the light source in the original image is adaptively adjusted to avoid the influence of the light source on color detection, so that the color information corresponding to the original image is more accurate. Therefore, after the original image is color-adapted, the detection accuracy of color information is improved in various extreme scenes caused by the light source, and the restoration effect is enhanced.

[0105] In one embodiment, there is a preset spatial conversion relationship between the color space of the color information and the color statistical space corresponding to the human eye color perception model, and the dimensions covered by the color statistical space include brightness, saturation and hue; image processing is performed on the original image according to the color information, including: converting the color information into the color statistical space according to the preset spatial conversion relationship to obtain color information of the color statistical space; and statistically analyzing the color distribution based on the color information of the color statistical space.

[0106] The color statistical space is a color space established and converted based on the human eye color perception model. The color statistical space is independent of the device and can still reuse the color statistical method under the HSV space with dimensions of hue, saturation, and lightness, so as to be able to cooperate with the color statistical method, so that this solution has a wider adaptability. Optionally, the color statistical space is the CIE lch space; the "CIELCh" space is a color space based on CIELAB, which uses polar coordinates C* (chromaticity, relative saturation) and h° (hue angle, hue angle in the CIELAB color wheel) instead of Cartesian coordinates a and b. The lightness L* of CIELAB remains unchanged.

[0107] By presetting the spatial conversion relationship, the color information can be quickly converted to a color space with brightness, saturation and hue, so that the color information obtained above can be substituted into various color statistical methods, making this solution have a wider range of applications.

[0108] In one embodiment, the step of generating a preset light source conversion relationship includes: obtaining a spectral sensitivity variation rule of an image sensor; determining each original image sample based on each light source, reflectance data set and spectral sensitivity variation rule, and determining each human eye perceived image sample based on each light source, reflectance data set and a human eye color perception model; determining a color space conversion relationship based on a conversion relationship between each human eye perceived image sample and each original image sample.

[0109] The spectral sensitivity change rule is used to characterize the change rule of the image sensor under the influence of the sensitivity influencing factor; the sensitivity influencing factor can be the light source of the shooting environment, or the reflectivity of the photographed object, and can cover both the light source of the shooting environment and the reflectivity of the photographed object. Optionally, the spectral sensitivity change rule can be a spectral sensitivity function (SSF), which can be obtained through a spectrometer strategy. The light source data set contains different light sources, and the reflectivity data set contains the reflectivity of different objects; the original image samples and the human eye perception image samples are both samples used to determine the color space conversion relationship. The original image samples are the images before the conversion, and the human eye perception image samples are the images after the conversion.

[0110] Each original image sample is determined based on the light source data set, the reflectance data set and the spectral sensitivity variation rule. Each original image sample is generated by calculation, so that there are more types of original image samples. Each standard light source image sample is determined based on the light source data set, the reflectance data set and the human eye color perception model, so that this solution can be widely applied to most light source scenes.

[0111] Specifically, the preset light source conversion relationship is the conversion from Raw RGB to CIE XYZ space, which is calibrated using a spectrometer. The measuring instrument can test and obtain the spectral sensitivity function curve of the mobile phone module. The curve is integrated with the existing light source / reflectivity data set (light source 300+, reflectivity 10w+) to obtain Raw RGB data above one million levels; wherein, the spectral sensitivity function curve is as follows Figure 4 shown.

[0112] In one embodiment, each light source, reflectance data set and spectral sensitivity variation rule is as follows: Figure 5 As shown in (a), (b) and (c) in FIG, the color channel value of each original image sample is obtained by convolution of these parameters; and each light source, reflectance data set and human eye color perception model are respectively as shown in Figure 5 As shown in (d), (e) and (f) of FIG, the color channel values ​​of each image sample perceived by the human eye are obtained by convolution of these parameters. Then, based on the color channel values ​​of each original image sample and the color channel values ​​of each image sample perceived by the human eye, the preset light source conversion relationship is calculated; the preset light source conversion relationship is the conversion matrix K, and the conversion matrix K is determined by the least squares method.

[0113] Correspondingly, such as Figure 6 As shown, according to the preset color space conversion relationship, the color channel value of the original image is converted to the color space of the human eye color perception model to obtain the original image in the human eye perception color space, including: in the original image, the color channel value of each pixel is determined; the color channel value of each pixel is respectively matrix multiplied with the conversion matrix K to obtain the original image in the human eye perception color space. Among them, in the original image, the color channel value of each pixel is determined, represented as Raw rgb1, Raw rgb2, Raw rgb3...Raw rgbN, N is a positive integer; for the original image in the human eye perception color space, its color channel value is represented as Raw Xyz1, Raw Xyz2, Raw Xyz3...Raw XyzN, N is a positive integer.

[0114] In order to more clearly illustrate the technical solution of this application, the research process and related technologies of this solution are described: Detecting and restoring the color of objects in images is one of the most important contents in the field of computer vision. When the algorithm detects the color and spatial distribution of the object, the existing object color can be enhanced or changed through the back-end algorithm; Figure 7 After detecting that the orange is orange-yellow in color, targeted color enhancement is performed to bring a better visual experience. Figure 7 The (a) image in the figure is the image that has not been enhanced. Figure 7 Figure (b) is the enhanced image.

[0115] Image signal processing includes: white balance (Auto White Balance, Abw), demosaic processing (Demo saic), color correction (Color Correction Matrix, CCM), gamma (Gamma) of RGB images in sequence, until the RGB image is converted to the YUV color space (Color Space Transformation, CST) and then image processing in the YUV color space.

[0116] The advantages of object color detection in the YUV domain include but are not limited to: first, the image signal has been processed at the front end. When there are no serious errors in the image signal processing, the object color tendencies of different devices are consistent, so the detection strategy is more unified; second, there are a large number of images with SRGB color domain on the Internet, which can be uniformly converted into YUV data for extensive verification.

[0117] There are generally two solutions for object color detection in the YUV domain: YUV-based object color detection determines the color of each pixel in the target image by counting the position of each pixel in the YUV. HSV-based object color detection first converts the image YUV data to the HSV space, Yuv->sRGB->HSV. Statistical color distribution in the HSV space is used to determine the color of the object, such as skin color.

[0118] However, if there is an error in image signal processing, the color detection of the object in the image may be abnormal due to the image signal processing. At this time, if the algorithm used for color detection is executed by any image signal processing method, the detection of the YUV domain image generated by the image signal processing will cause several problems. For example, Figure 8 In the normal white balance condition shown in Figure (a), the object color detection detects a high-saturation yellow; Figure 8 In the case of cold white balance shown in Figure (b), the object color detection detects low-saturation yellow; Figure 8 In the normal white balance condition shown in Figure (c), the object color detection detects a pink hue; Figure 8 In the white balance deviation shown in Figure (d), when detecting pure colors, the hue is seriously wrong.

[0119] In one embodiment, Fig. 9 As shown in the figure, color is the result of the combined effect of the sensor's sensitivity (SSF), the light source (spd), and the object's reflectivity. The object's color itself is expressed under white light (D65), that is, after removing the influence of the light source color. The figure below shows that the same group of objects present different colors under different light sources. Fig. 9Figure (a) is under the illumination of cold light source. Fig. 9 Figure (a) is under the illumination of warm light source.

[0120] In one embodiment, Fig.10 As shown in the figure, color information is the result of the superposition of the above three factors. When the device is fixed, the color of the object depends on the illumination and reflectivity. The reflectivity obtained is used to represent the true color of the object. Therefore, the illumination needs to be restored to white light, which has the characteristics of equal energy and no interference with reflectivity. The D65 light source can be used to represent white light.

[0121] In one embodiment, Fig.11 As shown, in the current architecture, algorithms related to the raw and RGB domains mostly use the object color detection results transmitted by YUV; there is a certain frame delay, and the detection results are affected by the front-end module in real time.

[0122] Correspondingly, such as Fig.12 As shown, color detection is moved into the white balance processing process, because the white balance processing process is the earliest color processing module in the ISP pipeline. Object color detection is performed in this module, and the result can cover the entire ISP without the problem of frame delay. Specifically, the process of object color detection can be implemented through steps 202-204.

[0123] In a specific embodiment, the Srgb domain is a known image, such as Fig.13 (a), (b), (c), and (d) in the figure; correspondingly, the original images are in the Raw rgb space, which are device-related images, namely (e) corresponding to (a), (f) corresponding to (b), (g) corresponding to (c), and (h) corresponding to (d).

[0124] In one embodiment, the spectral channel distribution energy of a multi-spectral color sensor is as follows: Fig.14 As shown, its wavelength coverage range is 350 to 1000 nm, and the sensor field of view angle exceeds 90 degrees. It can output the color information of the scene light source color in the CIE-XYZ color space in real time, so as to detect the scene light source information in the original image through the multispectral color sensor.

[0125] In one embodiment, after the color channel value is converted to the CIE-XYZ color space by presetting the light source conversion relationship, the influence of the light on the object color is removed by chromatic adaptation. Specifically, Bradford conversion is used; the object color XYZs under the current scene light source is converted to the object color XYZd under the D65 light source, and XYZd is used to characterize the color of the object under white light. Bradford is a linear conversion, and the conversion method is as follows:

[0126]

[0127] The calculation process of [M] is as follows: the preset color adaptation conversion weight matrix [MA] and its inverse matrix [MA]-1 are as follows;

[0128] [MA]=[0.8951000 0.2664000 -0.1614000

[0129] -0.7502000 1.7135000 0.0367000

[0130] 0.0389000 -0.0685000 1.0296000];

[0131] [MA] -1 =[0.9869929 -0.1470543 0.1599627 0.4323053 0.5183603 0.0492912

[0133] -0.0085287 0.0400428 0.9684867].

[0134]

[0135]

[0136] Among them, X WS ,Y WS ,Z WS Represents the reference light source XYZ of the scene where the original image is located. This set of values ​​is provided by the multispectral color sensor in. WD ,Y WD ,Z WD XYZ represents the target light source. This set of values ​​uses the standard values ​​of 0.95047, 1.00000, and 1.08883 under the D65 light source. So according to the above formula, ρ is calculated s ,γ s ,β s and ρ D ,γ D ,β D .

[0137] Based on this, the matrix M can be determined as follows:

[0138]

[0139] In one embodiment, the color space CIE Xyz where the human eye perception model is located is converted to the color statistical space CIE Lch space corresponding to the human eye perception model; after entering the CIE Lch space (brightness, saturation, hue space), the color distribution can be statistically analyzed using a statistical method similar to the HSV space.

[0140] In a specific embodiment, Fig.15 As shown, the object color detection is moved from the yuv domain to the raw domain, the XYZ of the output light source color of the multispectral sensor is obtained, the spectral sensitivity function SSF of the module is measured, the raw RGB and CIE XYZ information of the module are generated through the existing light source and reflectance data, the conversion matrix K is calculated, and the BradFord matrix M is calculated. The XYZ->Lch converted by the M matrix is ​​used for color statistics in the LCH space.

[0141] Based on this, the effects of this embodiment include:

[0142] First, by moving object color detection from the YUV domain to the RAW domain, the algorithm in the RAW / RGB domain (before the YUV domain) completely solves the interference of the image signal processing process on the detection result, and the stability is improved by more than 50%.

[0143] Secondly, in the process of object color detection in the raw domain, since the color adaptation conversion is performed, the influence of lighting is effectively removed, which can bring better detection accuracy and restoration effect in various extreme scenes. Specifically, the multispectral sensor provides the real color information Xyz of the light source. Then, the color of the detected object is converted through BradFord color adaptation to remove the influence of the original lighting and obtain the color of the object under white light (complete reaction reflectance). This can effectively correct the judgment of color under different lighting. For example, Fig.16 and Fig.17 As shown, Fig.16 There are reflections in (a) and (c) of the figure, and the color is darker. Correspondingly, Fig.16 Figures (b) and (d) are processed based on an embodiment of this case, and their colors are closer to the real scene; Fig.17 Figure (a) is the main camera perspective. If the blue wall under the yellow light is not illuminated, it may look like this: Fig.17 Figure (b) in the figure identifies the correct telephoto perspective, which is more likely to be Fig.17 (c) Figure 1 The same method can be used to identify incorrect telephoto angles.

[0144] Again, the conversion relationship of raw rgb->CIE xyz is obtained through spectrometer calibration, and then the conversion relationship of raw rgb->CIE xyz->CIE lab->CIE lch is further derived. In this way, the association between raw rgb and CIElch space can be directly established. This space is a device-independent color space and is very convenient for statistical color distribution.

[0145] In addition to Bradford simple chromatic adaptation to remove the influence of lighting, you can also consider using more complex chromatic adaptation methods, such as CIE CAT02. The advantage of CIE CAT02 is that you can decide the degree of removal when removing the influence of lighting. However, from the perspective of object color detection itself, Bradford simple chromatic adaptation is the most ideal method to completely remove lighting.

[0146] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0147] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0148] In one embodiment, Fig.18 As shown, an image processing device is provided, comprising:

[0149] An acquisition module 1802 is used to acquire an original image without image signal processing;

[0150] A detection module 1804 is used to perform color detection based on the original image to obtain color information corresponding to the original image;

[0151] The image processing module 1806 is used to perform image processing on the original image according to the color information.

[0152] In one embodiment, the detection module 1804 is used to:

[0153] According to a preset light source conversion relationship, the original image is subjected to chromatic adaptation processing to obtain a chromatic adaptation image;

[0154] Performing color detection on the chromatic adaptation image to obtain color information corresponding to the original image.

[0155] In one embodiment, the preset light source conversion relationship includes a preset chromatic adaptation conversion weight; the detection module 1804 is used to:

[0156] Detecting a reference light source in the original image;

[0157] According to the target chromatic adaptation conversion weights between the reference light source and the standard light source, the color channel values ​​of the original image are weightedly processed to obtain a chromatic adaptation image under the standard light source; wherein the target chromatic adaptation conversion weights are a set of preset chromatic adaptation conversion weights among the preset chromatic adaptation conversion weights.

[0158] In one embodiment, the detection module 1804 is used to:

[0159] Detecting scene light source information in the original image by a multispectral color sensor;

[0160] The reference light source is determined according to the scene light source information.

[0161] In one embodiment, the detection module 1804 is used to:

[0162] Performing white balance processing on the original image to obtain scene light source information in the original image;

[0163] The reference light source is determined according to the scene light source information.

[0164] In one embodiment, the preset light source conversion relationship includes a preset color space conversion relationship; the detection module 1804 is used to:

[0165] According to the preset color space conversion relationship, the color channel values ​​of the original image are converted into the color space of the human eye color perception model to obtain the original image in the human eye perception color space;

[0166] Based on the original image in the human eye perception color space, chromatic adaptation processing is performed to obtain a chromatic adaptation image.

[0167] In one embodiment, there is a preset space conversion relationship between the color space of the color information and the color statistical space corresponding to the human eye color perception model, and the dimensions covered by the color statistical space include brightness, saturation and hue; the image processing module 1806 is used to:

[0168] According to the preset space conversion relationship, the color information is converted into the color statistical space to obtain the color information of the color statistical space;

[0169] Color distribution is counted based on the color information in the color statistical space.

[0170] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0171] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.19 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0172] Those skilled in the art will understand that Fig.19The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0173] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0174] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0175] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: include: Acquire an original image without image signal processing; Performing color detection based on the original image to obtain color information corresponding to the original image; Image processing is performed on the original image according to the color information.

2. The method according to claim 1, characterized in that The performing color detection based on the original image to obtain color information corresponding to the original image includes: According to a preset light source conversion relationship, the original image is subjected to chromatic adaptation processing to obtain a chromatic adaptation image; Performing color detection on the chromatic adaptation image to obtain color information corresponding to the original image.

3. The method according to claim 2, characterized in that The preset light source conversion relationship includes a preset chromatic adaptation conversion weight; and performing chromatic adaptation processing on the original image according to the preset light source conversion relationship to obtain a chromatic adaptation image includes: Detecting a reference light source in the original image; According to the target chromatic adaptation conversion weights between the reference light source and the standard light source, the color channel values ​​of the original image are weightedly processed to obtain a chromatic adaptation image under the standard light source; wherein the target chromatic adaptation conversion weights are a set of preset chromatic adaptation conversion weights among the preset chromatic adaptation conversion weights.

4. The method according to claim 3, characterized in that The detecting the reference light source in the original image comprises: Detecting scene light source information in the original image by a multispectral color sensor; The reference light source is determined according to the scene light source information.

5. The method according to claim 3, characterized in that: The detecting the reference light source in the original image comprises: Performing white balance processing on the original image to obtain scene light source information in the original image; The reference light source is determined according to the scene light source information.

6. The method according to claim 2, characterized in that The preset light source conversion relationship includes a preset color space conversion relationship; and performing chromatic adaptation processing on the original image according to the preset light source conversion relationship to obtain a chromatic adaptation image includes: According to the preset color space conversion relationship, the color channel values ​​of the original image are converted into the color space of the human eye color perception model to obtain the original image in the human eye perception color space; Based on the original image in the human eye perception color space, chromatic adaptation processing is performed to obtain a chromatic adaptation image.

7. The method according to claim 1, characterized in that There is a preset space conversion relationship between the color space of the color information and the color statistical space corresponding to the human eye color perception model, and the dimensions covered by the color statistical space include brightness, saturation and hue; The performing image processing on the original image according to the color information includes: According to the preset space conversion relationship, the color information is converted into the color statistical space to obtain the color information of the color statistical space; Color distribution is counted based on the color information in the color statistical space.

8. An image processing device, characterized in that: include: An acquisition module, used for acquiring an original image without image signal processing; A detection module, used to perform color detection based on the original image to obtain color information corresponding to the original image; The image processing module is used to perform image processing on the original image according to the color information.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images