Image processing method, image processing device, storage medium, and electronic device

Abstract

The present disclosure provides an image processing method, device, storage medium and electronic equipment, and relates to the technical field of computer vision. The image processing method comprises: acquiring a to-be-displayed image and spectral data of a plurality of detection regions collected by a plurality of spectral sensors; determining response data of each sub-region in XYZ color space according to the spectral data of each detection region and a human eye response function, determining a first conversion parameter of each sub-region according to the response data of each sub-region in XYZ color space and the response data of each sub-region in RGB color space; determining a color adaptation conversion parameter in an LMS color space based on brightness information and color temperature information of a display environment of the to-be-displayed image; determining a color transformation parameter of each sub-region according to the first conversion parameter, the color adaptation conversion parameter and a second conversion parameter; and performing color transformation processing on the to-be-displayed image by using the color transformation parameter. The present disclosure improves the image processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In related technologies, image parameters are typically adjusted manually on a computer to ensure that the image display on the terminal screen matches the human eye's perception. However, this method, which involves manually adjusting image parameters, is clearly labor-intensive and time-consuming. Furthermore, because the screen size and various display parameters of the debugging equipment and the image display device may differ, the processed image may not achieve the desired effect, resulting in low image processing efficiency. Summary of the Invention

[0003] This disclosure provides an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device, thereby improving, at least to some extent, the problem of low image processing efficiency.

[0004] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0005] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring an image to be displayed collected by an image sensor, and spectral data of multiple detection regions collected by multiple spectral sensors; each spectral sensor corresponds to a detection region, and each detection region corresponds to a sub-region in the image to be displayed; determining response data of each sub-region of the image to be displayed in an XYZ color space based on the spectral data of each detection region and the human eye response function; determining a first conversion parameter of each sub-region based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space; the first conversion parameter being a conversion parameter between the XYZ color space and the RGB color space; determining a color adaptation conversion parameter in an LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed; determining a color transformation parameter of each sub-region based on the first conversion parameter, the color adaptation conversion parameter, and a second conversion parameter; the second conversion parameter being a conversion parameter between the LMS color space and the XYZ color space; performing color transformation processing on the image to be displayed using the color transformation parameter, and the color-transformed image to be displayed is used for display in the display environment.

[0006] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: a data acquisition module configured to acquire an image to be displayed acquired by an image sensor, and spectral data of multiple detection regions acquired by multiple spectral sensors; each spectral sensor corresponds to a detection region, and each detection region corresponds to a sub-region in the image to be displayed; and a first conversion parameter determination module configured to determine response data of each sub-region of the image to be displayed in the XYZ color space based on the spectral data of each detection region and the human eye response function, and to determine a first conversion parameter of each sub-region based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space; the first conversion parameter is... The conversion parameters between XYZ and RGB color spaces; the color adaptation conversion parameter determination module, configured to determine the color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed; the color transformation parameter determination module, configured to determine the color transformation parameters of each sub-region based on the first conversion parameter, the color adaptation conversion parameter, and the second conversion parameter of each sub-region; the second conversion parameter is the conversion parameter between the LMS color space and the XYZ color space; the image color transformation processing module, configured to perform color transformation processing on the image to be displayed using the color transformation parameters, and the image to be displayed after color transformation processing is used for display in the display environment.

[0007] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the image processing method of the first aspect described above and its possible implementations.

[0008] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor. The processor is configured to perform the image processing method of the first aspect and its possible implementations thereof by executing the executable instructions.

[0009] The technical solution disclosed herein has the following beneficial effects:

[0010] On the one hand, by obtaining the conversion parameters between the XYZ and RGB color spaces for each sub-region of the image to be displayed based on its spectral data, the conversion parameters for each sub-region are flexibly adjusted, improving the accuracy of color conversion. On the other hand, by determining the color adaptation conversion parameters within the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed, the image can be color-adjusted according to the environment of the display device, improving the image adjustment quality and ensuring that the image display effect on the screen is consistent with that of the human eye, thus achieving the expected display effect. Furthermore, compared to manually adjusting image parameters, this solution saves manpower and time costs, enhances portability, and further improves image processing efficiency.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0013] Figure 1 A flowchart illustrating the processing procedure of an image processor is shown.

[0014] Figure 2 This illustrates the system architecture of the operating environment for this exemplary embodiment;

[0015] Figure 3 A flowchart illustrating an image processing method in this exemplary embodiment is shown.

[0016] Figure 4 This diagram illustrates a set of spectral spectrometers and a set of spectral sensors in this exemplary embodiment;

[0017] Figure 5 This diagram illustrates the arrangement of the spectral sensors in this exemplary embodiment.

[0018] Figure 6 This diagram illustrates the process of obtaining color adaptation conversion parameters in this exemplary embodiment.

[0019] Figure 7 This diagram illustrates the process of color transformation of the image to be displayed based on the color transformation parameters of each sub-region of the image to be displayed in this exemplary embodiment.

[0020] Figure 8 A flowchart illustrating an image processing method in this exemplary embodiment is shown;

[0021] Figure 9 This diagram illustrates the updating of the image processor according to the color transformation matrix in this exemplary embodiment;

[0022] Figure 10 This diagram illustrates the structure of an image processing apparatus according to this exemplary embodiment.

[0023] Figure 11 A schematic diagram of the structure of an electron in this exemplary embodiment is shown. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0025] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] In related technologies, the color of an object is usually determined based on what the human eye actually sees, such as Figure 1As shown, manual adjustment of display parameters such as Auto White Balance (AWB), Color Corrected Matrix (CCM), gamma correction, and 3D lookup table is used to ensure that the colors displayed on the screen match the colors actually seen by the human eye. However, achieving a "what you see is what you get" display effect requires a significant amount of time, resulting in low efficiency in the image processing process. Furthermore, the entire image processing process relies heavily on manual adjustments, making the method unreusable and resulting in poor portability.

[0027] In view of one or more of the above-mentioned problems, this disclosure provides an exemplary embodiment of an image processing method. The following describes the method in conjunction with... Figure 2 The system architecture of the operating environment for this exemplary embodiment will be described.

[0028] refer to Figure 2 As shown, the system architecture 200 may include a terminal device 210 and a server 220. The terminal device 210 may be an electronic device with photographic capabilities, such as a smartphone, tablet, or camera. The terminal device 210 may include an image sensor and multiple spectral sensors, which may be integrated into the camera module of the terminal device 210 for acquiring images to be displayed and spectral data. The server 220 generally refers to the backend system providing illumination detection-related services for images in this exemplary embodiment, such as a server implementing an illumination detection method. The server 220 may be a single server or a cluster of multiple servers; this disclosure does not limit this. The terminal device 210 and the server 220 can be connected via a wired or wireless communication link for data interaction.

[0029] In one embodiment, the terminal device 210 can first acquire the image to be displayed collected by an image sensor, and the spectral data of multiple detection areas collected by multiple spectral sensors, wherein each spectral sensor acquires the spectral data of a detection area, and each detection area corresponds to a sub-region in the image to be displayed; then, a first conversion parameter for each sub-region is determined based on the spectral data of each detection area and the human eye response function; a color adaptation conversion parameter is determined based on the brightness and color temperature information of the display environment of the image to be displayed; a color transformation parameter for each sub-region is determined based on the first conversion parameter, the color adaptation conversion parameter, and the second conversion parameter; finally, the color transformation parameter is used to perform color transformation processing on the image to be displayed.

[0030] In one embodiment, terminal device 210 acquires an image to be displayed collected by an image sensor and spectral data of multiple detection areas collected by multiple spectral sensors, and then sends the spectral data of the multiple detection areas to server 220. After receiving the spectral data of the multiple detection areas sent by terminal device 210, server 220 determines a first conversion parameter for each sub-region based on the spectral data of each detection area and the human eye response function; determines a color adaptation conversion parameter based on the brightness and color temperature information of the display environment of the image to be displayed; determines a color transformation parameter for each sub-region based on the first conversion parameter, the color adaptation conversion parameter, and the second conversion parameter; and finally performs color transformation processing on the image to be displayed using the color transformation parameters.

[0031] As can be seen from the above, the illumination detection method for images in this exemplary embodiment can be executed by the terminal device 210 or the server 220 described above.

[0032] The following is combined with Figure 3 The image processing methods are explained. Figure 3 An exemplary flow of an image processing method is shown, including the following steps S310 to S350:

[0033] Step S310: Acquire the image to be displayed collected by the image sensor, and the spectral data of multiple detection areas collected by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the image to be displayed;

[0034] Step S320: Determine the response data of each sub-region of the image to be displayed in the XYZ color space based on the spectral data of each detection region and the human eye response function; determine the first conversion parameter of each sub-region based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space; the first conversion parameter is the conversion parameter between the XYZ color space and the RGB color space.

[0035] Step S330: Based on the brightness and color temperature information of the display environment of the image to be displayed, determine the color adaptation conversion parameters in the LMS color space;

[0036] Step S340: Determine the color transformation parameters for each sub-region based on the first transformation parameter, color adaptation transformation parameter, and second transformation parameter for each sub-region; the second transformation parameter is the transformation parameter between the LMS color space and the XYZ color space.

[0037] Step S350: Perform color transformation processing on the image to be displayed using color transformation parameters. The image to be displayed after color transformation processing is used for display in the display environment.

[0038] Based on the above method, on the one hand, the conversion parameters between the XYZ color space and the RGB color space of each sub-region in the image to be displayed are obtained based on the spectral data of each sub-region. The conversion parameters are flexibly adjusted according to the spectral data of each sub-region, improving the accuracy of color conversion. On the other hand, the color adaptation conversion parameters in the LMS color space are determined based on the brightness and color temperature information of the display environment of the image to be displayed. This allows for color adjustment of the image according to the environment of the display device, improving the image adjustment quality and ensuring that the image display effect on the screen is consistent with that of the human eye, thus achieving the expected display effect. Furthermore, compared to manually adjusting image parameters, this image processing method saves manpower and time costs, enhances portability, and further improves image processing efficiency.

[0039] The following is about Figure 3 Each step in the process will be explained in detail.

[0040] refer to Figure 3 In step S310, the image to be displayed is acquired by the image sensor, and the spectral data of multiple detection areas are acquired by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the image to be displayed.

[0041] An image sensor is a sensor that converts light signals into electrical signals, achieving imaging by quantitatively characterizing the light signals. This disclosure does not limit the specific type of image sensor; it can be a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge-Coupled Device) sensor. In one embodiment, the image sensor is located in the outgoing light path of an image filter and is used to sense the light signal passing through the image filter to generate an image of the subject to be displayed. The subject refers to the scene, object, or person being photographed directly in front of the camera module. The image to be displayed is the image obtained by the camera module from the subject; it can be a raw image, such as a RAW image, or an RGB or YUV image processed by an ISP (Image Signal Processor).

[0042] Generally, an image sensor consists of a number of photosensitive elements arranged in an array, with each photosensitive element corresponding to one pixel in the image to be displayed. The number of photosensitive elements represents the resolution of the image sensor. For example, if the photosensitive elements are arranged in an H×W array, where H represents the number of rows and W represents the number of columns, then the resolution of the image sensor can be H×W, and the size of the generated image to be displayed is also H×W, where H represents the image height and W represents the image width. For example, H is 3000 and W is 4000.

[0043] In one embodiment, the image filter may include a Bayer filter; the Bayer filter is a filter formed by arranging RGB monochromatic filters in an array, which can be located in the incident light path of the image sensor, so that the image sensor can receive monochromatic light (i.e., light with three different spectral ranges of R, G, and B) that has passed through the Bayer filter. This disclosure does not limit the arrangement of the Bayer filters; a traditional Bayer array arrangement or a quad-bayer array arrangement can be used.

[0044] The image to be displayed can be a RAW image captured by the aforementioned camera-capable terminal device 210 and output via an image sensor, or it can be an image already stored in the terminal device 210. This disclosure does not impose any special limitation on the source of the image to be displayed. For example, the image to be displayed can be a photo stored in the mobile phone's photo album that needs to be processed.

[0045] A spectral sensor can be a sensor used to acquire spectral data of an image to be displayed. The spectral sensor can be located in the output optical path of a spectral splitter to sense the light signal passing through the spectral splitter and obtain spectral data. This disclosure does not limit the specific type of spectral sensor; it can be a CMOS or CCD sensor, and its type can be the same as or different from that of the image sensor.

[0046] The aforementioned spectral splitter is used to separate light of a specific wavelength band from the incident light. It can be located in the incident light path of the spectral sensor, enabling the spectral sensor to receive the light of the specific wavelength band after passing through the spectral splitter, thereby sensing spectral data. In contrast, image filters typically only provide red, green, and blue monochromatic light within the visible light range, while spectral splitters can provide a wider variety of different wavelength bands of light over a larger spectral range (e.g., 350–1000 nm, covering the ultraviolet to infrared bands). The number of light bands that can be split by a spectral splitter is called the number of channels of the spectral splitter or spectral sensor. A spectral splitter can be an optical device such as a filter or a pyramid. Taking a filter as an example, in one embodiment, each set of spectral splitters can include L filters with different peak wavelengths (or center wavelengths), such that the incident light is separated into L different wavelength bands after passing through the spectral splitter, and the number of channels of the spectral splitter is L. If L is 1, meaning the number of channels in the spectrometer is 1, then the spectrometer is a single-spectrum spectrometer; if L is a positive integer not less than 2, meaning the number of channels in the spectrometer is greater than or equal to 2, then the spectrometer is a multispectral spectrometer. For example, L can be 13.

[0047] In one implementation, the L filters in each group of spectral splitters can be arranged in a p×q array, where p represents the number of rows and q represents the number of columns, and L=p×q. Figure 4 A schematic diagram of a spectral splitter is shown, which has 3×4 channels. The spectral splitter includes 12 filters arranged in a 3×4 array, denoted as C1~C12, indicating that they are used to filter the light in channels C1~C12. The peak wavelength and full width at half maximum (FWHM) of the light in each channel can be found in Table 1, which covers 12 important wavelength bands in the range of 350~1000nm.

[0048] Table 1

[0049]

[0050] In one embodiment, each set of spectral sensors may include L photosensitive elements, each used to sense the light signal filtered by the corresponding L filters in the spectral spectrometer, to obtain response data for L channels, i.e., spectral data for L bands. If L is 1, i.e., the number of channels of the spectral sensor is 1, then the spectral sensor is a monospectral sensor; if L is a positive integer not less than 2, i.e., the number of channels of the spectral sensor is greater than or equal to 2, then the spectral sensor is a multispectral sensor.

[0051] In one implementation, the L photosensitive elements in each group of spectral sensors can be arranged in a p×q array, where p represents the number of rows and q represents the number of columns, and L = p×q. For example, see reference... Figure 4 As shown, the spectral sensor may include 3×4 photosensitive elements, denoted as Z1~Z12, which correspond one-to-one with the filters C1~C12 of the spectral spectrometer, and receive optical signals from 12 channels respectively to obtain response data from 12 channels.

[0052] It should be noted that the spectral spectrometer and the spectral sensor can be considered as a single unit, referred to as a spectral sensor.

[0053] Each set of spectral splitters corresponds to a set of spectral sensors and a detection area. The detection area refers to a localized region within the object being photographed. During the shooting process, the light reflected from each detection area enters the camera module of the terminal device 210, is split by its corresponding set of spectral splitters, and finally enters its corresponding set of spectral sensors. The spectral sensors sense the light signal reflected from the detection area after being split by the spectral splitters, thus obtaining the spectral data of that detection area. The spectral splitters and spectral sensors can be configured correspondingly; for example, their positions can be aligned one-to-one along the optical axis, so that each set of spectral sensors receives the light transmitted through its corresponding set of spectral splitters (in some cases, each set of spectral sensors may also receive a small amount of light transmitted from adjacent spectral splitters, which can be eliminated through algorithms).

[0054] Each detection region can correspond to a sub-region in the image to be displayed. For example, by using K sets of spectral sensors to detect the spectral data of K detection regions, these K detection regions are equivalent to dividing the shooting scene into K blocks. Similarly, when an image sensor captures an image of the same subject to be displayed, this image can also be divided into K sub-regions, with each detection region corresponding to one sub-region.

[0055] This disclosure uses a spectral sensor to acquire spectral data of each detection region of the image to be displayed, which is beneficial for more refined optimization of the image to be displayed. For example, different illumination conditions in different sub-regions of the image to be displayed can be characterized separately, and then different processing can be applied to different sub-regions.

[0056] The above-mentioned K can be regarded as the resolution of the spectral sensor. Each group of spectral sensors outputs spectral data of a detection area, which can be regarded as a pixel of the spectral data. K groups of spectral sensors can output spectral data with a pixel of K. That is to say, a pixel of the spectral data is equivalent to a sub-area of the image to be displayed. In this exemplary embodiment, K < H×W, that is, the resolution of the spectral sensor is lower than that of the image sensor. Among them, the image sensor is used for imaging, and a high-resolution image sensor can generate a high-definition image to be displayed; while the spectral sensor is used for detecting spectral data, and the detection of spectral data does not need to be refined to the level of image pixels. By detecting the spectral data of K sub-areas in the image to be displayed through K groups of spectral sensors, the differences between different parts in the image to be displayed are characterized, and this can already meet the usage requirements. Therefore, in this exemplary embodiment, by combining the characteristics of high-definition imaging by the image sensor and spectral information detection by the spectral sensor, both a high-definition image to be displayed and relatively rich and detailed spectral information can be obtained. Moreover, increasing the resolution of the spectral sensor will lead to an increase in the manufacturing cost of the camera module in the terminal device. Controlling the resolution of the spectral sensor to be lower than (generally much lower than) the resolution of the image sensor is beneficial to cost control.

[0057] In one embodiment, the above-mentioned K groups of spectral sensors can be arranged in an m×n array, where m represents the number of rows and n represents the number of columns, then K = m×n. It should be understood that the K groups of spectral beam splitters can also be arranged in an m×n array. Figure 5 A schematic diagram of the arrangement array of the spectral sensors is shown, where m = 6 and n = 8. That is, the camera module of the terminal device can include 48 groups of spectral beam splitters and 48 groups of spectral sensors. Each group of spectral sensors can further include 3×4 photosensitive elements to output response data of 12 channels.

[0058] In one embodiment, H / W = m / n, that is, the ratio of the image sensor to the m×n groups of spectral sensors is the same. For example, the resolution of the image sensor is 3000 ×4000, and 48 groups of spectral sensors are arranged in a 6×8 array, and the aspect ratios of both are the same. This makes it easier for the m×n detection areas to correspond to the sub-areas in the image to be displayed. For example, each group of spectral sensors corresponds to a detection area, and this detection area corresponds to a sub-area of 500 ×500 in the image to be displayed, and the 6×8 detection areas respectively correspond to 6×8 sub-areas in the image to be displayed.

[0059] In one implementation, when setting the spectral sensor, considering the need to detect illumination conditions, m and n can be set to be positive integers not less than 3. This allows the image to be displayed to be divided into at least 9 sub-regions. The spectral data of each sub-region is detected, thereby determining the illumination conditions of each sub-region, thus fully detecting the different illumination conditions of different parts of the photographed object.

[0060] In one implementation, when performing operations such as feature extraction on the image to be displayed, a 3×3 extraction template (i.e., 3×3 sub-regions) may be used. This facilitates filtering, averaging, edge finding, and other operations on the RGB or spectral data within the 3×3 range, which is beneficial for achieving refined illumination analysis. To facilitate the use of the 3×3 extraction template, the resolution of the spectral sensor can be set to 6×8, meaning the camera module of the terminal device can include 48 sets of spectral sensors arranged in a 6×8 array.

[0061] In one embodiment, each set of spectral spectrometers may include L filters with different peak wavelengths; each set of spectral sensors includes L photosensitive elements, each used to sense the light signal filtered by the corresponding L filters; the spectral data of each detection region includes response data from L channels. Wherein, L is a positive integer not less than 2, meaning the number of channels in the spectral spectrometer or spectral sensor is at least 2. Referring to the above... Figure 4 As shown, L filters can be arranged in a p×q array, and p×q photosensitive elements can also be arranged in a p×q array, where p represents the number of rows and q represents the number of columns.

[0062] In one implementation, considering that the light intensity of a general light source gradually decreases in a circular diffusion manner, p=q can be set, that is, each group of spectral sensors arranges its photosensitive elements in a square array to conform to the circular diffusion law of light intensity, which makes it easier to detect the spectral data of each detection area (or sub-region) more accurately.

[0063] Spectral data can be spectral data collected by a spectral sensor based on a corresponding sub-region of the image to be displayed. This disclosure does not specifically limit the content of the spectral data. For example, the spectral data can be spectral data obtained by a spectral sensor directly collecting data from a sub-region of the image to be displayed, or it can be spectral data obtained by subtracting the image sensor's response from the spectral data obtained by the spectral sensor directly collecting data from a sub-region of the image to be displayed.

[0064] In one embodiment, the spectral data may include monospectral data and multispectral data. When generating multispectral data, the spectral sensor may correspond to multiple detectors, each of which can detect the spectrum of different bands in the detection area. The spectrum of different bands can form the multispectral data of the detection area.

[0065] After obtaining the spectral data for each detection area, continue to refer to... Figure 3 In step S320, the response data of each sub-region of the image to be displayed in the XYZ color space can be determined based on the spectral data of each detection region and the human eye response function. Based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space, the first conversion parameter of each sub-region is determined. The first conversion parameter is the conversion parameter between the XYZ color space and the RGB color space.

[0066] The aforementioned human eye response function can be a function that simulates the visual response of the human eye to monochromatic light of different wavelengths. This disclosure does not specifically limit the content of the human eye response function. For example, the human eye response function can be an XYZ response in an XYZ color space, or a visual function.

[0067] The aforementioned XYZ color space can reflect the standard response of the human eye to monochromatic light of different wavelengths, and can demonstrate the spectral power distribution response of long, medium, and short photoreceptor cone cells in the retina to light. The response data of the aforementioned XYZ color space can be the response of a sub-region of the image to be displayed in the XYZ color space. This disclosure does not impose any special limitations on the method of obtaining the XYZ color space response data. For example, the response data of each sub-region of the image to be displayed in the XYZ color space can be directly obtained through an XYZ sensor. Alternatively, the response data of each sub-region in the XYZ color space can be obtained based on the spectral data of each detection region and the human eye response function.

[0068] In one implementation, spectral data of each sub-region of the image to be displayed can first be acquired using a spectral sensor. Then, the response of the image sensor in the spectral data of each sub-region is subtracted to obtain the spectral data of each sub-region of the image to be displayed. The spectral data is then convolved with the aforementioned human eye response function to obtain the response data of each sub-region of the image to be displayed in the XYZ color space.

[0069] The aforementioned RGB color space response data can be the response of the image to be displayed in the RGB color space output by the image sensor through capturing the light signal of the scene being photographed. This disclosure does not impose any special limitation on the method of obtaining the RGB color space response data of each sub-region of the image to be displayed. For example, the RGB color space response data of each sub-region of the image to be displayed can be obtained based on spectral data and the image sensor, or it can be obtained from the image to be displayed itself.

[0070] In one implementation, the response data of each sub-region in the RGB color space can be determined based on the spectral data of each detection region and the response function of the image sensor; or the RGB data of each sub-region can be used as the response data of each sub-region in the RGB color space.

[0071] The response function of the image sensor can be the energy response state of the photoelectric conversion device of the image sensor in the wavelength range of visible or invisible light. This disclosure does not impose any special limitation on the method of obtaining the response function of the image sensor. For example, the response function of the image sensor can be obtained by measuring the image sensor with a spectrometer.

[0072] In this exemplary embodiment, spectral data of each sub-region of the image to be displayed can be acquired first using a spectral sensor, and then the response of the image sensor in the spectral data of each sub-region can be subtracted to obtain the spectral data of each sub-region of the image to be displayed; the response data of each sub-region in the RGB color space can be obtained by convolving the response function of the image sensor with the spectral data of each sub-region of the image to be displayed.

[0073] In one implementation, the mapping results of a 24-color card in the RGB color space and the mapping results in the XYZ color space can be obtained under lighting conditions such as D65, D50, TL84, and A, respectively. Then, based on these mapping results, a Color Correction Matrix (CCM) is obtained for each of the aforementioned lighting conditions. The interpolation results of the CCM under the D65, D50, TL84, and A lighting conditions are used as the RGB color space response data for each sub-region of the image to be displayed. However, the XYZ color space response data obtained by converting the RGB color space response data obtained by this method differs significantly from what the human eye perceives.

[0074] In one implementation, after obtaining the response data of each sub-region of the image to be displayed in the XYZ color space and the response data of each sub-region in the RGB color space, the first conversion parameter of each sub-region can be determined based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space.

[0075] The first conversion parameter is a conversion parameter between the XYZ color space and the RGB color space. In one embodiment, the first conversion parameter includes a first forward conversion matrix and a first inverse conversion matrix. The first forward conversion matrix is ​​a conversion matrix from the RGB color space to the XYZ color space, and the first inverse conversion matrix is ​​a conversion matrix from the XYZ color space to the RGB color space. Since the first conversion parameter of this exemplary embodiment is obtained based on the spectral data of each detection area of ​​the image to be displayed, the accuracy of the first conversion parameter is higher, which can avoid the response drift phenomenon caused by using heating and low illumination to achieve color space conversion in another embodiment.

[0076] In another implementation, the response data of the RGB color space is converted to the response data of the XYZ color space by increasing the temperature (above 35 degrees Celsius) and maintaining low illumination (e.g., below 1 lux). However, this method usually causes the response function of the image sensor to drift, resulting in poor conversion effect.

[0077] Continue to refer to Figure 3 In step S330, based on the brightness and color temperature information of the display environment of the image to be displayed, the color adaptation conversion parameters in the LMS color space are determined.

[0078] The display environment of the image to be displayed can be the environment of the terminal device where the screen on which the image is ultimately displayed is located. For example, when taking a picture of the view outside a window using a mobile phone or other imaging device indoors, the indoor environment where the mobile phone is located is the display environment of the image to be displayed; when an image taken by mobile phone A is sent to mobile phone B, the environment where mobile phone B is located at that time is the display environment; when an image taken in the morning is saved on mobile phone A and viewed at night, the environment where mobile phone A is located at night is the display environment. The brightness and color temperature information of the display environment can be obtained by the photosensor in the terminal device. This disclosure does not impose any special limitations on the method of obtaining the brightness and color temperature information of the display environment.

[0079] The LMS color space is a color space represented by the response of the three cones of the human eye. It is named after the response (sensitivity) peaks of the three cones of the human eye at long, medium, and short wavelengths. The LMS color space is typically used when estimating the color of an image under different light sources during color adaptation.

[0080] Color adaptation can refer to adaptation to light sources, or it can refer to the human eye's ability to adapt to changes in the white point under different lighting sources or observation conditions. When lighting conditions change, the human visual system can automatically adjust the relative sensitivity of the three types of cone cells in the retina to maintain the color appearance of objects as much as possible. Because of color adaptation, the human eye can automatically balance the color rendering of different light sources under mixed lighting conditions. In other words, the human eye perceives the same object as having different colors under different color temperatures and brightness environments. For example, from an environment with a color temperature of 5000K to an environment with a color temperature of 3000K, the human eye will match the actual color of the current object to the other color based on the brightness and color temperature of the environment.

[0081] The color adaptation conversion parameter is a conversion parameter determined based on the color adaptation result of the image to be displayed in the LMS color space. This disclosure can realize the color adaptation result of the human eye through the color adaptation conversion parameter. In other words, this disclosure can restore the display result of the image to be displayed in the display environment to the result presented by the image to be displayed in the environment in which it was photographed through the color adaptation conversion parameter.

[0082] In one implementation, such as Figure 6 As shown, determining the color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed may include steps S610~S620:

[0083] Step S610: Determine the illumination information of each sub-region based on the spectral data of each detection region;

[0084] Step S620: Based on the illumination information of each sub-region, the brightness information of the display environment, and the color temperature information, determine the color adaptation conversion parameters of each sub-region in the LMS color space.

[0085] The illumination information of each sub-region can reflect the illumination conditions of the shooting environment of the image to be displayed. This disclosure does not make any special limitation on the specific content of the illumination information of each sub-region. For example, the illumination information of each sub-region may include the color temperature, color deviation value and other index values ​​that can represent the light source information of each sub-region of the image to be displayed.

[0086] In one embodiment, the illumination information of each sub-region can be the color temperature of the shooting environment of the image to be displayed. In this exemplary embodiment, the shooting environment can be the actual environment in which the scene in the image to be displayed is located. For example, when taking a picture of the scene outside the window through a window with a mobile phone or other imaging device indoors, the environment in which the scene outside the window is located is the shooting environment of the image to be displayed; if an image taken by mobile phone A is sent to mobile phone B, and mobile phone A and the image to be captured are in the same environment, then the environment in which mobile phone A is located at this time is the shooting environment; if an image taken in the morning is saved in mobile phone A and viewed in the evening, then the environment in which mobile phone A is located in the morning is the shooting environment.

[0087] In this exemplary embodiment, in step S610, the color temperature of each sub-region of the image to be displayed in the shooting environment can be determined based on the spectral data of each detection area; in step S620, the color adaptation conversion parameters of each sub-region in the LMS color space are determined based on the color temperature of each sub-region in the shooting environment, the brightness information and color temperature information of the display environment.

[0088] Furthermore, in one embodiment, the response data of each sub-region of the image to be displayed in the LMS color space can be obtained first; based on the correspondence between the color temperature of each sub-region of the image to be displayed in the shooting environment and the color temperature of the display environment of each sub-region, the target screen color temperature of the sub-region in the current display environment is determined; the color appearance model is updated based on the brightness information of the display environment of the sub-region and the target screen color temperature to obtain the target color appearance model; the color adaptation result of each sub-region in the LMS color space is obtained by combining the target color appearance model and the response data of each sub-region of the image to be displayed in the LMS color space; the color adaptation conversion parameters are determined according to the ratio of the color adaptation result of each sub-region of the image to be displayed in the LMS color space to the response data of each sub-region in the LMS color space; in one embodiment, the color adaptation conversion parameters include a color adaptation conversion matrix.

[0089] Since the color adaptation conversion parameters are obtained based on the color temperature of the image to be displayed in the shooting environment, the brightness information and color temperature information of the display environment, the "what you see is what you get" visual effect can be achieved directly in an end-to-end manner, so that the color of the object seen by the user on the screen is consistent with the color of the object actually seen by the user, thus improving the user's visual experience.

[0090] In one embodiment, the color adaptation conversion parameters for each sub-region can be determined based on the color temperature of the image to be displayed in the shooting environment, the color gamut of the screen displaying the image, and the color temperature of the display environment. In this exemplary embodiment, the color temperature of each sub-region of the image to be displayed in the shooting environment and the color temperature of the display environment can be fused to obtain a fused color temperature; this fused color temperature is used as the target color temperature of the sub-region, and the color coordinates corresponding to the target color temperature are calculated; then, the color coordinates of white are obtained based on the brightness information of the display environment in which the sub-region of the image to be displayed is located; and the color adaptation conversion parameters are determined based on the ratio of the color coordinates corresponding to the target color temperature to the color coordinates of white.

[0091] Continue to refer to Figure 3 After obtaining the first conversion parameter and color adaptation conversion parameter of each sub-region, in step S340, the color transformation parameter of each sub-region can be determined according to the first conversion parameter, color adaptation conversion parameter and second conversion parameter of each sub-region; the second conversion parameter is the conversion parameter between LMS color space and XYZ color space;

[0092] The color transformation parameters can be the basis for color transformation processing of each sub-region of the image to be displayed. In one embodiment, the color transformation parameters can include a color transformation matrix.

[0093] In one embodiment, the second conversion parameter includes a second forward conversion matrix and a second inverse conversion matrix, wherein the second forward conversion matrix is ​​a conversion matrix from the XYZ color space to the LMS color space, and the second inverse conversion matrix is ​​a conversion matrix from the LMS color space to the XYZ color space.

[0094] Furthermore, in one embodiment, the second forward transformation matrix and the second inverse transformation matrix can be obtained according to the color appearance model. For example, the second forward transformation matrix and the second inverse transformation matrix can be determined according to the color appearance model CIE CAM02 or CIE CAM16.

[0095] In one implementation, determining the color transformation parameters for each sub-region based on the first transformation parameter, the color adaptation transformation parameter, and the second transformation parameter for each sub-region may include the following steps:

[0096] Multiply the first positive transformation matrix, the second positive transformation matrix, the color adaptation transformation matrix, the second inverse transformation matrix, and the first inverse transformation matrix of any sub-region in sequence to obtain the color transformation matrix of that sub-region.

[0097] The aforementioned color transformation matrix can be dynamically generated based on spectral data and changes in the display and shooting environments of the image to be displayed, thereby improving the flexibility of the color transformation process of the image to be displayed. After obtaining the color transformation matrix, in one embodiment, the color transformation matrix can be applied to the white balance processing and color correction processing of the image to be displayed.

[0098] In this exemplary embodiment, the color transformation matrix can be decomposed into a target white balance matrix and a target color correction matrix; wherein, the target white balance matrix can be a diagonal matrix, and the sum of each row of the target color correction matrix can be 1.

[0099] Generally, the target white balance matrix can be a 3 In one implementation, a diagonal matrix of 3 can be used to multiply the response data of each sub-region of the image to be displayed in the RGB color space with the target white balance matrix to obtain the white balance result of each sub-region of the image to be displayed; and multiply the white balance result of each sub-region of the image to be displayed with the target color correction matrix to obtain the color transformation result of each sub-region of the image to be displayed.

[0100] Since the color transformation matrix can be dynamically generated based on spectral data and changes in the display and shooting environments of the image to be displayed, the target white balance matrix and target color correction matrix can be dynamically obtained based on the color transformation matrix. Using the target white balance matrix and target color correction matrix to perform color transformation processing on the image to be displayed, under mixed light source conditions, it is not necessary to manually adjust the color attribute parameters of the image to be displayed, so that the color adaptation effect of the human eye can be simulated, and the image to be displayed achieves the visual effect of "what you see is what you get".

[0101] In one implementation, after each sub-region of the image to be displayed is processed by the target white balance matrix and the target color correction matrix, the processing results of the 24-color card or SG color card by the target white balance matrix and the target color correction matrix can be obtained. Then, the visual error is obtained based on the result of visually viewing the color card and the processing result of the color card by the target white balance matrix and the target color correction matrix. Finally, a three-dimensional lookup table is used to compensate for the visual error in order to obtain a color transformation result that is more consistent with the visual effect.

[0102] In another implementation, when performing color transformation on an image under mixed light source conditions, different proportions of light sources can be used to apply biased processing to the image. For example, when the proportion of low color temperature in the image is relatively high, the color transformation of the image can be biased towards low color temperature.

[0103] Continue to refer to Figure 3After obtaining the color transformation parameters, in step S350, the color transformation parameters can be used to perform color transformation processing on the image to be displayed. The image to be displayed after color transformation processing is used for display in the display environment.

[0104] In one embodiment, the above-described color transformation processing of the image to be displayed using color transformation parameters may include the following steps:

[0105] The target white balance matrix and target color correction matrix are used to perform white balance processing and color correction processing on the image to be displayed.

[0106] In one implementation, such as Figure 7 As shown, the above-described color transformation processing of the image to be displayed using color transformation parameters may include the following steps S710~S730:

[0107] Step S710: Use the color transformation parameters of each sub-region as the color transformation parameters of the reference points within each sub-region.

[0108] Step S720: By interpolating the color transformation parameters of reference points in adjacent sub-regions, the color transformation parameters of each pixel in the image to be displayed are obtained.

[0109] Step S730: Perform color transformation processing on the image to be displayed using the color transformation parameters of each pixel.

[0110] In this exemplary embodiment, in step S710, the color transformation parameters of each sub-region are respectively used as the color transformation parameters of the reference points in each sub-region. Further, in one embodiment, the target white balance matrix and target color correction matrix of each sub-region can be used as the color transformation parameters of the center pixel of each sub-region. After obtaining the color transformation parameters of the center pixel of each sub-region, in step S720, the target white balance matrix and target color correction matrix of the center pixel in adjacent sub-regions can be interpolated to obtain the target white balance matrix and target color correction matrix of each pixel in the image to be displayed, so as to determine the color transformation parameters of each pixel in the image to be displayed. After obtaining the color transformation parameters of each pixel in the image to be displayed, in step S730, the color transformation parameters of each pixel can be used to perform color transformation processing on the image to be displayed to obtain the display effect of the image to be displayed in the above display environment.

[0111] In one embodiment, the target white balance matrix and target color correction matrix of each pixel of the image to be displayed can be determined based on the bilinear interpolation result of the target white balance matrix and target color correction matrix of the center pixel in adjacent sub-regions. This disclosure does not impose any special limitation on the specific calculation method of the above interpolation operation.

[0112] Since existing image signal processors (ISPs) may not be able to achieve a display effect that matches the visual effect, in one embodiment, the above-mentioned white balance processing and color correction processing of the display image using the target white balance matrix and the target color correction matrix may include the following steps:

[0113] The target white balance matrix replaces the regular white balance matrix in the image signal processor, and the target color correction matrix replaces the regular color correction matrix in the image signal processor. The image signal processor then performs white balance and color correction processing on the image to be displayed.

[0114] Based on the above method, the accuracy of image color transformation is improved, saving labor and time costs, thereby improving image processing efficiency and enhancing the user's visual experience.

[0115] In one embodiment, an exemplary flow of the image processing method of this disclosure is as follows: Figure 8 As shown, image processing can be performed on the image to be displayed according to steps S801~S815:

[0116] Step S801: Obtain the image to be displayed;

[0117] Step S802: The detection areas of multiple spectral sensors are mapped one-to-one to the sub-regions of the image to be displayed;

[0118] Step S803: The spectral sensor acquires spectral data of the corresponding detection area;

[0119] Step S804: Determine the response data of each sub-region in the XYZ color space based on the spectral data of each detection region and the human eye response function;

[0120] Step S805: Determine the response data of each sub-region in the RGB color space based on the spectral data of each detection region and the response function of the image sensor; or use the RGB data of each sub-region as the response data of each sub-region in the RGB color space.

[0121] Step S806: Determine the first forward transformation matrix and the first inverse transformation matrix of each sub-region based on the response data of each sub-region in the XYZ color space and the response data in the RGB color space;

[0122] Step S807: Determine the illumination information of each sub-region based on the spectral data of each detection region;

[0123] Step S808: Based on the illumination information of each sub-region, the brightness information and color temperature information of the display environment are used to determine the color adaptation transformation matrix of each sub-region in the LMS color space;

[0124] Step S809: Determine the conversion parameters of each sub-region between the XYZ color space and the LMS color space as the second forward conversion matrix and the second inverse conversion matrix;

[0125] Step S810: Multiply the first positive transformation matrix, the second positive transformation matrix, the color adaptation transformation matrix, the second inverse transformation matrix, and the first inverse transformation matrix of any sub-region in sequence to obtain the color transformation matrix of any sub-region.

[0126] Step S811: Use the color transformation matrix of each sub-region as the color transformation matrix of the reference point within each sub-region.

[0127] Step S812: Interpolate the color transformation matrix of the reference points in adjacent sub-regions to obtain the color transformation matrix of each pixel in the image to be displayed.

[0128] Step S813: Decompose the color transformation matrix into a target white balance matrix and a target color correction matrix;

[0129] Step S814: Replace the conventional white balance matrix and conventional color correction matrix in the image signal processor with the target white balance matrix and target color correction matrix respectively to obtain the image signal processor to be displayed;

[0130] Step S815: The image to be displayed is processed using the image signal processor.

[0131] In one embodiment, the image to be displayed can be an image captured by a terminal device and to be processed by an ISP. The spectral data of each detection area can be the spectral data obtained by subtracting the response function of the image sensor from the spectral data obtained by the spectral sensor. The image to be displayed can be processed by the image processing method disclosed herein.

[0132] In this exemplary embodiment, a distribution method of 6 can be used. 8. Multiple spectral sensors acquire spectral data RL corresponding to each detection region of the image to be displayed, where each detection region corresponds to a sub-region in the image to be displayed. R can be the target in the image to be displayed, L can be the illumination information of the shooting environment, and "." can be an integral symbol. The response function S of the image sensor module can be obtained by measuring the image sensor module with a spectrometer. Based on the convolution result of the spectral data RL of each detection region and the response function S of the image sensor, the response data of each sub-region in the RGB color space is obtained. The XYZ color space of each sub-region of the image to be displayed is then acquired. The human eye response function in the color space is used to convolve the image sensor's response function S with the human eye response function to obtain the response data of each sub-region in the XYZ color space. Based on the response data of each sub-region of the image to be displayed in the RGB color space and the response data of each sub-region in the XYZ color space, a first forward transformation matrix Mrgb2xyz and a first inverse transformation matrix Mxyz2rgb are obtained, where Mrgb2xyz is the transformation matrix from the RGB color space to the XYZ color space, and Mxyz2rgb is the transformation matrix from the XYZ color space to the RGB color space.

[0133] The transformation matrix from the XYZ color space to the LMS color space can be obtained based on the Mcam02 matrix of the CIE CAM02 color appearance model. Combining the Mcam02 matrix with the response data of each sub-region of the image to be displayed in the XYZ color space, the response data (lms) of each sub-region in the LMS color space is obtained. Then, the illumination information of each sub-region of the image to be displayed is determined based on the spectral data of each detection region. Based on the illumination information of each sub-region, the brightness and color temperature information of the display environment, and the CIE CAM02 color appearance model, the result (lmsc) of each sub-region after color adaptation is determined. The color adaptation transformation matrix (Madp) can be obtained based on the result of (lmsc / lms).

[0134] The transformation matrix from LMS color space to XYZ color space is obtained based on the inv matrix of the color appearance model CIE CAM02. The lmsc of each sub-region of the image to be displayed can be converted to the response result of each sub-region in the XYZ color space based on the inv matrix. The color transformation matrix MadpE is obtained by sequentially multiplying Mrgb2xyz, Mcam02, Madp, inv, and Mxyz2rgb.

[0135] MadpE=Mrgb2xyz Mcam02 Madp inv Mxyz2rgb;

[0136] The MadpE matrix is ​​decomposed into a target white balance (WB) matrix and a target color correction matrix (CCM), such as... Figure 9 As shown, the ISP can be updated according to the MadpE matrix to achieve dynamic WB+CCM image processing.

[0137] The dynamic WB+CCM result of each sub-region of the image to be displayed can be used as the image processing result of the center pixel of each sub-region; bilinear interpolation is performed on the dynamic WB+CCM results of the center pixels of two adjacent sub-regions to obtain the dynamic WB+CCM result of each pixel in the image to be displayed, thus obtaining the image processing result of the image to be displayed.

[0138] Exemplary embodiments of this disclosure also provide an image processing apparatus. For example... Figure 10 As shown, the image processing apparatus 1000 may include:

[0139] The data acquisition module 1010 is configured to acquire the image to be displayed collected by the image sensor, and the spectral data of multiple detection areas collected by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the image to be displayed;

[0140] The first conversion parameter determination module 1020 is configured to determine the response data of each sub-region of the image to be displayed in the XYZ color space based on the spectral data of each detection region and the human eye response function, and to determine the first conversion parameter of each sub-region based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space; the first conversion parameter is the conversion parameter between the XYZ color space and the RGB color space.

[0141] The color adaptation conversion parameter determination module 1030 is configured to determine the color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed.

[0142] The color transformation parameter determination module 1040 is configured to determine the color transformation parameters of each sub-region based on the first transformation parameter, the color adaptation transformation parameter, and the second transformation parameter of each sub-region; the second transformation parameter is the transformation parameter between the LMS color space and the XYZ color space;

[0143] The image color transformation processing module 1050 is configured to determine the color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed.

[0144] In one embodiment, determining the color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed may include:

[0145] The illumination information of each sub-region is determined based on the spectral data of each detection region;

[0146] Based on the illumination information, display environment brightness information, and color temperature information of each sub-region, the color adaptation conversion parameters of each sub-region in the LMS color space are determined.

[0147] In one embodiment, the image processing apparatus 1000 may further include an RGB color space response data acquisition module, configured to determine the response data of each sub-region in the RGB color space based on the spectral data of each detection region and the response function of the image sensor; or to use the RGB data of each sub-region as the response data of each sub-region in the RGB color space.

[0148] In one embodiment, the first conversion parameter includes a first forward conversion matrix and a first inverse conversion matrix, wherein the first forward conversion matrix is ​​a conversion matrix from RGB color space to XYZ color space, and the first inverse conversion matrix is ​​a conversion matrix from XYZ color space to RGB color space; the second conversion parameter includes a second forward conversion matrix and a second inverse conversion matrix, wherein the second forward conversion matrix is ​​a conversion matrix from XYZ color space to LMS color space, and the second inverse conversion matrix is ​​a conversion matrix from LMS color space to XYZ color space; the color adaptation conversion parameter includes a color adaptation conversion matrix; and the color transformation parameter includes a color transformation matrix.

[0149] The above-mentioned determination of the color transformation parameters for each sub-region based on the first transformation parameter, color adaptation transformation parameter, and second transformation parameter for each sub-region may include:

[0150] Multiply the first positive transformation matrix, the second positive transformation matrix, the color adaptation transformation matrix, the second inverse transformation matrix, and the first inverse transformation matrix of any sub-region in sequence to obtain the color transformation matrix of any sub-region.

[0151] In one embodiment, the image processing apparatus may further include a matrix decomposition module configured to decompose the color transformation matrix into a target white balance matrix and a target color correction matrix; (the target white balance matrix is ​​a diagonal matrix, and the sum of each row of the target color correction matrix is ​​1).

[0152] In one embodiment, the above-described color transformation processing of the image to be displayed using color transformation parameters may include:

[0153] The target white balance matrix and target color correction matrix are used to perform white balance processing and color correction processing on the image to be displayed.

[0154] In one embodiment, the above-described white balance processing and color correction processing of the image to be displayed using the target white balance matrix and the target color correction matrix may include:

[0155] The target white balance matrix replaces the regular white balance matrix in the image signal processor, and the target color correction matrix replaces the regular color correction matrix in the image signal processor. The image signal processor then performs white balance and color correction processing on the image to be displayed.

[0156] In one embodiment, the above-described color transformation processing of the image to be displayed using color transformation parameters may include:

[0157] The color transformation parameters of each sub-region are used as the color transformation parameters of the reference points within each sub-region.

[0158] By interpolating the color transformation parameters of reference points in adjacent sub-regions, the color transformation parameters of each pixel in the image to be displayed are obtained.

[0159] The color transformation parameters of each pixel are used to perform color transformation processing on the image to be displayed.

[0160] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation, and therefore will not be repeated here.

[0161] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In an alternative embodiment, the program product can be implemented as a portable compact disc read-only memory (CD-ROM) including program code and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0162] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0163] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0164] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0165] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0166] Exemplary embodiments of this disclosure also provide an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, such as program code. The processor executes the executable instructions to perform the methods of this exemplary embodiment.

[0167] The following is for reference. Figure 11The following description uses a mobile terminal as an example to illustrate an electronic device. It should be understood that... Figure 11 The electronic device 1100 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0168] refer to Figure 11 As shown, the electronic device 1100 may include: a processor 1101, a memory 1102, a mobile communication module 1104, a wireless communication module 1105, a display screen 1106, a camera module 1107, an audio module 1108, a power supply module 1109, and a sensor module 1110.

[0169] Processor 1101 may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a display processing unit (DPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. The image processing method in this exemplary embodiment can be executed by a CPU. In one embodiment, the image sensor in the camera module 1107 can acquire an image to be displayed and acquire spectral data of multiple detection areas based on multiple spectral sensors. Each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the image to be displayed. After receiving the spectral data of each detection area, the CPU determines the response data of each sub-region of the image to be displayed in the XYZ color space based on the spectral data of each detection area and the human eye response function. Based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space, a first conversion parameter of each sub-region is determined. Then, based on the brightness and color temperature information of the display environment of the image to be displayed, a color adaptation conversion parameter in the LMS color space is determined. Based on the first conversion parameter, the color adaptation conversion parameter, and the second conversion parameter of each sub-region, a color transformation parameter of each sub-region is determined. The second conversion parameter is the conversion parameter between the LMS color space and the XYZ color space. Finally, based on the brightness and color temperature information of the display environment of the image to be displayed, the color adaptation conversion parameter in the LMS color space can be determined.

[0170] The memory 1102 can be used to store computer executable program code, which includes instructions. The processor 1101 executes various functional applications and data processing of the electronic device 1100 by running the instructions stored in the memory 1102. The memory 1102 can also store application data and various intermediate data, such as images, videos, and the aforementioned spectral data.

[0171] The communication function of electronic device 1100 can be implemented through mobile communication module 1104, antenna 1, wireless communication module 1105, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1104 can provide 3G, 4G, 5G and other mobile communication solutions for electronic device 1100. Wireless communication module 1105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for electronic device 1100.

[0172] The display screen 1106 is used to implement display functions, such as displaying user interfaces, images, videos, etc. In one embodiment, the display screen 1106 can be used to display the display result after the image processing method of this embodiment has processed the image to be displayed. The camera module 1107 is used to implement shooting functions, such as capturing images, videos, etc. For example, the camera module 1107 may include an image sensor and the above-mentioned spectral sensor. The image sensor can be used to acquire the image to be displayed. Combining multiple spectral sensors can more accurately acquire the spectral data of the image to be displayed, improving the detection accuracy and efficiency of the illumination information of the image, thereby improving the image processing efficiency. The audio module 1108 is used to implement audio functions, such as playing audio and acquiring voice. The power module 1109 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring the battery status. The sensor module 1110 may include one or more sensors, which are used to acquire status assessments of various aspects of the electronic device 1100.

[0173] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0174] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0175] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. An image processing method, characterized in that, include: The system acquires an image to be displayed collected by an image sensor, and spectral data of multiple detection areas collected by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the image to be displayed; each spectral sensor includes L photosensitive elements, which are used to sense the light signals filtered by the corresponding L filters in the spectral spectrometer, and obtain spectral data of L bands. Based on the spectral data of each detection region and the human eye response function, the response data of each sub-region of the image to be displayed in the XYZ color space is determined. Based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space, the first conversion parameter of each sub-region is determined. The first conversion parameter is the conversion parameter between the XYZ color space and the RGB color space. Based on the brightness and color temperature information of the display environment of the image to be displayed, the color adaptation conversion parameters in the LMS color space are determined; the determination of the color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed includes: determining the illumination information of each sub-region based on the spectral data of each detection region; Based on the illumination information of each sub-region, the brightness information and color temperature information of the display environment, the color adaptation conversion parameters of each sub-region in the LMS color space are determined; The color transformation parameters of each sub-region are determined based on the first transformation parameter, the color adaptation transformation parameter, and the second transformation parameter of each sub-region; the second transformation parameter is the transformation parameter between the LMS color space and the XYZ color space. The color transformation parameters are used to perform color transformation processing on the image to be displayed, and the color-transformed image is used for display in the display environment.

2. The method according to claim 1, characterized in that, The method further includes: Based on the spectral data of each detection region and the response function of the image sensor, determine the response data of each sub-region in the RGB color space; or use the RGB data of each sub-region as the response data of each sub-region in the RGB color space.

3. The method according to claim 1, characterized in that, The first conversion parameters include a first forward conversion matrix and a first inverse conversion matrix, wherein the first forward conversion matrix is ​​a conversion matrix from RGB color space to XYZ color space, and the first inverse conversion matrix is ​​a conversion matrix from XYZ color space to RGB color space; the second conversion parameters include a second forward conversion matrix and a second inverse conversion matrix, wherein the second forward conversion matrix is ​​a conversion matrix from XYZ color space to LMS color space, and the second inverse conversion matrix is ​​a conversion matrix from LMS color space to XYZ color space; the color adaptation conversion parameters include a color adaptation conversion matrix; the color transformation parameters include a color transformation matrix. The step of determining the color transformation parameters of each sub-region based on the first transformation parameter, the color adaptation transformation parameter, and the second transformation parameter of each sub-region includes: Multiply the first positive transformation matrix, the second positive transformation matrix, the color adaptation transformation matrix, the second inverse transformation matrix, and the first inverse transformation matrix of any sub-region in sequence to obtain the color transformation matrix of any sub-region.

4. The method according to claim 3, characterized in that, The method further includes: The color transformation matrix is ​​decomposed into a target white balance matrix and a target color correction matrix; the target white balance matrix is ​​a diagonal matrix, and the sum of the elements in each row of the target color correction matrix is ​​1; The step of performing color transformation processing on the image to be displayed using the color transformation parameters includes: The target white balance matrix and the target color correction matrix are used to perform white balance processing and color correction processing on the image to be displayed.

5. The method according to claim 4, characterized in that, The step of performing white balance processing and color correction processing on the image to be displayed using the target white balance matrix and the target color correction matrix includes: The target white balance matrix replaces the conventional white balance matrix in the image signal processor, and the target color correction matrix replaces the conventional color correction matrix in the image signal processor. The image signal processor then performs white balance processing and color correction processing on the image to be displayed.

6. The method according to claim 1, characterized in that, The step of performing color transformation processing on the image to be displayed using the color transformation parameters includes: The color transformation parameters of each sub-region are respectively used as the color transformation parameters of the reference points within each sub-region; The color transformation parameters of each pixel in the image to be displayed are obtained by interpolating the color transformation parameters of reference points in adjacent sub-regions. The color transformation of the image to be displayed is performed using the color transformation parameters of each pixel.

7. An image processing apparatus, characterized in that, include: The data acquisition module is configured to acquire the image to be displayed collected by the image sensor, and the spectral data of multiple detection areas collected by multiple spectral sensors; each spectral sensor corresponds to a detection area, and each detection area corresponds to a sub-region in the image to be displayed; each spectral sensor includes L photosensitive elements, which are used to sense the light signals filtered by the corresponding L filters in the spectral spectrometer, and obtain spectral data of L bands. The first conversion parameter determination module is configured to determine the response data of each sub-region of the image to be displayed in the XYZ color space based on the spectral data of each detection region and the human eye response function, and to determine the first conversion parameter of each sub-region based on the response data of each sub-region in the XYZ color space and the response data of each sub-region in the RGB color space; the first conversion parameter is the conversion parameter between the XYZ color space and the RGB color space. The color adaptation conversion parameter determination module is configured to determine color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed; the determination of color adaptation conversion parameters in the LMS color space based on the brightness and color temperature information of the display environment of the image to be displayed is configured to: determine the illumination information of each sub-region based on the spectral data of each detection region; and determine the color adaptation conversion parameters of each sub-region in the LMS color space based on the illumination information of each sub-region and the brightness and color temperature information of the display environment. The color transformation parameter determination module is configured to determine the color transformation parameters of each sub-region based on the first transformation parameter, the color adaptation transformation parameter, and the second transformation parameter of each sub-region; the second transformation parameter is the transformation parameter between the LMS color space and the XYZ color space. The image color transformation processing module is configured to perform color transformation processing on the image to be displayed using the color transformation parameters, and the image to be displayed after color transformation processing is used for display in the display environment.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 6 by executing the executable instructions.

Images