An ISP color inverse processing method and device and a storage medium

Abstract

The application discloses an ISP color inverse processing method and device and a storage medium. The method segments an RGB value range according to a preset step length to obtain a plurality of gamut cubes. An ISP processing procedure of each gamut cube can be approximated as a linear procedure and corresponds to an inverse processing conversion matrix. For each pixel point in a to-be-processed image, a gamut cube to which the pixel point belongs is determined, and then an inverse processing conversion matrix corresponding to the gamut cube is determined. The pixel point is converted into an ISP color inverse processed pixel value according to the inverse processing conversion matrix. All pixel points in the to-be-processed image are converted to obtain ISP color inverse processed image data. The scheme uses segmented linearization processing to fit the nonlinear characteristics of the ISP processing procedure, and the restoration degree is more accurate. Moreover, the corresponding inverse processing conversion matrix can be directly found by using the mapping relationship between the gamut cube and the inverse processing conversion matrix, the data processing efficiency is improved, and the real-time performance of image data processing is ensured.

Description

Technical Field

[0001] This application relates to the field of camera simulation testing technology, and in particular to an ISP color inverse processing method, apparatus and storage medium. Background Technology

[0002] Intelligent agents such as robots and cars are equipped with numerous sensors, among which cameras are crucial. As camera resolution and frame rates increase, integrating the entire Image Signal Processor (ISP) onto the camera's internal chip can lead to overheating, poor image quality, and the appearance of noise and dead pixels. Therefore, to address this, the complex ISP is often separated from the camera and integrated into the controller or a separate chip. Figure 1 As shown in (1), the ISP is an independent chip, and the system-on-chip (SoC) chip is set on the electronic control unit (ECU) control board. The camera module is connected to the ISP on the ECU board. Figure 1 As shown in (2), the ISP is integrated inside the SoC chip, the SoC chip is set on the ECU board, and the camera module is connected to the ISP inside the SoC chip.

[0003] In controller hardware-in-loop (HIL) testing, the camera simulation link scenario is as follows: Figure 2 As shown, the data source (video or image) is input to the controller under test through the simulation board. There are two types of data sources: one is video or image data rendered by scene simulation software, and the other is video or image data of the real scene captured by a camera. Both types of data sources are processed by the ISP. However, in Figure 1 In the simulation architecture shown, the data source, after being processed by the ISP (Image Signal Processor), is input to the controller under test (ECU) and then processed again by the ECU's ISP. This results in the image data received by the ECU's processing algorithm undergoing two ISP processes, leading to color deviations in the image content. Consequently, the controller test results are inaccurate and fail to meet testing requirements. Therefore, in this simulation scenario, the data source needs to undergo an inverse ISP color processing step before being input to the ECU. However, due to insufficient information and support from the ISP, only a near-inverse processing restoration can be achieved through technical means to meet the controller testing requirements.

[0004] Current ISP color inverse processing methods employ a model-based approach, requiring the acquisition of a large number of images to build a database, and then training a neural network to obtain an ISP color inverse processing model. However, this method requires capturing a large amount of diverse image data, resulting in high costs. Furthermore, for applications requiring real-time rendering in scene simulation software, the data source needs to be generated in real-time, and the data processed in real-time before being input to the controller under test, which may affect the real-time performance of the simulation test. Summary of the Invention

[0005] In view of the above problems, this application provides an ISP color inverse processing method, apparatus, and storage medium to solve at least some of the aforementioned problems. The specific solution is as follows:

[0006] The first aspect of this application provides an ISP color inverse processing method, including:

[0007] The RGB data range is segmented according to a preset step size to obtain multiple color gamut cubes;

[0008] For any pixel in the image to be processed, determine the target color gamut cube to which the pixel belongs based on the RGB value of the pixel.

[0009] Based on the inverse processing transformation matrix corresponding to each color gamut cube obtained in advance, the target inverse processing transformation matrix corresponding to the target color gamut cube is determined, and one inverse processing transformation matrix corresponds to one color gamut cube;

[0010] Based on the target inverse processing transformation matrix, the RGB values ​​corresponding to the pixels are converted into pixel values ​​after ISP color inverse processing.

[0011] In one possible implementation, the process of obtaining the inverse processing transformation matrix corresponding to each color gamut cube includes:

[0012] For any of the color gamut cubes, sampling pixels corresponding to the RGB value range of the color gamut cube are selected from the RGB data corresponding to the first calibration image to obtain a first image data matrix. The first calibration image is a RAW image containing the calibration color.

[0013] The RGB data corresponding to each sampled pixel is obtained from the RGB data corresponding to the second calibration image to obtain the second image data matrix. The second calibration image is the image obtained by ISP processing of the first calibration image.

[0014] Based on the first image data matrix and the second image data matrix, the inverse processing transformation matrix corresponding to the color gamut cube is obtained.

[0015] In one possible implementation, obtaining the inverse processing transformation matrix corresponding to the color gamut cube based on the first image data matrix and the second image data matrix includes:

[0016] The linear equation for transforming the first image data matrix into the second image data matrix is ​​constructed as follows:

[0017]

[0018] Where O represents the second image data matrix, and I represents the first image data matrix. This represents the ISP processing conversion matrix;

[0019] Solve the linear equation to obtain the ISP processing transformation matrix, and calculate the inverse matrix of the ISP processing transformation matrix to obtain the inverse processing transformation matrix corresponding to the color gamut cube.

[0020] In one possible implementation, solving the linear equation to obtain the ISP processing transformation matrix includes:

[0021] The ISP processing transformation matrix is ​​determined according to the following formula:

[0022]

[0023] in, Describe the transpose of matrix I. express The inverse matrix.

[0024] In one possible implementation, the first calibration image is an artificially calibrated color patch RAW image, which includes multiple standard color patches distributed in the central region of the entire image;

[0025] The second calibration image is an RGB image obtained by inputting the first calibration image into the ISP for processing.

[0026] In one possible implementation, the step of segmenting the RGB data range according to a preset step size to obtain multiple color gamut cubes includes:

[0027] For each color channel in the RGB channels of the image, the value range of each color channel is divided into m regions according to a preset step size;

[0028] By selecting any region of each color channel and combining them, a color gamut cube is obtained. Combining the entire RGB value range yields m×m×m color gamut cubes.

[0029] In one possible implementation, determining the target color gamut cube to which the pixel belongs based on the RGB value corresponding to the pixel includes:

[0030] The first region of the pixel in the R channel is determined based on the R channel value corresponding to the pixel.

[0031] The second region of the pixel in the G channel is determined based on the G channel value corresponding to the pixel.

[0032] The third region of the pixel in the B channel is determined based on the B channel value corresponding to the pixel.

[0033] The target color gamut cube of the pixel is determined based on the first region of the R channel, the second region of the G channel, and the third region of the B channel.

[0034] In one possible implementation, the target inverse processing transformation matrix corresponding to the target color gamut cube is determined based on the pre-obtained inverse processing transformation matrix corresponding to each color gamut cube, including:

[0035] The inverse processing transformation matrix corresponding to the first number of the target color gamut cube is obtained by querying the mapping table. Each color gamut cube corresponds to a unique number, which ranges from [1, k], where k is the total number of color gamut cubes.

[0036] A second aspect of this application provides an ISP color inverse processing apparatus, comprising:

[0037] The segmentation module is used to segment the RGB data range according to a preset step size to obtain multiple color gamut cubes;

[0038] The first determining module is used to determine the target color gamut cube to which any pixel in the image to be processed belongs based on the RGB value corresponding to the pixel.

[0039] The second determining module is used to determine the target inverse processing transformation matrix corresponding to the target color gamut cube based on the inverse processing transformation matrix corresponding to each color gamut cube obtained in advance, wherein one inverse processing transformation matrix corresponds to one color gamut cube;

[0040] The inverse processing module is used to convert the RGB value corresponding to the pixel point into the pixel value after ISP color inverse processing based on the target inverse processing transformation matrix.

[0041] A third aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the ISP color inverse processing method described in the first aspect or any implementation thereof.

[0042] The fourth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the ISP color inverse processing method described in the first aspect or any implementation thereof.

[0043] The ISP color inverse processing method provided in this application divides the RGB value range into multiple regions according to a preset step size, and combines different regions of each color channel to obtain multiple color gamut cubes. The ISP processing of each color gamut cube can be approximated as a linear process, solving for the inverse processing transformation matrix corresponding to each color gamut cube. Then, for each pixel in the image to be processed, the color gamut cube to which the pixel belongs is first determined, and then the pixel value of the pixel is converted to the ISP color inverse processing pixel value according to the inverse processing transformation corresponding to the color gamut cube, until the pixel values ​​of all pixels in the image to be processed are converted to the ISP color inverse processing pixel values, obtaining the ISP color inverse processing image data corresponding to the image. The above-mentioned ISP color inverse processing uses piecewise linearization to fit the nonlinear characteristics of the ISP processing process, which has a more accurate restoration degree compared to a single linear fitting method. Compared to complex polynomial fitting methods, this scheme directly uses the mapping relationship between the color gamut cube and the inverse processing transformation matrix to find the corresponding inverse processing transformation matrix, improving data processing efficiency and ensuring the real-time performance of image data processing. Attached Figure Description

[0044] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0045] Figure 1 A schematic diagram of the architecture of a controller provided in this application;

[0046] Figure 2 A schematic diagram of a hardware-in-the-loop test architecture for a controller provided in this application;

[0047] Figure 3 A flowchart of an ISP color inverse processing method provided in this application;

[0048] Figure 4 A schematic diagram of a color gamut cube provided in this application;

[0049] Figure 5 A flowchart of the process for obtaining the inverse transformation matrix provided in this application;

[0050] Figure 6A schematic diagram of an artificially calibrated color patch image provided in this application;

[0051] Figure 7 A schematic diagram of an ISP color inverse processing device provided in this application;

[0052] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0053] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] To better understand the embodiments of this application, a brief introduction to the technical terms involved in this application will be given first:

[0057] Hardware-in-the-loop (HIL) testing uses a real-time processor to run a simulation model to simulate the operating state of the controlled object and connects to the controller under test (DUT) through an I / O interface to perform comprehensive and systematic testing of the DUT.

[0058] The application scenario of this application is an intelligent driving vision control system or a robot vision control system, therefore the controlled object is the camera. Furthermore, a camera simulation board can be used to simulate a camera. This camera simulation board is a simulation testing tool used to receive video data transmitted from a PC, process the video data, and then inject the processed video data into the controller under test.

[0059] Please see Figure 3The diagram illustrates a flowchart of an ISP color inverse processing method provided in an embodiment of this application. This method can be applied to the processor of an electronic device, wherein the processor can be a CPU or a graphics processing unit (GPU), etc. Figure 3 As shown, the method may include the following steps:

[0060] S101 divides the RGB data range into segments according to a preset step size to obtain multiple color gamut cubes.

[0061] The Image Signal Processor (ISP) encompasses numerous and complex functions. For example, color-related functions include white balance, color correction, and gamma correction (used to adjust image contrast). Therefore, the ISP color mathematical model is not a simple linear equation but requires polynomial fitting. Using polynomial fitting complicates the ISP color inverse processing mathematical model, increasing the computational load in actual camera simulations and compromising the real-time performance of hardware-in-the-loop testing. To address this issue, this application's embodiments employ piecewise linearization (PWL) of the nonlinear equations. PWL refers to dividing the RGB (red, green, blue) color channel values ​​of the image into segments, approximating each segment as a linearized ISP color processing step.

[0062] In one exemplary embodiment, for each color channel in the RGB channels of an image, the numerical range of each color channel is divided into m regions according to a preset step size; then, any region of each color channel is selected and combined to obtain a color gamut cube, and the entire numerical range of RGB can be obtained into m×m×m color gamut cubes. For example, as Figure 4 As shown in the figure, each cube represents a color gamut cube obtained by combining any region selected from each of the RGB three channels. Figure 4 The colors are used only to illustrate linear piecewise processing from a mathematical perspective and do not have any symbolic meaning.

[0063] Taking an 8-bit deep camera as an example, the value range of each color channel is [0, 255]. Each color channel is divided into a grid according to a preset step size. For example, if the preset step size is 64, each color channel can be divided into 4 regions: [0, 63], [64, 127], [128, 191], and [192, 255]. Combining any region from each of the three color channels yields a color gamut cube, resulting in a total of 4 × 4 × 4 = 64 color gamut cubes. Each color gamut cube can be numbered (e.g., 1 to 64).

[0064] For example, if the preset step size is 32, each color channel can be divided into 8 regions. The three color channels can be combined to obtain 8×8×8=512 color gamut cubes. For example, each color gamut cube can be numbered, such as 1~512.

[0065] During testing or debugging, the preset step size can be adjusted according to the actual testing requirements. For example, the higher the test precision requirement (i.e., the more stringent the requirement), the smaller the step size and the more color gamut cubes are required; conversely, the lower the test precision requirement (i.e., the more lenient the requirement), the larger the step size and the fewer color gamut cubes are required. In addition, different manufacturers' cameras have different CMOS characteristics or ISPs, and their sensitivity to different colors (R, G, B) varies, so the preset step size also varies. This application does not impose any restrictions on this.

[0066] S102, for any pixel in the image to be processed, determine the target color gamut cube to which the pixel belongs based on the RGB value corresponding to the pixel.

[0067] In intelligent driving vision systems and robot vision systems, the image to be processed refers to the video frames (i.e., images) contained in the video captured by the camera; in other application scenarios, the image to be processed can be a static image captured by the camera.

[0068] Furthermore, the image to be processed in this application refers to an image after ISP processing, with colors that match human vision. Therefore, each pixel in the image to be processed includes components of the three color channels: RGB.

[0069] In an exemplary embodiment, for each pixel of the image to be processed, the region of the pixel in the R channel is determined according to the R channel value corresponding to the pixel. Similarly, the regions of the pixel in the G channel and B channel are determined respectively. The color gamut cube to which the pixel belongs is determined according to the regions of the three RGB channels.

[0070] For example, the RGB values ​​corresponding to pixel P are (0, 32, 32) in sequence. Taking the division of each color channel into 4 regions and the three channels into a total of 64 color gamut cubes as an example, pixel P belongs to the color gamut cube composed of R[0~63], G[0~63], and B[0~63].

[0071] In an exemplary embodiment, a number can be created for each color gamut cube, and the number corresponding to each color gamut cube and the regions of each RGB color channel included in that color gamut cube can be recorded. For example, taking the case where each color channel can be divided into 4 segments, color gamut cube 1 consists of R[0~63], G[0~63], and B[0~63], color gamut cube 2 consists of R[0~63], G[64,127], and B[0~63], color gamut cube 3 consists of R[0~63], G[0~63], and B[64,127], and so on. The mapping relationship between the numbers of all color gamut cubes and the region combinations of the three RGB channels can be recorded. For example, this mapping relationship can be stored in the form of a table.

[0072] S103, based on the inverse processing transformation matrix corresponding to each color gamut cube obtained in advance, determine the target inverse processing transformation matrix corresponding to the target color gamut cube, with one inverse processing transformation matrix corresponding to one color gamut cube.

[0073] After dividing the color gamut into cubes, the ISP color inverse processing process for each cube can be approximated as a linear equation. Each color gamut cube corresponds to an inverse processing transformation matrix, which, when multiplied by the pixel value in the ISP-processed image, yields the ISP color inverse processing pixel value for that pixel.

[0074] The inverse processing transformation matrix corresponds one-to-one with the color gamut cube, that is, each color gamut cube corresponds to one inverse processing transformation matrix, and each inverse processing transformation matrix corresponds to one color gamut cube.

[0075] For example, the inverse processing transformation matrix corresponding to each color gamut cube in the first calibration color patch image (the RAW image output by the image sensor, i.e., without ISP processing) and its corresponding ISP-processed second calibration color patch image can be used to solve for the matrix. For instance, the inverse processing transformation matrix can be solved using the formula O = I·M, where O represents the image data matrix of the second calibration color patch image, I represents the image data matrix of the first calibration color patch image, and M represents the ISP processing transformation matrix corresponding to the current color gamut cube. After solving for M, the inverse matrix M is then calculated. -1 That is, the inverse processing transformation matrix corresponding to the current color gamut cube.

[0076] In one exemplary embodiment, after solving for the inverse processing transformation matrix corresponding to each color gamut cube, the results can be stored in a table. For example, the table stores the color gamut cube number i and the inverse processing transformation matrix M. i , where i∈[1,k], and k is the total number of color gamut cubes.

[0077] After determining the target color gamut cube for the current pixel, the target inverse processing transformation matrix corresponding to the target color gamut cube can be further determined based on the mapping relationship between the color gamut cube and the inverse processing transformation matrix. For example, if it is determined in S102 that the current pixel belongs to the target color gamut cube 1, i.e., i=1, then the corresponding inverse processing transformation matrix M1 can be obtained by looking up the transformation matrix table.

[0078] S104, based on the target inverse processing transformation matrix, converts the RGB values ​​corresponding to the pixels into the pixel values ​​after ISP color inverse processing.

[0079] The pixel value after ISP color inverse processing is obtained by multiplying the RGB value corresponding to the pixel with the target inverse processing transformation matrix corresponding to the pixel.

[0080] After obtaining the pixel value of each pixel in the image to be processed after ISP color inverse processing, the ISP color inverse processed image data corresponding to the image is obtained.

[0081] Furthermore, the ISP color inverse processing image data can be injected into the controller under test (DUT) to complete the DUT test.

[0082] The ISP color inverse processing method provided in this embodiment divides the RGB value range into multiple regions according to a preset step size. Different regions of each color channel are combined to obtain multiple color gamut cubes. The ISP processing of each color gamut cube can be approximated as a linear process. An inverse processing transformation matrix is ​​set for each color gamut cube and solved to obtain the inverse processing transformation matrix corresponding to each color gamut cube. Then, for each pixel in the image to be processed, the color gamut cube to which the pixel belongs is first determined. Then, according to the inverse processing transformation corresponding to the color gamut cube, the pixel value of the pixel is converted into the pixel value after ISP color inverse processing. This process continues until the pixel values ​​of all pixels in the image to be processed are converted into pixel values ​​after ISP color inverse processing, thus obtaining the ISP color inverse processing image data corresponding to the image. Furthermore, the converted ISP color inverse processing image data can be injected into the controller under test for testing. The above-described ISP color inverse processing uses piecewise linearization to fit the nonlinear characteristics of the ISP processing process, which provides more accurate restoration compared to a single linear fitting method. Compared to complex polynomial fitting methods, this scheme directly uses the mapping relationship between the color gamut cube and the inverse processing transformation matrix to find the corresponding inverse processing transformation matrix, which improves data processing efficiency and ensures the real-time performance of image data processing.

[0083] Please see Figure 5 The diagram illustrates a flowchart of the process for obtaining the inverse transformation matrix corresponding to the color gamut cube, as provided in an embodiment of this application. Figure 5As shown, this process may include the following steps:

[0084] S201, for any color gamut cube, select sampling pixels from the RGB data corresponding to the RGB value range of the color gamut cube to obtain the first image data matrix.

[0085] The first calibration image is a RAW image containing calibration color patches. A RAW image refers to the raw photoelectric signal captured by the image sensor, which has undergone preliminary processing but has not yet undergone deep processing by the ISP (such as depixelation, noise reduction, sharpening, gamma compression, etc.).

[0086] In one exemplary embodiment, the first calibration image can be an artificially generated calibration color patch image, such as one generated by image generation software according to actual needs and stored in an electronic device. For example, the first calibration image can be stored in a camera simulation board, and the processor running the ISP color inverse processing method provided in this application can read the image data of the first calibration image from the camera simulation board. Alternatively, the first calibration image can be stored in the memory of the electronic device running the ISP color inverse processing method. This application does not limit the storage location of the first calibration image, as long as it satisfies the requirement that the electronic device running the ISP color inverse processing method can read the first calibration image from that storage location.

[0087] The artificially calibrated color patch image must include RGB standard colors. Furthermore, to avoid interference from processing modules such as lens shading correction (LSC) within the ISP, such as... Figure 6 As shown, standard color patches are distributed in the central 1 / 3 area of ​​the image. Furthermore, the preset step size for dividing the color gamut cube is determined based on the requirement that a color gamut cube must include at least 3 artificially calibrated color patches. In other words, any color gamut cube obtained by dividing the entire color gamut of the artificially calibrated color patch image includes at least 3 standard color patches.

[0088] For example, if the current color gamut cube is R[0~63], G[0~63], B[0~63], in order to calculate the inverse processing transformation matrix corresponding to this color gamut, pixels whose RGB values ​​are respectively within the numerical range of the color gamut cube can be selected from the first calibration image. The RGB values ​​of the selected pixels form the first image data matrix. For example, the following four pixels are selected: P1 rgb (10,10,10), P2 rgb (0,32,32), P3 rgb (0,63,32), P4 rgb (0,0,32), the RGB values ​​of these four pixels are used to construct the first image data matrix.

[0089] S202, obtain the RGB data corresponding to each sampled pixel from the RGB data corresponding to the second calibration image to obtain the second image data matrix.

[0090] The second calibration image is obtained by processing the first calibration image through an ISP. For example, the first calibration image can be injected into a controller containing an ISP and processed by the ISP to obtain the second calibration image.

[0091] In one exemplary embodiment, the first calibration image is an artificial calibration color patch image, the resolution of which is known, according to... Figure 6 A coordinate system is created as shown, with the image width direction as the x-axis, running from left to right, and the image height direction as the y-axis, running from top to bottom. The top left corner of the image is the origin. This makes it easy to determine the position of the standard color patch in the coordinate system. Furthermore, based on the coordinate position of this color patch, it is easy to determine the corresponding pixel value of the color patch after ISP processing in the second calibration image.

[0092] For example, the coordinate positions of selected sampling pixels in the first calibration image are determined, and the pixel values ​​corresponding to each coordinate position are determined in the second calibration image. The pixel values ​​corresponding to these coordinate positions in the second calibration image form a second image data matrix. Specifically, the position of the first pixel value (the pixel value at that coordinate position in the first calibration image) in the first image data matrix is ​​the same as the position of the second pixel value (the pixel value at that coordinate position in the second calibration image) in the second image matrix.

[0093] S203, based on the first image data matrix and the second image data matrix, obtain the inverse processing transformation matrix corresponding to the color gamut cube.

[0094] The linear equation for constructing the first image matrix to the second image matrix is ​​as follows:

[0095] (1)

[0096] Wherein, matrix O represents the second image data matrix, and matrix I represents the first image data matrix. The resulting matrix is ​​the transformation matrix, denoted as . , is used to characterize the ISP color processing process.

[0097] Among them, (I) r1 I g2 I b3 () represents the pixel value of pixel P1 in the first calibration image, where the coordinate position of pixel P1 is z1. (O) r1 O g2 O b3() represents the pixel value corresponding to the pixel at coordinate position z1 in the second calibration image. That is, (O r1 O g2 O b3 ) is (I r1 I g2 I b3 The pixel values ​​obtained after ISP processing.

[0098] Furthermore, Formula 1 can be simplified as follows:

[0099] (2)

[0100] For example, the transformation matrix can be solved using the least squares method. During the calculation process, the RGB values ​​in matrix O and matrix I are normalized. The formula is as follows:

[0101] (3)

[0102] in, Describe the transpose of matrix I. express The inverse matrix.

[0103] Further calculation inverse matrix This is determined to be the inverse processing transformation matrix corresponding to the color gamut cube. For example, it can be calculated using Gaussian elimination, the adjoint matrix method, or the block matrix method. .

[0104] It should be noted that the inverse processing transformation matrix is ​​not a complete ISP inverse process, but rather the best approximate solution obtained by reversing the ISP's color processing as much as possible.

[0105] After calculating the inverse transformation matrices for all color gamut cubes, a set of inverse transformation matrices is obtained. Each inverse transformation matrix corresponds to a color gamut cube, where the subscript of M indicates the number of the color gamut cube corresponding to that matrix. For example, Let represent the inverse processing transformation matrix corresponding to color gamut cube 1, where k represents the total number of color gamut cubes.

[0106] In an exemplary embodiment, the inverse processing transformation matrices corresponding to all color gamut cubes can be stored in the form of a table, which includes M i And the corresponding matrix, where i represents the number of the color gamut cube. After determining the number of the color gamut cube to which the pixel belongs, the inverse processing transformation matrix corresponding to that color gamut cube can be obtained by looking up this table.

[0107] The inverse processing transformation matrix method provided in this embodiment uses a RAW image of artificially calibrated color patches as the first calibration image, thus saving on RAW image acquisition equipment. Furthermore, the first calibration image is directly input to the controller under test (DUT), and the ISP in the DUT processes the input image to obtain the ISP-processed image, i.e., the second calibration image. Moreover, the standard color patches contained in the first calibration image do not affect the color restoration of the color patches (i.e., the ISP color inverse processing) to RGB during the ISP's conversion of the RAW image to an RGB image.

[0108] The above describes the ISP color inverse processing method embodiments provided in this application. The apparatus for performing the above method will be described below.

[0109] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an ISP color inverse processing device provided in an embodiment of this application. Figure 7 As shown, the device includes:

[0110] The segmentation module 101 is used to segment the RGB data range according to a preset step size to obtain multiple color gamut cubes.

[0111] The first determining module 102 is used to determine the target color gamut cube to which any pixel belongs based on the RGB value of the pixel in the image to be processed.

[0112] The second determining module 103 is used to determine the target inverse processing transformation matrix corresponding to the target color gamut cube based on the inverse processing transformation matrix corresponding to each color gamut cube obtained in advance, wherein one inverse processing transformation matrix corresponds to one color gamut cube.

[0113] The inverse processing module 104 is used to convert the RGB values ​​corresponding to the pixels into the pixel values ​​after ISP color inverse processing based on the target inverse processing transformation matrix.

[0114] In one possible implementation, the segmentation module 101 is specifically used to: for each color channel in the RGB channels of the image, divide the value range of each color channel into m regions according to a preset step size; select any region of each color channel to combine to obtain a color gamut cube, and the entire RGB value range can be obtained into m×m×m color gamut cubes.

[0115] In one possible implementation, the first determining module 102 is specifically used to: determine the first region of the pixel in the R channel based on the R channel value corresponding to the pixel; determine the second region of the pixel in the G channel based on the G channel value corresponding to the pixel; determine the third region of the pixel in the B channel based on the B channel value corresponding to the pixel; and determine the target color gamut cube of the pixel based on the first region of the R channel, the second region of the G channel, and the third region of the B channel.

[0116] In one possible implementation, the second determining module 103 is specifically used to: query the mapping relationship table to obtain the inverse processing transformation matrix corresponding to the first number of the target color gamut cube; wherein each color gamut cube corresponds to a unique number, the number range is [1, k], and k is the total number of color gamut cubes.

[0117] In one possible implementation, the process of obtaining the inverse transformation matrix corresponding to each color gamut cube may include:

[0118] For any color gamut cube, sampled pixels corresponding to the RGB value range of the color gamut cube are selected from the RGB data corresponding to the first calibration image to obtain the first image data matrix. The first calibration image is a RAW image containing the calibration colors.

[0119] The RGB data corresponding to each sampled pixel is obtained from the RGB data corresponding to the second calibration image to obtain the second image data matrix. The second calibration image is the image obtained after processing the first calibration image by an ISP. In an exemplary embodiment, the first calibration image is an artificial calibration color patch RAW image, which includes multiple standard color patches distributed in the central area of ​​the entire image. The second calibration image is the RGB image obtained after inputting the first calibration image into an ISP for processing.

[0120] Based on the first image data matrix and the second image data matrix, the inverse processing transformation matrix corresponding to the color gamut cube is obtained. In an exemplary embodiment, the inverse processing transformation matrix corresponding to the color gamut cube is obtained in the following manner:

[0121] The linear equation for transforming the first image data matrix into the second image data matrix is ​​constructed as follows: Where O represents the second image data matrix and I represents the first image data matrix. This represents the ISP processing transformation matrix;

[0122] Solving the linear equation yields the ISP processing transformation matrix, and calculating the inverse of the ISP processing transformation matrix gives the inverse processing transformation matrix corresponding to the color gamut cube. The ISP processing transformation matrix can be solved using the following formula: ,in, Describe the transpose of matrix I. express The inverse matrix.

[0123] The ISP color inverse processing device provided in this embodiment utilizes piecewise linearization to fit the nonlinear characteristics of the ISP processing process, achieving more accurate color reproduction compared to a single linear fitting method. Compared to complex polynomial fitting methods, this scheme directly uses the mapping relationship between the color gamut cube and the inverse processing transformation matrix to find the corresponding inverse processing transformation matrix, improving data processing efficiency and ensuring real-time image data processing.

[0124] This application also provides an electronic device in its embodiments. (See reference...) Figure 8 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as laptops, PDAs (Personal Digital Assistants), PADs (Tablet PCs), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0125] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit (CPU), a graphics processing unit (GPU), etc.) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage device 208 into a random access memory (RAM) 203. In this application, the CPU or GPU can execute any of the above-described ISP color inverse processing method embodiments according to the read program code.

[0126] When the electronic device is powered on, RAM 203 also stores various programs and data required for the operation of the electronic device. Processing device 201, ROM 202, and RAM 203 are interconnected via bus 204. Input / output (I / O) interface 205 is also connected to bus 204.

[0127] Typically, the following devices can be connected to I / O interface 205: input devices 206 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 207 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 208 including, for example, memory cards, hard drives, etc.; and communication devices 209. Communication device 209 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0128] Furthermore, this application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the ISP color inverse processing methods provided in this application.

[0129] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the ISP color inverse processing methods provided in this application.

[0130] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0133] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An ISP color inverse processing method, characterized in that, include: The RGB data range is segmented according to a preset step size to obtain multiple color gamut cubes; For any pixel in the image to be processed, the target color gamut cube to which the pixel belongs is determined based on the RGB value corresponding to the pixel. Based on the inverse processing transformation matrix corresponding to each color gamut cube obtained in advance, the target inverse processing transformation matrix corresponding to the target color gamut cube is determined, and one inverse processing transformation matrix corresponds to one color gamut cube; Based on the target inverse processing transformation matrix, the RGB values ​​corresponding to the pixels are converted into pixel values ​​after ISP color inverse processing.

2. The method according to claim 1, characterized in that, The process of obtaining the inverse transformation matrix corresponding to each color gamut cube includes: For any of the color gamut cubes, sampling pixels corresponding to the RGB value range of the color gamut cube are selected from the RGB data corresponding to the first calibration image to obtain a first image data matrix. The first calibration image is a RAW image containing the calibration color. The RGB data corresponding to each sampled pixel is obtained from the RGB data corresponding to the second calibration image to obtain the second image data matrix. The second calibration image is the image obtained by ISP processing of the first calibration image. Based on the first image data matrix and the second image data matrix, the inverse processing transformation matrix corresponding to the color gamut cube is obtained.

3. The method according to claim 2, characterized in that, Based on the first image data matrix and the second image data matrix, the inverse processing transformation matrix corresponding to the color gamut cube is obtained, including: The linear equation for transforming the first image data matrix into the second image data matrix is ​​constructed as follows: Where O represents the second image data matrix, and I represents the first image data matrix. This represents the ISP processing conversion matrix; Solve the linear equation to obtain the ISP processing transformation matrix, and calculate the inverse matrix of the ISP processing transformation matrix to obtain the inverse processing transformation matrix corresponding to the color gamut cube.

4. The method according to claim 3, characterized in that, Solving the linear equation to obtain the ISP processing transformation matrix includes: The ISP processing transformation matrix is ​​determined according to the following formula: in, Describe the transpose of matrix I. express The inverse matrix.

5. The method according to any one of claims 2-4, characterized in that, The first calibration image is an artificially calibrated color patch RAW image, which includes multiple standard color patches distributed in the central area of ​​the entire image; The second calibration image is an RGB image obtained by inputting the first calibration image into the ISP for processing.

6. The method according to any one of claims 1-4, characterized in that, The step of dividing the RGB data range into multiple color gamut cubes according to a preset step size includes: For each color channel in the RGB channels of the image, the value range of each color channel is divided into m regions according to a preset step size; By selecting any region of each color channel and combining them, a color gamut cube is obtained. Combining the entire RGB value range yields m×m×m color gamut cubes.

7. The method according to any one of claims 1-4, characterized in that, Determining the target color gamut cube to which the pixel belongs based on the RGB value corresponding to the pixel includes: The first region of the pixel in the R channel is determined based on the R channel value corresponding to the pixel. The second region of the pixel in the G channel is determined based on the G channel value corresponding to the pixel. The third region of the pixel in the B channel is determined based on the B channel value corresponding to the pixel. The target color gamut cube of the pixel is determined based on the first region of the R channel, the second region of the G channel, and the third region of the B channel.

8. The method according to any one of claims 1-4, characterized in that, Based on the pre-obtained inverse processing transformation matrix corresponding to each color gamut cube, determine the target inverse processing transformation matrix corresponding to the target color gamut cube, including: The inverse processing transformation matrix corresponding to the first number of the target color gamut cube is obtained by querying the mapping table. Each color gamut cube corresponds to a unique number, which ranges from [1, k], where k is the total number of color gamut cubes.

9. An ISP color inverse processing device, characterized in that, include: The segmentation module is used to segment the RGB data range according to a preset step size to obtain multiple color gamut cubes; The first determining module is used to determine the target color gamut cube to which any pixel in the image to be processed belongs based on the RGB value corresponding to the pixel. The second determining module is used to determine the target inverse processing transformation matrix corresponding to the target color gamut cube based on the inverse processing transformation matrix corresponding to each color gamut cube obtained in advance, wherein one inverse processing transformation matrix corresponds to one color gamut cube; The inverse processing module is used to convert the RGB value corresponding to the pixel point into the pixel value after ISP color inverse processing based on the target inverse processing transformation matrix.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the ISP color inverse processing method as described in any one of claims 1 to 8.

Images