Color gamut mapping and coding collaborative optimization method for R scene

Abstract

The invention relates to the technical field of video coding, in particular to a color gamut mapping and coding collaborative optimization method for an R scene, which comprises the following steps of: constructing a dynamic color gamut perception model, and generating a local color gamut mapping function adaptive to a current scene; executing preliminary color gamut mapping; extracting chromaticity gradient, saturation change rate and spatial frequency characteristics to construct a chromaticity distortion sensitivity graph; embedding the graph into an encoder rate distortion optimization module, and dynamically adjusting quantization parameters and conversion unit division; performing compression by adopting a standard coding framework; and iteratively optimizing the color gamut mapping function through a coding distortion feedback channel. According to the method, the color fidelity and the compression efficiency are remarkably improved, the average CIEDE2000 color difference is reduced by 32% under the same code rate, and multi-terminal heterogeneous display adaptation is supported.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and in particular to a method for color gamut mapping and coding co-optimization in R scenes. Background Technology

[0002] With the continuous evolution of extended reality technology, immersive visual experiences are becoming increasingly important in scenarios such as consumer electronics, remote collaboration, and industrial simulation. Among them, R scenarios (such as augmented reality (AR) and mixed reality (MR)) pose a dual challenge to high-fidelity color reproduction and efficient visual encoding. Color gamut mapping, as a key link connecting content creation and terminal display, needs to preserve the original color semantics as much as possible under limited display capabilities; while video or image encoding needs to maintain visual quality under bandwidth-constrained conditions. In traditional systems, these two are usually processed independently, lacking a collaborative optimization mechanism.

[0003] Existing color gamut mapping methods mostly employ static lookup tables or global compression strategies based on perceptual uniformity, which are difficult to adapt to dynamic lighting, viewing angle changes, and heterogeneous display characteristics of multiple devices in R scenes, and are prone to color distortion or loss of detail. Summary of the Invention

[0004] The purpose of this invention is to provide a method for co-optimizing color gamut mapping and encoding in R scenarios, so as to solve the technical problem that the existing color gamut mapping and video encoding processes are disconnected, making it difficult to balance color fidelity and compression efficiency in R scenarios.

[0005] This invention provides a method for co-optimization of color gamut mapping and encoding in R scenes, including: Construct a dynamic color gamut perception model for R scenarios, and generate a local color gamut mapping function that adapts to the current scenario in real time based on the spatial content features of the input image or video frame, lighting conditions and the color gamut boundary of the target display device. Perform preliminary color gamut mapping, transforming the original wide color gamut content to the color gamut range reproducible by the target display device according to the local color gamut mapping function, and generating mapped image data; Before encoding, the mapped image data is analyzed to extract the chromaticity gradient distribution, saturation change rate, and spatial frequency characteristics of the region adjacent to the color gamut boundary, and a chromaticity distortion sensitivity map is constructed. The chroma distortion sensitivity map is embedded into the rate distortion optimization module of the encoder, and the quantization parameters and transform block partitioning strategy are dynamically adjusted so that encoding resources are preferentially allocated to the chroma distortion sensitive area. Perform video or image encoding using a standard encoding framework. Based on the adjusted quantization parameters and block partitioning strategy, compress and encode the mapped image data to generate the final bitstream. A coding distortion feedback channel is established, the decoded reconstructed frame is used to calculate the chromaticity error distribution between the frame and the mapped image, and the error information is fed back to the dynamic color gamut perception model for iterative optimization of the local color gamut mapping function of subsequent frames.

[0006] In some embodiments, the dynamic color gamut perception model employs a region adaptive strategy based on content semantic segmentation to divide the image into high-saturation foreground regions, low-light background regions, and dynamic transition regions, and independently calculates the color gamut compression ratio and hue offset for each type of region.

[0007] In some embodiments, the color gamut boundary of the target display device is represented by a polygon vertex set in the CIE 1931 chromaticity coordinate system, with no less than 8 vertices, covering red, green, blue, cyan, magenta, yellow and white points, and the color gamut volume calculation accuracy is no less than 0.1%.

[0008] In some embodiments, the local color gamut mapping function employs a piecewise nonlinear compression strategy, performing progressive compression within the 0.95 to 1.0 times saturation range within the color gamut boundary. The compression slope decreases linearly with saturation, ensuring smooth color transitions in high-saturation regions.

[0009] In some embodiments, the chromaticity gradient distribution is calculated by using the a and b channels in the CIELAB color space to calculate the gradient magnitude in the horizontal and vertical directions, respectively. The gradient threshold is set to 3.5, and pixels exceeding this threshold are marked as chromaticity edge points. The saturation change rate is defined as the absolute value of the saturation difference between adjacent pixels divided by the spatial distance. When the change rate is greater than 0.15, it is determined to be a high-sensitivity region.

[0010] In some embodiments, the rate-distortion optimization module introduces a chromaticity weighting factor that is proportional to the local mean of the chromaticity distortion sensitivity map, with a scaling factor ranging from 0.8 to 1.2, to adjust the chromaticity distortion weight in the Lagrange multiplier.

[0011] In some embodiments, the transform block partitioning strategy forces the use of 4×4 minimum transform units in chroma distortion-sensitive regions, while allowing the use of 32×32 maximum transform units in low-sensitivity regions, in order to balance compression efficiency and chroma fidelity.

[0012] In some embodiments, the coding framework supports intra-frame prediction and inter-frame prediction modes, wherein the motion vector accuracy of inter-frame prediction is improved to 1 / 16 pixel, and independent motion compensation is enabled in the chroma component, with compensation accuracy consistent with that of the luminance component.

[0013] In some embodiments, the encoding distortion feedback channel employs a low-latency decoder, performing full-frame reconstruction only on key frames or scene switching frames, and macroblock-level local reconstruction on the remaining frames, with reconstruction latency controlled within 2 frames.

[0014] In some embodiments, the chromaticity error distribution is calculated using the CIEDE2000 chromaticity difference formula, with an error threshold set to 2.3. Regions exceeding this threshold are compensated with an additional saturation of 0.05 in the local color gamut mapping function of the next frame.

[0015] Compared with the prior art, the present invention has the following beneficial effects: By constructing a dynamic color gamut perception model and a coding distortion feedback mechanism, a closed-loop collaborative optimization of color gamut mapping and video coding is achieved, effectively solving the problem that color fidelity and compression efficiency are difficult to balance due to the separation of the two in the traditional process. In the R scenario, the present invention can adaptively adjust the mapping strategy according to the content semantics, lighting conditions and terminal characteristics, and guide the allocation of encoding resources through the chroma distortion sensitivity map, which can significantly suppress the color block and color deviation phenomenon at low bit rate. The coding distortion feedback mechanism allows the color gamut mapping function to be dynamically corrected as the reconstruction quality improves, further enhancing long-term visual consistency. Experiments show that, at the same bitrate, the present invention reduces the average CIEDE2000 color difference by 32% and improves the subjective visual score by 1.8 points compared to the independent processing scheme, with the maximum subjective visual score being 5 points. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a technical architecture diagram of the optimization method of the present invention; Figure 2 This is a schematic diagram illustrating the principle framework of the dynamic color gamut perception model and the collaborative optimization of encoding distortion feedback in this invention. Detailed Implementation

[0018] The following will be based on embodiments of the present invention. Figures 1-2 The technical solutions in the embodiments of the present invention will be clearly and completely described together. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0019] Example This embodiment provides a method for co-optimizing color gamut mapping and encoding in R scenes, specifically: First, a dynamic color gamut perception model for R scenarios is constructed. Based on the spatial content features of the input image or video frame, lighting conditions, and the color gamut boundary of the target display device, a local color gamut mapping function adapted to the current scene is generated in real time.

[0020] Understandably, the dynamic color gamut perception model employs a region-adaptive strategy based on content semantic segmentation, dividing the image into high-saturation foreground regions, low-light background regions, and dynamic transition regions, and independently calculating the color gamut compression ratio and hue shift for each type of region. This model first performs semantic segmentation on the input image, using a lightweight convolutional neural network structure to classify pixels. The network has no more than 12 layers, fewer than 1.5 MB of parameters, and an inference speed exceeding 60 frames per second on mobile GPUs, meeting the real-time processing requirements of R scenarios.

[0021] The segmentation results divide the image into three types of regions: high-saturation foreground regions typically contain interactive objects that users are interested in, such as virtual buttons or 3D models; low-light background regions refer to areas with weak ambient light and less color information; and dynamic transition regions are located at the boundary between the foreground and background, where color and brightness change drastically.

[0022] Specifically, for high-saturation foreground regions, the model calculates a higher color gamut compression ratio to preserve the integrity of color semantics; for low-light background regions, a lower compression ratio is used to save encoding resources; and for dynamic transition regions, a smooth transition compression strategy is adopted to avoid color banding.

[0023] The color gamut boundary of the target display device is represented by a polygon vertex set in the CIE 1931 chromaticity coordinate system, with no fewer than 8 vertices covering red, green, blue, cyan, magenta, yellow, and white points, and a color gamut volume calculation accuracy of no less than 0.1%. This polygon vertex set serves as one of the model's input parameters, ensuring that the generated local color gamut mapping function is strictly limited to the reproducible range of the target device. The model's initial parameters are obtained through offline training, with a training dataset containing no fewer than 50,000 R scene samples, covering indoor, outdoor, low-light, high dynamic range, and multi-device display combinations, ensuring the model has good generalization ability across various R scenarios.

[0024] Next, preliminary color gamut mapping is performed, which transforms the original wide color gamut content to the color gamut range that the target display device can reproduce according to the local color gamut mapping function, generating mapped image data.

[0025] Understandably, the local color gamut mapping function employs a piecewise nonlinear compression strategy, implementing progressive compression within the 0.95 to 1.0 times saturation range at the color gamut boundary. The compression slope decreases linearly with saturation, ensuring smooth color transitions in high-saturation regions.

[0026] This mapping process is performed in the CIELAB color space. Specifically, the input image is converted from its original color space (such as RGB or XYZ) to the CIELAB space, where L represents luminance and a and b represent chrominance. For each pixel, its saturation S = sqrt(a² + b²) is calculated. If S is less than or equal to 0.95 times the maximum saturation S_max, the pixel is within the safe color gamut and does not require compression. If S is greater than 0.95 times S_max but less than or equal to S_max, it enters the progressive compression range. The compressed saturation S' is calculated using the formula S' = S_max - k(S - 0.95S_max)², where k is the compression coefficient, which decreases linearly with S to ensure that the compression intensity gradually increases near the color gamut boundary, thus avoiding abrupt color changes. The mapped a' and b... The component is scaled along its original direction, i.e., a ' = a (S' / S), b ' = b (S' / S).

[0027] After mapping all pixels, the data is converted back to the target device's color space to generate mapped image data. The aforementioned process ensures that color details in highly saturated areas are preserved while effectively preventing color distortion caused by color gamut overflow.

[0028] Next, the mapped image data is analyzed before encoding to extract the chromaticity gradient distribution, saturation change rate, and spatial frequency characteristics of the region adjacent to the color gamut boundary, and a chromaticity distortion sensitivity map is constructed.

[0029] Understandably, the chromaticity gradient distribution calculates the horizontal and vertical gradient magnitudes using the a and b channels in the CIELAB color space, respectively. The gradient threshold is set to 3.5, and pixels exceeding this threshold are marked as chromaticity edge points. The calculation process uses the Sobel or Prewitt operator to perform convolution operations on the a and b channels, respectively, to obtain the horizontal gradients Gx_a and Gx_b and the vertical gradients Gy_a and Gy_b. Then, the gradient magnitude of each channel is calculated as G_a = sqrt(Gx_a² + Gy_a²) and G_b = sqrt(Gx_b² + Gy_b²), and the final chromaticity gradient magnitude is G_c = max(G_a, G_b).

[0030] It should be noted that pixels with G_c greater than 3.5 are marked as chroma edge points, and these areas are extremely sensitive to chroma distortion. At the same time, the saturation change rate is defined as the absolute value of the saturation difference between adjacent pixels divided by the spatial distance. When the change rate is greater than 0.15, it is judged as a highly sensitive area, and higher precision needs to be retained in the encoding.

[0031] The saturation change rate is calculated in a local neighborhood of 3×3 or 5×5. Each pixel is traversed with its 8-neighbor or 24-neighbor pixels. The absolute value of the saturation difference of all adjacent pairs is calculated and divided by the Euclidean distance (usually 1 or sqrt(2)). The maximum value is taken as the saturation change rate of the pixel.

[0032] Furthermore, it is also necessary to identify the regions near the color gamut boundary, that is, the pixel regions where the saturation S' after mapping is greater than 0.9 times S_max. These regions are prone to color block artifacts during the encoding and quantization process. Combining the above three types of information—chroma edge points, high saturation change rate regions, and regions near the color gamut boundary—the final chroma distortion sensitivity map is generated through weighted fusion.

[0033] Then, the chroma distortion sensitivity map is embedded into the rate distortion optimization module of the encoder, and the quantization parameters and transform block partitioning strategy are dynamically adjusted so that the encoding resources are preferentially allocated to the chroma distortion sensitive area.

[0034] Understandably, the rate-distortion optimization module introduces a chroma weighting factor, which is proportional to the local mean of the chroma distortion sensitivity map, with a scaling factor ranging from 0.8 to 1.2. This factor is used to adjust the chroma distortion weight in the Lagrange multipliers. In the standard rate-distortion optimization (RDO) process, the encoder selects the optimal coding mode by minimizing the Lagrange cost function J = D + λR, where D is the distortion, R is the code rate, and λ is the Lagrange multiplier.

[0035] This invention decomposes the total distortion D into luminance distortion D_luma and chrominance distortion D_chroma, i.e., D = D_luma + wD_chroma, where w is a chrominance weighting factor. The calculation of w is based on the local mean μ of the chrominance distortion sensitivity map within the current coding unit (CU), i.e., w = α μ and α are scaling factors, which are dynamically adjusted between 0.8 and 1.2. They can be adaptively selected according to the bitrate budget or content complexity. It should be noted that when μ is high, w increases, which means that the weight of chroma distortion in the total cost increases. The encoder will tend to choose a lower quantization parameter (QP) to reduce chroma distortion.

[0036] Meanwhile, the transform block partitioning strategy forces the use of 4×4 minimum transform units in chroma distortion-sensitive regions, while allowing the use of 32×32 maximum transform units in low-sensitivity regions, to balance compression efficiency and chroma fidelity. When making CU partitioning decisions, the encoder checks the average sensitivity of the current CU in the chroma distortion sensitivity map. If the average sensitivity exceeds a preset threshold (e.g., 0.7), the chroma component transform units (TUs) are forced to be partitioned into 4×4 sizes, even if this increases the bit rate overhead; conversely, if the average sensitivity is below another threshold (e.g., 0.3), 32×32 TUs are allowed to maximize compression efficiency.

[0037] Next, video or image encoding is performed using a standard encoding framework. Based on the adjusted quantization parameters and block partitioning strategy, the mapped image data is compressed and encoded to generate the final bitstream.

[0038] Understandably, the coding framework supports intra-frame prediction and inter-frame prediction modes. The motion vector accuracy for inter-frame prediction is improved to 1 / 16 pixel, and independent motion compensation is enabled in the chroma component, with compensation accuracy consistent with the luma component. The coding process follows standard frameworks such as H.265 / HEVC or AV1, but enhancements have been made to chroma processing. In the inter-frame prediction stage, the motion estimation module not only calculates high-precision motion vectors (MV) for the luma component but also independently calculates MVs for the chroma component.

[0039] Traditional coding standards typically set the chroma MV to half of the luminance MV, which can easily cause blurring at color boundaries. This invention improves the accuracy of the chroma MV to 1 / 16 of a pixel by using independent chroma motion estimation, making motion compensation of the chroma components more accurate and effectively reducing color ghosting.

[0040] During the transform and quantization stages, the encoder strictly follows a defined QP and TU partitioning strategy. For highly sensitive regions, a lower QP value (e.g., QP_chroma = QP_luma - 2) and a 4×4 TU are used to preserve more high-frequency chroma details; for low-sensitive regions, a higher QP value (e.g., QP_chroma = QP_luma + 1) and a larger TU are used to save bit rate.

[0041] Finally, an encoding distortion feedback channel is established, the decoded reconstructed frame is calculated and its chromaticity error distribution with the mapped image is calculated, and the error information is fed back to the dynamic color gamut perception model for iterative optimization of the local color gamut mapping function of subsequent frames.

[0042] Understandably, the encoding distortion feedback channel employs a low-latency decoder, performing full-frame reconstruction only for keyframes or scene-switching frames, while macroblock-level local reconstruction is used for the remaining frames, with reconstruction latency controlled within two frames. Furthermore, a lightweight decoder module is integrated within the encoder for real-time reconstruction of encoded frames. To control computational overhead and latency, the system employs a selective reconstruction strategy: for I-frames (keyframes) or P-frames where scene switching is detected, full-frame reconstruction is performed; for other P-frames, only macroblocks marked as highly sensitive in the chroma distortion sensitivity map are locally reconstructed.

[0043] The reconstructed image is compared with the generated mapped image. The chromaticity error distribution is calculated using the CIEDE2000 color difference formula, with an error threshold set at 2.3. Regions exceeding this threshold are compensated with an additional saturation of 0.05 in the local color gamut mapping function of the next frame. The CIEDE2000 formula comprehensively considers differences in brightness, chromaticity, and hue, and can more accurately reflect the color difference perceived by the human eye.

[0044] The calculated color difference map is sent back to the dynamic color gamut perception model. Based on the error distribution, the dynamic color gamut perception model fine-tunes the local color gamut mapping function of the next frame: in areas where the color difference exceeds 2.3, the dynamic color gamut perception model will slightly increase the saturation compensation by 0.05 to offset the chromaticity attenuation introduced by the encoding process, thereby achieving more accurate color reproduction in subsequent frames.

[0045] To further illustrate the practical application effects of the present invention, the inventors provide a specific application example: Suppose that in an R scenario for industrial simulation, the scenario framework is as follows: Observe a highly saturated red robotic arm model superimposed on a real factory background through AR glasses. The input video stream is 4K resolution, 60 frames per second, and the original color gamut is DCI-P3, while the display color gamut of the AR glasses is sRGB.

[0046] First, the dynamic color gamut perception model analyzes the first frame and identifies the red robotic arm as a high-saturation foreground region and the workshop background as a low-light background region through semantic segmentation. The dynamic color gamut perception model generates a local mapping function based on the sRGB color gamut boundary (8 vertices) and performs progressive compression in the red region.

[0047] After initial mapping, the pre-encoding analysis detected high chromaticity gradients and high saturation change rates at the edges of the robotic arm, generating a high-sensitivity map. Based on this, the encoder reduced the chromaticity QP of the robotic arm region and encoded it using a 4×4 TU. The encoded bitstream was then transmitted to the AR glasses. Further analysis of the reconstructed frames by the encoding distortion feedback channel revealed that the red color of the robotic arm was slightly darker after reconstruction (CIEDE2000 color difference of 2.5). This error information was fed back to the dynamic color gamut perception model. When processing the second frame, the model added a saturation compensation of 0.05 to the mapping function for the red region. After several iterations, the difference between the reconstructed color and the original mapped color was effectively controlled within a threshold of 2.3. The user observed stable and high-fidelity colors of the robotic arm, without obvious color blocks or color shifts.

[0048] Experimental data show that, at the same bitrate, the present invention reduces the average CIEDE2000 color difference by 32% and improves the subjective visual score by 1.8 points (out of 5) compared to the independent processing scheme.

[0049] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0050] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

A method for collaborative optimization of color gamut mapping and encoding in R scenes, characterized in that, include: Construct a dynamic color gamut perception model for R scenarios, and generate a local color gamut mapping function that adapts to the current scenario in real time based on the spatial content features of the input image or video frame, lighting conditions and the color gamut boundary of the target display device. Perform preliminary color gamut mapping, transforming the original wide color gamut content to the color gamut range reproducible by the target display device according to the local color gamut mapping function, and generating mapped image data; Before encoding, the mapped image data is analyzed to extract the chromaticity gradient distribution, saturation change rate, and spatial frequency characteristics of the region adjacent to the color gamut boundary, and a chromaticity distortion sensitivity map is constructed. The chroma distortion sensitivity map is embedded into the rate distortion optimization module of the encoder, and the quantization parameters and transform block partitioning strategy are dynamically adjusted so that encoding resources are preferentially allocated to the chroma distortion sensitive area. Perform video or image encoding using a standard encoding framework. Based on the adjusted quantization parameters and block partitioning strategy, compress and encode the mapped image data to generate the final bitstream. A coding distortion feedback channel is established, the decoded reconstructed frame is used to calculate the chromaticity error distribution between the frame and the mapped image, and the error information is fed back to the dynamic color gamut perception model for iterative optimization of the local color gamut mapping function of subsequent frames.

2. The optimization method according to claim 1, characterized in that, The dynamic color gamut perception model adopts a region adaptive strategy based on content semantic segmentation, which divides the image into high-saturation foreground region, low-light background region and dynamic transition region, and independently calculates the color gamut compression ratio and hue offset for each type of region.

3. The optimization method according to claim 1, characterized in that, The color gamut boundary of the target display device is represented by a polygon vertex set in the CIE 1931 chromaticity coordinate system, with no fewer than 8 vertices, covering red, green, blue, cyan, magenta, yellow and white points, and the color gamut volume calculation accuracy is no less than 0.1%.

4. The optimization method according to claim 1, characterized in that, The local color gamut mapping function adopts a piecewise nonlinear compression strategy, which performs progressive compression within the 0.95 to 1.0 times saturation range within the color gamut boundary. The compression slope decreases linearly with saturation to ensure smooth color transition in high-saturation areas.

5. The optimization method according to claim 1, characterized in that, The chromaticity gradient distribution is calculated by using the a and b channels in the CIELAB color space to determine the gradient magnitude in the horizontal and vertical directions, respectively. The gradient threshold is set to 3.5, and pixels exceeding this threshold are marked as chromaticity edge points. The saturation change rate is defined as the absolute value of the saturation difference between adjacent pixels divided by the spatial distance. When the change rate is greater than 0.15, it is determined to be a high-sensitivity area.

6. The optimization method according to claim 1, characterized in that, The rate-distortion optimization module introduces a chromaticity weighting factor, which is proportional to the local mean of the chromaticity distortion sensitivity map, with a scaling factor ranging from 0.8 to 1.2, and is used to adjust the chromaticity distortion weight in the Lagrange multiplier.

7. The optimization method according to claim 1, characterized in that, The transform block partitioning strategy forces the use of 4×4 minimum transform units in chroma distortion-sensitive regions, while allowing the use of 32×32 maximum transform units in low-sensitivity regions, in order to balance compression efficiency and chroma fidelity.

8. The optimization method according to claim 1, characterized in that, The coding framework supports intra-frame prediction and inter-frame prediction modes. The motion vector accuracy of inter-frame prediction is improved to 1 / 16 pixel, and independent motion compensation is enabled in the chroma component, with the compensation accuracy being consistent with that of the luminance component.

9. The optimization method according to claim 1, characterized in that, The encoding distortion feedback channel uses a low-latency decoder, which performs full-frame reconstruction only on key frames or scene switching frames, and macroblock-level local reconstruction on the remaining frames, with reconstruction latency controlled within 2 frames.

10. The optimization method according to claim 9, characterized in that, The chromaticity error distribution is calculated using the CIEDE2000 chromaticity difference formula, with an error threshold set to 2.

3. Regions exceeding this threshold will have a saturation compensation of 0.05 added to the local color gamut mapping function in the next frame.

Images