Image data processing method and related product

Abstract

The invention provides an image data processing method and system, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of image processing. The method is applied to an image sending end, and comprises the following steps: dividing a video frame image into a plurality of image blocks; determining a target image block from the plurality of image blocks, and calculating a linear relation parameter of a chrominance component relative to a brightness component of a pixel in the target image block; and obtaining supplementary information of the target image block according to the linear relation parameters, coding the supplementary information and the plurality of image blocks together, and sending the coded supplementary information and the plurality of image blocks to an image receiving end. According to the method, the image quality of the video frame image at the image receiving end is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image data processing method, system, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In scenarios such as remote desktops, video conferencing, and live streaming, image data needs to be compressed and encoded before being transmitted from the sending end to the receiving end. Traditional solutions use a generalized compression method to compress image data, resulting in poor image quality after reconstruction at the receiving end. Summary of the Invention

[0003] This disclosure provides an image data processing method, system, electronic device, computer-readable storage medium, and computer program product.

[0004] In a first aspect, embodiments of this disclosure propose an image data processing method applied to an image transmitting end, comprising: dividing a video frame image into multiple image blocks; determining a target image block from the multiple image blocks and calculating a linear relationship parameter between the chromaticity component and the luminance component of the pixels in the target image block; obtaining supplementary information of the target image block based on the linear relationship parameter, and encoding the supplementary information together with the multiple image blocks and sending it to an image receiving end.

[0005] In some embodiments, determining a target image block from a plurality of image blocks includes: determining at least one compression-sensitive image block from the plurality of image blocks based on the chromaticity component differences of pixels within the plurality of image blocks; calculating the linearity coefficient between the luminance component and the chromaticity component of pixels within the compression-sensitive image block; and determining the compression-sensitive image block with the linearity coefficient greater than a first threshold as the target image block.

[0006] In some embodiments, determining at least one compression-sensitive image block from a plurality of image blocks based on the chroma component differences of pixels within a plurality of image blocks includes: for each image block, calculating the chroma component differences of pixels within the image block based on the neighboring pixels of the pixels within the image block, and determining a target pixel from the plurality of pixels based on the chroma component differences of the plurality of pixels within the image block; and determining at least one compression-sensitive image block from the plurality of image blocks based on the number of target pixels within the plurality of image blocks.

[0007] In some embodiments, a video frame image includes multiple image regions, each image region including at least one image block; determining at least one compression-sensitive image block from multiple image blocks based on the chroma component differences of pixels within the multiple image blocks includes: using a thread-level parallel processing method, determining compression-sensitive image blocks in image blocks within different image regions in parallel based on the chroma component differences of pixels within image blocks in different image regions.

[0008] In some embodiments, calculating the linearity coefficients of the luminance and chrominance components of pixels within a compression-sensitive image block includes: obtaining the variance of the luminance component, the variance of the chrominance component, and the covariance of the luminance and chrominance components of pixels within the compression-sensitive image block based on the luminance and chrominance components of pixels within the compression-sensitive image block; and obtaining the linearity coefficient of the compression-sensitive image block based on the variance of the luminance component, the variance of the chrominance component, and the covariance of the luminance and chrominance components of pixels within the compression-sensitive image block.

[0009] In some embodiments, calculating the linear relationship parameters of the chromaticity components of pixels within a target image block relative to the luminance components includes: obtaining the linear relationship parameters of the chromaticity components of pixels within a target image block relative to the luminance components based on the variance of the luminance components, the mean of the luminance components, the mean of the chromaticity components, and the covariance between the luminance components and the chromaticity components.

[0010] In some embodiments, supplementary information of the target image block is obtained based on the linear relationship parameter, and the supplementary information is encoded together with multiple image blocks of the video frame image and sent to the image receiving end. This includes: compressing and encapsulating the linear relationship parameter to obtain supplementary information of the target image block; downsampling and compressing the chroma components of pixels in each image block of the multiple image blocks of the video frame image to obtain compressed chroma components, and generating compressed image data of the video frame image based on the compressed chroma components of each image block and the luminance components of pixels; during the encoding of the compressed image data, the supplementary information is inserted into the encoded bitstream corresponding to the compressed image data, and the encoded bitstream containing the compressed image data and supplementary information is sent together to the image receiving end.

[0011] In some embodiments, compressing and encapsulating the linear relationship parameters to obtain supplementary information of the target image block includes: quantizing the linear relationship parameters of the target image block to obtain quantized parameters; compressing the index of the target image block and the quantized parameters to obtain compressed parameters of the target image block; and encapsulating the compressed parameters of the target image block with the display timestamp corresponding to the video frame image to obtain supplementary information.

[0012] In some embodiments, the video frame image includes multiple target image blocks, wherein each target image block corresponds to a compressed parameter; the compressed parameter of the target image block and the display timestamp corresponding to the video frame image are encapsulated to obtain supplementary information, including: determining the length of the compressed parameter based on the number of multiple target image blocks; and encapsulating the compressed parameter of the target image block, the length of the compressed parameter, and the display timestamp corresponding to the video frame image to obtain supplementary information.

[0013] In some embodiments, the number of video frame images is multiple. After dividing the video frame images into multiple image blocks, the method further includes: generating an index for multiple image blocks based on the frame number of the video frame image corresponding to the multiple image blocks, the row number and column number of the image block in the corresponding video frame image.

[0014] Secondly, embodiments of this disclosure also provide another image data processing method, applied to an image receiving end, the method comprising: processing the received encoded bitstream to obtain video frame data and supplementary information; restoring the video frame data based on the supplementary information to obtain a video frame image; wherein, the supplementary information includes linear relationship parameters of the chromaticity components of pixels within a target image block relative to the luminance components, and the supplementary information is used to restore the chromaticity components in the target image block.

[0015] In some embodiments, restoring video frame data based on supplementary information to obtain a video frame image includes: decoding and upsampling the video frame data to obtain decompressed image data; wherein the decompressed image data includes the luminance components and decompressed chrominance components of pixels within multiple image blocks, and the multiple image blocks include a target image block and non-target image blocks other than the target image block; updating the decompressed chrominance components of the target image block based on the supplementary information to obtain an updated target image block; obtaining non-target image blocks based on the luminance components and decompressed chrominance components of pixels within the non-target image blocks; and obtaining a video frame image based on the updated target image block and non-target image blocks.

[0016] In some embodiments, updating the decompressed chroma components of the target image block based on supplementary information to obtain an updated target image block includes: decompressing the supplementary information to obtain the index of the target image block and the linear relationship parameters of the chroma components of the pixels within the target image block relative to the luminance components; and updating the decompressed chroma components of the target image block according to the index of the target image block and the linear relationship parameters to obtain the updated target image block.

[0017] In some embodiments, updating the decompressed chromaticity of a target image block according to its index and linear relationship parameters to obtain an updated target image block includes: determining the luminance component and decompressed chromaticity component of a pixel within the target image block according to its index; calculating the chromaticity information of a pixel within the target image block according to the linear relationship parameters and the luminance component of the pixel within the target image block; and replacing the chromaticity component of the target image block with the chromaticity information of the pixel within the target image block to obtain the updated target image block.

[0018] Thirdly, embodiments of this disclosure also provide an image data processing system, the system including an image transmitting end, the image transmitting end including: a division module, used to divide a video frame image into multiple image blocks; a calculation module, used to determine a target image block from the multiple image blocks and calculate the linear relationship parameter of the chromaticity component relative to the luminance component of the pixels in the target image block; an encoding module, used to obtain supplementary information of the target image block according to the linear relationship parameter, and encode the supplementary information together with the multiple image blocks of the video frame image and send it to the image receiving end.

[0019] In some embodiments, the system further includes an image receiving end, which includes: a separation module for processing the received encoded bitstream to obtain video frame data and supplementary information; and a restoration module for restoring the video frame image based on the supplementary information to obtain the video frame image; wherein the supplementary information includes linear relationship parameters between the chromaticity components and the luminance components of pixels within the target image block, and the supplementary information is used to restore the chromaticity components in the target image block.

[0020] Fourthly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the image data processing method as described in either the first or second aspect.

[0021] Fifthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the image data processing method as described in either the first or second aspect.

[0022] In a sixth aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the image data processing method as described in any of the first and second aspects.

[0023] This disclosure divides video frame images into blocks, allowing for processing of video frame images on a block-by-block basis. This enables differentiated processing methods for different image blocks, improving the accuracy of video frame image processing. Furthermore, this disclosure determines the linear relationship parameter between the chroma components and luminance components of pixels within a target image block, and encodes this parameter as supplementary information along with multiple image blocks before transmitting it to the image receiver. This allows the image receiver to directly recover the chroma components of the target image block based on the linear relationship between the luminance and chroma components using the received linear relationship parameter. Compared to recovering the chroma components of the target image block by decoding and decompressing the received chroma components, this disclosure reduces chroma component errors during image data transmission by recovering the chroma components using linear relationship parameters, thereby reducing chroma component distortion. Additionally, deriving the chroma components from the luminance components improves the signal-to-noise ratio of the video frame image, further enhancing the image quality at the image receiver. Attached Figure Description

[0024] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart of an image data processing method provided in this disclosure embodiment; Figure 2 A flowchart illustrating yet another image data processing method provided in this disclosure embodiment; Figure 3 This is a flowchart illustrating an image data processing method provided by an embodiment of the present disclosure in conjunction with a specific application scenario; Figure 4 This is a structural block diagram of an image data processing system provided in an embodiment of the present disclosure; Figure 5 A structural block diagram of another image processing system provided in this disclosure embodiment; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0026] In desktop scenarios such as desktop projection, remote desktop transfer, and screen recording, image data compression and recovery using traditional downsampling compression may have the following problems: Firstly, because the color difference between adjacent pixels of high-contrast elements (such as black text on a white background) in desktop scenes is large (reaching 60-80 gray levels), directly using average downsampling compression will dilute the abrupt change information, and thus bilinear interpolation cannot restore it, resulting in "color halo" at the edge of the element (the measured error can even reach 8-12 gray levels), which causes severe color distortion at the edge of text / icons and affects readability.

[0027] Secondly, in desktop scenarios, due to the abrupt change in chromaticity at the boundaries of solid color blocks in desktop rules, related technologies do not distinguish between the boundaries and the interior, and directly use downsampled mixed chromaticity values, resulting in a 2-3 pixel blur band after upsampling, with colors interpenetrating each other.

[0028] Thirdly, for table lines and icon outlines that are 1-2 pixels wide, downsampling is directly used, which causes the color features to be downsampled and averaged, and upsampling cannot recover them. This results in the loss of color in high-frequency fine lines, causing the lines to become lighter, distorted, or even disappear.

[0029] Fourthly, in desktop scenarios, the brightness and chromaticity of the core areas of the desktop (such as text and solid color blocks) are strongly correlated. However, related technologies do not utilize this feature but rely on general interpolation to restore the image, resulting in limited image restoration accuracy.

[0030] Fifthly, the generalized downsampling strategy is not compatible with the characteristics of "high frequency density and abrupt color changes" in desktop scenarios. In the generalized downsampling strategy, the signal-to-noise ratio (SNR) is only 30-35dB, which is lower than the requirement of more than 40dB in desktop scenarios, resulting in insufficient SNR during sampling conversion.

[0031] To address the problems mentioned above, this disclosure provides an image data processing method that can be applied to scenarios where there is a linear correlation between the luminance and chrominance components of pixels in image data, including but not limited to the desktop scenarios mentioned above.

[0032] Figure 1 This is a flowchart of an image data processing method provided in an embodiment of the present disclosure, applied to an image sending end, wherein the image sending end can be an encoding end.

[0033] See Figure 1 The process specifically includes the following steps: Step 101: Divide the video frame image into multiple image blocks.

[0034] Specifically, a video frame image, as a global whole, exhibits significant differences in characteristics (such as texture, brightness, chroma, motion state, etc.) across different regions. Processing the entire video frame image may result in the loss of local details and susceptibility to noise interference. This disclosure addresses this by dividing the image into blocks, decomposing the global problem into a series of local problems, thereby enabling refined processing within each image block and improving the accuracy of the image data.

[0035] In some embodiments, video frame images can be divided into multiple image blocks based on computational accuracy and CPU cache hit rate.

[0036] In this context, computational precision refers to the accuracy of video frame image processing. Specifically, for the same video frame image, the more image blocks it is divided into, the more refined the processing of that video frame image, and the greater the computational precision; conversely, the fewer image blocks it is divided into, the coarser the processing of that video frame image, and the lower the computational precision.

[0037] CPU cache hit rate refers to the percentage of times the CPU finds data in the cache within a certain period of time. A higher hit rate results in faster program execution and better performance.

[0038] Specifically, in computer architecture, the CPU performs extremely fast calculations, but reading data directly from main memory (RAM) is very slow. To resolve this contradiction, multiple levels of high-speed caches (L1, L2, L3, etc.) are set up between the CPU and main memory. These caches are small in capacity but extremely fast. When the CPU needs to read data, it first searches in the fastest L1 cache. If the data is found, it is used directly; this is called a "cache hit." If the data is not found, it searches step by step in the slower L2, L3 caches, and even main memory; this is called a "cache miss," which causes the CPU to "idle" waiting for hundreds of clock cycles, greatly reducing efficiency.

[0039] If the image patch is too small, it may cause the CPU to access pixels in a skipping and non-contiguous manner across the entire video frame when processing the image. This can lead to the data just loaded into the cache being overwritten by the new data needed later before it is fully utilized, resulting in frequent cache misses and the CPU spending most of its time waiting for data. Furthermore, the smaller the image patch, the higher the proportion of management overhead (such as function calls and loop control) in the processing of each image patch. Conversely, the larger the image patch, the more all the data of an image patch (including input pixels, intermediate calculation results, etc.) can be stored in the CPU's high-speed cache at once. In this way, the CPU hardly needs to access slow main memory when processing the image patch, resulting in higher efficiency.

[0040] This disclosure can balance computational accuracy and CPU cache hit rate to divide video frame images into multiple image blocks of appropriate size.

[0041] In addition, this disclosure may also divide image blocks based on the resolution, texture complexity, etc. of video frame images, or may adopt multi-level division strategies, etc., and this disclosure does not limit it in this way.

[0042] Step 102: Determine the target image block from multiple image blocks and calculate the linear relationship parameter between the chromaticity component and the luminance component of the pixels in the target image block.

[0043] Specifically, the target image block can be an image block in which the chromaticity component and the luminance component of the pixels are linearly correlated. For an image block in which the chromaticity component and the luminance component of the pixels are linearly correlated, the chromaticity component can be directly derived from the luminance component of the pixels in the image block.

[0044] In some embodiments, for each image block, the linear correlation between the chromaticity component and the luminance component of the pixels in the image block can first be determined based on the luminance component and chromaticity component of the pixels in the image block. Then, image blocks with a linear correlation higher than a preset level can be identified as target image blocks. Further, the linear relationship parameter between the chromaticity component and the luminance component of the pixels in the target image block is calculated, and the chromaticity component can be directly derived from the luminance component based on the linear relationship parameter.

[0045] Step 103: Based on the linear relationship parameters, obtain the supplementary information of the target image block, encode the supplementary information together with multiple image blocks, and send it to the image receiving end.

[0046] Specifically, when the image transmitting end encodes and sends the video frame image to the image receiving end, it not only encodes and sends multiple image blocks of the video frame image to the image receiving end, but also encodes and sends the linear relationship parameters of the target image block, which have a correlation between the luminance component and the chrominance component, as supplementary information of the target image block, along with multiple image blocks, to the image receiving end. In this way, the image receiving end can recover multiple image blocks by decoding and decompressing the received encoded bitstream; furthermore, for the target image block, it can directly deduce and recover the chrominance component of the target image block based on the linear relationship parameters of the luminance and chrominance components of the pixels within the target image block relative to the luminance component, and the luminance component itself.

[0047] This disclosure divides video frame images into blocks, allowing for processing of video frame images on a block-by-block basis. This enables differentiated processing methods for different image blocks, improving the accuracy of video frame image processing. Furthermore, this disclosure determines the linear relationship parameter between the chroma components and luminance components of pixels within a target image block, and encodes this parameter as supplementary information along with multiple image blocks before transmitting it to the image receiver. This allows the image receiver to directly recover the chroma components of the target image block based on the linear relationship between the luminance and chroma components using the received linear relationship parameter. Compared to recovering the chroma components of the target image block by decoding and decompressing the received chroma components, this disclosure reduces chroma component errors during image data transmission by recovering the chroma components using linear relationship parameters, thereby reducing chroma component distortion. Additionally, deriving the chroma components from the luminance components improves the signal-to-noise ratio of the video frame image, further enhancing the image quality at the image receiver.

[0048] It should be noted that the chromaticity components of a pixel include a blue chromaticity component and a red chromaticity component. The chromaticity components mentioned above and below in this application may include only the blue chromaticity component or the red chromaticity component, or they may include both. Correspondingly, the linear relationship parameters between the chromaticity components and the luminance components include a first linear relationship parameter between the blue chromaticity component and the luminance component and / or a second linear relationship parameter between the chromaticity component and the red chromaticity component; this disclosure does not limit this relationship.

[0049] Furthermore, when decoding and restoring encoded video frame images, areas such as text / icon edges, solid color block boundaries, or high-frequency fine lines in the video frame image may experience severe chroma distortion after decoding, which will significantly affect the image quality of the restored video frame image. However, other areas in the video frame image may not have a significant impact on the chroma of the image after decoding and restoration, and generally will not have a significant impact on the image quality of the restored video frame image.

[0050] Therefore, in response to the above Figure 1 In step 102, when determining the target image block, only the linear relationship parameters of the image block that may cause severe color distortion after decoding can be encoded and transmitted to the image receiver along with multiple image blocks. This reduces the amount of data transmitted and improves data transmission efficiency. A specific implementation method is given below.

[0051] In some embodiments, determining a target image block from a plurality of image blocks includes: determining at least one compression-sensitive image block from the plurality of image blocks based on the chromaticity component differences of pixels within the plurality of image blocks; calculating the linearity coefficient between the luminance component and the chromaticity component of pixels within the compression-sensitive image block; and determining the compression-sensitive image block with the linearity coefficient greater than a first threshold as the target image block.

[0052] In this context, a compression-sensitive image block refers to an image block whose chroma component reconstruction error exceeds a preset threshold after processing with a preset encoding / decoding standard. In other words, a compression-sensitive image block is an image block that is prone to chroma distortion after decoding. The chroma component reconstruction error exceeding the preset threshold can be either the reconstruction error of the blue chroma component or the red chroma component exceeding the preset threshold, or it can be the combined reconstruction error of both the blue and red chroma components exceeding the preset threshold; this disclosure does not limit this to either.

[0053] Specifically, a large difference in the chromaticity components of pixels within an image patch indicates a jump in the chromaticity values ​​of the pixels within the patch, which represents high-frequency chromaticity information. During image encoding and decoding, the image patch is transformed from the spatial domain to the frequency domain. The transformed high-frequency coefficients correspond to rapid changes in the image (such as edges and textures). However, during quantization, high-frequency coefficients are usually coarser quantized (i.e., using a larger quantization step size) to reduce the bit rate. Therefore, high-frequency information representing jumps in chromaticity is easily lost during quantization, leading to blurred chromaticity edges or ringing effects after decoding.

[0054] Therefore, this disclosure can determine whether an image block is a compression-sensitive image block by measuring the difference in the chromaticity components of pixels within the image block.

[0055] In some embodiments, compression-sensitive image blocks may include solid color image blocks, high-frequency detail image blocks, and image blocks that include images / text.

[0056] Among them, a solid color image block refers to an image block that is a solid color and includes its boundaries. Because the boundaries of solid color image blocks experience abrupt changes in chroma, pixel blurring bands will appear after decoding, and colors will interpenetrate each other, thus classifying them as distortable blocks.

[0057] High-frequency detail image blocks refer to image blocks containing high-frequency fine lines, such as table lines and icon outlines. Because high-frequency detail image blocks have large chroma component spatial gradients, they can blur details or produce ringing effects after encoding and compression, leading to lighter, distorted, or even disappeared lines, resulting in a significant decrease in subjective quality. Therefore, they are also classified as easily distorted blocks.

[0058] Image blocks that include images / text refer to areas containing text, sharp edges, or complex textures. Because image blocks that include images / text dilute abrupt changes during encoding and compression, making them unrecoverable during decoding, they can cause "colored halos" to appear at the edges of the decoded text / images, affecting readability. Therefore, they are also classified as volatile blocks.

[0059] Of course, compression-sensitive image blocks can also include flat chroma blocks. Since the chroma components of flat chroma blocks vary very little in space, even small errors during quantization will be converted into noticeable chroma deviations or chroma noise, thus they are easily distorted blocks, such as large areas of solid color backgrounds (blue sky, walls). This disclosure does not limit the types of compression-sensitive image blocks.

[0060] Furthermore, after determining the compression-sensitive image block, the linearity coefficient of the compression-sensitive image block can be used to determine whether there is a linear relationship between the luminance component and the chrominance component of the pixel in the compression-sensitive image block. Only compression-sensitive image blocks with a linear relationship can accurately deduce and restore the corresponding chrominance component based on the luminance component and the corresponding linear relationship parameter.

[0061] In some embodiments, the setting of the first threshold may be determined based on a preset threshold.

[0062] Specifically, as mentioned above, image blocks with reconstruction errors exceeding a preset threshold are considered compression-sensitive image blocks. The preset threshold measures the degree of chroma component error in these blocks. As long as the error between the derived chroma component and the original chroma component, based on the linear relationship parameter, is less than the preset threshold, the accuracy of video frame image restoration can be improved. A larger preset threshold results in a larger restored chroma error for the compression-sensitive image block, and correspondingly, a greater tolerance for the error between the derived chroma component and the original chroma component. For image blocks with higher linearity, the error between the derived chroma component and the original chroma component, obtained through the linear relationship parameter, is smaller, and the derived chroma component is more accurate. In this case, the linearity coefficient can be set smaller, meaning the first threshold can be set smaller. Conversely, a smaller preset threshold results in a smaller restored chroma error for the compression-sensitive image block, and correspondingly, a smaller tolerance for the error between the derived chroma component and the original chroma component. In this case, the linearity coefficient can be set larger, meaning the first threshold can be set larger. Therefore, the first threshold and the preset threshold can be negatively correlated.

[0063] In some embodiments, the first threshold can also be determined based on the precision of the decoded video frame image. Higher precision results in a higher linearity coefficient, allowing for a higher first threshold to be set; conversely, lower precision results in a lower linearity coefficient, allowing for a lower first threshold to be set. This disclosure does not limit the setting of the first threshold.

[0064] This embodiment first identifies compression-sensitive image blocks that are prone to distortion after processing with a preset encoding / decoding standard by using the chroma component difference of pixels within an image block. Then, based on the linearity coefficient of the luminance and chroma components of pixels in the compression-sensitive image block, it identifies target image blocks where the luminance and chroma components of pixels have a linear relationship. Thus, only the linear relationship parameters of the easily distorted image blocks with a linear relationship are encoded and transmitted to the image receiving end. This not only eliminates the boundary penetration and blur bands of pure color blocks, ensures smooth transitions, preserves the chroma characteristics of high-frequency fine lines, avoids the disappearance of distortion, and improves the signal-to-noise ratio of video frame images, further improving the image quality of video frame images at the image receiving end; it also reduces the amount of data transmitted through encoding, improving data transmission efficiency and accuracy.

[0065] Furthermore, compression-sensitive image blocks can be determined using the pixel neighborhood difference method. A specific implementation method is given below.

[0066] In some embodiments, determining at least one compression-sensitive image block from a plurality of image blocks based on the chroma component differences of pixels within a plurality of image blocks includes: for each image block, calculating the chroma component differences of pixels within the image block based on the neighboring pixels of the pixels within the image block, and determining a target pixel from the plurality of pixels based on the chroma component differences of the plurality of pixels within the image block; and determining at least one compression-sensitive image block from the plurality of image blocks based on the number of target pixels within the plurality of image blocks.

[0067] Specifically, for each image block, each pixel within the image block is traversed, and the absolute difference between each pixel and its neighboring pixels in terms of chromaticity components is calculated.

[0068] In some embodiments, the neighboring pixels of a pixel can be pixels adjacent to that pixel on one or more sides. For example, the chroma component difference of a pixel can be the absolute difference in chroma components between the pixel's right neighbor (horizontal direction) and its bottom neighbor (vertical direction). The chroma component differences between the last column and the last row of pixels in the image block can be ignored. Then, the high difference points of pixels in the image block are counted. If the difference in any direction (horizontal or vertical) of a pixel is greater than a preset difference threshold (e.g., 32), the pixel is considered a "high difference point". A "high difference point" pixel is a target pixel, meaning a pixel in the image block whose chroma component difference is greater than the preset difference threshold. Finally, based on the number of "high difference points" pixels in the image block, it is determined whether the image block is a compression-sensitive image block. Specifically, the total number of "high difference points" in the entire image block is counted. If the total number exceeds a small threshold (e.g., 1 or 3), the image block is determined to be a compression-sensitive image block (typically text, thin lines, or edge regions).

[0069] In some embodiments, the adjacent pixels of each pixel may also be the pixels corresponding to the right neighbor (horizontal direction) or the bottom neighbor (vertical direction) of the pixel, or the pixel neighborhood of the pixel, that is, a set of pixels that are directly adjacent to the pixel in space with the pixel as the center. The neighborhood pixels may be four-neighborhood or eight-neighborhood. This disclosure does not limit the adjacent pixels of a pixel.

[0070] The embodiments of this disclosure determine whether an image block is a compression-sensitive image block by using the difference in chromaticity components between adjacent pixels, which can improve the accuracy and efficiency of determining compression-sensitive image blocks.

[0071] Furthermore, this disclosure can also determine whether multiple image blocks in a video frame are compression-sensitive image blocks by using threads in parallel.

[0072] In some embodiments, a video frame image includes multiple image regions, each image region including at least one image block; determining at least one compression-sensitive image block from multiple image blocks based on the chroma component differences of pixels within the multiple image blocks includes: using a thread-level parallel processing method, determining compression-sensitive image blocks in image blocks within different image regions in parallel based on the chroma component differences of pixels within image blocks in different image regions.

[0073] Specifically, video image frames can be divided into multiple image regions according to preset rules, and each image region can be processed by a thread, with multiple threads corresponding to multiple image regions being processed in parallel.

[0074] The preset rules can be to divide the video image frame into regions according to rows, or according to columns, or according to the type of image in the video image frame (such as a solid color image, an image containing text, lines, etc.). This disclosure does not limit these rules.

[0075] This disclosure employs a multi-threaded parallel approach based on image regions. By determining compression-sensitive image blocks from multiple image blocks, the efficiency of determining compression-sensitive image blocks can be further improved.

[0076] In some embodiments, calculating the linearity coefficients of the luminance and chrominance components of pixels within a compression-sensitive image block includes: obtaining the variance of the luminance component, the variance of the chrominance component, and the covariance of the luminance and chrominance components of pixels within the compression-sensitive image block based on the luminance and chrominance components of pixels within the compression-sensitive image block; and obtaining the linearity coefficient of the compression-sensitive image block based on the variance of the luminance component, the variance of the chrominance component, and the covariance of the luminance and chrominance components of pixels within the compression-sensitive image block.

[0077] Specifically, first, all pixels within the compression-sensitive image block are traversed. Then, the sum of the luminance components of all pixels is calculated, and the mean of the luminance components is calculated based on the sum of the luminance components. Similarly, the sum of the chrominance components of all pixels is calculated, and the mean of the chrominance components is calculated based on the sum of the chrominance components. The calculation of the luminance and chrominance sums can be performed in parallel using the SIMD instruction set. Further, all pixels within the compression-sensitive image block are traversed again, and the variance and covariance are calculated using the obtained mean luminance and mean chrominance values. To obtain unbiased estimates, N-1 is used as the denominator when calculating the variance and covariance, where N is the total number of pixels within the compression-sensitive image block.

[0078] Among them, the luminance component variance is used to measure the dispersion of the luminance component of pixels within a compression-sensitive image patch; the chrominance variance is used to measure the dispersion of the chrominance component of pixels within a compression-sensitive image patch; and the covariance is used to measure the degree of co-variation between the luminance and chrominance components of pixels within a compression-sensitive image patch. The chrominance component can be a blue chrominance component and / or a red chrominance component.

[0079] In some embodiments, the linearity coefficient of a compression-sensitive image patch can be obtained using a correlation coefficient formula. The correlation coefficient formula can be the Pearson correlation coefficient formula: Pearson correlation coefficient r = Cov(Y,Cb) / sqrt(Var(Y)×Var(Cb)). Here, r is the Pearson correlation coefficient, Cov(Y,Cb) is the covariance between the luminance and chrominance components, Var(Y) is the variance of the luminance component, Var(Cb) is the variance of the chrominance component, sqrt() is the square root operation, and the value of r ranges from -1 to 1. The closer r is to 1, the higher the positive correlation between the chrominance and luminance components; the closer r is to -1, the higher the negative correlation between the chrominance and luminance components; and the closer r is to 0, the lower the correlation between the chrominance and luminance components.

[0080] In some embodiments, the correlation coefficient can be calculated based solely on the blue color difference component of the pixel.

[0081] This disclosure uses the Pearson correlation coefficient to determine the linearity coefficients of the luminance and chrominance components of pixels within a compression-sensitive image block, thus improving the accuracy of coefficient determination. Of course, this disclosure can also determine the linearity coefficients of the luminance and chrominance components through other methods, such as comparison with a pre-trained neural network model, a large language model, or a pre-set image database; this disclosure does not limit this method.

[0082] After determining the linearity coefficient of each compression-sensitive image patch, the target image patch can be identified from multiple compression-sensitive image patches based on the linearity coefficient. For example, a positive correlation threshold r0 = 0.8 can be preset. If the calculated correlation coefficient r is greater than or equal to this threshold (i.e., r ≥ 0.8), then the luminance and chrominance components of the compression-sensitive image patch are considered to have a significant positive linear relationship. Image patches that simultaneously meet the conditions of being prone to chrominance distortion and having a linear relationship are ultimately identified as target image patches. Optionally, the relevant information of the target image patch can be recorded in a shared array for subsequent processing.

[0083] Furthermore, the linear relationship parameters between the chromaticity components and the luminance components of pixels within a target image block can be specifically calculated.

[0084] In some embodiments, calculating the linear relationship parameters of the chromaticity components of pixels within a target image block relative to the luminance components includes: obtaining the linear relationship parameters of the chromaticity components of pixels within a target image block relative to the luminance components based on the variance of the luminance components, the mean of the luminance components, the mean of the chromaticity components, and the covariance between the luminance components and the chromaticity components.

[0085] Specifically, the linear relationship parameters between the chromaticity component and the luminance component of pixels within a target image patch can be obtained through linear regression fitting. The linear regression fitting can be a least-squares linear regression closed-loop solution formula. The linear relationship parameters include a first parameter and a second parameter, where the first parameter α_Cb = Cov(Y,Cb) / Var(Y); and the second parameter β_Cb = μ_Cb / 256 - α_Cb × (μ_Y / 256). Cov(Y,Cb) is the covariance between the luminance and chromaticity components, Var(Y) is the variance of the luminance component, μ_Cb is the mean of the chromaticity component, and μ_Y is the mean of the luminance component. When calculating β_Cb, the mean μ_Cb and μ_Y are 8-bit data (0-255), but for calculation accuracy, they are first divided by 256 to normalize them to the [0,1) interval (or approximate). Then, the previously calculated α_Cb (which may already be a fixed-point number) is used for calculation to obtain β_Cb.

[0086] In addition, the chromaticity components include blue chromatic difference components and red chromatic difference components. Correspondingly, the linear relationship parameters include the first linear relationship parameter corresponding to the blue chromatic difference component and the second linear relationship parameter corresponding to the red chromatic difference component. The calculation method is similar and will not be repeated here.

[0087] The embodiments of this disclosure can accurately obtain the linear relationship parameters between the chromaticity component and the luminance component of pixels within a target image block through linear regression fitting.

[0088] Furthermore, when calculating linear correlation parameters, a multi-linear parallel approach can be used to improve the calculation efficiency of linear correlation parameters.

[0089] Furthermore, after obtaining the linear relationship parameters between the chromaticity components and the luminance components of pixels within the target image block, a mapping table of linear correlation parameters and image blocks can be established for all linear correlation parameters. This mapping table only includes the target image block, and for non-target image blocks, there are no corresponding linear correlation parameters, so the parameters are not stored.

[0090] In response to the above Figure 1 In step 103, the supplementary information and the pixel information of the image block (including luminance component information and chrominance component information) can be compressed separately, and then encoded and sent to the image receiving end. A specific implementation method is given below.

[0091] In some embodiments, supplementary information of the target image block is obtained based on the linear relationship parameter, and the supplementary information is encoded together with multiple image blocks of the video frame image and sent to the image receiving end. This includes: compressing and encapsulating the linear relationship parameter to obtain supplementary information of the target image block; downsampling and compressing the chroma components of pixels in each image block of the multiple image blocks of the video frame image to obtain compressed chroma components, and generating compressed image data of the video frame image based on the compressed chroma components of each image block and the luminance components of pixels; during the encoding of the compressed image data, the supplementary information is inserted into the encoded bitstream corresponding to the compressed image data, and the encoded bitstream containing the compressed image data and supplementary information is sent together to the image receiving end.

[0092] First, for a target image block, a video frame image may include multiple target image blocks. The linear relationship parameters corresponding to each target image block can be encoded and compressed based on a preset encoding table, such as a preset Huffman code table, to obtain the compressed parameters of the corresponding target image block, thereby reducing the size of the linear relationship parameters. Then, the multiple compressed parameters are encapsulated into an SEI (Supplemental Enhancement Information Message) to form a supplementary information, which serves as supplementary information for the target image block in the video frame image.

[0093] Then, the pixel information of all image blocks in the video frame is compressed. Since the human eye is not sensitive to color but is sensitive to brightness, the brightness channel data in the image block is not processed, maintaining the original brightness resolution; while the chroma channel data in the image block can be compressed by downsampling to obtain compressed image data. Thus, the compressed image data is encoded and transmitted, reducing the bitrate of the image block (reducing the video size) while the visual quality remains almost unchanged.

[0094] During downsampling compression, 2×2 neighborhood average downsampling can be performed. The process is accelerated by sliding window summation. For each 2×2 window, the pixel values ​​of 4 chroma components (fixed number of points) are accumulated, and the average value is obtained by shifting it 2 bits to the right (equivalent to dividing by 4) to generate the compressed chroma components.

[0095] Finally, during the encoding and transmission of the compressed image data, the encapsulated supplementary information can be appended to the encoded bitstream corresponding to the compressed image data through a preset interface to obtain the bitstream of the video frame image, and then sent to the image receiving end.

[0096] This disclosure improves data transmission efficiency by compressing the linear relationship parameters and the pixel information of the image blocks separately, and then using the compressed parameters obtained after compressing the linear relationship parameters as supplementary information for the target image blocks. These compressed parameters, along with the bitstreams corresponding to the compressed image data of all image blocks, are sent to the image receiving end.

[0097] Furthermore, in some embodiments, the linear relationship parameters are compressed and encapsulated to obtain supplementary information of the target image block, including: quantizing the linear relationship parameters of the target image block to obtain quantized parameters; compressing the index of the target image block and the quantized parameters to obtain compressed parameters of the target image block; and encapsulating the compressed parameters of the target image block with the display timestamp corresponding to the video frame image to obtain supplementary information.

[0098] Specifically, the linear relationship parameters of the target image block are generally floating-point data. These parameters can be quantized first to reduce storage space, and then compressed. Additionally, multiple video frames are transmitted, each corresponding to supplementary information. By encapsulating the display timestamp of the video frame with the compressed parameters of the target image block, the compressed parameters can be bound to the corresponding video frame. This allows the receiving end to quickly and accurately locate the corresponding video frame after receiving the supplementary information.

[0099] After compressing and encapsulating the linear relationship parameters to obtain supplementary information for the target image block, the supplementary information can be stored in the memory buffer and marked as "auxiliary data" before being transmitted to the encoder along with multiple image block data of the video frame image.

[0100] Furthermore, a video frame image comprises multiple target image blocks, with each target image block corresponding to a compressed parameter. Therefore, the number of target image blocks may differ in different video frame images, and the corresponding compressed parameter length may also differ.

[0101] In some embodiments, the compressed parameters of the target image block and the display timestamp corresponding to the video frame image are encapsulated to obtain supplementary information, including: determining the length of the compressed parameters based on the number of multiple target image blocks; and encapsulating the compressed parameters of the target image block, the length of the compressed parameters, and the display timestamp corresponding to the video frame image to obtain supplementary information.

[0102] In other words, when encapsulating linear correlation parameters, the length of the compressed parameters can be encapsulated together with the display timestamps corresponding to the compressed parameters and video frame images, making the obtained supplementary information more comprehensive. As a result, when the image is restored at the image receiving end based on the supplementary information, the image restoration is more accurate.

[0103] Furthermore, depending on the application scenario, supplementary information may include other information, such as the type of supplementary information. In some application scenarios, multiple types of supplementary information may be included during bitstream transmission. By encapsulating the type of supplementary information within the supplementary information itself, the corresponding supplementary information can be quickly determined based on the type, improving data transmission efficiency.

[0104] In addition, after dividing the video frame image into multiple image blocks, the index of each image block can be obtained so that the corresponding image block can be accurately located in subsequent processing.

[0105] In some embodiments, the number of video frame images is multiple. After dividing the video frame images into multiple image blocks, the method further includes: generating an index for multiple image blocks based on the frame number of the video frame image corresponding to the multiple image blocks, the row number and column number of the image block in the corresponding video frame image.

[0106] Specifically, a 32-bit integer encoding of "32-bit frame number + 16-bit image block row number + 16-bit image block column number" can be used (to optimize memory storage). For example, the index of the image block in the 20th row and 30th column of the first frame can be (0x00000001<32)|(0x0014<<16)|0x001E.

[0107] Furthermore, the indices of each image block can be stored in an array, so that the indices of the image blocks correspond one-to-one with the positions of the image blocks.

[0108] This disclosure generates an index for an image block, which enables precise location of the corresponding image block within the video frame image, as well as its specific position within the corresponding video frame image, based on the index of the image block.

[0109] Furthermore, in the process of encapsulating the linear correlation parameters to obtain supplementary information, the index of the target image block corresponding to the linear correlation parameter can be compressed and encapsulated together with the corresponding linear correlation parameter.

[0110] against Figure 1 In step 101, before dividing the video frame image into multiple image blocks, the video frame image can be preprocessed first.

[0111] In some embodiments, dividing a video frame image into multiple image blocks includes: extracting the RGB components from the video frame image; converting the RGB components to obtain an image in a converted format; and dividing the image in the converted format into multiple image blocks.

[0112] Specifically, taking a desktop scenario as an example, the computer host captures video frame image data in BGRA (Blue-Green-Red-Alpha) format via a preset interface, avoiding image copying overhead. During component extraction, the Alpha channel is ignored (fixed at 255), and the RGB components are stored row-by-row in the CPU memory buffer. Memory alignment can be used during storage to allow subsequent access to the RGB components based on SIMD instructions, improving access efficiency.

[0113] Then, ITU-R BT.601 (a standard matrix) can be invoked to convert the RGB components into a YUV (Luminance(Y)-Chrominance(U,V)) image format. Furthermore, 16-bit fixed-point arithmetic can be implemented using a preset vector instruction set (avoiding floating-point errors and improving parallel efficiency). During normalization, software branching can be used to limit the chrominance component values ​​to the range of 0-255 (values ​​below 0 are set to 0, values ​​above 255 are set to 255). Additionally, the luminance and chrominance components can be stored in separate memory buffers. Finally, the converted YUV image is divided into multiple image blocks.

[0114] The data processing method disclosed herein, through the core design of "multi-threaded parallelism + SIMD acceleration + image block-level differential processing + SEI standard encapsulation", combined with the strong correlation characteristics of pixel luminance components, achieves a leapfrog improvement in the accuracy of high-frequency detail chromaticity restoration compared with existing image data processing methods.

[0115] Specifically, this solution divides the image into blocks and precisely selects target image blocks prone to chromatic distortion and exhibiting linear relationships. This ensures a high degree of matching between the linear fitting parameters and the local chromaticity variation patterns. The linear correlation parameters between the luminance and chromaticity components are calculated only for high-frequency areas prone to chromatic distortion, avoiding invalid calculations in low-correlation areas. Furthermore, during the restoration phase, the chromaticity component is accurately derived using the linear correlation parameters and the luminance component, completely resolving the pain point of existing technologies that "dilute abrupt changes through average downsampling." Real-world testing shows that the chromaticity error at text edges remains stable at a low grayscale level (generally between 0.9 and 1.7), the chromaticity loss rate of fine lines is significantly reduced to only 1.8%, and the sampling conversion signal-to-noise ratio is significantly improved to 52.3-54.7 dB, an improvement of ≥12 dB compared to existing solutions (signal-to-noise ratio of 36.8-39.5 dB). This allows for clear restoration of core details such as desktop text, table lines, and icon outlines.

[0116] Secondly, embodiments of this disclosure also provide another image data processing method, which can be applied to an image receiving end, wherein the image receiving end can be a decoding end.

[0117] Figure 2 A flowchart of yet another image data processing method provided in an embodiment of this disclosure.

[0118] See Figure 2 The process 200 may specifically include the following steps: Step 201: Process the received encoded bitstream to obtain video frame data and supplementary information.

[0119] Specifically, the encoded bitstream received by the image receiver includes two parts: video frame data and supplementary information. The video frame data refers to the pixel information corresponding to the video frame image, while the supplementary information refers to supplementary information about the pixel information of the target image block. Furthermore, the video frame data is compressed.

[0120] After receiving the bitstream, the image receiver can call a preset interface, such as the av_read_frame interface of FFmpeg (Fast Forwarding Multimedia Framework), to perform separation processing so that the video frame data and supplementary information are transmitted to different threads for processing.

[0121] Step 202: Reconstruct the video frame data based on the supplementary information to obtain the video frame image.

[0122] The supplementary information includes the linear relationship parameters between the chromaticity components and the luminance components of pixels within the target image block. This supplementary information is used to restore the chromaticity components in the target image block.

[0123] Specifically, the video frame data is the compressed data of the video frame image sent by the image transmitting end. This disclosure can restore the chromaticity component in the target image block based on the linear relationship between the luminance and chromaticity components of pixels within the target image block as indicated in the supplementary information, thereby obtaining the video frame image and improving the image quality of the final obtained video frame image.

[0124] The image data processing method provided in this embodiment of the present disclosure includes not only video frame data of each image block in the video frame image, but also supplementary information of the target image block in which the luminance component and chrominance component have a linear relationship within the pixel. Therefore, the image receiving end can correct the chrominance component in the video frame data according to the supplementary information of the target image block, thereby improving the image quality of the image at the image receiving end and enabling the image receiving end to more accurately reproduce the video frame image.

[0125] In some embodiments, restoring video frame data based on supplementary information to obtain a video frame image includes: decoding and upsampling the video frame data to obtain decompressed image data; updating the decompressed chroma component of the target image block based on the supplementary information to obtain an updated target image block; obtaining a non-target image block based on the luminance component and decompressed chroma component of pixels within the non-target image block; and obtaining a video frame image based on the updated target image block and the non-target image block.

[0126] The decompressed image data includes the luminance components and decompressed chrominance components of pixels within multiple image blocks. These multiple image blocks include target image blocks and non-target image blocks. Specifically, video frame data can be transmitted to the decoding thread, where FFmpeg's decoder can be called to decode the video frame data and obtain the decompressed image data. This involves decompressing the compressed chrominance components within the multiple image blocks to obtain the decompressed chrominance components. The luminance components of the image blocks are transmitted at their original resolution during the encoding and transmission process without compression, therefore, decompression of the luminance components of the image blocks is unnecessary.

[0127] Specifically, bilinear interpolation can be used to upsample and decompress the chrominance components. Furthermore, the interpolation results of multiple pixels can be computed in parallel using an instruction set to improve processing efficiency.

[0128] Since the luminance component of pixels within an image block is transmitted at its original resolution, no luminance loss occurs during transmission. Furthermore, there is a linear relationship between the luminance and chrominance components of pixels within the target image block. Therefore, the decompressed chrominance components in the target image block can be corrected based on supplementary information from the target image block. An updated target image block is then obtained based on the updated decompressed chrominance components and the luminance component of the target image block in the decompressed image data. For non-target image blocks, they are obtained based on the corresponding decompressed chrominance and luminance components in the decompressed image data. Thus, video frame images are reconstructed from the updated target and non-target image blocks, thereby improving the image quality of the final video frame images.

[0129] In addition, the target image block can be an image block in which the luminance component and chrominance component have a linear relationship in all pixels, or it can be an image block in which the luminance component and chrominance component have a linear relationship in pixels and are prone to chrominance distortion during encoding, decoding and transmission. For details, please refer to the description of the target image block above, which will not be repeated here.

[0130] In some embodiments, updating the decompressed chroma components of the target image block based on supplementary information to obtain an updated target image block includes: decompressing the supplementary information to obtain the index of the target image block and the linear relationship parameters of the chroma components of the pixels within the target image block relative to the luminance components; and updating the decompressed chroma components of the target image block according to the index of the target image block and the linear relationship parameters to obtain the updated target image block.

[0131] Specifically, the supplementary information received by the image receiver can be parsed by a parsing thread to restore the index of the target image block corresponding to the supplementary information and the corresponding linear correlation parameters. Based on the index in the supplementary information, the target image block can be determined from multiple image blocks in the decompressed image data. Then, the decompressed chromaticity components of the target image block in the decompressed image data can be updated according to the linear correlation parameters. Based on the updated chromaticity components and the corresponding luminance components, the target image block can be accurately recovered, resulting in the updated target image block.

[0132] In some embodiments, the supplementary information may further include display timestamp information corresponding to the video frame image where the target image block is located, so that the supplementary information and the video frame image can be synchronized according to the display timestamp information; further, the target image block in the corresponding video frame image is determined according to the index in the supplementary information. The specific content and technical effects can be found in the corresponding description above, and will not be repeated here.

[0133] In some embodiments, updating the decompressed chromaticity of a target image block according to its index and linear relationship parameters to obtain an updated target image block includes: determining the luminance component and decompressed chromaticity component of a pixel within the target image block according to its index; calculating the chromaticity information of a pixel within the target image block according to the linear relationship parameters and the luminance component of the pixel within the target image block; and replacing the chromaticity component of the target image block with the chromaticity information of the pixel within the target image block to obtain the updated target image block.

[0134] Specifically, based on the index in the supplementary information, the decompressed chroma components corresponding to the supplementary information can be determined from the decompressed chroma components of multiple image blocks. Thus, based on the linear relationship parameters and the corresponding decompressed chroma components, the original chroma components (i.e., the chroma components before compression) of the corresponding target image block can be obtained through linear regression fitting. Then, the original chroma components corresponding to the target image block can be directly used to replace the decompressed chroma components to obtain the updated chroma components.

[0135] In some embodiments, after obtaining the original chromaticity components of the corresponding target image block through linear regression fitting, in order to avoid overflow leading to color distortion, the recovered original chromaticity components can be normalized to limit the chromaticity components to the range of 0-255.

[0136] In some embodiments, after parsing the supplementary information, an index-parameter mapping table can be constructed based on the parsed index and the corresponding linear correlation parameters. This allows the target image block in the decompressed image data that needs to be updated for chroma components to be determined by querying the index-parameter mapping table during chroma component updates, thereby improving data processing efficiency.

[0137] Furthermore, in some embodiments, obtaining a video frame image based on the updated target image block and the non-target image block includes: performing an inverse color space conversion on the updated target image block and the non-target image block to obtain a BGRA format image, and rendering and displaying the BGRA format image to obtain a video frame image.

[0138] Specifically, the color space inverse conversion can be based on the aforementioned ITU-R BT.601 inverse matrix. The YUV format image blocks are converted to BGRA format image blocks in parallel through a vector instruction set. During the conversion process, 10 bits of intermediate precision are retained, and finally, the values ​​are rounded to 8 bits (0-255) for precision optimization and to reduce quantization errors. Finally, the BGRA format data is rendered to the display terminal (supporting multi-monitor expansion) through a preset interface. The rendering refresh rate is kept synchronized with the original frame rate to ensure no screen tearing or stuttering, so as to display and output the video frame images.

[0139] In addition, if the parameter parsing of a certain video frame fails or the image block matching rate is lower than the preset value, it can automatically switch to the full-frame normal upsampling mode to ensure display stability.

[0140] Figure 3 This is a flowchart illustrating an image data processing method provided by an embodiment of the present disclosure in conjunction with a specific application scenario. The method can be applied to an image data processing system, which includes an encoding end and a decoding end.

[0141] See Figure 3 This method is applied to desktop scenarios, and the process 300 specifically includes the following steps: Step 301: BGRA data input and preprocessing.

[0142] Among them, thread 1 (data reading thread) can be used for BGRA data input and preprocessing.

[0143] Specifically, the computer host captures desktop BGRA format image data through the first interface. This first interface can be a DirectX 12 (Microsoft Multimedia and Gaming Graphics Interface version 12) interface. This disclosure uses the DirectX 12 interface instead of the HDMI (High Definition Multimedia Interface) hardware interface, making desktop capture more universal in software editing scenarios and avoiding image copying overhead.

[0144] Then, the captured desktop BGRA format image data is processed by ignoring the alpha channel to obtain the RGB components. The RGB components are then stored row by row in the CPU memory buffer (contiguous physical address allocation, size: 1920×1080×3×1Byte=6.22MB), and memory alignment (e.g., 64-byte alignment) is used to optimize the access efficiency of subsequent SIMD instructions.

[0145] Step 302: Convert RGB to YUV444.

[0146] One approach is to use thread 2 (format conversion thread) to convert RGB to YUV444.

[0147] Specifically, a standard matrix (which can be an ITU-R BT.601 standard matrix) is invoked, and RGB data is converted to YUV444 format using a vector instruction set (which can be a CPUAVX2 vector instruction set) to perform 16-bit fixed-point operations, thereby avoiding floating-point errors and improving parallel efficiency. The specific calculation logic is as follows: Brightness Y: Y = (0.299 × R + 0.587 × G + 0.114 × B) × 256, where, after fixed-point processing, multiply by 256 and round down, subsequent calculations are all based on fixed-point numbers.

[0148] Chromaticity Cb: Cb = (0.492 × (BY / 256)) × 256 + 128; Chromaticity Cr: Cr = (0.877 × (RY / 256)) × 256 + 128, where adding 128 maps the Cb / Cr value range to 0-255 to conform to the encoding standard.

[0149] After fixed-point arithmetic, normalization is performed. Specifically, the Cb / Cr value can be limited to the range of 0-255 through software branching, where values ​​below 0 are set to 0 and values ​​above 255 are set to 255.

[0150] Finally, the output is stored. Specifically, the three components Y, Cb, and Cr are stored in independent memory buffers (each buffer size: 1920×1080×2Byte=4.15MB, 16-bit fixed-point storage), supporting random access to the data of each component.

[0151] Step 303: Divide the converted image into image blocks and allocate macroblock indexes.

[0152] This can be handled using thread 3 (block thread).

[0153] Specifically, taking into account both computational accuracy and CPU cache hit rate, the block size can be configured to 8×8 pixels. The image can be divided in the order of "from left to right and from top to bottom". A 1920×1080 image can be divided into 240 rows × 135 columns = 32400 blocks.

[0154] The macroblock index format can use a 32-bit integer encoding of "32-bit frame number + 16-bit block row number + 16-bit block column number". For example, the block index of the 20th row and 30th column of the first frame is (0x00000001<32)|(0x0014<<16)|0x001E. The index is stored in the array block_index

[32400] , which corresponds one-to-one with the block position. This macroblock index format can optimize memory storage.

[0155] Step 304: Perform high-frequency analysis and correlation determination on the multiple blocks that have been divided.

[0156] Among them, threads 4-8 (analysis threads) can be used to analyze multiple blocks in parallel, and thread 5 (region processing thread) can be used to process the image into regions.

[0157] Specifically, thread 5 can divide the image into 5 regions by rows, each region consisting of 48 rows of blocks. Then, the 5 analysis threads process the 5 regions in parallel, avoiding lock contention.

[0158] When performing high-frequency analysis, high-frequency chromaticity detection is performed first for each analysis thread.

[0159] Specifically, for each 8x8 block, a more direct pixel neighborhood difference method is used to determine whether it is a high-frequency region. More specifically, first, the pixels are traversed, iterating through each pixel within the block (except the last row and last column). Then, the difference is calculated: for each pixel, the absolute difference in chromaticity components (Cb / Cr) between it and its right neighbor (horizontal direction) and its bottom neighbor (vertical direction) is calculated. Finally, high difference points are counted; if the difference in any direction is greater than a preset threshold (e.g., 32), the pixel is considered a "high difference point".

[0160] Secondly, the high-frequency determination of blocks is performed. Specifically, the total number of "high difference points" in the entire block is counted. If the total number exceeds a small threshold (e.g., 1 or 3), the block is determined to be a "potential high-frequency block" (usually text, thin lines, or edge areas).

[0161] For potential high-frequency blocks, further correlation determination can be performed. This correlation determination can be based on the Pearson correlation coefficient.

[0162] Specifically, for the "potential high-frequency blocks" screened through high-frequency analysis, the system will further calculate the Pearson correlation coefficient r between its Y component and Cb component to quantify the degree of linear correlation between the two. The specific calculation process is as follows: First, calculate the mean: Iterate through the 64 pixels within the 8x8 block and calculate the mean value μ_Y of the Y component and the mean value μ_Cb of the Cb component respectively: μ_Y=(ΣY_i) / 64; μ_Cb=(ΣCb_i) / 64.

[0163] To improve computational efficiency, the summation process (Σ) can be processed in parallel using the SIMD instruction set.

[0164] Then, covariance and variance are calculated: the 64 pixels within the block are traversed again, and the covariance and variance are calculated using the previously obtained mean. To obtain an unbiased estimate, the denominator is N-1 (i.e., 63).

[0165] Covariance Cov(Y,Cb): measures the degree of co-variance between Y and Cb.

[0166] Cov(Y,Cb)=Σ[(Y_i-μ_Y)×(Cb_i-μ_Cb)] / 63; Variance Var(Y) and Var(Cb): These measure the degree of dispersion of the data.

[0167] Var(Y)=Σ[(Y_i-μ_Y)²] / 63; Var(Cb)=Σ[(Cb_i-μ_Cb)²] / 63.

[0168] Next, the correlation coefficient is calculated: Substitute the above results into the formula to calculate the Pearson correlation coefficient r. r = Cov(Y,Cb) / sqrt(Var(Y)×Var(Cb)). The value of this coefficient r ranges from -1 to 1. r close to 1 indicates a strong positive correlation, close to -1 indicates a strong negative correlation, and close to 0 indicates no linear correlation. Since this calculation involves square root and division, it is a floating-point operation, but because it is only performed on a small number of "potential high-frequency blocks" that have been screened, the overall computational load is controllable.

[0169] Finally, a threshold judgment and result recording are performed: a preset positive correlation threshold r0 = 0.8 is set. If the calculated correlation coefficient r is greater than or equal to this threshold (i.e., r ≥ 0.8), then the Y component and Cb component of the Block are considered to have a significant positive linear relationship. Blocks that simultaneously meet the two conditions of "potentially high frequency" and "high correlation" are ultimately determined to be "highly correlated high frequency Blocks". The system records their index in a shared array high_corr_blocks[] for subsequent processing.

[0170] Step 305: Calculate the linear parameters.

[0171] Among them, threads 9-12 (computation threads) can be used to calculate the linear parameters corresponding to multiple highly correlated and high-frequency blocks in parallel.

[0172] Specifically, for the Blocks in high_corr_blocks[], the four computing threads calculate the linear parameters in parallel by index segment (Cb and Cr both correspond to the two linear parameters α and β, and the Cb / Cr components are calculated separately).

[0173] Taking an 8×8 text block as an example (mean Y = 120×256, mean Cb = 180×256, Cov(Y,Cb) = -1200×256², Var(Y) = 4000×256²): Fixed-point number conversion: α_Cb=Cov(Y,Cb) / Var(Y)=(-1200×256²) / (4000×256²)=-0.3; β_Cb=μ_Cb / 256-α_Cb×(μ_Y / 256)=180-(-0.3)×120=216; Storage format: α and β are stored as 32-bit floating-point numbers. A mapping table is established: param_map[block_index]=(α_Cb,β_Cb,α_Cr,β_Cr). Blocks that are not matched do not store parameters.

[0174] Step 306: Parameter compression and SEI encapsulation.

[0175] Among them, thread 13 (encapsulated thread) can be used for parameter compression and SEI encapsulation.

[0176] Specifically, the first step is quantization, where the floating-point numbers α and β are multiplied by 1. Round down to a 16-bit integer (α range: -10~10, β range: 0~255, 16-bit integers can completely cover it), for example α=-0.3→-300000, β=216→21600000.

[0177] Then, based on the preset Huffman code table (optimized for the distribution of α and β quantization values), the combined data of "macroblock index (32 bits) + α_Cb (16 bits) + β_Cb (16 bits) + α_Cr (16 bits) + β_Cr (16 bits)" can be encoded, with a compression ratio of about 4:1 (original 32 bytes / block → compressed 8 bytes / block).

[0178] Next, SEI encapsulation is performed, which can be done according to the H.265 SEI standard to obtain the SEI message. The custom SEI payload type is 0x80, and the SEI message structure is shown in Table 1 below: Table 1. SEI Messages

[0179] Finally, the binding process is performed, and the SEI message is stored in a memory buffer and marked as "auxiliary data". It will then be passed to the encoder along with the video frame data.

[0180] Step 307: Normal downsampling and video encoding.

[0181] Among them, ordinary downsampling and video encoding processing can be performed through thread 14 (encoding thread).

[0182] Specifically, ordinary downsampling can be implemented in software: perform 2×2 neighborhood averaging downsampling on the Cb and Cr components of YUV444, and accelerate the process by using a sliding window summation. The specific steps are as follows: First, for each 2×2 window, accumulate 4 Cb pixel values ​​(fixed number of points), shift right by 2 bits (equivalent to dividing by 4) to obtain the average value, and generate NV12 format data (Y: 1920×1080, CbCr: 960×540, interleaved storage).

[0183] Then, the x265 encoder is invoked to encode the video. The configuration parameters include: bitrate control: CBR 8Mbps, I-frame interval 30 frames, and P-frames using multi-reference frame prediction (reference frame number = 4).

[0184] Finally, the encapsulated SEI message is appended to the corresponding video frame through the x265_sei_message interface of x265; and finally, it is encoded and output to generate an H.265 bitstream.

[0185] Steps 301 to 307 above constitute the core processing flow of the encoding end: "BGRA input → preprocessing → YUV conversion → Block processing → parameter calculation → SEI encapsulation → encoding output". Furthermore, the encoding segment can be processed in parallel using multi-threading (e.g., frame-level + block-level parallelism), which improves data processing efficiency.

[0186] Next, we will continue to introduce the image data processing flow with the decoding end as the main execution unit. The processing process of the decoding end is the reverse process of the processing process of the encoding end.

[0187] Step 311: Bit stream reception and separation.

[0188] Among them, thread 21 (bit stream parsing thread) can be used for bit stream reception and separation.

[0189] Specifically, the av_read_frame interface of FFmpeg can be called to separate the video frame data (AVPacket) and SEI message (SEI data of custom payload type 0x80 is extracted through av_packet_get_side_data) and transmit them to thread 22 (decoding thread) and thread 23 (parsing thread) respectively.

[0190] Step 312: Video Decoding and SEI Analysis.

[0191] Among them, thread 22 (decoding thread) and thread 23 (parsing thread) can be used for processing.

[0192] Specifically, the FFmpeg H.265 decoder (avcodec_send_packet + avcodec_receive_frame) can be called to decode the video frame data and output the raw NV12 format frame (Y: 1920×1080, CbCr: 960×540); then SEI parsing is performed. Specifically, first, the frame synchronization identifier (PTS value) in the SEI message is extracted and aligned with the PTS of the decoded video frame to avoid frame out-of-order issues; furthermore, the Huffman-coded parameter stream can be soft-decompressed to restore the macroblock index and the quantized α and β values, then divided by 1. Obtain the floating-point number; finally, construct the mapping table and establish a hash table (unordered_map4_t, ParamStruct>) for "macroblock index → ​​α_Cb, β_Cb, α_Cr, β_Cr", with a query time complexity of O(1).

[0193] Step 313: Normal upsampling and Block matching.

[0194] Among them, thread 24 (upsampling thread) and thread 25 (matching thread) can be used for processing.

[0195] Specifically, thread 24 is used for normal upsampling, where bilinear interpolation can be used for upsampling, accelerated by SIMD. Specifically, software bilinear interpolation upsampling is performed on the Cb and Cr components of NV12 to restore them to YUV444 format (basic version). The interpolation formula is: Cb(x,y)=[Cb(x / 2,y / 2)+Cb(x / 2+1,y / 2)+Cb(x / 2,y / 2+1)+Cb(x / 2+1,y / 2+1)] / 4.

[0196] Then, the interpolation results of the four pixels are calculated in parallel using the SSE4.1 instruction set to improve processing efficiency.

[0197] Block matching is performed using 25 threads per process. Specifically, the base version YUV444 is divided according to the same rules (8×8 pixels) at the encoding end. The macroblock index of each block is traversed, and the hash table is queried: if the index matches, it is marked as a "high-relevance, high-frequency block" and the corresponding parameters are recorded; if it does not match, it is marked as a "low-relevance block", and the upsampling result of the base version is retained.

[0198] Step 314: Perform high-quality color reproduction.

[0199] Among them, threads 26-30 (recovery threads) can be used to recover multiple highly correlated and high-frequency blocks in parallel, and thread 25 (region processing thread) can be used to process multiple highly correlated and high-frequency blocks in a partition.

[0200] Specifically, thread 25 (regional processing thread) is used to partition and process multiple highly correlated, high-frequency blocks. Consistent with the encoding end, five regional processing threads process blocks in different regions, recovering high-quality Cb and Cr based on luminance Y and linear parameters.

[0201] Specifically, the 16-bit fixed-point value of luminance Y is converted to a floating-point number (divided by 256) to ensure it matches the calculation precision of the α and β parameters; then, the restoration formula is used for completion and calculation; among them, the high-quality Cb / Cr restoration formula is: Cb_rec=α_Cb×(Y / 256)+β_Cb; Cr_rec=α_Cr×(Y / 256)+β_Cr.

[0202] Taking the Block example from the encoding end in the above steps as an example (Y fixed point number = 150 × 256 = 38400, α_Cb = -0.3, β_Cb = 216), the high-quality chroma restoration at the decoding end is as follows: Cb_rec=-0.3×(38400 / 256)+216=-0.3×150+216=171, the error between Cb and the original Cb value is ≤1 gray level.

[0203] Then, the results are normalized: the recovered Cb_rec and Cr_rec are limited to the range of 0-255 by software logic (set to 0 if below 0, set to 255 if above 255) to avoid color distortion caused by overflow; Finally, the data is updated by overwriting the corresponding positions in the base version YUV444 buffer with the high-quality recovered Cb / Cr values, while the low-correlation blocks retain the original upsampling results to ensure compatibility in non-high-frequency regions.

[0204] Step 315: Perform BGRA synthesis and output.

[0205] Specifically, the process is the reverse of the encoding process, requiring inverse color space conversion (AVX2 acceleration) to convert YUV444 data to RGB format. Specifically, this can be achieved using the ITU-R BT.601 inverse matrix and the AVX2 vector instruction set for parallel conversion of YUV444 to RGB format. The specific formula (rounded after floating-point operations) is as follows: R = (Y / 256) + 1.13983 × Cr_rec; G=(Y / 256)-0.39465×Cb_rec-0.58060×Cr_rec; B = (Y / 256) + 2.03211 × Cb_rec; Alpha=255 (fixed opacity); Then, the conversion result is optimized for precision. During the conversion process, 10 bits of intermediate precision are retained, and finally, the result is rounded to an 8-bit pixel value (0-255) to reduce quantization error.

[0206] Finally, the display output is performed, and the BGRA format data is rendered to the display terminal through the DirectX12 interface (supporting multi-monitor expansion). The rendering refresh rate is synchronized with the original frame rate (30fps), without screen tearing or stuttering.

[0207] In addition, this disclosure can also handle anomalies. If a frame parameter fails to be parsed or the block matching rate is less than 50%, it will automatically switch to the full-frame normal upsampling mode to ensure display stability.

[0208] Steps 311 to 315 above constitute the core process of the decoding end: "bit stream input → separation → decoding → upsampling → block matching → high-quality recovery → output". This can be the reverse process of the encoding end processing flow.

[0209] It should be noted that the processing procedures at the encoding end and the decoding end can also exist independently, and this disclosure does not limit this.

[0210] The data processing method disclosed herein, through its core design of "multi-threaded parallelism + SIMD acceleration + image block-level differential processing + SEI standard encapsulation," combined with the strong correlation characteristics between pixel luminance components, achieves multi-dimensional technical breakthroughs compared to existing image data processing methods. Specific beneficial effects are as follows: Firstly, the accuracy of high-frequency detail color reproduction has been significantly improved.

[0211] This solution divides the image into blocks and precisely selects target image blocks prone to chromatic distortion and exhibiting linear relationships. This ensures a high degree of matching between the linear fitting parameters and the local chromaticity variation patterns. The linear correlation parameters between the luminance and chromaticity components are calculated only for high-frequency areas prone to chromatic distortion, avoiding invalid calculations in low-correlation areas. Furthermore, the chromaticity component is accurately derived from the linear correlation parameters and the luminance component during the restoration phase, completely resolving the pain point of existing technologies that "dilute abrupt changes through average downsampling." Real-world testing shows that the chromaticity error at text edges remains stable at a low grayscale level (generally between 0.9 and 1.7), the chromaticity loss rate of fine lines is significantly reduced to only 1.8%, and the sampling conversion signal-to-noise ratio is significantly improved to 52.3-54.7 dB, an improvement of ≥12 dB compared to existing solutions (signal-to-noise ratio of 36.8-39.5 dB). This allows for clear restoration of core details such as desktop text, table lines, and icon outlines.

[0212] Secondly, CPU software encoding efficiency has been optimized to meet real-time transmission requirements.

[0213] To address the pain point of limited CPU software encoding capabilities, this solution adopts a layered parallel architecture: the encoding end uses multiple threads to handle "data reading, format conversion, block analysis, parameter calculation, and encoding output," while the decoding end uses multiple threads in parallel to complete "receive separation, decoding parsing, upsampling matching, and output restoration." Combined with SIMD instruction set acceleration (AVX2 / SSE4.2), the parallelism of core operations (YUV conversion, gradient calculation, and interpolation) is increased by 8 times. The final single-frame processing latency at the encoding end is 6.3-7.8ms, and at the decoding end it is 4.1-5.6ms, with a total end-to-end latency of ≤35ms. This represents a reduction of over 50% compared to existing CPU software encoding solutions (total latency 23.7-28.1ms), fully meeting the real-time requirements of desktop projection (≤100ms) and remote work (≤50ms).

[0214] Thirdly, the solution disclosed herein has strong compatibility, high portability, and is adapted to mainstream terminal environments.

[0215] This solution is developed based on the open-source standard framework (x265 encoder, FFmpeg decoder) and has no dedicated hardware dependency: the encoding end supports mainstream Intel / AMD CPUs (from i7-12700H to i9-13900K), and the decoding end is compatible with consumer PCs, ultrabooks, and other terminals; the software environment is adapted to 64-bit systems such as Windows 11 / 10 and Linux, and the development language is standard C / C++, which can be compiled by mainstream compilers such as MSVC and GCC. Furthermore, the solution supports dynamic parameter configuration (block size, number of threads, correlation coefficient threshold), allowing low-performance CPUs to downgrade to 16×16 Block + SSE4.2 instruction set operation, while high-performance CPUs can enable AVX2 + 32 threads for full-load operation. It adapts to different terminal computing power without code refactoring, and its portability far exceeds that of solutions relying on dedicated hardware.

[0216] Fourthly, it has low bandwidth overhead and is compatible with existing transmission links.

[0217] This solution employs SEI standard encapsulation and Huffman compression: it transmits only the α / β parameters of highly correlated blocks (each block is compressed to only 8 bytes, with a total parameter size of ≤80KB per frame). Compared to directly transmitting YUV444 format (adding 12.4MB of data per frame), the bandwidth overhead increases by ≤4.8%, and it is fully compatible with existing 5G and wired network YUV420 transmission links. SEI messages are transmitted bound to video frames, following the H.265 standard protocol. No modifications are required to existing encoder interfaces, transmission protocols, or receiver decoding logic, allowing for seamless integration into existing desktop projection and remote desktop systems with extremely low upgrade costs.

[0218] Fifthly, CPU resource usage is optimized to balance performance and power consumption.

[0219] This solution reduces redundant overhead through a differentiated computation strategy: parameter calculations are performed only on 1 / 3 to 1 / 4 of the highly correlated, high-frequency blocks within a frame (8000-10000 blocks / frame), while low-correlation regions retain ordinary sampling methods, avoiding complex calculations across the entire frame. Simultaneously, optimizations such as memory alignment (64-byte alignment) and sliding window summation improve CPU cache hit rate. On an i9-13900K, the CPU utilization per frame is only 22%-28%, a 30% reduction compared to existing CPU-based software encoding solutions (35%-42%). When adapted to low-power terminals such as thin and light laptops, it can simultaneously handle encoding and desktop operations without noticeable lag or power spikes, achieving a balance between "high precision + low resources."

[0220] Sixthly, it is highly robust and adaptable to complex scenarios and abnormal operating conditions.

[0221] This solution features a dual fault-tolerance mechanism: ① At the parameter transmission level, Huffman coding performs entropy encoding on the quantized α / β values, providing strong resistance to network packet loss. A small amount of parameter loss only affects the local image quality of the corresponding block and does not spread to the entire frame; ② At the decoding adaptation level, if SEI parameter parsing fails or the block matching rate is below 50%, it automatically switches to full-frame normal upsampling mode to ensure stable image output. Simultaneously, the solution supports adaptive processing for different desktop scenarios (text-intensive documents, multi-icon desktops, color gradient backgrounds) without requiring manual parameter adjustment, exhibiting robustness far exceeding traditional solutions that rely on fixed interpolation.

[0222] Based on the same inventive concept as the above-described solutions, this disclosure also provides an image data processing system.

[0223] Figure 4 This is a structural block diagram of an image data processing system provided in an embodiment of the present disclosure.

[0224] Further reference Figure 4 As an implementation of the methods shown in the above figures, the system embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0225] See Figure 4 The system includes an image transmitting end 410, which includes a segmentation module 411, a calculation module 412, and an encoding module 413. The segmentation module 411 divides a video frame image into multiple image blocks; the calculation module 412 determines a target image block from the multiple image blocks and calculates the linear relationship parameters between the chromaticity components and luminance components of the pixels within the target image block; the encoding module 413 obtains supplementary information of the target image block based on the linear relationship parameters, and encodes the supplementary information along with the multiple image blocks of the video frame image before sending it to the image receiving end.

[0226] In this embodiment, the specific processing of the partitioning module 411, the calculation module 412, and the encoding module 413 in the image data processing system, and the resulting technical effects, can be referred to respectively. Figure 1 The relevant descriptions of steps 101-103 in the corresponding embodiments will not be repeated here.

[0227] In some embodiments, the calculation module 412 is specifically used to determine at least one compression-sensitive image block from multiple image blocks based on the difference in chromaticity components of pixels within multiple image blocks; calculate the linearity coefficient between the luminance component and the chromaticity component of pixels within the compression-sensitive image block; and determine the compression-sensitive image block with a linearity coefficient greater than a first threshold as the target image block.

[0228] In some embodiments, the calculation module 412 is further configured to: for each image block, calculate the chromaticity component difference of pixels within the image block based on the adjacent pixels within the image block, and determine a target pixel from multiple pixels based on the chromaticity component difference of multiple pixels within the image block; and determine at least one compression-sensitive image block from multiple image blocks based on the number of target pixels within multiple image blocks.

[0229] In some embodiments, the video frame image includes multiple image regions, and each image region includes at least one image block; the calculation module 412 is further configured to: use a thread-level parallel processing method to determine the compression-sensitive image blocks in the image blocks in different image regions in parallel according to the chromaticity component difference of pixels in the image blocks in different image regions.

[0230] In some embodiments, the calculation module 412 is further configured to: obtain the variance of the luminance component, the variance of the chrominance component, and the covariance of the luminance component and the chrominance component of the pixels within the compression-sensitive image block based on the luminance component and the chrominance component of the pixels within the compression-sensitive image block; and obtain the linearity coefficient of the compression-sensitive image block based on the variance of the luminance component, the variance of the chrominance component, and the covariance of the luminance component and the chrominance component of the pixels within the compression-sensitive image block.

[0231] In some embodiments, the calculation module 412 is further configured to: obtain the linear relationship parameters of the chromaticity component relative to the luminance component of the pixels in the target image block based on the variance of the luminance component, the mean of the luminance component, the mean of the chromaticity component, and the covariance between the luminance component and the chromaticity component of the pixels in the target image block.

[0232] In some embodiments, the encoding module 413 is specifically used to: compress and encapsulate the linear relationship parameters to obtain supplementary information of the target image block; downsample and compress the chroma components of pixels in each image block of a video frame image to obtain compressed chroma components, and generate compressed image data of the video frame image based on the compressed chroma components of each image block and the luminance components of pixels; during the encoding process of the compressed image data, insert the supplementary information into the encoded bitstream corresponding to the compressed image data, and send the encoded bitstream containing the compressed image data and supplementary information to the image receiving end.

[0233] In some embodiments, the encoding module 413 is further configured to: quantize the linear relationship parameters of the target image block to obtain quantized parameters; compress the index of the target image block and the quantized parameters to obtain compressed parameters of the target image block; and encapsulate the compressed parameters of the target image block with the display timestamp corresponding to the video frame image to obtain supplementary information.

[0234] In some embodiments, the video frame image includes multiple target image blocks, wherein one target image block corresponds to one compressed parameter; the encoding module 413 is further configured to: determine the length of the compressed parameter according to the number of multiple target image blocks; and encapsulate the compressed parameter of the target image block, the length of the compressed parameter, and the display timestamp corresponding to the video frame image to obtain supplementary information.

[0235] In some embodiments, the number of video frame images is multiple, and the partitioning module 411 is further configured to: generate an index for multiple image blocks based on the frame number of the video frame image corresponding to the multiple image blocks, the row number and column number of the image block in the corresponding video frame image.

[0236] The image processing system provided in this disclosure divides video frame images into blocks, allowing for processing of video frame images on a block-by-block basis. This enables differentiated processing methods for different image blocks, improving the accuracy of video frame image processing. Furthermore, the system determines the linear relationship parameter between the chroma components and luminance components of pixels within a target image block, and encodes this parameter as supplementary information along with multiple image blocks before transmitting it to the image receiving end. This allows the image receiving end to directly recover the chroma components of the target image block based on the linear relationship between the luminance and chroma components using the received linear relationship parameter. The system's recovery of the chroma components using linear relationship parameters reduces chroma component errors during image data transmission, thereby reducing chroma component distortion. Additionally, the system's derivation of chroma components from luminance components improves the signal-to-noise ratio of the video frame image, further enhancing the image quality at the image receiving end.

[0237] Figure 5 This is a structural block diagram of another image processing system provided in an embodiment of the present disclosure.

[0238] See Figure 5 In some embodiments, the system includes not only an image transmitting end 410 but also an image receiving end 420. The image receiving end 420 includes a processing module 421 and a restoration module 423. The processing module 421 processes the received encoded bitstream to obtain video frame data and supplementary information. The restoration module 423 restores the video frame data based on the supplementary information to obtain a video frame image. The supplementary information includes linear relationship parameters between the chroma components and luminance components of pixels within a target image block, and is used to restore the chroma components in the target image block.

[0239] The image data processing system provided in this embodiment of the present disclosure includes not only video frame data of each image block in the video frame image, but also supplementary information of the target image block in which the luminance component and chrominance component have a linear relationship within the pixel. Therefore, the image receiving end can correct the chrominance component in the video frame data according to the supplementary information of the target image block, thereby improving the image quality of the image at the image receiving end and enabling the image receiving end to more accurately reproduce the video frame image.

[0240] In some embodiments, the restoration module 423 is specifically used to decode and upsample the video frame data to obtain decompressed image data; wherein, the decompressed image data includes the luminance components and decompressed chrominance components of pixels in multiple image blocks, and the multiple image blocks include target image blocks and non-target image blocks other than the target image blocks; based on supplementary information, the decompressed chrominance components of the target image blocks are updated to obtain updated target image blocks; non-target image blocks are obtained according to the luminance components and decompressed chrominance components of pixels in the non-target image blocks; and video frame images are obtained according to the updated target image blocks and non-target image blocks.

[0241] In some embodiments, the restoration module 423 is further configured to: decompress the supplementary information to obtain the index of the target image block and the linear relationship parameters of the chromaticity components of the pixels in the target image block relative to the luminance components; and update the decompressed chromaticity components of the target image block according to the index of the target image block and the linear relationship parameters to obtain the updated target image block.

[0242] In some embodiments, the restoration module 423 is further configured to: determine the luminance component and decompressed chrominance component of the pixels in the target image block according to the index of the target image block; calculate the chrominance information of the pixels in the target image block according to the linear relationship parameter of the target image block and the luminance component of the pixels in the target image block; replace the chrominance component of the target image block with the chrominance information of the pixels in the target image block to obtain the updated target image block.

[0243] Other specific implementation details and their technical effects of the image data processing system embodiments provided in this disclosure are the same as the implementation details and their technical effects of the image data processing method embodiments described above, and will not be repeated here.

[0244] This embodiment exists as a device embodiment corresponding to the above method embodiment. The image data processing system provided in this embodiment, through the core design of "multi-threaded parallelism + SIMD acceleration + image block-level differential processing + SEI standard packaging", combined with the strong correlation characteristics of pixel luminance components, achieves a leapfrog improvement in the accuracy of high-frequency detail color reproduction.

[0245] Specifically, the image data processing system provided in this disclosure, through image block division and precise screening of target image blocks prone to chromatic distortion and exhibiting linear relationships, ensures a high degree of matching between linear fitting parameters and local chromaticity variation patterns. It calculates the luminance-chromaticity component linear correlation parameters only for high-frequency areas prone to chromatic distortion, avoiding invalid calculations in low-correlation areas. Furthermore, in the restoration stage, the chromaticity component is accurately derived using the linear correlation parameters and luminance component, completely resolving the pain point of existing technologies where "average downsampling dilutes abrupt change information." Actual measurements show that the chromaticity error at text edges remains stable at a low grayscale level (generally between 0.9 and 1.7), the chromaticity loss rate of fine lines is significantly reduced to only 1.8%, and the sampling conversion signal-to-noise ratio is significantly improved to 52.3-54.7 dB, an improvement of ≥12 dB compared to existing solutions (signal-to-noise ratio of 36.8-39.5 dB), clearly restoring core details such as desktop text, table lines, and icon outlines.

[0246] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the image data processing method described in any of the above embodiments.

[0247] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 600 of this embodiment includes a processor 601 and a memory 602; wherein, the memory 602 is used to store computer execution instructions; the processor 601 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. For example, the electronic device 600 can be a general-purpose processor, a graphics processing device, a neural network computing device, or a graph neural network computing device.

[0248] In some embodiments, the memory 602 can be either standalone or integrated with the processor 601.

[0249] When the memory 602 is set up independently, the electronic device also includes a bus 603 for connecting the memory 602 and the processor 601.

[0250] It should be understood that the processor 601 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0251] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0252] Bus 603 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0253] This disclosure also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the steps of the image data processing method of any of the above method embodiments.

[0254] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image data processing method according to any of the above embodiments.

[0255] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0256] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0257] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0258] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods in the various embodiments of this application.

[0259] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0260] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0261] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0262] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0263] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0264] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image data processing method, characterized in that, Applied to an image transmitting end, the method includes: Divide the video frame image into multiple image blocks; A target image block is determined from the plurality of image blocks, and the linear relationship parameters of the chromaticity components of the pixels in the target image block relative to the luminance components are calculated. Based on the linear relationship parameters, supplementary information of the target image block is obtained, and the supplementary information and the multiple image blocks are encoded together and sent to the image receiving end.

2. The method according to claim 1, characterized in that, Determining the target image block from the plurality of image blocks includes: Based on the chromaticity component differences of pixels within the plurality of image blocks, at least one compression-sensitive image block is determined from the plurality of image blocks; Calculate the linearity coefficients of the luminance and chrominance components of the pixels within the compression-sensitive image block, and determine the compression-sensitive image blocks whose linearity coefficients are greater than a first threshold as target image blocks.

3. The method according to claim 2, characterized in that, The step of determining at least one compression-sensitive image block from the plurality of image blocks based on the chromaticity component differences of pixels within the plurality of image blocks includes: For each image block, the chromaticity component difference of the pixels in the image block is calculated based on the adjacent pixels of the pixels in the image block, and the target pixel is determined from the multiple pixels based on the chromaticity component difference of the multiple pixels in the image block. Based on the number of target pixels within the plurality of image blocks, at least one compression-sensitive image block is determined from the plurality of image blocks.

4. The method according to claim 2, characterized in that, The video frame image includes multiple image regions, and each image region includes at least one image block; The step of determining at least one compression-sensitive image block from the plurality of image blocks based on the chromaticity component differences of pixels within the plurality of image blocks includes: A thread-level parallel processing method is adopted to determine the compression-sensitive image blocks in different image regions in parallel based on the difference in chromaticity components of pixels within image blocks in different image regions.

5. The method according to claim 2, characterized in that, The calculation of the linearity coefficients of the luminance and chrominance components of pixels within the compressed sensitive image block includes: Based on the luminance and chrominance components of the pixels within the compression-sensitive image block, the variance of the luminance component, the variance of the chrominance component, and the covariance between the luminance and chrominance components of the pixels within the compression-sensitive image block are obtained. The linearity coefficient of the compression-sensitive image block is obtained based on the variance of the luminance component, the variance of the chrominance component, and the covariance between the luminance and chrominance components of the pixels within the compression-sensitive image block.

6. The method according to any one of claims 1 to 5, characterized in that, The calculation of the linear relationship parameters between the chromaticity components and the luminance components of pixels within the target image block includes: Based on the variance of the luminance component, the mean of the luminance component, the mean of the chrominance component, and the covariance between the luminance and chrominance components of the pixels within the target image block, the linear relationship parameters of the chrominance component relative to the luminance component of the pixels within the target image block are obtained.

7. The method according to claim 1, characterized in that, The step of obtaining supplementary information of the target image block based on the linear relationship parameter, and encoding the supplementary information together with multiple image blocks of the video frame image before sending it to the image receiving end includes: The linear relationship parameters are compressed and encapsulated to obtain supplementary information of the target image patch; The chroma components of pixels in each of the multiple image blocks of the video frame image are downsampled and compressed to obtain compressed chroma components. Based on the compressed chroma components of each image block and the luminance components of the pixels, compressed image data of the video frame image is generated. During the encoding process of the compressed image data, the supplementary information is inserted into the encoded bitstream corresponding to the compressed image data, and the bitstream containing the compressed image data and the supplementary information is sent to the image receiving end.

8. The method according to claim 7, characterized in that, The step of compressing and encapsulating the linear relationship parameters to obtain supplementary information for the target image patch includes: The linear relationship parameters of the target image patch are quantized to obtain quantized parameters; The index of the target image block is compressed with the quantized parameters to obtain the compressed parameters of the target image block; The supplementary information is obtained by encapsulating the compressed parameters of the target image block with the display timestamp corresponding to the video frame image.

9. The method according to claim 8, characterized in that, The video frame image includes multiple target image blocks, wherein each target image block corresponds to a compressed parameter; The supplementary information is obtained by encapsulating the compressed parameters of the target image block with the display timestamp corresponding to the video frame image, including: The length of the compressed parameters is determined based on the number of the plurality of target image blocks; The supplementary information is obtained by encapsulating the compressed parameters of the target image block, the length of the compressed parameters, and the display timestamp corresponding to the video frame image.

10. The method according to claim 8, characterized in that, The number of video frame images is multiple. After dividing the video frame images into multiple image blocks, the method further includes: An index is generated for each of the image blocks based on the frame number of the video frame image corresponding to the image block, the row number of the image block in the corresponding video frame image, and the column number of the image block.

11. An image data processing method, characterized in that, Applied to an image receiving end, the method includes: The received encoded bitstream is processed to obtain video frame data and supplementary information; Based on the supplementary information, the video frame data is restored to obtain a video frame image; wherein, the supplementary information includes a linear relationship parameter between the chroma component and the luminance component of the pixels in the target image block, and the supplementary information is used to restore the chroma component in the target image block.

12. The method according to claim 11, characterized in that, The step of restoring the video frame data based on the supplementary information to obtain a video frame image includes: The video frame data is decoded and upsampled to obtain decompressed image data; wherein, the decompressed image data includes the luminance components and decompressed chrominance components of pixels in multiple image blocks, and the multiple image blocks include the target image block and non-target image blocks other than the target image block; Based on the supplementary information, the decompressed chroma components of the target image block are updated to obtain the updated target image block; The non-target image block is obtained based on the luminance component and the decompressed chrominance component of the pixels within the non-target image block. Based on the updated target image block and the non-target image block, a video frame image is obtained.

13. The method according to claim 12, characterized in that, The step of updating the decompressed chroma components of the target image block based on the supplementary information to obtain the updated target image block includes: The supplementary information is decompressed to obtain the index of the target image block and the linear relationship parameters of the chromaticity component of the pixels in the target image block relative to the luminance component. Based on the index of the target image block and the linear relationship parameter, the decompressed chroma components of the target image block are updated to obtain the updated target image block.

14. The method according to claim 13, characterized in that, The step of updating the decompressed chromaticity brightness of the target image block according to the index of the target image block and the linear relationship parameter to obtain the updated target image block includes: Based on the index of the target image block, determine the luminance component and decompressed chrominance component of the pixels within the target image block; The chromaticity information of the pixels in the target image block is calculated based on the linear relationship parameters of the target image block and the luminance components of the pixels in the target image block. The updated target image block is obtained by replacing the chromaticity components of the target image block with the chromaticity information of the pixels within the target image block.

15. An image data processing system, characterized in that, The system includes an image transmitting end, which includes: The segmentation module is used to divide video frame images into multiple image blocks; The calculation module is used to determine a target image block from the plurality of image blocks and calculate the linear relationship parameters of the chromaticity component relative to the luminance component of the pixels in the target image block; The encoding module is used to obtain supplementary information of the target image block according to the linear relationship parameter, and encode the supplementary information together with multiple image blocks of the video frame image and send them to the image receiving end.

16. The system according to claim 15, characterized in that, It also includes an image receiver, which includes: The separation module is used to process the received encoded bitstream to obtain video frame data and supplementary information; The restoration module is used to restore the video frame data based on the supplementary information to obtain a video frame image; wherein, the supplementary information includes linear relationship parameters of the chroma components of pixels within the target image block relative to the luminance components, and the supplementary information is used to restore the chroma components in the target image block.

17. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image data processing method according to any one of claims 1-14.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the image data processing method according to any one of claims 1-14.

19. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the steps of the image data processing method according to any one of claims 1-14.

Images