Image data processing method, system, device, medium and product

Abstract

The disclosure provides an image data processing method, system, device, medium and product, and relates to the technical field of image processing. The method comprises the following steps: a CPU calls a graphic processor (GPU) to process color space information in initial image data, and obtains supplementary information of the initial image data; the CPU encodes the supplementary information of the initial image data, generates a supplementary information code stream of the initial image data, and calls the GPU to encode video frame image data, and generates a video frame code stream of the initial image data; wherein the video frame image data is obtained by downsampling the initial image data; and the CPU outputs an encoding code stream of the initial image data based on the supplementary information code stream and the video frame code stream of the initial image data. The method improves the efficiency and accuracy of data processing.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image data processing method, system, device, medium, and product. Background Technology

[0002] In scenarios such as remote desktops, video conferencing, and live streaming, traditional solutions rely on the central processing unit (CPU) to serially complete image encoding and data transmission, which results in low computing power utilization and insufficient overall computing efficiency. On the other hand, when using a graphics processing unit (GPU) to perform image encoding and transmission, it can only complete basic color conversion operations, limiting its encoding processing capabilities. Summary of the Invention

[0003] This disclosure provides an image data processing method, system, device, medium, and product.

[0004] In a first aspect, embodiments of this disclosure propose an image data processing method applied to a CPU in a terminal device. The method includes: the CPU calling a graphics processing unit (GPU) to process color space information in initial image data to obtain supplementary information of the initial image data, wherein the supplementary information is used to identify the linear relationship between chroma components and luminance components in the initial image data; the CPU encoding the supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data, and calling the GPU to encode video frame image data to generate a video frame bitstream of the initial image data; wherein the video frame image data is obtained by downsampling the initial image data; and the CPU outputting an encoded bitstream of the initial image data based on the supplementary information bitstream and the video frame bitstream.

[0005] Secondly, this disclosure also provides an image data processing method applied to a CPU in a terminal device. The method includes: the CPU receiving an encoded bitstream, wherein the encoded bitstream includes a video frame bitstream and a supplementary information bitstream; the CPU decoding the supplementary information bitstream to obtain supplementary information; and calling a GPU to decode the video frame bitstream to obtain video frame image data; the CPU calling the GPU to restore the video frame image data based on the supplementary information to obtain initial image data corresponding to the encoded bitstream, wherein the supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the initial image data.

[0006] Thirdly, this disclosure also provides an image data processing method applied to a GPU. The method includes: the GPU processing color space information in video frame image data to obtain supplementary information of initial image data, wherein the video frame image data is obtained by downsampling the initial image data, and the supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the initial image data; the GPU encoding the video frame image data to generate a video frame bitstream of the video frame image data; and the GPU sending the video bitstream of the video frame image data to a central processing unit (CPU), so that after the CPU encodes the supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data, it outputs the encoded bitstream of the initial image data based on the supplementary information bitstream and the video frame bitstream.

[0007] Fourthly, this disclosure also provides an image data processing method applied to a GPU. The method includes: the GPU receiving an encoded bitstream from a central processing unit (CPU), wherein the encoded bitstream includes a video frame bitstream and a supplementary information bitstream; the GPU decoding the video frame bitstream to obtain video frame image data; and the GPU restoring the video frame image data based on the supplementary information to obtain initial image data corresponding to the encoded bitstream, wherein the supplementary information is obtained by the CPU decoding the supplementary information bitstream, and the supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the initial image data, so as to indicate the restoration of the chroma component in the video frame image data.

[0008] Fifthly, embodiments of this disclosure also provide an image data processing system, the system including a central processing unit (CPU) and a graphics processing unit (GPU), wherein the GPU is used to process color space information in initial image data to obtain supplementary information of the initial image data, wherein the supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the initial image data; the CPU is used to encode the supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data; the GPU is also used to encode video frame image data to generate a video frame bitstream of the initial image data; wherein the video frame image data is obtained by downsampling the initial image data; the CPU is also used to output an encoded bitstream of the initial image data based on the supplementary information bitstream and the video frame bitstream of the initial image data.

[0009] In a sixth aspect, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the image data processing method described in any of the implementations of the first to fourth aspects.

[0010] In a seventh aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer, when executed, to implement the image data processing method described in any of the implementations of the first to fourth aspects.

[0011] Eighthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the image data processing method as described in any of the first to fourth aspects.

[0012] The image data processing method disclosed herein adopts a CPU and GPU collaborative hybrid processing architecture. The computational task of obtaining supplementary information from the initial image data is scheduled to run on the GPU, fully leveraging the powerful parallel computing power of the GPU and significantly improving the processing throughput efficiency of the basic image data. Furthermore, the encoding of supplementary information with low computational load is handled by the CPU, while the encoding of downsampling blocks with higher computational load is allocated to the GPU for parallel processing. The two types of encoding tasks can be carried out simultaneously and in parallel, further compressing the overall encoding time. In addition, the logical management of bitstream pairing and association and output is uniformly handled by the CPU. The bitstream matching and forwarding can be accurately completed based on the CPU's scheduling and peripheral interaction capabilities, avoiding the latency loss caused by frequent data transfer between multiple devices. Attached Figure Description

[0013] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart of an image data processing method provided in this disclosure embodiment; Figure 2 A flowchart illustrating yet another image data processing method provided in this disclosure embodiment; Figures 3a-3b The following are flowcharts of another image data processing method provided in this disclosure; Figure 4 This is a structural block diagram of an image data processing system provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0015] In a first aspect, this disclosure provides an image data processing method that can be applied to scenarios where there is a linear correlation between the luminance component and the chrominance component in the image data, including but not limited to desktop scenarios such as remote desktop, video conferencing, and live streaming.

[0016] Figure 1 The flowchart of an image data processing method provided in this embodiment of the present disclosure is applied to a CPU. Specifically, the image data processing method involves the CPU performing an image encoding operation and then sending the image data.

[0017] See Figure 1 The process specifically includes the following steps: Step 101: The CPU calls the graphics processing unit (GPU) to process the color space information in the initial image data to obtain supplementary information of the initial image data.

[0018] The supplementary information is used to identify the linear relationship between the chromaticity and luminance components in the initial image data.

[0019] Specifically, the initial image data refers to image data in YUV (Luminance(Y)-Chrominance(U,V)) format. For example, the initial image data can be image data in YUV444 format. Color space information refers to the parameter information associated with the luminance and chrominance components in the video frame image data.

[0020] In applications such as remote desktops, the initial image data includes image data in which the chromaticity components and luminance components have a linear correspondence. Therefore, this disclosure can also process the color space information in the initial image data to determine the linear correlation between the chromaticity components and luminance components in the initial image data, thereby obtaining supplementary information about the initial image data.

[0021] In the image data encoding process, the CPU can act as a logical control center, delegating some computational operations in the image encoding process to the GPU for collaborative execution. The CPU handles core management tasks such as overall process scheduling, logical control, and task distribution. If additional calculations of the linear relationship between chroma and luminance components and extraction of supplementary information are performed on the CPU, it will significantly consume CPU computing resources and slow down the overall encoding sequence. In contrast, the GPU possesses the advantage of parallel batch processing, allowing massive amounts of image data to be simultaneously processed for linear relationship fitting without consuming CPU main control computing power. Therefore, in this disclosure, the CPU can schedule the process of obtaining supplementary information from the initial image data to the GPU for execution, fully utilizing the GPU's computational advantages and improving the image data processing method.

[0022] Step 102: The CPU encodes the supplementary information of the initial image data to generate the supplementary information bitstream of the initial image data, and calls the GPU to encode the video frame image data to generate the video frame bitstream of the initial image data.

[0023] The video frame image data is obtained by downsampling the initial image data.

[0024] For example, the initial image data can be YUV444 format image data, and the video frame image data can be YUV420 format image data.

[0025] Specifically, the supplementary information bitstream refers to the bitstream obtained by encoding supplementary information; the video frame bitstream refers to the bitstream obtained by encoding video frame image data.

[0026] Optionally, the video frame stream can be H.264 / H.265 video data. H.264 and H.265 are two internationally recognized generations of video coding standards. The corresponding video data refers to the standardized compressed stream generated after processing according to the compression rules specified by these standards, such as intra / inter-frame prediction, transform quantization, and entropy coding. These are also the most commonly used video transmission carriers in remote desktop, video conferencing, and live streaming scenarios. The supplementary information stream can be a stream of custom Supplemental Enhancement Information (SEI) messages.

[0027] This disclosure encodes the linear correlation between the chroma and luminance components in the initial image data as additional auxiliary information along with the video frame image data; thus, in the subsequent decoding stage, the initial image data corresponding to each video frame image data can be reconstructed based on the linear relationship corresponding to the video frame image data, effectively improving the clarity of the restored image.

[0028] The encoding of supplementary information and the encoding of video frame image data are independent of each other and have no process constraints, allowing for parallel processing. The computational load of supplementary information is small and can be handled by the CPU; while the encoding of video frame image data has a higher computational load and can be scheduled for GPU execution, fully leveraging the parallel computing power of the GPU to improve the overall efficiency of image data processing.

[0029] Furthermore, in some embodiments, the statistics of supplementary information of the initial image data and the generation of video frame image data corresponding to the initial image data can be performed simultaneously using different threads to further improve the processing efficiency of image data, and this disclosure does not limit this.

[0030] Step 103: The CPU outputs the encoded bitstream of the initial image data based on the supplementary information bitstream of the initial image data and the video frame bitstream.

[0031] Specifically, the CPU can further bind and associate the supplementary information bitstream corresponding to the initial image data with the video frame bitstream, and merge and encapsulate them to form a complete encoded bitstream independent of the initial image data. The CPU then sends the complete encoded bitstream for subsequent transmission, storage, or processing.

[0032] The CPU is responsible for global logic scheduling, task coordination, and data distribution. Supplementary information transaction encoding and video frame image data encoding are generated by different hardware units. Since the two bitstreams originate from different sources, the CPU uniformly matches and binds the supplementary information bitstream of the initial image data with the video frame bitstream, accurately establishing a one-to-one mapping relationship and ensuring the accuracy of the bitstream pairing logic. Furthermore, bitstream association, encapsulation, splicing, and packet output are serial logical transactions with few branches and a well-organized process, resulting in low computational load. The CPU can efficiently complete these processes without the need for GPU parallel computing power. Moreover, the final integrated bitstream needs to be transmitted to external peripherals such as image receivers and network transmission interfaces. The CPU uniformly manages the underlying communication control, including peripheral interaction, network packet sending, and data forwarding. Direct output of the integrated bitstream by the CPU eliminates the need for secondary data transfer between the GPU and CPU, reducing additional transmission latency and bandwidth loss.

[0033] The image data processing method disclosed herein adopts a CPU and GPU collaborative hybrid processing architecture. The computational task of obtaining supplementary information from the initial image data is scheduled to run on the GPU, fully leveraging the powerful parallel computing power of the GPU and significantly improving the processing throughput efficiency of the basic image data. Furthermore, the encoding of supplementary information with low computational load is handled by the CPU, while the encoding of downsampling blocks with higher computational load is allocated to the GPU for parallel processing. The two types of encoding tasks can be carried out simultaneously and in parallel, further compressing the overall encoding time. In addition, the logical management of bitstream pairing and association and output is uniformly handled by the CPU. The bitstream matching and forwarding can be accurately completed based on the CPU's scheduling and peripheral interaction capabilities, avoiding the latency loss caused by frequent data transfer between multiple devices.

[0034] In some embodiments, the method further includes: the CPU calling the GPU to divide the initial image data into multiple image blocks.

[0035] There are multiple supplementary information entries, with one supplementary information entry corresponding to one image block.

[0036] Specifically, the initial image data includes multiple image blocks, with each image block corresponding to a supplementary information entry. This supplementary information is used to identify the linear relationship between the chromaticity and luminance components in the image block.

[0037] Specifically, the initial image data, as a global whole, exhibits significant differences in characteristics (such as texture, brightness, chroma, motion state, etc.) across different regions. Processing this initial image data as a whole may result in the loss of local details and subject it to noise interference. This disclosure addresses this by dividing the image into blocks, decomposing the global problem into a series of local problems, thereby enabling refined processing within each image block and improving the accuracy of the image data.

[0038] In some embodiments, the CPU invokes the GPU to process the color space information of multiple image patches in parallel to obtain supplementary information for the multiple image patches. Therefore, this disclosure can fully utilize the parallel computing capabilities of the GPU, improve the efficiency of obtaining supplementary information, and thus improve the efficiency of image data processing.

[0039] In some embodiments, the CPU can divide the initial image data based on computational precision and CPU cache hit rate to obtain multiple image blocks.

[0040] In this context, computational precision refers to the accuracy of processing the initial image data. Specifically, for the same initial image data, the more image blocks obtained, the more refined the processing of the initial image data, and the greater the computational precision; conversely, the fewer image blocks obtained, the coarser the processing of the initial image data, and the lower the computational precision.

[0041] CPU cache hit rate refers to the percentage of times the CPU finds data in the cache within a certain period of time. A higher hit rate results in faster program execution and better performance.

[0042] If the image patch is too small, it may cause the CPU to access pixels in a skipping and non-contiguous manner when processing the initial image data. This can lead to the data just loaded into the cache being overwritten by the new data needed later before it is fully utilized, resulting in frequent cache misses and the CPU spending most of its time waiting for data. Furthermore, the smaller the image patch, the higher the proportion of management overhead (such as function calls and loop control) in the processing of each image patch. Conversely, the larger the image patch, the more all the data of an image patch (including input pixels, intermediate calculation results, etc.) can be stored in the CPU's high-speed cache at once. In this way, when processing the image patch, the CPU hardly needs to access the slow main memory, resulting in higher efficiency.

[0043] This disclosure can balance computational accuracy and CPU cache hit rate by dividing the initial image data into multiple image blocks of appropriate size.

[0044] In addition, this disclosure may also divide image blocks based on the resolution, texture complexity, etc. of the initial image data, or may adopt a multi-layer partitioning strategy, etc., and this disclosure does not limit it in this regard.

[0045] Furthermore, in application scenarios such as remote desktops, the chromaticity components and luminance components of at least some image blocks within the initial image data exhibit a linear correspondence. Therefore, this disclosure can also process the color space information of each image block in the initial image data to determine the linear correlation between the chromaticity and luminance components in each image block, thereby obtaining supplementary information for the corresponding image block.

[0046] Furthermore, the CPU encodes the supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data, and calls the GPU to encode the video frame image data to generate a video frame bitstream of the initial image data. This includes: the CPU encodes the supplementary information of multiple image blocks separately to generate supplementary information bitstreams of each image block, and calls the GPU to encode multiple downsampled blocks separately to generate video frame bitstreams of each image block. The video frame image data includes multiple downsampled blocks, and one downsampled block corresponds to one image block.

[0047] Specifically, the video frame image data includes multiple downsampled blocks. This disclosure allows downsampling processing to be performed on multiple image blocks in the initial image data to obtain downsampled blocks corresponding to each image block. Specifically, one downsampled block is obtained from one image block through downsampling processing.

[0048] The supplementary information bitstream of an image block refers to the bitstream obtained by encoding the supplementary information of that image block; the video frame bitstream refers to the bitstream obtained by encoding each downsampled block. One piece of supplementary information corresponds to one supplementary information bitstream, and one downsampled block corresponds to one video frame bitstream.

[0049] This disclosure divides the initial image data into multiple image blocks and encodes the image data in units of image blocks, decomposing the global problem into a series of local problems, thereby enabling refined processing in each image block and improving the accuracy of the image data.

[0050] In some embodiments, the CPU outputs the encoded bitstream of the initial image block based on the supplementary information bitstream and the video frame bitstream of the initial image data, including: the CPU associating the supplementary information bitstream of each image block with the video frame bitstream of the corresponding image block to obtain the encoded bitstream of multiple image blocks, and outputting it.

[0051] Specifically, each image block corresponds to a supplementary information bitstream and a video frame bitstream. The CPU can further bind the supplementary information bitstreams generated for each image block with the corresponding video frame bitstream, and merge and encapsulate them to form an independent complete encoded bitstream for that image block. After all image blocks are encapsulated, the CPU splices and integrates the complete encoded bitstreams of multiple image blocks and sends them together for subsequent transmission, storage, or processing.

[0052] This disclosure migrates block-level adaptive sampling to the GPU side, significantly improving sampling speed by leveraging the multi-core parallel capabilities of the GPU while fully preserving the original algorithm's computational accuracy and image quality restoration. This achieves GPU acceleration for computationally intensive tasks and CPU processing for logic control tasks, particularly in high real-time scenarios such as remote desktops, video conferencing, and live streaming, balancing encoded image quality, image restoration clarity, and overall real-time processing. The embodiments provided in this disclosure improve computational speed while maintaining algorithm accuracy, meeting the dual requirements of real-time transmission and high image quality in desktop scenarios.

[0053] In some embodiments, the method further includes: the CPU determining a first locally available storage resource and determining a second storage resource in the GPU for storing supplementary information; after the CPU obtains the supplementary information of the initial image data, the method further includes: the CPU calling the GPU to write the supplementary information of the initial image data into the second storage resource; and copying the supplementary information in the second storage resource into the first storage resource.

[0054] Specifically, since the image data processing is handled collaboratively by the CPU and GPU, this disclosure allows for the creation of corresponding storage resources on both the CPU and GPU sides.

[0055] The first storage resource refers to the temporary storage resources accessible to the CPU, while the second storage resource refers to the storage resources in the GPU used to store supplementary information.

[0056] After the GPU processes each downsampled block to obtain the supplementary information for the corresponding image block, it writes this supplementary information into the second storage resource. Furthermore, since the encoding of the supplementary information is performed by the CPU, the CPU can also copy the supplementary information of each image block from the second storage resource to the first storage resource available locally on the CPU. Thus, the CPU can read the supplementary information of each image block from the first storage resource and encode it to obtain the supplementary information bitstream for the corresponding image block.

[0057] The embodiments provided in this disclosure create a storage resource on the CPU local side and a storage resource on the GPU side, respectively. The GPU can first write the obtained supplementary information into the second storage resource on the GPU side, and then the CPU copies the supplementary resource into the first storage resource accessible to the CPU for use in CPU-side parameter encoding. This enables CPU and GPU to process collaboratively, improving the processing accuracy and efficiency of image data.

[0058] In some embodiments, the CPU calls the graphics processing unit (GPU) to process the color space information in the initial image data to obtain supplementary information of the initial image data, including: for each image block, the CPU calls the GPU to calculate the chromaticity gradient, the correlation between the luminance component and the chromaticity component, and the linear relationship parameters corresponding to the image block to obtain supplementary information of the image block.

[0059] Specifically, the chromaticity gradient can characterize the degree of drastic chromaticity changes within an image patch. A large chromaticity gradient value indicates that the image patch includes highly distorted and sensitive areas such as text edges, thin lines, and color block boundaries; conversely, a gentle chromaticity gradient indicates that the image patch includes large areas of smooth, pure color.

[0060] For image blocks including the edges of Chinese characters / icons, the boundaries of solid color blocks, or high-frequency fine lines, decoding and restoring these blocks may result in severe chromaticity distortion, significantly impacting the image quality of the recovered image. Therefore, this disclosure uses the chromaticity gradient as a threshold to distinguish whether luminance-chromaticity linear compensation is needed for image blocks: high gradients pose a high risk of distortion, and linear parameters can be used to correct chromaticity during decoding; low gradient regions inherently suffer only minor chromaticity loss, simplifying the restoration logic and balancing computational load and image quality, thus eliminating the need for linear parameter correction. Simultaneously, the chromaticity gradient can help determine the fit of the linear model; regions with disordered gradients have low linear fitting reliability and can be flagged to reduce compensation weights, avoiding over-correction that introduces new noise.

[0061] The correlation between the luminance and chrominance components is used to quantify the strength of the binding between luminance and chrominance values ​​within an image patch: a higher correlation indicates a high degree of synchronization between changes in brightness and color intensity within the image patch, resulting in good linear model adaptability; extremely low correlation indicates no stable correspondence between the two, rendering linear compensation ineffective. Furthermore, correlation can be used as a criterion for encoding selection: weakly correlated image patches do not require the transmission of complete linear parameters, only marking them as not requiring compensation, saving bitstream bandwidth; strongly correlated image patches record complete linear parameters, ensuring effective decoding and restoration. In addition, this correlation can provide a confidence flag for the decoding end, adaptively adjusting the compensation amplitude based on the correlation magnitude during decoding to prevent color shifts caused by applying a linear model to weakly correlated blocks.

[0062] Linear parameters are used to quantify the corresponding mapping rules between luminance and chrominance within image blocks where there is a linear correlation between them. They are the core correction model for chrominance restoration at the decoding end. In remote desktop scenarios, YUV downsampling compresses chrominance resolution, and simple upsampling easily results in blurry text and color cast at the edges. The decoding end uses the linear correlation parameter between the luminance and chrominance components to inversely deduce the true chrominance value based on the restored accurate luminance value.

[0063] In this disclosure, the GPU can first be invoked to calculate the chromaticity gradient of the image patch. If it is determined from the obtained chromaticity gradient that the image patch needs to enable luminance-chromaticity linear compensation restoration, the correlation between the luminance component and the chromaticity component in the image patch is further calculated. If it is determined from the obtained chromaticity gradient that the image patch does not need to enable luminance-chromaticity linear compensation restoration, the correlation and linear relationship parameters corresponding to the image patch can be directly set to empty or invalid. Furthermore, if it is determined from the obtained chromaticity gradient that the image patch needs to enable luminance-chromaticity linear compensation restoration, and if the calculated correlation indicates that there is a linear correlation between the luminance component and the chromaticity component in the image patch, the linear relationship parameters corresponding to the image patch are further calculated. If the calculated correlation indicates that there is no linear correlation between the luminance component and the chromaticity component in the image patch, the linear relationship parameters corresponding to the image patch can be directly set to empty or invalid.

[0064] In some embodiments, the process of calling the GPU to calculate the chromaticity gradient, the correlation between the luminance component and the chromaticity component, and the linear relationship parameters corresponding to the image patch can select a 3×3 local neighborhood corresponding to the pixel as a feature statistics window. Within this limited neighborhood, the global statistics and calculations of the chromaticity gradient, luminance-chromaticity correlation, and linear restoration coefficient can be uniformly completed. Compared with the global statistical method, the 3×3 neighborhood window provided in this disclosure is small in size and has strong focusing, which can accurately lock the local texture features of the image and adapt to the feature distribution characteristics of high-frequency detail areas such as text edges, icon outlines, fine lines, and color block boundaries in remote desktop images.

[0065] Specifically, in the statistical process, firstly, the chromaticity data of all pixels within a 3×3 neighborhood is traversed, and the variation amplitude and direction of chromaticity values ​​are calculated point by point to obtain the overall chromaticity gradient characteristics of the neighborhood. This accurately represents the intensity of chromaticity fluctuations in the local area, effectively distinguishing between areas with smooth solid colors and textured edges prone to distortion. Secondly, based on the original data of the luminance and chromaticity components of all pixels within the 3×3 neighborhood, the correlation between the two sets of data is calculated, quantifying the correlation between luminance and chromaticity within the neighborhood. This accurately determines whether the local area satisfies the linear mapping condition, selecting effective image areas suitable for the linear restoration model. Based on the correlation verification, the corresponding linear restoration coefficients are further calculated by fitting the pixel data of the 3×3 neighborhood.

[0066] This disclosure, through comprehensive statistical analysis of chromaticity gradient, correlation, and linear relationship parameters in image blocks, can accurately extract local detail features of image blocks and generate precise compensation parameters that match the image quality distortion characteristics of the region. This provides reliable core data support for the decoding end to achieve refined chromaticity restoration, restore image detail texture, and correct edge color distortion.

[0067] The embodiments provided in this disclosure can significantly improve the color reproduction accuracy of text, thin lines, and color block boundaries, thereby improving image distortion at its source.

[0068] In some embodiments, the supplementary information includes validity information and restoration parameter information. The validity information is used to identify whether there is a linear relationship between the chroma components and the luminance components in the image patch, and the restoration parameter information is used to identify the linear relationship parameters between the chroma components and the luminance components in the image patch.

[0069] Specifically, validity information is used to identify whether a linear relationship exists between the chroma and luminance components in an image patch, that is, to indicate whether restoration parameter information needs to be transmitted. Restoration parameter information for an image patch is only required to be transmitted if the validity information indicates a linear relationship between the chroma and luminance components, and only then is the restoration parameter information in the corresponding image patch valid. The validity information can be determined based on the correlation in the above embodiments. In some embodiments, a correlation threshold can be preset. If the correlation value of an image patch is greater than the correlation threshold, it is determined that a linear relationship exists between the luminance and chroma components in the image patch, and thus the restoration parameter information in the corresponding image patch is valid; conversely, if the correlation value of an image patch is less than or equal to the correlation threshold, it is determined that a linear relationship does not exist between the luminance and chroma components, and thus the restoration parameter information in the corresponding image patch is invalid.

[0070] In some embodiments, the recovery parameter information may include linear relationship parameters of chroma components relative to luminance components in an image patch, i.e., linear relationship parameters in the above embodiments; in some embodiments, the recovery parameter information may also include recovery parameters for recovering chroma components based on luminance components, which is not limited in this disclosure.

[0071] In some embodiments, the supplementary information may also include the chromaticity gradient information described in the above embodiments, used to identify the validity of the validity information. Thus, the accuracy of the corresponding recovery parameter information can be determined based on the chromaticity gradient information and validity information of the image patch.

[0072] Furthermore, the CPU encodes the supplementary information of multiple image blocks to generate a supplementary information bitstream for each image block, including: the CPU reads the supplementary information of multiple image blocks; for each image block, the CPU encodes the validity information and the recovery parameter information to obtain the encoded information; and encapsulates the encoded information to obtain the supplementary information bitstream for the image block.

[0073] Specifically, after the CPU completes the feature calculation of each image block and generates the corresponding supplementary information, this disclosure further reads, encodes and encapsulates the supplementary information of all image blocks.

[0074] In some embodiments, the CPU performs effective block mask encoding on the validity information of each image block and differential encoding and quotient encoding on the recovery parameter information of the image block based on the statistical results of the supplementary information of each image block, thereby obtaining the supplementary information bitstream of the image block.

[0075] Specifically, validity information can be a binary identifier, indicating whether a linear relationship exists between the corresponding image blocks or whether the corresponding image block needs to transmit restoration parameters. Masking uses 1 bit or a simplified marker to indicate the valid or invalid status of the restoration parameter information corresponding to each image block. Multiple image blocks corresponding to the initial image data can form a mask bit sequence. 1 represents that the image block is valid and the backend needs to parse the restoration parameters; 0 represents that the image block is invalid and does not need to carry restoration parameters.

[0076] Differential coding refers to not directly transmitting the original absolute values ​​of parameters, but instead calculating the difference between the parameters of the current image block and the parameters of the previous image block, and only transmitting the difference data. In this case, adjacent image blocks within the same frame have similar texture features, the recovered parameter difference is much smaller than the original parameter magnitude, and the difference data is more concentrated, providing a better compression basis for subsequent entropy coding. This avoids the enormous bitstream pressure caused by transmitting complete floating-point parameters block by block, and the compression benefits are particularly high when the effective blocks of the entire frame are distributed contiguously.

[0077] Entropy coding refers to lossless variable-length compression algorithms (such as Huffman coding and arithmetic coding). It statistically analyzes the probability of occurrence of the difference values ​​after differential coding, assigning short bits to frequently occurring small differences and long bits to low-frequency large differences, adaptively allocating the code length according to probability to achieve lossless maximum compression. Since the parameter differences after differential coding are mostly concentrated around 0, with a highly uneven probability distribution, entropy coding can exploit probabilistic redundancy, maximizing bitrate compression without sacrificing the accuracy of the restored parameters. Furthermore, the restored parameter information is the core data for chroma restoration and must be transmitted losslessly. Entropy coding, being a lossless compression method, does not compromise the accuracy of linearity and correlation coefficients, ensuring stable decoding and image quality restoration results.

[0078] Furthermore, the CPU's mask encoding of validity information can be processed asynchronously by worker threads, thereby improving the encoding efficiency of supplementary information.

[0079] Specifically, the validity mask encoding logic is simple and highly independent, eliminating the need to wait for the recovery parameter differential and entropy encoding to complete. The CPU can dedicate a separate working thread to process the mask bit sequence of the entire frame image block, while the encoding of the recovery parameter information is performed synchronously on the main thread. The two types of encoding tasks proceed in parallel, eliminating the latency of serial queuing. This reduces the overall encoding time of the entire frame supplementary information, improving the real-time performance of encoding in high frame rate remote desktop and real-time conferencing scenarios.

[0080] Of course, this disclosure can also directly encode the recovery parameter information of each image block, and this disclosure does not limit this.

[0081] The embodiments provided in this disclosure can encode the validity information and restoration parameter information of image blocks separately according to the characteristics of the validity information and restoration parameter information of image blocks, thereby improving encoding efficiency and encoding accuracy.

[0082] In some embodiments, the method further includes: the CPU sending the raw video data to the GPU; the CPU calling the GPU to perform color space conversion on the raw video data to obtain initial image data; and the CPU calling the GPU to downsample the initial image data to obtain video frame image data.

[0083] Specifically, raw video data refers to the raw data acquired by the CPU. The format of raw video data is generally BGRA (Blue-Green-Red-Alpha) image data.

[0084] The CPU acquires BGRA format image data from the desktop, display buffer, or other image sources. Then, the CPU inputs the acquired BGRA image to the GPU as input data. Further, the GPU reads the received raw video data (BGRA format image data) and converts the BGRA format graphic data into YUV format image data through color space conversion to obtain initial image data (such as YUV444 format image data). Further, the GPU downsamples this initial image data to obtain video frame image data (such as YUV420 format image data).

[0085] In the embodiments provided in this disclosure, the CPU sends the raw video data to the GPU for preprocessing such as color space conversion and downsampling. Furthermore, the CPU directly calculates the supplementary information of each image block within the GPU, eliminating the need to send the data of each image block back to the CPU for further processing, thus reducing memory movement. If the calculation of supplementary information were handled by the CPU, a large amount of image block data would need to be repeatedly transferred between the CPU and the GPU, resulting in additional bus transmission latency and bandwidth consumption.

[0086] The embodiments provided in this disclosure delegate the preprocessing of the original video data (including color space conversion and downsampling) and the calculation of supplementary information for each image block to the GPU. While making full use of the parallel computing characteristics of the GPU, it also reduces memory movement and further improves data processing efficiency.

[0087] In some embodiments, the CPU calls the graphics processing unit (GPU) to process the color space information in the video frame image data to obtain supplementary information of the initial image data, including: the CPU calls the GPU to use thread-level parallel processing to process the color space information in multiple image blocks in parallel to obtain supplementary information for each of the multiple image blocks.

[0088] Specifically, an initial frame of image data is divided into several independent image blocks. The GPU, leveraging its massive lightweight parallel thread resources, allocates independent computation threads to different image blocks. These threads operate independently and synchronously, performing feature calculations such as chroma gradient, luminance-chroma correlation, and linear restoration coefficients to obtain supplementary information for the corresponding image block. The computational logic between different image blocks is independent of each other, allowing for complete parallel processing and batch output of all image blocks' complete supplementary information at once, fully utilizing the GPU's multi-threaded parallel computing power. Compared to CPU's serial block-by-block computation, this significantly reduces the overall time required to generate supplementary information. For example, each image block is a 2×2 chroma block. Each GPU thread uses a 2×2 pixel block as its basic processing unit, completing block-level statistics of supplementary information while generating video frame image data.

[0089] The embodiments provided in this disclosure, by allocating independent computing threads to different image blocks, allow each thread to process different image blocks synchronously without blocking each other, thereby obtaining supplementary information for each image block. This fully leverages the advantages of GPU's multi-threaded parallel computing power, further shortens the overall generation time of supplementary information, and improves the processing efficiency of image data.

[0090] Secondly, this disclosure also provides an image data processing method, which is applied to a CPU. Specifically, the image data processing method involves the CPU performing an image decoding operation to restore the initial image data.

[0091] Figure 2 A flowchart of yet another image processing method provided in an embodiment of this disclosure.

[0092] like Figure 2 As shown, the image processing method specifically includes the following steps: Step 201: The CPU receives the encoded bitstream.

[0093] The encoded bitstream includes the video frame bitstream and the supplementary information bitstream.

[0094] Step 202: The CPU decodes the supplementary information bitstream to obtain the supplementary information; and calls the GPU to decode the video frame bitstream to obtain the video frame image data.

[0095] Step 203: The CPU calls the GPU to restore the video frame image data based on the supplementary information to obtain the initial image data corresponding to the encoded bitstream.

[0096] The supplementary information is used to identify the linear relationship between the chromaticity and luminance components in the initial image data.

[0097] Specifically, the CPU receives the encoded bitstream, which consists of two parts: a video frame bitstream and a supplementary information bitstream. Then, the CPU decodes the supplementary information bitstream and sends the video frame bitstream to the GPU, which then decodes the video frame bitstream. The two decoding processes can be executed in parallel.

[0098] The supplementary information identifies the linear relationship between the chroma and luminance components of the initial image data before downsampling. Furthermore, in subsequent upsampling and reconstruction of the video frame image data by the CPU using the GPU, an initial reconstructed image is obtained. This initial reconstructed image is then further corrected using the supplementary information to obtain the final initial image patch. Specifically, for some image patches containing highly distortion-sensitive areas such as text edges, thin lines, and color block boundaries, downsampling followed by upsampling can lead to severe chroma distortion, significantly impacting the image quality of the restored image. Therefore, this disclosure also requires correction of the reconstructed initial image based on the supplementary information to obtain a clear reconstructed image.

[0099] It should be noted that the image data processing method provided in this disclosure is used to decode the encoded bitstream and restore the initial image data; the above Figure 1 The corresponding image data processing method is used to encode the initial image data to obtain the encoded bitstream. In other words, the image data processing method provided in this disclosure is similar to the one described above. Figure 1 The corresponding image data processing method embodiments are the opposite of the processing steps, but other specific details are similar. Therefore, other specific details of the embodiments of this disclosure can be found above. Figure 1 The corresponding implementation examples will not be repeated here.

[0100] The embodiments provided in this disclosure call the GPU to decode the video frame bitstream to obtain video frame image data, and then call the GPU to restore the decoded video frame image data. By scheduling most of the computation for decoding the encoded bitstream to the GPU, the computational advantages of the GPU can be fully utilized, improving image data processing efficiency. Furthermore, this disclosure schedules both the process of obtaining video frame image data and the process of restoring the video frame image data to the GPU, reducing memory movement and further improving data processing efficiency. Furthermore, this disclosure utilizes the CPU to decode the supplementary information bitstream to obtain supplementary information. The GPU can then correct the initial restored image obtained from the video frame image data based on this supplementary information, enabling a more accurate restoration of the initial image data corresponding to the encoded bitstream and improving the image quality of the restored initial image data. This disclosure improves data processing efficiency while ensuring the image quality of the restored initial image data through CPU and GPU collaboration in decoding and restoring the encoded bitstream.

[0101] In some embodiments, the video frame image data includes multiple downsampling blocks, and the number of supplementary information entries is multiple, with one downsampling block corresponding to one supplementary information entry.

[0102] Furthermore, the CPU calls the GPU to restore the video frame image data based on the supplementary information to obtain the initial image data corresponding to the encoded bitstream. This includes: the CPU calls the GPU to restore the chroma components in the corresponding downsampling blocks based on the linear relationship in each supplementary information to obtain the initial image data corresponding to the encoded bitstream. The initial image data includes multiple image blocks, and one image block corresponds to one downsampling block.

[0103] Specifically, the supplementary information stream can include multiple sub-information streams, each of which is decoded to obtain one piece of supplementary information; the video frame stream includes multiple sub-video streams. Each sub-video stream is decoded to obtain a downsampled block; and there is a one-to-one correspondence between the downsampled block and the supplementary information. A downsampled block refers to the image data obtained by downsampling the initial image data, and the supplementary information identifies the linear relationship between the chroma and luminance components of the initial image data before downsampling. Further, the CPU calls the GPU to upsample and restore each downsampled block to obtain an initial restored image, and simultaneously uses the supplementary information corresponding to each downsampled block to correct the corresponding initial restored image, obtaining the image block corresponding to that downsampled block.

[0104] This disclosure decomposes a global problem into a series of local problems by restoring the encoded bitstream on an image block basis, thereby enabling refined processing of each image block and improving the accuracy of image data.

[0105] In some embodiments, the CPU calls the GPU to decode the video frame bitstream, including: the CPU determining the decoding parameters required to decode the video frame bitstream and submitting the decoding parameters to the GPU, wherein the decoding parameters include at least one of the image parameters, quantization matrix and slice control information corresponding to the video frame bitstream; the CPU calls the GPU to decode the video frame bitstream based on the decoding parameters.

[0106] Among them, the image parameters record basic attributes such as video frame resolution, image block size, and color sampling format, which are used to guide the GPU to determine the decoding operation specifications and allocate corresponding video memory space to ensure that pixel channel resolution matches the image size; the quantization matrix is ​​a dedicated lookup table for residual inverse quantization. The GPU uses this matrix to perform inverse quantization restoration of the bitstream residual coefficients, which is a key parameter for restoring the original pixel accuracy and avoiding image distortion; the slice control information includes intra-frame slice division, data offset, and parallel scheduling identifier. The GPU uses this information to split multi-threaded parallel decoding tasks, which improves decoding speed while enhancing bitstream fault tolerance.

[0107] Specifically, image parameters are configuration parameters that describe the basic attributes of video frames and provide a unified computing benchmark for GPU decoding. They include at least one of the following: frame resolution, coding block size, YUV chroma sampling format, bit depth, color space identifier, and frame type (such as I-frame or P-frame).

[0108] Before decoding, the GPU relies on image parameters to determine the overall size of the image, divides it into decoding processing units, allocates corresponding amounts of video memory to store pixel data, and matches the processing logic for luminance and chrominance channels. If this parameter is missing, the GPU cannot determine the image size and sampling rules, which will result in problems such as pixel misalignment, image stretching, and disordered color channel resolution.

[0109] The quantization matrix is ​​a lookup table matrix used in the H.264 / H.265 coding standard for residual transform quantization. The encoder uses the quantization matrix to compress the residual data, thus reducing the bitrate. Decoding requires inverse quantization to recover the residual pixel values.

[0110] During the inverse transform stage, the GPU reads the quantization matrix and performs inverse quantization on the quantization residual coefficients in the bitstream to restore the pixel residual information before encoding. The quantization matrix directly determines the image restoration accuracy. Matrix mismatch will cause image blockiness, color banding, and detail blurring.

[0111] In addition, video frames are divided into several independent slices, and slice control information records the number of slices, the starting offset of each slice's data, the slice length, the parallel scheduling flag, the boundary synchronization flag, and other management information.

[0112] GPUs support multi-slice, multi-threaded parallel decoding, relying on slice control information to split decoding tasks and allocate independent threads to synchronously process different slice data. On the one hand, this significantly improves the parallelism of decoding and reduces the decoding time of a single frame. On the other hand, errors in a single slice of data will not affect the decoding of the entire frame, thus improving the fault tolerance of decoding.

[0113] In the embodiments provided in this disclosure, the CPU submits the decoding parameters required for decoding to the GPU, so that the GPU can accurately and efficiently decode the video frame bitstream according to the decoding parameters.

[0114] In some embodiments, the CPU decodes the supplementary information bitstream to obtain supplementary information, including: the CPU parses the supplementary information bitstream to determine the configuration parameters and compression parameter stream of the supplementary information bitstream; the CPU decodes the compression parameter stream based on the configuration parameters to obtain supplementary information.

[0115] Specifically, the configuration parameters include at least one of version, mode, and mask stream length. The version is the version number field of the supplementary information bitstream, used to identify the auxiliary information encoding specification version that accompanies the current supplementary information bitstream. The GPU distinguishes different algorithm iteration logics by reading the version, adapts to the corresponding parameter parsing and chroma reconstruction process, and avoids parsing errors caused by mixing old and new version bitstreams.

[0116] The mode represents the chroma compensation mode enabled in the current frame, used to indicate whether linear luminance and chroma restoration are enabled and to distinguish the compensation intensity strategies for different regions. Subsequent chroma restoration steps of the GPU can switch the corresponding computation branch based on this field to achieve adaptive image quality correction. This mode can correspond to the chroma gradient during the encoding process.

[0117] The mask stream length is used to record the total bit length of the subsequent valid block mask binary bit stream, providing a length boundary for segmented reading and buffering of mask data, preventing out-of-bounds reading of the bit stream and data misalignment, and ensuring the complete extraction of the mask sequence.

[0118] The compressed parameter stream includes the bitstream corresponding to validity information and recovery parameter information. For example, the compressed parameter stream includes the valid block mask stream and the recovery parameter stream.

[0119] The CPU parses the supplementary information stream, that is, it parses the extracted user-defined SEI message to obtain the configuration parameters and compression parameter stream; then the CPU decodes the compression parameter stream based on the configuration parameters.

[0120] In some embodiments, the CPU decodes the compressed parameter stream, including restoring the valid block bit stream to obtain a valid block mask, and decoding the restored parameter stream based on the valid block mask to restore the restored parameters of each valid block.

[0121] The effective block mask consists of a mask bit sequence composed of single-bit markers, each bit corresponding to an image block within the frame. A bit value of 1 indicates that the corresponding image block has valid linear restoration parameters, and subsequent restoration parameters need to be read to participate in chroma reconstruction; a bit value of 0 indicates that it is a flat invalid block, and the parameter parsing process is skipped directly to reduce redundant computational overhead.

[0122] The recovery parameter stream stores the linear mapping coefficients between the luminance and U and V chroma channels for each valid image block, which are the linear recovery parameters mentioned earlier. This parameter stream is compressed through differential coding and entropy coding. After the CPU completes the decompression, it sends the complete linear coefficients to the GPU for use at the decoding end to correct chroma distortion and restore text and edge details.

[0123] In some embodiments, the CPU decodes the compressed parameter stream and can restore the effective block bitstream using mixed run-length encoding (RLE) or Rice coding to obtain the effective block mask. Then, it restores the recovery parameters of each effective block using differential Rice or the original 16-bit mode.

[0124] Specifically, the CPU first extracts the run lengths corresponding to consecutive 0s and 1s in the mask bitstream using RLE run-length encoding, eliminating redundant bits caused by large areas of solid color. Then, it uses Rice variable-length entropy encoding to perform secondary compression on small integers of run lengths, significantly reducing the size of the mask bitstream. In the decoding stage, the CPU first performs Rice inverse decoding on the compressed mask bitstream to recover the run length sequence, and then reconstructs the complete valid block mask bit by bit using RLE inverse operation. The entire scheme is lossless, and the computational logic is lightweight, allowing for asynchronous parsing using independent threads without consuming decoding computing power for recovery parameters.

[0125] For the recovery parameters of each valid block, the CPU can use a differential Rice mode, which utilizes the spatial correlation of parameters from adjacent image blocks for differential preprocessing, and then performs Rice entropy coding on the concentrated small differences, achieving an extremely high compression ratio under normal desktop conditions. Alternatively, it can use the original 16-bit direct storage mode, which skips differential and variable-length compression and directly stores the original parameters in 16-bit fixed-length storage for complex images with severe texture and large parameter fluctuations, as a fallback solution to ensure controllable bitstream and lossless parameter recovery. When the CPU parses the SEI, it first reads the mode identifier, automatically switches to the corresponding decoding branch, and after completing the reconstruction of all linear recovery parameters, sends them to the GPU for chroma compensation.

[0126] In the embodiments provided in this disclosure, the CPU obtains configuration parameters and compressed parameter streams by parsing the supplementary information bitstream, and decodes the compressed parameter stream based on the configuration parameters, thereby improving the accuracy and efficiency of decoding.

[0127] In some embodiments, the method further includes: the CPU writing supplementary information to a locally available first storage resource and copying the supplementary information in the first storage resource to a second storage resource of the GPU; wherein the second storage resource also stores video frame image data.

[0128] Specifically, after the CPU parses the supplementary information, it performs parameter caching, writes the parsed supplementary information into the parameter data on the CPU side, and saves it in the first storage resource on the CPU side. Further, the supplementary information is written into the second storage resource on the GPU side through the first storage resource on the CPU side. Meanwhile, the video frame image data is decoded by the GPU. Thus, the video frame image data of each image block (including multiple downsampled blocks) is combined with the supplementary information, allowing the GPU to read the supplementary information and restore the corresponding downsampled blocks.

[0129] The embodiments provided in this disclosure create storage resources on both the CPU and GPU sides. After the CPU obtains the supplementary information through decoding, it first writes it to the local first storage resource and then copies it to the second storage resource on the GPU side. This allows the supplementary information and downsampled blocks of each image block to be combined on the GPU side. Subsequently, the GPU can be called to restore the corresponding downsampled blocks based on the supplementary information, thereby realizing collaborative processing between the CPU and GPU. This improves processing efficiency while ensuring processing accuracy.

[0130] In some embodiments, the supplementary information includes validity information and restoration parameter information. The validity information is used to identify whether there is a linear relationship between the chroma component and the luminance component in the image block, and the restoration parameter information is used to identify the linear relationship parameter between the chroma component and the luminance component in the image block. The CPU calls the GPU to restore the corresponding downsampled blocks based on each supplementary information to obtain the initial image data corresponding to the encoded bitstream. This includes: for each downsampled block, the CPU calls the GPU to determine the chroma component information and luminance component information corresponding to the downsampled block; in response to the validity information indicating that there is a linear relationship between the chroma component and the luminance component in the image block corresponding to the downsampled block, the downsampled block is restored according to the restoration parameter information, the chroma component information and the luminance component information of the downsampled block to obtain the image block corresponding to the downsampled block.

[0131] Specifically, the chromaticity component information corresponding to the downsampling block includes the chromaticity component value of the downsampling block, and the luminance component information includes the average luminance of the downsampling block.

[0132] A single downsampling block consists of multiple pixels; in the YUV color space, each pixel corresponds to a set of luminance and chrominance components, thus a downsampling block contains a batch of luminance and chrominance components. The GPU reads the luminance and chrominance components corresponding to all pixels within the downsampling block to perform calculations.

[0133] After the GPU parses the video frame stream, it can obtain relevant information about each downsampling block, including the chroma component information. The GPU can directly read the chroma component information of the downsampling block to obtain the chroma component values ​​and the luminance component values ​​of each pixel in the downsampling block. Furthermore, based on the luminance component values ​​of each pixel in the downsampling block, the average luminance of the downsampling block can be calculated.

[0134] For a downsampled block in an image block where there is a linear relationship between the chroma and luminance components, the downsampled block can be restored using the recovery parameter information and the chroma and luminance component information of the downsampled block to obtain the image block corresponding to the downsampled block.

[0135] It should be noted that the chromaticity components of the image patch include the blue chromaticity component (U) and the red chromaticity component (V). The chromaticity components mentioned above in this application, as well as the chromaticity components mentioned below, may include only the blue chromaticity component or the red chromaticity component, or they may include both the blue chromaticity component and the red chromaticity component. This disclosure does not limit this.

[0136] The restoration parameter information includes the restoration parameters for the chroma components, which can be performed pixel-by-pixel chroma enhancement restoration using the formulas: U'=U+alphaU×(Y-meanY) and V'=V+alphaV×(Y-meanY). Here, U' and V' represent the chroma component values ​​in the corresponding image block after downsampling and restoration, U and V represent the chroma components in the downsampled block, alphaU and alphaV represent the restoration parameters, Y represents the luminance component of the downsampled block, and meanY represents the average luminance of the downsampled block.

[0137] The embodiments provided in this disclosure are capable of accurately recovering downsampled data to obtain image blocks.

[0138] In some embodiments, the method further includes: the CPU calling the GPU to restore the initial image data in color space to obtain the original video data.

[0139] Specifically, the CPU issues a scheduling instruction to call the GPU to perform color space restoration and pixel format conversion on the restored initial image data, and finally outputs raw video data in BGRA format, thereby rendering the raw video data output by the GPU to the display terminal or for subsequent quality analysis; the initial image data is the restored YUV444 format image data.

[0140] In some embodiments, the method further includes: the CPU comparing the raw video data with the benchmark reference data to obtain a comparison result.

[0141] Specifically, this image data processing method also includes an image quality verification step: comparing the original BGRA format video data output after color restoration with the benchmark reference data pixel by pixel to generate a quantization comparison result.

[0142] The benchmark reference data consists of the raw data before the encoding process begins, without downsampling or compression. This benchmark reference data can be loaded in scenarios such as algorithm performance verification, objective image quality evaluation, and consistency verification of encoding and decoding results to quantitatively analyze the accuracy of image restoration after color restoration, and to complete distortion index statistics and restoration effect verification.

[0143] Figures 3a-3b The following are flowcharts of another image data processing method provided in this disclosure, which is applied to a hybrid CPU and GPU architecture. Figure 3a This image data processing method uses a hybrid CPU and GPU architecture to perform encoding operations; Figure 3b The image data processing method uses a hybrid CPU and GPU architecture to perform the decoding operation.

[0144] like Figure 3a As shown, this method is applied at the encoding end and specifically includes the following steps: Step 301: BGRA data input and GPU resource preparation.

[0145] Specifically, the CPU first performs image acquisition, including acquiring BGRA format image data from the desktop, display buffer, or other image sources.

[0146] Secondly, input resource access is performed. The CPU uses the acquired BGRA image as the GPU-side input texture for subsequent computation shaders to read.

[0147] Then, the encoding resource is acquired, and the CPU obtains the writable encoded input texture for the next frame from the hardware encoder.

[0148] In addition, the CPU also needs to prepare statistical resources. The CPU creates GPU statistical textures and corresponding temporary storage resources accessible to the CPU to store block-level statistical results for subsequent SEI parameter encoding.

[0149] Step 302: GPU preprocessing, NV12 generation, and block-level statistics are executed in parallel.

[0150] NV12 refers to a storage format of YUV420.

[0151] Specifically, the input image is processed using a compute shader thread group approach, enabling multiple pixel blocks to be executed in parallel on the GPU side, thus achieving parallel scheduling.

[0152] GPU preprocessing refers to color preprocessing. The GPU thread reads the BGRA input texture and generates the NV12 luminance and chrominance planes required by the hardware encoder.

[0153] Furthermore, block-level statistics are performed. In the same GPU preprocessing process, 2×2 chroma blocks are used as processing units to statistically analyze the chroma gradient, luminance-chroma correlation, and linear restoration coefficient within the 3×3 neighborhood.

[0154] Furthermore, the statistical results are output, and the chromaticity components (alphaU, alphaV) recovery parameters corresponding to each 2×2 chromaticity block are packaged into the GPU statistical texture in 16-bit format.

[0155] Finally, readback preparation is performed by copying the GPU statistical texture to CPU-accessible temporary resources to provide input for CPU-side parameter encoding and SEI packaging.

[0156] Step 303: Hardware video encoding and CPU-side SEI parameter generation pipeline processing.

[0157] Specifically, the CPU submits the video encoding, and the CPU calls the hardware encoder to encode the NV12 format GPU texture to generate H.264 / H.265 video stream packets.

[0158] The CPU reads the statistical results, maps temporary resources on the CPU side, and reads the block-level statistical results generated by the GPU.

[0159] The CPU performs parameter encoding. Based on the block-level statistical results, the CPU performs effective block mask encoding, alphaU / alphaV parameter differential encoding, and entropy encoding. The mask encoding can be processed asynchronously by the worker thread.

[0160] The CPU performs SEI encapsulation, encapsulating the encoded block-level parameters into user-defined SEI messages and saving them in frame order, waiting to be associated with the corresponding video frames for output.

[0161] The hardware encoding task is performed by the GPU video encoding unit, while the SEI parameter generation is performed by the CPU. The two form a CPU-GPU heterogeneous pipeline through frame order and buffer resources.

[0162] Step 304: Link the video stream with SEI and output it.

[0163] Specifically, the CPU performs SEI acquisition, obtaining the corresponding user-defined SEI message based on the current output frame order.

[0164] The CPU performs SEI association. During the bitstream output stage, the CPU associates the SEI message with the corresponding video frame bitstream packet, enabling the decoding end to recover block-level parameters frame by frame.

[0165] The CPU encodes and outputs a bitstream containing video frames and corresponding SEI information, enabling real-time transmission.

[0166] In addition, in local verification scenarios, SEI data and video frame bitstream packets can be sent to the producer-consumer queue, and the decoding thread can asynchronously perform decoding and recovery verification.

[0167] like Figure 3b The method is applied at the decoding end and specifically includes the following steps: Step 310: Bit stream reception, frame packet buffering and decoding thread scheduling.

[0168] Specifically, the CPU receives the bitstream, which includes H.264 / H.265 video frame data and user-defined SEI messages.

[0169] The CPU performs frame packet caching, saving the SEI data and the corresponding video frame bitstream packets in frame order to ensure that the subsequent recovery parameters correspond one-to-one with the video frames.

[0170] In addition, in local verification or real-time reception scenarios, a producer-consumer queue can be used, where the decoding thread in the CPU retrieves the frame packets to be processed from the queue, realizing pipeline parallelism between the encoding output thread and the decoding processing thread.

[0171] Step 320: Parse, extract SEI and submit hardware decoding.

[0172] Specifically, first, the CPU performs Network Abstraction Layer (NAL) parsing. The decoding thread in the CPU parses the NAL units in the H.264 / H.265 bitstream to identify parameter sets, video slices, and user-defined SEI messages.

[0173] Secondly, the CPU performs SEI extraction. When the CPU detects a user-defined SEI message, it saves the NAL data of that SEI for use in CPU-side parameter parsing.

[0174] Then, the CPU performs decoding submission, constructing parameters such as image parameters, quantization matrix, bitstream data, and slice control information required for DirectX Video Acceleration (DXVA) decoding of the video slices.

[0175] Next, the CPU submits the decoding task to the GPU video decoding unit through the video decoding interface and outputs decoded image resources in NV12 format.

[0176] After the hardware decoding task is submitted, it is executed by the GPU video decoding unit, while the CPU can continue to perform SEI parameter parsing and parameter upload preparation.

[0177] Step 330: CPU-side SEI parameter parsing and GPU parameter uploading.

[0178] Specifically, the CPU performs SEI parameter parsing, which involves parsing the extracted user-defined SEI message and reading the version, mode, mask stream length, valid block mask, and alphaU / alphaV parameter stream.

[0179] The CPU performs parameter decoding. It decodes the compressed parameter stream in SEI, restores the effective block mask by using a hybrid RLE / Rice method, and restores the alphaU and alphaV parameters of each effective block by using differential Rice or the original 16-bit mode.

[0180] The CPU caches parameters and writes the parsed alphaU and alphaV block-level recovery parameters into the CPU-side parameter array.

[0181] The CPU uploads parameters, writes block-level parameters to GPU textures or buffers through CPU-writable temporary resources, and copies them to parameter resources that can be read by GPU post-processing.

[0182] Finally, a synchronization and merging process is performed to ensure that the NV12 image resources and the corresponding frame's block-level parameter resources for the decoded output are ready before GPU post-processing.

[0183] Step 340: GPU post-processing, quality statistics, and display output.

[0184] Specifically, the GPU performs texture sampling. The GPU establishes reading channels for the luminance plane and chrominance plane of the decoded NV12 image data, and reads the block-level recovery parameter texture containing alphaU and alphaV.

[0185] The GPU performs basic restoration by reading the basic U / V values ​​corresponding to the current 2×2 block in the NV12 chromaticity plane and calculating the average brightness of the 2×2 block.

[0186] The GPU performs enhanced restoration. For blocks containing valid alpha parameters, the GPU performs pixel-by-pixel chroma enhancement restoration according to the formulas U'=U+alphaU×(Y-meanY) and V'=V+alphaV×(Y-meanY); where U' and V' represent the chroma component values ​​in the corresponding image block after downsampling restoration, U and V represent the chroma components in the downsampled block, alphaU and alphaV represent the restoration parameters, Y represents the luminance component of the downsampled block, and meanY represents the average luminance of the downsampled block.

[0187] In addition, the GPU can also access reference data. In scenarios such as image quality evaluation, result comparison, or implementation verification, reference image data can be accessed for comparison of restoration effects and quality statistics.

[0188] Furthermore, the GPU can also perform post-processing scheduling, executing chroma restoration, color output, and quality statistics in parallel through computation shaders to generate BGRA output resources.

[0189] Finally, the output is displayed, and the GPU output is rendered to a display terminal or used for subsequent quality analysis.

[0190] Figures 3a-3b The specific details of the corresponding embodiments are similar to those in the corresponding embodiments above, and will not be repeated here.

[0191] The embodiments provided in this disclosure first employ GPU block-level statistics and parametric texture optimization techniques.

[0192] Specifically, the embodiments of this disclosure include 2×2 chroma block processing: each GPU thread uses a 2×2 pixel block as the basic processing unit to complete block-level statistics while generating the NV12 luminance plane and chroma plane; 3×3 neighborhood access: read 3×3 neighboring pixels around the current 2×2 block for calculating chroma gradient, high-frequency determination and luminance-chroma correlation.

[0193] This disclosure also includes parameter packet storage: the two 16-bit recovery parameters, alphaU and alphaV, are packetized and written into the GPU statistical texture, reducing the bandwidth of statistical results and the amount of data read back by the CPU. Furthermore, this disclosure also includes planar resource access: encoded / decoded image resources are directly written and read using the UAV / SRV access methods of the NV12 luma plane and chroma plane, reducing intermediate format conversions.

[0194] Secondly, the embodiments disclosed herein have optimized parameter encoding and SEI compression.

[0195] Specifically, the embodiments of this disclosure include effective block mask encoding: the CPU side generates an effective block mask based on whether alphaU and alphaV are zero, and uses a hybrid RLE / Rice method to compress consecutive invalid blocks or effective block states.

[0196] This disclosure also includes alpha parameter encoding: the alphaU and alphaV parameters of the valid block are differentially expressed using the previous value, and Rice encoding or original 16-bit writing is selected according to the difference size.

[0197] This disclosure also includes SEI payload organization: the SEI payload contains a version number, a mode field, a mask stream length, a compressed mask stream, and a compressed alpha parameter stream, which facilitates the decoding end to quickly recover parameters.

[0198] In this embodiment, parallel encoding is implemented, the mask encoding can be executed asynchronously by the CPU working thread, and the alpha parameter encoding is completed in the main processing path, thereby reducing the SEI generation latency.

[0199] Furthermore, the embodiments disclosed herein also include GPU recovery and quality statistics optimization.

[0200] Specifically, embodiments of this disclosure include block-level parameter sampling: the GPU post-processing stage reads the parameter texture to obtain the alphaU and alphaV parameters corresponding to the current 2×2 block.

[0201] This disclosure also includes luminance-driven restoration: performing U / V restoration on each pixel based on the difference between the current luminance and the average luminance of the 2×2 block, preserving chromatic details of the desktop edges and text areas.

[0202] This disclosure includes a dual-output comparison: the GPU simultaneously outputs the enhanced BGRA result and the basic recovery result, facilitating comparison before and after recovery in implementation and verification scenarios.

[0203] This disclosure includes a quality statistical reduction: the computation shader reduces the error before and after recovery within the thread group, and writes the statistical results into a structured buffer for CPU to read back and analyze.

[0204] Finally, this disclosure also employs pipelined parallel technology.

[0205] Specifically, this disclosure adopts a CPU-GPU heterogeneous pipeline: the CPU is responsible for logic control, parameter encoding, SEI encapsulation and SEI parsing, while the GPU and video encoding / decoding unit are responsible for image preprocessing, hardware encoding, hardware decoding and recovery calculation, so as to realize the overlapping execution of heterogeneous tasks.

[0206] This disclosure implements multi-frame pipelined processing: encoding output, SEI parameter generation, bitstream buffering, decoding submission, and GPU post-processing can form a pipeline around the frame order, improving end-to-end throughput.

[0207] In addition, this disclosure also includes a synchronization and rendezvous mechanism: during the bitstream output stage, video bitstream packets and SEI messages are associated in frame order to ensure that the decoded image and corresponding block-level parameters are ready before GPU post-processing at the decoding end.

[0208] In addition, this disclosure also includes asynchronous operations: using asynchronous GPU operations, temporary resources, and queue scheduling to reduce CPU-GPU synchronization wait and reduce performance bottlenecks.

[0209] The embodiments provided in this disclosure fully port the block-level adaptive sampling algorithm to the GPU, significantly improving processing speed while maintaining algorithm accuracy. Furthermore, this disclosure designs a GPU block-level statistical texture and alpha parameter packaging strategy to reduce the bandwidth of statistical results and the amount of data read back by the CPU. Further still, this disclosure constructs a CPU-GPU hybrid processing architecture, enabling GPU acceleration for computationally intensive tasks and CPU processing for logic control tasks. In addition, this disclosure also implements GPU-parallel high-frequency chromaticity detection, luminance-chromaticity correlation determination, and alphaU / alphaV linear recovery parameter generation, significantly improving efficiency compared to CPU serial processing. Finally, this disclosure integrates a hardware encoder, a hardware decoder, and an SEI parameter processing pipeline to achieve an end-to-end GPU-accelerated video processing workflow.

[0210] The embodiments provided in this disclosure utilize GPU parallel computing to parallelize the block-level high-frequency analysis, correlation determination, and alpha linear recovery parameter generation, which originally required CPU serial processing. This fully leverages the massive parallelism of the GPU and combines it with a hardware encoder, hardware decoder, and CPU-side SEI parameter processing pipeline to achieve end-to-end GPU-accelerated processing. Simultaneously, supplementary information for each image block is obtained through block-level high-frequency analysis, correlation determination, and alpha linear recovery parameter generation, thereby restoring the image block based on the supplementary information and improving the restoration accuracy.

[0211] Thirdly, embodiments of this disclosure also provide an image data processing method applied to a GPU for performing encoding operations. The method includes: the GPU processing color space information in video frame image data to obtain supplementary information of the initial image data, wherein the video frame image data is obtained by downsampling the initial image data, and the supplementary information is used to identify the linear relationship between the chroma and luminance components in the initial image data; the GPU encoding the video frame image data to generate a video frame bitstream of the initial image data; and sending the video bitstream of the initial image data to a central processing unit (CPU), so that after the CPU encodes the supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data, it outputs an encoded bitstream of the initial image data based on the supplementary information bitstream and the video frame bitstream.

[0212] The embodiments provided in this disclosure are similar to those provided in the first aspect above, except that the execution subject is different. The embodiments in this disclosure are described with GPU as the execution subject, while the embodiments provided in the first aspect above are described with CPU as the execution subject. The specific implementation process is the same, and will not be repeated here.

[0213] Fourthly, this disclosure also provides an image data processing method applied to a GPU for performing decoding operations. The method includes: the GPU receiving an encoded bitstream from a central processing unit (CPU), wherein the encoded bitstream includes a video frame bitstream and a supplementary information bitstream; the GPU decoding the video frame bitstream to obtain video frame image data; and the GPU reconstructing the video frame image data based on the supplementary information to obtain initial image data corresponding to the encoded bitstream, wherein the supplementary information is obtained by the CPU decoding the supplementary information bitstream, and the supplementary information is used to identify the linear relationship between the chroma and luminance components in the initial image data, so as to indicate the reconstruction of the chroma components in the video frame image data.

[0214] The embodiments provided in this disclosure are similar to those provided in the second aspect above, except that the execution subject is different. The embodiments in this disclosure are described with GPU as the execution subject, while the embodiments provided in the first aspect above are described with CPU as the execution subject. The specific implementation process is the same, and will not be repeated here.

[0215] Based on the same inventive concept as the above-described solutions, this disclosure also provides an image data processing system.

[0216] Fifthly, embodiments of this disclosure also provide an image data processing system.

[0217] Figure 4 This is a structural block diagram of an image data processing system provided in an embodiment of the present disclosure.

[0218] Further reference Figure 4 As an implementation of the methods shown in the above figures, the system embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0219] See Figure 4 The image data processing system includes a central processing unit (CPU) 401 and a graphics processing unit (GPU) 402. The GPU 402 processes the color space information in the initial image data to obtain supplementary information, which identifies the linear relationship between the chroma and luminance components in the initial image data. The CPU 401 encodes the supplementary information to generate a supplementary information bitstream. The GPU 402 also encodes video frame image data to generate a video frame bitstream of the initial image data, where the video frame image data is obtained by downsampling the initial image data. The CPU 401 further outputs the encoded bitstream of the initial image data based on the supplementary information bitstream and the video frame bitstream.

[0220] In this embodiment, the specific processing of CPU 401 and GPU 402 in the image data processing system and the resulting technical effects can be found in [reference 1]. Figure 1 The relevant descriptions of steps 101-103 in the corresponding embodiments will not be repeated here.

[0221] In some embodiments, the CPU 401 is further configured to call the GPU to divide the initial image data into multiple image blocks; wherein, the number of supplementary information entries is multiple, with one supplementary information entry corresponding to one image block.

[0222] CPU401 is further used to encode the supplementary information of multiple image blocks separately to generate the supplementary information bitstream of each image block; GPU402 is further used to encode multiple downsampling blocks separately to generate the video frame bitstream of each image block, wherein the video frame image data includes multiple downsampling blocks, and one downsampling block corresponds to one image block.

[0223] In some embodiments, the CPU401 is further configured to associate the supplementary information bitstream of each image block with the video frame bitstream of the corresponding image block to obtain the encoded bitstream of multiple image blocks and output it.

[0224] In some embodiments, the CPU is further configured to: determine a first locally available storage resource and determine a second storage resource in the GPU for storing supplementary information; further, the GPU is further configured to write supplementary information of the initial image data into the second storage resource; the CPU is further configured to copy the supplementary information in the second storage resource into the first storage resource.

[0225] In some embodiments, for each image block, the GPU is further used to calculate the chromaticity gradient, the correlation between the luminance component and the chromaticity component, and the linear relationship parameters corresponding to the image block, to obtain supplementary information about the image block.

[0226] In some embodiments, the supplementary information includes validity information and restoration parameter information. The validity information is used to identify whether there is a linear relationship between the chroma components and the luminance components in the image block, and the restoration parameter information is used to identify the linear relationship parameters between the chroma components and the luminance components in the image block. Specifically, the CPU is further configured to: read the supplementary information of multiple image blocks. Further, for each image block, the CPU is specifically configured to encode the validity information and the restoration parameter information to obtain encoded information; and to encapsulate the encoded information to obtain the supplementary information bitstream of the image block.

[0227] In some embodiments, the CPU is further configured to send the raw video data to the GPU; the GPU is further configured to perform color space conversion on the raw video data to obtain initial image data; and to downsample the initial image data to obtain video frame image data.

[0228] In some embodiments, the GPU is also used to perform parallel processing on the color space information of multiple image blocks using a thread-level parallel processing method to obtain supplementary information for each of the multiple image blocks.

[0229] In some embodiments, the CPU is further configured to receive the encoded bitstream, wherein the encoded bitstream includes a video frame bitstream and a supplementary information bitstream; the CPU is further configured to decode the supplementary information bitstream to obtain multiple pieces of supplementary information; the GPU is further configured to decode the video frame bitstream to obtain video frame image data including multiple downsampled blocks; wherein one downsampled block corresponds to one piece of supplementary information; the GPU is further configured to restore the corresponding downsampled blocks based on each piece of supplementary information to obtain the initial image data corresponding to the encoded bitstream, wherein the initial image data includes multiple image blocks, one image block corresponds to one downsampled block, the supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the image block, and the GPU restores the chroma component in the downsampled block according to the linear relationship.

[0230] In some embodiments, the CPU is specifically used to determine the decoding parameters required to decode the video frame bitstream and submit the decoding parameters to the GPU, wherein the decoding parameters include at least one of the image parameters, quantization matrix and slice control information corresponding to the video frame bitstream; the GPU is specifically used to decode the video frame bitstream based on the decoding parameters.

[0231] In some embodiments, the CPU is specifically used to parse the supplementary information bitstream, determine the configuration parameters and compression parameter stream of the supplementary information bitstream, and decode the compression parameter stream based on the configuration parameters to obtain multiple pieces of supplementary information.

[0232] In some embodiments, the CPU is further configured to write supplementary information to a locally available first storage resource and copy the supplementary information in the first storage resource to a second storage resource of the GPU; wherein the second storage resource also stores video frame image data.

[0233] In some embodiments, the supplementary information includes validity information and restoration parameter information. The validity information is used to identify whether there is a linear relationship between the chroma and luminance components in the image block, and the restoration parameter information is used to identify the linear relationship parameters between the chroma components and luminance components in the image block. Specifically, the GPU determines the chroma component information and luminance component information corresponding to each downsampled block; in response to the validity information indicating a linear relationship between the chroma and luminance components in the image block corresponding to the downsampled block, it restores the downsampled block according to the restoration parameter information, the chroma component information, and the luminance component information of the downsampled block to obtain the image block corresponding to the downsampled block.

[0234] In some embodiments, the GPU is also used to restore the initial image data in color space to obtain the original video data.

[0235] The specific details of the system provided in this disclosure are similar to those in the corresponding method embodiments described above, and will not be repeated here.

[0236] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the image data processing method described in any of the above embodiments.

[0237] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 5 As shown, the electronic device 500 of this embodiment includes a processor 501 and a memory 502; wherein, the memory 502 is used to store computer execution instructions; the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments. For example, the electronic device 500 can be a general-purpose processor, a graphics processing device, a neural network computing device, or a graph neural network computing device.

[0238] In some embodiments, the memory 502 can be either standalone or integrated with the processor 501.

[0239] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0240] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0241] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0242] Bus 503 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0243] This disclosure also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the steps of the image data processing method of any of the above method embodiments.

[0244] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image data processing method according to any of the above embodiments.

[0245] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0246] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0247] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0248] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods in the various embodiments of this application.

[0249] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0250] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0251] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0252] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0253] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0254] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image data processing method, characterized in that, The image data processing method is applied to the central processing unit (CPU) in a terminal device, and the method includes: The CPU calls the graphics processing unit (GPU) to process the color space information in the initial image data to obtain supplementary information of the initial image data. The supplementary information is used to identify the linear relationship between the chromaticity component and the luminance component in the initial image data. The CPU encodes the supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data, and calls the GPU to encode the video frame image data to generate a video frame bitstream of the initial image data, wherein the video frame image data is obtained by downsampling the initial image data; The CPU outputs the encoded bitstream of the initial image data based on the supplementary information bitstream of the initial image data and the bitstream of the video frames.

2. The method according to claim 1, characterized in that, The method further includes: The CPU calls the GPU to divide the initial image data into multiple image blocks; wherein, the supplementary information consists of multiple entries, with one entry of supplementary information corresponding to one image block; The CPU encodes supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data, and calls the GPU to encode video frame image data to generate a video frame bitstream of the initial image data, including: The CPU encodes the supplementary information of the plurality of image blocks respectively to generate a supplementary information bitstream for each image block, and calls the GPU to encode the plurality of downsampling blocks respectively to generate a video frame bitstream for each image block. The video frame image data includes a plurality of downsampling blocks, and one downsampling block corresponds to one image block.

3. The method according to claim 2, characterized in that, The CPU, based on the supplementary information bitstream of the initial image data and the video frame bitstream, outputs the encoded bitstream of the initial image block, including: The CPU associates the supplementary information bitstream of each image block with the video frame bitstream of the corresponding image block to obtain the encoded bitstream of the multiple image blocks, and then outputs it.

4. The method according to claim 1, characterized in that, The method further includes: The CPU determines a first locally available storage resource and a second storage resource in the GPU for storing the supplementary information; After the CPU obtains supplementary information from the initial image data, the method further includes: The CPU calls the GPU to write supplementary information of the initial image data into the second storage resource; The CPU copies the supplementary information from the second storage resource to the first storage resource.

5. The method according to claim 2, characterized in that, The CPU calls the graphics processing unit (GPU) to process the color space information in the initial image data to obtain supplementary information of the initial image data, including: For each image block, the CPU calls the GPU to calculate the chromaticity gradient, the correlation between the luminance component and the chromaticity component, and the linear relationship parameters corresponding to the image block, thereby obtaining supplementary information about the image block.

6. The method according to claim 5, characterized in that, The supplementary information includes validity information and recovery parameter information. The validity information is used to identify whether there is a linear relationship between the chroma components and the luminance components in the image block, and the recovery parameter information is used to identify the linear relationship parameters between the chroma components and the luminance components in the image block. The CPU encodes the supplementary information of the plurality of image blocks to generate a supplementary information bitstream for each image block, including: The CPU reads supplementary information from the plurality of image blocks; For each image block, the CPU encodes the validity information and the recovery parameter information to obtain encoded information; and encapsulates the encoded information to obtain the supplementary information bitstream of the image block.

7. The method according to claim 1, characterized in that, The method further includes: The CPU sends the raw video data to the GPU; The CPU calls the GPU to perform color space conversion on the original video data to obtain the initial image data; The CPU calls the GPU to downsample the initial image data to obtain the video frame image data.

8. The method according to claim 2, characterized in that, The CPU calls the graphics processing unit (GPU) to process the color space information in the initial image data to obtain supplementary information of the initial image data, including: The CPU calls the graphics processing unit (GPU) to perform parallel processing on the color space information of multiple image blocks to obtain supplementary information for each of the multiple image blocks.

9. An image data processing method, characterized in that, The image data processing method is applied to the CPU in a terminal device, and the method includes: The CPU receives the encoded bitstream, wherein the encoded bitstream includes a video frame bitstream and a supplementary information bitstream; The CPU decodes the supplementary information bitstream to obtain supplementary information; and calls the GPU to decode the video frame bitstream to obtain video frame image data. The CPU calls the GPU to restore the video frame image data based on the supplementary information to obtain the initial image data corresponding to the encoded bitstream, wherein the supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the initial image data.

10. The method according to claim 9, characterized in that, The video frame image data includes multiple downsampling blocks, and the supplementary information includes multiple entries, with one downsampling block corresponding to one supplementary information entry. The CPU invokes the GPU to reconstruct the video frame image data based on the supplementary information to obtain the initial image data corresponding to the encoded bitstream, including: The CPU calls the GPU to restore the chroma components in the corresponding downsampling blocks based on the linear relationship in each of the supplementary information, so as to obtain the initial image data corresponding to the encoded bitstream. The initial image data includes multiple image blocks, and one image block corresponds to one downsampling block.

11. The method according to claim 9, characterized in that, The CPU calls the GPU to decode the video frame bitstream, including: The CPU determines the decoding parameters required to decode the video frame bitstream and submits the decoding parameters to the GPU. The decoding parameters include at least one of the image parameters, quantization matrix, and slice control information corresponding to the video frame bitstream. The CPU calls the GPU to decode the video frame stream based on the decoding parameters.

12. The method according to claim 9, characterized in that, The CPU decodes the supplementary information bitstream to obtain supplementary information, including: The CPU parses the supplementary information bitstream to determine the configuration parameters and compression parameter stream of the supplementary information bitstream; The CPU decodes the compressed parameter stream based on the configuration parameters to obtain the supplementary information.

13. The method according to claim 9, characterized in that, The method further includes: The CPU writes the supplementary information into a locally available first storage resource and copies the supplementary information from the first storage resource to a second storage resource of the GPU; wherein the second storage resource also stores the video frame image data.

14. The method according to claim 10, characterized in that, The supplementary information includes validity information and recovery parameter information. The validity information is used to identify whether there is a linear relationship between the chroma components and the luminance components in the image block, and the recovery parameter information is used to identify the linear relationship parameters between the chroma components and the luminance components in the image block. The CPU calls the GPU to restore the corresponding downsampling blocks based on the supplementary information to obtain the initial image data corresponding to the encoded bitstream, including: For each of the downsampling blocks, The CPU calls the GPU to determine the chroma component information and luminance component information corresponding to the downsampling block; In response to the validity information indicating a linear relationship between the chroma and luminance components in the image block corresponding to the downsampled block, the downsampled block is restored based on the recovery parameter information, the chroma component information, and the luminance component information of the downsampled block to obtain the image block corresponding to the downsampled block.

15. The method according to claim 9, characterized in that, The method further includes: The CPU calls the GPU to restore the initial image data in color space to obtain the original video data.

16. An image data processing method, characterized in that, The image data processing method is applied to a GPU, and the method includes: The GPU processes the color space information in the video frame image data to obtain supplementary information of the initial image data. The video frame image data is obtained by downsampling the initial image data. The supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the initial image data. The GPU encodes the video frame image data to generate a video frame bitstream of the video frame image data; The GPU sends the video bitstream of the video frame image data to the central processing unit (CPU), so that the CPU encodes the supplementary information of the initial image data, generates the supplementary information bitstream of the initial image data, and outputs the encoded bitstream of the initial image data based on the supplementary information bitstream and the video frame bitstream.

17. An image data processing method, characterized in that, The image data processing method is applied to a GPU, and the method includes: The GPU receives an encoded bitstream from the central processing unit (CPU), wherein the encoded bitstream includes a video frame bitstream and a supplementary information bitstream; The GPU decodes the video frame bitstream to obtain video frame image data; The GPU restores the video frame image data based on the supplementary information to obtain the initial image data corresponding to the encoded bitstream. The supplementary information is obtained by the CPU decoding the supplementary information bitstream. The supplementary information is used to identify the linear relationship between the chroma component and the luminance component in the initial image data, so as to indicate the restoration of the chroma component in the video frame image data.

18. An image data processing system, characterized in that, The system includes a central processing unit (CPU) and a graphics processing unit (GPU), wherein, The GPU is used to process the color space information in the initial image data to obtain supplementary information of the initial image data, wherein the supplementary information is used to identify the linear relationship between the chromaticity component and the luminance component in the initial image data; The CPU is used to encode the supplementary information of the initial image data to generate a supplementary information bitstream of the initial image data; The GPU is also used to encode video frame image data to generate a video frame bitstream of the initial image data; wherein the video frame image data is obtained by downsampling the initial image data; The CPU is also used to output the encoded bitstream of the initial image data based on the supplementary information bitstream and the video frame bitstream of the initial image data.

19. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the image data processing method according to any one of claims 1-17.

20. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the image data processing method according to any one of claims 1-17.

21. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the steps of the image data processing method according to any one of claims 1-17.

Images