Video processing apparatus, method and storage medium

Abstract

The present disclosure provides a video processing device, method and storage medium. The device comprises a graphics processor and a controller, the graphics processor comprising a decoding engine and a post-processing engine. The controller configures a shared surface to receive decoded data output by the decoding engine and as a sampling reading target of the post-processing engine; creates a synchronization primitive and associates it with a decoding task for the shared surface, so that the synchronization primitive enters a completion state after the decoding task is completed; generates post-processing task submission information including cache access control information corresponding to the shared surface; submits the post-processing task according to the submission information, so that the post-processing task is executed after the decoding task is completed as indicated by the synchronization primitive, and the graphics processor processes a reading request issued by the post-processing engine for the shared surface according to a cache access strategy indicated by the cache access control information, so that the post-processing engine obtains the decoded data and performs a post-processing operation, thereby reducing data movement, storage bandwidth occupation and resource coordination overhead.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a video processing apparatus, method and storage medium. Background Technology

[0002] After video streams are decoded, they typically require post-processing such as rotation, color space conversion, and color balancing. In existing video processing systems, decoding and post-processing tasks are often performed by different processing units. Decoding results are usually first written to the decoding output buffer and then read by the post-processing engine according to its supported resource formats. To accommodate the access methods of different processing units, the system may transfer data between system memory and graphics memory, or reallocate post-processing input objects in graphics memory and copy the decoding results. These processes increase interconnect bandwidth and storage bandwidth usage and prolong the processing chain from decoding output to the post-processing stage. When multiple video streams run concurrently, decoding output objects and post-processing input objects may also occupy storage resources simultaneously, increasing the pressure on resource allocation and task scheduling. Furthermore, the decoding write and post-processing read operations need to coordinate task completion status, buffer access order, and data accessibility status in the buffer hierarchy. Therefore, how to reduce the data connection and resource coordination overhead between video decoding and video post-processing is a technical problem that needs to be solved in the field of video processing. Summary of the Invention

[0003] This disclosure provides a video processing apparatus including a graphics processor (GPU) and a controller. The GPU includes a decoding engine and a post-processing engine. The controller configures a shared surface to receive decoded data output by the decoding engine and use it as a sampling target for the post-processing engine. It creates a synchronization primitive and associates it with a decoding task for the shared surface, causing the synchronization primitive to enter a state indicating completion of the decoding task upon completion. It generates submission information for the post-processing task, including cache access control information corresponding to the shared surface. The post-processing task is submitted according to the submission information. The post-processing task executes when the state of the synchronization primitive indicates completion of the decoding task. During execution, the GPU processes the read requests issued by the post-processing engine for the shared surface according to the cache access policy indicated by the cache access control information, and the post-processing engine obtains the decoded data from the shared surface and performs post-processing operations.

[0004] This disclosure also provides a video processing method, which includes: configuring a shared surface to receive decoded data output by a decoding engine and use it as a sampling target for a post-processing engine; creating a synchronization primitive and associating the synchronization primitive with a decoding task for the shared surface, such that the synchronization primitive enters a state indicating the completion of the decoding task after the decoding task is completed; generating submission information for a post-processing task, the submission information including cache access control information corresponding to the shared surface; submitting the post-processing task according to the submission information, wherein the post-processing task is executed when the state of the synchronization primitive indicates the completion of the decoding task, and during execution, processing the read request issued by the post-processing engine for the shared surface according to the cache access policy indicated by the cache access control information, and enabling the post-processing engine to obtain decoded data from the shared surface and perform post-processing operations.

[0005] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described video processing method.

[0006] This disclosure provides a video processing apparatus, method, and storage medium. A controller configures a shared surface as the decoding data writing object of the decoding engine and the sampling reading target of the post-processing engine, enabling decoding and post-processing tasks to complete data connection around the same shared surface, thereby reducing the processing steps of separately allocating post-processing input objects and submitting explicit data copying tasks. The controller creates synchronization primitives and associates them with decoding tasks, so that the synchronization primitives enter a completed state after the decoding task is completed. The controller submits post-processing tasks according to submission information carrying cache access control information, and the post-processing tasks are executed after the synchronization primitives indicate that the decoding task is completed. When executing post-processing tasks, the graphics processor processes the read requests issued by the post-processing engine for the shared surface according to the corresponding cache access strategy. Thus, the access objects of the shared surface, the execution dependencies between decoding and post-processing tasks, and the cache access method during post-processing reads can be coordinated in the same control flow, thereby reducing storage bandwidth consumption and resource coordination overhead, and reducing intermediate processing steps before the decoded output enters the post-processing stage. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the structure of the video processing system provided in this disclosure.

[0008] Figure 2 This is a schematic diagram of the structure of the video processing apparatus provided in this disclosure.

[0009] Figure 3 This is a flowchart illustrating the video processing method provided in this disclosure.

[0010] Figure 4 This is a schematic diagram of the shared surface configuration process provided in this disclosure.

[0011] Figure 5 This is a schematic diagram of the command queue scheduling structure of a video processing device.

[0012] Figure 6 This is a schematic diagram of the graphics processor cache hierarchy provided in this disclosure.

[0013] Figure 7 This is a schematic diagram of the graphics processor sampling structure. Detailed Implementation

[0014] The technical solutions in this disclosure will now be clearly and completely described with reference to the accompanying drawings.

[0015] Figure 1 This is a schematic diagram of the structure of the video processing system provided in this disclosure, such as... Figure 1 As shown, the video processing system 100 includes a video input subsystem 110, a host processing subsystem 120, a graphics storage subsystem 130, a video output subsystem 140, an interconnection subsystem 150, and a video processing device 160. The video input subsystem 110, host processing subsystem 120, graphics storage subsystem 130, video output subsystem 140, and video processing device 160 are communicatively connected through the interconnection subsystem 150.

[0016] The video input subsystem 110 includes a video source 111 and a video input interface 112. The video source 111 provides a video stream to be processed, including captured video, local video, or network video. The video input interface 112 receives the video stream to be processed and provides it to the host processing subsystem 120 or the video processing device 160.

[0017] The host processing subsystem 120 includes a processor 121 and a system memory 122. The processor 121 runs application programs, drivers, or video processing runtime programs. The system memory 122 stores the programs executed by the processor 121, as well as task descriptions, resource information, and task states generated during video processing.

[0018] The graphics storage subsystem 130 includes a graphics memory 131. The graphics memory 131 provides storage resources for video decoding and video post-processing.

[0019] The video output subsystem 140 includes at least one of a display device 141 and a network output device 142. The display device 141 displays the processed video image, and the network output device 142 sends the processed video content to a remote device or a communication peer.

[0020] The interconnect subsystem 150 includes a bus that transmits video data, task requests, resource access requests, and task status between different components of the video processing system 100.

[0021] The video processing device 160 is located in the video processing link between the video input subsystem 110 and the video output subsystem 140, and is communicatively connected to the video input interface 112, the host processing subsystem 120, the graphics storage subsystem 130, and the video output subsystem 140, respectively. The video processing device 160 receives the video stream to be processed, performs video decoding and post-processing, and outputs the processed video content. In a real-time communication scenario, the video stream to be processed can be a Web Real-Time Communication (WebRTC) video stream. Video hardware decoding can offload decoding operations from the Central Processing Unit (CPU) to the graphics processing unit (GPU), and the decoding results typically reside in Video Random Access Memory (VRAM). Post-processing operations such as rotation and color balancing can be performed by the shader cores or dedicated bit block transfer units (Blitters) in the GPU's 3D engine, and the decoding results can be read.

[0022] Figure 1 The video processing node 190 and the buffer conversion module 191 are also shown to illustrate the locations where buffer conversion may occur between video decoding and video post-processing.

[0023] In one cross-domain copy path, data in the decoded output buffer is transferred to system memory 122 via interconnect subsystem 150, processed by processor 121 or other image processing architecture, and then transferred to graphics memory 131. This path involves round-trip data transfer between system memory 122 and graphics memory 131.

[0024] Taking a 3840×2160 resolution, 8-bit per component, and 60 frames per second (fps) luma-chroma 4:2:0 (YUV420) video stream as an example, the raw frame data rate, excluding stride alignment and metadata, is approximately 0.75 GB / s. Actual storage access volume is also affected by bit depth, planar stride, storage alignment, and the number of read / write operations. Continuous cross-domain replication consumes 150 Mbps of the interconnect subsystem's transmission bandwidth and forces the decoded output to undergo intermediate steps such as data download, host-side processing, and data upload before entering the post-processing stage, thus adversely affecting end-to-end latency-sensitive real-time communication scenarios such as WebRTC.

[0025] In one internal reallocation path, cache conversion module 191 allocates a post-processing input object in graphics memory 131 and copies data from the decoded output object to that post-processing input object. The post-processing engine then reads the post-processing input object, and this path still generates additional graphics memory reads and writes. For mobile devices or multi-instance cloud servers, additional graphics memory reads and writes increase power consumption and reduce overall energy efficiency.

[0026] When processing multiple video streams concurrently, different video streams may occupy decoding output objects and post-processing input objects respectively. The number of intermediate objects increases with the number of video stream instances, thereby consuming graphics storage resources and potentially increasing the probability of storage fragmentation or resource allocation failure. In this path, the same decoded data may reside separately in the decoding output object and the post-processing input or output object, resulting in double memory residency; in multi-instance WebRTC concurrent scenarios with limited graphics memory resources, this may also lead to the risk of Out of Memory (OOM).

[0027] exist Figure 1 In the illustrated operation, video source 111 provides the video stream to be processed to video input interface 112. Processor 121 generates a video processing task based on the program in system memory 122 and submits the video processing task to video processing device 160 through interconnect subsystem 150. Video processing device 160 performs video decoding and video post-processing, and video output subsystem 140 receives the processed video content.

[0028] When establishing a shared surface between video decoding and video post-processing, the video processing apparatus 160 can reduce the intermediate object configuration and explicit data copying steps that the decoded output must pass through before entering the post-processing stage. In multi-instance video processing scenarios, with the number of intermediate objects reduced, the same graphics storage resources can be reserved for more video processing tasks. For 4K frame rate (e.g., frame rate not less than 60 frames per second) WebRTC video streams, provided that the resource format and storage layout are compatible and the driver supports direct sharing, this disclosure can shorten the flow path of video data within the graphics processor and reduce the number of graphics memory read / write operations, thereby reducing the load on the interconnect bus and the video memory controller, reducing system power consumption while meeting target image quality requirements, and improving the concurrent processing capability of multi-instance video streams.

[0029] Figure 2 This is a schematic diagram of the structure of the video processing apparatus provided in this disclosure, such as... Figure 2As shown, the video processing device 160 includes a graphics processor 161 and a controller 162. The graphics processor 161 includes a decoding engine 1611 and a post-processing engine 1612. The decoding engine 1611 has decoding capabilities such as entropy decoding, inverse transformation, and motion compensation. The output of the decoding engine 1611 is connected to the decoding write path in the graphics memory. The post-processing engine 1612 has processing capabilities such as image rotation, color space conversion, and color balancing. The post-processing engine 1612 can be implemented using a programmable shader unit or fixed-function post-processing hardware such as a dedicated bit block transfer unit (Blitter). The input of the post-processing engine 1612 is connected to the sampling read path in the graphics memory. The controller 162 is communicatively connected to the graphics processor 161 and has resource configuration, task submission, and timing scheduling functions. In terms of resource configuration, controller 162 can configure access attributes of the shared surface and register them with each engine; in terms of task scheduling, controller 162 can submit tasks to the decoding engine and post-processing engine respectively and manage the execution order; in terms of synchronization, controller 162 can generate and detect synchronization status flags to coordinate execution dependencies between different engines.

[0030] Figure 2 Shared surface 163 is also shown. Shared surface 163 is a logical resource in the graphics memory, possessing both decode-write and sample-read attributes: it is registered as a write target for decoding engine 1611 and a sample-read target for post-processing engine 1612. The decode-write path and sample-read path establish a correspondence with shared surface 163 through the same shared surface identifier and access the storage area corresponding to shared surface 163. Shared surface 163 in... Figure 2 The position in the code is used to express the access mapping relationship, and does not limit the physical encapsulation position of the shared surface 163.

[0031] In this disclosure, the graphics processor 161 provides the computing resources required for video decoding and post-processing. The decoding engine 1611 executes decoding tasks and writes the decoded data generated by the decoding tasks to a shared surface 163; the post-processing engine 1612 executes post-processing tasks and uses the shared surface 163 as a sampling target. The controller 162 is communicatively connected to the graphics processor 161 and performs task configuration and scheduling for both the decoding engine 1611 and the post-processing engine 1612.

[0032] Figure 3 This is a flowchart illustrating the video processing method provided in this disclosure. Based on the aforementioned video processing apparatus, the method is executed by the controller 162 in coordination with the decoding engine 1611 and the post-processing engine 1612, and specifically includes the following steps: In step S310, the controller 162 configures the shared surface 163 to receive the decoded data output by the decoding engine 1611 and use it as the sampling target of the post-processing engine 1612.

[0033] Specifically, controller 162 calls the video memory resource allocation interface to create a shared surface 163 and registers it with both the decoding channel and the post-processing channel. When registering with the decoding channel, the shared surface 163 is marked as a decoding write target, enabling the decoding engine 1611 to write the decoded data to this shared surface 163. When registering with the post-processing channel, the shared surface 163 is marked as a post-processing sampling read target, enabling the post-processing engine 1612 to initiate sampling reads from this shared surface 163. The same shared surface identifier is used for both registrations, and access descriptions suitable for decoding writes and sampling reads are established respectively. With compatible resource formats and storage layouts, this configuration does not require the controller to create a separate post-processing input surface or submit an explicit data copy task.

[0034] In step S320, controller 162 creates a synchronization primitive.

[0035] Controller 162 creates a synchronization primitive and associates it with a decoding task targeting shared surface 163. Controller 162 then submits the decoding task to decoding engine 1611, which performs a decoding write to shared surface 163. The synchronization primitive is in an incomplete state during the decoding task execution; upon completion, the synchronization state associated with the decoding task is updated, causing the synchronization primitive to enter a state indicating the completion of the decoding task. The synchronization primitive is used to establish an execution dependency between decoding engine 1611 and post-processing engine 1612, preventing post-processing engine 1612 from reading decoded data from shared surface 163 before the decoding task is complete.

[0036] In step S330, controller 162 generates submission information for the post-processing task.

[0037] The controller 162 constructs submission information for the post-processing task based on the type of the post-processing task and the attributes of the shared surface 163. The submission information includes cache access control information corresponding to the shared surface 163. This cache access control information is used to ensure that when the post-processing task reads the shared surface 163, the graphics processor 161 processes the read request issued by the post-processing engine 1612 for the shared surface 163 according to the corresponding cache access strategy. Specifically, by specifying how decoded and written data is reused in the post-processing read path within the cache hierarchy, the cache hit rate is maximized, and the situation where data is first written back to video memory and then read from video memory is reduced. The cache access control information corresponds to the shared surface 163 and is submitted along with the post-processing task as part of its submission information.

[0038] It should be noted that synchronization primitives can be created and associated with the decoding task before or at the time of submission; submission information can be constructed before, during, or after the decoding task is executed. Post-processing tasks can be submitted either after the synchronization primitive enters the completion state, or they can be submitted in advance and wait for the synchronization primitive before execution; regardless of the method used, post-processing tasks are executed when the state of the synchronization primitive indicates that the decoding task has been completed.

[0039] In step S340, controller 162 submits the post-processing task.

[0040] The post-processing task is executed when the state of the synchronization primitive indicates that the decoding task has been completed. In one implementation, the controller 162 detects the state of the synchronization primitive. When the state of the synchronization primitive indicates that the decoding task has not yet been completed, the controller 162 waits and does not submit the post-processing task to avoid read-write conflicts. When the state of the synchronization primitive switches to indicate that the decoding task has been completed, the controller 162 submits the post-processing task according to the constructed post-processing task submission information. When executing the post-processing task, the graphics processor 161 processes the read request issued by the post-processing engine 1612 for the shared surface 163 according to the cache access policy indicated by the cache access control information, and enables the post-processing engine 1612 to obtain the decoded data from the shared surface 163 according to the cache access policy, execute the corresponding post-processing operations, and finally output the processing result to the subsequent display or encoding stage.

[0041] In this disclosure, controller 162 configures shared surface 163 as both a decoding write target and a sampling read target, enabling decoding and post-processing tasks to complete data connection around the same shared surface 163. This reduces the need for separately allocating post-processing input objects and submitting explicit data copying tasks. Controller 162 creates synchronization primitives and associates them with decoding tasks, ensuring the synchronization primitives enter a completed state upon completion of the decoding task. Controller 162 submits post-processing tasks according to submission information carrying cache access control information. Post-processing tasks execute upon completion of the decoding task as indicated by the synchronization primitives. During post-processing task execution, graphics processor 161 processes read requests from post-processing engine 1612 for shared surface 163 according to a corresponding cache access strategy. Thus, the access objects of shared surface 163, the execution dependencies between decoding and post-processing tasks, and the cache access method during post-processing reads can be coordinated within the same control flow, reducing storage bandwidth consumption and resource coordination overhead, and minimizing intermediate processing steps before the decoded output enters the post-processing stage.

[0042] Figure 4 This is a schematic diagram of the shared surface configuration process provided in this disclosure. Figure 4 As shown, the shared surface configuration process includes the following steps: In step S410, controller 162 configures shared surface 163 based on composite allocation flag.

[0043] The composite allocation flag endows shared surface 163 with decoding write and sampling read attributes. The decoding write attribute enables shared surface 163 to receive decoded data output by decoding engine 1611, while the sampling read attribute enables shared surface 163 to serve as a sampling read target for post-processing engine 1612. The resource format, planar layout, stride, storage layout information, and access attributes of shared surface 163 are configured to meet the compatibility requirements of the decoding write path and the sampling read path. Specifically, controller 162 forms a composite allocation flag based on the resource requirements of the video task to be processed and writes the composite allocation flag into the creation parameters, resource description, or attribute configuration content of shared surface 163.

[0044] In the driver interface implementation, when creating the shared surface 163, the controller 162 passes the composite allocation flag as a creation parameter or surface attribute to the underlying allocation interface. The underlying allocation interface allocates the shared surface 163 according to the composite allocation flag and records the decoding write attribute and sampling read attribute of the shared surface 163.

[0045] In the Video Acceleration API (VA-API) implementation, controller 162 can create shared surfaces managed by VA-API using the vaCreateSurfaces function. When importing external buffers, controller 162 can provide information such as memory type, external buffer descriptor, and offset and stride of each plane through VA surface attributes (VASurfaceAttrib) when calling vaCreateSurfaces; when exporting shared surfaces, controller 162 can obtain the Direct Rendering Manager (DRM) PRIME shared handle or the Direct Memory Access Buffer (DMA-BUF) file descriptor through the vaExportSurfaceHandle function. Internal creation, external buffer import, and surface export are optional implementation paths and are not required to be executed simultaneously during the same surface creation process.

[0046] In the Gallium driver binding implementation, composite allocation flags are converted into a pipeline binding flag combination at the driver layer, including PIPE_BIND_VIDEO_DECODE_DPB and PIPE_BIND_SAMPLER_VIEW. PIPE_BIND_VIDEO_DECODE_DPB corresponds to the Decode Picture Buffer (DPB) binding flag, and PIPE_BIND_SAMPLER_VIEW corresponds to the sampled view binding flag. Shared surface 163 can also have PIPE_BIND_RENDER_TARGET binding capability to support shared surface 163 as a rendering target simultaneously.

[0047] In step S420, controller 162 registers shared surface 163 as a decoding write target.

[0048] The composite allocation flags include a Decode Picture Buffer (DPB) flag, which establishes the write relationship between the shared surface 163 and the decoding task. The controller 162 writes the resource identifier, resource description, or resource handle of the shared surface 163 into the registration content on the decoding task side, and makes the registration content carry the DPB flag. After receiving the decoding task, the decoding engine 1611 writes the decoding data into the shared surface 163 according to the registration result.

[0049] In step S430, the controller 162 registers the shared surface 163 as the sampling and reading target.

[0050] The composite allocation flag also includes a sampling view binding flag, which is used to establish a sampling and reading relationship between the shared surface 163 and the post-processing task. The controller 162 writes the resource identifier, resource description, or resource handle of the shared surface 163 into the registration content on the post-processing task side, and makes the registration content carry the sampling view binding flag. After the post-processing task is submitted, the post-processing engine 1612 uses the shared surface 163 as the sampling and reading target according to the registration result.

[0051] In this disclosure, the decoding write target and the sampling read target correspond to the same shared surface identifier. The shared surface identifier is used to identify the shared surface 163 and can be represented as a surface object identifier, resource handle, resource descriptor, memory object identifier, or driver resource identifier. The controller 162 associates the registration results of the decoding task side and the post-processing task side, so that the two registration results reference the same shared surface identifier.

[0052] There is no necessary order between steps S420 and S430. They can be executed in parallel or in any order, as long as both steps complete the registration for the same shared surface identifier.

[0053] In the resource management implementation, the two registration results reference the same underlying memory object. The decoding task side and the post-processing task side establish access descriptions respectively, but the two access descriptions reference the same resource handle, and no separate input resources are allocated for the post-processing task.

[0054] In the cross-engine sharing implementation, controller 162 can export the underlying video memory object corresponding to the shared surface 163 as a DMA-BUF file descriptor through the DRM PRIME sharing mechanism. The decoding task side and the post-processing task side establish access descriptions based on this file descriptor, respectively. The handle passing itself is used to share the underlying storage object and its metadata; when the resource format and storage layout are compatible, controller 162 does not need to submit an explicit data copying task from the decoding output object to the independent post-processing input object.

[0055] In this disclosure, the composite allocation flag enables the shared surface 163 to simultaneously possess both decoding write attributes and sampling read attributes, allowing the decoding engine 1611 and the post-processing engine 1612 to use the shared surface 163 according to their respective access methods. The same shared surface identifier ensures that the decoding write object and the sampling read object remain consistent at the resource identifier level, thereby reducing resource conversion and explicit data copying between different surface objects.

[0056] Figure 5 This is a schematic diagram of the command queue scheduling structure of a video processing device, such as... Figure 5 As shown, in the video processing apparatus 160, the graphics processor 161 includes a decoding engine 1611, a post-processing engine 1612, a command queue 1613 for decoding tasks, and a command queue 1614 for post-processing tasks. The controller 162 submits tasks to both command queues. Specifically, the command queue 1613 for decoding tasks carries decoding task commands and schedules the decoding engine 1611 to perform write operations on the shared surface 163. The command queue 1614 for post-processing tasks carries post-processing task commands and schedules the post-processing engine 1612 to sample and read decoded data from the shared surface 163.

[0057] Synchronization primitives include at least one of fences and semaphores. A fence is a task completion synchronization object that indicates the completion of a write to the shared surface 163 by the command queue 1613 of the decoding task. When a fence is used, the controller 162 creates the fence and associates it with the decoding task; after the decoding task is completed, the fence enters a state indicating that the decoding task is complete. The controller 162 can read the fence state through polling, callbacks, or event waiting.

[0058] A semaphore is used to establish an inter-queue synchronization object that creates execution dependencies between the command queue 1613 of the decoding task and the command queue 1614 of the post-processing task. When using a semaphore, the controller 162 associates the semaphore's signaling operation with the command queue 1613 of the decoding task and the semaphore's wait operation with the command queue 1614 of the post-processing task. After completing the write to the shared surface 163, the command queue 1613 of the decoding task performs a signaling operation on the semaphore; before executing the corresponding post-processing command, the command queue 1614 of the post-processing task performs a wait operation on the semaphore, and after the wait condition is met, it schedules the post-processing engine 1612 to read the shared surface 163. The signaling operation causes the semaphore to enter a state where the wait condition is met, and the wait operation is used to confirm whether the wait condition has been met. The semaphore does not carry decoded data, nor does it change the data content in the shared surface 163; it only transmits execution dependencies. The semaphore wait operation allows the command queue 1614 of the post-processing task to pause the execution of the corresponding post-processing command before the wait condition is met, and to continue execution after the wait condition is met; after the synchronization relationship between queues is configured, the controller 162 does not need to continuously poll the semaphore status.

[0059] In one implementation, the synchronization primitive includes both a fence and a semaphore. After completing the write to the shared surface 163, the command queue 1613 of the decoding task sets the fence and performs a signal operation on the semaphore. The controller 162 determines the submission timing of the post-processing task based on the decoding task completion status indicated by the fence, and the command queue 1614 of the post-processing task determines the execution timing of the post-processing task based on the inter-queue synchronization condition indicated by the semaphore. When the fence and semaphore are used together, they provide corresponding synchronization criteria for the submission and execution phases of the post-processing task, respectively.

[0060] Figure 6 This is a schematic diagram of the graphics processor cache hierarchy provided in this disclosure. Figure 6 As shown, the graphics processor 161 includes a decoding engine 1611, a post-processing engine 1612, and a cache level 1615. The cache level 1615 is a single-level or multi-level cache structure (e.g., a level 2 cache) located between the execution structure of the graphics processor 161 and the graphics memory 131. The cache level 1615 receives write data generated by the decoding engine 1611 and read requests generated by the post-processing engine 1612. The decoding engine 1611, the cache level 1615, and the shared surface 163 form a decoding write path, and the shared surface 163, the cache level 1615, and the post-processing engine 1612 form a post-processing read path.

[0061] In this disclosure, controller 162 generates submission information for post-processing tasks and controls the cache access strategy when the post-processing task reads the shared surface 163 through cache access control information in the submission information. Specifically, the cache access strategy may include a target cache level, which can be determined according to the cache structure of graphics processor 161 and the cache attributes supported by the driver, such as a level 2 cache. The driver can write cache attributes, cache access hints, or cache residency hints corresponding to the target cache level into the cache access control information, so that the post-processing task's read request for the shared surface 163 preferentially utilizes the corresponding target cache level. For decoded data that the decoding engine 1611 has written to the shared surface 163 and can still obtain from the target cache level, it can be provided to the post-processing engine 1612 through the target cache level; for decoded data that cannot be obtained from the target cache level, it can be read from the storage area corresponding to the shared surface 163.

[0062] By using cache access control information, the graphics processor 161 can utilize the target cache level to process read requests issued by the post-processing engine 1612 for the shared surface 163 during post-processing task execution. When valid decoded data exists in the target cache level, the graphics processor 161 can utilize that target cache level to respond to the read request, thereby increasing the likelihood that decoded data written to the cache hierarchy will be reused in the post-processing read path, reducing the likelihood of decoded data being written back to graphics memory and then read from graphics memory again, and reducing the memory bandwidth usage of the post-processing process. Even if the target cache level is not hit, the graphics processor 161 can still obtain decoded data from the storage area corresponding to the shared surface 163 to ensure the normal execution of the post-processing task.

[0063] In one driver interface implementation, cache residency control information can serve as a specific implementation of cache access control information. When the target graphics processor and driver support the corresponding cache control features, controller 162 can provide cache residency hints to the Global Memory Controller (GMEM Controller) through the L2 residency control interface, prompting that decoded data be preferentially retained in the L2 cache, thereby increasing the likelihood that decoded data remains accessible in cache level 1615. The specific interface name, calling method, and cache semantics can be determined based on the target platform.

[0064] In one instruction attribute implementation, the compiler can generate load instructions with cache hint attributes. For example, on NVIDIA platforms that support the corresponding cache access control features, built-in load functions with cache hint attributes, such as __ldcs and __ldcg, can be used; on AMD or other platforms, interfaces or instructions with similar cache access functions provided by the corresponding instruction set, driver, or compiler can be used to enable the post-processing task to read shared surface 163 according to the corresponding cache access attributes. The availability of the aforementioned cache hint instructions or L2 resident control interfaces depends on whether the target graphics processor, driver, and compiler support the corresponding cache control features. Cache hint attributes are used to influence the cache access method on the post-processing read side and do not guarantee that decoded data will continuously reside in the L2 cache.

[0065] In one task attribute implementation, controller 162 writes cache residency control information into the resource description, access attributes, or task submission description of the post-processing task. Graphics processor 161 determines the target cache level based on the corresponding description and, during the execution of the post-processing task, uses the target cache level to process read requests issued by post-processing engine 1612 for shared surface 163.

[0066] In an access path where no cache residency control information is configured, the decoded data generated by the decoding engine 1611 can first enter the cache level 1615 and then be written to the shared surface 163 in the graphics memory 131. When the post-processing engine 1612 performs a post-processing task, it can read data from the shared surface 163 in the graphics memory 131 and re-enter the read data into the cache level 1615. This path can include memory access from the cache level 1615 to the graphics memory 131, and memory access from the graphics memory 131 to the cache level 1615.

[0067] In the access path configured with cache residency control information, after the decoding task writes to shared surface 163, the cache residency control information can increase the likelihood that the decoded data remains accessible through cache level 1615 during post-processing task execution. When post-processing engine 1612 reads shared surface 163, it can utilize the existing accessible data in cache level 1615, thereby reducing the need to retrieve the same data from graphics memory 131 again. Even when no available decoded data exists in cache level 1615, post-processing engine 1612 can still retrieve the decoded data from graphics memory 131 corresponding to shared surface 163.

[0068] When synchronization primitives and cache residency control information are used together, the synchronization primitives are used to constrain the execution time of the post-processing task, so that the post-processing task is executed after the state indication of the synchronization primitive is completed, so that the post-processing engine 1612 reads the shared surface 163 after the decoding task is completed; the cache residency control information is used to control the cache access strategy adopted by the post-processing task when reading the shared surface 163, so that the graphics processor 161 uses the target cache level to process the read request issued by the post-processing engine 1612 for the shared surface 163.

[0069] In this disclosure, synchronization primitives enable post-processing tasks to execute after decoding tasks are completed, thereby coordinating the access order of the two tasks to the same shared surface 163. Cache resident control information enables the graphics processor 161 to utilize cache level 1615 to process read requests from the post-processing engine 1612 for the shared surface 163, thereby increasing the likelihood that the post-processing engine 1612 will utilize already accessible data in cache level 1615 and reducing the need to retrieve the same data from lower-level storage resources. Reduced storage read / write activity also correspondingly reduces the power consumption overhead from storage accesses.

[0070] Figure 7 This is a schematic diagram of the graphics processor sampling structure, such as... Figure 7 As shown, the graphics processor 161 includes a post-processing engine 1612 and a texture sampler 1616. The post-processing engine 1612 includes a shader unit 1612A. The texture sampler 1616 connects the storage access path of the shared surface 163 and the sampling processing path of the shader unit 1612A.

[0071] Shader unit 1612A performs programmable video post-processing. Shader unit 1612A samples and reads decoded data from shared surface 163 and performs at least one of the following post-processing operations on the decoded data: rotation processing, color space conversion processing, and color balance processing.

[0072] Texture sampler 1616 is configured to perform layout decoding on the storage address of the decoded data in shared surface 163 based on a sampling view of shared surface 163, and use the sampling result obtained from the layout decoding as the decoded data sampled and read from shared surface 163 by post-processing engine 1612. The sampling view describes the sampling access method of shared surface 163, including format information, plane information, size information, stride information, component mapping information, and storage layout information. The sampling view does not constitute another independent data object, nor does it require the controller to submit a task to copy the decoded data in shared surface 163 to an independent post-processing input object. Layout decoding refers to the texture sampler 1616 resolving the tile layout, address reordering layout, or other storage address mapping relationship according to the sampling view to determine the actual storage address corresponding to the logical sampling position. In the tile layout implementation, texture sampler 1616 determines the tile index and tile offset based on the sampling position; in the address reordering layout implementation, texture sampler 1616 determines the actual storage address according to the Swizzle rules in the sampling view.

[0073] In this disclosure, shader unit 1612A submits a sampling request containing sampling coordinates and plane selection relationships to texture sampler 1616. Texture sampler 1616 reads the sampling view, determines information such as data format, target plane, and storage layout, decodes the sampling coordinates into actual storage addresses, and reads the sampling data from shared surface 163 according to the actual storage addresses. Subsequently, texture sampler 1616 organizes the sampling results according to the component mapping relationship in the sampling view and provides the sampling results to shader unit 1612A. The layout decoding process is automatically completed by hardware, and shader unit 1612A does not need to directly calculate the tile index or swizzle address.

[0074] In one shader interface implementation, shader unit 1612A uses the texel fetch function to obtain texel data for a specified plane according to integer coordinates. This interface name is used to describe one implementation of the sampling instruction and does not constitute a necessary limitation.

[0075] In this disclosure, the decoded data includes luma-interleaved chroma dual-plane format data. This format data includes luma plane data and interleaved chroma plane data, with both planes corresponding to the same video frame or video processing unit. The luma plane data carries the luma component, and the interleaved chroma plane data carries the chroma components stored in an interleaved order. In the NV12 format implementation, the luma-interleaved chroma dual-plane format data includes one luma plane and one interleaved chroma plane. The luma plane carries the Y component, and the interleaved chroma plane carries interleaved Cb and Cr components. This interleaved chroma plane is also referred to as the UV plane in the NV12 format. When the interleaved chroma plane is measured in texels containing two chroma components, the number of sampling points in the horizontal and vertical directions is half the number of sampling points in the corresponding luma plane. The byte stride of the interleaved chroma plane is determined by the number of components contained in the chroma texels, the component bit width, storage alignment, and resource format.

[0076] When sampling decoded data, shader unit 1612A samples the luma plane data and interleaved chroma plane data in the luma-interleaved chroma dual-plane format data, respectively. Specifically, shader unit 1612A generates luma plane sampling requests and interleaved chroma plane sampling requests for the same target processing location. Texture sampler 1616 determines the corresponding storage address based on the sampling view or plane description of each plane, and returns the luma sampling result and chroma sampling result respectively.

[0077] The luminance plane sampling coordinates and the interleaved chrominance plane sampling coordinates can be converted according to the ratio of the number of plane sampling points: u_uv = u_y × (W_uv / W_y); v_uv = v_y × (H_uv / H_y); Where (u_y, v_y) represents the sampling coordinates of the luminance plane data, (u_uv, v_uv) represents the sampling coordinates of the interleaved chrominance plane data, W_y and H_y represent the number of horizontal and vertical sampling points in the luminance plane, respectively, and W_uv and H_uv represent the number of horizontal and vertical sampling points in the interleaved chrominance plane, respectively. In the NV12 format example, the ratio of the number of sampling points for both is 1 / 2.

[0078] The specific execution process of dual-plane sampling in shader unit 1612A is as follows: First, the shader unit 1612A is configured with two independent texture sampling binding points: the first binding point binds to the luma plane texture in a single-channel 8-bit normalized unsigned integer format, covering the entire image width and height; the second binding point binds to the interleaved chroma plane texture in a dual-channel 8-bit normalized unsigned integer format, with both width and height being half that of the luma plane. Simultaneously, an output image binding point is configured, with an output format of four-channel 8-bit normalized format.

[0079] Secondly, the shader unit 1612A receives configuration parameters through a push constant block, including: the scaling factor of the interlaced chromaticity plane relative to the luma plane (0.5 in both the horizontal and vertical directions), the effective size of the luma plane, and the effective size of the interlaced chromaticity plane.

[0080] The shader unit 1612A initiates operations at a single-pixel parallel execution granularity, with each shader program instance corresponding to a target output pixel. The execution process for each instance is as follows: First, the integer coordinates of the output pixel corresponding to the current instance are obtained; if these coordinates exceed the valid range of the output image, the current instance is terminated directly, and no further operations are performed. Next, the sampling coordinates of the luminance plane are calculated: the output pixel coordinates are clamped to the valid size range of the luminance plane in both the horizontal and vertical directions (not less than 0 and not exceeding the plane size minus 1), obtaining the luminance plane sampling coordinates; the single-channel luminance value Y is read from the corresponding coordinates of the luminance texture using the texel acquisition function. Then, the sampling coordinates of the chroma plane are calculated: the output pixel coordinates are multiplied by the chroma scaling factor in the corresponding direction, rounded, and then clamped to the valid size range of the chroma plane, obtaining the chroma plane sampling coordinates; the dual-channel chroma values ​​are read from the corresponding coordinates of the chroma texture using the texel acquisition function, where the first channel is the blue difference component Cb, and the second channel is the red difference component Cr. Finally, component mapping and output are performed: the read luminance component Y is mapped to the red channel (R channel) of the output pixel, the blue difference component Cb is mapped to the green channel (G channel), the red difference component Cr is mapped to the blue channel (B channel), and the alpha channel is set to 1.0. This four-channel vector is then written to the corresponding pixel position in the output image. This channel mapping is only used to pass the original YUV components within the shader and does not represent a complete color space conversion.

[0081] After the state indication of the synchronization primitive indicates that the decoding task is complete and the post-processing task is scheduled for execution, the post-processing engine 1612 schedules the shader unit 1612A to execute the post-processing task. The shader unit 1612A makes a sampling request to the texture sampler 1616 according to the target processing position. The texture sampler 1616 completes the layout decoding and returns the sampling result. The shader unit 1612A then performs the corresponding post-processing operation.

[0082] During rotation processing, shader unit 1612A determines the source sampling position in shared surface 163 based on the target output position and rotation relationship. When using inverse coordinate mapping, the source sampling position can be determined according to the following relationship: x_s=cosθ·(x_o-x_c)+sinθ·(y_o-y_c)+x_c; y_s=-sinθ·(x_o-x_c)+cosθ·(y_o-y_c)+y_c; Where (x_o, y_o) represents the target output position, (x_s, y_s) represents the source sampling position, θ represents the rotation angle, and (x_c, y_c) represents the rotation center. When using normalized coordinates and taking the image center as the rotation center, the rotation center is (0.5, 0.5). The luma plane and the interlaced chroma plane can use the same normalized sampling position; when using integer coordinates in units of pixels on the luma plane, the sampling coordinates of the interlaced chroma plane are converted according to the ratio of the number of sampling points in the interlaced chroma plane to that in the luma plane. In the NV12 format example, the conversion ratio in both the horizontal and vertical directions is 1 / 2.

[0083] The forward rotation matrix corresponds to the aforementioned inverse coordinate mapping. The forward mapping can be expressed as: x_o=cosθ·(x_s-x_c)-sinθ·(y_s-y_c)+x_c; y_o=sinθ·(x_s-x_c)+cosθ·(y_s-y_c)+y_c. When generating the target image, in order to determine the source sampling position corresponding to each target output position, the above-mentioned forward rotation relationship can be inverted to obtain the aforementioned reverse sampling mapping with opposite positive and negative signs.

[0084] In the color space conversion process, shader unit 1612A combines the luminance and chrominance components into an input component vector, and generates the output components in the target color space according to the following relationship: v_out = M·v_in + b; Where v_in includes the luminance and chrominance components, M represents the transformation matrix, b represents the offset vector, and v_out represents the transformed components. The transformation matrix is ​​determined based on the color standard, value range, and target output color space of the input video. Taking the International Telecommunication Union Radiocommunication Sector Recommendation BT.601 (ITU-R BT.601), a limited studio range input, and a target red-green-blue (RGB) color space as an example, the specific transformation matrix form is as follows:

[0085] Where Y is the luminance component, Cb and Cr are two chromaticity components in the staggered chromaticity plane; 16 and 128 are the reference offsets of the luminance and chromaticity components within a finite range, respectively.

[0086] In color balance processing, shader unit 1612A performs gain adjustment, offset adjustment, white balance adjustment or hue adjustment on the sampled results or components after color space conversion based on the component adjustment parameters in the post-processing task.

[0087] Shader unit 1612A can execute multiple post-processing operations consecutively in the same post-processing task. For example, shader unit 1612A first determines the rotated source sampling position, then obtains the luminance and chrominance components, and then performs color space conversion and color balance processing.

[0088] When the sampling results are continuously used in the operation within the shader unit 1612A, it is not necessary to write and reread the independent intermediate surface between each post-processing operation.

[0089] Texture sampler 1616 performs layout decoding based on the sampling view, enabling decoded data stored in Tile or Swizzle layout to be retrieved according to the logical sampling position used by shader unit 1612A.

[0090] Shader unit 1612A samples luminance plane data and interleaved chrominance plane data respectively, so that the number of sampling points, stride and storage layout of the two planes participate in address determination respectively.

[0091] The sampling results obtained from the layout decoding are provided to the shader unit 1612A. The post-processing task can perform post-processing operations using the decoded data in the shared surface 163 without requiring the controller to pre-copy the decoded data as another linear post-processing input object.

[0092] This disclosure also provides a computer-readable storage medium storing at least one instruction that is executed by a processor to implement the video processing methods described in the above embodiments.

[0093] Those skilled in the art will understand that, in one or more of the examples above, the functions described in this disclosure can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0094] It should be noted that the technical solutions described in this disclosure can be combined arbitrarily as long as they do not conflict.

[0095] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video processing apparatus, characterized in that, Includes a graphics processor and a controller, the graphics processor including a decoding engine and a post-processing engine, the controller being configured to: Configure a shared surface to receive the decoded data output by the decoding engine and use it as the sampling target of the post-processing engine; Create a synchronization primitive and associate the synchronization primitive with a decoding task for the shared surface, such that the synchronization primitive enters a state indicating the completion of the decoding task after the decoding task is completed; The submission information for the post-processing task is generated, and the submission information includes cache access control information corresponding to the shared surface; The post-processing task is submitted according to the submission information. The post-processing task is executed after the decoding task is completed as indicated by the state of the synchronization primitive. During execution, the graphics processor processes the read request issued by the post-processing engine for the shared surface according to the cache access policy indicated by the cache access control information, and the post-processing engine obtains the decoded data from the shared surface and performs post-processing operations.

2. The video processing apparatus according to claim 1, characterized in that, The cache access strategy includes the target cache level; The cache access control information instructs the graphics processor to process read requests issued by the post-processing engine for the shared surface during the execution of the post-processing task using the target cache level.

3. The video processing apparatus according to claim 1, characterized in that, The controller is also configured to: The shared surface is configured based on a composite allocation flag, which gives the shared surface decoding write attributes and sampling read attributes.

4. The video processing apparatus according to claim 3, characterized in that, The composite allocation flag includes a decoding reference image buffer binding flag and a sampling view binding flag; The decoding reference image buffer binding flag registers the shared surface as the decoding write target of the decoding engine, and the sampling view binding flag registers the shared surface as the sampling read target of the post-processing engine. The decoding write target and the sampling read target correspond to the same shared surface identifier.

5. The video processing apparatus according to claim 1, characterized in that, The synchronization primitives include at least one of a fence and a semaphore; The fence is a task completion synchronization object that indicates the command queue of the decoding task to complete the writing to the shared surface; The semaphore is used to establish an inter-queue synchronization object that creates an execution dependency between the command queue of the decoding task and the command queue of the post-processing task.

6. The video processing apparatus according to claim 1, characterized in that, The post-processing engine includes a shader unit that samples and reads the decoded data from the shared surface and performs at least one of the following post-processing operations on the decoded data: rotation processing, color space conversion processing, and color balance processing.

7. The video processing apparatus according to claim 6, characterized in that, The decoded data includes luma-interlaced chroma dual-plane format data, and the shader unit samples the luma plane data and interlaced chroma plane data in the luma-interlaced chroma dual-plane format data respectively.

8. The video processing apparatus according to claim 1, characterized in that, The graphics processor also includes a texture sampler, which is configured to: Based on the sampling view of the shared surface, the storage address of the decoded data in the shared surface is laid out and decoded, and the sampling result obtained by the layout decoding is used as the decoded data sampled and read by the post-processing engine from the shared surface.

9. A video processing method, characterized in that, include: Configure a shared surface to receive the decoded data output by the decoding engine and use it as the sampling target for the post-processing engine; Create a synchronization primitive and associate the synchronization primitive with a decoding task for the shared surface, such that the synchronization primitive enters a state indicating the completion of the decoding task after the decoding task is completed; The submission information for the post-processing task is generated, and the submission information includes cache access control information corresponding to the shared surface; The post-processing task is submitted according to the submission information. The post-processing task is executed after the decoding task is completed as indicated by the state of the synchronization primitive. During execution, the post-processing engine processes the read request issued by the post-processing engine for the shared surface according to the cache access policy indicated by the cache access control information, and enables the post-processing engine to obtain the decoded data from the shared surface and perform post-processing operations.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the video processing method as described in claim 9.

Images