System and method for rendering differential video on a graphics display

Abstract

A differential video rendering system comprising: a graphics processing unit (GPU); a graphics display coupled to the GPU; a video decoder configured to decode a bitstream of encoded data into a plurality of sets of decoded blocks; at least one processor configured to: generate, based on a first set of the plurality of sets of decoded blocks, a first differential video frame comprising a plurality of sets of differential regions, normalize each set of the plurality of sets of differential regions to a fixed size block to provide a normalized plurality of sets of differential regions, map respective sets of the normalized plurality of sets of differential regions to align with respective tile size regions of a plurality of tile size regions conforming to the GPU, generate a hierarchical region tree based on the normalized plurality of sets of differential regions mapped to the plurality of tile size regions, and generate a plurality of optimal regions based on the hierarchical region tree satisfying a predefined criterion corresponding to a predefined optimal region number and a predefined efficiency parameter; and a graphics rendering engine configured to render, based on the plurality of optimal regions and a set of differential regions, the first differential video frame on the graphics display.

Description

Technical Field

[0001] This disclosure relates to a system and method for rendering differential video on a graphics display. Specifically, this disclosure relates to a differential video rendering system and method that utilizes region information to partially render differential video frames on a graphics display. Background Technology

[0002] During graphics rendering, image textures can be mapped to different geometries, such as cubes, rectangles, spheres, and cylinders, using a graphics application programming interface (API). These geometries vary depending on the user interface (UI) scenario and the use cases required by the application. When the image is a series of consecutive video frames and the texture is mapped to any geometry, it is called video texturing. Video texturing is a common technique for rendering YUV color-coded image data to a graphics window. 360 Video incorporates a similar approach to map the texture of captured 360YUV frames to a defined area of ​​a sphere shape.

[0003] Video is typically displayed on a dedicated video plane for any system. It does not require separate color space conversion because the input and output formats remain the same. However, some UI and graphics applications require rendering YUV frames (such as video frames) on a graphics plane that is in RGB format.

[0004] The related video texturing method requires rendering video frames on arbitrary shapes according to the final UI requirements. This video rendering technique involves a large amount of video images or video data from the central processing unit (CPU) memory to the graphics processing unit (GPU) memory. It also requires each frame to be copied to GPU memory before it can be displayed. As video resolution increases, CPU-GPU bandwidth becomes an increasingly critical resource, and any scarcity can lead to a decrease in system performance. Furthermore, increased memory access will result in higher power consumption for the device.

[0005] Furthermore, related video texturing methods utilize GPUs (Open GL ES library) to read complete video frames, decode video data, and render the complete frames. Based on existing video texturing methods, such as... Figure 1 As shown, the entire decoded frame generated from the video decoding based on the input data is read by the GPU and further processed through the GPU's graphics pipeline. The decoded frame includes both changed and unchanged pixels.

[0006] When the entire decoded frame, including both changed and unchanged pixels, passes through the GPU's graphics pipeline, multiple GPU cycles are wasted rendering the unchanged pixels, resulting in limited system resource availability. Furthermore, a full frame update will be required, so the GPU and CPU may need high Double Data Rate (DDR) random access memory (RAM) bandwidth and increased memory accesses. This leads to reduced system performance and ultimately audio glitches due to lower overall system performance. Several video texturing features cannot be productized on low-end systems due to the high bandwidth utilization. Low-end systems may also experience synchronization issues between audio and video playback due to the high bandwidth utilization.

[0007] Furthermore, to address the issue of high bandwidth utilization, existing video texturing methods disclose video rendering using compressed textures. Decoded frames are compressed to generate decoded compressed frames, and these decoded compressed frames are passed through the GPU's graphics pipeline for rendering on a graphics display. However, the compression or decompression of decoded frames can lead to the loss of pixel information, resulting in low-quality images at higher resolutions. Additionally, it can cause format support issues and limit applicability to normal video input. Summary of the Invention

[0008] Technical issues

[0009] Therefore, a system and method are needed that can reduce overall system bandwidth requirements and enhance rendering performance to render decoded video frames on a graphics display by minimizing GPU DDR access. In other words, a system and method are needed that can improve CPU-GPU DDR bandwidth by minimizing increased memory access and rendering only the few pixels required for the decoded frame.

[0010] Technical solution

[0011] This summary is provided to introduce some concepts in a simplified format, which will be further described in the detailed embodiments below. This summary is not intended to identify key or fundamental concepts, nor is it intended to define the scope of this disclosure.

[0012] Additional aspects will be set forth in part in the description which follows, and will become apparent in part from the description itself, or may be learned by practice of the embodiments presented.

[0013] According to one aspect of this disclosure, a differential video rendering system includes: a graphics processing unit (GPU); a graphics display coupled to the GPU; a video decoder configured to decode a bitstream of encoded data into a plurality of decoded block sets; at least one processor configured to: generate a first differential video frame including a plurality of differential region sets based on a first set of the plurality of decoded block sets; normalize each set of the plurality of differential region sets into fixed-size blocks to provide a normalized plurality of differential region sets; map corresponding sets of the normalized plurality of differential region sets to align with corresponding tile-size regions among a plurality of tile-size regions conforming to the GPU; generate a hierarchical region tree based on the normalized plurality of differential region sets mapped to the plurality of tile-size regions; and generate a plurality of optimal regions based on the hierarchical region tree satisfying predefined criteria corresponding to a predefined number of optimal regions and a predefined efficiency parameter; and a graphics rendering engine configured to render the first differential video frame on the graphics display based on the plurality of optimal regions and a set of differential regions.

[0014] The corresponding best region among multiple best regions can include a set of tile-sized regions and satisfy predefined criteria.

[0015] The differential video rendering system may also include a central processing unit (CPU) configured to: determine the predefined optimal number of regions based on experimental values ​​associated with at least one of the GPU's clock speed, the CPU's clock speed, and the number of processing cores included in the GPU; and determine the predefined efficiency parameter based on system variable parameters corresponding to at least one of the GPU's clock speed, the GPU's bandwidth, the memory configuration coupled to the GPU, the width of the memory bus, and the number of processing cores included in the GPU, wherein the predefined efficiency parameter corresponds to the GPU's processing capability to process the maximum number of differential regions with minimum bandwidth.

[0016] The predefined number of optimal regions can correspond to the maximum number of optimal regions that can be passed to the GPU's rendering pipeline without affecting the overall performance of the GPU.

[0017] At least one processor may also be configured to: determine a first number of tile-size regions among the plurality of tile-size regions, including a minimum number of differential regions; perform a marking process to mark the first number of tile-size regions having the minimum number of differential regions as dirty tiles; and generate a list of the dirty tiles based on the marking process.

[0018] At least one processor may also be configured to: generate a block list comprising the plurality of tile-size regions based on the list of dirty tiles; select a root block from the block list, wherein the root block is a superset of all blocks in the block list; select a second number of tile-size regions in the plurality of tile-size regions in the block list in an ordered order; add the selected second number of tile-size regions to the root block in an ordered order until the number of sub-regions of the root block exceeds the predefined criterion corresponding to the predefined optimal number of regions; and generate a first level of the hierarchical region tree based on adding the selected second number of tile-size regions to the root block.

[0019] At least one processor may also be configured to: select a third number of tile size regions from the plurality of tile size regions in the block list, wherein the third number of tile size regions is adjacent to the second number of tile size regions; add the selected third number of tile size regions to the first level of the hierarchical region tree in an ordered order; determine that at least one sub-region of the root block at the first level exceeds at least one of the predefined optimal region number and the predefined efficiency parameter; and divide the at least one sub-region exceeding at least one of the predefined optimal region number and the predefined efficiency parameter into a first plurality of sub-sub-regions, such that each sub-sub-region in the first plurality of sub-sub-regions satisfies the predefined criteria.

[0020] At least one processor may also be configured to generate a second level of a hierarchical region tree based on dividing at least one sub-region into a first plurality of sub-sub-regions, wherein the second level of the hierarchical region tree may include the first plurality of sub-sub-regions, and wherein the second level may correspond to a level following the first level of the hierarchical region tree.

[0021] The hierarchical region tree may include multiple levels, and the multiple levels may include at least a first level. At least one processor may also be configured to: determine whether any sub-region at each level of the multiple levels exceeds at least one of the predefined optimal region number and the predefined efficiency parameter; and divide the sub-region exceeding at least one of the predefined optimal region number and the predefined efficiency parameter into a second plurality of sub-regions, such that each sub-region at the corresponding level of the multiple levels satisfies the predefined criteria.

[0022] The bottom level of the hierarchical region tree may include leaf blocks, and the at least one processor may also be configured to: generate multiple optimal regions from the root block toward the leaf block based on the segmentation of at least one sub-region and at least one sub-sub-region in any sub-sub-region, and the multiple optimal regions may be generated from the root block toward the leaf block, such that each of the multiple optimal regions has an efficiency greater than or equal to a predefined efficiency parameter.

[0023] At least one processor may also be configured to: arrange the first level and the second level from the top of the root block toward the leaf block in the order in which the first level and the second level were generated; and generate the hierarchical region tree based on the arrangement.

[0024] The at least one processor may also be configured to: generate a second differential video frame based on a second set of a plurality of decoded block sets, wherein the generation of the first differential video frame may occur before the generation of the second differential video frame, a first number of tile-size regions in a plurality of tile-size regions may correspond to reused tiles, a second number of tile-size regions in a plurality of tile-size regions may correspond to dirty tiles, reused tiles may consist entirely of reused blocks, reused blocks may correspond to blocks with the same pixel values ​​in the first and second differential video frames, and dirty tiles may include reused blocks.

[0025] At least one processor may also be configured to: generate a block list comprising the plurality of tile-sized regions based on the list of reused tiles and the list of dirty tiles; select a root block from the block list, wherein the root block is a superset of all blocks in the block list; select a first set of dirty tiles and the reused tiles in an ordered order; add the selected first set of dirty tiles and the reused tiles to the root block in an ordered order until the number of sub-regions of the root block exceeds the predefined optimal number of regions, wherein each reused tile in the first set of reused tiles is added to the root block as a separate sub-region; and generate a first level of the hierarchical region tree based on adding the selected first set of dirty tiles and the reused tiles to the root block.

[0026] At least one processor may also be configured to: select a second set of each of the dirty tiles and the reused tiles, wherein the second set of dirty tiles is adjacent to a first set of dirty tiles; add the selected second set of dirty tiles and reused tiles in an ordered order to the first level of the hierarchical region tree; determine that at least one sub-region of the root block at the first level exceeds at least one of the predefined optimal region number and the predefined efficiency parameter; and divide the at least one sub-region exceeding at least one of the predefined optimal region number and the predefined efficiency parameter into a first plurality of sub-sub-regions, such that each sub-sub-region in the first plurality of sub-sub-regions satisfies the predefined criteria.

[0027] At least one processor may also be configured to generate a second level of a hierarchical region tree based on dividing at least one sub-region into a first plurality of sub-sub-regions, wherein the second level of the hierarchical region tree may include the first plurality of sub-sub-regions, and wherein the second level may correspond to a level following the first level of the hierarchical region tree.

[0028] The hierarchical region tree may include multiple levels, and the multiple levels may include at least a first level. At least one processor may also be configured to: determine whether any sub-region at each level of the multiple levels exceeds at least one of the predefined optimal region number and the predefined efficiency parameter; and divide the sub-region exceeding at least one of the predefined optimal region number and the predefined efficiency parameter into a second plurality of sub-regions, such that each sub-region at the corresponding level of the multiple levels satisfies the predefined criteria.

[0029] The bottom level of the hierarchical region tree may include leaf blocks, and the at least one processor may also be configured to: generate multiple optimal regions from the root block toward the leaf block based on the segmentation of at least one sub-region and at least one sub-sub-region in any sub-sub-region, and the multiple optimal regions may be generated from the root block toward the leaf block, such that each of the multiple optimal regions has an efficiency greater than or equal to a predefined efficiency parameter.

[0030] At least one processor may also be configured to: arrange the first and second levels in the order of generation from the top of the root block toward the leaf blocks; and generate the hierarchical region tree based on the arrangement.

[0031] Each of the first and second differential video frames can correspond to either a static video frame or a dynamic video frame.

[0032] At least one processor may also be configured to map a normalized set of differential regions to align with a set of tile-sized regions, based on a tile-based rendering process. Attached Figure Description

[0033] These and other features, aspects, and advantages of certain embodiments of the present disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1 This illustrates the problems in existing video texturing methods.

[0035] Figure 2 The system architecture of a differential video rendering system according to an embodiment of the present disclosure is shown.

[0036] Figure 3 An embodiment according to this disclosure is shown. Figure 2 An exemplary implementation of a differential video rendering system.

[0037] Figure 4 The method steps of a differential video rendering method according to an embodiment of the present disclosure are shown.

[0038] Figure 5 An embodiment according to this disclosure is shown. Figure 4 A detailed flowchart of the differential frame generation process.

[0039] Figure 6 An embodiment according to this disclosure is shown. Figure 4 An example of the differential frame generation process.

[0040] Figure 7 The method steps of a region preprocessing operation in a differential video rendering method according to an embodiment of the present disclosure are shown.

[0041] Figure 8 An example of a region preprocessing operation of a differential video rendering method according to an embodiment of the present disclosure is shown.

[0042] Figures 9A to 9B Examples of tile-based rendering processes and dirty tile marking processes corresponding to region preprocessing operations of a differential video rendering method according to embodiments of the present disclosure are shown.

[0043] Figures 10A to 10B Examples of determining the predefined optimal number of regions (N) and predefined efficiency parameters (E) corresponding to the GPU of a differential video rendering system according to embodiments of the present disclosure are shown.

[0044] Figure 11 An embodiment according to this disclosure is shown. Figure 4 The first part of the connected component formation process in the differential video rendering method.

[0045] Figure 12AAn example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0046] Figure 12B An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0047] Figure 13 An embodiment according to this disclosure is shown. Figure 11 The second part of the process of forming connected components.

[0048] Figure 14A An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0049] Figure 14B An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0050] Figure 15A An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0051] Figure 15B An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0052] Figure 16A An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0053] Figure 16B An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0054] Figure 17A An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0055] Figure 17B An example of generating a hierarchical region tree according to an embodiment of the present disclosure is shown.

[0056] Figure 18 An example of a dynamic differential video frame according to an embodiment of the present disclosure is shown.

[0057] Figure 19 An embodiment according to this disclosure is shown. Figure 4 Another example of the differential frame generation process.

[0058] Figure 20 Another example of a region preprocessing operation of a differential video rendering method 400 according to an embodiment of the present disclosure is shown.

[0059] Figure 21 A flowchart illustrating the connected component formation process of a reference dynamic differential video frame according to an embodiment of the present disclosure is shown.

[0060] Figure 22A An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0061] Figure 22B An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0062] Figure 23A An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0063] Figure 23B An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0064] Figure 24A An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0065] Figure 24B An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0066] Figure 25A An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0067] Figure 25B An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0068] Figure 26A An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0069] Figure 26B An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0070] Figure 27 An embodiment according to this disclosure is shown. Figure 21 An example of the process of forming connected components.

[0071] Figure 28A The embodiments of the present disclosure are shown as follows. Figure 21 An example illustration of a hierarchical region tree formed as a result of the process of forming connected components.

[0072] Figure 28B The embodiments of the present disclosure are shown as follows. Figure 21An example illustration of a hierarchical region tree formed as a result of the process of forming connected components.

[0073] Figure 29 An embodiment according to this disclosure is shown. Figure 4 The flowchart shows the optimal region formation process of the differential video rendering method.

[0074] Figure 30 An example of generating an optimal set of regions according to an embodiment of this disclosure is shown.

[0075] Figure 31A An example of the optimal set of regions according to embodiments of this disclosure is shown.

[0076] Figure 31B An example of the optimal set of regions according to embodiments of this disclosure is shown.

[0077] Figure 32A Exemplary advantages of the differential video rendering method according to embodiments of the present disclosure over prior art video rendering methods are shown.

[0078] Figure 32B Exemplary advantages of the differential video rendering method according to embodiments of the present disclosure over prior art video rendering methods are shown.

[0079] Figure 32C Exemplary advantages of the differential video rendering method according to embodiments of the present disclosure over prior art video rendering methods are shown.

[0080] Figure 33A This illustrates yet another exemplary advantage of the differential video rendering method according to embodiments of the present disclosure over prior art video rendering methods.

[0081] Figure 33B This illustrates yet another exemplary advantage of the differential video rendering method according to embodiments of the present disclosure over prior art video rendering methods.

[0082] Figure 33C This illustrates yet another exemplary advantage of the differential video rendering method according to embodiments of the present disclosure over prior art video rendering methods.

[0083] Figure 34 Another system architecture according to an embodiment of this disclosure is shown.

[0084] Figure 35 Another exemplary implementation of an embodiment according to this disclosure is shown.

[0085] Furthermore, the elements in the accompanying drawings are shown for simplicity and may not necessarily be drawn to scale. For example, the flowcharts illustrate the method according to the most prominent steps involved to help improve understanding of various aspects of this disclosure. Additionally, regarding the construction of the device, one or more components of the device may have already been represented in the drawings by related technical symbols, and the drawings may show only those specific details relevant to understanding embodiments of this disclosure so as not to obscure the drawings with details that would be obvious to a person of ordinary skill in the art from the description herein. Detailed Implementation

[0086] Although illustrative implementations of embodiments of this disclosure are shown below, any number of techniques, whether currently known or existing, can be used to implement the embodiments. This disclosure should in no way be limited to the illustrative implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.

[0087] As used herein, the term "some" is defined as "no," "one," "more than one," or "all." Therefore, the terms "no," "one," "more than one," "more than one but not all," or "all" all fall under the definition of "some." The term "some embodiments" can refer to no embodiment, one embodiment, several embodiments, or all embodiments. Therefore, the term "some embodiments" is defined as meaning "no embodiment, one embodiment, more than one embodiment, or all embodiments."

[0088] The terminology and structure used herein are intended to describe, teach, and illustrate certain embodiments and their specific features and elements, and do not limit, constrain, or reduce the spirit and scope of the claims or their equivalents.

[0089] More specifically, unless otherwise stated, any terms used herein (such as, but not limited to, “including,” “contains,” “has,” “composes,” and their grammatical variations) do not specify exact limitations or constraints, and certainly do not preclude the possibility of adding one or more features or elements, and furthermore, unless otherwise stated, should not be construed as excluding the possibility of removing one or more of the listed features and elements.

[0090] Regardless of whether a feature or element is used only once, or in any manner, it may still be referred to as "one or more features," "one or more elements," "at least one feature," or "at least one element." Furthermore, unless otherwise stated, the use of the terms "one or more" or "at least one" features or elements does not preclude the absence of that feature or element.

[0091] Unless otherwise defined, all terms used herein, in particular any technical and / or scientific terms, may be considered to have the same meaning as commonly understood by one of ordinary skill in the art.

[0092] As is common in the art, embodiments can be described and illustrated around blocks that perform desired functions. These blocks (which may be referred to herein as unit modules, or as such as drivers, controllers, devices, engines, etc.) can be physically implemented by analog or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and can be driven by firmware and software. The circuitry included in a block can be implemented by dedicated hardware or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware for performing some functions of the block and a processor for performing other functions of the block. Each block of an embodiment can be physically divided into two or more interactive and discrete blocks. Similarly, the blocks of an embodiment can be physically combined into more complex blocks.

[0093] The embodiments will now be described in detail with reference to the accompanying drawings.

[0094] Figure 2 The system architecture of a differential video rendering system according to an embodiment of the present disclosure is shown. Figure 2 A system 200 for partially rendering differential video frames on a graphics display is shown. System 200 includes an input unit 202, a video decoder 204, a hardware unit 206, a central processing unit (CPU) 208, a neural processing unit (NPU) 210, an audio digital signal processor (audio DSP) 212, a graphics processing unit (GPU) 214, a graphics engine 216, a multimedia engine 218, a memory 220, an application interface 222, and an output unit 224. These components are coupled to each other. Each of the video decoder 204, CPU 208, NPU 210, and GPU 214 is communicatively coupled to memory 220 to read instructions from or write instructions to memory 220. Here, without departing from the scope of this disclosure, graphics engine 216 may also be referred to as a "graphics rendering engine".

[0095] Input unit 202 receives a bitstream of encoded data and transmits the received bitstream of encoded data to video decoder 204. Input unit 202 may include suitable logic, circuitry, and / or interfaces that can be configured to act as an input interface between the user and the system. Input unit 202 may also include various input devices that can be configured to communicate with different operating components of system 200. Examples of input interfaces include, but are not limited to, network applications, media applications, or broadcast applications. Input interfaces may include interfaces different from those described above.

[0096] The video decoder 204 decodes the bitstream of encoded data into multiple sets of decoded blocks and stores these sets in the memory 220. The encoded data, for example, corresponds to the video data received by the input unit 202.

[0097] Hardware unit 206 includes an optimal region generator unit 206A, which includes an optimal region module 206B, a video processing engine 204C, a differential generator 206D, and a capture module 206E. The differential generator 206D generates differential video frames based on multiple sets of decoded blocks. Each of the generated differential video frames includes multiple sets of differential regions. Furthermore, the differential generator 206D normalizes each of the multiple sets of differential regions into blocks of a fixed size. Here, without departing from the scope of this disclosure, the optimal region generator unit 206A may also be referred to as a "region generator unit," and the differential generator 206D may also be referred to as a "differential frame generator."

[0098] The optimal region generator unit 206A maps a corresponding set from a normalized set of differential regions to a corresponding tile-size region among a plurality of tile-size regions conforming to GPU 214. In an embodiment, the plurality of tile-size regions conforming to GPU 214 may mean that the plurality of tile-size regions conform to or are compatible with GPU 214. The optimal region generator unit 206A further generates a hierarchical region tree based on the normalized set of differential regions mapped to the plurality of tile-size regions. Furthermore, the optimal region generator unit 206A generates a plurality of optimal regions based on a hierarchical region tree that satisfies predefined criteria corresponding to a predefined number of optimal regions (N) and a predefined efficiency parameter (E) of GPU 214. The corresponding optimal region among the plurality of optimal regions comprises a set of tile-size regions and satisfies the predefined criteria corresponding to N and E. The descriptions of N and E will be further described in detail with reference to some examples of the embodiments.

[0099] The CPU 208 is the hardware that controls the overall operation and function of the system 200. For example, the CPU 208 implements the operating system (OS), calls the graphics application programming interface (API) of the GPU 214, and executes the driver for the GPU 214. In addition, the CPU 208 can execute various other applications installed on the system 200, such as video applications, game applications, web browsing applications, etc.

[0100] The NPU 210 can be a microprocessor specifically designed to accelerate machine learning algorithms, such as by operating predictive models like artificial neural networks (ANNs) or random forests (RF). However, the NPU 210 is not limited to the examples mentioned above. The NPU 210 can operate on any other artificial intelligence model as needed.

[0101] GPU 214 is a graphics-dedicated processor that performs a graphics pipeline. In one example, GPU 214 can be implemented as hardware that performs a 3D graphics pipeline to display 3D objects of a 3D image as 2D images for display. For example, GPU 214 can perform various functions such as rendering, shading, blending, lighting, and generating pixel values ​​for the pixels to be displayed. In one example, GPU 214 can perform a tile-based rendering process. In this context, the term "tile-based" means that each frame of a moving image is divided into multiple tiles, and rendering is performed on a tile-by-tile basis. The tile-based rendering process updates only specific tiles at any given time. Each tile is only a part of the entire frame buffer and can be stored on on-chip RAM. Performing a tile-based rendering process results in a reduction in the bandwidth required by GPU 214 for depth testing and for blending transparent fragments, and is therefore available to GPU 214 without any access to any external memory.

[0102] The audio DSP 212 decodes the encoded audio data received via the input unit 202 and transmits it to the output unit 224 (e.g., a speaker, headphones).

[0103] Examples of output unit 224 are not limited to those described above. Output unit 224 may include a graphical user interface (GUI) and / or an interface that can be configured to act as an output interface between the user and system 200. A GUI may refer to a graphical display provided on the display (e.g., screen) of an electronic device. A GUI may include at least one window, at least one icon, at least one scroll bar, and any other graphical items for the user to input commands to the device. It should be understood that exemplary embodiments may include various types of GUIs with various shapes, designs, and configurations. Other examples of output unit 224 may include graphics devices / display devices, computer screens, alarm systems, computer-aided design / computer-aided manufacturing (CAD / CAM) systems, video game consoles, smartphone displays, displays mounted on dashboards in automobiles, or any other type of data output device.

[0104] The graphics engine 216 includes a renderer and a partial rendering engine. The graphics engine 216 renders differential video frames on a graphics display based on an optimal region generated by the optimal region generator unit 206A and a set of differential regions from differential video frames generated by the differential generator 106D.

[0105] Multimedia engine 218 includes a multimedia player, such as an audio / video (AV) player. Examples of multimedia players are not limited to those described above, and multimedia engine 218 may include media players other than AV players. Furthermore, multimedia engine 218 provides an interface for configuring and controlling multimedia applications installed on system 100.

[0106] Memory 220 is hardware that stores various types of data processed in system 200. For example, memory 220 may store data processed by video decoder 204, CPU 208, and GPU 214, or data to be processed by video decoder 204, CPU 208, and GPU 214. Furthermore, memory 220 may store application data and drivers to be executed by components of system 200 (i.e., CPU 208, NPU 210, GPU 214, etc.). Memory 220 may include random access memory (RAM) (such as dynamic random access memory (DRAM) or static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disc storage device, hard disk drive (HDD), solid-state drive (SSD), or flash memory. Additionally, memory 120 may include external storage devices accessible by system 200. According to embodiments of this disclosure, memory 220 may further include double data rate synchronous dynamic random access memory (SDRAM), double data rate 2 SDRAM, double data rate 3 SDRAM, or double data rate 4 SDRAM (double data rate or double data rate 2, 3 or 4 DDR synchronous random access memory, or DDR / DDR2 / DDR3 / DDR4 SDRAM).

[0107] Application interface 222 can be configured as a video graphics application interface for a user to play back media content on system 200. Application interface 222 can be configured to have a dynamic interface that can change according to user preferences and the configuration of system 200. According to some example embodiments of this disclosure, application interface 222 can correspond to the user interface of one or more applications installed on system 200. For example, application interface 222 can be an interface for virtual reality (VR) 360, an advertising interface, or a content viewer interface. Examples of application interface 222 are not limited to the above examples; application interface 222 can be any interface of one or more applications installed on system 200.

[0108] Now refer to the attached diagram. Figure 3 , Figure 3 An embodiment according to this disclosure is shown. Figure 2 An exemplary implementation of a differential video rendering system. (Refer to...) Figure 2 describe Figure 3 Furthermore, for the sake of brevity, in the preferred embodiment, similar reference numerals are used throughout the figures to denote corresponding features. Figure 3An exemplary implementation of the differential video rendering system shows that the bitstream of encoded data received by input unit 202 is decoded into multiple sets of decoded blocks. After decoding the bitstream of encoded data, differential generator 206D generates differential video frames based on the multiple sets of decoded blocks and further normalizes the differential video frames into fixed-size blocks. Furthermore, after normalizing the differential video frames into fixed-size blocks, optimal region generator unit 206A performs frame data preprocessing operations on the fixed-size blocks.

[0109] Frame data preprocessing operations may include mapping corresponding sets from a set of normalized differential regions to align with regions of corresponding tile size, based on a tile-based rendering process.

[0110] After mapping the corresponding sets in a set of normalized differential regions, the optimal region generator unit 206A performs an optimal region formation process, which may include generating a hierarchical region tree based on the set of normalized differential regions mapped to tile-size regions, and generating an optimal region based on the hierarchical region tree. Each of the optimal regions includes a set of tile-size regions that satisfy predefined criteria corresponding to N and E. The optimal region corresponds to a portion of the video data that is a region of interest.

[0111] The generated optimal region is transferred to the memory of GPU 214. Furthermore, graphics engine 216 combines the optimal region with the differential region group of the differential frame. After combining the optimal region with the differential region group, graphics engine 216 uses this combination to render differential video frames on a graphics display including 3D scene data.

[0112] According to embodiments of this disclosure, the aforementioned frame data preprocessing operations and optimal region formation process result in the minimum optimal rectangular region required to fully utilize the GPU 214 to render the next differential video frame on the graphics display. This optimizes the bandwidth of the GPU 214 by limiting memory access only to the updated region and by partially rendering the differential video frame to the graphics display by avoiding unchanged pixels corresponding to previous differential video frames. Therefore, the differential video rendering system of this disclosure can reduce overall system bandwidth requirements and enhance the rendering performance of the GPU 214 by minimizing GPU DDR access.

[0113] Figure 4 The method steps of a differential video rendering method according to embodiments of the present disclosure are illustrated. In such... Figure 4 In the implementation shown, this topic relates to the differential video rendering method 400 in system 200. References will be made to... Figure 2 describe Figure 4 Furthermore, for the sake of brevity, in the preferred embodiment, similar reference numerals are used throughout the drawings to denote corresponding features.

[0114] The differential video rendering method 400 may include, at block S402, receiving a bitstream of encoded data by input unit 202. As an example, input unit 202 receives encoded video data via an input interface. The flow of the differential video rendering method 400 now proceeds to block S404.

[0115] At box S404, after receiving the encoded video data, the differential video rendering method 400 may include decoding the encoded video data into a set of multiple decoded blocks. As an example, the video decoder 204 decodes the encoded video data into a set of multiple decoded blocks. The flow of the differential video rendering method 400 now proceeds to box S406.

[0116] At box S406, after decoding the encoded video data, the differential video rendering method 400 may include: generating a differential video frame comprising multiple sets of differential regions based on a first set of multiple sets of decoded blocks. As an example, the differential generator 206D generates differential video frames based on multiple sets of decoded blocks. Now, according to the accompanying drawings... Figure 5 The further processing steps of the differential video rendering method 400 at box S406 are described in detail. Figure 5 A detailed flowchart of the process performed by the differential generator 206D is shown. After generating multiple sets of decoded blocks at box S406 A1, the differential video rendering method 400 may include, at box S406 A2, normalizing the multiple sets of decoded blocks into fixed-size blocks. As an example, the differential generator 206D normalizes the multiple sets of decoded blocks into fixed-size blocks. As a result, at box S406 A3, the differential video rendering method 400 may include generating a list of blocks of fixed size.

[0117] According to the attached diagram Figure 6 Describes the process performed by the difference generator 206D at box S406. Figure 6 An embodiment according to this disclosure is shown. Figure 4 An example of the differential frame generation process.

[0118] The 206D differential generator uses motion vector information of blocks within differential video frames to identify skipped macroblocks. Such blocks can have varying sizes, for example... Figure 6As shown on the left-hand side, differential video frames can include, for example, blocks of different sizes, namely 4×4, 4×8, 8×4, 4×16, 8×8, 8×16, 16×8, and 16×16. In the following text, each differential video frame includes a set of blocks of different sizes. In the following text, blocks of different sizes (i.e., 4×4, 4×8, 8×4, 4×16, 8×8, 8×16, 16×8, and 16×16) can be referred to as "multiple differential region sets". As an example, a first set of multiple differential region sets (i.e., for example, blocks 4×4, 4×8, 4×16, and 8×8) differs from a second set of multiple differential region sets (i.e., for example, blocks 4×16, 8×8, 8×16, 16×8, and 16×16). The above examples of blocks of different sizes are not limited to the size configurations described above; the block sizes can differ from the size configurations described above.

[0119] The differential generator 206D predicts frames based on blocks of different sizes and motion vectors from previous differential video frames. Depending on the encoding format of the encoded data, the motion vectors can refer to multiple previous video frames. The differential generator 206D converts the motion vectors into spatial information (x, y, w, h) that corresponds only to one previous frame in the display order.

[0120] The difference generator 206D further normalizes multiple sets of difference regions into blocks of fixed size. For example, as Figure 6 As shown on the right-hand side, blocks of different sizes (i.e., 4×4, 4×8, 8×4, 4×16, 8×8, 8×16, 16×8, 16×16) are normalized to fixed-size blocks of 16×16. According to an example embodiment of this disclosure, blocks of different sizes can also be normalized to fixed-size blocks of 32×32 or any other size. Specifically, the difference generator 206D organizes multiple sets of difference regions in an easily manageable format.

[0121] Now, refer to it again. Figure 4 After normalizing the multiple decoded block sets, the differential video rendering method 400 now proceeds to frame S408. Frame S408 may include multiple method steps S408 A, S408 B, and S408 C. (Refer to...) Figures 7 to 19 The steps in block S408 are described in detail. At block S408 A, the differential video rendering method 400 may include a region preprocessing operation performed by the optimal region generator unit 206A on a normalized set of multiple differential regions, which may be referred to as a frame data preprocessing operation.

[0122] Now for reference Figure 7 , Figure 7 An embodiment according to this disclosure is shown. Figure 4The differential video rendering method 400 includes the following method steps for region preprocessing. The region preprocessing operation at frame S408A may include a series of method steps S408A2 to S408A12. At frame S408A2, the differential video rendering method 400 may include dividing a differential video frame (hereinafter referred to as the "first differential video frame") into a grid of multiple tile-sized regions conforming to the GPU 214. After dividing the first differential video frame, the procedural region preprocessing operation now proceeds to frame S408A4.

[0123] At frame S408 A4, the differential video rendering method 400 may further include mapping corresponding sets among the normalized sets of differential regions to align with corresponding tile-size regions among the multiple tile-size regions conforming to GPU 214. After mapping the corresponding sets among the normalized sets of differential regions, the process region preprocessing operation now proceeds to frame S408 A6.

[0124] At frame S408 A6, the differential video rendering method 400 may further include: determining a first number of tile-size regions that include a minimum number of differential regions among a plurality of tile-size regions.

[0125] After determining the first number of tile-size regions, the process now proceeds to frame S408A8. At frame S408A8, the differential video rendering method 400 may further include: performing a marking process to mark the first number of tile-size regions having the minimum number of differential regions as dirty tiles.

[0126] After the marking process is executed, the flow now proceeds to box S408 A10. At box S408 A10, the differential video rendering method 400 may further include generating a list of dirty tiles based on the execution of the marking process. After generating the list of dirty tiles, the flow now proceeds to box S408 A12.

[0127] At box S408 A12, the differential video rendering method 400 may further include: generating a block list comprising regions of multiple tile sizes based on a list of dirty tiles. As an example, reference will be made to the accompanying drawings. Figure 8 , Figure 9A and Figure 9B The preprocessing operation of the aforementioned region at box S408 A of the differential video rendering method 400 is described.

[0128] Figure 8 An example of a region preprocessing operation of a differential video rendering method 400 according to an embodiment of the present disclosure is shown. Figure 8The image shows the first differential video frame being divided into a grid of multiple tile-sized regions conforming to the GPU 214. As an example, the optimal region generator unit 206A divides the first differential video frame into a grid of 32×32 tile-sized regions. Furthermore, in... Figure 8 The image shows a 16×16 normalized fixed-size block. Furthermore, according to... Figure 8 The optimal region generator unit 206A maps the corresponding normalized fixed-size blocks to align them with the corresponding 32×32 tile-size regions conforming to the GPU 214. Here, the 16×16 normalized fixed-size block refers to "a set of multiple normalized differential regions", and the 32×32 tile-size region refers to "multiple tile-size regions conforming to the GPU 214".

[0129] Figures 9A to 9B Examples of a tile-based rendering process and a dirty tile marking process according to embodiments of the present disclosure are shown. Figure 9A The diagram shows 16×16 normalized fixed-size blocks mapped onto a 32×32 tile-sized region by the optimal region generator unit 206A on a per-tile basis. Furthermore, in... Figure 9B In the example, a 32×32 tile-size region with a minimum number of 16×16 normalized fixed-size blocks is labeled as a dirty tile. For example, the optimal region generator unit 206A labels a 32×32 tile-size region with at least one 16×16 normalized fixed-size block as a dirty tile. Figure 8 , Figure 9A and Figure 9B The exemplary process shown corresponds to the method steps in blocks S408 A2 to S408 A12.

[0130] Figures 10A to 10B An example of determining the predefined optimal number of regions (N) and predefined efficiency parameter (E) corresponding to the GPU 214 of system 400 according to an embodiment of this disclosure is shown. Here, N corresponds to the maximum number of optimal regions that can be passed to the rendering pipeline of GPU 214 without affecting the overall performance of GPU 214, and E corresponds to the processing capability of GPU 214 to process the most differential regions with minimum bandwidth.

[0131] According to embodiments of this disclosure, CPU 208 determines N based on experimental values ​​associated with at least one of the following: the clock speed of GPU 214, the clock speed of CPU 208, and the number of processing cores included in GPU 214. Furthermore, CPU 208 determines E based on at least one corresponding system variable parameter associated with at least one of the following: the clock speed of GPU 214, the bandwidth of GPU 214, the memory configuration coupled to GPU 214, the width of the memory bus, and the number of processing cores included in GPU 214.

[0132] According to embodiments of this disclosure, CPU 208 determines N using an iterative method defined below:

[0133] For each variation factor (GPU clock, cores)

[0134] 1. For n (maximum N→1)

[0135] i.CHECK FPS

[0136] ii.IF FPS == PREV_FPS

[0137] 1. N=1

[0138] 2. PREV_FPS = FPS

[0139] 3. Continue

[0140] iii.BREAK

[0141] 2. RETURN n;

[0142] in,

[0143] Maximum N - screen size or tile size

[0144] FPS - Frames per second

[0145] PREV_FPS - Previous frame rate

[0146] According to embodiments of this disclosure, CPU 208 determines E using an iterative method defined below:

[0147] For varying factors (GPU clock, cores, DDR speed, and bandwidth)

[0148] 1. For E(0.0→0.5) / / Step size = 0.05

[0149] i. For the input range set (1→n)

[0150] ii. Execute the pipeline and measure BW

[0151] iii. Prepare a chart for each E

[0152] 2. Select E from the chart that has the minimum bandwidth for the most differential regions.

[0153] CPU 208 selects a value for E that has the minimum bandwidth for the most differential regions, and E represents the processing capability of GPU 214 to process the most differential regions with the minimum bandwidth.

[0154] Now, refer to it again. Figure 4 Following the region preprocessing operation at box S408A, the differential video rendering method 400 now proceeds to box S408B. Box S408B describes the connected component formation process for generating a hierarchical region tree. At box S408B, the differential video rendering method 400 may further include generating a hierarchical region tree based on a normalized set of multiple differential regions mapped to multiple tile-sized regions. The optimal region generator unit 206A generates the hierarchical region tree through the connected component formation process S408B. The connected component formation process S408B, in conjunction with the reference... Figures 11 to 17B This is performed in a series of steps as described below. Figures 11 to 17B The component formation process S408 B in the case of differential frames of static video is explained.

[0155] Figure 11 It shows Figure 4 The connected component formation process at frame S408B of the differential video rendering method 400. The connected component formation process S408B may include a series of method steps S408B2 to S408B26. The connected component formation process S408B begins with the optimal region generator unit 206A at... Figure 7 The list of blocks generated at frame S408 A12.

[0156] At box S408 B2, the connected component formation process S408 B may include: selecting a root block (b) from a block list, where the root block is a superset of all blocks in the block list. The flow of the connected component formation process S408 B now proceeds to box S408 B4.

[0157] After selecting the root block, the connected component formation process S408 B at the box may include: selecting a second number of tile-size regions from a plurality of tile-size regions in an ordered sequence. The flow of the connected component formation process S408 B now proceeds to box S408 B6. In the following text, the number of tile-size regions refers to "dirty tile regions".

[0158] At box S408 B6, the connected component formation process S408 B may further include: adding the selected second number of tile-sized regions in an ordered order to the root block until the number of sub-regions of the root block exceeds a predefined standard corresponding to N.

[0159] In other words, the root block is initially empty. Furthermore, tile-sized regions are selected one by one and added to the root block. Newly added tile-sized regions become direct children of the root block, until the number of tile-sized regions exceeds the N value specified in iterative method 1 as described above.

[0160] Now refer to the attached diagram. Figure 12A , Figure 12B Explanation of the process of forming connected components at S408 B2, S408 B4, S408 B6 and S408 B8: S408 B. Figures 12A to 12B An example of generating a first level of a hierarchical region tree according to an embodiment of this disclosure is shown. For example, to illustrate the generation of the hierarchical region tree, the value of N is considered to be 6. Figures 12A to 12B As shown on the left-hand side, 18 dirty tiles exist in the differential video frame, and the value of N = 6. The optimal region generator unit 206A initially selects a root tile from the generated tile list, and further selects a second number of tile-size regions in an ordered order from multiple tile-size regions in the tile list. As an example, the optimal region generator unit 206A selects tile-size regions 1 to 6 as the second number of tile-size regions from tile-size regions 1 to 18. Furthermore, the optimal region generator unit 206A adds the selected tile-size regions 1 to 6 to the root tile in an ordered order until the number of sub-regions of the root tile (i.e., R1 to R6) exceeds the predefined value N = 6. Here, the value of N used is an example and is not limited to the example above. The value of N depends on the type and processing power of the GPU 214 and can be calculated based on iterative method 1 as described above.

[0161] After adding the selected tile-size regions 1 to 6 to the root block, at box S408 B8, the connected component formation process S408 B may include: generating a first level of the hierarchical region tree based on adding a second number of selected tile-size regions to the root block. As an example, the optimal region generator unit 206A generates level 1, which includes sub-regions R1 to R6 of the root block, such as... Figures 12A to 12B As shown.

[0162] After generating the first level of the hierarchical region tree, at box 408B10, the connected component formation process S408B may include: selecting a third number of tile-size regions adjacent to the second number of tile-size regions from a plurality of tile-size regions in the block list. The process now proceeds to box 408B12.

[0163] At box 408B12, the connected component formation process S408B may include: adding the selected third number of tile-sized regions in an ordered order to the first level of the hierarchical region tree. The process now proceeds to box 408B14.

[0164] At box 408B14, the connected component formation process S408B may include: determining that at least one subregion of the root block at the first level exceeds a predefined criterion corresponding to at least one of N and E. The process now proceeds to box 408B16.

[0165] At box 408B16, the connected component formation process S408B may include: dividing at least one sub-region exceeding a predefined standard into a first plurality of sub-sub-regions, such that each sub-sub-region in the first plurality of sub-sub-regions satisfies the predefined standard corresponding to N and E.

[0166] Figure 13 An embodiment according to this disclosure is shown. Figure 11 The subsequent part of the connected component formation process S408 B. According to Figure 13 The process of connecting component formation S408 B now proceeds to box S408 B18. After segmenting at least one sub-region, at box S408 B18, connecting component formation S408 B may include: generating a second level of a hierarchical region tree based on segmenting at least one sub-region into a first plurality of sub-sub-regions.

[0167] After forming the next level, repeat the operations performed at boxes 408B14 and 408B16 until each of the sub-regions and sub-sub-regions of the root block satisfies the predefined criteria corresponding to N and E.

[0168] like Figure 13 As shown in box S408 B20, the optimal region generator unit 206A can generate multiple levels of a hierarchical region tree. The multiple levels may include at least a first level. The flow of the connected component formation process S408 B now proceeds to box S408 B22. At box S408 B22, the connected component formation process S408 B may include: determining whether any sub-region at each of the multiple levels exceeds a predefined criterion corresponding to at least one of N and E. The flow of the connected component formation process S408 B now proceeds to box S408 B24.

[0169] At block S408 B24, the connected component formation process S408 B may include: dividing a sub-region exceeding a predefined criterion into a second plurality of sub-regions, such that each sub-region at a corresponding level in the plurality of levels satisfies the predefined criteria corresponding to N and E. The flow of the connected component formation process S408 B now proceeds to block S408 B26.

[0170] At box S408 B26, the connected component formation process S408 B may include: arranging multiple levels from the top of the root block toward the leaf block in a generation order of multiple levels. As a result of this arrangement, a hierarchical region tree is formed. As an example, the optimal region generator unit 206A may arrange the first and second levels of the hierarchical region tree from the top of the root block toward the leaf block in a generation order of the first and second levels of the hierarchical region tree.

[0171] Now refer to the attached diagram. Figures 14A to 17B Explanation of the connected component formation process at boxes 408B10 to 408B24, S408 B. Figures 14A to 17B An example of generating a hierarchical region tree according to an embodiment of this disclosure is shown. For example... Figures 12A to 12B As shown, tile size region 8 is a neighbor of sub-region R1 at level 1. Furthermore, tile size regions 7, 9, and 10 are neighbors of sub-region R2 at level 1, and tile size region 11 is a neighbor of sub-region R5 at level 1. Therefore, the optimal region generator unit 206A further selects tile size regions 8, 7, 9, 10, and 11 as a third number of tile size regions, and adds the selected tile size regions 8, 7, 9, 10, and 11 to the corresponding sub-regions R1, R2, and R5, respectively, as follows. Figures 14A to 14B as well as Figures 15A to 15B As shown.

[0172] After adding the selected tile size regions 7, 9, and 10 to subregion R2, the optimal region generator unit 206A determines that the E constraint for subregion R2 is violated, and therefore divides subregion R2 into sub-subregions R7 and R8, as follows. Figures 15A to 15B As shown on the right-hand side, each of the subregions R7 and R8 satisfies predefined criteria N and E. As an example, subregions R7 and R8 satisfy each of constraints E and N.

[0173] In addition, such as Figure 12A as well as Figure 16A , Figure 16B As shown, tile size regions 12, 13, and 14 are neighbors of sub-region R5 at level 1, which includes tile size regions 5 and 11. Therefore, the optimal region generator unit 206A further selects tile size regions 12, 13, and 14 and adds them to sub-region R5. After adding the selected tile size regions 12, 13, and 14 to sub-region R5, the optimal region generator unit 206A further determines that one of the constraints E and N for sub-region R5 is violated, and therefore divides sub-region R5 into sub-sub-regions R9 and R10, as shown. Figures 17A to 17BAs shown on the right-hand side, each of the subregions R9 and R10 satisfies predefined criteria N and E. As an example, subregions R9 and R10 satisfy each of constraints E and N.

[0174] In addition, such as Figure 12A as well as Figures 17A to 17B As shown, tile size regions 15, 17, and 18 are neighbors of sub-region R6, which includes tile size region 6, at level 1, and size region 16 is a neighbor of sub-region R3, which includes tile size region 3, at level 1. Therefore, the optimal region generator unit 206A further selects tile size regions 15, 17, and 18, and adds the selected tile size regions 15, 17, and 18 to sub-region R5. Furthermore, the optimal region generator unit 206A selects tile size region 16 and adds it to sub-region R3. Adding the selected tile size regions to the corresponding sub-regions of the root block results in the generation of a hierarchical region tree that satisfies predefined criteria N and E. Thus, the connected component formation process S408 B of the differential video rendering method 400 is completed. Furthermore, without departing from the scope of this disclosure, the hierarchical region tree may be referred to as an "R-tree" in the accompanying drawings. Hereinafter, it will be used as... Figures 18 to 28B The component formation process S408 B in the case of dynamic video differential frames is explained.

[0175] In the case of dynamic video, motion estimation is used to predict reusable blocks from the previous frame N in frame N+1. Blocks from frame N that have absolutely zero difference in pixel values ​​and differ only in positional information (X, Y) in frame N+1 are considered reusable blocks. Reusable blocks do not need to be read from memory 220 again and can be reused by GPU 214 during rendering. In the following text, frame N may be referred to as the "first differential video frame" and frame N+1 may be referred to as the "second differential video frame".

[0176] In the region preparation of the hierarchical region tree, reuse blocks are processed individually in a way that allows them to be combined only with matching reuse regions. If no matching reuse region is found in the second differential video frame, it is added to the dirty tile region along with its location information based on efficiency.

[0177] According to embodiments of this disclosure, the differential generator 206D generates frame N+1 after generating frame N. As an example, the differential generator 206D generates frame N based on a first set of decoded blocks from a plurality of decoded block sets, and generates frame N+1 based on a second set of decoded blocks from a plurality of decoded block sets.

[0178] Figure 18 An example of a dynamic differential video frame according to an embodiment of the present disclosure is shown. Figure 18 This includes frame N and frame N+1. For example... Figure 18As shown, frame N+1 differs from frame N in its position information (x1, y1), (x2, y2), and (x3, y3). The blocks representing the position information (x1, y1), (x2, y2), and (x3, y3) in frame N+1 have absolutely zero difference in pixel values ​​relative to frame N, and only the position information (X, Y) differs in frame N+1. These blocks are considered reused blocks.

[0179] Figure 19 A reference dynamic differential video frame according to an embodiment of the present disclosure is shown. Figure 4 Another example of the differential frame generation process. In this disclosure Figure 6 The differential frame generation process described here is almost similar to Figure 19 The differential frame generation process only occurs during the differential frame generation process. Figure 19 Furthermore, in scenarios where there are reusable blocks of the same size as the fixed-size 16×16 blocks, they differ from each other. Therefore, the difference generator 206D performs the same operation as the referenced above. Figure 6 The process described is similar to the one described above, therefore references are omitted. Figure 19 Detailed explanation of the differential frame generation process.

[0180] Figure 20 Another example of a region preprocessing operation of a differential video rendering method 400 according to an embodiment of the present disclosure is shown. Figure 7 , Figure 8 The differential video rendering method described in section 400, specifically the region preprocessing S408 A, is almost similar to... Figure 20 Regional preprocessing operations. Only in Figure 20 The description further describes mapping 16x16 normalized fixed-size blocks, including reused blocks, to differentiate them from each other in a scene aligned on a 32x32 tile-size region on a per-tile basis. Hereinafter, a reused block may be referred to as a "reused tile" without departing from the scope of this disclosure. Therefore, the differential video rendering method 400 may also include generating a block list comprising multiple tile-size regions based on a list of dirty tiles and a list of reused tiles. (References omitted) Figure 20 Detailed explanation of the area preprocessing operations.

[0181] In addition, according to Figure 20 A tile-sized region consisting entirely of reused blocks is called a reused tile, and a tile-sized region partially composed of reused blocks is called a dirty tile. In an embodiment, a tile-sized region "partially composed" of reused blocks may include one or more reused blocks as well as other blocks. As an example, Figure 20 The tile size regions other than tile size regions 10 and 11 are dirty tiles, and tile size regions 10 and 11 are reused tiles. Therefore, according to Figure 7The region preprocessing S408 A and the optimal region generator unit 206A generate a block list including regions of multiple tile sizes based on the list of dirty tiles and the list of reused tiles.

[0182] Figure 21 A flowchart illustrating the connected component formation process of a reference dynamically differential video frame according to an embodiment of the present disclosure is shown. The connected component formation process of the reference dynamically differential video frame begins with a block list generated based on a list of dirty tiles and a list of reused tiles. For clarity, Figure 21 The process of forming connected components is used with Figure 4 The connected component formation process S408 B is represented by the same reference numerals. Figure 21 The connected component formation process S408 B includes a series of method steps S2100 to S2116. For ease of understanding, reference will be made to... Figures 22A to 27 Explain steps S2100 to S2116. Figures 22A to 27 An embodiment according to this disclosure is shown. Figure 21 An example of the connected component formation process S408 B.

[0183] At box S2100, the connected component formation process S408 B may include: selecting a root block from a block list, where the root block is a superset of all blocks in the block list. The block list includes a list of dirty blocks and a list of reused blocks. This process is similar to... Figure 11 The selection process is performed at box S408 B2. Initially, the root block is empty. Then, blocks are selected one by one from the block list and added to the root block. The newly added blocks become direct children of the root block. Hereinafter, without departing from the scope of the invention, a block may also be referred to as a "block-size region".

[0184] At box S2100, the connected component formation process S408 B may further include: selecting a first set of each of dirty blocks and reused blocks from the block list in an ordered order, and adding the first set of selected dirty blocks and reused blocks to the root block in an ordered order until the number of subregions of the root block exceeds a predefined criterion corresponding to N.

[0185] As an example, refer to Figures 22A to 22B The optimal region generator unit 206A selects tile size regions 1 to 6 from tile size regions 1 to 18, and adds tile size regions 1 to 6 to the root block in an ordered manner until the number of sub-regions (i.e., R1 to R6) of the root block exceeds the predefined value N = 6. The optimal region generator unit 206A further selects tile size regions 7, 8, 9, and 10.

[0186] The connected component formation process S408 B now proceeds from box S2100 to box S2102. (Reference) Figures 23A to 23BAt box S2102, the optimal region generator unit 206A checks whether the selected tile size regions 7, 8, 9, and 10 include any reused tiles. Figures 23A to 23B As shown, tile size regions 7, 8, and 9 are dirty tiles, and tile size region 10 is a reused tile. Therefore, if the result at box S2102 is yes, the process proceeds to box S2104, where the optimal region generator unit 206A checks for any matching reused regions in the neighbors of reused tile 10, and if found, inserts the reused tile separately from the dirty tile region (i.e., including sub-regions of the dirty tile), as shown in box S2106. As an example, as Figures 23A to 23B As shown, reused block 10 indicates that it includes a reused area and is inserted separately from sub-regions R1 to R5 into sub-region R6. Furthermore, as... Figures 24A to 24B As shown, reused tile 11 is a matching reused region among the neighbors of reused tile 10, and is inserted into the sub-region R6 that includes reused tile 10. Therefore, each time the optimal region generator unit finds any reused tile, it adds such a tile as a separate sub-region to the root block.

[0187] Now, returning to box S2102, if the result at box S2102 is negative, the optimal region generator unit 206A inserts tile-sized regions 7, 8, and 9 into their corresponding adjacent sub-regions R1 and R2. For example... Figures 22A to 22B As shown, tile size region 8 is a neighbor of subregion R1, and tile size regions 7 and 9 are neighbors of subregion R2. Therefore, tile size region 8 is inserted into subregion R1, and tile size regions 7 and 9 are inserted into subregion R2. Here, subregions R1 and R2 contain dirty tiles and can be considered as dirty regions. Therefore, according to Figure 21 The box S2108 inserts the tile size regions 7, 8, and 9 into the dirty region.

[0188] The connected component formation process S408 B now proceeds from boxes S2106 and S2108 to box S2110. At box S2110, the optimal region generator unit 206A determines whether either the reused region or the dirty region is full. For example, the optimal region generator unit 206A determines whether either the reused region or the dirty region violates or exceeds a predefined criterion corresponding to N. If the result of box S2110 is negative, the optimal region generator unit 206A continues to add adjacent tile-sized regions to the corresponding sub-regions of the root block, such that each sub-region satisfies a predefined criterion E, where E is less than or equal to a specific efficiency E', for example... Figure 21 As shown in S2112.

[0189] The connected component formation process S408 B now proceeds from box S2112 to box S2114. At box S2114, the optimal region generator unit 206A determines whether any sub-region includes the number of tile-size regions that violate the predefined criterion E. As an example, the optimal region generator unit 206A determines whether any sub-region exceeds the predefined criterion E. If the result of box S2114 is negative, the connected component formation process S408 B now proceeds from box S2114 to box S2116.

[0190] Furthermore, if the result of box S2114 is yes, then the optimal region generator unit 206A further selects a second set of each of the dirty tiles and the reused tiles, wherein the second set of dirty tiles is adjacent to the first set of dirty tiles. As an example, such as... Figures 22A to 22B as well as Figures 25A to 25B As shown, tile size regions 12 and 14 are neighbors of subregion R5, which includes tile size regions 5 and 6. Similarly, tile size region 13 is a neighbor of reused tile 11 in subregion R6. Therefore, tile size regions 12 and 14 are added to subregion R5, and tile size region 13 is added to subregion R6. When tile size region 13 is added to subregion R6, the connected component formation process S408 B now proceeds from box S2114 to box S2116. Furthermore, if the result of box S2110 is yes, the connected component formation process S408 B now proceeds from box S2110 to box S2116.

[0191] At box S2116, the optimal region generator unit 206A divides each of the full dirty regions and reused regions into one or more sub-regions. For example, the optimal region generator unit 206A divides each sub-region that exceeds at least one of the predefined criteria N and E into one or more sub-regions, such that each of the one or more sub-regions satisfies the predefined criteria N and E, and further, the optimal region generator unit 206A also updates the value of N based on the iterative method 1 described above.

[0192] Furthermore, if the result of box S2114 is yes, the optimal region generator unit 206A repeats processes S2100 to S2112 until the result of box S2112 becomes no. Additionally, each time the result of box S2114 becomes no, the optimal region generator unit 206A performs a segmentation operation on any level of the sub-region and sub-sub-region that violates constraint E. Furthermore, each time the result of box S2110 becomes yes, the optimal region generator unit 206A performs a segmentation operation on any level of the sub-region and sub-sub-region that violates constraint N, updating the value of N after segmentation.

[0193] Based on the example above, Figure 21The connected component formation process S408 B may include: generating a first level of a hierarchical region tree based on adding a first set of selected dirty tiles and reused tiles to the root block, and generating a second level of the hierarchical region tree based on dividing at least one sub-region into one or more sub-sub-regions. The first level of the hierarchical region tree is generated as a result of adding reused tiles and dirty tiles to the root block, and the second level of the hierarchical region tree is generated as a result of dividing sub-regions exceeding at least one of predefined criteria N and E.

[0194] The segmentation operation at box S2116 can also be performed on each of the multiple levels generated as the result of one or more iterations of the process at boxes S2100 to S2114. As an example, the optimal region generator unit 206A can perform segmentation on levels Level1, Level... i ...Level k Perform a partitioning operation, where each of these levels is arranged in order from the top of the root block towards the leaf blocks. Leaf blocks represent the lowest level, or Level. k Furthermore, these levels are arranged in descending order to form a hierarchical region tree. As an example, the second level is the level following the first level in the hierarchical region tree. This arrangement results in the formation of a hierarchical region tree. As an example, the optimal region generator unit 206A can arrange the first and second levels of the hierarchical region tree from the top of the root block toward the leaf blocks in the order in which the first and second levels of the hierarchical region tree were generated.

[0195] The optimal region generator unit 206A performs a segmentation operation for each of the multiple levels until each of the sub-regions and sub-sub-regions satisfies predefined criteria N and E at each level of the multiple levels.

[0196] Reference Figures 25A to 27 Further explanation of the segmentation operation. For example... Figures 25A to 25B As shown, sub-region R6 at level 1 is divided into sub-sub-regions R7, R8, and R9 to satisfy the predefined criteria N and E of sub-region R6. Furthermore, as... Figures 26A to 26B As shown, a tile-size region 15 is added separately as R10 from the reused regions R7 and R8 to satisfy the standard E of sub-region R6. Furthermore, as... Figures 25A to 25B , Figures 26A to 26B and Figure 27 As shown, tile sizes 12, 14, 17, and 18 are added to sub-region R5, which includes dirty tiles 5 and 6. Therefore, sub-region R5 becomes overcrowded and violates at least one of predefined criteria N and E. Consequently, sub-region R5 is divided into sub-sub-regions R12 and R13 to satisfy at least one of predefined criteria N and E.

[0197] therefore, Figure 21 The process of forming connected components, S408 B, results in the formation of a hierarchical region tree that includes multiple levels.

[0198] Now for reference Figures 28A to 28B It illustrates an embodiment of the present disclosure as Figure 21 An example illustration of the hierarchical region tree formed as a result of the connected component formation process S408 B. Figure 28A An example of a tile-size region including dirty and reused areas is shown. For example, Figure 28A The area of ​​size 9 in the map represents a reused area. Furthermore, in... Figures 28A to 28B The diagram shows an example representation of a hierarchical region tree. The topmost node I in the hierarchical region tree corresponds to the root block. The hierarchical region tree includes levels Level1, Level2, Level3, Level4, Level5, and Level6. i Level k The Level 1 of the hierarchical region tree includes i nodes (R1, e1), (R2, e2), ..., (R... N e i These nodes correspond to sub-regions directly connected to the root block, and are direct children of the root block. Level of the hierarchical region tree. i This includes sub-sub-regions of the sub-region at Level 1, and includes child nodes. For example, (R 11 e 11 ), (R 12 e 12 ) and (R 13 e 13 () is a child node of node (R1, e1). The bottom level (i.e., Level) of the hierarchical region tree. k This includes leaf blocks. The Level of a hierarchical region tree. k Each of the tile sizes in the area.

[0199] Now, refer to it again. Figure 4 After generating the hierarchical region tree S408 B, the differential video rendering method 400 proceeds to box S408 C. Box S408 C describes the optimal region forming process S408C for generating multiple optimal regions. At box S408 C, the differential video rendering method 400 may further include generating multiple optimal regions based on the hierarchical region tree satisfying predefined criteria N and E. The optimal region generator unit 206A generates multiple optimal regions through the optimal region forming process S408 C. The optimal region forming process S408 C also refers to the accompanying drawings. Figure 29 Performed in a series of steps as described.

[0200] Figure 29 An embodiment according to this disclosure is shown. Figure 4The flowchart of the optimal region formation process S408C of the differential video rendering method 400 is shown. The optimal region formation process S408C includes a series of method steps S2900 to S2910. The optimal region formation process S408C begins at the first level of the hierarchical region tree generated by the optimal region generator unit 206A.

[0201] The optimal region formation process S408C begins with the root block of the hierarchical region tree. The root block of the hierarchical region tree is the largest single region covering all actual pixel data in the form of dirty tiles and reused tiles. Level 1 of the hierarchical region tree is the smallest optimal region that includes at least all sub-regions of the root block. Furthermore, the leaf blocks in the last level of the region tree are the most efficient because they only cover dirty tiles. Additionally, if the optimal region includes any reused regions, those reused regions are considered dirty regions.

[0202] At block S2900 of the optimal region formation process S408 C, the optimal region generator unit 206A initializes the optimal region set (S) using all sub-regions at level 1 and resolves the hierarchical region tree using breadth-first search (BFS). Resolution begins at the root block of the hierarchical region tree and explores all nodes at the current level of the hierarchical region tree before moving to the node at the next level. The flow of the optimal region formation process S408 C now proceeds to block S2902.

[0203] At box S2902, the optimal region generator unit 206A selects a sub-region at the first level of the hierarchical region tree and adds the first level, including the sub-region, to S. After adding the first level, including the sub-region, to S, the optimal region generator unit 206A calculates the overall efficiency (E') of the first level and the total number of optimal regions (N') of the first level. The flow of the optimal region formation process S408 C now proceeds to box S2904. Here, the overall efficiency of each level of the hierarchical region tree can be given by the following formula: E L = Number of dirty tiles in region R / Total number of tiles in R. Here, R indicates the region that exists at the corresponding level in the hierarchical region tree.

[0204] At box S2904, the optimal region generator unit 206A determines whether the value of N' exceeds the predefined standard N. Furthermore, the optimal region generator unit 206A determines whether the value of E' exceeds the predefined standard E. If the determination at box S2904 is yes and the individual efficiency of the sub-region exceeds the predefined standard E, then the optimal region generator unit 206A replaces the sub-region with its sub-sub-region at the next level in S. Here, the individual efficiency of the sub-region at the corresponding level of the hierarchical region tree can be given by the following formula: (∑e SL +(EE R ))>E, where

[0205] e SL It represents the individual efficiency of the subregion at level L of the hierarchical region tree.

[0206] E is the overall efficiency of the hierarchical region tree, and

[0207] E R It refers to the efficiency of the entire region.

[0208] Furthermore, if the value of N' exceeds the predefined criterion N, the optimal region generator unit 206A can perform a rearrangement of S. Additionally, if the determination result at box S2904 is negative, the optimal region generator unit 206A can add the remaining sub-regions of level 1 to S, and can recalculate E' and N' after adding them to check if they satisfy each of the predefined criteria N and E. The flow of the optimal region formation process S408C now proceeds to box S2908.

[0209] At frame S2908, the optimal region generator unit 206A determines whether all sub-regions at the first level of the hierarchical region tree have been resolved. If the determination at frame S2908 is negative, the optimal region generator unit 206A repeats the process from frames S2902 to S2908 to prepare a final set of optimal regions for rendering on the graphics display. Conversely, if the determination at frame S2908 is positive, the optimal region generator unit 206A transmits the final set of optimal regions prepared based on the process from frames S2902 to S2908 to the graphics engine 216 for rendering on the graphics display. Therefore, the optimal region generator unit 206A uses at least one sub-region and at least one sub-sub-region formed as a result of the segmentation operation to generate one or more optimal regions from the root block of the hierarchical region tree toward the leaf blocks of the hierarchical region tree using a BFS (Browsing-First Search) approach.

[0210] The final set of optimal regions comprises a list of optimal regions, and each optimal region in the list satisfies predefined E and N criteria. Furthermore, one or more optimal regions are generated from the root block toward the leaf blocks such that the E' of each of the one or more optimal regions is greater than or equal to E.

[0211] Figure 30 An example of generating a final optimal set of regions according to embodiments of this disclosure is shown. Figure 30 As shown, the hierarchical region tree includes levels Level1, Level2, and Level3. i Level k The Level 1 of the hierarchical region tree includes nodes (R1, e1), (R2, e2), (R3, e3), and (R... n e nWhen performing parsing at Level 1 of the hierarchical region tree to find the optimal region, it is determined that node (R1, e1) needs to be split to achieve a specific efficiency E'. Therefore, node (R1, e1) is split into child nodes (R... 11 e 11 ), (R 12 e 12 ) and (R 13 e 13 Similarly, at levels Level1 and Level... i Level k At each level of the hierarchical region tree, a partitioning operation is performed such that the efficiency of each node from the root block to the leaf block is greater than or equal to E. As an example, a partitioning operation is performed such that each node after partitioning satisfies E' in descending order from the root block to the leaf block.

[0212] Once a split operation is performed at a level in the hierarchical region tree, the final optimal set of regions can be generated based on the sum of the nodes after splitting at the corresponding level in the hierarchical region tree.

[0213] As an example, according to Figure 30 The final optimal set of regions can be determined as follows:

[0214] Final optimal region set (S) = {(R)} 11 ,e 11 ),(R 12 ,e 12 ),(R 13 ,e 13 ),(R2,e2),(R3,e3),(R n ,e n )}.

[0215] Therefore, the cumulative efficiency of the child nodes of the hierarchical region tree improves the overall efficiency of the hierarchical region tree.

[0216] Figures 31A to 31B An example of a final optimal set of regions generated as a result of the optimal region formation process S408 C according to an embodiment of the present disclosure is shown. Figures 31A to 31B This describes the optimal set of regions that maximize efficiency while also adhering to a predefined criterion N. As an example, such as... Figures 31A to 31BAs shown, the final optimal region set generated as a result of the optimal region formation process S408 C includes four optimized sub-regions, namely {(R1,R2),(R3),(R4),(R5,R6)}. Therefore, the total number of sub-regions at level 1 of the hierarchical region tree is optimized through the optimal region formation process S408 C. Furthermore, these optimized regions can lead to a reduction in the overall bandwidth requirements of system 200 and an enhancement in the rendering performance of GPU 214 when rendering decoded video frames on a graphics display by graphics engine 216.

[0217] Now, refer to it again. Figure 4 Following the optimal region formation process S408C, the flow of the differential video rendering method 400 now proceeds to box S410. At box S410, the differential video rendering method 400 may include combining the final optimal region set with a set of differential regions. As an example, the graphics engine 216 performs the optimal region rendering process at box S410, and thus combines the final optimal region set with the differential region set. The flow of the differential video rendering method 400 now proceeds to box S412.

[0218] At frame S412, the differential video rendering method 400 may include rendering differential video frames on a graphics display based on a combination of a final optimal region set and a differential region group. As an example, graphics engine 216 renders differential video frames on the graphics display of output unit 224 based on a combination of a final optimal region set and a differential region group.

[0219] According to the differential video rendering method 400 and differential video rendering system 200 described above, compared with the existing full-decoded frame rendering method, the decoded frame can be partially rendered to the graphics display. Therefore, the pixels of the decoded frame are required for rendering only to a minimum, and thus, GPU DDR access can be minimized, and CPU-GPU DDR bandwidth can be improved.

[0220] Compared to existing full-frame rendering methods, the final optimal region set prepared by process S408, which includes processes S408A, S408B, and S408C of differential video rendering method 400, provides less unchanged region area, thereby saving a significant amount of DDR bandwidth, and the reduced rendering volume leads to improved performance of the differential video rendering system.

[0221] Because the final optimal set of regions can include only the minimum number of pixels required for rendering the decoded frames, the rendering performance of the differential video rendering system 200 can be improved, and the DDR bandwidth used by the GPU during rendering can also be reduced.

[0222] Furthermore, the GPU rendering pipeline may stall due to the large number of regions used for rendering. In such scenarios, the differential video rendering method 400 and differential video rendering system 200 of this disclosure can prevent GPU rendering pipeline stalls by applying a minimum optimal set of regions for rendering. Moreover, CPU consumption can be reduced simultaneously. Furthermore, the differential video rendering method 400 of this disclosure traverses the tree downwards until the GPU efficiency increases within the constraint of the maximum number of rectangles "N" as shown in the example above, thereby providing maximum efficiency within the allowable limit of the GPU's maximum region.

[0223] Figure 32A , Figure 32B , Figure 32C Exemplary advantages of the differential video rendering method 400 according to embodiments of the present disclosure over prior art video rendering methods are illustrated. From Figure 32A , Figure 32B , Figure 32C It can be seen that the video rendering methods of related technologies lead to problems such as high redundancy, high GPU DDR bandwidth access, and high rendering load per frame during full-frame rendering. However, from Figure 32A , Figure 32B , Figure 32C As can be seen, in contrast to existing video rendering methods, the differential video rendering method 400 can reduce redundancy by 20-30%, overcome GPU pipeline stalls, improve GPU rendering performance, reduce DDR bandwidth, and improve the efficiency of video rendering systems.

[0224] Figure 33A , Figure 33B , Figure 33C This illustrates another exemplary advantage of the differential video rendering method 400 according to embodiments of the present disclosure over prior art video rendering methods. From Figure 33A , Figure 33B , Figure 33C It can be seen that the rendering performance of each existing video rendering method is very low. However, from Figure 33A , Figure 33B , Figure 33C As can be seen, the differential video rendering method according to the embodiment has high rendering performance while maintaining low CPU utilization.

[0225] Furthermore, the differential video rendering method 400 can be implemented in low-end systems, such as systems with specifications or capabilities lower than other systems. The differential video rendering method 400 overcomes problems related to audio glitches in low-end systems and problems related to the commercialization of video texturing features in low-end systems by reducing bandwidth utilization and increasing memory access.

[0226] Furthermore, the differential video rendering method 400 can be implemented in various video rendering technologies. For example, the differential video rendering method 400 can be implemented in advanced direct TV applications, video texturing solutions (such as video ads on user interfaces), VR 360, animated video-based user interfaces that display animated video on graphical objects, and multi-view planes that support graphics rendering. Implementations of the differential video rendering method 400 are not limited to the examples described above. The differential video rendering method 400 can be implemented in any other video texturing solution different from the video texturing solutions described above.

[0227] Figure 34 A representative architecture 3400 is shown, providing the tools and development environment for the technical implementation of System 200 described herein. Figure 34 These are non-limiting examples, and it will be understood that many other architectures can be implemented to facilitate the functionality described herein. Architectures can be implemented in, for example... Figure 34 It runs on hardware of architecture 3400, which includes a processor, memory and various dedicated hardware components.

[0228] Architecture 3400 can include an operating system, libraries, frameworks, or middleware. The operating system can manage hardware resources and provide common services. The operating system can include, for example, a kernel, services, and drivers that define hardware interface layers. Drivers can be responsible for controlling the underlying hardware or interfaced with the underlying hardware. For example, depending on the hardware configuration, drivers can include display drivers, camera drivers, etc. Drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0229] The hardware interface layer includes libraries, which may include system libraries (e.g., the C standard library) that provide functions such as memory allocation, string manipulation, and mathematical functions. Additionally, libraries may include API libraries, such as audiovisual media libraries (e.g., multimedia databases supporting the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), databases (e.g., SQLite providing various relational database functionalities), and web libraries (e.g., WebKit providing web browsing functionality).

[0230] Middleware can provide higher-level public infrastructure, such as various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Middleware can provide a wide range of additional APIs that can be utilized by applications and / or other software components / modules, some of which may be specific to a particular operating system or platform.

[0231] As used in this disclosure, the term "module" can refer to a specific unit including hardware, software, and firmware, or any combination thereof. For example, a module can be used interchangeably with a cell, logic, logic block, component, or circuit. A module can be the smallest unit or part of performing one or more specific functions. Modules can be formed mechanically or electronically. For example, the "module" disclosed herein can include at least one of the following: known or future ASIC (Application-Specific Integrated Circuit) chips, FPGA (Field-Programmable Gate Array) devices, and programmable logic devices.

[0232] Furthermore, architecture 3400 depicts an aggregation of mechanisms based on audio / video processing devices and ML / NLP-based mechanisms according to embodiments of this subject. The user interface defined as input and interaction 3401 refers to the overall input. It may include one or more of the following: touchscreen, microphone, camera, etc. First hardware module 3402 depicts dedicated hardware for the ML / NLP-based mechanisms. In the example, first hardware module 3402 may include one or more of a neural processor, FPGA, DSP, GPU, etc.

[0233] The second hardware module 3412 describes dedicated hardware for performing data segmentation and transmission. The ML / NLP-based framework and API 3404 corresponds to the hardware interface layer for performing ML / NLP logic 3406 based on the underlying hardware. In the example, the framework can be one or more of the following: Tensorflow, Café, NLTK, GenSim, ARM computing, etc. The simulation framework and API 3414 can include one or more of the following: audio kernel, audio suite, Unity, Unreal, etc.

[0234] Database 3408 describes a pre-trained database. Database 3408 can be remotely accessed via the cloud by ML / NLP logic 3406. In other examples, database 3408 may reside partly in the cloud and partly on a device, based on the use of statistics.

[0235] Another database 3418 involves storage. Database 3418 can be accessed remotely via the cloud. In other examples, database 3418 may reside partly in the cloud and partly on the device, based on usage statistics.

[0236] A rendering module 3405 is provided for rendering audio output and triggering further utility operations. The rendering module 3405 can function as a display / touchscreen, monitor, speaker, projection screen, etc.

[0237] General-purpose hardware and driver module 3403 corresponds to, for example Figure 35The mentioned computing system 3500, and instantiated drivers for general-purpose hardware units and special-purpose units (3402, 3412).

[0238] In this example, the ML mechanism under this architecture 3400 can be remotely accessible and cloud-based, thus allowing remote access via a network connection. Audio / video processing devices can be configured for remote access to the NLP / ML module, and the analog module can include skeleton components such as microphones, cameras, screens / monitors, speakers, etc.

[0239] Furthermore, at least one of the multiple modules of the mesh network can be implemented via AI based on ML / NLP logic 3406. AI-related functions can be performed via non-volatile memory, volatile memory, and a processor constituting the first hardware module 3402 (i.e., dedicated hardware for ML / NLP-based mechanisms). The processor can include one or more processors. Here, the one or more processors can be general-purpose processors (such as central processing units (CPUs), application processors (APs), etc.), graphics-only units (such as graphics processing units (GPUs)), vision processing units (VPUs), and / or AI-specific processors (such as neural processing units (NPUs)). The aforementioned processors collectively correspond to... Figure 35 The processor is 3502.

[0240] One or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0241] Here, "provided through learning" means by applying learning logic / techniques to multiple learning data sets to create predefined operating rules or AI models with desired characteristics. "Acquired through training" refers to obtaining predefined operating rules or AI models configured to perform desired features (or purposes) by training a basic AI model with multiple training data sets using training techniques. Learning can be performed within the device (i.e., architecture 3400 or system 3500) that performs the AI ​​according to the embodiment, and / or can be implemented via a separate server / system.

[0242] AI models can consist of multiple neural network layers. Each layer has multiple weight values, and neural network layer operations are performed by calculating the results of the previous layer and the operations on the multiple weights. Examples of neural networks include, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), and Deep Q-Networks.

[0243] ML / NLP Logic 3406 is a method for training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0244] Figure 35 Another exemplary implementation according to the embodiments is shown, as well as yet another typical hardware configuration of architecture 3400 in the form of computer system 3500. Computer system 3500 may include a set of instructions that can be executed to cause computer system 3500 to perform any one or more of the disclosed methods. Computer system 3500 may operate as a stand-alone device, or may be connected to other computer systems or peripheral devices, for example, using a network.

[0245] In a networked deployment, computer system 3500 can operate as a server or client user computer in a server-client user network environment, or as a peer-to-peer (or distributed) network environment. Computer system 3500 can also be implemented as or incorporated across various devices, such as personal computers (PCs), tablet PCs, personal digital assistants (PDAs), mobile devices, handheld computers, laptop computers, desktop computers, communication equipment, wireless telephones, landline telephones, network equipment, network routers, switches, or bridges, or any other machine capable of executing (sequentially or otherwise) a set of instructions that specifies the actions to be taken by that machine. Furthermore, while a single computer system 3500 is shown, the term "system" should also be understood to include any collection of systems or subsystems that individually or jointly execute one or more sets of instructions to perform one or more computer functions.

[0246] Computer system 3500 may include processor 3502, such as a central processing unit (CPU), graphics processing unit (GPU), or both. Processor 3502 may be a component in a variety of systems. For example, processor 3502 may be part of a standard personal computer or workstation. Processor 3502 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), servers, networks, digital circuits, analog circuits, combinations thereof, or other devices now known or to be developed in the future for analyzing and processing data. Processor 3502 may implement software programs, such as manually generated (i.e., programmed) code.

[0247] Computer system 3500 may include memory 3504, such as memory 3504 that can communicate via bus 3508. Memory 3504 may include (but is not limited to) computer-readable storage media such as various types of volatile and non-volatile storage media, including (but not limited to) random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc. In one example, memory 3504 includes a cache or random access memory for processor 3502. In alternative examples, memory 3504 is separate from processor 3502, such as processor cache memory, system memory, or other memory. Memory 3504 may be an external storage device or database for storing data. Memory 3504 is operable to store instructions executable by processor 3502. The functions, actions, or tasks shown or described in the figures can be performed by a processor 3502 programmed to execute the instructions stored in memory 3504. Functions, actions, or tasks are independent of a specific type of instruction set, storage medium, processor, or processing strategy, and can be executed by software, hardware, integrated circuits, firmware, microcode, etc., operating individually or in combination. Similarly, processing strategies can include multiprocessing, multitasking, parallel processing, etc.

[0248] As shown in the figure, the computer system 3500 may or may not include a display 3510, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flat panel display, a solid-state display, a cathode ray tube (CRT), a projector, a printer, or other display devices now known or to be developed in the future for outputting specific information. The display 3510 may serve as an interface between the user and the functions of the processor 3502, or specifically as an interface with software stored in the memory 3504 or the drive unit 3516.

[0249] Additionally, computer system 3500 may include an input device 3512 configured to allow a user to interact with any component of system 3500. Computer system 3500 may also include a disk drive or optical drive, for example, in drive unit 3516. The disk drive may include computer-readable medium 3522 in which one or more instruction sets 3524, such as software, may be embedded. Furthermore, the instructions 3524 may embody one or more methods or logic as described. In a specific example, during execution by computer system 3500, the instructions 3524 may reside wholly or at least partially within memory 3504 or processor 3502.

[0250] Embodiments may involve a computer-readable medium that includes or receives and executes instructions 3524 in response to a propagation signal, enabling devices connected to network 3526 to transmit voice, video, audio, images, or any other data on network 3526. Furthermore, instructions 3524 may be sent or received on network 3526 via communication interface 3520 (which may be, for example, a communication port) or using bus 3508. Communication interface 3520 may be part of processor 3502 or may be a separate component. Communication interface 3520 may be created in software or may be a physical connection in hardware. Communication interface 3520 may be configured to connect to network 3526, external media, display 3510, or any other component or combination thereof in system 3500. Connection to network 3526 may be a physical connection, such as a wired Ethernet connection, or may be established wirelessly as discussed later. Similarly, additional connections to other components of system 3500 may be physical or may be established wirelessly. Network 3526 can optionally be directly connected to bus 3508.

[0251] Network 3526 may include a wired network, a wireless network, an Ethernet AVB network, or a combination thereof. The wireless network may be a cellular telephone network, an 802.11, 802.16, 802.20, 802.1Q, or WiMax network. Furthermore, network 3526 may be a public network (e.g., the Internet), a private network (e.g., an intranet), or a combination thereof, and may use a variety of networking protocols currently available or developed in the future, including but not limited to TCP / IP-based networking protocols. The system is not limited to operation using any particular standard and protocol. For example, standards for transmission over the Internet and other packet-switched networks (e.g., TCP / IP, UDP / IP, HTML, and HTTP) may be used.

[0252] While this disclosure has been described in specific language, it is not intended to impose any limitation. It will be apparent to those skilled in the art that various working modifications can be made to the method to achieve the inventive concept taught herein.

[0253] The accompanying drawings and the foregoing description provide examples of embodiments. Those skilled in the art will understand that one or more of the described elements can be well combined into a single functional element. Alternatively, certain elements may be divided into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of processes described herein may be changed and is not limited to the manner described herein.

[0254] Furthermore, the operations in any flowchart do not need to be performed in the order shown; nor is it necessary to perform all actions. Moreover, actions that do not depend on other actions can be performed in parallel with other actions. The scope of the embodiments is by no means limited to these specific examples. Many variations (whether or not explicitly given in the specification) are possible, such as differences in structure, size, and material use. The scope of the embodiments is at least as broad as given by the appended claims.

[0255] The benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, any benefits, advantages, solutions to problems, and any components that may cause any benefit, advantage, or solution to occur or become more apparent should not be construed as key, essential, or fundamental features or components of any or all claims.

Claims

1. A differential video rendering system, comprising: Graphics Processing Unit (GPU); A graphics display, coupled to the GPU; A video decoder is configured to decode a bitstream of encoded data into a set of multiple decoded blocks; At least one processor is configured as follows: A first differential video frame is generated based on a first set of the plurality of decoded block sets. The first differential video frame includes a plurality of differential region sets. Each of the plurality of difference region sets is normalized into a fixed-size block to provide a normalized plurality of difference region sets. The corresponding sets in the normalized multiple differential region sets are mapped to align with the corresponding tile size regions in the multiple tile size regions conforming to the GPU. A hierarchical region tree is generated based on the normalized set of multiple differential regions mapped to the multiple tile-size regions, and Multiple optimal regions are generated based on a hierarchical region tree that satisfies predefined criteria corresponding to a predefined number of optimal regions and a predefined efficiency parameter; and a graphics rendering engine is configured to render the first differential video frame on the graphics display based on the multiple optimal regions and a set of differential regions.

2. The differential video rendering system according to claim 1, wherein, The corresponding optimal region among the multiple optimal regions includes a set of tile-sized regions and satisfies the predefined criteria.

3. The differential video rendering system according to claim 2 further includes a central processing unit (CPU), wherein the CPU is configured as follows: The predefined optimal number of regions is determined based on experimental values ​​associated with at least one of the GPU's clock speed, the CPU's clock speed, and the number of processing cores included in the GPU; and The predefined efficiency parameter is determined based on at least one system variable parameter corresponding to the GPU's clock speed, the GPU's bandwidth, the memory configuration coupled to the GPU, the memory bus width, and the number of processing cores included in the GPU, wherein... The predefined efficiency parameter corresponds to the GPU's processing capability to handle the most differential regions with the minimum bandwidth.

4. The differential video rendering system according to claim 3, wherein, The predefined number of optimal regions corresponds to the maximum number of optimal regions that can be passed to the GPU's rendering pipeline without affecting the overall performance of the GPU.

5. The differential video rendering system according to claim 3, wherein, The at least one processor is further configured to: Determine a first number of tile-size regions among the plurality of tile-size regions, including the minimum number of difference regions; Perform a marking process to mark the first number of tile-sized regions having the minimum number of difference regions as dirty tiles; as well as A list of dirty tiles is generated based on the marking process.

6. The differential video rendering system according to claim 5, wherein, The at least one processor is further configured to: Generate a block list that includes the multiple block size regions based on the list of dirty blocks; Select a root block from the block list, wherein the root block is a superset of all blocks in the block list; Select a second number of tile size regions from the plurality of tile size regions in the block list in an ordered manner; The selected second number of tile-sized regions are added to the root block in an ordered manner until the number of sub-regions of the root block exceeds the predefined standard corresponding to the predefined optimal number of regions; and The first level of the hierarchical region tree is generated by adding the selected second number of tile-sized regions to the root block.

7. The differential video rendering system according to claim 6, wherein, The at least one processor is further configured to: Select a third number of tile size regions from the plurality of tile size regions in the block list, wherein the third number of tile size regions is adjacent to the second number of tile size regions; Add the selected third number of tile-sized regions to the first level of the hierarchical region tree in an ordered order; Determine that at least one sub-region of the root block at the first level exceeds at least one of the predefined optimal region number and the predefined efficiency parameter; and The at least one sub-region exceeding at least one of the predefined optimal region number and the predefined efficiency parameter is divided into a first plurality of sub-sub-regions, such that each sub-sub-region in the first plurality of sub-sub-regions satisfies the predefined criteria.

8. The differential video rendering system according to claim 7, wherein, The at least one processor is further configured to generate a second level of the hierarchical region tree based on dividing the at least one sub-region into the first plurality of sub-sub-regions. Wherein, the second level of the hierarchical region tree includes the first plurality of sub-regions, and The second level corresponds to the level following the first level of the hierarchical region tree.

9. The differential video rendering system according to claim 8, wherein, The hierarchical region tree includes multiple levels. Wherein, the plurality of levels include at least the first level, and The at least one processor is further configured to: Determine whether any sub-region at each of the plurality of levels exceeds at least one of the predefined optimal number of regions and the predefined efficiency parameter; and The sub-region that exceeds at least one of the predefined optimal region number and the predefined efficiency parameter is divided into a second plurality of sub-regions, such that each sub-region at the corresponding level of the plurality of levels satisfies the predefined criteria.

10. The differential video rendering system according to claim 9, wherein, The bottom level of the hierarchical region tree includes leaf blocks. The at least one processor is further configured to: generate the plurality of optimal regions from the root block toward the leaf block based on the segmentation of the at least one sub-region and at least one sub-sub-region of any sub-sub-region, and The plurality of optimal regions are generated from the root block toward the leaf block, such that each of the plurality of optimal regions has an efficiency greater than or equal to the predefined efficiency parameter.

11. The differential video rendering system according to claim 8, wherein, The at least one processor is further configured to: Arrange the first and second levels from the top of the root block toward the leaf block in the order in which the first and second levels are generated; as well as The hierarchical region tree is generated based on the aforementioned arrangement.

12. The differential video rendering system according to claim 3, wherein, The at least one processor is further configured to generate a second differential video frame based on a second set of the plurality of decoded block sets. The generation of the first differential video frame occurs before the generation of the second differential video frame. Among these, a first number of tile-size regions within the plurality of tile-size regions correspond to reused tiles. Among them, the second number of tile-size regions in the plurality of tile-size regions corresponds to dirty tiles. The reused tiles are composed entirely of reused blocks. Wherein, the reused block corresponds to a block in the first differential video frame and the second differential video frame that has the same pixel value, and The dirty block includes the reuse block.

13. The differential video rendering system according to claim 12, wherein, The at least one processor is further configured to: A block list including the multiple tile size regions is generated based on the list of reused tiles and the list of dirty tiles; Select a root block from the block list, wherein the root block is a superset of all blocks in the block list; Select the first set of the dirty blocks and the reused blocks in an ordered order; The selected dirty tiles and the first set of reused tiles are added to the root block in ordered order until the number of sub-regions of the root block exceeds the predefined optimal number of regions, wherein each reused tile in the first set of reused tiles is added to the root block as a separate sub-region; and The first level of the hierarchical region tree is generated by adding the first set of the selected dirty tiles and the reused tiles to the root block.

14. The differential video rendering system according to claim 13, wherein, The at least one processor is further configured to: Select a second set of each of the dirty blocks and the reused blocks, wherein the second set of dirty blocks is adjacent to the first set of dirty blocks; The second set of the selected dirty tiles and the reused tiles are added to the first level of the hierarchical region tree in an ordered order; Determine that at least one sub-region of the root block at the first level exceeds at least one of the predefined optimal region number and the predefined efficiency parameter; and The at least one sub-region exceeding at least one of the predefined optimal region number and the predefined efficiency parameter is divided into a first plurality of sub-sub-regions, such that each sub-sub-region in the first plurality of sub-sub-regions satisfies the predefined criteria.

15. The differential video rendering system according to claim 14, wherein, The at least one processor is further configured to generate a second level of the hierarchical region tree based on dividing the at least one sub-region into the first plurality of sub-sub-regions. Wherein, the second level of the hierarchical region tree includes the first plurality of sub-regions, and The second level corresponds to the level following the first level of the hierarchical region tree.

Images