Video streams decoding and rendering

Abstract

A method and system for rendering, the method comprising: receiving by a processor a video stream comprising a set having a first order of YUV and chroma frames, the YUV frames outnumbering the chroma frames, and wherein each frame depicts an area or a respective portion thereof, a first chroma frame affecting a first part of the area; copying information of the first chroma frame and providing the information to a second processor for decoding; receiving a YUV frames sequence, each affecting a respective second part of the area; rendering each of the YUV frames, while maintaining the collection of second parts affected by the YUV frames received after the first chroma frame; receiving processing results of the first chroma frame; and rendering the first part according to the processing results, excluding the collection of second parts, whereby the stream is rendered in an order different from the first order.

Description

VIDEO STREAMS DECODING AND RENDERINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of provisional patent application No. 63 / 553,725, entitled “STREAMING VIDEO CODEC” filed February 15, 2024, which is hereby incorporated by reference in its entirety without giving rise to disavowment.FIELD OF THE INVENTION

[0002] The disclosure relates to the field of video decoding in general, and to enabling efficient video decoding and rendering on a browser, in particular.BACKGROUND

[0003] Video decoders are hardware and / or software entities designed to receive compressed video data, e.g., encoded frames, and decode the frames to reconstruct the original video frames for rendering or for further processing.

[0004] YUV and Chroma are specific color representation formats used in video encoding and decoding.

[0005] In YUV frames, also referred to as luma frames, the Y (luminance) channel is fully sampled, and the U and V (chrominance) channels are also sampled at the same rate as Y (i.e., no chroma subsampling).

[0006] The chroma frames comprise fuller color information, and therefore comprise only the chrominance components, U which represents the difference between blue and the luminance, and V which represents the difference between red and the luminance.

[0007] It is appreciated that the YUV frames and the chroma frames have the same format and can thus be decoded in the same manner, which enhances the consistency, efficiency, interoperability, and reduced complexity.

[0008] Remote Desktop Protocol (RDP) is a proprietary protocol developed by Microsoft™, used to enable remote access to a computer's desktop environment over a network connection. Thus, RDP allows a user to connect to a remote computer as if the user is physically sitting in front of it, interacting with its graphical user interface (GUI). Among others, RDP may be used for remote administration, technical support, and accessingworkstations or servers from a distant location. RDP typically supports strong encryption, thereby ensuring secure data transmission between the client and server.

[0009] Using the RDP protocol, the graphical user interface of the remote system is transmitted by a server over the network, after the actual process of encoding images for transmission. The process may involve breaking down the screen content into YUV color components (luma and chroma). The screen may be broken into a stream combined of chroma frames and luma frames at changing ratios, where the luma frames outnumber the chroma frames. In some situations, the higher the frame rate, the lower is the ratio between the number of chroma frames and the number of luma frames. For example, at higher frame rates, the ratio may be about 15%, and at lower frame rates the ratio may be about 25%.

[0010] When the RDP frames received at a client device, they need to be and are rendered.SUMMARY

[0011] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods which are meant to be exemplary and illustrative, not limiting in scope.

[0012] There is provided, in accordance with an embodiment of the disclosed subject matter, a method for rendering, the method comprising: receiving by a first processor a video stream comprising a set having a first order of YUV frames and chroma frames, wherein the YUV frames outnumber the chroma frames, and wherein each of the YUV frames and the chroma frames depicts a display area or a respective portion thereof, the chroma frames comprising a first chroma frame affecting a first part of the display area; copying information of the first chroma frame and providing the information to a second processor for decoding; receiving a sequence comprising at least one YUV frame, each YUV frame of the at least one YUV frame affecting a respective second part of the display area, wherein accumulation of the respective second part of each of the at least one YUV frame forms a collection of second parts; rendering each YUV frame of the at least one YUV frame, while maintaining the collection of second parts affected by the at least one YUV frame received since the first chroma frame was received; receiving processing results of the first chroma frame, the processing results comprising a decoding of the information; and rendering the first part of the display area in accordance with the processing results of the first chroma frame, said rendering excluding the collection of second parts, whereby rendering of the video stream is performed in a second order, the second order different from the first order. Within the method, a collection of second parts is optionally maintained for each received chroma frame, and wherein the collection of second parts is used when rendering the chroma frame. Within the method, when a chroma frame and a YUV frame are received at times differing in at most a predetermined period of time, the YUV frame is optionally rendered, and the respective second part of the YUV frame is optionally not added to the collection of second parts. Within the method, each chroma frame is optionally processed by the second processor using a separate thread. Within the method, a first chroma frame and a second chroma frame are optionally processed in parallel. Within the method, the video stream is optionally a Remote Desktop Protocol (RDP) stream, and the method is optionally utilized for the rendering the RDP stream over a browser. Within the method, the browser optionally has limited access to contents of the chroma frames, thereby preventing the browser from performing the second processing. Within the method, processing the first chroma frameoptionally comprises at least one operation from the group consisting of: color space conversion, data restructuring, texture preparation, synchronization calculations, buffer management, optimization, error checking and validation. Within the method, the first processor is optionally a graphic processing unit (GPU) and the second processor is optionally a central processing unit (CPU). The method can further comprise: receiving by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received; receiving by the first processor after the processing results of the first frame are received, second processing results of the second chroma frame; rendering each YUV frame of at least one YUV frame received between when the first processing results were received and the second processing results were received; and rendering the area in accordance with the processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received. The method can further comprise: receiving by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received; rendering each YUV frame of at least one YUV frame received between when the second chroma frame was received and when the second processing results were received; receiving by the first processor, second processing results of the second chroma frame, before the processing results of the first frame are received; and rendering the area in accordance with the second processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received. Within the method, exclusion of the collection of second parts optionally comprises subtracting the collection of second parts from an area affected by the first chroma frame. The method can further comprise sending an acknowledgement signal after the first chroma frame is received and before the processing results are received, thereby eliminating slowdown by a sender of the first chroma frame, and enabling the sender to send a second chroma frame. Within the method, a proportion between a number of the chroma frames and a number of the YUV frames is optionally lower when a frame rate is higher. Within the method, a ratio of a number of the chroma frames and a number of the YUV frames is optionally between 10% and 35%. Within the method, rendering is optionally performed using a shader that combines luminosity from a luma texture and chrominance components from the processed Chroma frame.

[0013] There is provided, in accordance with another embodiment of the disclosed subject matter, a system having a processor, the processor being configured to perform the steps of:receiving by a first processor a video stream comprising a set having a first order of YUV frames and chroma frames, wherein the YUV frames outnumber the chroma frames, and wherein each of the YUV frames and the chroma frames depicts a display area or a respective portion thereof, the chroma frames comprising a first chroma frame affecting a first part of the display area; copying information of the first chroma frame and providing the information to a second processor for decoding; receiving a sequence comprising at least one YUV frame, each YUV frame of the at least one YUV frame affecting a respective second part of the display area, wherein accumulation of the respective second part of each of the at least one YUV frame forms a collection of second parts; rendering each YUV frame of the at least one YUV frame, while maintaining the collection of second parts affected by the at least one YUV frame received since the first chroma frame was received; receiving processing results of the first chroma frame, the processing results comprising a decoding of the information; and rendering the first part of the display area in accordance with the processing results of the first chroma frame, said rendering excluding the collection of second parts, whereby rendering of the video stream is performed in a second order, the second order different from the first order. Within the system, the collection of second parts is optionally maintained for each received chroma frame from the chroma frames, and wherein the collection of second parts is used when rendering the chroma frame. Within the system, when a chroma frame from the chroma frames and a YUV frame from the YUV frames are received at times differing in at most a predetermined period of time, the YUV frame is rendered, and the respective second part of the YUV frame is not added to the collection of second parts. Within the system, the processor is optionally further configured to: receive by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received; receive by the first processor after the processing results of the first frame are received, second processing results of the second chroma frame; render each YUV frame of at least one YUV frame received between when the first processing results were received and the second processing results were received; and render the area in accordance with the processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received. Within the system, the processor is optionally further configured to: receive by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received; render each YUV frame of at least one YUV frame received between when the second chroma frame was received and when the second processing results were received; receive by the first processor,second processing results of the second chroma frame, before the processing results of the first frame are received; and render the area in accordance with the second processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received. Within the system, each chroma frame from the chroma frames is optionally processed by the second processor using a separate thread.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Exemplary embodiments are illustrated in referenced figures. Dimensions of components and features shown in the figures are generally chosen for convenience and clarity

[0015] Fig. 1 is a flowchart of steps taken in an exemplary scenario in which an RDP stream is to be rendered by a browser, in accordance with some exemplary embodiments of the disclosure;

[0016] Fig. 2 is a flowchart of steps taken in an exemplary scenario in which an RDP stream comprising two chroma frames is to be rendered by a browser, in accordance with some exemplary embodiments of the disclosure;

[0017] Fig. 3 is a flowchart of steps taken in another exemplary scenario in which an RDP stream comprising two chroma frames is to be rendered by a browser, in accordance with some exemplary embodiments of the disclosure;

[0018] Fig. 4 is a flowchart of steps in a method for rendering a video stream, in accordance with some exemplary embodiments of the disclosure;

[0019] Fig. 5 is an illustration of a representation of a YUV444 frame in three planes, in accordance with some exemplary embodiments of the disclosure; and

[0020] Fig. 6 is a representation of packing the YUV444 into two YUV420 frames, in accordance with some exemplary embodiments of the disclosure.DETAILED DESCRIPTION

[0021] YUV444 is a color space that describes color as a combination of luma (Y) and two Chroma components (U and V). Luma represents the brightness of the color, while Chroma represents the difference of the red and blue from the luminance.

[0022] A stream comprising YUV frames (also referred to as luma frames) and chroma frames may be received at an RDP client and needs to be decoded and rendered.

[0023] Video decoders used in real-time streaming may need to be able to complete their rendering operation within a predefined time duration, also referred to as the latency requirement. For example, in some cases the frame rate may be 30 FPS, and therefore rendering should be performed within 33ms. However, in some situations, for example when code is being executed on CPU and not on GPU, when color conversion is performed back and forth, or due to any other reason, the naive solution of processing each frame as it is received and in accordance with the frame order, may violate the latency requirement.

[0024] In some exemplary embodiments, the video stream comprises two interleaved frame streams, including a YUV stream and a Chroma stream. The YUV stream (e.g., YUV420, also referred to as the luma stream) may represent a frame using YUV model, including a luma component (Y) and two chroma components (U and V). The Chroma stream (e.g., Chroma420) may represent chroma information only (excluding luma information), but may be enclosed into all Y, U, V planes.

[0025] As the RDP stream does not always need constantly high resolution, in some situations the luma stream and the chroma stream are interleaved, with the frames of the luma stream outnumbering the frames of the chroma stream. For example, the number of chroma frames per time unit may be between 15% and 25%, or between 10% and 30% of the number of YUV frames. The ratio may depend also on the frame rate, for example in high frame rates the number of chroma frames per luma frames may be smaller than in low frame rate.

[0026] One technical problem associated with the disclosure is that a browser, which may often need to display an RDP stream, may not have full access to the resources of the underlying Graphical Processing Unit (GPU). In particular, the browser may be unable to access the decoding capabilities of the GPU, which can prevent the browser from decoding the chroma frames. Browsers, which are designed to provide a secure and sandboxed environment for web applications, often have restricted access to the underlying hardwareresources of the device they are running on. This restriction is particularly pronounced when it comes to accessing the full capabilities of the Graphical Processing Unit (GPU).

[0027] GPUs are specialized processors optimized for handling complex graphical operations, including video decoding. They typically offer hardware-accelerated decoding capabilities that can significantly improve performance and efficiency when processing video streams. However, due to security considerations and the need to maintain a consistent cross-platform experience, browsers are generally not given direct access to these low -level GPU functions.

[0028] In the context of RDP streams, which often involve real-time video transmission, the inability to utilize the GPU's decoding capabilities can be particularly problematic. Chroma frames, which contain color information separate from luminance data, require specific decoding processes that are typically optimized for GPU execution. Without access to these specialized decoding functions, the browser must rely on software-based decoding methods, which can be significantly slower and less efficient.

[0029] This limitation can lead to several issues, such as increased CPU usage (e.g., software-based decoding places a higher burden on the CPU, potentially affecting overall system performance), higher latency (e.g., the decoding process may take longer, leading to delays in displaying the RDP stream), reduced frame rates (e.g., the browser might struggle to keep up with high frame rate streams, resulting in dropped frames or stuttering playback), increased power consumption (e.g., software decoding is generally less power-efficient than hardware-accelerated decoding, which can be a concern for mobile devices or devices with limited power sources which are not connected to a power grid), scalability issues (e.g., as the resolution and complexity of RDP streams increase, the performance gap between hardware and software decoding becomes more pronounced), or the like. Addressing this technical problem requires innovative solutions that can work within the constraints of browser environments while still providing efficient decoding and rendering of RDP streams, particularly for chroma frames.

[0030] Another technical problem associated with the disclosure is that the naive solution of synchronously copying the chroma frame to a memory unit of an associated Central Processing Unit (CPU), decoding the chroma frame by the CPU, and transferring the results back to the GPU for rendering, will generally not comply with the latency requirements. Thetwo-way transmission in addition to the processing time can make rendering the stream rather slow.

[0031] Another technical problem associated with the disclosure is that the server is awaiting an acknowledgement (ACK) from the client for each chroma frame before transmitting the next one. This acknowledgement process may introduce delays in certain situations. For example, when processing of a chroma frame is delayed on the client side, the corresponding ACK may also be delayed. As a result, the server may slow down the transmission of subsequent frames. This slowdown in frame transmission may potentially impact the overall display quality and smoothness of the video stream.

[0032] One technical solution may involve processing and rendering each YUV frame as it is received. Additionally, the solution may include maintaining, for each chroma frame, a collection of areas affected by YUV frames received after that chroma frame. Each affected area may comprise one or more rectangles, with the accumulation generating a collection of rectangles. In some aspects, the rectangles may have sides parallel to the axes of the display area. It is noted, however, that in some embodiments, the area may be represented using other means, such as polygons or non-geometric shapes, and the disclosed subject matter is not limited to specific representation.

[0033] When a chroma frame is received, it may be copied from the handling GPU's memory to the CPU memory for processing. This approach may address limitations where the browser has restricted access to certain GPU functionality and frame content. Copying the data may be enabled using methods such as the CopyToQ function of the of the browser's WebCodecs Application Program Interface (API), and which uses the VideoFrame. CopyToQ method that copies the otherwise opaque object's data into the CPU memory.

[0034] The CPU may then process the chroma frame. Once processing is complete, the CPU may provide the decoded (and optionally further processed) data back to the GPU for rendering.

[0035] The chroma frame may then be rendered. In some implementations, rendering may be performed only for areas affected by the chroma frame that are not included in the accumulated collection of rectangles affected by subsequently received YUV frames. By subtracting the areas updated by later YUV frames, only the non-updated areas may be rendered.

[0036] Excluding certain rectangles may help prevent image degradation by avoiding the rendering of outdated chroma information. Additionally, this partial rendering approach may potentially improve rendering speed in some cases.

[0037] In some aspects, two or more chroma frames may be processed concurrently. For example, a second chroma frame may be received before a first chroma frame is rendered. Concurrent processing may occur regardless of whether the processing results of the second chroma frame are received before or after those of the first chroma frame. In such scenarios, each chroma frame may be rendered without the areas affected by YUV frames received after that specific chroma frame.

[0038] In some cases, a chroma frame may be received together with a YUV frame, where "together" may refer to a time difference smaller than a threshold, such as less than 1ms, 5ms, 100ms, or the like. In some cases, the threshold may depend on the Frame Per Second (FPS) of the stream. For example, in a stream of 30 FPS, the threshold may be about 33.33 ms or less (e.g., 1ms, 5ms, 30ms, 33ms, 33.33ms, 35ms, 40ms). In situations where the two frames are received together, the chroma frame may potentially be used to enhance the color resolution of the area affected by both frames. In current technologies the YUV frame may be rendered first, replacing the current view to provide an initial image, and then the chroma frame data may be merged into the existing view as the protocol decrees, to refine and improve the color information of the rendered area, thus possibly delaying rendering of the stream. However, in accordance with the current solution, instead of delaying the stream rendering, further received YUV frames are rendered, and the chroma frame, once ready for rendering, is used to enhance the latest view that was rendered before the chroma frame has finished processing.

[0039] Another technical solution of the disclosure relates to sending an acknowledgment (ACK) associated with a chroma frame to the transmitting server before the chroma frame is rendered, for example once the chroma frame is received. This may be done for each chroma frame in the video stream.

[0040] Sending an early ACK may enable the server to transmit the next chroma frame sooner, potentially before the first chroma frame is fully processed and rendered. This approach may help reduce delays in the stream transmission and improve the overall presentation quality. By decoupling the ACK from the rendering process, the system can maintain a more continuous flow of data, which may be particularly beneficial in scenarioswith limited processing resources, high-latency networks, or when usage of GPU is limited and adds latency. Additionally, this method may allow for more efficient parallel processing of multiple chroma frames, as subsequent frames can be received and queued while earlier ones are still being rendered.

[0041] One technical effect of the disclosure is a solution to the technical challenge of rendering video streams in environments where full access to chroma frame data may be restricted, such as in a browser. This solution can be implemented using code executable in JavaScript within a browser-based execution environment. By addressing limitations on GPU access, the method enables efficient video decoding and rendering even when hardware acceleration is unavailable.

[0042] Another technical effect of the disclosure is the rendering of an RDP stream within a browser, in an efficient manner and with minimal delays. The YUV frames are rendered as soon as they are received, and since chroma frames are typically transmitted when the frame is rather stable, there is no significant degradation of the video quality in rendering the chroma frame at a later time, without having to delay the rendering of YUV frames received while the chroma frames are being processed. The disclsoeds subjcet matter allows continuous rendering of YUV frames while chroma frames are being processed

[0043] For the avoidance of doubt, it is clarified that the subject matter is not limited to such an environment, a specific codec, a protocol, or the like.

[0044] Rendering efficiency may be further enhanced by reducing the areas to be rendered based on chroma frames. This can be accomplished by subtracting areas affected by YUV frames received after a given chroma frame, avoiding unnecessary "backdating" of these areas. The method dynamically adjusts the rendering process to account for the most up-to- date frame information.

[0045] Additional efficiency may be gained by sending acknowledgments to the video stream sender immediately after receiving each chroma frame, rather than waiting until after rendering. This approach can help minimize delays in transmitting subsequent chroma frames, potentially improving overall stream performance.

[0046] While this method may result in some frames being rendered out of their original transmission order, the disclosed approach aims to maintain video quality. By intelligently managing the rendering sequence and focusing on the most current frame data, the method balances efficiency with visual fidelity in video stream processing.

[0047] The disclosed subject matter may provide for one or more technical improvements over any pre-existing technique and any technique that has previously become routine or conventional in the art. Additional technical problem, solution and effects may be apparent to a person of ordinary skill in the art in view of the present disclosure.

[0048] Referring now to Fig. 1, showing a flowchart of steps taken in an exemplary scenario in which an RDP stream is rendered by a browser, in accordance with some exemplary embodiments of the disclosure.

[0049] The flowchart illustrates the interaction between two main components: a codec 100, which may be executed by a GPU, and a chroma analyzer 104, which may be executed by a CPU. In some implementations, the chroma analyzer 104 may be a separate thread executed by the CPU.

[0050] The process begins at step 108, where a YUV frame is received by the GPU. At step 112, this YUV frame is rendered by the GPU.

[0051] This sequence of receiving and rendering YUV frames is repeated in steps 116 and 120, and again in steps 124 and 128, demonstrating the continuous processing of incoming video data.

[0052] At step 132, the system receives both a YUV frame and a chroma frame that affect the same area of the display at substantially the same timing. In some cases, the frames are indicated to be within the stream at the same time, for example, by having an identical timestamp, having an identical order within the stream or the like. In some cases, the frames may have similar timestamp, such as having a time different therebetween that is lesser than a threshold. In some cases, these frames may be received simultaneously or within a short time interval, for example within a few milliseconds of each other.

[0053] Step 134 involves the browser copying the chroma frame from the GPU memory to the CPU memory. This transfer allows the chroma analyzer 104 to process the frame.

[0054] At step 136, the YUV frame that was received along with the chroma frame is rendered. It's worth noting that in some implementations, the order of steps 134 and 136 may be reversed.

[0055]

[0056] The process continues at step 140, where another YUV frame is received. This frame affects an area referred to as Rect 1, which may comprise one or more rectangles of the display. Step 144 involves rendering this YUV frame that affects Rect 1.

[0057] Similarly, at step 148, the system receives yet another YUV frame affecting an area called Rect 2, which may also comprise one or more rectangles. This frame is then rendered at step 152.

[0058] Step 154 marks the receipt of the processing results for the chroma frame. At step 156, the system renders the chroma frame. However, this rendering is applied only to the area affected by the chroma frame, excluding Rect 1 and Rect 2, which were modified after the chroma frame was initially received. This exclusion process may involve subtracting Rect1 and Rect 2 from the area affected by the chroma frame. In some cases, Rect 1 and / or Rect2 may not be affected by the chroma frame at all, and subtraction may not change the affected area. In other cases, subtracting the rectangles reduces the area being rendered.

[0059] The process then continues at step 160 with the receipt of another YUV frame, which is rendered at step 164. This cycle of receiving and rendering YUV frames may continue until another chroma frame is received, at which point the process for handling chroma frames would be repeated.

[0060] Referring now to Fig. 2, showing a flowchart of steps taken in an exemplary scenario in which an RDP stream comprising two chroma frames is rendered by a browser, in accordance with some exemplary embodiments of the disclosure.

[0061] The scenario in Fig. 2 involves concurrent processing of two chroma frames. In addition to codec 100 and chroma analyzer 1 (104), which are analogous to codec 100 and chroma analyzer 104 of Fig. 1, another analyzer such as chroma analyzer 2 (200) may be executed, for example as another thread spawned by the CPU.

[0062] At step 204, a YUV frame and a chroma frame may be received, similar to step 132 of Fig. 1.

[0063] At step 208, the chroma frame may be copied from the GPU memory to the CPU memory, to be processed by chroma analyzer 1 (104).

[0064] Additionally, at step 212, the YUV frame may be rendered. In some implementations, steps 208 and 212 may be executed in a reverse order.

[0065] At step 216 a YUV frame may be received, which affects one or more rectangles referred to as Rect 1. At step 220, this YUV frame may be rendered.

[0066] At step 224 a YUV frame and a chroma frame 2, affecting one or more rectangles termed as Rect 2 may be received as in step 204 above. At step 228 the chroma frame may be copied from the GPU memory to the CPU memory and processed, for example by another thread spawned and executed by the CPU, as in step 208 above. At step 232 the YUV frame received at step 224 may be rendered.

[0067] At step 236, a YUV frame affecting one or more rectangles termed Rect 3 may be received, and at step 240 this YUV frame may be rendered.

[0068] At step 244, the processing results of the first chroma frame received at step 204 may be received from the CPU.

[0069] At step 248, the area affected by the first chroma frame may be rendered, excluding the areas affected by the YUV frames received since the chroma frame was received at step 204, i.e., Rect 1, Rect 2 and Rect 3. In some aspects, Rect 1, Rect 2 and Rect 3 are subtracted from the area affected by the chroma frame received at step 204, as these areas may have been updated after the chroma frame was received.

[0070] At step 252, another YUV frame may be received, which affects one or more rectangles referred to as Rect 4, and at step 256 this YUV frame may be rendered.

[0071] At step 260, the processing results of the second chroma frame received at step 224 may be received from the CPU.

[0072] At step 264, the area affected by the second chroma frame may be rendered, excluding the areas affected by the YUV frames received since the chroma frame was received at step 224, i.e., Rect 3 and Rect 4. In some implementations, Rect 3 and Rect 4 are subtracted from the area affected by the chroma frame received at step 224, as these areas may have been updated after the chroma frame was received.

[0073] In some aspects, for each chroma frame, a registry of the areas affected by frames received between the time the chroma frame was received and the time the chroma frame is rendered may be maintained. Rendering the chroma frame may exclude these areas, as they may have been affected by later received frames.

[0074] Referring now to Fig. 3, showing a flowchart of steps taken in an exemplary scenario in which an RDP stream comprising two chroma frames is rendered by a browser, in accordance with some exemplary embodiments of the disclosure.

[0075] Codec 100, chroma analyzer 1 (104) and chroma analyzer 2 (200) may be configured similarly to the components shown in Fig. 2.

[0076] Steps 204, 208, 212, 216, 220, 224, 228, 232, 236 and 240 may be performed in a manner similar to that described for Fig. 2.

[0077] In contrast to the scenario depicted in Fig. 2, at step 344 the processing results of the second chroma frame may be received before the results of the first chroma frame. The only YUV frame received between receiving the second chroma frame and receiving its processing results may be the YUV frame received at step 236, which affects Rect 3.

[0078] At step 348, the second chroma frame may be rendered, excluding the areas overlapping with Rect 3.

[0079] At step 252, another YUV frame may be received which affects Rect 4, and may be rendered at step 256.

[0080] Then at step 360 the results of processing the first chroma frame may be received, and the first chroma frame may be rendered, excluding the areas affected by all rectangles affected by YUV frames received after the first chroma frame was received.

[0081] Thus, at step 364, the first chroma frame may be rendered, excluding Rect 1, Rect 2, Rect 3 and Rect 4.

[0082] In some aspects, under various combinations of chroma frames and their results, each frame may be rendered according to the area it affects, excluding areas affected by YUV frames received since the chroma frame is received, and regardless of other chroma frames.

[0083] The following code listings show exemplary pseudo-code for subtracting a collection of one or more rectangles from another collection of one or more rectangles, as may be done for example in rendering the first chroma frame at step 364.

[0084] Listing 1 below shows an exemplary pseudo-code for defining a rectangle by its left and right values (for example its boundaries on the X axis) and its top and bottom values (for example its boundaries on the Y axis): type Rect { left: integer;top: integer; right: integer; / / inclusive width = right - left + 1 bottom: integer; / / inclusive height = bottom - top + 1}Listing 1

[0085] Listing 2 below shows an exemplary pseudo-code for calculating whether a given point is inside a given rectangle: function isPoint!nRect(rect: Rect, x: integer, y: integer): boolean { if(x < rect. left) { return false;} if(x > rect. right) { return false;} if(y < rect. top) { return false;} if (y > rect.bottom) { return false;} return true;}Listing 2

[0086] Listing 3 below shows an exemplary pseudo-code for calculating how many of rectangle two's corners are inside rectangle 1, and returning a mask representing the same: let A_POINT = 0x01; let B_POINT = 0x02; let C_POINT = 0x04; let D_POINT = 0x08; function getPointMaskAndCount(one: Rect, two: Rect): [integer, integer] { let pointsCount = 0; let pointMask = 0;}Listing 3

[0087] Listing 4 below shows an exemplary pseudo-code for subtracting from one rectangle (rectangle one) another rectangle (rectangle two). It is appreciated that if there is no overlap between the rectangles, the code returns rectangle 1, and if rectangle two is contained fully or partially in Rectangle 1, the code returns a list of portions of Rectangle 1 not contained in Rectangle 2: function rectSubtract(one: Rect, two: Rect): Rect]] { let left 1 = one. left; let topi = one. top; let right 1 = one. right; let bottoml = one. bottom; let left2 = two. left; let top2 = two. top; let right2 = two. right; let bottom2 = two.bottom; / / how many corners of rectangle two are contained in rectangle one let [pointsCount, pointMask] = getPointMaskAndCount(one, two); / / how many corners of rectangle one are contained in rectangle two let [reverseCount, reverseMask] = getPointMaskAndCount(two, one); / / iterating through the different combinations if (points Count = = = 0) { if (reverseCount = = = 2) { / / rectangle two covers one side of rectangle one if ((reverseMask & A_POINT) && (reverseMask & B_POINT)) { return [Rect(leftl, bottom2, rightl, bottoml ), ];} else if ((reverseMask & B_POINT) && (reverseMask & C_POINT)) { return [Rect(leftl, topi, left2, bottoml ),];} else if ((reverseMask & C_POINT) && (reverseMask & D_POINT)) { return [Rect(leftl, topi, rightl, top2),];} else if ((reverseMask & D_POINT) && (reverseMask & A_POINT)) { return [Rect(right2, topi, rightl, bottoml),];}} else if (reverseCount = = = 4) { return []; / / rectangle two contains rectangle one, nothing to return}} else if (pointsCount = = = !) { / / rectangle two covers one corner of rectangle 1 if(pointMask & A_POINT) { return [Rect(leftl, topi, left2, bottoml ),Rect(left2, topi, rightl, top2),];} else if (pointMask & B_P01NT) { return [Rect(leftl, topi, right2, top2),Rect(right2, topi, rightl, bottoml),];} else if (pointMask & C_P01NT) { return [Rect(leftl, bottom2, right2, bottoml ),Rect(right2, topi, rightl, bottoml),];} else if (pointMask & D_P01NT) { return [Rect(leftl, topi, left2, bottoml ),Rect(left2, bottom2, rightl, botoml ),];}} else if (pointsCount = = = 2) { / / rectangle one covers a side of rectangle 2 if ((pointMask & A_POINT) && (pointMask & B_POINT)) { return [Rect(leftl, topi, rightl, top2),Rect(leftl, top2, left2, bottoml ),Rect(right2, top2, rightl, bottoml),];} else if ((pointMask & B_P01NT) && (pointMask & C_P01NT)) { return [Rect(leftl, topi, right2, top2),Rect(right2, topi, rightl, bottoml),Rect(leftl, bottom2, right2, bottoml ),];} else if ((pointMask & C_P01NT) && (pointMask & D_P01NT)) { return [Rect(leftl, topi, left2, bottom2),Rect(right2, topi, rightl, bottom2),Rect(leftl, bottom2, rightl, bottoml ),];} else if ((pointMask & D_P01NT) && (pointMask & A_P01NT)) { return [Rect(leftl, topi, left2, bottoml ),Rect(left2, topi, rightl, top2),Rect(left2, bottom2, rightl, bottoml ),];}} else if (pointsCount = = = 4) { / / rectangle two is fully contained in rectangle one, / / and rectangle two does not cover any corner of rectanlgle one return [Rect(leftl, topi, left2, bottoml ),Rect(left2, topi, rightl, top2),Rect(left2, bottom2, rightl, bottoml ),Rect(right2, top2, rightl, bottom2),];} / / For two valid rects this should not happenassertFail]" Invalid rect overlap");}Listing 4

[0088] As can be appreciated, the subtraction has an 0(1) computational time and space complexity. In some exemplary embodiments, one may decide on the new set of Rects to create by iterating over all the possible overlaps categorized by the second Rect's points that are inside the first Rect. For example, if there is a single point of two in one, we know that we should return two Rects that don't overlap with two and cover the rest of one.

[0089] Listing 5 below is an exemplary pseudo-code describing how to subtract a set of Rects from another is provided: / / result = setOne - setTwo function rectSetSubtract(setOne: Rect]], setTwo: Rect]]) { let isetl = setOne. shallowCopy(); let iset2 = new Rect]]; for (let twolndex = 0; twolndex < setTwodength; ++twolndex) { let two = setTwo[twoIndex] ; for (let onelndex = 0; onelndex < isetl ength; ++onelndex) { let one = isetl [onelndex] ; let result = rectSubtract( one, two ); for (let rectlndex = 0; rectlndex < resultdength; ++ ectlndex) { iset2.push( result [rectlndex] );}} / / truncate isetl isetl. truncate]); / / copy iset2 to isetl for (let rectlndex = 0; rectlndex < iset2.length; ++ rectlndex) [ isetl. push] iset2 [rectlndex] );} / / truncate iset2 iset2. truncate]);} return isetl;}Listing 5

[0090] The time complexity of this pseudo-code is 0(NM), where N and M represent the sizes of the rectangle sets. The space complexity is also 0(NM). This is because the first rectangle set is updated iteratively using two temporary sets. Additionally, the resulting setmay increase by up to 4 rectangles on each iteration of the second rectangle. In some implementations, the algorithm may involve iterating through different combinations of rectangle overlaps, categorized by how many points of the second rectangle are inside the first rectangle. For example, if there is a single point of overlap, the algorithm may return two non-overlapping rectangles that cover the remaining area of the first rectangle. This approach can potentially optimize the subtraction process for certain geometric configurations.

[0091] Referring now to Fig. 4, showing a flowchart of steps in a method for rendering a video stream, in accordance with some exemplary embodiments of the disclosure.

[0092] At step 400, a first processor, such as a GPU, may receive a video stream comprising YUV frames and chroma frames in a first order. The YUV frames may outnumber the chroma frames in some implementations.

[0093] Each of the YUV frames and the chroma frames may depict a display area or a respective portion thereof. The chroma frames may comprise a first chroma frame affecting a first part of the display area. In some cases, one or more chroma frames may be received together with a YUV frame, e.g., at a time difference not exceeding a threshold or for example within a short time interval.

[0094] At step 404, which may occur at any time before the first chroma frame is rendered, an acknowledgment (ACK) may be sent to the party transmitting the video stream. This may help avoid delay in the transmission of subsequent chroma frames.

[0095] At step 408, information from the first chroma frame may be copied from a storage space associated with the first processor and provided to a second processor, such as a CPU, for decoding. In some implementations, if multiple chroma frames are processed in parallel, for example when a second chroma frame is received before a first one has finished processing, each chroma frame may be processed by a separate processing thread.

[0096] The method may be useful for rendering RDP streams in a browser environment, where access to GPU resources and chroma frame contents may be limited.

[0097] It is appreciated that processing the chroma frames may comprise any one or more of the following: decoding, color space conversion, data restructuring, texture preparation, synchronization calculations, buffer management, optimization, error checking and validation, or the like.

[0098] At step 412, a YUV frame may be received by the first processor. The YUV frame may affect a respective second part of the display area. The second part may comprise one or more rectangles. In some implementations, these rectangles may be aligned with axes of the display area.

[0099] An accumulation of the respective second parts from multiple YUV frames may form a collection of second parts, which may be represented as a collection of rectangles.

[0100] At step 416, the YUV frame may be rendered. Steps 412 and 416 may be repeated for additional received YUV frames.

[0101] The system may maintain a collection of second parts affected by the sequence of YUV frames received since the chroma frame was received.

[0102] At step 420, the first processor may receive processing results for the chroma frame, including decoded information embedded within the chroma frame.

[0103] At step 424, the part of the display area affected by the chroma frame may be rendered in accordance with the processing results of the chroma frame. However, this rendering may exclude the collection of second parts, such that areas updated after the chroma frame was received are not rendered. This exclusion process may involve subtracting one set of rectangles from another.

[0104] This method may provide for out-of-order rendering of the video stream. Chroma frames may not be rendered immediately upon receipt, but rather when their processing results become available and after other YUV frames which were subsequently received were rendered.

[0105] In some implementations, when a second chroma frame is received after the first chroma frame, the system may render each chroma frame while excluding the areas affected by YUV frames received since that respective chroma frame was received. This approach may be applied regardless of whether the processing results of the first chroma frame or the second chroma frame arrive first. By selectively rendering only the non-overlapping areas, the system may help maintain visual consistency and potentially improve rendering efficiency.

[0106] The rendering process may involve tracking the affected areas for each chroma frame separately. When rendering a particular chroma frame, the system may subtract the collection of areas affected by subsequently received YUV frames from the area to berendered for that chroma frame. This subtraction process may help ensure that the most up- to-date frame information is displayed, while avoiding unnecessary re -rendering of areas that have already been updated by more recent YUV frames.

[0107] Referring now to Fig. 5, showing a representation of a YUV444 frame in three planes, where Y444, U444, and V444 are the Y, U, and V planes of a source YUV444 frame, according to the RDP spec. The resolution of these planes may be specified by width w and height h. In some aspects, the Y444 plane represents the luminance information at full resolution, while the U444 and V444 planes contain the chrominance information, also at full resolution. This YUV444 representation provides a complete color description for each pixel in the frame. The width w and height h parameters define the dimensions of each plane, which in some cases may be identical across all three planes. The structure of YUV444 frames may involve three separate planes for Y, U, and V components. This structure may form a basis for various subsampling and encoding techniques that may be used in video compression and transmission protocols like RDP.

[0108] Referring now also to Fig. 6, showing a representation of packing of the YUV444 frame represented in Fig. 5 into two YUV420 frames, comprising a YUV frame (main view) and a chroma frame (auxiliary view), which together represent the full frame. The areas marked as Bl to B9 make up the Y, U, and V planes of the two YUV420 macroblocks representing the main and auxiliary views. These areas may be related to Y444, U444, and V444 as follows:Main View:Area Bl: Y420 (x, y) = Y444 (x, y) where (x, y) is within the range of [0, w-1] x [0, h-1] Area B2: U420 (x, y) = U444 (2x, 2y) where (x, y) is within [0, w / 2-1] x [0, h / 2-1] Area B3: Y420 (x, y) = V444 (2x, 2y) where (x, y) is within [0, w / 2-1] x [0, h / 2-1]Auxiliary View:Area B4: Y420 (x, y) = U444 (2x+l, y) where (x, y) is within [0, w / 2-1] x [0, h-1] Area B5: Y420 (w / 2+x, y) = V444 (2x+l, y) where (x, y) is within [0, w / 2-1] x [0, h-1] Area B6: U420 (x, y) = U444 (4x, 2y+l) where (x, y) is within [0, w / 4-1] x [0, h / 2-1] Area B7: U420 (w / 4+x, y) = U444 (4x, 2y+l) where (x, y) is within [0, w / 4-1] x [0, h / 2-1] Area B8: V420 (x, y) = V444 (4x+2, 2y+l) where (x, y) is within [0, w / 4-1] x [0, h / 2-1] Area B9: V420 (w / 4+x, y) = V444 (4x+2, 2y+l) where (x, y) is within [0, w / 4-1] x [0, h / 2-

[0109] In some exemplary embodiments, the following methods may be implemented and used for rendering, once the rectangles to be updated are determined:

[0110] pushRect(rect): pushes 4 vertices into WebGL buffers containing only position information in [0,W)x[0,H) space where W, H are the video coded width and height respectively.

[0111] drawLumaO : Treats the videoFrame currently set in the frame texture as a luma RDP AVC444 frame and draws it onto a luma texture in a passthrough manner. The luma texture is in a YUV color space.

[0112] drawChroma(): Treats the videoFrame currently set in the frame texture as chroma RDP AVC444 frame and merges it onto a chroma texture using the current luma texture and the algorithm in the RDP spec adapted into a WebGL shader. The chroma texture is in a YUV color space.

[0113] draw(): If drawChroma() was called just before this call - the chroma texture is used, otherwise, the luma frame is used. The texture is drawn only in the current rects set with the pushRect() call. The color conversion YUV— >RGB is applied.

[0114] setYUVBuffer(arrayBuffer): sets a GL_LUMINANCE 2D WebGL texture. The shader responsible for merging luma and chroma frames samples the ID texture correctly.In some exemplary embodiments, VideoFrame. copyTo() copies the frame YUV data in a packed fashion. This means the frame's width is the row pitch, which has width * height Y bytes, followed by 2(width / 2 * height / 2) bytes representing interlaced U and V bytes. So for a 4x4 image, we have the following byte sequence:YYYYYYYYYYYYYYYYUVUVUVUV containing 16 bytes of Y component, and 8 bytes of 4 interlaced U and V components.

[0115] setYUVBuffer(arrayBuffer) is called after the VideoFrame. copyTo() operation is completed. Before calling VideoFrame. copyToQ the frame's rects are saved, and the Rect- Subtract operation is applied to them until the operation completes. Then we call pushRect(rect) with the subtracted rects and setYUVBuffer(arrayBuffer), followed by drawChroma() and draw().

[0116] The frame texture may be created from a VideoFrame object. A Video Frame may be imported into two WebGL textures. Both imported textures may be imported into theframe WebGL texture. The luminosity texture may be sampled into the R component and the chroma texture is sampled into the G and B components.

[0117] Listing 6 below shows shader pseudo-code for drawing a YUV texture of the VideoFrame object.Aversion 100 precision highp float;#extension {RequiredExtension} : require uniform {EumaTextureSamplerType} ySampler; uniform {ChromaTextureSamplerType} uvSampler; varying vec2 v_texcoord;vec3 sampleFrame(vec2 st){ vec3 result = vec3(0.0); result.r = {LumaSamplerFunction}(ySampler, st).r; result. gb = {ChromaTextureSamplerType}(uvSampler, st).rg; return result;} void main( void ) { gl_FragColor = vec4(sampleFrame(v_texcoord), 1.0);}Listing 6

[0118] The chroma texture may be created from the frame and the luma textures. The frame texture holds YUV Chroma information (decoded from a Chroma RDP frame). The luma texture holds the previously decoded Luma RDP frame in the stream. Listing 7 below is an example of a shader pseudo code:Aversion 100 precision highp floa t ; uniform sampler2D frameSampler ; uniform sampler2D lumaSampler ; uniform floa t framewidth ; uniform floa t frameHeigh t ; uniform vec4 texTransform; varying vec2 v texcoord;void main (void) { vec2 st = floor(v texcoord) ;vec3 lumaColor = texture2D (lumaSampler , trans formTexCoord (st) ) . rgb ; float cy = lumaColor. r; float cu = c b4 + c b6 + c b8 ; float cv = c b5 + c b7 + c b9; vec3 chromaColor = vec3(cy, cu , cv) ; float a = sign (b4 factor + b5 factor + b6 factor + b7 factor + b8 factor + b9 factor) ; gl FragColor = vec4 (a * chromaColor + (1.0 - a) * lumaColor, 1.0) ;}Listing 7

[0119] For each area, a factor of zero is created if the current pixel does not belong to it. See each bx_factor variable. It is also sampled according to the spec's formulas, considering that the Chroma texture frame is subsampled in the U (G) and V (B) components. Then the pixel is combined by taking the luminosity from the luma texture, and U component from the relevant U area (either B4, B6, or B8), and the Y component from the relevant Y area (either B5, B7, or B9). The area samples are summed, and this is applicable as the irrelevant areas are already zeroed out. The currently calculated chromaColor or the lumaColor sample from the luma texture may be outputted. If none of the area factors were one, the lumaColor is outputted. Otherwise, the chromaColor is outputted.

[0120] It is noted that in some cases, the VideoFrame. copyTo() operation may be synchronous and block the calling thread. In such cases, the operation is invoked in a separate thread (e.g., using Web Workers API).

[0121] The present disclosed subject matter may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosed subject matter.

[0122] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0123] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to anexternal computer or external storage device via a network, for example, the Internet, a Local Area Network (LAN), and a Wide Area Network (WAN). The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0124] Computer readable program instructions for carrying out operations of the present disclosed subject matter may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language, aspect oriented programming language, procedural programming language, or the like. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. The remote computer may be connected to the user's computer through any type of network, including a LAN, a WAN, or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosed subject matter.

[0125] Aspects of the present disclosed subject matter are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosed subject matter. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0126] The computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. The computer readable program instructions may be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0127] The computer readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0128] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosed subject matter. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0129] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosed subject matter. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that theterms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0130] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present disclosed subject matter has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosed subject matter in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosed subject matter. The embodiment was chosen and described in order to best explain the principles of the disclosed subject matter and the practical application, and to enable others of ordinary skill in the art to understand the disclosed subject matter for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

CLAIMSWhat is claimed is:

1. A method for rendering, the method comprising: receiving by a first processor a video stream comprising a set having a first order of YUV frames and chroma frames, wherein the YUV frames outnumber the chroma frames, and wherein each of the YUV frames and the chroma frames depicts a display area or a respective portion thereof, the chroma frames comprising a first chroma frame affecting a first part of the display area; copying information of the first chroma frame and providing the information to a second processor for decoding; receiving a sequence comprising at least one YUV frame, each YUV frame of the at least one YUV frame affecting a respective second part of the display area, wherein accumulation of the respective second part of each of the at least one YUV frame forms a collection of second parts; rendering each YUV frame of the at least one YUV frame, while maintaining the collection of second parts affected by the at least one YUV frame received since the first chroma frame was received; receiving processing results of the first chroma frame, the processing results comprising a decoding of the information; and rendering the first part of the display area in accordance with the processing results of the first chroma frame, said rendering excluding the collection of second parts, whereby rendering of the video stream is performed in a second order, the second order different from the first order.

2. The method of Claim 1, wherein a collection of second parts is maintained for each received chroma frame, and wherein the collection of second parts is used when rendering the chroma frame.

3. The method of Claim 1, wherein when a chroma frame and a YUV frame are received at times differing in at most a predetermined period of time, the YUV frame is rendered, and the respective second part of the YUV frame is not added to the collection of second parts.

4. The method of Claim 1, wherein each chroma frame is processed by the second processor using a separate thread.

5. The method of Claim 1, wherein a first chroma frame and a second chroma frame are processed in parallel.

6. The method of Claim 1, wherein the video stream is a Remote Desktop Protocol (RDP) stream, and wherein the method is utilized for the rendering the RDP stream over a browser.

7. The method of Claim 6 wherein the browser has limited access to contents of the chroma frames, thereby preventing the browser from performing the second processing.

8. The method of Claim 1, wherein processing the first chroma frame comprises at least one operation from the group consisting of: color space conversion, data restructuring, texture preparation, synchronization calculations, buffer management, optimization, error checking and validation.

9. The method of Claim 1, wherein the first processor is a graphic processing unit (GPU) and the second processor is a central processing unit (CPU).

10. The method of Claim 1, further comprising: receiving by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received; receiving by the first processor after the processing results of the first frame are received, second processing results of the second chroma frame; rendering each YUV frame of at least one YUV frame received between when the first processing results were received and the second processing results were received; and rendering the area in accordance with the processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received.

11. The method of Claim 1, further comprising: receiving by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received;rendering each YUV frame of at least one YUV frame received between when the second chroma frame was received and when the second processing results were received; receiving by the first processor, second processing results of the second chroma frame, before the processing results of the first frame are received; and rendering the area in accordance with the second processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received.

12. The method of Claim 1, wherein exclusion of the collection of second parts comprises subtracting the collection of second parts from an area affected by the first chroma frame.

13. The method of Claim 1, further comprising sending an acknowledgement signal after the first chroma frame is received and before the processing results are received, thereby eliminating slowdown by a sender of the first chroma frame, and enabling the sender to send a second chroma frame.

14. The method of Claim 1, wherein a proportion between a number of the chroma frames and a number of the YUV frames is lower when a frame rate is higher.

15. The method of Claim 1, wherein a ratio of a number of the chroma frames and a number of the YUV frames is between 10% and 35%.

16. The method of Claim 1, wherein rendering is performed using a shader that combines luminosity from a luma texture and chrominance components from the processed Chroma frame.

17. A system having a processor, the processor being configured to perform the steps of: receiving by a first processor a video stream comprising a set having a first order of YUV frames and chroma frames, wherein the YUV frames outnumber the chroma frames, and wherein each of the YUV frames and the chroma frames depicts a display area or a respective portion thereof, the chroma frames comprising a first chroma frame affecting a first part of the display area; copying information of the first chroma frame and providing the information to a second processor for decoding; receiving a sequence comprising at least one YUV frame, each YUV frame of the at least one YUV frame affecting a respective second part of the displayarea, wherein accumulation of the respective second part of each of the at least one YUV frame forms a collection of second parts; rendering each YUV frame of the at least one YUV frame, while maintaining the collection of second parts affected by the at least one YUV frame received since the first chroma frame was received; receiving processing results of the first chroma frame, the processing results comprising a decoding of the information; and rendering the first part of the display area in accordance with the processing results of the first chroma frame, said rendering excluding the collection of second parts, whereby rendering of the video stream is performed in a second order, the second order different from the first order.

18. The system of Claim 17, wherein the collection of second parts is maintained for each received chroma frame from the chroma frames, and wherein the collection of second parts is used when rendering the chroma frame.

19. The system of Claim 17, wherein the processor is further configured to: receive by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received; receive by the first processor after the processing results of the first frame are received, second processing results of the second chroma frame; render each YUV frame of at least one YUV frame received between when the first processing results were received and the second processing results were received; and render the area in accordance with the processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received.

20. The system of Claim 17, wherein the processor is further configured to: receive by the first processor, a second chroma frame after the first chroma frame was received and before the processing results of the first chroma frame are received;render each YUV frame of at least one YUV frame received between when the second chroma frame was received and when the second processing results were received; receive by the first processor, second processing results of the second chroma frame, before the processing results of the first frame are received; and render the area in accordance with the second processing results of the second chroma frame, said rendering excluding second parts affected by YUV frames received after the second chroma frame was received.

Images