Graphics service provider and method for data stream transmission of image-generated content via a remote graphics processing service

Abstract

Graphic design service providers (120), with: a single-frame capture unit (134) which is designed to capture single frames from image-generated content at a single-frame rate; an encoder (136) trained to encode captured individual frames at the frame rate; and to cause a processor (124) configured to transmit encoded frames when the image-generated content is at least partially changed, and to cause a frame-skip message to be transmitted, wherein the frame-skip message is configured to cause the frame-acquisition unit (134) to refrain from acquisition and the encoder (136) to refrain from encoding when the image-generated content is unchanged, wherein the frame-skip message is generated and stored in the memory before the generation of the image-generated content.

Description

TECHNICAL AREA

[0001] This application is generally aimed at computer graphics processing and in particular at performance-efficient, memory-efficient and time-efficient remote image generation, single-image capture and encoding. BACKGROUND

[0002] The use of personal computing devices was originally concentrated in the field of research, providing powerful tools on the desktop calculators of researchers, engineers, analysts, and clerical staff. This use has evolved from mere number and word processing to highly programmable, interactive tools suitable for production-level applications and real-time graphics generation, enabling incredibly detailed computer-aided design, development, and visualization. More recently, personal computing technology has taken on a key role in media and gaming output, a development fueled by the evolution of mobile computing devices. The personal calculator is no longer confined to desktop or even mobile computers of the real world.Robust networks and the miniaturization of computing power have enabled mobile devices, such as cordless phones and tablet computers, which claim a large share of the personal computer market.

[0003] Mobile computing has changed the conventional notions of information access and media distribution. Network-enabled devices are the new norm, connecting a multitude of devices across a variety of networks. This has led to a wealth of conventional or "common" content, as well as unconventional, amateurish, or homemade content.If this is taken further, not only will this content be available on virtually any mobile device, but mobile devices will not only be used as conventional output devices, but will also act as media distribution tools, providing access to and forwarding, or "sending out," a vast amount of content to one or more display devices, including televisions, computer monitors, projectors, or any device capable of receiving, decoding, and displaying content sent as a data stream. Although mobile devices are typically viewed as clients, mobile devices, or indeed virtually any computing device, can act as "media service providers."

[0004] In a typical service provider-client telegraphics processing setup, the graphics content is stored, retrieved, and generated on a service provider or server. Frames of content generated as images or image-generated content are captured and encoded, generally at a rate specified by a monitoring device or simply part of a configuration. The captured and encoded frames are then converted into packets and transmitted over a network to a client as a video stream (often with audio). The client simply decodes the video stream and displays the content. Such a "thin client" application can easily be ported to a variety of platforms.

[0005] With the continued development of mobile computing technology, increasingly focused on content access and distribution, the role of mobile devices will expand further. Typical boundaries between client and service provider will continue to blur, and more and more people will use their mobile devices as both client and service provider, depending on the content they are interested in. In this field, US 8429699 B2 and US 2010 / 0027663 A1 are known. US 2010 / 0290528 A1 describes a video processing unit with a pre-processor and a processor. US 2012 / 0057636 A1 describes a method for processing video data in which blocks in the image with specific pixel values ​​are removed before an encoding step. OVERVIEW

[0006] One aspect provides a graphics service provider according to claim 1. In one embodiment, the service provider comprises: (1) among other things, a frame capture unit configured to capture frames with image-generated or image-generated content at a frame rate, (2) an encoder configured to encode received frames at the frame rate, and (3) a processor configured to cause encoded frames to be transmitted when the image-generated content is at least partially changed, and to transmit a frame-skip message, wherein the frame-skip message is configured to cause the frame capture unit to refrain from capturing and to cause the encoder to refrain from encoding when the image-generated content is changed.

[0007] A further aspect provides a graphics service provider for the data stream transmission of image-generated content of a video stream according to claim 8. In one embodiment, the service provider comprises: (1) a memory configured to store a pre-calculated frame-skip message indicating to a client to reuse a previously transmitted frame of the video stream; (2) a graphics generation unit configured to generate the image-generated content and, if the image-generated content is at least partially new, to indicate this by means of a control bit; (3) a central processing unit (CPU) configured to use the control bit to instruct a frame capture unit and an encoder to cease operation on the image-generated content and to cause the pre-calculated frame-skip message to be transmitted to the client. SHORT DESCRIPTION

[0008] Reference is now made to the following descriptions in conjunction with the accompanying drawings, in which: Fig. 1 is a block view of an embodiment of a remote graphics processing system with a service provider client; Fig. 2 is a block view of an embodiment of a graphics service provider; and Fig. 3 a flowchart of an embodiment of a method for data stream transmission of image-generated content via a remote graphics processing service provider. DETAILED DESCRIPTION

[0009] As the role of media service provider expands to a growing number of computing devices, limitations emerge that were previously overlooked in conventional service provider-client models. Conventional service providers are centrally located, grid-powered, and typically rely on externally provided network access, cooling, and possibly external storage. While such an arrangement is not immune to power, memory, or latency limitations, these limitations are significantly more pronounced in mobile devices when they assume a service provider role. Mobile devices, including smartphones, tablets, laptops, and others, generally rely to some extent on battery power.Many mobile devices are also limited to built-in storage, or can only be expanded through a certain type of removable media, such as SD flash or microSD cards. Given these limitations, mobile device manufacturers and software developers are cautious about introducing unnecessary processes or memory operations. When a mobile device is viewed as a service provider, even modest improvements in efficiency regarding power consumption, memory bandwidth, and latency can be crucial.

[0010] This recognizes that capture and encoding stages in remote graphics processing can be bypassed if the image generation process is idle and the generated content is not modified. In a graphics service provider, processing is typically divided between a central processing unit (CPU) and a graphics processing unit (GPU). The GPU performs the image generation tasks, including frame capture and, in some cases, encoding. The CPU runs an application, generates scene data and image generation instructions, or can retrieve content from memory and pass it to the GPU for image generation. Furthermore, the CPU processes captured and encoded frames to prepare them for transmission to the client.In some embodiments, the CPU manages the operation of the image generation stage, the acquisition stage, and the encoding stage by enabling and disabling them, by clock frequency scaling, or by other mechanisms. By detecting when the generated content remains unchanged, the acquisition and encoding stages can be disabled or at least slowed down to a rate comparable to the content's update rate. It is further recognized herein that this can be accomplished by maintaining a control bit which, when set, indicates that new content has been generated. The control bit can be maintained by the image generation stage and read by the acquisition and encoding stages, or the control bit can be maintained by the CPU, which manages the operation of the acquisition and encoding stages based on the control bit.

[0011] This recognizes that by bypassing data acquisition and encoding, resource-intensive memory read / write cycles, along with the processing cycles required for encoding, are eliminated. These savings reduce power consumption, memory bandwidth usage, and latency.

[0012] Although the service provider has bypassed unnecessary processes, the client still requires that individual frames be displayed at the specified frame rate. It is recognized here that a frame-skip message can be transmitted instead of a captured and encoded individual frame. The frame-skip message is typically built into any encoding scheme used for data stream transmission. For example, the h.264 standard provides a protocol to instruct the client to reuse the previously received and decoded individual frame. That is, the current individual frame is to be displayed further; the generated content has not changed. It is further recognized here that the frame-skip message can be pre-computed independently of the image generation, capture, and encoding stages. For example, the CPU can pre-compute an h.264 - Pre-calculate the single-frame skip message and can easily transmit it whenever the control bit is active, bypassing the capture and encoding stages.

[0013] It is further recognized herein that the efficiency achieved when image-generated content remains unchanged can also be achieved when the image-generated content is new but well-known. In these cases, capture and encoding can be avoided as long as an efficient encoding pattern can be pre-calculated instead of capturing and encoding a single frame. It is further recognized herein that the efficiency is most valuable when a mobile device is the service provider, but it can be applied equally to conventional, centralized service provider arrangements. The scale of such arrangements could reach a point that would justify such modest improvements in performance and storage efficiency at the individual client level.

[0014] Before introducing various embodiments of the graphics service provider and the method for data stream transmission of image-generated content, a remote service provider-client graphics processing system is described in which the graphics service provider and the method can be integrated or executed.

[0015] Fig. Figure 1 is a block view of an embodiment of a remote service provider-client graphics processing system 100. The system 100 comprises a network 110 over which a service provider 120 and a client 140 communicate with each other. The service provider 120 represents the central collection of content, processing, and image generation resources. The client 140 is a consumer of this content and these resources. In certain embodiments, the service provider 120 is freely scalable and has the capacity to provide this content and these services to many clients simultaneously by advantageously utilizing the parallel and partitioned processing and image generation resources.In addition to any limitations regarding the performance, memory bandwidth, and latency of the service provider 120, the scalability of the service provider 120 is limited to the capacity of the network 110 in such a way that above a certain threshold number of clients, limitations in network bandwidth necessitate that the service decreases on average for all clients.

[0016] The service provider 120 comprises a network interface card (NIC) 123, a central processing unit (CPU) 124, and a GPU 130. When selected in the service provider 120, or in certain embodiments upon request by the client 140, graphics content is retrieved from memory via an application running in the CPU 124. As is typical for graphics applications, such as games, the CPU 124 reserves itself to perform higher-level operations, such as determining the position, movement, and collision of objects in a given scene. From these higher-level operations, the CPU 124 generates image-generating instructions, which, when combined with the scene data, can be executed by the GPU 130. For example, the image-generating instructions and data can define the scene geometry, lighting, shading, surface texture, motion, and camera parameters for the scene.

[0017] The GPU 130 comprises a graphics generation unit 132, a frame capture unit 134, and an encoder 146. The graphics generation unit 132 performs image generation procedures according to the image generation instructions generated by the CPU 124, resulting in a stream of video frames for the scene. These raw video frames are captured by the frame capture unit 134 and encoded by the encoder 136. The encoder 136 formats the raw video stream for transmission, possibly applying a video compression algorithm such as the H.264 standard agreed upon by the International Telecommunication Union for Telecommunication Standardization (ITU-T), or the MPEG-4 Extended Video Encoder (AVC) standard of the International Organization for Standardization / International Electrotechnical Commission (ISO / IEC).Alternatively, the video stream can be encoded into a Windows Media Video (WMV) format, a VP8 format, or another video encoding format.

[0018] CPU 124 prepares the encoded video stream for transmission, which is then forwarded to NIC 122. NIC 122 contains circuitry required for communication over network 110 using a network protocol, such as the Internet, Wi-Fi, or Internet Protocol (IP). NIC 122 provides the physical layer and the foundation for the software layer of the network interface of the service provider 120.

[0019] Client 140 receives the transmitted video stream for display. Client 140 can represent a variety of personal computing devices, including a desktop or mobile personal computer, a tablet computer, a smartphone, or a television. Client 140 contains a NIC 143, a decoder 144, a video generation unit 146, a display 148, and a CPU 150. Similar to NIC 122, NIC 142 contains circuitry required for communication over the network 110 and provides the physical layer and the foundation for the software layer of Client 140's network interface. Client 140 receives the transmitted video stream via NIC 142. The CPU 150 unpacks the received video stream and prepares it for decoding.

[0020] The video stream is then decoded by decoder 144. Decoder 144 should be compatible with encoder 136 in that each uses the same formatting or compression scheme. For example, if encoder 136 uses the ITU-T H.264 standard, decoder 144 should also use it. Decoding can be performed by a client CPU or a client GPU, depending on the specific client physical device. After decoding, all that remains in the video stream are the raw, image-generated frames. These frames are processed by a basic video generation unit 146, just as any other video data stream is. The generated video can then be displayed on the display 148.

[0021] Following the description of a remote service provider-client graphics processing system in which the graphics service provider and the method for data stream transmission of generated content can be used or executed, various embodiments of the graphics service provider and the method are now described.

[0022] Fig. 2 is a single image view of an embodiment of a graphics service provider 200, for example, the service provider 120 from Fig. 1. The graphics service provider 200 comprises the NIC 123, the CPU 124 and the GPU 130, all of which are made up of Fig. 1. Furthermore, the graphics service provider 200 includes a memory 210, a data bus 220, and a control bus 230. The GPU 130 includes the graphics generation unit 132, the frame capture unit 134, and the encoder 136, as in Fig. 1.

[0023] As in the service provider 120 from Fig. The basic function of the graphics service provider 200 encompasses content generation, frame capture, and frame encoding for subsequent transmission to a client. The CPU 124, the graphics generation unit 132, the frame capture unit 134, and the encoder 136 all read from and write to memory 210 via the data bus 220. The data bus 220 is the primary transmission path for moving content through the image generation, capture, and decoding processes. While the data bus 220 can be used to transfer data directly from one processing unit to the next, more commonly, for large amounts of data, data is transferred from one processing unit to memory 210 via the data bus 220. Once the data is in memory, it can be accessed by any processing unit connected to the data bus 220.The CPU 124 manages the operation of the graphics generation unit 132, the single image acquisition unit 134 and the encoder 136 via the control bus 230.

[0024] CPU 124 executes an application that generates or retrieves from memory image-generating instructions and scene data intended for image generation. Typically, scene data is stored in memory 210 and used later during image generation. The graphics generation unit (GPU) 132 accesses the scene data in memory 210 via data bus 220 and executes the image-generating instructions on the scene data to create the generated content. The generated content is then transferred to memory 210 via data bus 220. The GPU 132 generally only operates when the on-screen content has changed. Numerous methods are available to conserve image-generating resources when the content remains unchanged. If the generated content changes, at least partially, a control signal or control bit is set.

[0025] The frame capture unit 134 and the encoder 136 are configured to operate at a frame rate specified by the CPU 124. The frame rate is often the result of a user setting, a quality of service (QoS) process, a negotiation between the service provider 200 and the client, or some other configuration. The frame rate is not only the rate at which individual frames of generated image content are captured and encoded, but also the rate at which they are transmitted, and likely also decoded and displayed. Such an arrangement is sensitive to processing delays and is suboptimal under network conditions. The frame capture unit 134 "captures" image-generated content by periodically copying the image-generated content to a buffer in the memory 210.In certain embodiments, the control bit is set when newly generated content is transmitted to the acquisition stage. Once the acquisition has taken place, the control bit is reset. When the single-frame acquisition unit 134 copies the generated content and the control bit is reset, the single-frame acquisition unit knows that the generated content has remained unchanged since the last acquisition. In the embodiment shown... Fig. 2. The CPU 124 manages the control bit. When the graphics generation unit 132 produces new generated content, the control bit is set. The CPU 124 then uses the control bit to enable and disable the frame capture unit 134 and the encoder 136 via the control bus 230. Alternatively, the CPU 124 can apply clock frequency scaling based on the control bit to control the rate at which the frame capture unit 134 and the encoder 136 operate. Clock frequency scaling allows a clocked device to be sped up or slowed down according to its workload and capacity. In this case, the CPU 124 can reduce the rate at which the frame capture unit 134 operates so that no frames are captured unless the generated content changes at least partially.Bypassing the single-image capture stage reduces power consumption and storage bandwidth by avoiding unnecessary read and write cycles.

[0026] Encoder 136 accesses the buffer in memory 210 via data bus 220 and encodes the stored frame. For example, many video stream implementations use h.264 encoding. The decoded frame is then written to the frame buffer in memory 210 via data bus 220. Similar to frame capture unit 134, the operation of encoder 136 is controlled by CPU 124 via control bus 230. CPU 124 reduces the rate at which encoder 136 operates based on the control bit. When the control bit is set, the generated content has changed at least partially, and new data has been stored in the buffer in memory 210. When the control bit is reset, the generated content remains unchanged, and no new data has been written to the buffer in memory 210.The coding stage is bypassed, thereby reducing power consumption by avoiding unnecessary processing.

[0027] CPU 124 retrieves encoded frames from memory 210 via data bus 220 and prepares them for transmission via NIC 142, or "packs" them for transmission. This preparation typically involves forming data packets from the frame buffer and possibly additional encoding for the transmission protocol. If the capture and encoding stages are bypassed, as indicated by the control bit over control bus 230, no new frames are ready for packaging and transmission. Instead, a frame-skip message is sent to the client. The frame-skip message indicates to the client that no update is necessary and that the client should either maintain the current display or use the previous frame. The client maintains its frame rate regardless of whether the content is modified, generated, captured, or encoded.The frame-skip message is generally built into the encoding scheme, for example, an h.264 frame-skip message. The frame-skip message is a predefined message and can be pre-calculated by CPU 124 before frame generation and stored in memory 210. After calculation, CPU 124 can resend the frame-skip message via NIC 122 as soon as any on-screen content is inactive or unchanged.

[0028] Fig. Figure 3 is a flowchart of an embodiment of a method for transmitting a data stream of image-generated content via a remote graphics processing service provider. The method begins in a start step 310. In an acquisition step 320, it is determined whether the generated content is unchanged. Based on this determination, a control signal is triggered. In certain embodiments, the control signal is a control bit embedded in a control message or that can form its own message. The control signal can also be an independent signal that is distributed to one or more processing devices. In a decision step 330, a decision is made based on the control signal from the acquisition step 320. If the control signal indicates that the generated content has changed, at least partially, then the method proceeds to the acquisition and encoding step 340 and to step 350 for transmitting the single image.If the control signal indicates that the generated content is unchanged, then the procedure proceeds to step 360 for the bypass and to step 370 for the transmission of a frame-skipping message.

[0029] In alternative embodiments, the generated content may be partially modified, but the generated content is well known, and an efficient encoding pattern can be pre-calculated to represent this modification. In this case, in these alternative embodiments, the process would proceed to bypass step 360. Instead of proceeding to the frame-skipping step 370, the pre-calculated frame is transmitted. This reduces the power consumption, memory bandwidth, and latency that normally occur during acquisition and encoding.

[0030] It is the embodiment from Fig.Step 3 continues; if the generated content has been at least partially modified, individual frames of the generated content are captured and encoded in capture and encoding step 340. Capture of an individual frame typically occurs at a specified frame rate. Encoding follows accordingly, so that the captured individual frames are encoded at the specified frame rate. In frame transmission step 350, the captured and encoded individual frame is transmitted to the client, who expects individual frames at a specified frame rate. The process then ends in step 380.

[0031] If the generated content remains unchanged, image generation is inactive, and newly generated content is not passed to the capture and encoding stages. The capture and encoding stages are bypassed in bypass step 360. Subsequently, in the send-frame-skip step 370, a pre-calculated frame-skip message is transmitted to the client. When the client receives the frame-skip message, it interprets it as a command to continue displaying or to reuse the previously captured and encoded frame. The frame-skip message can be stored in memory, calculated during initialization, calculated at scene loading time, or at any other time prior to image generation, capture, and encoding. The process ends in step 380.

Claims

[1] Graphic design service providers (120), with: a single-frame capture unit (134) which is designed to capture single frames from image-generated content at a single-frame rate; an encoder (136) trained to encode captured individual frames at the frame rate; and to cause a processor (124) configured to transmit encoded frames when the image-generated content is at least partially changed, and to cause a frame-skip message to be transmitted, wherein the frame-skip message is configured to cause the frame-acquisition unit (134) to refrain from acquisition and the encoder (136) to refrain from encoding when the image-generated content is unchanged, wherein the frame-skip message is generated and stored in memory before the generation of the image-generated content. [2] Graphics service provider (120) according to claim 1, wherein the graphics service provider (120) is a mobile computing device. [3] Graphics service provider (120) according to claim 1 or 2, wherein the single image acquisition unit (134) and the encoder (136) are contained in a graphics processing unit (130) which is communicatively connected to the processor (124). [4] Graphics service provider (120) according to one of claims 1-3, wherein the processor (124) includes the single-image acquisition unit (134) and the encoder (136). [5] Graphics service provider (120) according to one of claims 1-4, which further comprises a graphics display unit (132) configured to generate the image-generated content and configured to detect whether the image-generated content has been at least partially changed. [6] Graphics service provider (120) according to claim 5, wherein the graphics display unit (132) is configured to set a control bit when new content is generated and is sent to the frame capture unit (134), and wherein the processor (124) is configured to use the control bit to control the frame capture unit (134) and the encoder (136). [7] Graphics service provider (120) according to one of claims 1-5, wherein the single-frame skip message is a pre-calculated H.264 skip message. [8] Graphics service provider (120; 200) for data stream transmission of image-generated content of a video stream, with: a memory (210) configured to store a pre-calculated frame-skip message indicating to a client (140) to reuse a previously transmitted frame of the video stream; a graphics display unit (132) configured to generate the image-generated content, and if the image-generated content is at least partially new, to indicate this by means of a control bit; and a central processing unit (CPU) (124) configured to use the control bit to instruct a frame acquisition unit (134) and an encoder (136) to refrain from performing any operation on the image-generated content and to cause the pre-computed frame-skip message to be sent to the client (140), the frame-skip message being generated and stored in memory prior to the generation of the image-generated content. [9] Graphics service provider (120; 200) according to claim 8, wherein the graphics generation unit (132) is set up in a graphics processing unit (GPU) (130). [10] Graphics service provider (120; 200) according to claim 9, wherein the GPU (130) includes the single-frame acquisition unit (134) and the encoder (136).

Images