Systems and methods for adjusting one or more parameters of a GPU

Abstract

Describes a method for adjusting the complexity of content rendered by a Graphics Processing Unit (GPU). The method includes processing an image frame of a game scene by the GPU. The method further includes tracking one or more metrics regarding the processing of the image frame during the processing of the image frame. During the processing of the image frame, the method includes sending a Quality Adjustment Signal (QAS) to a shader associated with a game engine. The QAS is generated based on the one or more metrics associated with the processing by the GPU. During the processing of the image frame, the method includes, upon receiving the QAS, adjusting, by the shader, one or more shader parameters, wherein adjusting the one or more shader parameters changes the complexity level of the image frame being processed by the GPU.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for adjusting one or more parameters of a graphics processing unit (GPU). Background Art

[0002] Today, many games and simulations facilitate multiple players to participate in the same scene of a game simultaneously. The multi-player aspect of such games provides a rich gaming experience in which players can communicate, cooperate, compete, and / or otherwise interact with each other and influence and interact with the shared collective gaming environment of each other. Players in a multi-player game are connected via a network connection, such as a local area network (LAN) or a wide area network (WAN).

[0003] During the multi-player game process, it is necessary to accommodate a large number of players in an online multi-player game while maintaining a high-quality gaming experience for each player. Summary of the Invention

[0004] Embodiments of the present disclosure provide systems and methods for adjusting one or more parameters of a graphics processing unit (GPU).

[0005] Other aspects of the present disclosure will become apparent from the following detailed description in conjunction with the accompanying drawings, which illustrate by way of example the principles of the embodiments described in the present disclosure.

[0006] In one embodiment, one or more parameters of the GPU are adjusted during the operation of rendering an image frame. For example, during the rendering of an image frame rather than after the rendering of the image frame, the details of the image frame are reduced.

[0007] In one embodiment, the systems and methods described herein provide shader data parameters including quality parameters. The shader selects a shader data parameter value during shader execution. For example, the shader modifies the ray iteration count or the level of detail of virtual objects such as virtual grass or virtual leaves based on the busyness of the graphics processing unit (GPU). The GPU executes the shader. Again, for example, the level of detail is determined based on the distance of the virtual object in the virtual scene. Illustratively, the level of detail is lower when the virtual object is farther away along the depth dimension in the virtual scene compared to when the virtual object is closer in the depth dimension. A quality parameter matrix with multiple values of multiple shader data parameters is generated and these values are applied during the rendering operation performed by the GPU executing the shader.

[0008] In one embodiment, a method for adjusting the complexity of content rendered by a GPU is described. The method includes processing, by the GPU, an image frame of a game scene. The method further includes tracking, during the processing of the image frame, one or more metrics regarding the processing of the image frame. During the processing of the image frame, the method includes sending a Quality Adjustment Signal (QAS) to a shader associated with a game engine. The QAS is generated based on the one or more metrics associated with the processing by the GPU. During the processing of the image frame, the method includes, upon receiving the QAS, adjusting, by the shader, one or more shader parameters, wherein adjusting the one or more shader parameters changes the complexity level of the image frame being processed by the GPU.

[0009] In one embodiment, a server for adjusting the complexity of content rendered by a GPU is described. The GPU executes a shader to process an image frame of a game scene. The GPU tracks one or more metrics regarding the processing of the image frame. The server includes a processing unit coupled to the GPU. The processing unit generates the QAS and sends it to a shader associated with a game engine. The QAS is sent to the shader while the image frame is being processed and is generated based on the one or more metrics associated with the GPU processing. Upon receiving the QAS, the shader adjusts one or more shader parameters. The adjustment of the one or more shader parameters occurs while the image frame is being processed, and the adjustment of the one or more shader parameters changes the complexity level of the image frame being processed by the GPU.

[0010] In one embodiment, a system for adjusting the complexity of content rendered by a GPU is described. The system includes a server node. The server node includes a GPU. The GPU processes an image frame of a game scene. The GPU tracks one or more metrics regarding the processing of the image frame. The GPU executes a shader. The server node further includes a processing unit coupled to the GPU. The processing unit generates the QAS and sends it to a shader associated with a game engine. The QAS is sent to the shader while the image frame is being processed. The QAS is generated based on the one or more metrics associated with the processing by the GPU. Upon receiving the QAS, the shader adjusts one or more shader parameters. The adjustment of the one or more shader parameters occurs while the image frame is being processed, and the adjustment of the one or more shader parameters changes the complexity level of the image frame being processed by the GPU. The system includes a client device communicating with the server node via a computer network.

[0011] Some advantages of the systems and methods herein include taking into account network latency, the temperature of the GPU, the power associated with the GPU, and receiving user input during the rendering of an image frame to save the rendering time of the image frame. For example, if it is determined that the network latency increases during the rendering of an image frame, the complexity of the image frame is reduced to increase the rendering rate of the image frame. The increase in the image frame rendering rate takes into account the network latency, for example, eliminating the impact of the network latency. As another example, when it is determined that the power associated with the GPU or the temperature of the GPU or both are high, the complexity during the rendering of the image frame is reduced. The reduction in complexity reduces the temperature of the GPU and the power associated with the GPU. The reduction in power also reduces the temperature of the GPU.

[0012] In addition, taking into account the temperature of the GPU and the power associated with the GPU increases the life cycle of the GPU. When the GPU is consuming a large amount of power or when the temperature of the GPU is high, increasing or maintaining the load of the GPU reduces the life cycle. By reducing the complexity, the temperature of the GPU and the power associated with the GPU are reduced to increase the life cycle of the GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The various embodiments of the present invention can be best understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0014] Figure 1 is a diagram of an embodiment of the system, showing the change of shader parameter values based on the processing power usage of a graphics processing unit (GPU).

[0015] Figure 2 is a diagram of an embodiment of the system, showing the adjustment of the value of one or more parameters of the shader based on network latency.

[0016] Figure 3 is a diagram of an embodiment of the system, showing the generation of a quality adjustment signal (QAS) for adjusting one or more parameters based on the amount of power consumed by the GPU.

[0017] Figure 4 is a diagram of an embodiment of the system, showing the generation of QAS based on the temperature of the GPU.

[0018] Figure 5 is Figure 2 a diagram of an embodiment of the system of, showing the generation of a QAS signal based on the latency of user input received from a client device.

[0019] Figure 6A is a diagram of an embodiment of a table for showing examples of parameters.

[0020] Figure 6BA diagram of an implementation of a mapping between one of the metrics associated with a GPU and a parameter.

[0021] Figure 6C A diagram of an implementation of a method, showing details of modifying virtual objects in an image frame during GPU execution of a rendering operation.

[0022] Figure 6D A diagram of an implementation of an image frame, showing adjustment of values of one or more parameters in different parts of the image frame.

[0023] Figure 7 A diagram of an implementation of a system, showing playing a game using multiple nodes.

[0024] Figure 8 A diagram of an implementation of a flowchart conceptually showing various operations performed to stream a cloud video game to a client device.

[0025] Figure 9 A block diagram of an implementation of a game console that is compatible with handover of a display device of a client device and capable of communicating with a game server via a computer network.

[0026] Figure 10 A diagram of an implementation of a head-mounted display (HMD).

[0027] Figure 11 Shows an implementation of an information service provider (INSP) architecture. Detailed Description

[0028] Describes systems and methods for adjusting one or more parameters of a graphics processing unit (GPU). It should be noted that various implementations of the present disclosure are implemented without some or all of the specific details. In other cases, well-known process operations are not described in detail to avoid obscuring various implementations of the present invention.

[0029] Figure 1It is a diagram of an implementation of system 100, showing the change in the parameter values of shader A based on the processing power utilization rate of GPU A. System 100 includes memory device system A, central processing unit (CPU) A, GPU A, video encoder A, and shader component A. As described herein, examples of memory device systems include one or more memory devices coupled to each other. Illustratively, a memory device is a device from which data is read or to which data is written. The memory device can be a read-only memory (ROM) device, or a random access memory (RAM) device, or a combination thereof. Further illustratively, the memory device includes flash memory, cache, or redundant array of independent disks (RAID). For example, as used herein, a CPU is an electronic circuit that implements or executes multiple instructions of a computer program, such as a game engine, by performing arithmetic, logical, control, and input / output (I / O) functions specified by instructions. Further illustratively, the CPU executes instructions to determine variables, such as the position, orientation, size, and location of virtual objects in the virtual scene of a game. Examples of CPUs include processors, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and programmable logic devices (PLDs), and these terms may be used interchangeably herein. As used herein, examples of virtual objects include virtual guns, virtual users, virtual weapons, virtual vehicles, virtual sports equipment, virtual background objects, and one or more virtual grass blades. In this document, virtual background objects are sometimes referred to as game content context. Examples of game content context include virtual objects such as virtual grass blades and virtual bushes. Sometimes, the terms game content context and game content may be used interchangeably in this document. As used herein, examples of GPUs include electronic circuits such as processors or ASICs or PLDs or microcontrollers that apply rendering programs to generate image frames for output to a display device for displaying images on the display device.

[0030] As used herein, examples of video encoders include compressors that compress multiple image frames to output one or more encoded image frames. Illustratively, the video encoder receives raw image frames to output I, P, and B frames. Further illustratively, the video encoder applies the H.264 standard to compress multiple image frames. An example of an image frame is a still image. In this document, the rate at which multiple image frames, such as multiple images, are rendered is sometimes referred to as the frame rate. For example, the frame rate is measured as the number of image frames rendered per second or frames per second (fps).

[0031] Memory device system A, CPU A, GPU A, and video encoder A are coupled to each other via communication medium A. Illustratively, shader component A, memory device system A, CPU A, GPU A, and video encoder A are components of a system-on-chip (SoC). Again illustratively, shader component A, memory device system A, CPU A, GPU A, and video encoder A are components of a node such as a server or a game console of a server system. As used herein, examples of communication media include wired or wireless communication media. Examples of wired communication media include buses, cables, and silicon. Examples of wireless communication media include media for transmitting data via wireless transmission protocols such as Bluetooth TM or Wi-Fi TM media.

[0032] As used herein, examples of shader components include processors, or ASICs, or PLDs, or controllers, or computer programs or parts of computer programs. For example, shader component A is located outside of GPU A and is coupled to GPU A via communication medium A. Again for example, shader component A is part of GPU A. Further illustratively, shader component A is a component within GPU A or part of a computer program executed by GPU A. Again for example, shader component A is a component or part of shader A.

[0033] Memory device system A stores game engine A and shader A. As described herein, examples of game engines include game computer programs, computer programs for generating virtual reality (VR) scenes, computer programs for generating augmented reality (AR) scenes, physical software programs for applying physical laws to generate VR or AR scenes, or rendering computer programs for applying rendering operations to generate VR or AR scenes, or combinations of two or more of them. Illustratively, physical laws are applied to collision detection or collision response. Examples of virtual scenes include VR scenes and AR scenes. For example, a virtual scene includes one or more virtual objects and one or more virtual backgrounds. Illustratively, a virtual scene includes multiple virtual users, multiple virtual trees, multiple virtual weapons held by virtual users, a virtual sky, and a virtual plane of a video game.

[0034] As used herein, examples of shaders include computer programs for rendering multiple raw image frames. Illustratively, GPU executes a shader to generate a specified level of intensity, texture, or color, or combinations of two or more of them, for all pixels in an image frame. Again illustratively, shader A is part of game engine A. It should be noted that in one embodiment, the terms intensity, brightness, and shade may be used interchangeably herein. In one embodiment, the pixel texture of an image frame is an arrangement of colors or intensities in the image frame.

[0035] Shader component A includes shader parameter adjuster A and quality matrix A. For example, as used herein, the shader parameter adjuster of a shader component is part of the computer program of the shader component. As another example, the shader parameter adjuster of a shader component is part of the processor, ASIC, or PLD that implements the shader component. As yet another example, shader component A is part of the computer program of shader A.

[0036] As used herein, the quality matrix includes multiple values of one or more parameters (such as P1, P2, and P3). An example of parameter P1 is the amount of detail within an image frame. Illustratively, when a virtual object is represented by a first number of pixels (such as 40 or 50) within an image frame, the amount of detail within the image frame is less. When the virtual object is represented by a second number of pixels (such as 100 or 200), the amount of detail is less compared to another amount of detail within the same image frame. The first number of pixels is less than the second number of pixels. Both the first number of pixels and the second number of pixels occupy the same amount of display area on the display screen of the client device. As another illustration, the amount of detail in an image frame with 10 grass blades within the image frame is less than another amount of detail in an image frame provided with 20 grass blades. Compared to 10 grass blades, 20 grass blades occupy more pixels in the image frame, but occupy the same amount of display area on the display screen of the client device. The display area of the display screen is measured as the product of the width of the display area and the length of the display area. An example of parameter P2 is the distance at which a virtual object is represented in the virtual scene of an image frame. Illustratively, parameter P2 is the distance at which a virtual object is represented in the virtual scene of an image frame in the depth dimension. An example of parameter P3 is the number of ray iterations performed during a rendering operation that renders at least a portion of the image frame. This portion includes the virtual object being rendered. In this document, ray iteration is sometimes referred to as ray tracing. During each ray iteration, shader component A calculates the color and intensity of the pixels within the image frame. For multiple ray iterations, the color and intensity of the pixels within the image frame are calculated multiple times. The rendering operation is performed by GPU A to process (such as generate or output) one or more image frames, which will be further described below.

[0037] CPU A includes processing unit A. As used herein, the processing unit includes a processor, or an ASIC, or a PLD, or a controller. Illustratively, the processing unit within the CPU is part of a PLD or ASIC or controller that performs the functions described herein as being performed by the processing unit.

[0038] During the process of playing a game, CPU A executes game engine A to determine variables of the models of virtual objects in an image frame with a virtual scene, such as shape, size, position, and orientation. CPU A sends an indication that the variable determination is completed to GPU A via communication medium A. Upon receiving the indication of completion, GPU A performs a rendering operation to determine the color and shading of each pixel of the image frame.

[0039] During the period when GPU A performs a rendering operation for generating (such as rendering) an image frame, processing unit A determines the busy level of GPU A. For example, once GPU A starts to perform the rendering operation, GPU A sends a signal to processing unit A. Upon receiving the signal, processing unit A determines that the rendering operation has started and begins to determine the busy level of GPU A.

[0040] For example, in order to determine the busy level of GPU A, processing unit A sends a request to GPU A to obtain the frame rate at which GPU A generates image frames (such as image frames 102A, 102B, and 102C). Image frames 102A - 102C are unencoded image frames of the game, such as raw image frames. The request is sent to GPU A via communication medium A. Upon receiving the request, GPU A provides the frame rate to CPU A via communication medium A. Processing unit A determines that the frame rate is greater than a predetermined frame rate to further determine that GPU A is busy. Illustratively, GPU A is busy generating image frames for multiple games. Again illustratively, GPU A is busy generating image frames 102A - 102C for the game. Continuing with this example, when GPU A is busy, GPU A is using, for example, a large amount of processing power of GPU A that is greater than a predetermined amount. The predetermined amount of power is an example of a predetermined power threshold level. On the other hand, processing unit A determines that the frame rate is less than the predetermined frame rate to further determine that GPU A is not busy. When GPU A is not busy, GPU A is using, for example, a small amount of processing power of GPU A that is less than the predetermined amount of power. The amount of processing power of GPU A is an example of the power consumed by GPU A.

[0041] When determining that GPU A is busy, processing unit A generates a quality adjuster signal (QAS) and sends it to shader parameter adjuster A via communication medium A. When GPU A is processing (such as rendering or generating) image frames 102A - 102C, the QAS signal is sent from processing unit A to shader component A. For example, during the rendering of image frame 102A, processing unit A sends the QAS signal to shader component A.

[0042] Upon receiving the QAS, the shader parameter adjuster A applies content rules to adjust the value of one or more of the parameters P1 - P3 of one or more of the image frames 102A - 102C being generated by GPU A. For example, the shader parameter adjuster A reduces the resolution of virtual objects in the image frame 102A, such as the amount of detail, or increases the distance of virtual objects within the image frame 102A in the depth dimension, such that a lower number of pixels of the virtual object occupy the image frame 102A, or reduces the number of ray iterations used to generate the pixel colors and intensities representing the virtual object in the image frame 102A, or a combination thereof. It should be noted that when the resolution of the virtual object in the image frame 102A is reduced, the resolution of the image frame 102A is also reduced. As another example, the shader parameter adjuster A reduces the amount of detail of the virtual object in the image frame 102A to a preset level of detail, or increases the distance of the virtual object within the image frame 102A in the depth dimension to a preset distance, or reduces the number of ray iterations used to generate the pixel colors and intensities representing the virtual object in the image frame 102A to a preset number of ray iterations, or a combination thereof. The preset level of detail, preset distance, and preset number of ray iterations are stored in the memory device system A.

[0043] As yet another example, the shader parameter adjuster A reduces the amount of detail of the virtual objects in the image frames 102A - 102C having virtual objects, or increases the distance of the virtual objects within the image frame in the depth dimension, or reduces the number of ray iterations used to generate the pixel colors and intensities representing the virtual objects in the image frames 102A - 102C, or a combination thereof. As yet another example, the shader parameter adjuster A reduces the amount of detail of the virtual objects in the image frames 102A - 102C having virtual objects to a preset level of detail, or increases the distance of the virtual objects within the image frames 102A - 102C in the depth dimension to a preset distance, or reduces the number of ray iterations used to generate the pixel colors and intensities representing the virtual objects in the image frames to a preset number of ray iterations, or a combination thereof. As yet another example, when generating a part of the image frame 102A, such as a pixel or a block, and not generating the remaining part of the image frame 102A, such as the remaining pixels or remaining blocks, the shader parameter adjuster A adjusts the value of one or more of the parameters P1 - P3. The shader parameter adjuster A provides the adjusted value of one or more of the parameters P1 - P3 to the shader A via the communication medium A.

[0044] When receiving adjusted values of one or more of the parameters P1 - P3, shader A outputs image frames with the adjusted values of the parameters P1 - P3, such as image frames 102A - 102C. It should be noted that one or more parameters are adjusted before the generation of the image frames 102A - 102C is completed. The image frames 102A - 102C are unencoded image frames. Shader A sends the image frames 102A - 102C to video encoder A via communication medium A. Video encoder A encodes the image frames 102A - 102C, such as compresses them, to output encoded image frames, such as encoded image frames 104A and 104B. On the other hand, when it is determined that GPU A is not busy, processing unit A does not generate QAS, and shader parameter adjuster A does not adjust any of the parameters P1 - P3.

[0045] In one embodiment, shader parameter adjuster A does not adjust the values of the parameters P1 - P3 before the image frames 102A - 102C are rendered, such as generated or produced or output. For example, before GPU A performs a rendering operation to generate the image frames 102A - 102C, CPU A does not send instructions to GPU A to adjust the values of the parameters P1 - P3.

[0046] In one embodiment, shader parameter adjuster A does not adjust the values of the parameters P1 - P3 after the image frames 102A - 102C are rendered. For example, after the image frame 102A is rendered to re - render the image frame 102A, shader parameter adjuster A does not adjust the value of the image frame 102A.

[0047] In one embodiment, as described herein, the shader is part of a game engine. For example, shader A is part of game engine A.

[0048] In one embodiment, the functions performed by a processing unit, such as processing unit A, as described herein are performed by a shader component, such as shader component A.

[0049] In one embodiment, the functions performed by the CPU as described herein are not performed by a processing unit having a CPU, but by another processor of the CPU, such as the main processing unit.

[0050] In one embodiment, GPU A includes a frame rate counter that counts the number of images generated by GPU A within a predetermined time period, such as per second or every two seconds.

[0051] In one embodiment, the shader parameter adjuster A applies content rules to determine which of the multiple virtual objects in a virtual scene is more complex than the remaining virtual objects. For example, the shader parameter adjuster A determines that the virtual vehicle track in the virtual scene has a higher level of detail than the virtual grass blades in the virtual scene. When determining that the virtual vehicle track is more detailed, the shader parameter adjuster A reduces the level of detail of the virtual vehicle track without reducing the level of detail of the grass blades. Examples of a higher level of detail include a greater number of pixels consumed in the virtual scene of image frame 102A than by the virtual grass blades. Other examples of a higher level of detail also include having a greater number of components or parts or elements within image frame 102A. Illustratively, the virtual vehicle track has railroad ties, railroad tracks, railroad fasteners, and railroad ballast, and the virtual grass blade has a blade.

[0052] As another example, the shader parameter adjuster A initially reduces the level of detail of the virtual vehicle track and then reduces the level of detail of the grass blades. As yet another example, the shader parameter adjuster A reduces the level of detail of the virtual vehicle track to less than a first predetermined level and reduces the level of detail of the virtual grass blades to less than a second predetermined level. The first predetermined level is lower than, equal to, or higher than the second predetermined level.

[0053] Figure 2 FIG. is a diagram of an embodiment of system 200, showing adjustment of the values of parameters P1 - P3 based on network latency. System 200 includes memory device system A, communication medium A, CPU A, GPU A, transmission (TX) unit A, stream engine A, computer network 202, multiple client devices A and B, receive (RX) unit A, and shader component A. Transmission unit A includes video encoder A and multiple frame buffers. One frame buffer stores image frames 102A - 102C, and another frame buffer stores encoded image frames 104A and 104B. The frame buffers are coupled to video encoder A. Receive unit A includes decoder A and multiple frame buffers. Decoder A is coupled to the frame buffers of receive unit A. As used herein, examples of a decoder include components for decoding an encoded image frame such as decompressing to output an unencoded image frame. Illustratively, the decoder applies the H.264 standard to convert P, I, and B image frames into raw image frames.

[0054] As used herein, examples of a streaming engine include a network interface controller, such as a network interface card that applies an Internet communication protocol such as Transmission Control Protocol / Internet Protocol (TCP / IP). A receiving unit A is coupled to a communication medium A. Similarly, a streaming engine A is coupled to the communication medium A. Examples of a computer network 202 include a wide area network (WAN) such as the Internet, or a local area network (LAN) such as the Internet, or a combination thereof. As used herein, a client device is a device operated by a user to access a game executed using a game engine A. As used herein, examples of a client device include a computer, a mobile phone, a tablet, a smartphone, a smart TV, a game console, and a head-mounted display (HMD), etc. As used herein, an HMD is a display device worn by a user to view a virtual scene (such as a VR scene or an AR scene). The VR scene or the AR scene is generated when the game engine A is executed.

[0055] Client device A is operated by user A, and client device B is operated by another user B. A user account server assigns user account A to user A, which will be further described below. Similarly, the user account server assigns user B to another user account B. User A logs in to user account A to access a game from one or more nodes, which will be further described below. Similarly, user B logs in to user account B to access a game from one or more nodes.

[0056] During a rendering operation in which frames 102A - 102C are rendered, processing unit A sends a signal via communication medium A to streaming engine A to send a predetermined number of test packets (such as one or two packets), which carry an instruction indicating that client device A is to send back a predetermined number of test packets to processing unit A. When the signal is sent to streaming engine A, processing unit A starts a counter or a timer to count the network latency, such as the amount of time it takes to send a predetermined number of packets from streaming engine A to client device A and the amount of time it takes for streaming engine A to receive a predetermined number of packets from client device A. For example, upon receiving a signal from GPU A indicating that a rendering operation is being performed to generate one or more of the image frames 102A - 102C, processing unit A sends a signal to streaming engine A to send a predetermined number of test packets, which carry an instruction indicating that client device A is to send back a predetermined number of test packets. Upon receiving the signal from processing unit A, streaming engine A generates a predetermined number of test packets, applies the Internet protocol to embed the instruction within the test packets, and sends the test packets to client device A via computer network 202.

[0057] Client device A is the device for which image frames 102A - 102C are being rendered. For example, in response to receiving an input signal from client device A via computer network 202, CPU A determines variables of virtual objects in the virtual scene. Additionally, based on the variables determined by CPU A, GPU A determines the colors and intensities of the virtual objects in the virtual scene. For another example, upon receiving a request to play a game from client device A via computer network 202, GPU A renders virtual objects for the game.

[0058] Upon receiving a predetermined number of test packets with instructions, client device A parses the test packets to obtain the instructions. Client device A applies the Internet protocol to the instructions to generate a predetermined number of return test packets and sends the return test packets to stream engine A via computer network 202. Upon receiving the predetermined number of return test packets, stream engine A applies the Internet protocol to parse the return test packets to obtain the instructions. Stream engine A sends the instructions to processing unit A via receiving unit A. Upon receiving the instructions, processing unit A shuts down a counter or timer that has been started to determine the network latency when transmitting a predetermined number of test packets from stream engine A to client device A via computer network 202 and from client device A to stream engine A via computer network 202. For example, the network latency is the amount of time between the operation of stream engine A sending a predetermined number of test packets to client device A and the operation of stream engine A receiving a predetermined number of return test packets.

[0059] Processing unit A determines whether the network latency is greater than a predetermined amount of network latency stored in memory device system A. The predetermined amount of network latency is an example of a predetermined network latency threshold level. When determining that the network latency is greater than the predetermined amount of network latency (such as threshold Th), processing unit A generates a QAS and sends it to shader component A via communication medium A. In response to receiving the QAS, shader parameter adjuster A of shader component A adjusts the value of one or more of the parameters P1, P2, and P3 in the manner described herein and provides the adjusted values to shader A via communication medium A to generate image frames 102A - 102C. On the other hand, when determining that the network latency is not greater than the predetermined amount of network latency, processing unit A does not generate a QAS.

[0060] Video encoder A receives image frames 102A - 102C output from shader A, and encodes the image frames to generate encoded image frames 104A and 104B. Streaming engine A receives the encoded image frames 104A and 104B from video encoder A, applies Internet protocol to the encoded image frames 104A and 104B to generate one or more packets, and transmits the one or more packets to client device A via computer network 202. Client device A includes a video decoder for decoding the encoded image frames 104A and 104B to generate image frames 102A - 102C. Client device A displays image frames 102A - 102C on a display device of client device A. As used herein, examples of display devices include liquid crystal displays (LCDs), light emitting diode (LED) displays, and plasma displays.

[0061] Figure 3 FIG. is a diagram of an embodiment of system 300, showing the generation of QAS based on the amount of power consumed by GPU A. System 300 includes memory device system A, communication medium A, power measurement device (PMD) A, GPU A, power supply A, CPU A, and shader component A. As used herein, examples of power measurement devices include voltage sensors, current sensors, or power sensors. A current sensor includes one or more current shunt resistors to measure current. Power supply A and PMD A are coupled to communication medium A. In addition, PMD A is coupled to the input of GPU A to measure the power at the input of GPU A. For example, PMD A is coupled to the power input pin of GPU A.

[0062] During the execution of rendering operations by GPU A to generate one or more of image frames 102A - 102C, PMD A measures the amount of power consumed by GPU A at the input of GPU A, such as the power consumption of GPU A, to output power measurement data. For example, PMD A includes a sampler or analog - to - digital converter (ADC) to convert the analog value of the measured power amount into a digital value of power measurement data.

[0063] The power measurement data is transmitted via communication medium A to processing unit A. Processing unit A receives the power measurement data and determines whether the power measurement data has a value lower than a predetermined power threshold, where the predetermined power threshold is another example of a predetermined power threshold level. For example, when rendering operations are being performed to generate image frame 102A, processing unit A determines that the power measurement data is greater than the predetermined power threshold. The predetermined power threshold is stored in memory device system A.

[0064] When it is determined that the power measurement data is lower than a predetermined power threshold, processing unit A does not generate a QAS signal. On the other hand, when it is determined that the power measurement data is greater than the predetermined power threshold, processing unit A generates a QAS signal. The QAS signal is generated while or during the generation of one or more of frames 102A - 102C by performing a rendering operation. Processing unit A sends the QAS to shader parameter adjuster A via communication medium A. Upon receiving the QAS, shader parameter adjuster A adjusts the value of one or more of parameters P1 - P3 of one or more of the image frames of image frames 102A - 102C in the manner described herein. The adjusted value is sent from shader parameter adjuster A to shader A via communication medium A, thereby rendering one or more of the image frames of image frames 102A - 102C with the adjusted value in the manner described herein.

[0065] In one embodiment, a plurality of power measurement devices are coupled to the input of GPU A to measure the power consumed by GPU A.

[0066] Figure 4 FIG. is a diagram of an embodiment of system 400, showing the generation of QAS based on the temperature of GPU A. System 400 includes memory device system A, communication medium A, GPU A, CPU A, temperature sensor A, and shader component A. As used herein, examples of temperature sensors include thermocouples, thermistors, resistance temperature detectors, and semiconductor sensors. Temperature sensor A is coupled to GPU A and communication medium A. For example, temperature sensor A is attached to the surface of the semiconductor chip of GPU A.

[0067] During the generation of one or more frames 102A - 102C by GPU A, temperature sensor A senses the temperature of GPU A to output temperature data and provides the temperature data to processing unit A via communication medium A. For example, temperature sensor A includes a sampler or ADC to convert the analog temperature signal into digital data of the temperature of GPU A, thereby providing temperature data. Processing unit A receives the temperature data and determines whether the temperature of GPU A exceeds a predetermined temperature threshold TMP THR stored in memory device system A. For example, processing unit A includes a temperature analyzer to determine whether the temperature of GPU A exceeds the predetermined temperature threshold TMPTHR. The temperature threshold TMP THR is an example of a predetermined temperature threshold level. When it is determined that the temperature of GPU A exceeds the predetermined temperature threshold, processing unit A generates a QAS signal and sends the QAS signal to shader parameter adjuster A. Upon receiving the QAS, shader parameter adjuster A adjusts the values of one or more parameters P1 - P3 of one or more of the image frames 102A - 102C in the manner described herein and provides the adjusted values to shader A via communication medium A. Upon receiving the adjusted values, shader A generates one or more frames with adjusted values of one or more of the parameters P1 - P3. On the other hand, when it is determined that the temperature of GPU A does not exceed (e.g., is not greater than) the predetermined temperature threshold, processing unit A does not generate a QAS signal.

[0068] Figure 5 FIG. is a diagram of an embodiment of system 200, showing the generation of a QAS signal based on the latency of a user input received from client device A. Herein, the latency of a user input is sometimes referred to as user input latency. User input latency occurs when GPU A is rendering image frames 102A - 102C and CPU A receives a user input such as user input A2 from a client device via computer network 202.

[0069] User A selects or moves one or more buttons of client device A, such as touch screen buttons or hardware buttons or the joystick of a hand-held controller (HHC), and / or performs a movement of a body part of User A to generate user input A1. Examples of body parts include hands, faces, fingers, and legs. For example, user input A1 indicates that a virtual object in a virtual scene will move from one position to another, or from one orientation to another, or jump or dodge or bend during the execution of game engine A1. User input A1 is sent from client device A to streaming engine A via computer network 202. Streaming engine A sends user input A1 to CPU A via receiving unit A. CPU A executes game engine A to determine variables of virtual objects in the virtual scene based on user input A1. For example, upon receiving user input A1 indicating that a virtual object will jump, CPU A determines that the virtual object will perform a jumping motion. As another example, upon receiving user input A1 indicating that a virtual object will move from a first position to a second position, CPU A determines that the virtual object will move from the first position to the second position.

[0070] CPU A sends variables to GPU A via communication medium A. Upon receiving the variables, GPU A applies rendering operations to the variables to generate one or more of the image frames 102A - 102C. For example, image frame 102A has a virtual scene. During the time when GPU A renders one or more of the image frames 102A - 102C, CPU A receives user input A2 from client device A via computer network 202, stream engine A, and receiving unit A. In response to user A's selection of one or more buttons on client device A or in response to the movement of user A's body part, client device A generates user input A2. User input A2 triggers the modification of one or more of the parameters P1 - P3 of one or more of the image frames 102A - 102C that are being rendered. For example, user input A2 indicates to CPU A that one or more additional image frames other than image frames 102A - 102C will be rendered by GPU A based on user input A2, and after receiving user input A2, the time taken to render one or more of the image frames 102A - 102C is less than the amount of time used to render one or more of the image frames 102A - 102C before receiving user input A2. Illustratively, CPU A generates additional values for the variables of the virtual objects based on user input A2. For example, CPU A determines the additional positions and additional orientations of the virtual objects to be rendered in image frames 102A - 102C. CPU A sends the additional positions and additional orientations to GPU A via communication medium A. Upon receiving the additional positions and additional orientations, GPU A applies rendering operations to generate one or more additional image frames. It should be noted that after receiving user input A1, processing unit A continuously receives user input A2. For example, no other user input is received between the receipt of user inputs A1 and A2.

[0071] In response to receiving user input A2, processing unit A sends a query signal to GPU A via communication medium A to determine whether GPU A is generating one or more of the image frames 102A - 102C based on user input A1. Upon receiving the query signal, GPU A sends a response signal to processing unit A via communication medium A, indicating that GPU A has not completed generating one or more of the image frames 102A - 102C. For example, GPU A sends a response signal indicating that GPU A is still rendering image frame 102A and has not completed rendering image frame 102A. Upon receiving the response signal, processing unit A generates a QAS signal and sends the QAS signal to shader parameter adjuster A via communication medium A. Upon receiving the QAS signal, shader parameter adjuster A modifies the value of one or more of the parameters P1 - P3 of one or more of the image frames 102A - 102C in the manner described herein, and provides the modified value to shader A via communication medium A. For example, shader parameter adjuster A reduces the level of detail of virtual objects in the virtual scene of image frame 102A. Shader A renders one or more of the image frames 102A - 102C according to the modified value.

[0072] In one embodiment, processing unit A includes a timer to determine the user input waiting time, such as the time difference between when processing unit A receives user input A2 and when it receives user input A1. When determining that the user input waiting time is less than a predetermined user input waiting time, such as a predetermined time difference, processing unit A generates a QAS signal. On the other hand, when determining that the user input waiting time is greater than the predetermined user input waiting time, the processing unit does not generate a QAS signal.

[0073] Figure 6A is a diagram of an embodiment of table 600, showing examples of parameters P1, P2, and P3. An example of any one of the parameters P1, P2, and P3 is the image frames 102A - 102C( Figure 5) the resolution (R) of at least a portion such as a virtual object within any one of the image frames. Illustratively, a resolution with an R1 value is lower than a resolution with an R2 value, and a resolution with an R2 value is lower than a resolution with an R3 value. For example, an R2 value refers to a higher pixel count than an R1 value, and an R3 value refers to a higher pixel count than an R2 value. Further illustratively, an R3 value refers to an a3×b3 resolution, an R2 value refers to an a2×b2 resolution, and an R1 value refers to an a1×b3 resolution, where each of a1, a2, a3, b1, b2, and b3 is an integer. Additionally, a3 is greater than a2, a2 is greater than a1, and b3 is greater than b2, and b2 is greater than b1. As used herein, the resolution of a portion of an image frame refers to the number of pixels used to represent that portion in the image frame. For example, the resolution refers to the height and width of that portion of the image frame. The value of the resolution R ranges from R1 to Rn, where n is an integer greater than 1. The resolution of a portion of an image frame is an example of the complexity of the image frame. For example, a decrease in the pixel resolution R reduces the complexity level such as the amount of detail of the virtual object represented by the pixels, and an increase in the resolution R increases the complexity level of the virtual object.

[0074] Another example of any one of the parameters P1 - P3 is the ray iteration count (RIC), which is the number of iterations performed to render at least a portion such as a virtual object of any one of the image frames 102A - 102C. Illustratively, the ray iteration count to be applied to render the pixels of a virtual object in the virtual scene representing image frame 102A determined by shader parameter adjuster A is less than the ray iteration count to be applied to render the pixels of another virtual object in the virtual scene of image frame 102A determined by shader parameter adjuster A. Further illustratively, the ray iteration count is the number of times shader parameter adjuster A calculates the pixel color and intensity of the virtual object in image frame 102A.

[0075] The ray iteration count is another example of the complexity of an image frame. For example, a reduction in the ray iteration count used to render a portion of an image frame reduces the complexity level of that portion, and an increase in the number of ray iterations increases the complexity level.

[0076] The value of the ray iteration count ranges from RIC1 to RICn, where n is an integer greater than 1. It should be noted that RIC1 represents fewer ray iteration counts compared to RIC2, and RIC2 represents fewer ray iteration counts compared to RIC3.

[0077] Another example of any one of the parameters P1 - P3 is the number of virtual objects (VOR) rendered in any one of the image frames 102A - 102C. Illustratively, upon receiving the QAS, shader parameter adjuster A (Figure 5 )Apply content rules Figure 5 )to render a fewer number of virtual objects in image frame 102A as compared to the amount of virtual objects to be rendered in image frame 102A determined prior to receiving the QAS. Further illustratively, CPU A Figure 5 )determines that two virtual objects are to be rendered in the virtual scene of image frame 102A Figure 5 ). CPU A sends an instruction via communication medium A Figure 5 )to GPUA to render two virtual objects. Upon receiving the instruction, GPU A begins to render two virtual objects. However, upon receiving the QAS signal, shader parameter adjuster A applies content rules Figure 5 )to determine that one of the two virtual objects is to be rendered instead of two virtual objects. This one virtual object among the two virtual objects has less or more detail as compared to the other virtual object among the two virtual objects. GPU A does not render the other virtual object and renders this virtual object in the virtual scene of image frame 102A.

[0078] The value of the number of rendered virtual objects (VOR) ranges from VOR1 to VORn, where n is an integer greater than 1. It should be noted that VOR1 represents a fewer number of virtual objects as compared to VOR2, and VOR2 represents a fewer number of virtual objects as compared to VOR3.

[0079] The number of virtual objects rendered in an image frame is another example of the complexity of the image frame. For example, a reduction in the number of virtual objects rendered within an image frame reduces the complexity level of the image frame, and an increase in the number of virtual objects increases the complexity level.

[0080] Another example of any one of the parameters P1 - P3 includes a priority (PRTY), and the virtual object will be rendered in any one of the image frames 102A - 102C according to this priority. Illustratively, during the rendering operation, before the second virtual object in the virtual scene of image frame 102A is rendered by GPU A, the first virtual object in the virtual scene of image frame 102A is rendered by GPU A. Before the rendering operation, CPU A instructs GPU A that the second virtual object will be rendered at a position further in the depth dimension than the first virtual object. Compared with the depth of the first virtual object, the second virtual object has a greater depth, such as a greater distance in the depth dimension, and is rendered after the first virtual object. In another example, during the rendering operation, before the second virtual object in the virtual scene of image frame 102A is rendered by GPU A, the first virtual object in the virtual scene of image frame 102A is rendered by GPU A. Before the rendering operation, CPU A instructs GPU A that the first virtual object will be rendered at a position more prominent within one or more of the image frames 102A - 102C than the position of the second virtual object in one or more of the image frames 102A - 102C. Compared with the amount of distance of the second virtual object in the depth dimension, a smaller amount of distance of the first virtual object in the depth dimension is an example of the prominent position of the first virtual object. In this article, the smaller amount of distance of the first virtual object is sometimes referred to as a closer distance, and in this article, the amount of distance of the second virtual object is sometimes referred to as a farther distance. For example, the depth dimension is measured along an axis perpendicular to the display plane of the image frame 102A, 102B, or 102C. The display plane of the image frame is the plane of the display screen that displays the image frame. The rendering priority PRTY is determined by the shader parameter adjuster A.

[0081] The value range of the rendering priority is from PRTY1 to PRTYn, where n is an integer greater than 1. It should be noted that compared with PRTY2, PRTY1 represents a lower rendering priority, and compared with PRTY3, PRTY2 represents a lower rendering priority. The change in the priority of rendering virtual objects in one or more of the image frames 102A - 102C is an example of the change in the complexity level of one or more of the image frames 102A - 102C.

[0082] Another example of any one of the parameters P1 - P3 includes a quality parameter (QP) that provides weights to a resolution R, a ray iteration count RIC, a number of rendered virtual objects (VOR), and a rendering priority PRTY applied to the virtual objects in any one of the image frames 102A - 102C. Illustratively, shader parameter adjuster A applies a weight W1 to the resolution R, a weight W2 to the ray iteration count RIC, a weight W3 to the number of rendered virtual objects VOR, a weight W4 to the priority PRTY, and sums the weighted resolution, weighted ray iteration count, weighted number of rendered virtual objects, and weighted priority to determine the value of the quality parameter, where each of W1, W2, W3, and W4 is a real number. Further illustratively, shader parameter adjuster A multiplies the value of the weight W1 by the value of the resolution R, the weight W2 by the value of the ray iteration count RIC, the weight W3 by the value of the number of rendered virtual objects VOR, and the weight W4 by the priority PRTY to calculate the value of the quality parameter. Shader parameter adjuster A sends the value of the quality parameter to GPU A to be applied to image frames 102A, 102B, or 102C during a rendering operation.

[0083] The value of the quality parameter QP ranges from QP1 to QPn, where n is an integer greater than 1. It should be noted that QP1 represents a lower rendering quality compared to QP2, and QP2 represents a lower rendering quality compared to QP3.

[0084] Figure 6B is a diagram of an embodiment of a mapping 650 between metrics associated with GPU A ( Figure 5 ) and the parameters P1, P2, or P3. Shader parameter adjuster A ( Figure 5 ) applies the mapping 650 to determine the values of the parameters P1, P2, or P3. The mapping is an example of a content rule ( Figure 5 ) that shader parameter adjuster A applies to determine the values of one or more of the parameters P1, P2, and P3. Examples of metrics include the amount of power consumed by GPU A, network latency, the temperature of GPU A, user input latency, or a combination of two or more of them.

[0085] Shader parameter adjuster A receives a QAS, such as QAS A, from processing unit A. QAS A is generated when the power consumed by GPU A has a value PWR1, the network latency has a value NL1, GPU A has a temperature value temp1, the user input latency has a value UL1, or a combination of two or more of them. For example, user input A2 ( Figure 5 ) is received before one or more of the image frames 102A - 102C (Figure 5 ) When the user input waiting time has a value of UL1. After receiving QAS A, the shader parameter adjuster A accesses the mapping 650 from the memory device system A( Figure 5 ) to determine that the resolution of at least a portion of the image frame 102A is R1, the ray iteration count of this portion is RTC1, the quality parameter is QP1, the number of virtual objects rendered in the image frame 102A is VOR1, and the priority of the virtual objects in the image frame 102A is PRTY1.

[0086] Similarly, the shader parameter adjuster A receives a QAS from the processing unit A, such as QAS B. QAS B is generated when the power consumed by GPU A has a value of PWR2, the network waiting time has a value of NL2, GPU A has a temperature value of temp2, the user input waiting time has a value of UL2, or a combination of two or more of them. For example, before one or more of the image frames in the image frames 102A - 102C( Figure 5 ) are being rendered and have completed rendering, when the user input A2( Figure 5 ) is received, the user input waiting time has a value of UL2. After receiving QAS B, the shader parameter adjuster A accesses the mapping 650 from the memory device system A( Figure 5 ) to determine that the resolution of at least a portion of the image frame 102A is R2, the ray iteration count of this portion is RTC2, the quality parameter is QP2, the number of virtual objects rendered in the image frame 102A is VOR2, and the priority of the virtual objects in the image frame 102A is PRTY2.

[0087] Figure 6C is a diagram of an embodiment of method 660, showing the details of modifying virtual objects in the image frame 102A( Figure 5 ) during the rendering operation performed by GPU A( Figure 5 ). In method 660, CPU A instructs GPU A to draw virtual grass blades, draw virtual shrubs, and draw virtual clouds within the image frame 102A via communication medium A( Figure 5 ). CPU A further sets the details of the blades to D1, and sets the priority of drawing the blades to PRTY1, and provides the details and priority to GPU A. Then, CPU A instructs GPU A to execute shader A( Figure 5 ). During the execution of shader A, GPU A changes the detail level from D1 to value D2. The detail level is part of the quality parameter, which is modified by GPU A during the execution of shader A. The quality parameter is based on the busyness of GPU A during the execution of shader A, the temperature of GPU A during the execution of shader A, the amount of power consumed by GPU A during the execution of shader A, and whether it is from the client device A(Figure 5 )Receives user input A2 to request a less complex or smoother image frame playback, such as image frames 102A - 102C( Figure 5 ).

[0088] It should be noted that in one embodiment, when CPU A generates instructions for drawing virtual grass blades, drawing virtual shrubs, and drawing virtual clouds, CPU A does not generate instructions for GPU A to change quality parameters. For example, before GPU A executes shader A to draw virtual grass blades, virtual shrubs, and virtual clouds, CPU A does not instruct GPU A to change quality parameters. Instead, during the execution of shader A to draw virtual grass blades, virtual shrubs, and virtual clouds, the network latency, the temperature of GPU A, the amount of power consumed by GPU A, or the user input latency, or a combination of two or more of them, is determined, and based on this determination, GPU A adjusts the quality parameters to generate one or more of the image frames 102A - 102C.

[0089] Figure 6D FIG. is a diagram showing an embodiment of image frames 692, 694, 696, 698, and 699, showing the adjustment of the value of one or more of the parameters P1 - P3 in different parts of the image frames. Each of the image frames 692, 694, 696, 698, and 699 is an example of any one of the image frames 102A - 120C( Figure 5 ).

[0090] Each of the image frames 692 - 698 has a part A and a part B. Part B has adjusted values of one or more of the parameters P1 - P3, and part A does not have adjusted values. For example, the number of times ray tracing is performed to render part A is greater than the number of times ray tracing is performed to render part B. For example, part A is generated before shader A( Figure 5 ) generates part B. As another example, part A is generated after shader A generates part B. As yet another example, part A is generated while part B is being generated. Part A in image frame 692 is above part B and has a smaller size than part B. Part A in image frame 694 is above part B and has the same size as part B. Part A in image frame 696 is above part B and has a larger size than part B. In image frame 698, part B is surrounded by part A and has a smaller size compared to the size of part A.

[0091] Image frame 699 has multiple parts A1, A2, and B. For example, parts A1 and A2 of image frame 699 are generated by shader A before shader A generates part B of image frame 699. For example, after shader A generates part B of image frame 699, shader A generates parts A1 and A2 of image frame 699. Yet another example is that parts A1 and A2 of image frame 699 are generated by shader A while shader A generates part B of image frame 699. Parts A1 and A2 have adjusted values of one or more of the parameters P1 - P3, and part B does not have adjusted values of one or more of the parameters P1 - P3. For example, the adjusted value of part A1 is different from the adjusted value of part A2. Another example is that the adjusted value of part A1 is the same as (such as equal to) the adjusted value of part A2. Yet another example is that the adjusted value of part A1 is substantially the same as (such as within a predetermined range) the adjusted value of part A2.

[0092] In one embodiment, in one or more of the image frames 692 - 694, part A is below part B.

[0093] In one embodiment, parts A of image frames 692 - 698 have adjusted values of one or more of the parameters P1, P2, or P3, and part B does not have adjusted values.

[0094] In one embodiment, part B of image frame 699 has adjusted values of one or more of the parameters P1 - P3, and parts A1 and A2 of image frame 699 do not have adjusted values of one or more of the parameters P1 - P3.

[0095] Figure 7 FIG. is a diagram of an embodiment of system 700, showing the use of nodes A and B for distributed gaming. System 100 includes a user account server 702, a node assembly server 704, a switch system 706, nodes A and B, a computer network 202, and client devices A and B.

[0096] As used herein, a node is a hardware server or a game console that executes game engine A. For example, a node has a housing separate from the housing of another node. Another example is that a node is placed on a rack in a data center, which is different from the rack on which another node is placed within the data center. In one embodiment, as used herein, a server includes a processor and a memory device. The processor is coupled to the memory device.

[0097] As used herein, a switch system includes one or more switches that facilitate data transfer between the node assembly server 704 and one or more of nodes A and B. For example, the switch system is a switch fabric. The switch fabric has a large amount of bandwidth between nodes and is often dynamically reconfigured and allows for quality of service (QoS). As another example, the switch system includes a multiplexer that selects between nodes A and B to execute game engine A. As yet another example, the switch system includes one or more transistors that facilitate data transfer between the node assembly server 704 and one or more of nodes A and B. As still another example, the switch system includes one or more switches, each of which changes position between an open position and a closed position. The open position of the switch decouples the node assembly server 704 from the node coupled to the switch. The closed position of the switch couples the node assembly server 704 to the node coupled to the switch.

[0098] Memory device system A includes GPU memory device A, which includes a frame buffer for storing encoded image frames 104A and 104B and a frame buffer for storing image frames 102A - 102C ( Figure 5 ). GPU memory device A is coupled to communication medium A.

[0099] When login information such as a username and password is received from user A via an input device (e.g., a handheld controller, a camera, etc.) of client device A, client device A generates game request 708A and sends game request 708A and the login information to user account server 702 via computer network 202. Similarly, when login information is received from user B via an input device of client device B, client device B generates game request 708B and sends game request 708B and the login information to user account server 702 via computer network 202.

[0100] User account server 702 determines whether client device A is authorized to access game engine A based on the login information received with game request 708A. For example, after authenticating the login information received with game request 708A, user account server 702 determines that client device A is authorized to access game engine A. In addition, user account server 702 determines whether client device B is authorized to access game engine A based on the login information received with game request 708B. When it is determined that user accounts A and B are authorized to access game engine A, user account server 702 sends a signal to node assembly server 704 to allow the execution of game engine A.

[0101] When the node assembly server 704 receives a signal from the user account server 702, it selects, via the switch system 706, one or both of nodes A and B that will execute game engine A to initialize one or both of nodes A and B. For example, the node assembly server 704 sends a signal to the control input of the switch system 706 to couple to one of nodes A and B. When a signal is received at the control input, the switch system 706 closes the position of one switch in the switch to connect the node assembly server 704 to node A and opens the position of its other switch to disconnect the node assembly server 704 from node B.

[0102] Execute game engine A to transfer data such as encoded frames from one or both of nodes A and B to client devices A and B via computer network 202. When data such as encoded frames are being transferred from node A to client devices A and B, GPU A is a shared resource. GPU A is shared between client devices A and B. After sending game request 708A from client device A to node A, the user inputs A1 and A2 shown above with reference to Figure 5 are sent from client device A to node A.

[0103] Upon receiving the encoded image frames 104A - 104B ( Figure 5 ) via computer network 202 ( Figure 5 ), client device A includes a decoder that decodes, such as decompresses, the encoded image frames 104A and 104B to generate image frames 102A - 102C. The processor (such as GPU) of client device A displays image frame 102A to render the virtual scene of the game and also displays image frames 102B and 102C on the display device of client device A.

[0104] It should be noted that, in one embodiment, system 700 includes multiple nodes in addition to those Figure 7 shown. For example, system 700 includes 50 nodes, or 25 nodes, or 5 nodes.

[0105] Figure 8 is an embodiment of a flowchart conceptually showing the various operations performed to stream cloud video games to client device A or B ( Figure 7 ) As node A or B ( Figure 7) For example, the game server 802 executes a video game and generates raw (uncompressed) video 804 and audio 806. The video 804 and audio 806 are captured and encoded for streaming, as indicated by reference numeral 808 in the figure shown. Encoding provides compression of the video and audio streams to reduce bandwidth usage and optimize the gaming experience. Examples of encoding formats include H.265 / MPEG-H, H.264 / MPEG-4, H.263 / MPEG-4, H.262 / MPEG-2, WMV, VP6 / 7 / 8 / 9, etc.

[0106] The encoded audio 810 and the encoded video 812 are further sub-packaged into network packets, as indicated by reference numeral 814, for transmission over a computer network 202 ( Figure 7 ) such as the Internet. In some embodiments, the network packet encoding process also employs a data encryption process to provide enhanced data security. In the implementation shown, audio packets 816 and video packets 818 are generated for transmission over a computer network 820 (which is an example of the computer network 202).

[0107] The game server 802 also generates haptic feedback data 822, which is also sub-packaged into network packets for network transmission. In the implementation shown, haptic feedback packets 824 are generated for transmission over the computer network 820.

[0108] The foregoing operations of generating the original video, audio, and haptic feedback data are performed on the game server 802 in the data center, and the operations of encoding the video and audio and packetizing the encoded audio / video and haptic feedback data for transmission are performed by a streaming engine (such as Streaming Engine A) in the data center. As noted, the audio, video, and haptic feedback packets are transmitted over the computer network 820. As indicated by reference numeral 826, the client device A or B disassembles, such as by parsing, the audio packet 816, the video packet 818, and the haptic feedback packet 824 to extract the encoded audio 828, the encoded video 830, and the haptic feedback data 832 from the network packets at the client device. If the data has been encrypted, the data is also decrypted. Then, as indicated by reference numeral 834, the client device decodes the encoded audio 828 and the encoded video 830 to generate client-side original audio and video data for rendering on the display device 840 of the client device. The haptic feedback data 832 is processed by a processor of the client device (such as client device A or B) to generate a haptic feedback effect at the controller device 842 or other interface device (such as an HMD, etc.), and the haptic effect can be rendered through the controller device 842 or other interface device. An example of a haptic effect is the vibration or rumbling of the controller device 842. The controller device 842 is an example of a handheld controller having a plurality of buttons. The controller device 842 is also part of the client device A or B. For example, the client device A or B includes the controller device 842 and a game console. As another example, the client device A or B includes the controller device 842 and an HMD. The controller device 842 communicates with the HMD via a wireless protocol such as TM Bluetooth or Wi-Fi TM . As yet another example, the client device A or B includes the controller device 842, a game console, and an HMD. The game console communicates with the HMD via a wireless protocol or via a wired medium such as a cable, and the controller device 842 communicates with the game console via a wireless protocol or via a wired medium.

[0109] It should be understood that the video game responds to user inputs, such as user inputs A1 and A2 ( Figure 5) Therefore, a program flow similar to the one described herein for transmitting and processing user input is executed but in the opposite direction from client device A or B to game server 802. As shown, controller device 842 or another input device, such as a body part of user A, etc., or a combination thereof generates input data 844, which is an example of any one of user inputs A1 and A2. This input data 844 is packetized at client device A or B for transmission over computer network 820 to the data center. Game server 802 unpacks and reassembles the input data packets 846 to define input data 848 on the data center side. Input data 848 is fed into game server 802, which processes input data 848 to update the saved data of the game state of the video game.

[0110] During the transmission of audio packets 1016, video packets 818, and tactile feedback packets 824 via computer network 820, in some embodiments, the data transmission over computer network 820 is monitored to ensure quality of service. For example, as indicated by reference numeral 850, the network condition of computer network 820, including both upstream and downstream network bandwidths, is monitored and the game stream is adjusted in response to changes in the available bandwidth. That is, as indicated by reference numeral 852, the encoding and decoding of network packets are controlled based on the current network condition.

[0111] Figure 9 is compatible with the handover of the display device of client device A or B and is capable of communicating via computer network 820 ( Figure 8 ) with game server 802 ( Figure 8 ) of an embodiment of game console 900. Game console 900 is located within the data center or at the location where user A is located. In some embodiments, game console 900 is used to execute a game displayed on client device A or B such as an HMD. Game console 900 is equipped with various peripheral devices that can be connected to game console 900. Game console 900 has a dynamic random access memory (XDRAM) unit 926, a unit processor 928, a reality synthesizer graphics processing unit 930 having a dedicated video random access memory (VRAM) unit 932, and an input / output (I / O) bridge 934. Game console 900 also has a Blu ray Disc read-only memory (BD-ROM) disc reader 940 for reading from disc 940a, and a removable plug-in hard disk drive (HDD) 936 that can be accessed through I / O bridge 934. Optionally, game console 900 also includes a memory card reader 938 for reading compact flash memory cards, memory memory cards, etc., that can be similarly accessed through I / O bridge 934. I / O bridge 934 is also connected to Wireless link port 918, Institute of Electrical and Electronics Engineers (IEEE) 802.9b / g wireless network (Wi-Fi TM ) port 920, Gigabit Ethernet port 922, and Universal Serial Bus (USB) 2.0 port 924 capable of supporting connections.

[0112] In operation, the I / O bridge 934 processes all wireless, USB, and Ethernet data, including data from game controllers 842 and / or 903 and from HMD 905. For example, when user A is playing a game generated by executing a portion of game engine A, the I / O bridge 934 receives input data from game controllers 842 and / or 903 and / or from HMD 905 via a link and directs the input data to the unit processor 928, which accordingly updates the current state of the game, such as variables. For example, a camera in HMD 905 captures the pose of user A to generate an image representing the pose. This image is another example of user input A1 or A2 ( Figure 5 ). Each of game controllers 842 and 903 is an example of a handheld controller.

[0113] In addition to game controllers 842 and 903 and HMD 905, the wireless, USB, and Ethernet ports also provide connections for other peripheral devices, such as remote control 904, keyboard 906, mouse 908, portable entertainment device 910 (e.g., Sony Playstation entertainment device, etc.), camera 912 (e.g., camera, etc.), microphone headset 914, and microphone 915. In some embodiments, such peripheral devices are wirelessly connected to game console 900. For example, portable entertainment device 910 communicates via a Wi-Fi TM ad-hoc connection, while microphone headset 914 communicates via a link.

[0114] Providing these interfaces means that game console 900 may also be compatible with other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, Internet Protocol (IP) voice telephones, mobile telephones, printers, and scanners.

[0115] Furthermore, a conventional memory card reader 916 is connected to game console 900 via USB port 924, enabling the memory card 948 used by game console 900 to be read. Game controllers 842 and 903 and HMD 905 are operable to via Link 918 communicates wirelessly with game console 900 or is connected to USB port 924, thereby also receiving power for charging the batteries of game controllers 842 and 903 and HMD 905. In some embodiments, each of game controllers 842 and 903 and HMD 905 includes a memory, a processor, a memory card reader, a permanent memory such as flash memory, a light emitter such as an illuminated spherical segment, a light-emitting diode (LED), or an infrared lamp, a microphone and a speaker for ultrasonic communication, a sound-insulating cavity, a digital camera, a built-in clock, a recognizable shape such as a spherical segment facing game console 900, and a wireless device using a protocol such as Wi-Fi TM and so on.

[0116] Game controller 842 is a controller designed to be used with both hands by User A, and game controller 903 is a single-handed controller with an accessory. HMD 905 is designed to fit on User A's head and / or in front of the eyes. In addition to one or more analog joysticks and traditional control buttons, each of game controllers 842 and 903 is susceptible to three-dimensional position determination. Similarly, HMD 905 is susceptible to three-dimensional position determination. Thus, in some embodiments, in addition to or instead of traditional button or joystick commands, the postures and movements of User A of game controllers 842 and 903 and HMD 905 are translated into inputs such as input signals A1 or A2 to game engine A ( Figure 5 ). Optionally, other wireless-enabled peripheral devices, such as Playstation TM portable devices, etc. are used as controllers. For Playstation TM portable devices, additional game or control information, such as control instructions or number of lives, etc. is provided on the display screen of the device. In some embodiments, other alternative or auxiliary control devices are used, such as a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown), a custom controller, etc. Examples of custom controllers include one or several large buttons (also not shown) for quick response quiz games.

[0117] Remote control 904 is also operable to communicate wirelessly with game console 900 via link 918. Remote control 904 includes controls that are suitable for the operation of Blu Ray TM optical disc BD-ROM reader 940 and the navigation of the optical disc content.

[0118] In addition to traditional pre-recorded and recordable CDs and the so-called Super Audio CDs, Blu Ray TMThe optical disc BD-ROM reader 940 is operable to read CD-ROMs compatible with the game console 900. In addition to traditional pre-recorded and recordable DVDs, Blu Ray TM the optical disc BD-ROM reader 940 is also operable to read digital video disc-ROMs (DVD-ROMs) compatible with the game console 900. Blu Ray TM the optical disc BD-ROM reader 940 is further operable to read BD-ROMs compatible with the game console 900, as well as traditional pre-recorded and recordable Blu-ray discs.

[0119] The game console 900 is operable to provide audio and video generated or decoded via the Reality Synthesizer Graphics Unit 930 to a display and sound output device 942, such as a monitor or a television, through an audio connector 950 and a video connector 952. The display and sound output device has a display screen 944 and one or more speakers 946, or via a wireless link port 918 to the display device of the HMD 905. The sound output device 942 is an example of the display device of client device A or B. In various embodiments, the audio connector 950 includes traditional analog and digital outputs, while, differently, the video connector 952 includes component video, S-video, composite video, and one or more high-definition multimedia interface (HDMI) outputs. Thus, the video output can be in formats such as Phase Alternating Line (PAL) or National Television System Committee (NTSC), or in high-definition formats such as 2220p, 1080i, or 1080p. Audio processing such as generation, decoding, etc. is performed by the unit processor 908. The operating system of the game console 900 supports 5.1 surround sound, theater surround sound (DTS), and the decoding of 7.1 surround sound from optical discs.

[0120] In some embodiments, a camera, such as the camera 912, includes a single charge-coupled device (CCD), an LED indicator, and a hardware-based real-time data compression and encoding device, such that the compressed video data is transmitted in a suitable format, such as based on the Moving Picture Experts Group (MPEG) standard within the image, for decoding by the game console 900. The LED indicator of the camera 912 is arranged to be illuminated in response to appropriate control data from the game console 900, for example to indicate adverse lighting conditions, etc. Some embodiments of the camera 912 are connected via USB in various ways, or Wi-Fi TMThe communication port is connected to the game console 900. Various embodiments of the camera include one or more associated microphones and are also capable of transmitting audio data. In several embodiments of the camera, the CCD has a resolution suitable for high-definition video capture. In use, the images captured by the camera are incorporated into the game or interpreted as game control inputs. In another embodiment, the camera is an infrared camera suitable for detecting infrared light.

[0121] In various embodiments, suitable software such as device drivers is provided to successfully communicate data with a peripheral device such as a camera or a remote control via one of the communication ports in the game console 900.

[0122] In some embodiments, the aforementioned system devices including the game console 900, the HHC, and the HMD 905 enable the HMD 905 to display and capture video of an interactive session of the game. The system devices facilitate the execution of the game engine A to initiate an interactive session of the game, and the interactive session defines the interactivity between user A and the game. The system devices further determine the initial position and orientation of the HHC and / or the HMD 905 operated by user A. The game console 900 determines the current state of the game based on the interactivity between user A and the game. The system devices track the position and orientation of the HHC and / or the HMD 905 during the interactive session of user A with the game. The system devices generate an audience video stream 905 of the interactive session based on the current state of the game and the tracked position and orientation of the HHC and / or the HMD. In some embodiments, the HHC renders the audience video stream on the display screen of the HHC or the display device of client device A. In various embodiments, the HMD 905 renders the audience video stream on the display screen of the HMD 905.

[0123] Reference Figure 10 , a diagram showing the components of the HMD 1002 is shown. The HMD 1002 is an example of the HMD 905 ( Figure 9 ). The HMD 1002 includes a processor 1000 for executing program instructions. A memory device 1002 is provided for storage. Examples of the memory device 1002 include volatile memory, non-volatile memory, or a combination thereof. The included display device 1004 provides a visual interface, such as displaying image frames generated from saved data, etc., that user A ( Figure 1 ) views. A battery 1006 is provided as the power source for the HMD 1002. The motion detection circuit 1008 includes any one of various motion sensing hardware, such as a magnetometer 1010, an accelerometer 1012, and a gyroscope 1014.

[0124] An accelerometer is a device used to measure acceleration and the reaction force of gravity sensing. Uniaxial and multi - axis models can be used to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense inclination, vibration, and shock. In one embodiment, three accelerometers 1012 are used to provide the direction of gravity, and the direction of gravity gives an absolute reference for two angles such as world - space pitch and world - space roll.

[0125] A magnetometer measures the strength and direction of the magnetic field near the HMD 1002. In some embodiments, three magnetometers 1010 are used in the HMD 1002 to ensure an absolute reference for the world - space yaw angle. In various embodiments, the magnetometer is designed to span the Earth's magnetic field of ±80 microteslas. The magnetometer is affected by metal and provides a yaw measurement that does not change with the actual yaw. In some embodiments, due to metal in the real - world environment, the magnetic field is distorted, which results in a distorted yaw measurement. In various embodiments, information from other sensors (e.g., gyroscopes 1014, cameras 1016, etc.) is used to calibrate this distortion. In one embodiment, the accelerometer 1012 is used together with the magnetometer 1010 to obtain the tilt and azimuth angles of the HMD 1002.

[0126] A gyroscope is a device used to measure or maintain orientation based on the principle of angular momentum. In one embodiment, instead of a single gyroscope 1014, three gyroscopes provide information about movement across the respective axes (x, y, and z) based on inertial sensing. Gyroscopes help detect rapid rotation. However, in some embodiments, the gyroscope drifts over time in the absence of an absolute reference. This triggers a periodic reset of the gyroscope, which can be done using other available information about position / orientation determined by, for example, object - based visual tracking, accelerometers, magnetometers, etc.

[0127] A camera 1016 is provided for capturing images and image streams of the real - world environment such as a room, a cabin, a natural environment, etc. around user A. In various embodiments, more than one camera is included in the HMD 1002, including a rear - facing camera, e.g., a camera facing away from user A when user A is viewing a display of the HMD 1002, etc., and a front - facing camera, e.g., a camera facing user A when user A is viewing a display of the HMD 1002, etc. Additionally, in several embodiments, a depth camera 1018 is included in the HMD 1002 for sensing depth information of objects in the real - world environment.

[0128] The HMD 1002 includes a speaker 1020 for providing audio output. Additionally, in some embodiments, the included microphone 1022 is used to capture audio from the real-world environment, including sounds from the surrounding environment and the speech of user A, etc. The HMD 1002 includes a haptic feedback circuit 1024, such as a vibration device, to provide haptic feedback to user A. In one embodiment, the haptic feedback circuit 1024 is capable of causing the movement and / or vibration of the HMD 1002 to provide haptic feedback to user A.

[0129] An LED 1026 is provided as a visual indicator of the status of the HMD 1002. For example, the LED can indicate battery level, power-on, etc. A card reader 1028 is provided to enable the HMD 1002 to read information from and write information to a memory card. The included USB interface 1030 is an example of an interface for implementing peripheral device connections or connections to other devices such as other portable devices, computers, etc. In various embodiments of the HMD 1002, any one of various interfaces can be included to achieve stronger connectivity of the HMD 1002.

[0130] Includes WiFi TM circuit 1032 for connecting to the Internet via wireless networking technology. Additionally, the HMD 1002 includes Bluetooth TM circuit 1034 for enabling wireless connections to other devices. In some embodiments, a communication link 1036 is included for connecting to other devices. In one embodiment, the communication link 1036 uses infrared transmission for wireless communication. In other embodiments, the communication link 1036 uses any one of various wireless or wired transmission protocols to communicate with other devices.

[0131] Includes input buttons / sensors 1038 for providing an input interface to user A ( Figure 1 )). Any one of various types of input interfaces, such as buttons, touchpads, joysticks, trackballs, etc. is included. In various embodiments, an ultrasonic communication circuit 1040 is included in the HMD 1002 for facilitating communication with other devices via ultrasonic technology.

[0132] Includes a biosensor 1042 to enable the detection of physiological data from user A or B. In one embodiment, the biosensor 1042 includes one or more dry electrodes for detecting the bioelectrical signals of user A or B through the user's skin.

[0133] The foregoing components of the HMD 1002 are described only as exemplary components that may be included in the HMD 1002. In various embodiments, the HMD 1002 includes or does not include some of the various foregoing components.

[0134] Figure 11 An embodiment of an Information Service Provider (INSP) architecture is shown. The INSP 1102 provides a variety of information services to users A, B, C, and D who are geographically dispersed and connected via a computer network 1106 (such as a LAN, WAN, or a combination thereof, etc.). The computer network 1106 is an example of the computer network 820( Figure 8 ). Examples of WANs include the Internet, and examples of LANs include intranets. User A operates a client device 1120-1, user B operates another client device 1120-2, user C operates yet another client device 1120-3, and user D operates yet another client device 1120-4.

[0135] In some embodiments, each of the client devices 1120-1, 1120-2, 1120-3, and 1120-4 includes a central processing unit (CPU), a display, and an input / output (I / O) interface. Examples of each of the client devices 1120-1, 1120-2, 1120-3, and 1120-4 include a personal computer (PC), a mobile phone, a netbook, a tablet computer, a gaming system, a personal digital assistant (PDA), a gaming console 900( Figure 9 ), and a display device, an HMD 1202( Figure 11 ), a gaming console 900, and an HMD 1002( Figure 10 ), a desktop computer, a laptop computer, a smart TV, etc. Each of the client devices 1120-1, 1120-2, 1120-3, and 1120-4 is an example of a client device A( Figure 7 ) or a client device B( Figure 7 ). In some embodiments, the INSP 1102 identifies the type of the client device and adjusts the communication method employed.

[0136] In some embodiments, an INSP provides one type of service, such as updating stock prices, or provides multiple types of services, such as broadcast media, news, sports, games, etc. In addition, the services provided by each INSP are dynamic, that is, services can be added or removed at any point in time. Therefore, the INSP that provides a specific type of service to a specific individual can change over time. For example, when the client device 1120-1 is in the hometown of user A, the client device 1120-1 is served by an INSP near the client device 1120-1, and when user A travels to a different city, the client device 1120-1 is served by a different INSP. The INSP in the hometown will transmit the requested information and data to the new INSP, so that the information "follows" the client device 1120-1 to the new city, making the data closer to the client device 1120-1 and easier to access. In various embodiments, a master-server relationship is established between the master INSP that manages the information of the client device 1120-1 and the server INSP that directly interacts with the client device 1120-1 under the control of the master INSP. In some embodiments, when the client device 1120-1 moves around the world, data is transferred from one ISP to another ISP so that the INSP is in a better position to serve the client device 1120-1 that delivers these services.

[0137] INSP 1102 includes an Application Service Provider (ASP) 1108 that provides computer-based services to customers via a computer network 1106. Software provided using the ASP model is sometimes also referred to as on-demand software or Software as a Service (SaaS). A simple form of providing access to computer-based services (such as customer relationship management, etc.) is through the use of standard protocols such as the Hypertext Transfer Protocol (HTTP). The application software resides on the vendor's server and is accessed by each of the client devices 1120-1, 1120-2, 1120-3, and 1120-4 through a web browser using, for example, Hypertext Markup Language (HTML), through dedicated client software provided by the vendor, and / or other remote interfaces such as thin clients.

[0138] Services delivered over a wide geographic area often use cloud computing. Cloud computing is a style of computing in which dynamically scalable and usually virtualized resources are provided as a service over a computer network 1106. Users A, B, C, and D do not need to be experts in the technical infrastructure in the "cloud" that supports them. In some implementations, cloud computing is divided into different services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services typically provide general business applications accessible from a web browser online, while the software and data are stored on servers. The term cloud is used as a metaphor for the computer network 1106, for example using servers, storage, and logic as depicted in a computer network diagram based on the computer network 1106, and is an abstract concept of the complex infrastructure it hides.

[0139] Further, INSP 1102 includes a Game Processing Provider (GPP) 1110, sometimes also referred to herein as a game processing server, which is used by client devices 1120-1, 1120-2, 1120-3, and 1120-4 to play single and multi-player video games. Most video games played over the computer network 1106 run via a connection to a game server. Typically, games use a dedicated server application that collects data from client devices 1120-1, 1120-2, 1120-3, and 1120-4 and distributes the data to other clients operated by other users. This approach is more efficient and effective compared to peer-to-peer arrangements, but uses a separate server to host the server application. In some implementations, GPP 1110 establishes communication between client devices 1120-1, 1120-2, 1120-3, and 1120-4, and these client devices exchange information without further relying on the centralized GPP 1110.

[0140] A dedicated GPP is a server that runs independently of the clients. Such servers typically run on dedicated hardware located in a data center, thus providing more bandwidth and dedicated processing power. For most PC-based multi-player games, a dedicated server is the method of hosting the game server. Large multi-player online games run on dedicated servers, which are typically hosted by the software company that owns the game name, thus allowing them to control and update the content.

[0141] A Broadcast Processing Server (BPS) 1112, sometimes referred to herein as a broadcast processing provider, distributes audio or video signals to viewers. Broadcasting to a very small group of viewers is sometimes referred to as narrowcasting. The last leg of the broadcast distribution is the manner in which the signal reaches the client devices 1120-1, 1120-2, 1120-3, and 1120-4, and in some embodiments, the signal is distributed radio-wise to an antenna and receiver by a radio or television station, or distributed via cable television or cable radio or via the station's "wireless cable". In various embodiments, especially through multicasting which allows sharing of signals and bandwidth, a computer network 1106 also brings radio or television signals to the client devices 1120-1, 1120-2, 1120-3, and 1120-4. Historically, in several embodiments, broadcasting was defined by geographical regions, e.g., national broadcasting, local broadcasting, etc. However, with the popularity of high-speed Internet, since content can reach almost any country in the world, broadcasting is no longer restricted by geographical location.

[0142] A Storage Service Provider (SSP) 1114 provides computer storage space and related management services. The SSP 1114 also provides regular backup and archiving. By providing storage as a service, the client devices 1120-1, 1120-2, 1120-3, and 1120-4 use more storage than when storage is not used as a service. Another major advantage is that the SSP 1114 includes a backup service, and the client devices 1120-1, 1120-2, 1120-3, and 1120-4 do not lose data when their hard drives fail. Further, in some embodiments, multiple SSPs have all or partial copies of the data received from the client devices 1120-1, 1120-2, 1120-3, and 1120-4, thus allowing the client devices 1120-1, 1120-2, 1120-3, and 1120-4 to access data in an efficient manner regardless of the location of the client devices 1120-1, 1120-2, 1120-3, and 1120-4 and the type of the client. For example, when user A is on the move, user A accesses personal files via a home computer as well as a mobile phone.

[0143] Communication provider 1116 provides connections to client devices 1120-1, 1120-2, 1120-3, and 1120-4. One type of communication provider 1116 is an Internet service provider (ISP) that provides access to computer network 1106. The ISP uses data transmission technologies suitable for delivering Internet protocol datagrams, such as dial-up, digital subscriber line (DSL), cable modem, fiber optic, wireless, or dedicated high-speed interconnections, to connect client devices 1120-1, 1120-2, 1120-3, and 1120-4. In some embodiments, communication provider 1116 also provides messaging services such as email, instant messaging, and short message service (SMS) text. Another type of communication provider is a network service provider (NSP) that sells bandwidth or network access by providing direct backbone access to computer network 1106. Examples of network service providers include telecommunications companies, data carriers, wireless communication providers, Internet service providers, cable television operators that provide high-speed Internet access, and the like.

[0144] Data switch 1118 interconnects several circuits within INSP 1102 and connects these circuits to client devices 1120-1, 1120-2, 1120-3, and 1120-4 via computer network 1106. In various embodiments, data switch 1118 covers a small area where all circuits of INSP 1102 are aggregated, or a large geographical area when different circuits are geographically dispersed. For example, data switch 1102 includes fast gigabit Ethernet in a data center cabinet, or an intercontinental virtual LAN.

[0145] In some embodiments, wireless technologies can be used to facilitate communication between nodes A and B and client devices A and B. Such technologies can include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which the service area covered by a provider is divided into small geographical areas called cells. Analog signals representing sound and images are digitized in a telephone, converted into a bit stream by an analog-to-digital converter, and transmitted. All 5G wireless devices in a cell communicate with a local antenna array and a low-power transceiver (transmitter and receiver) in the cell via radio waves through a frequency channel assigned by the transceiver from a frequency pool reused in other cells. The local antenna is connected to the telephone network and the Internet via a high-bandwidth fiber optic or wireless backhaul connection. As in other cell networks, a mobile device crossing from one cell to another is automatically transferred to the new cell. It should be understood that the 5G network is just an example type of communication network, and embodiments of the present invention can utilize earlier generations of wireless or wired communication, as well as newer generations of wired or wireless technologies after 5G.

[0146] It should be noted that, in various embodiments, one or more features of some of the embodiments described herein are combined with one or more features of one or more other embodiments described herein.

[0147] The embodiments described in this disclosure can be implemented using a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor - or programmable - based consumer electronics, minicomputers, mainframe computers, and the like. In one implementation, the embodiments described in this disclosure are implemented in a distributed computing environment where tasks are performed by remote processing devices linked through a wired or wireless network.

[0148] In view of the above embodiments, it should be understood that the embodiments described in this disclosure can employ various computer - implemented operations involving data stored in a computer system. These operations require physically manipulating physical quantities. Any operations forming part of the embodiments described in this disclosure are useful machine operations. Some of the embodiments described in this disclosure also relate to apparatuses or devices for performing these operations. The devices are specially constructed for the required purposes or are general - purpose computers selectively activated or configured by a computer program stored in the computer. Specifically, in one embodiment, it may be more convenient to use various general - purpose machines in conjunction with computer programs written according to the content herein, or to construct more specialized devices to perform the required operations.

[0149] In one implementation, some of the embodiments described in this disclosure are embodied as computer - readable code on a computer - readable medium. A computer - readable medium is any data storage device that stores data, which is then read by a computer system. Examples of computer - readable media include hard disk drives, network - attached storage (NAS), ROM, RAM, compact disc read - only memory (CD - ROM), recordable CD (CD - R), rewritable CD (CD - RW), magnetic tape, optical data storage devices, non - optical data storage devices, and the like. For example, the computer - readable medium includes computer - readable tangible media distributed on network - coupled computer systems such that the computer - readable code is stored and executed in a distributed manner.

[0150] In addition, although some of the above embodiments are described with respect to a gaming environment, in some embodiments, other environments such as video conferencing environments, etc., are used instead of gaming.

[0151] Although method operations are described in a specific order, it should be understood that other arrangement operations can be performed between the operations, or, as long as the processing of the covered operations is performed in the desired manner, the operations can be adjusted so that they occur at slightly different times, or can be distributed in a system that allows processing operations at various intervals associated with the processing.

[0152] Although the foregoing embodiments in the present disclosure have been described in some detail for the sake of clear understanding, it will be apparent that certain changes and modifications can be made within the scope of the appended claims. Accordingly, these embodiments are to be considered illustrative rather than restrictive, and the embodiments are not limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method for adjusting the complexity of content rendered by a Graphics Processing Unit (GPU), which includes: receiving, by the GPU, an instruction from a Central Processing Unit (CPU) to generate an image frame of a game scene; generating, in response to receiving the instruction from the CPU, a first rendered portion of the image frame by the GPU; During the generation of the first rendered portion of the image frame and prior to the generation of the second rendered portion of the image frame, track one or more metrics regarding the image frame Processed one or more metrics; during the generation of the first rendered portion of the image frame and before the generation of the second rendered portion of the image frame, determining one or more shader parameters that will modify the second rendered portion of the image frame, wherein based on the one or more metrics, determining the one or more shader parameters that will modify the second rendered portion of the image frame; and during the generation of the second rendered portion of the image frame, modifying, by the shader, the one or more shader parameters of the second rendered portion of the image frame, wherein the one or more shader parameters of the second rendered portion are modified without modifying one or more shader parameters of the first rendered portion of the image frame, and wherein modifying the one or more shader parameters of the second rendered portion changes the complexity level of the image frame being generated by the GPU.

2. The method according to claim 1, wherein during the generation of the first rendered portion of the image frame, tracking the one or more metrics to identify the amount of power consumed by the GPU, the method further includes: analyzing the amount of power consumed by the GPU during the generation of the first rendered portion of the image frame to determine whether the amount of power exceeds a predetermined threshold level, wherein modifying the one or more shader parameters of the second rendered portion occurs when it is determined that the amount of power exceeds the predetermined threshold level.

3. The method according to claim 1, wherein the one or more shader parameters of the second rendered portion include: the number of times of performing ray tracing during the generation of the second rendered portion of the image frame, or the resolution of the second rendered portion of the image frame during the generation of the second rendered portion of the image frame, or the number of virtual objects rendered during the generation of the second rendered portion of the image frame, or the priority order of rendering the virtual objects during the generation of the second rendered portion of the image frame, or a combination of two or more of them.

4. The method according to claim 3, wherein the priority order is based on the distance at which the virtual object will be rendered, and wherein one virtual object that will be rendered at a closer distance among the virtual objects has a higher priority than another virtual object that will be rendered at a farther distance among the virtual objects.

5. The method according to claim 1, wherein modifying the one or more shader parameters includes reducing the complexity level by: reducing the number of times of performing ray tracing during the generation of the second rendered portion of the image frame when it is determined that the amount of power consumed by the GPU is greater than a predetermined level ； or When it is determined that the amount of power consumed by the GPU is greater than the predetermined level, reduce the resolution of the second rendering portion of the image frame; Or When it is determined that the amount of power consumed by the GPU is greater than the predetermined level, reduce the number of virtual objects rendered within the second rendering portion of the image frame; or When it is determined that the amount of power consumed by the GPU is greater than the predetermined level, prioritize the rendering of the virtual objects; or A combination of two or more of them.

6. The method according to claim 1, wherein one or more metrics are tracked to identify the amount of latency in transmitting packets between the node and the client device during the generation of the first rendering portion of the image frame, and the method further Comprises: During the generation of the first rendering portion of the image frame, analyze the amount of latency to determine whether the amount of latency exceeds a predetermined threshold level, wherein the modification of the one or more shader parameters of the second rendering portion occurs when it is determined that the amount of latency exceeds the predetermined threshold level.

7. The method according to claim 1, wherein the modification of the one or more shader parameters of the second rendering portion includes reducing the complexity level by: When it is determined that the amount of latency in transmitting packets between the node and the client device is greater than a predetermined level, reduce the number of times ray tracing is performed within the second rendering portion of the image frame ； Or When it is determined that the amount of latency is greater than the predetermined level, reduce the resolution of the second rendering portion of the image frame; Or When it is determined that the amount of latency is greater than the predetermined level, reduce the number of virtual objects rendered within the second rendering portion of the image frame; Or When it is determined that the amount of latency is greater than the predetermined level, prioritize the rendering of the virtual objects within the second rendering portion of the image frame; or A combination of two or more of them.

8. The method according to claim 1, wherein one or more metrics are tracked to identify the temperature amount of the GPU during the generation of the first rendering portion of the image frame, and the method further Comprises: During the generation of the first rendering portion of the image frame, analyze the temperature amount to determine whether the temperature amount exceeds a predetermined threshold level, wherein the modification of the one or more shader parameters of the second rendering portion occurs when it is determined that the temperature amount exceeds the predetermined threshold level.

9. The method according to claim 1, wherein the modification of the one or more shader parameters of the second rendering portion includes reducing the complexity level by: When it is determined that the temperature amount of the GPU is greater than a predetermined level during the generation of the first rendering portion of the image frame, reduce the number of times ray tracing is performed within the second rendering portion of the image frame ； Or When it is determined that the temperature amount is greater than the predetermined level, reduce the resolution of the second rendering portion of the image frame; Or When it is determined that the temperature amount is greater than the predetermined level, reduce the number of virtual objects rendered within the second rendering portion of the image frame; Or When it is determined that the temperature amount is greater than the predetermined level, prioritize the rendering of the virtual objects within the second rendering portion of the image frame; or A combination of two or more of them.

10. The method according to claim 1, wherein modifying the one or more shader parameters includes reducing the complexity level by: Reducing the number of times ray tracing is performed within the second rendering portion of the image frame; or Reducing the resolution of the second rendering portion of the image frame; or Reducing the number of virtual objects rendered within the second rendering portion of the image frame; or Prioritizing the rendering of the virtual objects within the second rendering portion of the image frame; or A combination of two or more of them.

11. The method according to claim 1, wherein the second rendering portion of the image frame includes a plurality of pixels, and color, shading, and texture are provided to each of the pixels during the generation of the second rendering portion of the image frame.

12. The method according to claim 1, wherein during the generation of the first rendering portion of the image frame, one or more metrics are tracked to determine whether user input is received via a computer network, and the method further includes: Analyzing the user input to determine whether to modify the one or more shader parameters of the second rendering portion, wherein analyzing the user input includes determining whether to generate a next image frame based on the user input, and when it is determined that a next image frame is to be generated, performing the modification of the one or more shader parameters of the second rendering portion.

13. The method according to claim 1, wherein in response to a quality adjuster signal QAS received from a processor, the modification of the one or more shader parameters of the second rendering portion of the image frame is performed, wherein the GPU is coupled to the processor.

14. A server for adjusting the complexity of content rendered by a graphics processing unit GPU, which includes: The GPU, which is configured to receive instructions from a central processing unit (CPU) to generate image frames of a game scene, wherein the GPU is configured to track during the generation of a first rendering portion of the image frame and prior to the generation of a second rendering portion of the image frame Regarding the processing of the image frame one or more metrics Wherein, the GPU is configured to determine, during the generation of the first rendering portion of the image frame and before the generation of the second rendering portion of the image frame, that one or more shader parameters of the second rendering portion of the image frame are to be modified, wherein it is determined that the one or more shader parameters of the second rendering portion of the image frame are to be modified based on the one or more metrics. The GPU is configured to modify one or more shader parameters of the second rendering portion of the image frame, wherein the one or more shader parameters of the second rendering portion of the image frame are modified without modifying one or more shader parameters of the first rendering portion of the image frame, and wherein the modification of the one or more shader parameters of the second rendering portion of the image frame changes the complexity level of the image frame.

15. The server according to claim 14, wherein the second rendering portion of the image frame includes a plurality of pixels, and wherein color, shading, and texture are provided to each of the pixels during the generation of the second rendering portion of the image frame. Wherein when the first rendering portion of the image frame is being generated, the processing unit is configured to: Track the one or more metrics to identify the amount of power consumed by the GPU; Analyze the amount of power consumed by the GPU to determine whether the amount of power exceeds a predetermined threshold level, wherein the GPU is configured to modify the one or more shader parameters of the second rendering portion when it is determined that the amount of power exceeds the predetermined threshold level.

16. The server according to claim 14, wherein the one or more shader parameters of the second rendering portion of the image frame include the number of times ray tracing is performed during the generation of the second rendering portion of the image frame, or the resolution of the second rendering portion of the image frame, or the number of virtual objects rendered during the generation of the second rendering portion of the image frame, or the priority order of rendering the virtual objects during the processing of the image frame, or a combination of two or more of them.

17. The server according to claim 14, wherein when the second rendering portion of the image frame is being processed by the GPU, in order to modify the one or more shader parameters of the second rendering portion, the shader is configured to reduce the complexity level by: Reducing the number of times ray tracing is performed within the second rendering portion of the image frame; or Reducing the resolution of the second rendering portion of the image frame; or Reducing the number of virtual objects rendered within the second rendering portion of the image frame; or Prioritizing the rendering of a plurality of virtual objects within the image frame; or A combination of two or more of them.

18. A system for adjusting the complexity of content rendered by a graphics processing unit (GPU), which comprises: A central processing unit (CPU); And The GPU, which is configured to receive instructions from the CPU to generate image frames of a game scene, wherein the GPU is configured to execute a shader to generate a first rendered portion of the image frame in response to receiving the instructions, and wherein the GPU is configured to track Regarding the processing of the image frame one or more metrics during the generation of the first rendered portion of the image frame and prior to the generation of a second rendered portion of the image frame, and wherein the GPU is configured to determine, during the generation of the first rendered portion of the image frame and prior to the generation of the second rendered portion of the image frame, one or more shader parameters that will modify the second rendered portion of the image frame, and wherein, based on the one or more metrics, the one or more shader parameters that will modify the second rendered portion of the image frame are determined, and wherein the CPU is configured to generate a quality adjustment signal QAS and send the QAS to the GPU based on the determination that the one or more shader parameters will be modified, and wherein the QAS is sent to the GPU while the first rendered portion of the image frame is being generated and prior to the second rendered portion of the image frame being generated, and Wherein the GPU is configured to modify one or more shader parameters of the second rendering portion of the image frame upon receiving the QAS, wherein the one or more shader parameters of the second rendering portion of the image frame are modified without modifying one or more shader parameters of the first rendering portion of the image frame, and wherein the modification of the one or more shader parameters of the second rendering portion changes a complexity level of the image frame being generated by the GPU.

19. The system of claim 18, wherein the CPU is configured to determine whether a user input is received via a computer network during the generation of the first rendering portion of the image frame, and wherein the CPU is configured to analyze the user input to determine whether to generate the quality adjuster signal.

20. The system of claim 18, wherein, in order to modify the one or more shader parameters while the second rendering portion of the image frame is being generated by the GPU, the GPU is configured to reduce the complexity level by: Reducing the number of times ray tracing is performed on the second rendering portion of the image frame; or Reducing the resolution of the second rendering portion within the image frame; or Reducing the number of virtual objects rendered within the second rendering portion of the image frame; or Prioritizing the rendering of multiple virtual objects within the second rendering portion of the image frame; or A combination of two or more thereof.

Images