Dead Reckoning and Latency Improvements in 3D Game Streaming Scenarios

Abstract

The examples described herein generally relate to systems and methods for streaming video games at a client device. The client device may transmit video game controls to a streaming server. The client device may receive a video stream from the streaming server, the video stream encoding video images generated in response to the video game controls. The client device may determine that a video image of the video stream to be displayed in a frame has not been fully received at a specified time prior to display of the frame. The client device may determine an image transform based on a history of the video image and a motion vector for the video stream. The client device may apply an image transform to a portion of one or more images corresponding to a previous frame. The client device may display an alternative video image in a frame that includes a portion of the transformed image.

Description

Background Art

[0001] The present disclosure relates to streaming video games, and more particularly to mitigating latency in streaming video games.

[0002] A streaming video game may be defined as a video game in which a client device provides control commands to a server running the game via a network, the server acts on the control commands to generate a video stream, and the video stream is sent to the client device via the network for display on the client device's screen. A streaming video game may differ from a more conventional video game in which the client device draws the video images presented to the user even though the server performs some processing. For example, in a multiplayer game, the server may calculate the interactions between the players, but the server typically sends scene information to the client device to draw the video images. Compared to running the game on the client device, a streaming video game may provide the advantage of centralized game processing and allow for thinner, less complex client devices and less complex client software. For example, a streaming video game may be suitable for mobile devices that may lack a dedicated graphics processing card.

[0003] Streaming video games may encounter delays and delay variations (jitter) in the communication network used to stream the video. That is, due to network delays, the client device may not receive the video packet at the expected time. Due to jitter, the actual time of receiving the video packet may be difficult to predict. In particular, although wireless networks provide sufficient bandwidth to stream video, wireless networks may experience greater delays and jitter than wired networks with similar bandwidth.

[0004] Unlike conventional streaming video (e.g., movies), buffering may not be suitable for streaming video games because the video is based on user input, so the video images to be rendered in future frames are not available in advance. In addition, unlike conventional multi-player games where client devices run parallel game engines, client devices for streaming games may not have information about the current game state. Therefore, the client device may freeze when no new video images are received for display.

[0005] Therefore, there is a need in the art for improved graphics processing for video game streaming on computer devices. Summary of the invention

[0006] A simplified overview of one or more implementations of the present disclosure is given below to provide a basic understanding of such implementations. This overview is not an extensive overview of all contemplated implementations, and is neither intended to identify key or important elements of all implementations nor to define the scope of any or all implementations. Its sole purpose is to present some concepts of one or more implementations of the present disclosure in a simplified form as a prelude to a more detailed description presented later.

[0007] One example implementation relates to a computer device for playing a streaming video game. The computer device may include: a memory storing data and instructions; a central processing unit in communication with the memory; a network interface communicatively coupled to a communication network; a client application; and a display control component. The computer device may be operable to transmit video game controls to a streaming server via the network interface. The computer device may be operable to receive a video stream from the streaming server via the network interface, the video stream encoding video images generated in response to the video game controls. The computer device may be operable to determine that a video image of the video stream to be displayed in a frame has not been fully received at a specified time before the display of the frame. The computer device may be operable to determine an image transformation based at least on a history of the video image and a motion vector for the video stream. The computer device may be operable to apply the image transformation to at least a portion of one or more images corresponding to one or more previous frames to generate a substitute video image. The computer device may be operable to display a substitute video image in the frame including at least a portion of the transformed image in response to a determination that the video image to be displayed in the frame has not been fully received at the specified time.

[0008] Another example implementation relates to a method for playing a streaming video game on a computer device. The method may include transmitting video game control commands to a streaming server. The method may include receiving a video stream from a streaming server, the video stream encoding video images generated in response to video game controls. The method may include determining that a video image of a video stream to be displayed in a frame has not been fully received at a specified time prior to display of the frame. The method may include determining an image transform based at least on a history of the video image and a motion vector for the video stream. The method may include applying the image transform to at least a portion of one or more images corresponding to one or more previous frames to generate a substitute video image. The method may include displaying a substitute video image in a frame that includes at least a portion of a previously displayed image that has been transformed, in response to a determination that a video image to be displayed in a frame has not been fully received at a specified time.

[0009] Another example implementation relates to a computer-readable medium including code executable by one or more processors for playing a streaming video game on a computing device. The computer-readable medium may include code for transmitting video game controls to a streaming server. The computer-readable medium may include code for receiving a video stream from a streaming server, the video stream encoding a video image generated in response to the video game controls. The computer-readable medium may include code for determining that a video image of a video stream to be displayed in a frame has not been fully received at a specified time before the display of the frame. The computer-readable medium may include code for determining image reprojection based on at least an image of the video stream corresponding to a previous frame and a motion vector selected based on a camera transformation matrix for the previous frame. The computer-readable medium may include code for applying image reprojection to at least a portion of one or more images of a video stream corresponding to one or more previous frames to generate a substitute video image. The computer-readable medium may include code for displaying a substitute video image in a frame including at least a portion of the reprojected image in response to a determination that the video image to be displayed in the frame has not been fully received at the specified time.

[0010] Additional advantages and novel features associated with implementations of the present disclosure will be set forth in part in the following description and will in part become apparent to those skilled in the art upon examination of the following or through practice and study. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the attached picture:

[0012] Figure 1 is a schematic block diagram of an example system for playing a streaming video game according to an implementation of the present disclosure;

[0013] Figure 2 is a diagram illustrating processes and communications for playing a streaming video game according to an implementation of the present disclosure;

[0014] Figure 3 is an example of a video stream of a streaming video game including substitute video frames according to an implementation of the present disclosure;

[0015] Figure 4 is a flow chart of a method for playing a streaming video game according to an implementation of the present disclosure; and

[0016] Figure 5 is a schematic block diagram of an example computer device according to implementations of the present disclosure. DETAILED DESCRIPTION

[0017] The present disclosure provides a system and method for streaming video games. As used herein, the term "streaming video game" refers to an arrangement in which a streaming server executes a video game, generates an image, and sends a video stream encoding the image to a streaming client device. The streaming server can generate an image by performing a drawing process on the streaming client device. However, the streaming server may not actually display the generated image on a screen or monitor. Instead, the streaming server can encode the image into a video stream. The video stream can encode an image corresponding to a video frame. A frame can be a continuous time period for displaying an image. The encoding can include an image and / or indicate change information (e.g., a motion vector) of the image corresponding to each frame. For network communication, the encoded video stream can be divided into a plurality of video packets, each of which includes a portion of the video stream. The video packet may be lost in transmission, may arrive out of order, or may be delayed so that when the video packet arrives, the time for displaying the corresponding frame has passed.

[0018] For example, in one implementation, the present disclosure provides systems and methods for reducing the impact of network latency on a user experience at a streaming client device. Specifically, the present disclosure addresses the problem of a streaming client device not receiving video packets in time to present an image corresponding to a frame on a display of the streaming client device. The present disclosure provides a dead reckoning method in which, when one or more video packets are not received in time, a substitute image is generated based on a history of generated images and motion vectors for a video stream.

[0019] The stream client device may determine that at least a portion of the video data (e.g., one or more video packets) used to display a video frame has not been received at a time before the video frame is displayed. The stream client device may determine an image transformation based on a history of the video frame and a motion vector for the video stream. The stream client may apply the image transformation to at least a portion of one or more previous frames to generate a substitute video frame. If the video data has not yet arrived at the display time, the stream client may display a substitute video frame instead of the unreceived video frame. Therefore, when no new video frame is available, instead of freezing the image on the display, the stream client may predict what the lost video frame may look like as a substitute image. Although the predicted substitute image is unlikely to be an exact copy of the lost image, the substitute can improve the user experience by filling the video frame to smooth the motion. Once the stream client receives the current video packet, the stream client may display the video stream from the server in the correct frame based on the current video packet.

[0020] Determining the image transformation can be performed by the operating system of the streaming device with or without the help of the video game client application. Without the help of the video game client application, the operating system can analyze the video stream to determine the image transformation. For example, the video stream can be encoded to include images and / or motion vectors indicating the transformation of certain pixels between images. The operating system can analyze the images to identify objects and estimate the depth of the objects. The operating system can then extrapolate the motion vector from the image of the previous frame to the current frame corresponding to the image that was not received to determine the image transformation.

[0021] With the help of the video game client application and / or the streaming server, the operating system can use known information about the image to improve the image transformation. For example, the video game can provide a camera matrix indicating the pose of the virtual camera and the change of the pose. Since the camera pose can be the main source of motion between video frames, the camera matrix can be used to more accurately predict the image transformation. As another example, the video game can provide some information about the geometry of the video game image. For example, the video game can identify the location of the object. The operating system can cache the object texture and apply the cached texture to the object after the image transformation.

[0022] Reference now Figure 1 , the example computer system 100 may include a computer device 110 and a streaming server 160, the computer device 110 providing a video image for presentation in a corresponding video frame on a display 116, and the streaming server 160 streaming a video game to the computer device 110. The streaming server 160 may include a game engine 162, which processes various inputs (e.g., control commands 158) from one or more users and updates a game state. The game state may refer to the state of any feature of a video game, such as an environment, a character, an object, or its attributes. The streaming server 160 may also include a video encoder 164, which generates and encodes a video stream 166 for the computer device 110. The video stream 166 may encode images of information generated for a specific user, for example, based on the perspective of the user's character, or including applicable elements of the game state for the user. The streaming server 160 may be implemented, for example, as one or more hardware computer servers, or one or more virtual servers provided by a cloud service.

[0023] Computer device 110 may include any mobile or fixed computer device that may be connectable to a network for streaming video games. Computer device 110 may be, for example, a computer device (such as a desktop or laptop or tablet computer), a cellular phone, a game console / device, a mixed reality or virtual reality device, a music device, a television, a navigation system, a camera, a personal digital assistant (PDA) or handheld device, or any other computer device having wired and / or wireless connection capabilities to one or more other devices.

[0024] The computer device 110 may include a memory 112 and a CPU 114 configured to control the operation of the computer device 110. The memory 112 may be configured to store data and / or computer executable instructions that define an operating system 140 and / or a client application 150 and / or are associated with the operating system 140 and / or the client application 150, and the CPU 114 may execute the operating system 140 and / or the client application 150, which may be a streaming game client application. Examples of the memory 112 may include, but are not limited to, computer-usable memory types such as random access memory (RAM), read-only memory (ROM), tapes, disks, optical disks, volatile memory, non-volatile memory, and any combination thereof. The memory 112 may store a local version of an application executed by the CPU 114.

[0025] CPU 114 may include one or more processors for executing instructions. Examples of CPU 114 may include, but are not limited to, any processor specifically programmed as described herein, including a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SoC), or other programmable logic or state machine. CPU 114 may include other processing components, such as an arithmetic logic unit (ALU), registers, and a control unit.

[0026] The display 116 may be connected to a video port 122 that communicates with the CPU 114 and a graphics processing unit (GPU) 120 of the computer device 110. Image processing may be divided between the CPU 114 and the GPU 120 in a manner that enables the determination and application of image transforms to be provided. For example, the CPU 114 may determine when an image transform is to be applied, and the GPU 120 may apply the image transform to each pixel of one or more previous images.

[0027] Computer device 110 may also include input device 118 for receiving input from a user. Input device 118 may include, for example, a mouse, keyboard, touch screen, touchpad, buttons, game controller, digital pen, head mounted display, wearable device, or other device that provides input.

[0028] The computer device 110 may also include a video port 122 integrated with the CPU 114 and / or the GPU 120. The video port 122 may be, for example, a VGA, DVI, HDMI, or DisplayPort output. The video port 122 may provide a video signal that may be utilized by the display 116, which may include a monitor or a virtual reality head mounted display (HMD). For example, the video port 122 may periodically scan out a video buffer in the GPU memory 126. The CPU 114, the GPU 120, and the video port 122 may perform general graphics processing operations for the computer device 110.

[0029] The computer device 110 may also include a network interface 124, which may include any wired or wireless network interface (e.g., Ethernet or Wi-Fi). As another example, the computer device 110 may optionally include a universal serial bus (USB) interface. The network interface 124 of the computer device 110 may be connected to a corresponding network interface of the streaming server 160 via a network 170. The network 170 may be a communication network for transmitting packets. For example, the network 170 may be a local area network (LAN), a wide area network (WAN), or the Internet. The network 170 may utilize various protocols, including an Internet protocol (IP). Therefore, the computer device 110 may transmit data (e.g., data packets) to the streaming server 160 via the network 170 and receive data (e.g., data packets) from the streaming server 160.

[0030] The network 170 may contribute to the latency of the communication. Latency in the context of streaming video games may refer to the duration of the time period between when a user makes an input and when the user sees an image updated on his screen (e.g., display 116). High latency may make the game unplayable. Latency in streaming situations includes many components, including network latency, draw time, image encoding time, etc., and may exceed 100ms. Another characteristic of network communication is jitter. Jitter measures the variability of latency. From the perspective of a streaming video game player, jitter manifests as an uneven response to input and may cause a jarring gaming experience. Wireless networks such as wi-fi and mobile networks typically have higher jitter than wired networks. Bandwidth may refer to how much data can be transmitted between a client (e.g., computer device 110) and a server (e.g., streaming server 160) per second on average. Bandwidth may be asymmetric, with the bandwidth from client to server typically being much lower than the bandwidth in the opposite direction. Lossy network protocols such as user datagram protocol (UDP) are typically used for network streaming. This is because protocols with error checking and correction capabilities, such as the Transmission Control Protocol (TCP), have worse latency guarantees. Using protocols such as UDP may result in some packets being lost in transit.

[0031] The operating system 140 may include instructions (such as the client application 150) stored in the memory 112 and executed by the CPU 114. The operating system 140 may include a display controller 142 for controlling the GPU 120. For example, the display controller 142 may provide control commands to the GPU 120 to perform one or more specific graphics processing operations, such as drawing a source image or performing adjustments. The display controller 142 may include: a streaming engine 144 for receiving and decoding the video stream 166; a delay module 146 for detecting delays in the video stream 166; and a re-projection module 148 for generating image adjustments to be applied to one or more images in one or more previous video frames when a current video frame has not been received.

[0032] The computer device 110 may also include a client application 150 that includes instructions stored in the memory 112 and executed by the CPU 114. The application 150 may be, for example, a streaming video game client application that communicates with the streaming server 160. For example, the application 150 may include a control component 152 that receives user input from a user and transmits control commands 158 to the streaming server 160. In one implementation, the application 150 may also maintain some state information about the video game. For example, the application 150 may maintain a camera matrix 154 that indicates the pose of the camera. The application 150 may update the camera matrix 154 based on the server camera matrix received from the streaming server 160. In addition, in one implementation, the application 150 may include a geometry 156 that indicates the geometry of one or more objects that may be drawn in the video frame. In some implementations, the application 150 may provide the camera matrix 154 and / or the geometry 156 to the reprojection module 148 to determine the image transformation.

[0033] Reference now Figure 2 , diagram 200 shows an example timing of processing and communication at the computer device 110 and the streaming server 160. At 202, the computer device 110 (e.g., GPU 120 and display 116) can display a previous image. The previous image can correspond to a previous frame in the video stream. The previous image can be based on, for example, video information received from the streaming server. As discussed in further detail below, the previous image can also be based on an image transformation applied to another video frame. In either case, the previous image can refer to an image that has been displayed on the display 116, for example, in a previous frame. In addition, the video packet of the image to be displayed in the frame may arrive later. Although the image may arrive too late to be displayed in the correct frame, the image can be used for image transformation and can be referred to as an image corresponding to the previous frame.

[0034] At 204, the computer device 110 (e.g., the control component 152) can receive user input for controlling the video game from the user. For example, the input can be provided via the input device 118 (e.g., a mouse, keyboard, touch screen, controller, or other device that enables user interaction). At 206, the control component 152 can pass the user input as a control command 158 to the streaming server 160. The control command 158 can be a raw input (e.g., mouse movement and button presses), or the control component 152 can perform some processing of the input. For example, the control component 152 can convert mouse movement or button presses into game commands. That is, instead of transmitting raw input data, the control component 152 can transmit game commands, such as commands for moving a character forward or activating an ability. Therefore, the control command 158 can include one or both of the raw input command and the game command.

[0035] At 208, the streaming server 160 may perform a game state update. The game state update may include any command executed by the game engine 162. For example, the game state update may include interactions between one or more users, environments, and objects. The game state update may be based on the control command 158.

[0036] At 210, the streaming server 160 may perform video rendering for each user. The streaming server 160 may generate an image based on the user's camera view. For example, the camera view may include the environment and any objects that the user character can see. The streaming server 160 may not necessarily display the generated image at the streaming server 160. The streaming server 160 may generate the image in a memory.

[0037] At 212, the streaming server 160 may perform video encoding on the drawn video image. Various video encoding techniques are known in the art for reducing the size of the video for transmission. Such video encoding is important for streaming video games to reduce the network bandwidth used by the game. Video encoding techniques may generate various descriptive data that can be decoded to regenerate the encoded image. For example, video encoding may include motion vectors for pixels or groups of objects.

[0038] At 214, computer device 110 may determine image adjustments to apply to one or more previous images. Computer device 110 may determine the image adjustments concurrently with processing at streaming server 160. Thus, at 216, computer device 110 may make the image adjustments available when a packet for a next video frame is about to arrive.

[0039] At 218, the computer device 110 can determine whether the video packet of the next video frame has been received. Action 218 can occur at a specified time before the display of the next video frame. For example, the specified time can be a time that allows decoding of video packets and / or performing image adjustments. If all video packets have been received at the specified time, then at 222, the computer device 110 can decode the image based on the received video packets. If the video packets have not been received in time at the specified time, then at 220, the computer device 110 can apply the determined image adjustment to one or more images associated with one or more previous frames to generate a substitute image. In one implementation, the decision at action 218 can occur before adjusting the image at action 220 or decoding the image at action 222. Since actions 220 and 222 can utilize the same resources (e.g., GPU 120 and memory 126) to determine which action to perform, resources can be saved. In an alternative implementation, both actions 220 and 222 can be performed to attempt to generate two images. If the decoding process fails (e.g., due to not receiving the video packets in time), a substitute image can be selected.

[0040] At 224, if the video packet is not received in time before the specified time, the computer device 110 can display a substitute image, or if the video packet is received in time, the computer device 110 can display the received image. Thus, the computer device 110 can display an image generated by the streaming server or a substitute image depending on whether the video packet is received in time. For example, in a case with high latency, the computer device 110 can display a substitute image.

[0041] Reference now Figure 3 , example video stream 300 includes video frames 310, 330, and 350. Video frames 310 and 330 may be streaming frames generated by streaming server 160. Video frame 350 may be a substitute frame generated by computer device 110 if a corresponding video frame does not arrive from streaming server 160 in time for display.

[0042] As an example, video frame 310 may include an image of corridor 312-a that intersects a vertical corridor and terminates at wall 314-a. For example, a video game may allow a user to control a character walking along corridor 312-a. Corridor 312-a may be defined by a left wall and a right wall. The right wall may include a window 316-a, and the left wall may include a doorway 318-a.

[0043] Video frame 330 may be the next video frame after video frame 310. For example, if control command 158 moves the character forward, video frame 330 may be generated. In video frame 330, the previous position of each object (e.g., in video frame 310) is shown as a dotted line. Motion vectors 332, 334, 336 for indicating the direction and magnitude of the motion are shown as arrows. For example, wall 314-b may have moved downward in video frame 330 as indicated by motion vector 332, while window 316-b may have moved downward and to the right as indicated by motion vector 334, and doorway 318-b may have moved downward and to the left as indicated by motion vector 336. The size of wall 314-b, window 316-b, and doorway 318-b in video frame 330 may also become larger and larger as the camera moves toward each of the objects. Size information may also be included in the video encoding.

[0044] The video encoding of video frame 330 may identify each of wall 314-b, window 316-b, and doorway 318-b as a group of pixels, and indicate a corresponding motion vector 332. Since most pixels are the same in video frame 310 and video frame 330, providing the moved groups of pixels and motion vectors may provide significant compression for video encoding.

[0045] The video encoded motion vector 332 may also be used to determine an image transformation to generate a substitute image, which may be displayed as a video frame 350 if a video packet is not received in time from the streaming server 160. Video frame 350 shows an example substitute image based on motion vectors 332, 334, 336. For example, in one implementation, the image transformation may include applying motion vectors 332, 334, 336 to the same object or pixel group as the previous frame. As shown, in video frame 350, wall 314-c may be moved further downward, window 316-c may be moved further downward and to the right, and doorway 318-c may be moved further downward and to the left. Pixels that are beyond the screen space after the image transformation may not be drawn. For example, the portion of doorway 318 that extends beyond the bottom edge may not be displayed. Applying the motion vector from the previous video frame 330 may also result in an area 352 without color information because all pixels in area 352 have been moved and no new information is received. Computer device 110 may make a best-effort guess as to the color to use for region 352 based on neighboring groups of pixels (eg, wall 314 - c and the left and right walls of hallway 312 - c ).

[0046] although Figure 3An example of image transformation based on the history of video frames is shown, but other techniques can be used to determine image transformation. For example, given a camera transformation and a per-pixel depth value, a point in world space can be calculated. A color from the current color frame can be assigned to each point. Given a different camera transformation, these colored points can be projected into the screen space. This type of image transformation can be called reprojection. Reprojection can include, for example, tilting or otherwise adjusting the image according to a change in position information or camera posture. For example, the computer device 110 may have per-pixel depth and color and a camera transformation matrix from a previous frame, and packets in the video stream 166 may be lost. The lost packets may correspond to a certain area on the screen where the computer device 110 does not have its current data now. Through the current frame transformation, the computer device 110 can use reprojection to fill the lost area with colors from one or more previous frames. One benefit of managing camera transformations and using reprojection is that reprojection allows all image motion in a frame due to camera motion to be factored out and represented as a single change in posture motion, thereby leaving only independent motion sources in the scene for management, transmission, processing, etc., without having to rely entirely on encoding. For example, with reprojection, since the motion vectors 332, 334, 336 are derived from player / camera motion, and the rest of the scene is static, the above description of the encoder can be omitted. Figure 3 Therefore, unless other scene elements move independently of the player in corridor 312, re-projection can use only the camera transform. Re-projection module 148 can determine the camera transform and per-pixel depth values ​​in several ways.

[0047] Depth reconstruction is a process for reconstructing depth information from a flat 2D image (e.g., video frame 330). Several algorithms are known in the art to reconstruct depth information from a single image, an image plus camera transformation, a stereo image, an image sequence, etc. This is an active research area in image processing and robotics, and a new artificial intelligence-based approach can be used for depth reconstruction. The depth information of the scene can be cached and reused in subsequent frames. The depth information can also be interactively refined and updated so that the reprojection module 148 maintains a representation of the depth in the 3D scene. The depth information can be represented as voxels, or it can be described using parameters. In one implementation, the camera transformation matrix for the complete frame can be estimated and refined based on feature observation correspondences across multiple color image frames. In addition, the motion vectors of the camera transformation matrix across frames can be determined and extrapolated for subsequent frames.

[0048] In some implementations, the streaming server 160 may provide additional information for determining the image transformation via the client application 150. For example, the game engine 162 of the streaming server 160 may report the camera matrix 154 to the application 150 so that the streaming server 160 and the computer device 110 have the latest understanding of the camera transformation. In one implementation, the camera matrix 154 may be a 4×4 floating point matrix and may not significantly increase bandwidth consumption. In one implementation, since the bandwidth for transmitting the camera matrix 154 is small, the streaming server 160 may perform redundant transmissions of the camera matrix 154 in multiple packets of a frame. Therefore, if one of the packets containing the matrix does not arrive in time, the camera matrix 154 can be recovered from another packet in the frame. In one implementation, the application 150 may update the camera matrix 154 in parallel with the game engine 162. For example, in some games, the control command 158 may directly control the camera matrix 154 (e.g., the control command 158 moves the virtual camera). The application 150 can update the camera matrix 154 without having to perform the more complex game state updates and drawing operations performed by the streaming server 160 .

[0049] When available, the camera matrix 154 can assist the streaming engine 144 in picking good motion vectors, and can also help generate motion vectors at a higher granularity in the streaming server 160 and the computer device 110. Knowing the camera transform can also help separate the background scene from the moving objects. That is, knowing the camera matrix 154 can make the depth buffer reconstruction on the computer device 110 more robust.

[0050] Another example of additional information that can be provided by the streaming server 160 is depth information. The reprojection module 149 can use the depth information from the streaming server 160 instead of relying on depth reconstruction, or can use the depth information to enhance the depth reconstruction. Having ground truth depth information on the computer device 110 can reduce the complexity of depth reconstruction. However, if per-pixel depth is transmitted, the depth buffer information may significantly increase bandwidth consumption. One technique to reduce the bandwidth of the depth buffer information is to transmit only changes to the depth information after taking into account changes in the camera position. As described above, reprojection can be used to factor out camera motion.

[0051] Another example of additional information that may be provided by the streaming server 160 is the geometry 156. Since the geometry of an object is unlikely to change quickly, the transmission of the geometry 156 may be performed infrequently, and the information may be compressed or packaged. For example, the geometry 156 may be transmitted with video frames that have few changes, resulting in fewer and / or smaller packets. The client application 150 may also include the geometry 156 stored at the computer device 110. For example, the client application 150 may include the geometry 156 for frequently used objects.

[0052] Another example of additional information that may be provided by the streaming server 160 includes additional motion vectors beyond those included in the video encoding. For example, motion vectors may be provided for spatial screen space regions of a frame at various levels of granularity, from coarse regions to per-pixel or sub-sample granularity. In another implementation using geometry 156, motion vectors for scene geometry may be provided by the streaming server 160.

[0053] Machine learning can be used by the reprojection module 148 for image segmentation to further refine the range of the coarse motion vector information and / or improve depth reconstruction using the improved range attribute. For example, a machine learning model (such as a deep convolutional neural network) can perform image classification. For example, a neural network can perform clustering on images of a video stream to identify groups of pixels that experience the same motion vector. Groups of pixels can represent objects with fixed depth relationships between pixels. In another example, a machine learning classifier can be trained using examples of segmentation generated by the game engine 162, in which objects are identified. The reprojection module 148 can combine coarse-grained motion vectors (e.g., generated by a video encoder 164 or a game engine 162, which may be constrained by a network bandwidth budget for transmission) with fine-grained image segmentation performed on the client computer device 110.

[0054] Reference now Figure 4 , an example method 400 for a computer device 110 to perform a streaming video game is provided, wherein a streaming server 160 generates video images and streams the video images to the computer device 110. For example, when a packet including a streaming video image does not arrive at the computer device 110 in time for display, the method 400 can be used to reduce the impact of latency on the streaming video game. The actions shown in the method 400 can overlap in time. For example, at a certain moment, two actions can be performed by different components. The execution of the actions can also be staggered across components. In addition, the actions shown in the method 400 can be performed in different ways. Figure 4 are executed in the order shown.

[0055] At 410 , method 400 may include transmitting the video game control command to the streaming server. For example, client application 150 running on computer device 110 may execute control component 152 to transmit control command 158 to streaming server 160 via network interface 124 .

[0056] At 420, method 400 may include receiving a video stream encoding video images generated in response to the video game control. For example, display controller 142 may execute streaming engine 144 to receive video stream 166 encoding video images generated in response to control command 158. Streaming engine 144 may receive video stream 166 from streaming server 160 via network interface 124.

[0057] At 430, method 400 may include determining that a video image of a video stream to be displayed in a frame has not been completely received at a specified time prior to display of the frame. For example, display controller 142 may execute delay module 146 to determine that a video image of video stream 166 to be displayed in a frame has not been completely received at a specified time prior to display of the frame. For example, at the specified time, delay module 146 may determine whether all packets of the frame have been received, and determine that the video image has not been completely received when any packets are lost.

[0058] At 440, method 400 may include determining an image transformation based at least on a history of the video image and a motion vector for the video stream. For example, reprojection module 148 may determine an image transformation based at least on a history of previous frames and a motion vector for the video stream. In one implementation, the image transformation may be a reprojection. At 442, action 440 may include extracting a motion vector from a video encoding of the video stream. For example, reprojection module 148 may extract a motion vector from a video encoding of video stream 166. At 444, action 440 may include determining a depth value of an image point corresponding to one or more previous video frames. For example, reprojection module 148 may determine a depth value of a point of a previously displayed image or an image received too late to be displayed in a previous frame. In one implementation, reprojection module 148 may perform a depth reconstruction algorithm. In another implementation, at 448, reprojection module 148 may receive depth information from streaming server 160 via application 150. In another implementation, reprojection module 148 may apply a machine learning model trained for a video game to an image to determine a depth value. At 449, action 440 may include maintaining a camera matrix indicating the position of the virtual camera relative to the video image. For example, client application 150 may maintain camera matrix 154 indicating the position of the virtual camera relative to the video image.

[0059] At 450, method 400 may include applying an image transform to at least a portion of an image corresponding to one or more previous frames to generate a replacement image. For example, reprojection module 148 may control GPU 120 to apply an image transform to at least a portion of one or more images corresponding to one or more previous frames. Reprojection module 148 may send a command to GPU 120 along with the image transform and the portion of one or more previously displayed images to which the image transform is to be applied. GPU 120 may generate a replacement image by performing an image transform on one or more previously displayed images. At 452, action 450 may include reprojecting at least a portion of a previous video frame based on a depth value of a previous video frame and a camera transform. For example, GPU 120 may reproject at least a portion of an image corresponding to previous video frame 330 based on a depth value of a previous video frame and camera matrix 154.

[0060] At 460, method 400 may include displaying a substitute video image in a frame, the substitute video image including at least a portion of the transformed image. For example, display 116 may display video frame 350 including at least a portion of the transformed image of video frame 330. Display 116 may scan out the video image from GPU 120 via video port 122.

[0061] Reference now Figure 5 , shows an example computer device 110 according to one implementation, and Figure 1 In comparison, the example computer device 110 includes additional component details. In one example, the computer device 110 may include a processor 48 for performing processing functions associated with one or more of the components and functions described herein. The processor 48 may include a single group or multiple groups of processors or a multi-core processor. In addition, the processor 48 may be implemented as an integrated processing system and / or a distributed processing system. In one implementation, for example, the processor 48 may include a CPU 114 and a GPU 120. In one example, the computer device 110 may include a memory 50 for storing instructions executable by the processor 48 to perform the functions described herein. In one implementation, for example, the memory 50 may include a memory 112 and / or a memory 126.

[0062] In addition, the computer device 110 may include a communication component 52 that utilizes the hardware, software, and services described herein to provide for establishing and maintaining communications with one or more parties. The communication component 52 may communicate between components on the computer device 110 and between the computer device 110 and external devices (such as devices located in a communication network and / or devices connected serially or locally to the computer device 110). For example, the communication component 52 may include one or more buses, and may also include a transmission chain component and a reception chain component associated with a transmitter and a receiver, respectively, operable to interface with an external device. In one implementation, for example, the communication component 52 may include a network interface 124.

[0063] In addition, the computer device 110 may include a data repository 54, which may be any suitable combination of hardware and / or software, that provides mass storage of information, databases, and programs employed in conjunction with the implementations described herein. For example, the data repository 54 may be a database for the operating system 140 ( Figure 1 ) and / or application 150( Figure 1 ) data repository.

[0064] The computer device 110 may also include a user interface component 56 that is operable to receive input from a user of the computer device 110 and is also operable to generate output for presentation to the user. The user interface component 56 may include one or more input devices, including but not limited to a keyboard, a numeric keypad, a mouse, a touch-sensitive display, navigation keys, function keys, a microphone, a voice recognition component, any other mechanism capable of receiving input from a user, or any combination thereof. In addition, the user interface component 56 may include one or more output devices, including but not limited to a display, a speaker, a tactile feedback mechanism, a printer, any other mechanism capable of presenting output to a user, or any combination thereof.

[0065] In one implementation, the user interface component 56 may transmit and / or receive messages corresponding to the operation of the operating system 140 and / or the application 150. In addition, the processor 48 executes the operating system 140 and / or the application 150, and the memory 50 or the data repository 54 may store them.

[0066] As used in this application, the terms "component", "system" etc. are intended to include computer-related entities, such as but not limited to hardware, firmware, a combination of hardware and software, software, or execution software. For example, a component can be but not limited to a process, a processor, an object, an executable file, an execution thread, a program and / or a computer running on a processor. As an illustration, both the application and the computer device running on a computer device can be components. One or more components can reside in an execution process and / or thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. A component can communicate by a local and / or remote process, such as according to a signal with one or more data packets, such as data from a component interacting with another component in a local system, a distributed system and / or interacting with other systems via a signal through a network (such as the Internet).

[0067] Additionally, the term "or" is intended to mean an inclusive or rather than an exclusive or. That is, unless specified otherwise or clear from context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following: X employs A; X employs B; or X employs both A and B. Additionally, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from context to be directed to the singular.

[0068] Various implementations or features may be presented in terms of systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that the various systems may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Combinations of these methods may also be used.

[0069] The various illustrative logic, logic blocks, and actions of the methods described in conjunction with the embodiments disclosed herein may be implemented or executed with a specifically programmed one of a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, intended to perform the functions described herein. The general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computer devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In addition, at least one processor may include one or more components operable to perform one or more of the above steps and / or actions.

[0070] In addition, the steps and / or actions of the method or algorithm described in conjunction with the implementation disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. In addition, in some implementations, the processor and the storage medium may reside in an ASIC. In addition, the ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside in a user terminal as discrete components. In addition, in some embodiments, the steps and / or actions of the method or algorithm may reside on a machine-readable medium and / or a computer-readable medium as a code and / or instruction or any combination or set of codes and / or instructions, and the machine-readable medium and / or the computer-readable medium may be incorporated into a computer program product.

[0071] In one or more implementations, the described functions can be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, including any media that helps to transfer a computer program from one place to another. Storage media can be any available media that a computer can access. As an example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. Disks and optical disks used herein include compact disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein disks usually copy data magnetically, and optical disks usually copy data optically by lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0072] Although the implementation of the present disclosure has been described in conjunction with the examples of the present disclosure, it will be appreciated by those skilled in the art that changes and modifications may be made to the above implementations without departing from the scope of the present invention. Other implementations will be apparent to those skilled in the art by considering the specification or practicing according to the examples disclosed herein.

Claims

1. A computer device comprising: Memory, which stores data and instructions; a central processing unit in communication with the memory; a network interface communicatively coupled to a communication network; as well as A client application and display control component, in communication with the memory, the central processing unit and the network interface, is operable to: transmitting video game control commands to a streaming server via the network interface; receiving a video stream from the streaming server via the network interface, the video stream encoding video images generated in response to the video game control commands; determining an image transform based at least on a history of the video image and a motion vector for the video stream; applying the image transform to at least a portion of one or more images corresponding to one or more previous frames to generate a substitute video image, the substitute video image including at least a portion of the transformed previously displayed image, prior to a specified time at which a subsequent frame for the video stream is scheduled to arrive; determining that a video image of the video stream to be displayed in the subsequent frame has not been completely received at the specified time; as well as In response to the determination that the video image to be displayed in the subsequent frame has not been completely received at the specified time, the substitute video image including the at least a portion of the transformed previously displayed image is displayed in the subsequent frame instead of the video image of the video stream. 2 . The computer device of claim 1 , wherein the display control component is configured to extract the motion vector from the encoding of the video stream.

3. The computer device of claim 1, wherein the image transformation is a re-projection of the portion of the one or more images corresponding to the one or more previous frames.

4. The computer device of claim 3, wherein the display control component is configured to determine a depth value for a point of the one or more images corresponding to the one or more previous frames, wherein the re-projection is based on the depth value.

5. The computer device of claim 4, wherein the display control component is configured to apply a machine learning model trained for the video game to the one or more images corresponding to the one or more previous frames to determine the depth value.

6. A computer device according to claim 1, wherein the display control component is configured to apply a machine learning model to perform segmentation on the one or more images corresponding to the one or more previous frames and apply the motion vector to the identified image segmentation.

7. The computer device of claim 1, wherein the client application is configured to receive depth information from the streaming server.

8. The computer device of claim 1, wherein the client application is configured to maintain a camera matrix indicating a position of a virtual camera relative to the video image, wherein the image transformation is based on the camera matrix.

9. The computer device of claim 8, wherein the client application is configured to periodically receive an updated camera matrix from the streaming server.

10. The computer device of claim 1, wherein the client application is configured to receive a geometric representation corresponding to the video image from the streaming server.

11. The computer device of claim 1 , wherein the display control component is configured to: determining whether all packets for the frame have been received at the specified time; and Upon any packet loss, it is determined that the video image of the video stream to be displayed in the frame has not been completely received.

12. A method for playing a streaming video game on a computer device, comprising: transmitting video game control commands to a streaming server; receiving a video stream from the streaming server, the video stream encoding video images generated in response to the video game control commands; determining an image transform based at least on a history of the video image and a motion vector for the video stream; applying the image transform to at least a portion of one or more images corresponding to one or more previous frames to generate a substitute video image, the substitute video image including at least a portion of the transformed previously displayed image, prior to a specified time at which a subsequent frame for the video stream is scheduled to arrive; determining that a video image of the video stream to be displayed in the subsequent frame has not been completely received at the specified time; as well as In response to the determination that the video image to be displayed in the subsequent frame has not been completely received at the specified time, the substitute video image including the at least a portion of the transformed previously displayed image is displayed in the subsequent frame instead of the video image of the video stream.

13. The method of claim 12, wherein determining the image transform comprises extracting the motion vectors from the encoding of the video stream.

14. The method of claim 12, wherein the image transformation is a re-projection of the portion of the one or more images corresponding to the one or more previous frames.

15. The method of claim 14, wherein determining the image transformation comprises determining a depth value for a point of the one or more images corresponding to the one or more previous frames, wherein the re-projecting is based on the depth value.

16. The method of claim 15, wherein determining the depth value comprises applying a machine learning model trained for the video game to the one or more images corresponding to the one or more previous frames to determine the depth value.

17. The method of claim 12, wherein determining the image transform comprises maintaining a camera matrix indicating a position of a virtual camera relative to the video image, wherein the image transform is based on the camera matrix.

18. The method of claim 17, wherein maintaining the camera matrix comprises periodically receiving an updated camera matrix from the streaming server.

19. The method of claim 12, wherein determining the image transformation comprises: receiving, from the streaming server, a geometric representation corresponding to the video image; as well as The geometric representation is plotted to determine the depth value.

20. A computer-readable medium comprising code executable by one or more processors for playing a streaming video game on a computing device, the code comprising code for: transmitting video game control commands to a streaming server; receiving a video stream from the streaming server, the video stream encoding video images generated in response to the video game control commands; determining an image reprojection based on one or more images of the video stream corresponding to one or more previous frames and a motion vector selection based on a camera transformation matrix for the one or more previous frames; applying the image re-projection to at least a portion of the one or more images of the video stream corresponding to one or more previous frames to generate a substitute video image, the substitute video image comprising at least a portion of the re-projected image, prior to a specified time at which a subsequent frame of the video stream is scheduled to arrive; determining that a video image of the video stream to be displayed in the subsequent frame has not been completely received at the specified time; as well as In response to the determination that the video image to be displayed in the subsequent frame has not been completely received at the specified time, displaying the substitute video image including the at least a portion of the re-projected image in the subsequent frame instead of the video stream. The video image.

21. The computer-readable medium of claim 20, wherein the code for determining the image re-projection comprises code for determining depth values ​​for points of the one or more images corresponding to the one or more previous frames, wherein the image re-projection is based on the depth values.

Images