Image streaming method and image streaming system

Abstract

This provides an image streaming method for streaming image content from a remote computing device to a client device. [Solution] The method includes the step of a client device, which is an entertainment device 10, receiving one or more masked images from a remote computing device, such as a cloud server 15. The masked images include a first image in which one or more parts are masked. The method also includes the step of reconstructing the first image from the masked images using a MAE model (Masked Autoencoder Model) in the entertainment device. The MAE model is trained to reconstruct images from masked images. The method further includes the step of outputting the reconstructed one or more first images for display in the entertainment device.

Description

Technical Field

[0001] The present invention relates to an image streaming method and system for streaming image content from a remote computing device to a client device.

Background Art

[0002] Streaming of video content has become more common over time due to technological advances that support it. For example, high-speed Internet has become widespread, and more efficient video codecs have been developed. This has made it possible to efficiently and effectively deliver on-demand content such as movies and television programs to users.

[0003] Rather than being limited to streaming existing video content, it is also considered desirable that a user can stream video content corresponding to software running remotely. Cloud computing and cloud gaming mechanisms can be a favorable option for many users in that users can utilize more advanced processing hardware (e.g., on a server) without directly purchasing hardware themselves. This advantage allows users of low-performance devices (such as mobile phones, portable game consoles, and old computers) to also access content that does not meet basic processing requirements as long as they have an Internet connection.

[0004] While cloud computing has great advantages, it also has several limitations that can adversely affect the user experience. One of these is latency. In an environment with high latency, the delay that occurs between a user's input and the response thereto may become so long as to cause input errors. Similarly, in game play, longer reaction time to events may reduce user performance.

[0005] Several methods have been implemented to reduce latency associated with content streaming. One of these is the use of edge servers, which shortens the content transmission path, thereby reducing communication time between the server and client and lowering latency. However, there is still a demand for further latency reduction in content streaming environments.

[0006] The present invention aims to alleviate or reduce these problems at least partially. [Overview of the project] [Problems that the invention aims to solve]

[0007] The present invention aims to mitigate at least some of these problems. [Means for solving the problem]

[0008] Various aspects and features of the present invention are defined in the appended claims and accompanying description and include at least the following: -As a first embodiment, an image streaming method according to claim 1 is provided. -Another aspect is provided: an image streaming system according to claim 16. [Brief explanation of the drawing]

[0009] A more complete understanding of this disclosure and its many associated benefits can be gained by referring to the following detailed description and attached drawings. [Figure 1] This is a diagram illustrating the entertainment system in general terms. [Figure 2] This is a schematic flowchart illustrating an image streaming method. [Figure 3] This flowchart outlines the operations performed by the masked autoencoder. [Figure 4] This figure shows an example of an image processed by a masked autoencoder. [Figure 5A]This is a schematic diagram illustrating the image. [Figure 5B] This figure schematically shows a partially masked image corresponding to the image in Figure 5A. [Figure 6] This figure schematically shows the images used in the image streaming method shown in Figure 2. [Figure 7] This diagram schematically illustrates further image streaming methods. [Modes for carrying out the invention]

[0010] Image streaming methods and systems are disclosed. The following description provides many specific details to fully understand embodiments of the present invention. However, those skilled in the art will understand that these specific details are not necessarily required to carry out the invention. Conversely, for clarity, specific details well known to those skilled in the art are omitted where appropriate.

[0011] In embodiments of the present invention, a suitable system and / or platform for implementing the methods and techniques described herein may be an entertainment system comprising an entertainment device (i.e., a client device) and a cloud server (i.e., a remote computing device).

[0012] Referring to the drawings, identical or corresponding parts are denoted by the same reference numeral. Figure 1 shows an example of an entertainment system including an entertainment device 10 and a cloud server 15.

[0013] The entertainment device 10 may be, for example, a computer or a video game console. The entertainment device 10 includes a central processing unit 20. The central processing unit 20 may be a single-core or multi-core processor. The entertainment device also includes a graphics processing unit (GPU) 30. The GPU may be physically separate from the CPU or integrated with the CPU as a system-on-a-chip (SoC).

[0014] The GPU can work with the CPU as needed to process data, generate video (image data) and, if necessary, audio, and output them via AV output. If necessary, audio may be generated in conjunction with an audio processor (not shown).

[0015] Video and optional audio are presented on a television or similar device. If the television supports it, the video is stereoscopic. Audio is presented on a home theater system in one of several formats, such as stereo, 5.1 surround sound, or 7.1 surround sound. Similarly, video and audio are presented on a head-mounted display unit 120 worn by user 1.

[0016] This entertainment device features RAM40, which may have separate RAM for the CPU and GPU, and / or shared RAM. The RAM is either physically isolated or integrated as part of the SoC. Additional storage is provided by disk50, which may take the form of an external or internal hard drive, an external solid-state drive, or an internal solid-state drive.

[0017] This entertainment device transmits or receives data via appropriate data ports 60, such as USB® ports, Ethernet® ports, Wi-Fi® ports, and Bluetooth® ports. It can also receive data via an optical drive 70.

[0018] Audio / visual output from this entertainment device is typically provided through one or more A / V ports 90, or through a wired or wireless data port 60.

[0019] As an example of a device that displays an image output from an entertainment device, a head-mounted display "HMD" 120 worn by a user 1 can be mentioned. Images output from an entertainment device can also be displayed using various other devices (for example, a conventional television display connected to an A / V port 90).

[0020] If the components are not integrated, they are appropriately connected via a dedicated data link or bus 100.

[0021] Interaction with the device is typically provided using one or more handheld controllers 130, 130A, and / or in the case of an HMD, one or more VR controllers 130A-L, R. A user typically interacts with the system, the content displayed by the system, and the rendered virtual environment by providing input via the handheld controllers 130, 130A. For example, when playing a game, the user can move within the virtual environment of the game by providing input using the handheld controllers 130, 130A.

[0022] In an embodiment of the present disclosure, the entertainment device 10 generates one or more images of a virtual environment for display (for example, via a television or the HMD 120).

[0023] Accordingly, FIG. 1 shows an example of a data processing apparatus suitable for executing an application such as a video game and generating a video game image for display. The image is output via a display device such as a television, other similar monitors, and / or an HMD (for example, the HMD 120). More generally, the data processing apparatus can receive user input, execute an instance of a video game in response thereto, and render an image for display to the user.

[0024] In this example, the entertainment device 10 is provided with a cloud server accessible via an internet connection, and / or a remote computing device as part of the service 15. The cloud server 15 may include one or more GPUs and / or other suitable hardware components for rendering images.

[0025] In embodiments of the present invention, the methods and techniques described herein may be implemented at least partially using an autoencoder.

[0026] An autoencoder is a type of unsupervised machine learning model that uses one or more artificial neural networks to learn an efficient representation of unlabeled input data. Autoencoders can be used to encode various types of data, including images, videos, text, and audio.

[0027] An autoencoder can include an encoder neural network that encodes input data into a reduced representation (also known as a "latent space representation") and a decoder neural network that aims to reconstruct the input data from the encoded latent space representation. The latent space is typically lower dimensional than the input data. Therefore, the latent space generated by the encoder provides a more efficient and compressed representation that requires less memory than the input data.

[0028] An encoder neural network consists of one or more layers that transform input data into a lower-dimensional representation. The encoder neural network receives input data, and its final layer outputs a lower-dimensional representation of the input data, i.e., the latent space (also called the "bottleneck layer").

[0029] A decoder neural network may consist of one or more layers that transform data from latent space and generate output data with the same dimensionality as the data input to the encoder. The decoder aims to reconstruct the data originally input to the encoder neural network from the latent space representation of the data.

[0030] The neural networks of encoders and / or decoders typically consist of multiple hidden layers. For example, an encoder may consist of multiple hidden layers that progressively reduce the representation of the input data. Using deep neural networks (i.e., those with a large number of hidden layers) for encoders and / or decoders can improve the performance of autoencoders and, in some cases, reduce the amount of training data required.

[0031] The neural networks for encoders and decoders are typically trained simultaneously. During training, the autoencoder aims to minimize the mismatch between the input data to the encoder and the reconstructed output data generated by the decoder by tuning its internal parameters (e.g., weights and biases of the encoder and decoder neural networks) and optimizing (e.g., minimizing) the loss / error function. It will be understood that the specific loss function and the algorithm used to optimize it may vary depending on the nature and application of the autoencoder model. For example, a root mean square error loss function optimized using gradient descent may be used. Sparse autoencoders may also be used to promote sparseness in the latent space representation (compared to the input) and prevent the learning of automorphic functions. For example, a sparse autoencoder can be implemented by modifying the loss function to include a sparse regularization penalty.

[0032] Returning to Figure 1, in the streaming scenario, the cloud server 15 generates an image and transmits it for display by the entertainment device 10. For example, the cloud server 15 renders an image for a video game and transmits the rendered image for display by, for example, the HMD 120 of the entertainment device 10. In some cases, the entertainment device 10 can further transmit user input received by its peripherals (e.g., a handheld controller 130 or 130A) to the cloud server 15, which in turn allows the cloud server 15 to generate an image in response to the received user input.

[0033] Such a streaming configuration has the advantage of shifting the processing related to content generation (e.g., video game content) from the entertainment device 10 to the cloud server 15. By leveraging the cloud server 15's superior computing resources, it becomes possible to deliver more computationally intensive content even if the user's device lacks computing power (for example, if the user uses a mobile phone to interact with the content). However, streaming images from the cloud server 15 to the entertainment device 10 can cause latency issues if there is a delay before the image reaches the entertainment device 10. This problem is particularly noticeable with high-quality (e.g., 4K, and / or virtual reality) image content, making it difficult for users to interact with such content.

[0034] Embodiments of this disclosure relate to a method for reducing latency when streaming images from a remote computing device (e.g., a cloud server 15) to a client device (e.g., an entertainment device 10), particularly in low-bandwidth environments. In this disclosure, the remote computing device sends a partially masked image (e.g., 90% of pixels are masked) to the client device. The client device reconstructs the original (i.e., "unmasked") image from the masked image using a MAE model (Masked Autoencoder Model) trained to reconstruct images from partially masked images. The client device then outputs the reconstructed image for display (e.g., to an HMD 120 of an entertainment device). In this way, the approach improves the balance between latency and quality of the image output to the client. By streaming a partially masked image rather than a complete image, the amount of data streamed to the client is reduced, latency is reduced, and the image streaming becomes more resilient to changes in bandwidth. At the same time, by reconstructing the image on the client device side using the MAE model, information within the image can be efficiently restored, providing the user with a high-quality display image. This method achieves its goal by intentionally masking (i.e., modifying) images before transmission, counterintuitively reducing the bandwidth required for image transmission to the client device.

[0035] This method achieves an extremely high information recovery rate on the client terminal, enabling image reconstruction with sufficient accuracy even while masking a large portion of the streaming image (e.g., 90%). This is in contrast to existing super-resolution-based methods. Existing methods downsample images on the server side and upsample them on the client side using super-resolution technology. In this case, to avoid artifacts in the upsampled image, the ratio of transmitted data to displayed data usually needs to be set much higher (e.g., 2:1, in contrast to 10:1 in 90% MAE). Furthermore, image processing using MAE models is more efficient than many high-performance super-resolution machine learning models, reducing client-side inference time. Therefore, this method can be seen as achieving a better balance than existing super-resolution technologies in achieving the conflicting requirements of low latency and high image quality in a streaming environment.

[0036] This disclosure is particularly applicable to the streaming of interactive content such as video games, where the low latency achieved by this method is especially advantageous. However, this disclosure is also beneficial for streaming other types of content. For example, when streaming live content (e.g., sports broadcasts or entertainment videos), the low latency achieved by this method reduces latency and the risk of lag and other latency-induced artifacts in the content. Similarly, even with static content such as movie streaming, this approach improves streaming efficiency by reducing the amount of data that needs to be streamed from the cloud to the client, and allows for a reduction in the size of the buffer used to store content on the client side during streaming.

[0037] In this specification, the terms “masking” or “masking” mean removing information from an image. For example, masking (or masking) a portion of an image means removing information associated with the masked area (or masked area) from the image. Specifically, this involves setting the image elements corresponding to the masked area (or masked area) to zero or a constant value (e.g., a value indicating gray). Thus, the masking process (or masking process) produces an incomplete version of the image, and this incomplete information can be stored and / or transmitted with less data than the complete image.

[0038] Figure 2 shows an example of an image streaming method according to one or more embodiments of the present disclosure.

[0039] One or more steps of the image streaming method of the present disclosure may be performed by a remote computing device, and one or more steps may be performed by a client device. In the example of Figure 2, steps 210 and 220 are performed by a cloud server 15 (acting as an example of a remote computing device, and both terms are used interchangeably hereafter), and steps 230, 240, and 250 are performed by an entertainment device 10 (acting as an example of an entertainment device, and both terms are used interchangeably hereafter).

[0040] The example in Figure 2 illustrates an image processing method for a single image for the sake of simplicity. However, it will be understood that the techniques described herein are applicable to multiple images, such as multiple image frames for content (e.g., a video game).

[0041] The image processing method in Figure 2 includes the steps of: generating a masked version 202-M of image 202; transmitting the masked image 202-M from the cloud server 15 to the entertainment device 10; receiving the masked image 202-M on the entertainment device 10; generating a reconstructed image 202-R using the MAE model based on the masked image 202-M; and outputting the reconstructed image 202-R for display to the user.

[0042] Considering the steps performed using the method in Figure 2 in more detail, they are as follows:

[0043] Step 210 generates a masking (first) image 202-M for the content. The masked image 202-M is a partially masked version of the (first) original (i.e., unmasked) image 202. Image 202 can include any type of image, such as a photograph or a computer-generated (e.g., rendered) image.

[0044] Image 202 may be an image frame for content. Using the technology described with reference to Figure 2, it is possible to provide multiple content image frames to the user in real time.

[0045] In the example in Figure 2, generating the masked image 202-M involves processing the original (i.e., unmasked) image 202 to mask some or more of its portions. In other words, masking is applied as a post-processing step to the generated image 202 to obtain the partially masked image 202-M.

[0046] The advantage of generating the masked image 202-M in this way is that it can be seamlessly integrated with existing streaming pipelines that generate (render) the original image 202 for streaming to client devices. Therefore, the only change required on the server side (e.g., the cloud) is to partially mask the generated original image 202, and this masked image 202-M can be sent to the client, as described in relation to step 220 below.

[0047] Masking one or more portions of image 202 involves setting the values ​​of the pixels (or voxels in the case of a 3D image) corresponding to the masked portions to zero or a constant. Each masked region may consist of one or more elements (e.g., pixels or voxels) of image 202. Each masked region may be the same size and / or dimensions (e.g., 5x5 pixels in width x height), or it may be different in size and / or dimensions. Each masked region may consist of a single pixel of image 202. Alternatively, each masked region may consist of a pixel patch within image 202. Each patch can be any suitable shape, such as a rectangle or a square, and consists of multiple pixels of image 202. For example, the width x height dimensions of each pixel patch may be 2x2, 4x4, 8x8, 8x4, or 16x16 pixels. Masking image 202 in patch units rather than individual elements can improve the efficiency of the masking process and potentially improve the performance of reconstruction by the MAE model. An exemplary image divided into patches and partially masked is shown in Figure 5.

[0048] The selection of the image portion to be masked in image 102 can be performed in a random and / or uniform manner.

[0049] One or more regions of image 202 may be masked in a random manner. In other words, the masked regions can be randomly distributed across the entire masked image 202-M. For example, mask probabilities (e.g., 50%, 75%, 90%) can be assigned to one or more parts of image 202, and each image region can be masked based on an evaluation of a random function using the mask probabilities for the relevant image parts. For example, some image parts (such as those depicting the background) can be assigned high mask probabilities, while other image parts (such as those depicting objects like text within the image) can be assigned low mask probabilities. Alternatively, a uniform mask probability can be used across the entire image 202.

[0050] Alternatively, or additionally, one or more portions of image 202 may be masked in a uniform manner. In other words, the masked portions may be uniformly distributed across the entire masked image 202-M. For example, every N consecutive pixels in image 202, M pixels may be masked. For instance, pixels could be masked alternately (N=2, M=1) or every 4 pixels could be masked by 3 pixels (N=4, M=3). As illustrated in more detail in Figure 7, uniform masking distribution can further improve integration with existing streaming pipelines. This is because uniform masking is particularly compatible with streaming pipelines that utilize server-side downsampling. Similar to random masking, uniform masking also allows for assigning different mask ratios to different portions of image 202.

[0051] In some cases, the first portion of image 202 can be masked randomly, while the second portion can be masked uniformly.

[0052] The masking of image 202 in step 210 involves determining the mask ratio for image 202 (i.e., the proportion of image 202 to be masked). One or more different mask ratios can be used for different parts of image 202. The mask ratio can then be used to randomly and / or uniformly mask parts of image 202. In random masking, the mask ratio defines the probability that a particular part of the image or its components (e.g., pixels) will be masked.

[0053] In the examples of this disclosure, the masking ratio can be at least 0.5, at least 0.7, or at least 0.9 (i.e., 90% of image 202 is masked). For example, the masking ratio can be 0.5, 0.6, 0.7, 0.75, 0.8, 0.85, or 0.9. It will be understood that increasing the masking ratio reduces the amount of data that needs to be transmitted between the cloud server 15 and the entertainment device 10, but may degrade the quality of the image reconstructed on the client side. For example, excessive masking of image 202 may result in artifacts in the reconstructed image 202-R.

[0054] The mask ratio can be predetermined, for example, based on experimental testing of the quality of reconstructed images (in terms of both the quality of individual images and latency) at various mask ratios. For example, the optimal mask ratio can be selected by evaluating an empirically determined cost function related to latency and quality (e.g., user-perceived quality determined using an appropriate function) for multiple mask ratios and multiple images 202.

[0055] Alternatively, or additionally, the mask ratio may be updated in real time according to the current streaming conditions. This allows the method to appropriately respond to changing streaming conditions (e.g., bandwidth fluctuations between the cloud server 15 and the entertainment system 10) and improve the image quality output to the user on the client side. The mask ratio is determined depending on one or more characteristics of the cloud server 15 (i.e., the server) and / or the entertainment device 10 (i.e., the client), and / or the communication characteristics between the cloud server 15 and the entertainment device 10. For example, the mask ratio is determined depending on one or more of the frame rate of the image 202 transmitted from the cloud server 15 to the entertainment device 10, the communication bandwidth between the cloud server 15 and the entertainment device 10, and / or the computational load of the entertainment device 10. The mask ratio may be determined based on a function empirically determined based on these requirements (i.e., frame rate, bandwidth, and client-side computational load).

[0056] Considering the frame rate, the mask ratio can be increased as the frame rate increases during the transmission of image 202 from the server to the client. This contributes to reducing client-side latency because the increased masking reduces the amount of data transmitted per frame, offsetting the increase in the number of frames per unit time (e.g., second). This method prevents excessive fluctuations in the streaming bitrate between different frame rates. The change in the mask ratio in response to changes in frame rate can be determined based on an empirically determined function that associates the frame rate with the mask ratio. For example, if the initial frame rate is 60 frames per second (fps), the mask ratio can be set to 0.7, and when the frame rate increases to 120 fps, the mask ratio can be increased to 0.9.

[0057] Considering bandwidth, the masking ratio can be increased if the communication bandwidth between the server and client decreases. In this way, this configuration can adapt to changing network conditions and maintain low latency for images delivered to the user. For example, if the bandwidth changes from 100 Mbps to 50 Mbps, the masking ratio can be increased from 0.5 to 0.9 to reduce the amount of data that needs to be transmitted for each image (e.g., each image frame of content such as a video game). Conversely, if the bandwidth increases due to a decrease in network load, the masking ratio can be decreased to improve the quality of the image reconstructed on the client device.

[0058] When considering the computational load on the client side, this load includes processing load on the client device and / or memory load (CPU and GPU utilization, memory access time and usage, etc.). In some cases (for example, when the MAE model encoder is located on the client side and the encoder bears the majority of the client-side computational cost), the mask ratio can be increased as the client-side computational load increases. Setting a higher mask ratio reduces the amount of input data to the MAE model encoder, thereby reducing the processing and memory costs in the reconstruction of image 202-R using the MAE model. On the other hand, in other cases (for example, when the MAE model decoder is located on the client side and the associated computational cost is high), the mask ratio may be decreased as the client-side computational load increases in order to simplify the decoding process.

[0059] By adaptively changing the mask ratio based on streaming parameters in this way, this approach can optimally utilize the current streaming context, prevent excessive latency (by increasing the mask ratio as needed), and provide high-quality output images (by decreasing the mask ratio) where the network and / or computational conditions allow.

[0060] The masking of the original image 202 in step 210 involves selecting and masking one or more areas of the image 202 according to their importance to the client device user. Less important areas of the image may be masked more extensively than more important areas. This improves the accuracy of restoring the important areas and, at a given overall mask ratio, improves the quality of the reconstructed image 202-R that the user perceives. The significance of an image portion may be determined based on one or more of the following, for example: objects associated with (i.e., displayed) the image portion, game context data of the image portion, the user's gaze direction in the entertainment system 10 (determined, for example, using the HMD 120), and / or the placement of the image portion within the image 202.

[0061] Considering game context data, image 202 could be an image of a video game being played by a user of entertainment system 10. Game context data can include data about the degree to which different objects in image 202 are related to the video game being played. For example, game context data can be used to determine which objects displayed in image 102 the user is most likely to pay attention to (e.g., the character the user is controlling, or the character the user is currently fighting), or the direction the user is likely to move in within the virtual environment (e.g., based on the current objective in the game). Corresponding image regions within image 102 can be identified and a lower mask ratio applied to them than to other regions. For example, in image 102, a lower mask ratio can be applied to the part of the image corresponding to a more prominent character (e.g., an enemy boss the user is fighting) (e.g., a patch) than to the part of the image corresponding to a less prominent character (e.g., the user's ally character). It will be understood that game context data can provide a wide range of indicators of the relevance of image regions. For example, in a role-playing game (RPG), game context data could indicate a character speaking to the user, and the part of the image related to that speaking character might be masked to a lower degree.

[0062] By considering gaze data, for example, a lower mask ratio is applied to the image portion corresponding to the user's gaze position. Considering the user's gaze allows this method to evaluate the saliency of image portions in a more individualized, real-time manner, improving the perceived quality of the client-side reconstructed image 202-R.

[0063] Considering the position within the image, reduced masking can be applied to the central portion of image 102, while increased masking can be applied to the peripheral portion. Position-based splendor determination can be a computationally less burdensome alternative to gaze-based splendor determination, for example, when gaze data is unavailable.

[0064] The selection of the area to be masked in image 202 may include selecting a specific area (e.g., pixels) of image 202 to be masked. For example, it may be possible to mask specific pixels corresponding to a landscape determined based on game context data. Alternatively, or in addition, the selection of the area to be masked in image 202 may include setting one or more different mask ratios for different parts of image 202. For example, a higher mask ratio may be set for image parts that are judged to be less prominent compared to more prominent image parts, and the prominentness is determined, for example, based on the user's line of sight.

[0065] Referring to Figures 5A and 5B, these figures show an exemplary original image 500 and the corresponding masked image 500-M, masked using the techniques described herein. In this example, image 500 is an image for a video game.

[0066] As shown in Figure 5A, image 500 is divided into multiple patches 510. In Figures 5A and 5B, patch 510 is drawn in a grid pattern and includes patches 510-1 and 510-2. Furthermore, a portion 520 of image 500 is determined to be more prominent, for example, depending on whether the user's line of sight coincides with the portion 520 of the image and / or whether the portion 520 depicts an in-game character that is deemed relevant to the user.

[0067] Figure 5B shows the corresponding masked image 500-M. Patch 510-1 is not masked, while patch 510-2 is masked. In the example in Figure 5B, the masked patch 510 is randomly distributed within image 500-M. Furthermore, a different mask ratio is applied to the prominent portion 520 of image 500 than that applied to the rest of image 500. In this explanatory example, a mask ratio of 0.5 is used for the rest of image 500, and a mask ratio of 0.25 is used for the prominent portion 520 of the image.

[0068] In Figures 5A and 5B, the patches are enlarged and the mask ratio is set relatively low for illustrative purposes. However, it should be understood that in reality, the patches may be substantially smaller (e.g., about 2-100 pixels on each side), and the mask ratio may be substantially higher (e.g., 0.9 for the rest of the image and 0.7 for the prominent parts of the image).

[0069] Returning to Figure 2, as mentioned above, in the example in Figure 2, masking is performed as a post-processing step for image 102, and the masked image 102-M is generated.

[0070] In another example, masking can be integrated into the image generation process (e.g., the image rendering pipeline). For instance, to generate a masked image 102-M, the portion to be masked can be selected before or during the generation of image 102, and only the unmasked portion of image 102 can be fully generated to obtain the masked image 102-M. This eliminates the need to generate (render) the portion of the image to be masked, thus reducing the computational cost of generating the masked image 102-M and further improving the efficiency of this streaming method. This method allows the cloud server 15 to generate the masked image 102-M more efficiently and quickly, further reducing latency.

[0071] Using the example of rendering and generating image 102, an alternative example can be implemented as follows: Before generating image 102, select one or more regions of image 102 to be masked. For example, these can be randomly selected based on a predetermined mask ratio, or selected according to the user's viewing position as described herein. Then, by omitting at least part of the rendering operation for the masked portion and fully rendering only the unmasked portion of image 102, the masked image 102-M can be generated. At least part of the rendering operation can be omitted because the masked portion is not sent to the client and therefore does not need to be fully rendered. For example, it is not necessary to calculate the final pixel values ​​of the masked portion. For example, at least part of the shading (e.g., fragment shading, and / or lighting operations) and / or rasterization can be omitted for the masked portion. Each masked region may be rendered to a different degree depending on its relevance to the rendering of the unmasked region.

[0072] In some cases (i.e., in certain masked areas), part of the rendering operation of the masked area may be performed to the extent necessary for rendering the unmasked area (for example, to the extent that the geometry or lighting of the masked area affects the lighting of the unmasked area). This can apply, for example, when the masked part constitutes part of the same virtual environment as the unmasked part. In this case, the rendering of the masked part may affect the rendering of the unmasked part, at least partially. For example, when the lighting of the masked part affects the lighting of the unmasked part.

[0073] In some cases, simplifying the rendering process for masked areas can reduce the computational cost of rendering. For example, a less computationally intensive shading algorithm can be used for masked areas than for unmasked areas. This has the advantage of enabling more efficient rendering that takes into account the interaction between masked and unmasked areas (e.g., through lighting).

[0074] Furthermore, if the rendering of the unmasked areas is less dependent on (or independent of) the rendering of the masked areas, further efficiency can be achieved by not performing any rendering operations on the masked areas. For example, this may be applicable when the masked area is an overlay (e.g., an in-game menu in a video game) and does not interact with adjacent unmasked areas (e.g., through lighting or other means), and / or when the masked area is part of the background scenery (e.g., see the top two lines of patch 510 in image 500 of Figure 5A).

[0075] As discussed herein, a high percentage (e.g., 90%) of the pixels in image 102 may be masked. Therefore, it will be understood that at least partially omitting rendering operations on the masked pixels can significantly improve the efficiency of image generation and, more generally, the image streaming method.

[0076] In some cases, it is possible to combine the example in Figure 2 with the alternative examples described above. For example, first generate the first part of image 102, and then mask a portion of it. Next, when generating the second part of image 102, integrate the masking process into the second part generation process, so that only the unmasked areas of the second part are generated.

[0077] As discussed herein, this technology is particularly applicable to live streaming such as cloud gaming. In some cases, step 210 includes generating (e.g., rendering) a masked image 202-M depending on one or more user inputs received from peripheral devices (e.g., controllers 130) operated by a user of the entertainment system 10. The user inputs are processed by the entertainment system 10 and / or the cloud server 15. The user inputs are fed into a game engine, which can determine the generated image 202 to display to the user based on the user inputs.

[0078] Referring to Figure 2, step 220 includes sending the masked image 202-M generated in step 210 from the cloud server 15 to the entertainment device 10.

[0079] The masked image 202-M is transmitted, for example, via an internet connection. It will be understood that the masked image 202-M is encoded, for example, using an appropriate codec, before transmission.

[0080] The masked image 202-M transmitted in step 220 may contain the masked image 202-M itself (typically encoded using an image transmission codec).

[0081] Alternatively, as will be described later, it is also possible to process the masked image 202-M using the encoder of the MAE model before transmission, transmit the latent spatial representation of the masked image 202-M to the entertainment device 10, and decode it using the decoder of the MAE model located on the entertainment device 10. Distributing the MAE model in this way between the cloud server 15 (i.e., the remote computing device) and the entertainment device 10 (i.e., the client device) has the advantage that the amount of data transmitted between the server and the client is reduced because the latent spatial representation is more compact than the masked image 202-M, thereby further reducing the latency of this streaming method.

[0082] Step 230 includes the entertainment device 10 receiving the masked image 202-M sent from the cloud server 15 in step 220.

[0083] Similar to step 220, step 230 may include decoding the masked image 202-M using an appropriate codec.

[0084] Step 240 includes reconstructing an image from the masked image 202-M received in step 230 to obtain the reconstructed image 202-R. Step 240 is performed by the entertainment device 10.

[0085] Image reconstruction in step 240 is performed using an MAE model. The MAE model is an autoencoder and can be implemented and trained using the techniques described herein for autoencoders. The MAE model is trained to reconstruct an image from a partially masked image by predicting missing parts (i.e., masked parts) of the image. The MAE model is particularly well-suited to this task because it can learn an efficient latent space representation of a partially damaged / masked input and recover information within the input. By leveraging the information recovery capabilities of the MAE model, this method can significantly reduce the bandwidth required for image streaming and lower latency. For example, even if 90% of the image pixels are masked, the MAE model may be able to reconstruct the complete image 202 with sufficient accuracy (i.e., the difference between the reconstructed image 202-R and the original image 202 is sufficiently small).

[0086] Referring to Figure 3, an example architecture of the MAE model is shown. The MAE model consists of an encoder 320 and a decoder 340.

[0087] The encoder 320 receives a masked image 310 (e.g., masked image 202-M). The encoder 320 processes the masked image 310 and outputs a latent space representation 330 of the input masked image 310. The input latent space representation 330 is also called the latent space representation 330.

[0088] The encoder 320 may consist of one or more neural networks. For example, the encoder 320 may include one or more convolutional layers that capture spatial features of the input masked image 310 and one or more fully connected layers that compress the data into a low-dimensional latent space. The encoder 320 may use one or more nonlinear activation functions to learn more complex patterns within the input masked image 310.

[0089] The latent space representation 330 output from the encoder 320 is a compressed representation of the input masked image 310 and is typically lower dimensional than the input masked image 310. The latent space representation 330 aims to capture the most important features of the input masked image 310 while discarding redundant information.

[0090] This latent spatial representation 330 is input to the decoder 340. The decoder 340 reconstructs the original (i.e., unmasked) image corresponding to the masked image 310 from the latent spatial representation 330. Unlike the decoder of a conventional autoencoder, the MAE decoder 340 aims to reconstruct a completely unmasked version of the image.

[0091] The decoder 340 may have a configuration that reflects the structure of the encoder 320. The decoder 340 may consist of one or more neural networks. For example, the decoder 340 may include one or more deconvolutional layers that reverse the convolutional operations performed by the encoder 310, and one or more fully connected layers that increase the dimension of the latent space representation to the dimension of the original image. Similar to the encoder 320, the decoder 340 may use one or more nonlinear activation functions to reconstruct more complex patterns in the image.

[0092] The decoder 340 outputs a reconstructed image 350, which is a fully reconstructed version of the masked image 310. In this reconstructed image, the masked areas are filled with image data.

[0093] Therefore, the MAE model aims to learn a robust latent space representation 330 that captures the underlying structure of the masked input image 310, even though a portion of the input image is missing (i.e., masked). This allows the original unmasked image to be reconstructed by the MAE decoder 340.

[0094] The encoder 320 and decoder 340 may be trained simultaneously. During training, the MAE model adjusts its internal parameters (such as the weights and biases of the encoder and decoder neural networks) to optimize (minimize) the reconstruction error, aiming to minimize the discrepancy between the original (unmasked) image input to the encoder 320 and the reconstructed image 350 output by the decoder 340. The reconstruction error is calculated using any appropriate loss function, such as the root mean square error (MSE).

[0095] As an example, both the encoder 320 and decoder 340 of the MAE model can be placed on the client side, such as an entertainment device 10. In this example, a remote computing device sends masked image data 310,202-M to the client device. The client device then processes the image data of the masked image 310 using both the encoder 320 and decoder 340 of the MAE model to obtain a reconstructed image 340,202-R. This configuration has the advantage of enabling seamless integration with existing streaming servers, as the remote server can simply stream the image data using existing image codecs, etc. Since all MAE processing is performed on the client device, the complexity on the server side is minimized.

[0096] In another example, the encoder 320 and decoder 340 of the MAE model can be distributed between the server and the client. The MAE encoder 320 is located on the server side (i.e., a remote computing device such as a cloud server 15), and the MAE decoder 340 is located on the client side (i.e., a client device such as an entertainment device 10). On the server side, the MAE encoder 320 receives the masked image 310,202-M and outputs a reduced latent spatial representation 330 of the masked image. The latent spatial representation 330 is then sent from the server to the client, where it is decoded using the MAE decoder 340 to obtain the reconstructed image 350,202-R. This distributed MAE model configuration allows the latent spatial representation 330 to be transmitted with less data than the image data of the masked image 310,202-M, thus further reducing the latency of image streaming.

[0097] It will be understood that different MAE models can be used depending on the characteristics of the streaming image and the masking method implemented in step 210. Relevant image characteristics include, for example, one or more of the following: image quality (e.g., resolution), frame rate, and / or motion of objects in the image. Relevant aspects of the masking method include, for example, one or more of the following: mask ratio, and / or masking distribution (e.g., uniform or random). For example, different MAE models can be used for 4K images and 1080x720p images. Different MAE models can be used when using random masking and when using uniform masking. Also, different MAE models can be used for different mask ratios. Each MAE model can be trained for a specific case (e.g., a specific masking method).

[0098] In some cases, it is possible to adaptively select the appropriate MAE model to use for reconstructing an unmasked image in real time during image data streaming from the server to the client. For example, the cloud server 15 can monitor and detect the characteristics of the image 202 currently being sent to the entertainment device 10, and / or the current masking technique (e.g., at regular intervals or when a trigger occurs), and select one of several MAE models based on the detected image characteristics and / or masking technique. The cloud server 15 can then send the selected MAE model (e.g., the decoder 340 only, or both the encoder 320 and decoder 340, depending on whether image data or latent space data is being sent for a masked image) to the entertainment device 10 for use in image reconstruction. The transmission of the selected MAE model may include the transmission of a trained MAE model (e.g., its weights and biases) for use in inference. This ensures that the most appropriate MAE model for the currently transmitted image is used for image reconstruction, further improving the quality of the image reconstructed on the client side.

[0099] Returning to Figure 2, step 250 includes the entertainment device 10 (i.e., the client device) outputting the reconstructed image 202-R, which was output by the MAE model in step 240, for display to the user.

[0100] The output of the reconstructed image 202-R may include transmitting image 202-R for display on another display device. The reconstructed image 202-R may be displayed to a user of an entertainment device, for example, using an HMD120.

[0101] Referring to Figure 6, an exemplary image 202 processed using the method of Figure 2 is shown. The original image 202 is processed 210 to produce a partially masked image 202-M. The masked image 202-M is then input to a MAE model that reconstructs image 202 240 to obtain a reconstructed image 202-R. As shown in Figure 4, the reconstructed image 202-R is very similar to the original image 202. At the same time, generating the reconstructed image 202-R using the technique described herein requires less bandwidth than directly transmitting the original image 202 to the entertainment device 10. Therefore, this MAE-based streaming technique improves the balance between output image quality and latency, especially in low-bandwidth streaming scenarios. In low-bandwidth streaming scenarios, low bandwidth becomes a communication bottleneck.

[0102] Referring to Figure 4, further examples of image sets 410, 420, and 430 generated using the image streaming techniques discussed here are shown. Each image set 410, 420, and 430 consists of the following from left to right: images of masked pixels 411, 421, and 431; visualizations of the masked images (with black pixels masked) 412, 422, and 432; reconstructed images 413, 423, and 433; and the original (unmasked) images 414, 424, and 434. The original images 414, 424, and 434 correspond to the original image 202. The reconstructed images 413, 423, and 433 correspond to reconstructed image 202-R. The masked images 412, 422, and 432 correspond to masked image 202-M.

[0103] As shown in Figure 4, even though a significant percentage of pixels are masked in the masked images 412, 422, and 432, the MAE model can accurately reconstruct the masked pixels, so the reconstructed images 413, 423, and 433 are very similar to the original images 414, 424, and 434.

[0104] In the example shown in Figure 2, the masking of image 202 is performed on a remote computing device (e.g., cloud server 15), and the masked image 202-M is sent to a client device (e.g., entertainment device 10).

[0105] Referring to Figure 7, an example of an alternative image streaming method based on the embodiments of this disclosure is shown.

[0106] In the example in Figure 7, instead of masking image 202, the cloud server 15 (i.e., the remote computing device) downsamples image 202 260 and transmits the downsampled image 202-D to the entertainment device 10 (i.e., the client device). As shown in Figure 7, uniformly masking image 202 using a predetermined mask ratio to obtain the masked image 202-M can be considered equivalent to downsampling image 202 260 using a downsampling coefficient corresponding to the mask ratio. For example, as shown in Figure 7, masking image 202 using a mask ratio of 0.75 (i.e., masking 3 pixels every 4 pixels) is considered equivalent to downsampling each dimension of height (h) and width (w) by a factor of 2. Masking and downsampling are equivalent in that data from the same pixel is removed. However, masking masks specific pixels, while downsampling completely removes specific pixels. Therefore, downsampling and masking provide equivalent information compression, allowing the downsampled image 202-D to be transmitted with a similar amount of data as the masked image 202-M.

[0107] On the client side, the entertainment device 10 upsamples the downsampled image 202-D by adding the masked portion to the downsampled image 202-D 270. In other words, the entertainment device 10 restores the spatial resolution of the downsampled image 202-D to the spatial resolution of the original image 202 and fills in the new pixels as the masked portion. Rather than interpolating new pixels as in typical upsampling techniques, the entertainment device 10 simply masks the new pixels. In this way, the entertainment device generates the masked image 202-M. Subsequently, a reconstructed image 202-R is generated based on the masked image 202-M using the same technique described in Figure 2.

[0108] Therefore, similar to the image streaming method in Figure 2, the image streaming method in Figure 7 reduces the amount of data that needs to be transmitted from the cloud server 15 to the entertainment device 10, thereby reducing latency. Furthermore, the image streaming method in Figure 7 further improves integration with existing streaming pipelines by having the cloud server 15 simply transmit downsampled images, with all MAE-specific masking, encoding, and decoding performed on the client side. This method enables seamless integration of the image streaming method in Figure 7 with existing server-side streaming pipelines.

[0109] In some cases, the MAE-based reconstruction approach described here may only be invoked if bandwidth availability falls below a predetermined threshold. In such cases, by default, the original image 202 is sent from the remote computing device to the client device, and only if it is detected that bandwidth availability has fallen below the threshold (and / or latency has exceeded the threshold) is the masked image 202-M sent instead and reconstructed by the client device.

[0110] It will be understood that the technology described herein is applicable to all types of images, including 2D and 3D images. This technology is particularly applicable to virtual reality (VR) or augmented reality (AR) applications where low latency is especially desired.

[0111] Referring to Figure 2, in a schematic embodiment of the present invention, an image streaming method for streaming image content (i.e., images) from a remote computing device 15 to a client device 10 includes the following steps:

[0112] Step 230 includes receiving one or more (partially) masked images from the remote computing device 15 (and in the client device 10). Each masked image comprises a first image and one or more portions (e.g., patches) of the first image that are masked, as described in other parts of this specification.

[0113] Step 240 includes, on the client device 10, reconstructing one or more first images from one or more masked images using a masked autoencoder (MAE) model. The MAE model is trained to reconstruct images from (partially) masked image versions, as described elsewhere in this specification.

[0114] Step 250 includes outputting one or more reconstructed first images (on a client device) for display (to the user), as described later in this specification.

[0115] Those skilled in the art will understand that variations of the above methods corresponding to the operation of various embodiments of the methods and / or apparatus described herein and claimed herein are within the scope of this disclosure. These include, but are not limited to, the following:

[0116] -One or more masked images include multiple masked image frames for content. The step of reconstructing one or more first images from one or more masked images also includes the step of reconstructing multiple image frames in real time from the masked image frames for display to the user, as described herein.

[0117] - The MAE model includes an encoder that encodes multiple masked images into a single latent space representation, and a decoder that decodes the latent space representations of the multiple masked images to reconstruct multiple first images. This is as described elsewhere in this specification.

[0118] -The reception of one or more masked images 230 includes the step of receiving a latent space representation of the masked images. Further reconstruction of one or more initial images 240 is performed using a MAE model decoder, where the MAE encoder is located on a remote computing device, and the MAE decoder is located on a client device. The reception of the masked images 230 includes receiving an encoded masked image (i.e., the reception of the masked images 230 includes receiving an encoded masked image). The above is as described elsewhere in this specification.

[0119] - Receiving one or more masked images 230 includes receiving image data of one or more masked images. Reconstructing one or more first images 240 includes the steps of encoding the image data of one or more masked images into a latent space representation using a MAE encoder, and decoding the latent space representation of one or more masked images using a MAE model decoder to reconstruct one or more first images. That is, both the MAE encoder and the MAE decoder are located in the client device. The above is as described elsewhere in this specification.

[0120] -Step 240 of reconstructing one or more first images from one or more masked images includes the step of supplementing the image data of one or more masked portions and reconstructing the image data of the corresponding portions of the first images. The above is as described elsewhere in this specification.

[0121] -The method further includes the steps of generating one or more masked images on a remote computing device (210) and transmitting one or more masked images from the remote computing device to a client device (220). The above is as described elsewhere in this specification.

[0122] -In this case, the step of generating a masked image may include, if necessary, generating multiple first images and processing multiple first images to mask a portion thereof. This is as described elsewhere in this specification.

[0123] -In this case, optionally, the step of generating one or more masked images further includes the step of selecting one or more portions of the first image to be masked, rendering the unmasked portions of the first image, and at least partially omitting the rendering operation for the masked portions of the first image. The above is as described elsewhere in this specification.

[0124] -In this case, optionally, the method further includes the steps of receiving one or more user inputs from a peripheral device operated by a user of a client device, and generating one or more masked images on a remote computing device based on the user inputs. The foregoing is as described elsewhere in this specification.

[0125] - The step of generating a masked image includes selecting one or more portions of a first image to be masked. Herein, one or more portions are selected based on the user's line of sight of the client device, game context data relating to the portion of the first image from the video game engine (if the image is a video game image), or objects associated with the portion of the image. This is as described elsewhere in this specification.

[0126] -In this case, optionally, the step includes selecting one or more portions of the first image to be masked. This includes setting one or more mask ratios for one or more portions of the first image. This setting depends on the viewing direction or game context data. The above is as described elsewhere in this specification.

[0127] - The step of generating a masked image includes determining a mask ratio for the first image, depending on one or more of the following: the frame rate of the masked image (i.e., image content), the communication bandwidth between the telecomputing device and the client device, and the computational load of the client device. This is as described elsewhere in this specification.

[0128] -In this case, optionally, the method further includes the steps of: a remote computing device selecting an MAE model from a plurality of candidate MAE models based on one or more characteristics of an image; and transmitting the selected MAE model from the remote computing device to the client device for use in reconstructing a first image on the client device. The above is as described elsewhere in this specification.

[0129] -In this case, optionally, the step of generating a masked image includes the step of randomly selecting one or more portions of the first image to be masked (i.e., the masked portions are randomly distributed across the entire first image). This is as described elsewhere in this specification.

[0130] -In this case, optionally, the step of generating the masked image includes a step of uniformly masking a portion of the elements of the first image (i.e., the masked portions are uniformly distributed throughout the first image, for example, by masking 3 pixels every 4 pixels). This is as described elsewhere in this specification.

[0131] -The method further includes the step of displaying one or more reconstructed first images to the user of the client device. The above is as described elsewhere in this specification.

[0132] - Step 230 of receiving a masked image includes, on the client device, receiving a downsampled version of the first image from a remote computing device, and upsampling the downsampled first image by adding new mask elements to the downsampled first image. Here, step 240 of reconstructing the first image includes reconstructing the first image from the upsampled first image using a MAE model. The above is as described elsewhere in this specification.

[0133] -One or more first images are images for a video game. Here, step 250, which outputs a reconstructed first image, includes the step of outputting a reconstructed first image for display to a user of the video game. The above is as described elsewhere in this specification.

[0134] -One or more images are multiple images for content. This is as described elsewhere in this specification.

[0135] -For a given first image, one or more masking regions include at least 50%, preferably at least 70%, and more preferably at least 90% of the first image. The above is as described elsewhere in this specification.

[0136] Referring again to Figure 2, in another schematic embodiment of the present invention, an image streaming method for streaming image content (i.e., images) from a remote computing device 15 to a client device 10 includes the following steps:

[0137] Step 210: Generate one or more masked images on a remote computing device, as described separately herein.

[0138] Step 220: Send one or more masked images from the remote computing device to the client device, as described separately in this specification.

[0139] It will be understood that the above method can be implemented by adding or replacing conventional hardware, or dedicated hardware, as appropriately adjusted by software directives.

[0140] Therefore, necessary adaptations to existing parts of devices equivalent to prior art can be implemented in the form of computer program products containing processor-executable instructions stored on floppy disks, optical disks, hard disks, solid-state disks, PROMs, RAMs, flash memory, or any combination thereof, or as hardware such as ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other configurable circuits, in a form suitable for use in adapting devices equivalent to prior art. Separately, such computer programs can be transmitted via data signals over networks such as Ethernet, wireless networks, the Internet, or any combination thereof.

[0141] Therefore, referring to Figure 1, an example of a conventional device is the entertainment device 10. Based on this, an image streaming system that streams image content from a remote computing device 15 to a client device 10 may include the following:

[0142] The client device 10 includes: an input processing unit (e.g., CPU 20) configured to receive one or more masked images from a remote computing device (e.g., by appropriate software instructions), each masked image containing a first image, with one or more portions of the first image masked; an image reconstruction processor (e.g., CPU 20) configured, for example, by appropriate software instructions, to reconstruct one or more first images from the one or more masked images using a masked autoencoder (MAE) model, the MAE model being trained to reconstruct images from masked versions of images; and an output processing unit (e.g., CPU 20) configured to output one or more reconstructed first images for output (e.g., by appropriate software instructions).

[0143] The image streaming system may further include a remote computing device 15. The remote computing device 15 can be configured (for example, by appropriate software instructions) to generate one or more masked images.

[0144] The above description discloses and illustrates only one embodiment of the present invention. As those skilled in the art will understand, the present invention can be embodied in other specific forms without departing from its spirit or essential features. Accordingly, the disclosure of the present invention is intended to be illustrative and not to limit the scope of the invention or other claims. The disclosures contained herein (including readily identifiable variations of the teachings set forth herein) partially define the scope of the terms of the claims herein and do not attribute any inventive subject matter to the public.

Claims

1. An image streaming method for streaming image content from a remote computing device to a client device. A step of receiving one or more masked images from the remote computing device, wherein the masked images include a first image in which one or more parts are masked. The client device comprises the step of reconstructing the first image from the masked image using a MAE model (Masked Autoencoder Model), wherein the MAE model is trained to reconstruct an image from a masked image. An image streaming method characterized by comprising the step of outputting one or more reconstructed first images for display in the client device.

2. The one or more masked images include multiple masked image frames for content, The image streaming method according to claim 1, characterized in that the step of reconstructing one or more first images from the masked images includes the step of reconstructing a plurality of image frames from the plurality of masked image frames in real time for display to a user.

3. The aforementioned MAE model is The image streaming method according to claim 1 or 2, comprising: an encoder that encodes one or more masked images into a latent space representation; and a decoder that decodes the latent space representations of one or more masked images to reconstruct one or more first images.

4. The step of receiving one or more masked images includes receiving a latent spatial representation of the masked images. The image streaming method according to claim 3, characterized in that the step of reconstructing one or more first images includes the step of reconstructing one or more first images by decoding the latent space representations of one or more masked images using a MAE decoder.

5. The step of receiving one or more masked images includes the step of receiving image data of the one or more masked images, The step of reconstructing one or more first images is: The image streaming method according to claim 3, comprising the steps of: encoding image data of one or more masked images into a latent space representation using a MAE encoder; and decoding the latent space representations of one or more masked images using a MAE decoder to reconstruct one or more first images.

6. The remote computing device includes the steps of generating one or more masked images, The image streaming method according to any one of claims 1 to 5, further comprising the step of transmitting one or more masked images from the remote computing device to the client device.

7. The image streaming method according to claim 6, characterized in that the step of generating one or more masked images includes the steps of generating one or more first images and processing the one or more first images to mask one or more parts of the one or more first images.

8. The step of generating one or more masked images is: The steps include selecting one or more portions of the first image to be masked, The unmasked portion of the selected area will be rendered in the rendering step, The image streaming method according to claim 6, comprising the step of at least partially omitting rendering operations on the masked portion of the selected portion.

9. The steps include receiving one or more user inputs from a peripheral device operated by the user of the client device, The image streaming method according to any one of 6 to 8, further comprising the step of generating one or more masked images based on user input in the remote computing device.

10. The step of generating one or more masked images includes the step of selecting one or more portions of the first image to be masked, The image streaming method according to any one of 6 to 9, characterized in that one or more portions selected as the mask target are selected based on the gaze direction of the user of the client device or game context data from a video game engine relating to the first image.

11. The image streaming method according to any one of 6 to 10, characterized in that the step of generating one or more masked images includes determining a mask ratio for the first image based on one or more of the frame rate of the masked image, the communication bandwidth between the remote computing device and the client device, or the computational load on the client device.

12. The remote computing device selects an MAE model from a plurality of candidate MAE models depending on one or more characteristics of the image. The image streaming method according to any one of 6 to 11, further comprising the step of transmitting a selected MAE model for use in reconstructing the first image on the client device from the remote computing device to the client device.

13. The step of receiving one or more masked images is: The steps include: receiving a downsampled version of the first image from the remote computing device on the client device; The process includes the step of upsampling the downsampled first image by adding a new masked element to the downsampled first image, The image streaming method according to any one of claims 1 to 12, characterized in that the step of reconstructing the first image includes the step of reconstructing the first image from the upsampled first image using an MAE model.

14. The image streaming method according to any one of claims 1 to 13, characterized in that the masked area of ​​the first image is at least 50%, preferably at least 70%, and more preferably at least 90% of the first image.

15. A computer program that includes instructions for causing a computer system to execute the image streaming method described in any one of claims 1 to 12.

16. An image streaming system for streaming image content from a remote computing device to a client device, Equipped with a client device, The aforementioned client device, An input processor that receives one or more masked images from the remote computing device, wherein the masked image includes a first image in which one or more parts are masked. An image reconstruction processor that reconstructs one or more first images from one or more masked images using a MAE model (Masked Autoencoder Model), wherein the MAE model is trained to reconstruct images from masked images. An image streaming system comprising an output processor that outputs one or more reconstructed first images for display.

17. With additional remote computing devices, The image streaming system according to claim 16, characterized in that the remote computing device generates one or more masked images.

Images