Cache cloud rendering
By rendering and caching augmented reality frames on cloud servers and performing cache hits and geometric manipulation on mobile devices, the problem of insufficient processing power of mobile devices is solved, enabling lower motion-to-photon latency and better immersive experiences .
Patent Information
- Application Number
- CN202380034654.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-20
- Filing Date
- 2023-04-20
- Publication Date
- 2024-11-26
AI Technical Summary
When mobile computing devices process and display augmented reality media, battery life and processing power are limited, resulting in increased motion-to-photon delay (M2P time), affecting the immersive experience.
The cloud server is used to render and cache frames, allocate processing workloads, reduce the graphics processing unit (GPU) load of mobile devices, and optimize frame rendering through cache hits and geometric manipulation, and use extended field of view to render large fields of view to reduce latency.
Extends the battery life of mobile devices, reduces M2P time, improves the quality of augmented reality experience, avoids disorders and motion sickness, and enhances immersion.
Smart Images

Figure CN119032383A_ABST
Abstract
Description
[0001] Priority declaration
[0002] This patent application claims the benefit of priority to U.S. Application No. 17 / 725,139, filed on April 20, 2022, which is incorporated herein by reference in its entirety. Background Art
[0003] Augmented reality (AR) is an interactive experience in which the real-world environment is enhanced by computer-generated sensory information. The hardware components used for AR are: processors, displays, sensors, and input devices. Modern mobile computing devices such as smartphones and tablet computers contain these elements, which typically include cameras and micro-electromechanical systems (MEMS) sensors such as accelerometers, GPS, and solid-state compasses, making them suitable AR platforms. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] In the drawings, which are not necessarily drawn to scale, similar reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numeral refer to the figure number in which the element is first introduced. Some embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which:
[0005] Figure 1 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.
[0006] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality according to some examples.
[0007] Figure 3 is a flow chart depicting a method 300 for presenting assessment information according to one embodiment.
[0008] Figure 4 is a flow chart depicting a method 400 for presenting assessment information according to one embodiment.
[0009] Figure 5 Aspects of the subject matter according to one implementation are shown.
[0010] Figure 6 Aspects of the subject matter according to one implementation are shown.
[0011] Fig. 7A is an interface flow chart according to one implementation.
[0012] Figure 7B Aspects of the subject matter according to one implementation are shown.
[0013] Figure 8 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.
[0014] Fig. 9 is a block diagram illustrating a software architecture in which examples may be implemented.
[0015] Fig.10 is a diagrammatic representation of a processing environment according to some examples. DETAILED DESCRIPTION
[0016] As discussed above, mobile computing devices are often used to process and display a wide variety of multimedia, ranging from simple music playback to complex depth and object tracking AR and other graphically rich experiences. While mobile computing devices may be able to generate such content for display, they are severely limited in terms of battery life and maximum processing power. Therefore, in situations where the display of AR media may require a large processing workload, there may be a delay between the position of the user device and the time when that position is processed for display of the AR media. This delay is referred to as motion-to-photon latency, or "M2P time." High M2P times make for a poor augmented or virtual reality experience due to increased lag, and may even cause disorientation and motion sickness, further destroying the immersive experience.
[0017] A system for saving power and rendering time (e.g., for rendering static or dynamic scenes in an AR environment) to reduce M2P time is described herein. The disclosed system can utilize a cloud server to render and cache frames to distribute processing workloads and increase the battery life of a mobile computing device while reducing the workload on a graphics processing unit (GPU) of a mobile computing device (hereinafter referred to as a "user device"). "Rendering" is the process of generating an image from a two-dimensional (2D) or three-dimensional (3D) model. In certain embodiments as discussed herein, a rendered image may be referred to as a "frame". For example, a frame may be a single static image, or in some embodiments may be part of a sequence of frames, such as a video frame. The format of a frame may include a raster format (i.e., JPEG, TIFF, GIF, BMP) and a vector format (CGM, SVG, etc.), as well as one or more live streaming formats (HTTP Live Streaming (HLS), Real-Time Messaging Protocol (RTMP), WebRTC, Secure Reliable Transport (SRT), and Real-Time Streaming Protocol (RTSP)). In some embodiments, a frame may need to be encoded so that a sequence of frames can be received to generate a video stream.
[0018] In some example embodiments, a user device may submit a request to render a frame to a cached cloud rendering system (hereinafter referred to as the "system"), wherein the request includes image data and visual inertial odometry (VIO) data indicating an orientation and position of the user device. In response to receiving the request, the system renders the frame based on the request while accessing a cache pool of previously rendered frames to determine whether the request can be satisfied by performing geometric manipulations on the previously rendered frames. A "cache hit" refers to a situation where a previously rendered frame is identified from the cache of previously rendered frames based on the request and attributes of the previously rendered frame.
[0019] In the event of a cache hit, the system performs geometry manipulation on the identified previously rendered frame based on the VIO data from the request. Warping a previously rendered frame takes less time and is more power efficient than rendering a new frame. Thus, M2P time is improved by reducing the amount of time the system takes to provide the requested frame to the user device.
[0020] In the absence of a cache hit, meaning that the previously rendered frame was not identified from the cache of previously rendered frames, a newly rendered frame initiated in parallel with accessing the cache is returned to the user device. In either scenario, the newly rendered frame can be added to the cache of previously rendered frames regardless of whether the system detects a cache hit.
[0021] In some embodiments, a cache of previously rendered frames ("cache") may be maintained within a cloud server, where the size of the cache may be defined based on properties of the user device. For example, the number of frames maintained within the cache may vary based on factors such as: device resolution; compression type; number of bits; and a rendering configuration, where the rendering configuration defines the number of frames per second displayed at the user device. In some embodiments, the size of the cache may be defined by a system administrator associated with the system.
[0022] The system may be configured to purge one or more frames from the cache of previously rendered frames in response to detecting a trigger condition. In order to conserve system resources, the system may routinely purge old or irrelevant frames from the cache of previously rendered frames. The trigger conditions may include: location conditions; time conditions; and environmental conditions. For example, each frame in the cache of previously rendered frames may include time data, such as a timestamp, indicating the time when the frame was originally rendered. Thus, the frame may be purged from the cache of previously rendered frames when a threshold time period (e.g., 30 seconds, 5 minutes, etc.) from the timestamp expires.
[0023] In some embodiments, the system can be configured such that geometric manipulation operations are performed on previously rendered frames in multiple stages. For example, the system can apply "late warping" in which a sample of motion or posture information, such as VIO data, can be obtained immediately before the user device displays the requested frame, and this information can be used to correct the requested frame to compensate for any changes in device orientation, thereby further reducing M2P time. For example, a first geometric manipulation can be performed on a previously rendered frame at the cloud based on VIO data received in the request, while a second geometric manipulation is performed at the user device in response to receiving a frame from the cloud and based on updated VIO data from the user device.
[0024] A side effect of post-warping may be that the updated frame may require samples outside of the rendered frame. Therefore, in some embodiments, the system may be configured to render the requested frame with an extended field of view that can be displayed from the user device. Typically, when rendering operations are performed locally at the user device, extending the field of view of the rendered frame must be balanced with minimizing the power required for the user device to render the frame and the amount of processing workload to be performed by the GPU, since rendering additional pixels that are not actually displayed would be wasteful. Therefore, by rendering the requested frame on the cloud, a much larger field of view can be rendered without imposing additional processing or power consumption requirements on the user device.
[0025] Thus, example methods and systems relate to a cloud rendering system configured to generate and display a cache of media at a user device. The examples represent possible variations only. Unless otherwise explicitly stated, components and functions are optional and may be combined or subdivided, and operations may vary in order or be combined or subdivided. In the following description, for purposes of illustration, many specific details are set forth to provide a thorough understanding of the example implementations. However, it will be apparent to those skilled in the art that the subject matter may be practiced without these specific details.
[0026] Networked computing environment
[0027] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of user devices 106, each of the multiple instances hosting a number of applications, including a messaging client 108. Each messaging client 108 is communicatively coupled to other instances of the messaging client 108 and a messaging server system 104 via a network 102 (e.g., the Internet).
[0028] The message client 108 is capable of communicating and exchanging data with another message client 108 and a message server system 104 via the network 102. The data exchanged between the message clients 108 and between the message clients 108 and the message server system 104 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).
[0029] The message server system 104 provides server-side functions to specific message clients 108 via the network 102. Although certain functions of the message system 100 are described herein as being performed by the message client 108 or by the message server system 104, the location of certain functions within the message client 108 or within the message server system 104 may be a design choice. For example, it may be technically preferred to initially deploy certain technologies and functions within the message server system 104, but later migrate the technologies and functions to the message client 108 of the user device 106 that has sufficient processing power.
[0030] The message server system 104 supports various services and operations provided to the message clients 108. Such operations include sending data to the message clients 108, receiving data from the message clients 108, and processing data generated by the message clients 108. As examples, the data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the message system 100 is activated and controlled by functions available via the user interface (UI) of the message client 108.
[0031] Turning now specifically to the message server system 104, an application program interface (API) server 112 is coupled to the application server 110 and provides a programming interface to the application server 110. The application server 110 is communicatively coupled to a database server 116, which facilitates access to a database 122 that stores data associated with messages processed by the application server 110. Similarly, a web server 124 is coupled to the application server 110 and provides a web-based interface to the application server 110. To this end, the web server 124 processes incoming network requests via the hypertext transfer protocol (HTTP) and several other related protocols. In some embodiments, the database 122 may include a decentralized database.
[0032] The application program interface (API) server 112 receives and sends message data (e.g., commands and message payloads) between the user device 106 and the application server 110. Specifically, the application program interface (API) server 112 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the message client 108 to activate the functions of the application server 110. The application program interface (API) server 112 exposes various functions supported by the application server 110, including: account registration; login function; sending messages from a specific message client 108 to another message client 108 via the application server 110; sending media files (e.g., images or videos) from the message client 108 to the message server 114 and for possible access by another message client 108; setting of media data collections (e.g., stories); retrieving the friend list of the user of the user device 106; retrieving such collections; retrieving messages and content; adding and removing entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the message client 108).
[0033] The application server 110 hosts a plurality of server applications and subsystems, including, for example, a message server 114, an image processing server 118, and a social network server 120. The message server 114 implements a plurality of message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from a plurality of instances of the message client 108. As will be described in further detail, text and media content from a plurality of sources may be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the message client 108. Given the hardware requirements of other processor and memory intensive data processing, such processing may also be performed on the server side by the message server 114.
[0034] The application server 110 also includes an image processing server 118 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 114 .
[0035] The social network server 120 supports various social networking functions and services and makes these functions and services available to the messaging server 114. Examples of functions and services supported by the social network server 120 include identifying other users of the messaging system 100 who have relationships with a particular user or who the particular user is "following," as well as also identifying interests and other entities of a particular user.
[0036] System Architecture
[0037] Figure 2 1 is a block diagram showing additional details about the messaging system 100 according to some examples. Specifically, the messaging system 100 is shown to include a messaging client 108 and an application server 110. The messaging system 100 contains a number of subsystems that are supported on the client side by the messaging client 108 and on the server side by the application server 110. These subsystems include, for example, a transient timer system 202, a collection management system 204, an augmentation system 206, a map system 210, a game system 212, and a cached cloud rendering system 214.
[0038] The transient timer system 202 is responsible for implementing temporary or time-limited access to content by the message client 108 and the message server 114. The transient timer system 202 includes a plurality of timers that selectively enable access (e.g., for presentation and display) to messages and associated content via the message client 108 based on duration and display parameters associated with a message or a collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 202 are provided below.
[0039] The collection management system 204 is responsible for managing collections or collections of media (e.g., collections of text, image video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of an event related to the content). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing an icon that provides notification of the existence of a particular collection to the user interface of the messaging client 108.
[0040] In addition, the collection management system 204 also includes a curation interface 208, which enables collection managers to manage and curate specific content collections. For example, the curation interface 208 enables event organizers to curate collections of content related to specific events (e.g., deleting inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to users for including user-generated content in a collection. In such a case, the collection management system 204 operates to automatically pay such users for using their content.
[0041] The enhancement system 206 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. For example, the enhancement system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The enhancement system 206 is operable to provide media overlays or enhancements (e.g., image filters) to the messaging client 108 based on the geographic location of the user device 106. In another example, the enhancement system 206 is operable to provide media overlays to the messaging client 108 based on other information such as social network information of the user of the user device 106. The media overlays may include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) at the user device 106. For example, the media overlay may include text or images that may be superimposed on top of a photo taken by the user device 106. In another example, the media overlay includes a location identification overlay (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Cafe). In another example, the augmentation system 206 uses the geographic location of the user device 106 to identify a media overlay that includes the name of a business at the geographic location of the user device 106. The media overlay may include other logos associated with the business. The media overlay may be stored in the database 122 and accessed through the database server 116.
[0042] In some examples, the enhancement system 206 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify situations in which a particular media overlay should be provided to other users. The enhancement system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.
[0043] In other examples, the enhancement system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations via a bidding process. For example, the enhancement system 206 associates the media overlays of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.
[0044] The mapping system 210 provides various geo-location functions and supports the presentation of map-based media content and messages by the messaging client 108. For example, the mapping system 210 enables the display of user icons or avatars on a map to indicate the current or past locations of the user's "friends", as well as media content (e.g., a collection of messages including photos and videos) generated by such friends within the context of the map. For example, a message posted by a user to the messaging system 100 from a particular geographic location can be displayed to a particular user's "friends" on the mapping interface of the messaging client 108 within the context of that particular location on the map. The user can also share his or her location and status information with other users of the messaging system 100 (e.g., using an appropriate status avatar) via the messaging client 108, where the location and status information is similarly displayed to selected users within the context of the mapping interface of the messaging client 108.
[0045] The gaming system 212 provides various gaming functions within the context of the messaging client 108. The messaging client 108 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the messaging client 108 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite other users to participate in playing a particular game by sending an invitation to such other users from the messaging client 108. The messaging client 108 also supports both voice and text messaging (e.g., chat) within the context of gaming, provides leaderboards for gaming, and also supports providing in-game rewards (e.g., gaming coins and items).
[0046] Functionality provided by the cached cloud rendering system 214 may include: receiving a request for a rendered frame from a user device, the request including at least image data and device orientation information such as gyroscope data, accelerometer data, and VIO data; generating a cache of previously rendered frames; accessing the cache of previously rendered frames in response to the request; identifying a rendered frame from the cache of previously rendered frames based on the image data; applying a transformation to the rendered frame based on the VIO data; and causing a presentation of the rendered frame to be displayed at the user device based on the transformation.
[0047] Figure 3 1 is a flow chart illustrating the operation of the cached cloud rendering system 214 in performing a method 300 for causing a rendered frame to be displayed at a user device according to one embodiment. The operation of the method 300 may be described above with respect to Figure 2 The described messaging system 100 may be executed by one or more subsystems such as the cached cloud rendering system 214. Figure 3 As shown in , method 300 includes one or more operations 302 , 304 , 306 , 308 , and 310 .
[0048] At operation 302, the cached cloud rendering system 214 receives a request to render a frame from the user device 106, wherein the request includes at least image data, device orientation information such as VIO data, and media data of a media object in some embodiments. For example, a user of the user device 106 may provide input to present AR content, and in response to the input, the user device 106 may send a request to the cached cloud rendering system 214 to render a frame based on the content of the request. Thus, a frame may be rendered based on image data, device orientation information, and media data.
[0049] In response to receiving a request to render a frame from the user device 106, the cached cloud rendering system 214 accesses a cache of previously rendered frames (i.e., the database 122) at operation 304. For example, the cached cloud rendering system 214 can maintain the database 122 and routinely clear and update the database 122 with frames rendered by the cached cloud rendering system 214.
[0050] At operation 306, the cached cloud rendering system 214 identifies a previously rendered frame from the cache of previously rendered frames based on the attributes of the request received from the user device 106. In some implementations, the cached cloud rendering system 214 may identify the previously rendered frame based on a similarity of the previously rendered frame to the requested frame.
[0051] For example, each frame in the cache of previously rendered frames may include a set of attributes, such as: metadata; image features; device orientation data; location data; and time data. The cached cloud rendering system 214 may perform a comparison of the set of attributes for each frame from the cache of previously rendered frames and a corresponding set of attributes associated with the request to identify "similar" frames.
[0052] At operation 308, the cached cloud rendering system 214 applies a transformation to the previously rendered frame based on the device orientation information received in the request from the user device 106. For example, the cached cloud rendering system 214 may apply one or more geometric transformations to the identified frame so that the pixels of the frame can be relocated from their (x, y) space coordinates to a new set of coordinates, where the new set of coordinates can be based on the device orientation data received in the request. The geometric transformation can thus move (translate), rotate, and scale the geometry of the frame. The transformations that can be performed by the cached cloud rendering system 214 may include: geometric transformations, such as translation, scaling, rotation, and affine; perspective transformations; transposition; shearing; and distortion.
[0053] At operation 310, the cached cloud rendering system 214 may cause display of a rendering of the previously rendered frame at the user device 106 based on the transformation. In some implementations, causing display of the rendering of the previously rendered frame may include applying an additional transformation to the previously rendered frame.
[0054] For example, in some embodiments, the cached cloud rendering system 214 can enable the user device 106 to access updated device orientation information (e.g., VIO data) in response to the user device 106 receiving a previously rendered frame to apply post-warping to the previously rendered frame to correct for any last-minute changes in the device orientation.
[0055] Figure 4 4 is a flow chart illustrating the operation of the cached cloud rendering system 214 when performing a method 400 for adding a rendered frame to a cache of previously rendered frames according to one embodiment. The operation of the method 400 may be described above with respect to Figure 2 The described messaging system 100 may be executed by one or more subsystems such as the cached cloud rendering system 214. Figure 4 As shown, method 400 includes one or more operations 402 , 404 , and 406 .
[0056] At operation 402, as in operation 302 of method 300, the cached cloud rendering system 214 receives a request to render a frame from the user device 106. In some implementations, operation 404 of method 400 may be performed concurrently with operation 304 of method 300, where the cached cloud rendering system 214 accesses a cache of previously rendered frames (i.e., database 122).
[0057] At operation 404, the cached cloud rendering system 214 generates a new rendered frame based on the request received from the user device 106. The new rendered frame can be generated based on the image data from the request and other information that can be provided in the request, such as the identification of the media object and device orientation information. In some embodiments, operation 404 can be performed simultaneously with operation 304 of method 300.
[0058] At operation 406 , the cached cloud rendering system 214 adds the new rendered frame to the cache of previously rendered frames (ie, the database 122 ).
[0059] Figure 5 1 is a flow chart illustrating the operation of the cached cloud rendering system 214 in executing the method 500 for generating and clearing a cache of previously rendered frames according to one embodiment. The operation of the method 500 may be described above with respect to Figure 2 The described messaging system 100 may be executed by one or more subsystems such as the cached cloud rendering system 214. Figure 5 As shown, method 500 includes one or more operations 502 and 504 .
[0060] According to certain example implementations discussed herein, the cached cloud rendering system 214 maintains and periodically updates a cache of previously rendered frames. The amount of data stored in the cache or the number of cached frames may depend on properties of the user device 106, such as the display resolution of the user device 106 or a rendering configuration that may be provided to the cached cloud rendering system 214 by the user device 106. For example, the rendering configuration may provide indications of: frames per second (e.g., 20fps, 30fps, 60fps); rendering format; number of bits; device type; and compression type.
[0061] To ensure that frames stored in the cache of previously rendered frames are relevant to requests that may be received from the user device 106, the cached cloud rendering system 214 may routinely purge or update frames from the cache of previously rendered frames. Thus, at operation 502, the cached cloud rendering system 214 may detect a trigger condition, where the trigger condition may be based on one or more of: time data; location data; image attributes; and a result of a previous request for a rendered frame from the user device 106, where the result may be based on whether a cache hit was detected. Thus, at operation 504, the cached cloud rendering system 214 may purge one or more frames from the cache of previously rendered frames in response to detecting the trigger condition.
[0062] For example, each frame in the cache of previously rendered frames may include: image data; time data, such as a timestamp; orientation data, such as VIO data; and position data. In response to receiving a request from the user device 106, the cached cloud rendering system 214 may perform a comparison of the attributes of each frame in the cache of previously rendered frames with the corresponding attributes of the request to determine whether there is a cache hit. When it is determined that there is no cache hit (i.e., a cache miss), the cached cloud rendering system 214 may clear one or more frames from the cache of previously rendered frames.
[0063] In some embodiments, the trigger condition can be based on a comparison of image features of the image data from the request with corresponding image features of frames in the cache from previously rendered frames. For example, upon determining that the image features of the image data from the request are different from the image features of the frames in the cache from previously rendered frames, the cached cloud rendering system 214 can clear the cache of one or more frames and update the cache with the newly rendered frames.
[0064] In some implementations, the trigger condition may be the expiration of a threshold time period. For example, each frame in the cache from previously rendered frames may include a timestamp. Upon determining that the threshold time period has passed the time indicated by the timestamp, the cached cloud rendering system 214 may clear the frame and update the cache with the newly rendered frame.
[0065] In some implementations, the trigger condition may be based on the location of the user device 106. For example, each frame in the cache of previously rendered frames may include location data, and the request received from the user device 106 may include location data indicating the current location of the user device 106. Upon receiving the request to render a frame from the user device 106, the cached cloud rendering system 214 may determine that the location of the user device 106 is greater than a threshold distance from the location indicated by one or more frames from the cache of previously rendered frames. Upon determining that the location of the user device 106 is greater than the threshold distance from the location associated with one or more frames from the cache of previously rendered frames, the cached cloud rendering system 214 may clear the cache of one or more frames.
[0066] Figure 6 600 is a flowchart illustrating the operation of the cached cloud rendering system 214 in response to a request received from a user device 106 according to one embodiment. The operations depicted in the flowchart 600 may be described above with respect to Figure 2 The described messaging system 100 may be executed by one or more subsystems such as the cached cloud rendering system 214 .
[0067] like Figure 6 As shown, at operation 602, the user device 106 sends a frame request to the cloud server. In response to receiving the request from the user device 106, at operation 604, the cached cloud rendering system 214 determines whether the cache of previously rendered frames can satisfy the request, and at operation 606, concurrently with operation 604, the cached cloud rendering system 214 begins rendering the requested frame to add the rendered frame to the cache of previously rendered frames at operation 608.
[0068] The cached cloud rendering system 214 determines whether there is a cache hit. If there is a cache hit, and the cache of previously rendered frames can satisfy the frame request, then at operation 612, the cached cloud rendering system 214 performs geometry manipulation on the cached frame identified from within the cache of previously rendered frames, and at operation 614, returns the cached frame to the user device 106 for display.
[0069] However, if there is a cache miss, then at operation 610 , the cached cloud rendering system 214 returns the frame rendered in operation 606 to the user device 106 for display.
[0070] Fig. 7A is a diagram 700a depicting a method of rendering a portion of a frame rendered by a cached cloud rendering system 214 using an extended field of view, according to one embodiment.
[0071] As shown in graph 700a, rendered frame 702 may be rendered using an extended field of view 704 that is larger than a maximum display field of view 708 of displayed frame 710 at user device 106. Thus, based on view direction 706, displayed frame 710 may correspond to a portion of rendered frame 702.
[0072] Figure 7B 700b is a diagram depicting a method of performing geometry manipulation on a frame rendered by a cached cloud rendering system 214 with an extended view, according to one embodiment.
[0073] For example, as discussed above, in some embodiments, the cached cloud rendering system 214 can be configured to render the requested frame using an extended field of view that can be displayed by the user device so that geometric manipulations can be applied to the rendered frame to display a portion of the rendered frame in response to a subsequent request received from the user device 106.
[0074] Therefore, if Figure 7B As seen in FIG. 700 b , the cached cloud rendering system 214 may access a previously rendered frame 714 , where the previously rendered frame 714 is rendered using an extended field of view. The cached cloud rendering system 214 may then perform geometric manipulation on the previously rendered frame 714 , such as by changing the field of view angle 712 and the view direction 716 , to generate a displayed frame 718 based on a portion of the previously rendered frame 714 .
[0075] Machine Architecture
[0076] Figure 8800 within which instructions 810 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 800 to perform any one or more of the methods discussed herein. For example, the instructions 810 may cause the machine 800 to perform any one or more of the methods described herein. The instructions 810 convert a general purpose, unprogrammed machine 800 into a specific machine 800 that is programmed to perform the functions described and illustrated in the manner described. The machine 800 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 may operate in the performance of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 810 specifying actions to be taken by the machine 800. In addition, although only a single machine 800 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 810 to perform any one or more of the methods discussed herein. For example, the machine 800 may include a user device 106 or any of several server devices that form part of the message server system 104. In some examples, the machine 800 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.
[0077] The machine 800 may include a processor 804, a memory 806, and input / output I / O components 638 that may be configured to communicate with each other via a bus 840. In an example, the processor 804 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 808 and a processor 812 that execute instructions 810. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 8Multiple processors 804 are shown, but the machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.
[0078] The memory 806 includes a main memory 814, a static memory 816, and a storage unit 818, all of which are accessible by the processor 804 via the bus 840. The main memory 806, the static memory 816, and the storage unit 818 store instructions 810 that embody any one or more of the methodologies or functions described herein. The instructions 810 may also reside, in whole or in part, within the main memory 814, within the static memory 816, within the machine-readable medium 820 within the storage unit 818, within at least one of the processors 804 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 800.
[0079] The I / O components 802 may include various components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 802 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be appreciated that the I / O components 802 may include Figure 8 820 , and the like. In various examples, the I / O component 802 may include a user output component 826 and a user input component 828. The user output component 826 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 828 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a pointing-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.
[0080] In other examples, the I / O component 802 may include a biometric component 830, a motion component 832, an environment component 834, or a position component 836, as well as various other components. For example, the biometric component 830 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 832 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).
[0081] Environmental components 834 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., an infrared sensor that detects nearby objects), gas sensors (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.
[0082] With respect to cameras, user device 106 may have a camera system including, for example, a front-facing camera on the front surface of user device 106 and a rear-facing camera on the rear surface of user device 106. The front-facing camera may, for example, be used to capture still images and videos of a user of user device 106 (e.g., “selfies”), which may then be enhanced using the enhancement data (e.g., filters) described above. The rear-facing camera may, for example, be used to capture still images and videos in a more traditional camera mode, which may similarly be enhanced using the enhancement data. In addition to the front-facing camera and the rear-facing camera, user device 106 may also include a 360° camera for capturing 360° photos and videos.
[0083] Additionally, the camera system of user device 106 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or penta rear camera configurations on the front and back sides of user device 106. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.
[0084] The location component 836 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or a barometer that detects air pressure from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.
[0085] A variety of technologies can be used to achieve communication. The I / O component 802 also includes a communication component 838 that is operable to couple the machine 800 to the network 822 or device 824 via a corresponding coupling or connection. For example, the communication component 838 may include a network interface component or other suitable device that interfaces with the network 822. In other examples, the communication component 838 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low power consumption), Device 824 may be another machine or any of a variety of peripherals (eg, a peripheral coupled via USB).
[0086] In addition, the communication component 838 can detect the identifier or include a component that is capable of operating to detect the identifier. For example, the communication component 838 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a data matrix, a data symbol (Dataglyph), a MaxiCode, PDF417, an Ultra Code, a UCC RSS-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 838, such as a location obtained via an Internet Protocol (IP) geolocation, a location obtained via a NFC smart tag detection component, and a location obtained via a NFC smart tag detection component. Location derived from signal triangulation, location derived via detection of NFC beacon signals that can indicate a specific location, etc.
[0087] Various memories (e.g., main memory 814, static memory 816, and memory of processor 804) and storage unit 818 may store one or more sets of instructions and data structures (e.g., software) that implement or are used by any one or more of the methods or functions described herein. When executed by processor 804, these instructions (e.g., instructions 810) cause various operations to implement the disclosed examples.
[0088] The instructions 810 may be sent or received via a network interface device (e.g., a network interface component included in the communication component 838) using a transmission medium and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over the network 822. Similarly, the instructions 810 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with the device 824 using a transmission medium.
[0089] Software Architecture
[0090] Fig. 9 900 is a block diagram illustrating a software architecture 904 that may be installed on any one or more of the devices described herein. The software architecture 904 is supported by hardware, such as a machine 902 including a processor 920, a memory 926, and an I / O component 938. In this example, the software architecture 904 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 904 includes layers such as an operating system 912, a library 910, a framework 908, and an application 906. In operation, the application 906 invokes an API call 950 through the software stack and receives a message 952 in response to the API call 950.
[0091] The operating system 912 manages hardware resources and provides public services. The operating system 912 includes, for example, a kernel 914, a service 916, and a driver 922. The kernel 914 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 914 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. The service 916 can provide other public services for other software layers. The driver 922 is responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, the driver 922 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.
[0092] The library 910 provides a common low-level infrastructure used by the application 906. The library 910 may include a system library 918 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 910 may include an API library 924, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 910 may also include various other libraries 928 to provide many other APIs to the application 906.
[0093] The framework 908 provides a common high-level infrastructure used by the applications 906. For example, the framework 908 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 908 can provide a wide range of other APIs that can be used by the applications 906, some of which can be specific to a particular operating system or platform.
[0094] In an example, applications 906 may include a home application 936, a contacts application 930, a browser application 932, a book reader application 934, a positioning application 942, a media application 944, a messaging application 946, a game application 948, and a variety of other applications such as third-party applications 940. Application 906 is a program that executes the functions defined in the program. Various programming languages can be used to create one or more of the applications 906 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party applications 940 (e.g., provided by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or other mobile operating system. In this example, the third-party application 940 can activate the API call 950 provided by the operating system 912 to facilitate the functions described in this article.
[0095] Processing components
[0096] Now go to Fig.10 , a diagrammatic representation of a processing environment 1000 is shown including a processor 1002, a processor 1006, and a processor 1008 (eg, a GPU, a CPU, or a combination thereof).
[0097] According to embodiments discussed herein, the processor 1002 is shown coupled to a power source 1004 and is shown to include a processor 1002 that is operatively configured to perform the following operations: Figure 3 Method 300, Figure 4 Method 400, Figure 5 Method 500 and Figure 6 The (permanently configured or temporarily instantiated) modules of the operations discussed in flowchart 600, namely, X component 1010, Y component 1012, and Z component 1014.
[0098] Glossary
[0099] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.
[0100] "Communications network" refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.
[0101] "Component" refers to a logic, device or physical entity with boundaries defined by function or subroutine calls, branch points, APIs or other technical definitions that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be packaged functional hardware units designed for use with other components and can be part of a program that generally performs specific functions in related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. The hardware component can be a special processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It should be recognized that it can be determined for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a circuit system of a temporary configuration (e.g., configured by software). Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering the implementation of the hardware component being temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured into different special-purpose processors (e.g., including different hardware components) at different times. Software accordingly configures one or more specific processors to, for example, constitute a specific hardware component at one time and constitute different hardware components at different times. Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components (e.g., by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware component can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors 1004 or the parts implemented by the processor. In addition, the one or more processors can also operate to support the execution of related operations in the "cloud computing" environment or as "software as a service" (SaaS) operation. For example, at least some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations can be distributed between processors, not only resident in a single machine, but deployed across several machines. In some example embodiments, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other example embodiments, a processor or a part implemented by a processor can be distributed across several geographical locations.
[0102] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.
[0103] "Ephemeral messages" are messages that are accessible for a limited duration. Ephemeral messages can be text, images, videos, etc. The access time for ephemeral messages can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is transient.
[0104] "Machine storage media" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media", "device storage media", and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media", "computer storage media", and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media (at least some of which are encompassed by the term "signal media").
[0105] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.
[0106] "Signal medium" refers to any intangible medium that can store, encode or carry instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.
Claims
1. A method comprising: receiving a request for a rendered frame from a user device, the request comprising at least image data and visual inertial odometry (VIO) data; accessing a cache of previously rendered frames in response to the request; identifying a rendered frame from the cache of previously rendered frames based on the image data; applying a transform to the rendered frame based on the VIO data; as well as A presentation of the rendered frame is caused to be displayed at the user device based on the transformation.
2. The method according to claim 1, further comprising: In response to the request, generating a new rendered frame based on the image data and the VIO data, the generating the new rendered frame being performed simultaneously with accessing the cache of the previously rendered frame; as well as The newly rendered frame is added to the cache of previously rendered frames.
3. The method according to claim 1, wherein: The method further comprises: The cache of previously rendered frames is cleared in response to a trigger condition.
4. The method according to claim 3, wherein: The request is a first request including first image data, the first image data including a first set of image characteristics, and clearing the cache of the previously rendered frame further comprises: receiving a second request including second image data, the second image data including a second set of image features; performing a comparison of the first set of image features with the second set of image features; and A cache of the previously rendered frames is cleared based on the comparison.
5. The method according to claim 3, wherein: The presentation of the assessment includes display of the first value and the second value.
6. The method according to claim 1, wherein: The transform is a first transform, and causing presentation of the rendered frame to be displayed at the user device further comprises: causing the user device to generate updated VIO data; and The user device is caused to apply a second transform to the rendered frame based on the updated VIO data.
7. The method according to claim 1, wherein: The size of the cache of previously rendered frames is based on properties of the user device.
8. A system comprising: one or more processors; as well as a memory comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a request for a rendered frame from a user device, the request comprising at least image data and visual inertial odometry (VIO) data; accessing a cache of previously rendered frames in response to the request; identifying a rendered frame from the cache of previously rendered frames based on the image data; applying a transform to the rendered frame based on the VIO data; and A presentation of the rendered frame is caused to be displayed at the user device based on the transformation.
9. The system according to claim 8, wherein: The operations also include: In response to the request, generating a new rendered frame based on the image data and the VIO data, the generating the new rendered frame being performed concurrently with accessing the cache of the previously rendered frame; and The newly rendered frame is added to the cache of previously rendered frames.
10. The system according to claim 8, wherein: The operations also include: The cache of previously rendered frames is cleared in response to a trigger condition.
11. The system according to claim 10, wherein: The request is a first request including first image data, the first image data including a first set of image characteristics, and clearing the cache of the previously rendered frame further comprises: receiving a second request including second image data, the second image data including a second set of image features; performing a comparison of the first set of image features with the second set of image features; and A cache of the previously rendered frames is cleared based on the comparison.
12. The system according to claim 8, wherein: The user device comprises a display comprising a first resolution, and wherein the rendered frame comprises a second resolution, wherein the second resolution is greater than the first resolution.
13. The system according to claim 8, wherein: The transform is a first transform, and causing presentation of the rendered frame to be displayed at the user device further comprises: causing the user device to generate updated VIO data; and The user device is caused to apply a second transform to the rendered frame based on the updated VIO data.
14. The system according to claim 8, wherein: The size of the cache of previously rendered frames is based on properties of the user device.
15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: receiving a request for a rendered frame from a user device, the request comprising at least image data and visual inertial odometry (VIO) data; accessing a cache of previously rendered frames in response to the request; identifying a rendered frame from the cache of previously rendered frames based on the image data; applying a transform to the rendered frame based on the VIO data; as well as A presentation of the rendered frame is caused to be displayed at the user device based on the transformation.
16. The non-transitory machine-readable storage medium of claim 15, wherein: The operations also include: In response to the request, generating a new rendered frame based on the image data and the VIO data, the generating the new rendered frame being performed concurrently with accessing the cache of the previously rendered frame; and The newly rendered frame is added to the cache of previously rendered frames.
17. The non-transitory machine-readable storage medium of claim 15, wherein: The operations also include: The cache of previously rendered frames is cleared in response to a trigger condition.
18. The non-transitory machine-readable storage medium of claim 17, wherein: The request is a first request including first image data, the first image data including a first set of image characteristics, and clearing the cache of the previously rendered frame further comprises: receiving a second request including second image data, the second image data including a second set of image features; performing a comparison of the first set of image features with the second set of image features; and A cache of the previously rendered frames is cleared based on the comparison.
19. The non-transitory machine-readable storage medium of claim 15, wherein: The user device comprises a display comprising a first resolution, and wherein the rendered frame comprises a second resolution, wherein the second resolution is greater than the first resolution.
20. The non-transitory machine-readable storage medium of claim 15, wherein: The transform is a first transform, and causing presentation of the rendered frame to be displayed at the user device further comprises: causing the user device to generate updated VIO data; and The user device is caused to apply a second transform to the rendered frame based on the updated VIO data.
Citation Information
Cited By
Cached cloud rendering
US12646135B2
