Technologies used to manage the generation and rendering of user interfaces on client devices

Abstract

Techniques and systems for processing user interface content are provided. For example, a server computer may receive a user interface event corresponding to a user interface of a device (e.g., a client device, another server computer, or other device). An application associated with the user interface event may be determined, and an interface model may be generated using the application associated with the user interface event. The interface model defines state information of one or more graphical objects of the user interface. The state information is obtained from the user interface event. The server computer may send the interface model to the device, which enables the device to render the user interface.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Patent Application No. 62 / 770,973, filed on November 23, 2018, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to techniques and systems for managing the generation and rendering of user interfaces, and more particularly to improving the efficiency of generating and rendering user interfaces using a server-based system. Summary of the invention

[0004] In some examples, described herein are techniques and systems for implementing a user interface, for which generation and rendering are managed by a centralized or server-based computer system (e.g., a cloud-based server system, a distributed computer system, or other remote computer-based system) away from a client device. A server-based system can implement techniques for managing user interfaces across multiple devices, thereby improving the efficiency of managing a common user interface across a system with many client devices. The user interface may include any appropriate type of physical or virtual user interface that can facilitate receiving input via an input device, such as any type of graphical user interface (GUI), an electronic program guide (EPG), or a content guide for accessing media content, a user interface for a game application, and / or any other appropriate type of user interface. In some cases, a user interface may be generated for each application on a client device. For example, a client device may display a first user interface of a first application, a second user interface of a second application, a third user interface of a third application, etc.

[0005] The server-based user interface can be implemented and / or managed in part by a server system, in part by a client device or a combination thereof, so as to render a user interface. For example, the server system can determine the graphical elements (also referred to as graphical objects) to be displayed, where and how to display different graphical elements, how to react to user interface events (also referred to as interface events), and other user interface actions. The client device can relay the interface event to the server system, and can render graphics for the user interface based on the user interface layout information received from the server system and other attributes associated with the user interface (for example, an interface model as described herein). The server system may include one or more server computers. In some examples, the server system may include a cloud-based computer system. The client device may include any appropriate device, for example, a media device (for example, a set-top box or other console device), a television connected to the Internet, a mobile device (for example, a cellular phone, a laptop computer, a tablet computer, a wearable device or other mobile device). The client device can be connected to the server computer system using a wired or wireless network.

[0006] The client device can send interface events to the server system for processing. The interface events can be generated by the client device based on user input received through the user interface (e.g., events specific to the application to which the user interface applies), based on system-level events (e.g., events related to the user interface at the client device, such as volume adjustments or changes to dynamic interfaces), and / or based on application-level events. The server system can identify the application and / or system to which the received user interface event applies (e.g., for system-level events), and can determine any changes in the user interface of the application and / or system based on the received interface event.

[0007] In some examples, the server system may generate a user interface model (also referred to herein as an interface model) that describes how the user interface should look at any time. For example, the interface model may be generated based on received user interface events (e.g., one or more key presses, finger movements, device orientation and / or position, etc.). In some cases, the interface model may be based on other events, such as other remote client events or events received from devices or systems other than the client device. For example, other events may be based on events such as a video being started or ended, video access control being denied, a phone ringing, receiving an email or other message, etc. In some cases, the server system may generate an interface model that defines state information (e.g., the user interface layout and / or other properties of one or more graphic objects) for one or more graphic objects of the user interface. In some implementations, the interface model may be defined in whole or in part by a structure or syntax that may define or represent the logical structure of a document (e.g., the structure of an application's user interface), as well as the way in which the document is accessed, generated, rendered, and / or manipulated. For example, the interface model may include a document object model (DOM) that defines a structure for presenting components including a user interface.

[0008] In some cases, the server system can improve the efficiency of generating and rendering the user interface at the client device. In at least one embodiment, the server system can optimize the interface model before sending the interface model to the client device. For example, the server system can determine the difference between the current user interface (e.g., the current state information of the user interface, such as the current state information of the current user interface, the current attributes of the graphical elements, etc.) and the previous user interface (e.g., the previous state information generated before the current state information of the current user interface), and can generate an intermediate interface model (also referred to herein as an "incremental interface model") indicating the difference or delta between the current user interface and the previous user interface. The server system can send the incremental interface model to the client device, indicating only the difference, rather than sending the entire interface model that describes the state information of the entire user interface.

[0009] The interface model or the incremental interface model can be sent from the server system to the client device. The graphics rendering engine of the client device can render the user interface based on the received interface model. For example, the client device can read the interface model received for a given user interface, and can render the interface defined by the interface model. The client device can update the user interface based on the increment indicating the properties and / or functions of the changed user interface. In some cases, if the user interface changes by a certain amount (e.g., a complete change of the user interface), the interface model can indicate the properties and / or functions of the reusable user interface. The use of the interface model enables the client device to avoid determining the state information of the user interface (e.g., attributes such as layout) or the processing of the change of the state information, for example, performing layout calculations. For example, the server system can pre-calculate the properties of the graphic elements represented by the interface model (e.g., graphic element position, size, color and / or other properties), so that the client device does not need to determine the user interface state information (e.g., perform layout calculations, graphic attribute calculations, etc.) and other processing required to determine the properties of the user interface. Instead, the interface model is transmitted to the client device, and the client device can implement a routine (e.g., a graphics routine) to generate a user interface based on the interface model. For example, using the interface model, a client can convert a tree of graphics objects from the interface model into a list of graphics routine calls.

[0010] According to at least one example, a method for processing user interface content is provided. The method includes receiving a user interface event by a server computer. The user interface event corresponds to a user interface of a client device. The method also includes determining an application associated with the user interface event and generating an interface model using the application associated with the user interface event. The interface model defines state information for one or more graphical objects of the user interface. The state information is obtained from the user interface event. The method also includes sending the interface model by the server computer to the client device. The interface model enables the client device to render the user interface.

[0011] In another example, a system for processing user interface content is provided. The system includes one or more processors and a memory accessible to the one or more processors and storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: receive a user interface event, the user interface event corresponding to a user interface of a client device; determine an application associated with the user interface event; generate an interface model using the application associated with the user interface event, wherein the interface model defines state information for one or more graphical objects of the user interface, the state information being obtained from the user interface event; and send the interface model to the client device, the interface model enabling the client device to render the user interface.

[0012] In another example, a non-transitory computer-readable medium of a server computer stores instructions thereon, which, when executed by one or more processors, cause the one or more processors to: receive a user interface event, the user interface event corresponding to a user interface of a client device; determine an application associated with the user interface event; generate an interface model using the application associated with the user interface event, wherein the interface model defines state information for one or more graphical objects of the user interface, the state information being obtained from the user interface event; and send the interface model to a client device, the interface model enabling the client device to render the user interface.

[0013] In some aspects, the above method, apparatus and computer-readable medium may further include compressing the interface model. The interface model sent to the client device is a compressed interface model.

[0014] In some aspects, the interface model is a first interface model, and compressing the interface model includes: determining a difference between state information of one or more graphical objects and previous state information of the one or more graphical objects; and generating a second interface model, the second interface model including data of the one or more graphical objects having state information different from the previous state information.

[0015] In some cases, generating the interface model includes generating a portion of the interface model. The portion of the interface model is associated with a portion of the user interface.

[0016] In some aspects, the above methods, apparatus, and computer-readable media may also include: determining that the application is triggered by a client device; determining that the application is not running on the server computer; and adding the application to a list of applications accessible to multiple client devices using the server computer.

[0017] According to at least another example, a method for processing user interface content is provided. The method includes receiving a user interface event by a server computer. The user interface event corresponds to a user interface of a client device. The method also includes determining an application associated with the user interface event and generating an interface model using the application associated with the user interface event. The interface model defines state information for one or more graphical objects of the user interface. The state information is obtained from the user interface event. The method also includes determining a difference between the state information of the one or more graphical objects and previous state information of the one or more graphical objects. The method also includes generating a second model, the second model including data of the one or more graphical objects having state information different from the previous state information. The method also includes sending a second interface model to the client device by the server computer. The second interface model enables the client device to render the user interface.

[0018] In another example, a system for processing user interface content is provided. The system includes one or more processors and a memory accessible to the one or more processors and storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: receive a user interface event, the user interface event corresponding to a user interface of a client device; determine an application associated with the user interface event; generate an interface model using the application associated with the user interface event, wherein the interface model defines state information for one or more graphical objects of the user interface, the state information being obtained from the user interface event; determine a difference between the state information of the one or more graphical objects and previous state information of the one or more graphical objects; generate a second model, the second model including data of the one or more graphical objects having state information different from the previous state information; and send the second interface model to the client device, the second interface model enabling the client device to render the user interface.

[0019] In another example, a non-transitory computer-readable medium of a server computer stores instructions thereon, which, when executed by one or more processors, cause the one or more processors to: receive a user interface event, the user interface event corresponding to a user interface of a client device; determine an application associated with the user interface event; generate an interface model using the application associated with the user interface event, wherein the interface model defines state information for one or more graphical objects of the user interface, the state information being obtained from the user interface event; determine a difference between the state information of the one or more graphical objects and previous state information of the one or more graphical objects; generate a second model, the second model including data of the one or more graphical objects having state information different from the previous state information; and send the second interface model to the client device, the second interface model enabling the client device to render the user interface.

[0020] In some aspects of any of the above examples, the interface model (and / or the second interface model) includes a hierarchical data structure having one or more graphic objects. Each of the one or more graphic objects may have one or more attributes defined by state information. In some aspects, the one or more graphic objects include at least one of a box, a line, an image, an icon, text, and a content item to be displayed in a user interface. In some aspects, each of the one or more graphic objects includes one or more attributes defined by state information. The one or more attributes of the one or more graphic objects may include at least one of a position, a shape, and a color of the graphic object.

[0021] In some aspects of any of the above examples, the above methods, apparatus, and computer-readable media may also include determining state information based at least on a user interface event.

[0022] In some aspects of any of the above examples, an application associated with the user interface event is determined based on a connection between the server computer and the client device.

[0023] In some aspects of any of the above examples, the client device comprises a media device, and wherein the user interface comprises a media content guide.

[0024] The examples disclosed herein for example methods, apparatus, and computer-readable media may be implemented individually or in any combination.

[0025] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0026] The foregoing and other features and embodiments will become more apparent with reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Illustrative embodiments of the present application are described in detail below with reference to the following drawings:

[0028] Figure 1 is a block diagram illustrating an example of a network environment according to some embodiments;

[0029] Figure 2 is a block diagram illustrating an example of a server system and a client device according to some embodiments;

[0030] Figure 3 is a diagram illustrating an example of a message flow between a server system and a client device according to some embodiments;

[0031] Figure 4 is a diagram showing an example of an interface model according to some embodiments;

[0032] Figure 5 is a diagram showing another example of an interface model according to some embodiments;

[0033] Figure 6 is a diagram illustrating an example of an incremental interface model according to some embodiments;

[0034] Figure 7 is a diagram illustrating an example of a change in a user interface based on an incremental interface model according to some embodiments;

[0035] Figure 8 is a flow chart illustrating an example of a process for generating an incremental interface model according to some embodiments;

[0036] Fig. 9 is a flow chart illustrating an example of a process for processing user interface content according to some embodiments;

[0037] Fig.10 is a flow chart illustrating another example of a process for processing user interface content according to some embodiments; and

[0038] Fig.11 is a block diagram illustrating an example of a computing system architecture in accordance with some embodiments. DETAILED DESCRIPTION

[0039] Some aspects and embodiments of the present disclosure are provided below. It is obvious to those skilled in the art that some of these aspects and embodiments can be applied independently, and some of them can also be applied in combination. In the following description, for the purpose of explanation, in order to provide a thorough understanding of the embodiments of the application, specific details are set forth. However, it is obvious that various embodiments can be practiced without these specific details. The accompanying drawings and descriptions should not be restrictive.

[0040] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0041] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that the embodiments may be practiced without these specific details. For example, in order not to obscure the embodiments due to unnecessary details, circuits, systems, networks, processes and other components may be shown as components in block diagram form. In other cases, in order to avoid obscuring the embodiments, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details.

[0042] In addition, please note that various embodiments can be described as processes, which are depicted as flow charts, process views, data flow diagrams, structure diagrams or block diagrams. Although flow charts can describe these operations as sequential processes, many of these operations can be performed in parallel or concurrently. In addition, the order of these operations can be rearranged. The process terminates when its operation is completed, but may have additional steps that are not included in the figure. The process may correspond to a method, function, process, subroutine, subprogram, etc. When the process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

[0043] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying (one or more) instructions and / or data. Computer-readable media may include non-transient media in which data can be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or through wired connections. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or storage devices. Computer-readable media may store thereon codes and / or machine-executable instructions, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted by any appropriate means including memory sharing, message passing, token passing, network transmission, etc.

[0044] Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) that perform the required tasks may be stored in a computer-readable or machine-readable medium. (One or more) processors may perform the necessary tasks.

[0045] The user interface can be displayed by the device, and can allow the user to interact with the device and / or the software application (also referred to as application in this article) executed by the device by interacting with the graphical elements of the user interface. For example, the user can provide input to manipulate, control, select and / or interact with different graphical elements of the user interface by providing one or more inputs provided by a device capable of receiving input from the user. The input may include, for example, touch input, gesture input, device auxiliary input and / or voice input, etc. The user interface can be used for various purposes. For example, the user can use the user interface to obtain services or items (for example, obtain news articles, order movies or performances by interacting with the application, order food by interacting with the application, etc.). In another example, the user can interact with the game by interacting with the user interface of the game application.

[0046] In some cases, it is possible to generate, render a user interface or perform a combination of the two on a client device. The client device may include any device connected to a wired and / or wireless network, such as a set-top box, a television set connected to the Internet, a mobile device (e.g., a cellular phone, a laptop computer, a tablet computer or other mobile device), a desktop computer, a wearable device (e.g., a smart watch, a fitness tracker, a virtual reality headset, an augmented reality headset or other wearable devices) or other devices. For example, the user interface solution implemented by the client device can be implemented by running the client user interface code (e.g., as a native application or using HTML / JavaScript) on the client device, wherein the user interface code generates and renders the user interface. In such an implementation, the client user interface code needs to perform layout calculations and other calculations of the user interface, requiring the client device to process all user interface events and determine the user interface layout and other user interface attributes (e.g., the color of the graphic object, the edge information of the graphic object, the speed of the video to be played in the user interface, etc.) based on these events. In some cases, the client device can interact with the back-end server to process data and perform database access (e.g., for programming data, video-on-demand data, etc.).

[0047] However, in some cases, such a client device implementation may result in poor performance characteristics of the user interface. In some examples, the poor performance may be due to the communication of information and / or content between the client device and the server. For example, there may be delays when the user interface is started and / or during the deployment of the user interface. In one example, it often takes a significant amount of time to start (e.g., load, compile JavaScript, etc.) the user interface code itself. In another example, the user interface code may also take a significant amount of time to be ready for use. In some cases, the user interface code may also need to pre-fetch large amounts of data (e.g., to make the application faster when in use).

[0048] Another problem of existing user interface solutions is bandwidth utilization. For example, for user interface management, existing solutions require a large amount of bandwidth, which is expensive relative to network resources, especially in systems with many users. In an illustrative example, the loading time of an application depends on bandwidth, and when using a slow network (e.g., an asymmetric digital subscriber line (ADSL) network, a 3G network, or other network), bandwidth may be limited. Bandwidth delays cause the generation and / or rendering of a user interface to degrade in performance. Due to the cost required when paying by usage (e.g., a certain amount of dollars per gigabyte), bandwidth can be expensive. In some cases, in order to compensate for bandwidth costs, a solution is to use more memory to better cache code and data, which may also be an expensive solution. The technology described in this article can reduce the required bandwidth. This reduction for communication scale will improve the efficiency of managing the user interface, thereby improving runtime and reducing the resources required for managing the user interface.

[0049] This problem is further exacerbated by systems with many client devices, each of which manages its own interface and must maintain communications with a server for application-related data, etc. These problems become even more acute when the user interface changes rapidly, requiring many updates and frequent communications with one or more servers.

[0050] User interface realization based on client device may also cause other problems.For example, compared with developing features on server system, developing features on multiple client devices needs more computing resources.For example, various client devices (for example, set-top box, smart TV, mobile device etc.) may not have the ability of sharing resources with each other.In an illustrative example, if for each client device, the media content provider needs the storage of 1 gigabyte, and there are 1 million users (therefore at least 1 million client devices), then required storage capacity will be at least 1 quadrillion bytes (petabyte).In addition, factors such as different equipment restrictions, platforms, supported versions of code, versions of user interface make it difficult to manage user interface at client side.

[0051] In some cases, the user interface can be implemented by one or more server computers (also referred to as servers in this article). For example, a cloud-based system may include one or more server computers, which generate and render a user interface, which is then sent to a client device. For media-based applications (e.g., TV applications, streaming applications, streaming game applications, etc.), the user interface can be sent together or separately with a content stream (e.g., a media stream). A cloud-based system can implement a complete user interface, including logic, graphics rendering, video conversion, etc. Such a solution can allow less software to be installed on the client device (also requiring less software upgrades), and may require less testing, which helps to shorten the time to market. However, for various reasons, it is challenging to move user interface processing (e.g., views, controllers, etc.) to a cloud-based system. For example, for each active client device, a server computer may need to have a memory for its graphics, processing power for rendering graphics, and processing power for compressing graphics (e.g., for compressed video, images, and / or any other graphics compression scheme). Compared with the actual application "logic" code itself, the hardware intensity of this solution is about 50-100 times higher. This solution can also use a large amount of bandwidth from one or more server computers. In addition, it may be desirable to keep user interactions in near real-time (e.g., the time for a display to follow a user input is less than 20 milliseconds (ms), 50 ms, 100 ms, or other minimum duration), which is difficult with cloud-based systems. Cloud-based systems may also be difficult to implement (especially in real-time) and even more difficult to scale to larger deployments without using a large amount of central processing unit (CPU) and bandwidth resources to perform the desired functions (e.g., managing interaction events through the user interface). The amount of CPU and bandwidth required to implement such a system (even higher in real-time) may be prohibitively expensive. It may also be difficult to send multiple different user interfaces and user interface changes on a regular basis on the server side.

[0052] Described herein are systems, methods, and computer-readable media for implementing a user interface, for which generation and rendering are managed by a centralized or server-based computer system (e.g., a cloud-based server system, a distributed computer system, or other remote computer-based system) that includes one or more server computers and is remote from a client device. A server-based system can implement techniques for managing user interfaces across multiple devices, thereby improving the efficiency of managing a common user interface across a system with many client devices. The server-based user interface described herein can be implemented and / or managed in part by a server system (including one or more server computers) and in part by a client device or a combination thereof to render a user interface. For example, a client device can generate a user interface event (e.g., based on user input received through a user interface, based on a system-level event, and / or based on an application-level event), and can send the interface event to a server system. The server system can identify the application and / or system to which the interface event is to be applied (e.g., for a system-level event), and can determine the state information of the user interface obtained from any applicable interface event (e.g., the layout implementation required to determine the layout of the interface user interface, determine other features of the graphical objects of the interface, etc.). The layout implementation can include functions such as layout calculation. In some cases, the server system can generate an interface model that structurally defines the state information of the user interface, and can send the interface model (or a compressed version of the interface model) to the client device. The client device can then read the interface model and can render the interface defined by the interface model.

[0053] The state information of the user interface may include state information of one or more objects (e.g., graphic objects and any other objects) of the user interface. The state information may define the layout of the graphic objects in the user interface (e.g., the position, orientation, etc. of each graphic object) and the attributes or properties of the graphic objects (e.g., the colors, fonts, images, videos, etc. used). The attributes or properties may also be used to change the state of the object. In an illustrative example using a video object as an example of a graphic object, the "speed" attribute or property in the object model may control the playback speed of the video (e.g., a value of 1 for normal, 0 for pause, etc.). In another example, the "onSpeedChange" property in the object model may indicate to the client device that the server system is interested in video events such as video start, video end, video failure, etc.

[0054] The user interface generated using the server-based system described in this article can include physical or virtual user interfaces of any appropriate type that can promote receiving input via input devices.The illustrative example of user interface includes any type of graphical user interface (GUI), an electronic program guide (EPG) or content guide for accessing media content, a user interface for game applications, a user interface for bank applications and / or a user interface of any other appropriate type.In some cases, multiple applications can be stored on the client device.Each application can have a separate user interface that allows a user to interact with the application.For example, smart television sets or set-top boxes can include a first application for a first stream service, a second application for a second stream service, and a third application that allows a user to mirror the content displayed on a mobile device to a television set.In such an example, the first application can have a first user interface, and the second application can have a second user interface, and the third application can have a third user interface.

[0055] The server-based user interface system described herein can be used in any type of system (e.g., a client-server system) that includes a first device (e.g., acting as a server computer) and a second device (e.g., a client device), wherein the first device generates and / or manages an interface model and communicates (e.g., sends or transmits) the interface model (or a compressed version of the interface model) or provides access to the interface model or a compressed version of the interface model (e.g., by sending a link referencing a location from which the interface model can be accessed) to the second device, so that the second device can present (e.g., by rendering and / or displaying) a user interface defined by the interface model. Examples of systems that can implement a server-based user interface include: a media system in which a content provider provides a content guide (e.g., an electronic program guide and / or other user interfaces for interacting with media content, such as video content (video on demand content, streaming video content, and / or other video content), audio content (audio on demand content, streaming audio content, and / or other audio content), augmented reality content, virtual reality content, and / or other media) to subscribers of the media system, a database service provider that provides a user interface to subscribers for managing a database maintained by a data service provider, a case management system that provides a user interface to subscribers for interacting with a case management system or accessing cases maintained by the case management system, a security management system that provides a security management interface, a system that provides interfaces for augmented reality and / or virtual reality devices, Internet of Things (IoT) devices, networked devices, and / or other devices, any other type of interface that provides a client view, and the like.

[0056] Figure 11 is a block diagram illustrating an example of a network environment 100. The network environment 100 includes a server system 101 and client devices 110a, 110b to 110n. The server system 101 may include one or more server computers that may process user interfaces and / or other data to generate user interface data. The client devices 110a, 110b to 110n may include any suitable number of client devices, such as one or more client devices, connected to the server system 101 (e.g., via a wired or wireless network). Figure 2 Further details of the example server system 201 and client device 210 are described.

[0057] In some cases, server system 101 may include a cloud infrastructure system (also referred to as a cloud network) that provides cloud services to one or more client devices (e.g., client devices 110a-110n). In some cases, server system 101 may include a system configured to provide software as a service (SaaS), or may communicate with one or more servers that provide SaaS services. In a SaaS-based system, a third-party provider may host applications and make them available to users over a network. For example, Figure 1 As shown, server system 101 may communicate with one or more server computers 140 via network 136 (eg, receiving interface events 147 and sending interface models 149 as described in greater detail herein).

[0058] The server system 101 can access user interface data specific to an application and / or website. The application and / or website can access services (e.g., cloud services, SaaS services, etc.) provided by the server system 101 through a communication network (e.g., the Internet, a WiFi network, a cellular network, and / or using another appropriate other communication network). In some embodiments, the services provided by the server system 101 may include many services that users of the infrastructure of the server system can use on demand. The services provided by the server system 101 can be dynamically scaled to meet the needs of its users. The network of the server system 101 may include one or more server computers and / or other devices and / or systems. In some cases, one or more server computers and / or other devices and / or systems constituting the network are different from on-premises computers, server computers and / or other devices and / or systems that may be located on site (e.g., the user's home, workplace, location for holding an event, or other location). In some cases, one or more server computers and / or other devices and / or systems constituting the network of the server system 101 may be provided in an enterprise system. In some cases, a single entity (e.g., a cloud provider, SaaS provider, etc.) may implement server system 101, and services may be provided by different entities having accounts (e.g., cloud accounts, SaaS accounts, etc.) with multiple users accessing the services via client devices.

[0059] One or more server computers of the server system 101 can communicate with one or more client devices 110a-110n using a network 134 such as a wireless network, a wired network, or a combination of a wired network and a wireless network. A wireless network can include any wireless interface or combination of wireless interfaces (e.g., Internet, cellular such as 3G, Long Term Evolution (LTE) or 5G, a combination thereof, and / or other appropriate wireless networks). A wired network can include any wired interface (e.g., fiber optic, Ethernet, Ethernet over power line, Ethernet over coaxial cable, digital signal line (DSL), or other appropriate wired network). Wired and / or wireless networks can be implemented using various routers, access points, bridges, gateways, etc. that can connect the server system 101 and client devices 110a-110n to the network.

[0060] One or more client devices 110a-110n may include any device that can be connected to a wired and / or wireless network, such as a set-top box, a television connected to the Internet, a mobile device (e.g., a cellular phone, a laptop or notebook computer, a tablet computer or other mobile device), a desktop computer, a wearable device (e.g., a smart watch, a fitness tracker, a virtual reality headset, an augmented reality headset or other wearable device), a camera (e.g., a digital camera, an IP camera, a video camera, a camera phone, a video phone or other suitable capture device), a digital media player, a video game console, a video streaming device, other media devices and / or any other suitable type of device. In some cases, one or more client devices 110a-110n may be configured to obtain or capture media data. The media data may include video, audio, images, any combination thereof and / or any other type of media. One or more client devices 110a-110n may include local interface generators 112a, 112b to 111n and display devices 130a, 130b to 130n. The display devices 130a, 130b to 130n may display server-based user interfaces 132a, 132b to 132n.

[0061] As described above, the server-based user interface can be generated in part by the server system 101 and in part by a client device (e.g., one of the client devices 110a-110n), which will ultimately display the user interface or provide the user interface for display by another device. For example, the interface generator 102 of the server system 101 can receive interface events (e.g., one or more of the interface events 107a, 107b to 107n) from the local interface generator (e.g., one or more of the local interface generators 112a, 112b to 112n) of the client device (e.g., any one or more of the client devices 110a, 110b to 110n). For example, the interface event can be generated by the client device based on user input received through the user interface (e.g., events specific to the application to which the user interface applies), based on system-level events (e.g., events that apply to the client device as a whole, such as volume adjustment), and / or based on application-level events. The server system can identify the application and / or system to which the received user interface event applies (e.g., for system-level events), and can determine the graphical elements to be displayed, where and how to display different graphical elements, how to react to the user interface event (e.g., determine any changes in the user interface of the application based on the received interface event), and other user interface actions. Upon receiving the user interface information (e.g., interface model 109a, interface model 109b to interface model 109n, as described in more detail below) from the server system 101, the client device can generate and render a user interface based on the user interface information. The user interface can be rendered in the application in which the user interface event is identified.

[0062] In some implementations, the server system 101 can maintain "global" system objects for different types of devices. The global system object of a device can be queried by an application to determine information about the device. The "global" system object can provide the application with information about the static and dynamic state of the device. In some cases, some or all of the static and / or dynamic state information can be cached by the server system. The server system can send an information request to the client device, and the client device can reply with state information in response to the request. In some cases, some state information can be proactively sent to the server system as an event (e.g., a smart card has been removed, etc.). For differences in devices (e.g., screen size, device type, etc.), the application can generate different user interface properties (e.g., layout and other attributes) based on device information (e.g., screen size, device type, etc.) from the "global" system object.

[0063] In some examples, the server-based user interface can be generated in part by the server system 101 and in part by another server system (including one or more server computers). In some implementations, the other server system can include a SaaS system that hosts one or more applications available to one or more users. The other server system can provide the server-based user interface associated with the application to the client device for display by the client device or another device (e.g., a display) in communication with the client device. For example, as described above and as Figure 1 As shown, the server system 101 can communicate with one or more server computers 140 via the network 136. For example, the interface generator 102 of the server system 101 can receive interface events (e.g., one or more interface events 147) from a local interface generator (not shown) of a server computer in one or more server computers 140. The interface events are similar to the interface events described above with respect to the client devices 110a, 110b to 110n. The server system can identify the application and / or system (e.g., for system-level events) to which the received user interface event applies, and can determine the graphical elements to be displayed, where and how to display different graphical elements, how to react to the user interface event (e.g., determine any changes in the user interface of the application based on the received interface event), and other user interface actions. The server system 101 can then provide the user interface information to the server computer (in one or more server computers 140), and the server computer can generate and render the user interface based on the user interface information (e.g., the interface model, as described in more detail below). Although various examples are described herein discussing communications between server systems (e.g., server systems 101 and / or 201) and client devices (e.g., client devices 110a, 110b, 110c, and / or 210), the same concepts apply to communications between multiple server systems (e.g., between server system 101 and one or more server computers 140).

[0064] In some examples, server system 101 can generate an interface model that describes what the user interface looks like. Figure 1As shown, the server system 101 can generate an interface model 109a and send it to the client device 110a, generate an interface model 109b and send it to the client device 110b, until the interface model 109n is generated and sent to the client device 110n. For example, the interface model can be generated based on received user interface events (e.g., one or more key presses, finger movements, device orientation and / or position, etc.). In some cases, the interface model can also be based on other events, such as other remote client events that may not be based on explicit user input, or events received from devices or systems other than the client device. Examples of other events can be based on events such as a video being started or ended, video access control being denied, a phone ringing, an email or other message being received. In some cases, the server system can generate an interface model that logically defines one or more states of the user interface (e.g., the state of one or more graphical objects of the user interface) in a structured manner. In some implementations, the interface model may be defined in whole or in part by a structure or syntax that may define or represent the logical structure of a document (e.g., the structure of an application's user interface), as well as the manner in which the document is accessed, generated, rendered, and / or manipulated. For example, the interface model may include a document object model (DOM) that defines a structure for presenting components that comprise a user interface.

[0065] In some cases, the server system can improve the efficiency of generating and rendering the user interface at the client device. In at least one embodiment, the server system can optimize the interface model before sending the interface model to the client device. For example, the server system 101 can generate an intermediate interface model (also referred to as an "incremental interface model" herein) that indicates the difference or increment between the current user interface determined by the server system 101 (e.g., current state information of one or more graphical objects of the current user interface) and the previous user interface (e.g., previous state information of one or more graphical objects of a previously generated user interface). The server system can send the incremental interface model to the client device, indicating only the difference, rather than sending the entire interface model describing the entire user interface.

[0066] The server system 101 may send an interface model or an incremental interface model that has changes or differences from a previous interface model to a client device. The client device may render a user interface based on the received interface model. In some cases, a graphics rendering engine in a local interface generator of the client device may read an interface model received for a user interface of a given application, and may render an interface defined by the interface model. For example, using the interface model, the graphics rendering engine may convert a graphical object tree from the interface model into a list of graphical routine calls. When an incremental interface model is implemented, the client device may update the user interface based on an increment indicating the properties and / or functionality of the changed user interface. In some cases, if the user interface changes by a certain amount (e.g., a complete change of the user interface), the interface model may indicate the properties and / or functionality of a reusable user interface.

[0067] Figure 2 is a diagram showing an example of a server system 201 in communication with a client device 210. The server system 101 includes various components, including: a user interface server engine 204, an application user interface engine 206, a memory 203, and a central processing unit (CPU) 205. The application user interface engine 206 communicates with an application domain engine 208. The client device 210 also includes various components, including: a user interface client engine 214, a user input engine 216, a graphics rendering engine 218, a media synthesis engine 220, a decryption engine 222, and a decoding engine 224. In some cases, the media synthesis engine 220, the decryption engine 222, and the decoding engine 224 are optional components that can be used only when needed. For example, some user interfaces do not include media, in which case the media synthesis engine 220, the decryption engine 222, and / or the decoding engine 224 may not be used. In some cases, such as Figure 2 As shown, for example, when the user interface implemented by the graphics rendering engine 218 does not include any media, the output of the graphics rendering engine 218 can be provided directly to the display 215. In some cases, the client device may not include the media synthesis engine 220, the decryption engine 222, and / or the decoding engine 224. In some implementations, depending on the type of device, the client device 210 may also include a user input device 213 and / or a display 215 (the optional nature of these components is shown by the dashed outline of the boxes representing these components). For example, a mobile device and a networked television may include a display, while a set-top box may not include a display.

[0068] The components of the server system 201 and the client device 210 may include and / or may be implemented using electronic circuits or other electronic hardware, which may include: one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other appropriate electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. Although the server system 201 and the client device 210 are shown as including certain components, those of ordinary skill in the art will recognize that the server system 201 and / or the client device 210 may include more than one component. Figure 2 For example, in some cases, the server system 201 and / or the client device 210 may also include: one or more memory devices (e.g., RAM, ROM, cache, buffer, etc.), a processing device, one or more buses, and / or Figure 2 Other devices shown in FIG.

[0069] Server system 201 and Figure 1 The server system 101 is similar to and can perform the same or similar functions. Similarly, the client device 210 is similar to Figure 1 The client devices 110a-110n can perform the same or similar functions. Figure 1 As described, the server-based user interface can be generated in part by the server system 201 and in part by the client device 210. The interface generator 202 of the server system 201 can execute the user interface logic. For example, the application user interface engine 206 of the interface generator 202 can determine what graphic elements to display, where and how to display different graphic elements, the functions of these graphic elements, how to handle user interface events and other user interface actions. It can be assumed that the client device 210 has sufficient graphics processing performance to display certain content, for example, display video and synthesize the video with some on-screen display (OSD). For example, the client device 210 can have at least a graphics processing unit (GPU) or other graphics software and / or hardware that can support graphics acceleration (e.g., two-dimensional or three-dimensional graphics acceleration). In some cases, the GPU or other graphics software and / or hardware can be part of the graphics rendering engine 218.

[0070] Figure 3 is a diagram showing an example of a message flow between the server system 201 and the client device 210, and will be related to Figure 2 The client device 210 may generate an interface event 207, which may (for example, Figure 3The interface event 207 may be associated with or include a navigation command (e.g., moving a selection box from a first graphical element to a second graphical element, moving to a different menu, etc.), a selection command (e.g., selecting a graphical element corresponding to a movement, displaying a volume graphical element, etc.), and / or any other appropriate command. For example, in some cases, the interface event 207 may include low-level events, such as control selections, input interactions, gesture commands, voice commands, orientation and / or position of the client device 210 or the user input device 213, and / or other low-level events corresponding to one or more navigation commands, selection commands, and / or other commands.

[0071] Interface events 207 may be generated in response to input received from user input device 213. User input device 213 may be part of client device 210, or may be a separate device that communicates (wired and / or wirelessly) with client device 210. User input device 213 may include any appropriate input device, such as a remote control, a touch screen, a microphone, a keypad, a mouse, a keyboard, a device that can generate gesture input, a head-mounted display, and / or any other input device. A user may navigate on a user interface and may use user input device 213 to select items presented by the user interface. User input engine 216 may receive input commands from user input device 213. User input engine 216 may include an input interface for receiving input commands. In some cases, different user input engines 216 may be included for different types of user input devices. In response to receiving an input command, user input engine 216 may generate interface events 207, and interface events 207 may be sent to user interface client engine 214. In some cases, user input engine 216 may send input commands to user interface client engine 214 , and user interface client engine 214 may generate interface events 207 .

[0072] In some cases, the interface event may be a system-level event. A system-level event is an interface event related to a system (e.g., a client device or another server system) rather than a specific application. Even when one or more other applications are running for the user, a system-level interface event may be routed to a system-level application (e.g., a smart TV application). An example of a system-level interface event may include a volume adjustment input provided by a remote control button. Another example of a system-level event may include a color adjustment that affects the color of an item displayed by a client device. Other illustrative examples of system-level events from a device include: a phone is turned from a horizontal orientation to a vertical orientation and / or from a vertical orientation to a horizontal orientation, a smart card is removed, the device is turned off (e.g., power is turned off, sleep is entered in a low-power mode, etc.), the device battery is low (e.g., the battery charge is less than a battery charge threshold, e.g., less than 10%, 20% or other appropriate charge levels), the device enters a certain mode (e.g., night mode, where the color of the interface changes, notifications or alarms are turned off, etc.), the application is suspended by the user, and the application is continued by the user. Illustrative examples of system-level events generated by the server and sent to the user interface client engine 214 may include a connection to the client device (and user) being lost, a server shutting down, an application being suspended, an application being resumed, and the like.

[0073] In some cases, the interface event may be an application-level event. An application-level event is an interface event related to an application. For example, an application may generate an application-level event even when no user input is received. In various illustrative examples, the application-level event may include a change in a version of an application, a change in a display setting of an application, and / or any other application-level event.

[0074] The user interface server engine 204 of the server system 201 may receive the interface event 207 from the user interface client engine 214 of the client device 210. The user interface server engine 204 may then dispatch a message to the specific user of the client device 210 that is generated by an appropriate application (e.g., Figure 3 1, 2 to 3) using one or more interface events. Figure 3At step 304 shown, the user interface server engine 204 may send one or more interface events to the application user interface engine 206. In some implementations, one interface event may be included in each message to the server system 201. In some implementations, multiple interface events may be included in each message to the server system 201. In some cases, the interface events may be throttled or filtered so as not to overload the amount of messages to the server system 201. For example, in response to a user moving a finger on a touch screen (e.g., of a smartphone), many events may be generated (e.g., 50 events per second). These events may be filtered to a certain number of interface events per time period (e.g., a maximum of six interface events per second).

[0075] The current session between the client device 210 and the application can be identified using any appropriate technology (e.g., using a socket, an identifier associated with the user and the application, etc.). In some implementations, the server system 201 can keep a socket open for each connected user, for example, a first socket for a first user, a second socket for a second user, a third socket for a third user, etc. In some cases, multiple server computers can be used, and a load balancer can manage communications from a client device to multiple server computers. The load balancer can maintain a "sticky connection" between a specific server computer and a specific client device. For example, an IP sticky connection can be used to identify a server instance hosting a user's session with an application, which ensures that requests from a single user are distributed to the server on which they are launched. The server system 101 can then maintain an association between the open socket and the current application and the user.

[0076] For each user / connection, the user interface server engine 204 determines which application is the current application and which process is currently processing the application. In some cases, an application can register an event of interest to the application. In this case, if a user has more than one application running for the user, the server system 101 can dispatch the event to the application registered for the event. An application determined to be suitable for a user interface event can be referred to as an active application. In some cases, the server system 201 can have an application lifecycle engine (not shown) that can filter interface events and dispatch interface events to active applications for a given user and a client device from which the event is received. As described above, some events are system-level events (e.g., events related to the system rather than the application itself) and may not be (by default) routed to active applications.

[0077] As mentioned above, in Figure 3At step 304, the UI server engine 204 may send one or more UI events to the application UI engine 206, which may interpret the one or more events. In some implementations, the UI logic of the application UI engine 206 may asynchronously request one or more domain functions (in the case of a domain engine 208) from the application domain engine 208. Figure 3 In one illustrative example, a service provider's "pay phone bill" user interface may be implemented by server system 201, but application user interface engine 206 may make a request (e.g., using a URL of the service provider) to the service provider's application domain engine 208 (e.g., one or more servers and / or databases) to access user credentials, user credits, etc. In this case, server system 201 does not have to store all data associated with the provider of the application (but may cache certain data in some cases, which may provide faster performance). In another example, a media streaming service application may be implemented by server system 201. The streaming service user interface application itself may not maintain a list of movies to watch, but may request a list of movies from one or more servers and / or databases of the streaming service provider. In some cases, the application domain logic of application domain engine 208 may be physically separated from the user interface logic of application user interface engine 206, such as by running the domain logic as a separate service or microservice (e.g., in a different subnet). For example, for a chess game, the domain logic may handle the artificial intelligence (AI) of the game. In some cases, the application user interface logic of the application user interface engine 206 can bypass the domain logic (e.g., to increase speed). In some examples, the user interface logic can manage some form of caching. For example, the EPG application does not have to request the EPG microservice (e.g., in the application domain engine 208) a thousand times per second to determine what is on a certain TV channel. Rather, the application user interface engine 206 can cache to temporarily maintain such data locally.

[0078] The application user interface engine 206 can interact with one or more applications stored in the memory of the server system or in an external memory. Figure 3In the example shown, one or more applications may include application 1, application 2 to application n. In some cases, the application user interface engine 206 may include an application runtime engine (e.g., a sandbox) with application user interface logic. The sandbox may provide a secure environment. Each secure environment may be device-specific, application-specific, operating system (OS)-specific, etc. In some implementations, each application may have a dedicated application user interface engine 206. In some cases, it may be safer to run each application in its own application user interface engine 206. In some implementations, a single application user interface engine 206 may interact with multiple applications. In some cases, running different applications in the same application user interface engine 206 may use fewer resources. In some cases, multiple application user interface engines may run concurrently. In some cases, applications with high usage may use multiple dedicated application user interface engines (e.g., when each runtime engine has restrictions). In some examples, applications from a common application developer or provider may share a common application user interface engine. In some cases, a certain application user interface logic may share a process. For example, an EPG application may share the same application user interface process with a TV remote control application so as to easily share the cache of two applications in RAM. In another example, a niche application with a small number of concurrent users could share processes to save money.

[0079] During operation, application UI engines (e.g., sandboxes) may be started, monitored, and shut down as applications are added, removed, and / or updated, and as users start and / or stop applications. An application UI process should be able to handle a certain number of users (e.g., several thousand concurrent users). As an application process approaches its user limit, the UI server engine 204 may start additional processes for the application for the next number of users (e.g., the next several thousand users). For example, a first user using an application may cause an application UI engine to start, and then another application UI engine may be started when there are too many users running the application in that application UI engine.

[0080] Based on one or more received interface events (e.g., one or more key presses, finger movements, device or input device orientation and / or position, etc.), the application user interface engine 206 can determine how the graphical elements (or graphical objects) of the user interface associated with the application should be displayed. For example, the application user interface engine 206 can determine various states of the user interface, such as the layout of the graphical objects of the user interface, the color of the graphical objects, fonts, image information, video information, and other functions. Changes can include graphical changes, changes in controls and / or functions (e.g., adding or removing controls and / or functions), changes in the entire user interface, and the like. It can be determined by the application user interface engine 206 at the application level whether a specific interface event (e.g., touch input, key press, etc.) is a navigation command, a selection command, or other appropriate command. In some cases, various states of the user interface can be determined based on other remote client events, which may or may not be based on user input, such as events such as a video being started or ended, video access control being denied, a phone ringing, and the like. For example, a graphical element including a video can be removed in response to the end of a video. Available graphical elements and other user interface content for each application may be obtained from memory accessible to the server system 201 (e.g., memory 203 on the server system 201 or memory on one or more other local and / or remote servers). Using the available graphical elements and other user interface content, the application user interface engine 206 may determine what to display based on one or more received interface events, such as graphical content such as a box, line, text, image, address information (e.g., a uniform resource locator (URL)), etc., for a graphical asset.

[0081] The application user interface engine 206 can determine which user interface graphical objects (or elements) to display, remove and / or change based on specific interface events. For example, an application can register certain events (such as key presses, scroll events, zooming, playing video, etc.) so that a specific function will be called when a certain interface event occurs. In response to the received events and the corresponding functions called, the application user interface engine 206 can change the state of the application, and can generate a new interface model that defines the properties of the user interface based on the state change. Any appropriate framework can be used to implement application changes based on received interface events. An illustrative example of a framework implemented by the server system 201 is shown in Table 1 below.

[0082]

[0083]

[0084] Using the framework in Table 1, based on the interface event, the UI application can generate a DOM or update a DOM when it receives an interface event, and can send the DOM or incremental DOM to a client device or a server system. The names "event handler", "reducer", and "renderer" used in Table 1 are standards in UI frameworks (such as the React framework). In an illustrative example, the framework shown in Table 1 below can be used for a UI cloud application. The client device may include a client device 210, the server system in the rightmost column of Table 1 may include one or more server computers 140, the UI system corresponds to a user interface server engine 204, and the UI application corresponds to an application user interface engine 206.

[0085] Figure 8 800 is a flowchart showing an example of a process 800 for generating an incremental interface model. The process 800 will be described together with the framework shown in Table 1. At box 802, the process 800 includes receiving an interface event. For example, referring to the framework in Table 1, the client device or server system can receive input corresponding to the interface event. The client device or server system can send the interface event to the UI system. At box 804, the process 800 can determine the application associated with the interface event. For example, referring to Table 1, the UI system can find the correct event handler and the correct application / user context (including information associated with the user). The application can maintain different event handlers for each interface event registered by the application. For example, a first event handler can be maintained for a first interface event (e.g., a key press, a mouse move, a touch event on a touch screen, etc.), a second event handler can be maintained for a second interface event, a third event handler can be maintained for a third interface event, and so on. The event handler can be a function of the application, wherein the function is called when an interface event corresponding to the event handler occurs (e.g., an event handler function associated with a key press is called when a key press event is received by the server system). The event handler may generate an action based on the event. The UI system may dispatch or implement the action. The dispatch of the action may cause the user interface to be updated based on the action, which may cause the state of the user interface to change. For example, the UI system may send the previous state and the action to the UI application, and the UI application may implement a reducer to update the state. The reducer returns the new state (based on the action) to the UI system.

[0086] At box 806, process 800 can determine user interface state information based on interface events. At box 808, process 800 can determine whether the state of the user interface has changed. If it is determined at box 808 that the state of the user interface has changed, the process generates an incremental interface model at box 810. If it is determined at box 808 that the state of the user interface has not changed, process 800 can end at box 812. Referring to Table 1, when the state of the user interface changes, the renderer is called. For example, if the state has changed compared to the previous state of the user interface, the state is sent to the renderer of the UI application to convert the state into a new interface model (e.g., a document object model (DOM) or other interface model as described below). The new DOM is then sent to the UI system. In an implementation using incremental DOM, the UI system calculates incremental DOM based on the difference between the previous user interface and the current user interface. The UI system can send incremental DOM to a client device or a server system. In the case of not using incremental DOM, the UI system can send a complete DOM to a client device or a server system. The client device or the server system can update the user interface (e.g., perform a graphical update of the user interface) based on the DOM or incremental DOM.

[0087] In some cases, the user interface logic of the application user interface engine 206 can request user interface changes at any time, not just when a client event is received. In some cases, when a single event (e.g., a key press) is received, the application can request more than one user interface change (e.g., two or more changes). For example, a first graphical change can be implemented by the application domain engine 208 when the event is received (thereby allowing quick feedback to the viewer), and a second graphical change can be implemented when the domain logic of the application domain engine 208 responds to the information request from the application user interface engine 206. In some cases, the application user interface engine 206 can update the user interface from time to time, such as to push new information, after a timeout, and / or in response to the occurrence of other circumstances.

[0088] The application user interface engine 206 can generate an interface model that describes the state information of the user interface. For example, when the application user interface engine 206 is ready to answer a request from the user interface server engine 204, the application user interface engine 206 can generate a new or modified interface model that describes the layout and other features of the objects of the user interface. The application user interface engine 206 can then pass the interface model back to the user interface server engine 204 (in the Figure 3 308 of the embodiment of the present invention).

[0089] In some implementations, as described above, the interface model may include a document object model (DOM). The DOM may define or represent the logical structure of a document (e.g., the structure of an application's user interface), as well as the way in which the document is accessed and manipulated. For example, the DOM may include an object tree, each of which has properties (e.g., properties such as color, position, size, etc.). In some cases, the DOM may define or represent the logical structure and accessibility and manipulation information of a hypertext markup language (HTML) document (e.g., for a web page or web application), an extensible markup language (XML) document, an extensible hypertext markup language (XHTML) document, and / or other documents. The DOM is a cross-platform and language-independent application programming interface that views a document as a tree structure in which each node includes an object representing a portion of the document. One or more objects may represent elements of a user interface, and its elements may be interactive, resulting in the generation of interface events. For example, a DOM generated for defining a user interface may include a tree of graphic objects (e.g., one or more boxes, lines, images, icons, text, videos, etc.) having attributes (e.g., one or more positions, colors, URLs, etc.). Using DOM, the application user interface engine 206 can browse the structure of the document, and can add, modify and / or delete elements and content in the document (e.g., based on received user interface events and / or other events, such as remote client events). For example, using DOM, many items of the document can be accessed, changed, deleted and / or added based on received user interface events. In some implementations, an interface model (e.g., DOM) can be defined based on the characteristics of the device, operating system (OS) type, version of the application and / or some other changes of the client. For example, an interface model with differences in user interfaces between device types, OS types, application versions, etc. can be generated. In such an implementation, a single interface model or an incremental interface model (including UI changes) can be defined for multiple clients according to devices, applications, OS, etc. In some cases, different interface models can be generated based on the characteristics of the changing devices, or a single model can be used for all different features.

[0090] In some cases, server system 201 can automatically optimize the interface model (e.g., DOM) before sending the interface model to client device 210. For example, user interface server engine 204 can compress the interface model (e.g., DOM) as much as possible before sending the interface model to client device 210. Compression of the interface model can include generating an incremental interface model by determining the difference between a current interface model (defining the current user interface) and a previous user interface model (defining the previous user interface) determined by application user interface engine 206. In some cases, the interface model can be further compressed by applying one or more standard compression algorithms (e.g., zip compression to place the DOM in a .zip file format).

[0091] As described above, in some cases, the application UI engine 206 can generate a new interface model (e.g., DOM), and the UI server engine 204 can determine the difference ("delta") between the new interface model and the previous interface model based on the received interface events. It is usually simpler for the application to have the UI server engine 204 determine the incremental interface model (e.g., for a given state, the application has a given appearance). In some cases, the application UI engine 206 can generate the incremental interface model. For example, some applications may prefer to have the application UI engine 206 manage the delta (e.g., the application will indicate what is new and what is different).

[0092] In some cases, the user interface server engine 204 can send the interface model 209 (eg, DOM) to the user interface client engine 214 of the client device 210 (in Figure 3In some cases, the user interface server engine 204 may send an incremental interface model (e.g., incremental DOM) to the user interface client engine 214 to indicate only the changes or differences between the current user interface and the previously generated user interface (e.g., the last user interface generated before the current user interface), without sending the entire interface model describing the entire user interface. In some implementations, the interface model (e.g., DOM) or the incremental interface model (e.g., incremental DOM) may be sent as an object in a format such as a text-based JavaScript Object Notation (JSON) object or other appropriate object for representing structured data. In some cases where DOM is used as an example of an interface model, the incremental DOM may include the structure of the entire DOM, but only the leaves (or nodes) of the DOM that include the changed graphical elements may include data. The leaves of the DOM that do not include the changes may be left empty (without data) in the incremental DOM. As further described below, the client device client code (of the user interface client engine 214) may optimize a portion of the screen to be redrawn based on the received incremental DOM. In some examples, a complete interface model (e.g., complete DOM) may always be sent when a change to the interface model is generated. In some examples, a complete interface model can be sent only when a user navigates from one application to another and / or in other situations (e.g., when an application is first launched, etc.). In some examples, an incremental interface model can be sent when an application changes its interface model (e.g., based on processing by the application user interface engine 206).

[0093] In some examples, the interface model (e.g., DOM) or incremental interface model can be compressed by reusing certain properties shared by graphical elements (e.g., menu items) across the user interface. For example, in order to simplify the rendering of the user interface by the client device, the server system 201 can use a constant "style" that can be reused without "rules". For example, different menu items can share multiple properties, or some properties may have a small amount of differences (e.g., the title and x-coordinate position of the menu item are different, while the color, font, size, height, width, etc. are the same, or the menu item may be different only when it is highlighted, etc.). In such an example, the server system 201 can be designed so that it does not use cascading style sheets (CSS), which may be very demanding on computing resources (e.g., CPU) in some cases. CSS is a mechanism for adding styles (e.g., fonts, colors, spacing, etc.) to user interfaces or other electronic media (e.g., web documents, etc.). The constant style implemented by the server system 201 can be executed using "HTML" class syntax.

[0094] Using DOM as an example of an interface model, the complete DOM can be called Intra-DOM (I-DOM or iframe), and the incremental DOM can be called Predictive-DOM (P-DOM or pframe). An example of I-DOM (intra) corresponding to the PoC main menu (uncompressed) is as follows:

[0095] ["iframe",{"styles":{"backgroundColor":"#e4e4e4"},

[0096] "photo1":{"styles":{"top":"1%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth ":"1px","onclick":"yes","onmouseenter":"yes","backgroundColor":"white","borderStyle":"solid","boxShadow":"grey 10px 5px5px","borderColor":"gray"},"label0":{"styles":{"left":"10%","top":"20%","text":"My 1st Application"}}},

[0097] "photo2":{"styles":{"top":"12%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"ye s","backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label1":{"styles":{"left":"10%","top":"20%","text":"Photo carousel 1"}}},

[0098] "photo3":{"styles":{"top":"23%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"yes","backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label2":{"styles":{"left":"10%","top":"20%","text":"Photo animated"}}},

[0099] "pong":{"styles":{"top":"34%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"yes","backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label3":{"styles":{"left":"10%","top":"20%","text":"Pong"}}},

[0100] "connect4":{"styles":{"top":"45%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"yes","backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label4":{"styles":{"left":"10%","top":"20%","text":"Connect Four"}}},

[0101] "tele":{"styles":{"top":"56%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"yes","backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label5":{"styles":{"left":"10%","top":"20%","text":"Program Guide"}}},

[0102] "radio":{"styles":{"top":"67%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"yes","backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label6":{"styles":{"left":"10%","top":"20%","text":"Radios"}}},

[0103] "clock2":{"styles":{"top":"78%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"yes", "backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label7":{"styles":{"left":"10%","top":"20%","text":"Clock2"}}},

[0104] "abra":{"styles":{"top":"89%","left":"20%","width":"60%","height":"10%","transition":"all1.00s","borderWidth":"1px","onclick":"yes","onmouseenter":"yes"," backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none","borderColor":"gray"},"label8":{"styles":{"left":"10%","top":"20%","text":"Zapper"}}}} ]

[0106] An example of a P-DOM(inter) generated when user a moves down one item (e.g., an input event corresponding to an arrow key press) is as follows:

[0107] ["pframe",{

[0108] "photo1":{"styles":{"backgroundColor":"#e4e4e4","borderStyle":"hidden","boxShadow":"none"}},

[0109] "photo2":{"styles":{"backgroundColor":"white","borderStyle":"solid","boxShadow":"grey10px 5px 5px"}}} ]

[0111] As shown in the P-DOM ("pframe") above, only two menu items are listed (i.e., the changed menu items), with only the properties that have changed. In this case, there are no objects added or deleted, only property changes. The property changes in the example P-DOM above include the background color, border style, and box shadow color of "photo1" and "photo2". If an object is added in the current user interface model, it will only be added in the incremental DOM. If an object in a previous user interface model is deleted in the current user interface model, the object can be marked as "deleted" in the incremental DOM of the current user interface model.

[0112] Here is another example of a P-DOM that corresponds to scrolling up a list of movies. <down-arrow>On key press, the top movie in the list disappears and another movie at the bottom is added to the list. For the "movie0-12785" object, it is marked as "deleted" to be removed from the client screen. In addition, the "boxShadow" is also removed from "movie0-40418". Another option would be to set "boxShadow" to "none". In the example below, the movies "movie0-53986" and "movie0-30877" are only moved vertically, and only the "top" property of these movies is refreshed. The properties of the two movies "movie0-40418" and "movie0-37828" change more drastically when they gain or lose focus, and therefore receive more changes.

[0113]

[0114]

[0115]

[0116] etc.

[0117] In some implementations, the server system 201 can manage two or more interface models (e.g., two or more DOMs). For example, the server system 201 can manage a system DOM for a global system application and an application DOM for an application. In such an example, the global system application can control a client device (e.g., an operator application in the case of pay TV), while the user can mainly interact with the application. In another example, an application (e.g., a messaging, email, or alarm application) can be displayed on top of the current application (e.g., an EPG, a game application, or other application). For example, an application can be displayed as a pop-up on the current application. A pop-up application can provide an option to jump to a full-screen related application (e.g., a pop-up application can indicate that an email has been received, and an option to jump directly to a complete email application). In such an example, even when only one application is visible to the user, the server system 201 can manage multiple interface models for multiple applications in parallel. Managing multiple interface models allows quick switching from one application user interface to another application user interface.

[0118] As described above, the user interface server engine 204 of the server system 201 can send the interface model 209 (e.g., DOM) or a portion thereof with changes or differences from a previous interface model (e.g., an incremental interface model, such as an incremental DOM) to the user interface client engine 214 of the local interface generator 212 of the client device 210 (in the Figure 3 At step 310 of ). The graphics rendering engine of the client device can render the user interface based on the received interface model or incremental interface model. Using an object model (e.g., DOM) can enable the client device to avoid processing to determine the state information of the user interface (e.g., features such as the layout of the user interface or changes in the layout), such as performing layout calculations. For example, the application user interface engine 206 can pre-calculate the properties of the graphic elements (e.g., the position, size, color and / or other properties of the graphic elements) represented by the interface model or incremental interface model (e.g., DOM or incremental DOM), so that the client device 210 does not need to perform layout calculations and / or other processing required to determine the properties of the user interface. Alternatively, the client device can implement a routine (e.g., a graphics routine) to generate a user interface based on the DOM or changes in the DOM. The client device 210 can read the object model or incremental interface model received for a given user interface, and can render the user interface defined by the object model or incremental interface model. For example, using the DOM or incremental DOM, the graphics rendering engine 218 of the client device 210 can convert a graphics object tree from the DOM or incremental DOM into a list of graphics routine calls. Based on the interface properties (e.g., state information) described in the object model or incremental interface model, the graphics rendering engine 218 can generate drawing commands. In some cases, when the client device 210 receives the incremental DOM, the graphics rendering engine 218 can optimize the part of the screen to be redrawn based on the received incremental DOM. Receiving the incremental DOM can simplify the client device 210's decision on which part of the screen should be redrawn. In some cases, the graphics rendering engine 218 knows nothing about interface events and can perform graphics automation based on the DOM or incremental DOM. In some examples, the graphics rendering engine 218 can perform animations. For example, the properties sent in the DOM can be used to describe the animation. In an illustrative example, the graphics rendering engine 218 can perform only transition animations, in which the application can provide new properties to the DOM object associated with the time it takes to make the transition. Only transition animations (or "transition" animations) are a type of animation in CSS3 / HTML. For example, instead of just changing the properties immediately (e.g., the upper left position of an object), it can be expressed in the application that changes will occur according to time parameters (e.g., changes will occur within "n" seconds). The time parameter can be expressed by setting the "transition=time" property for the object.

[0119] In some cases, the object model or incremental object model (e.g., DOM or incremental DOM) can be layered by the server system 201, or can be sent to the client device 210 without being layered by the server system 201. Layering is the action of finding where to display objects in the user interface. When layering, many positioning decisions need to be made. For example, an interface object (e.g., a graphical object or element) has properties or attributes that indicate characteristics such as the first object is to the right of the second object, the third object is below the fourth object, etc. Because the positions of the objects are interdependent, when changing the properties, the layer will cause many calculations. In one example, the "y" position of the first object can depend on the size of the font of the second object located near the first object. In some examples, object properties that have side effects on adjacent objects can be prohibited, which may be sufficient for designing user interfaces for full-screen applications (e.g., user interfaces for phones, tablets, televisions, media devices such as set-top boxes or other console devices, etc.). In some examples, layering can be performed by the server system 201 to generate a layered interface model (e.g., a layered DOM), and the layered interface model can be sent to the client device 210. In some cases, a layering plug-in may be implemented between a user interface application and a user interface server system, which will be responsible for layering the objects without changing the rest of the system.

[0120] The client device 210 may also receive media content (e.g., video, audio, graphics, images, combinations thereof, or other media) (also referred to herein as "media") from one or more media sources 217. Since the client device 210 renders a UI that supports the presentation of media content, the server system 201 may not need to access one or more media sources 217, which can reduce the processing burden of the server system 201. The local interface generator 212 of the client device 210 can integrate media content from one or more media sources 217 into the rendered graphics. For example, when the received media is encrypted, the decryption engine 222 can decrypt the media (and / or any other media). For example, the decoding engine 224 can decode the video using any appropriate encoding technology based on the format of the received video (e.g., H.265 / High Efficiency Video Coding (HEVC), H.264 / Advanced Video Coding (AVC), MPEG, or other video formats). The media synthesis engine 220 can combine the rendered user interface from the graphics rendering engine 218 with the decrypted and / or decoded media from the decoding engine 224. The combined data may then be sent to the display 215. As previously described, in some implementations, the display 215 may be part of the client device 210. For example, the client device 210 may be a mobile device, and the display 215 may be a display screen of the mobile device. In another example, the client device 210 may be a networked television (e.g., a television with a WiFi modem and transceiver), and the display 215 may be a display screen of the television. In some implementations, the display 215 may be separate from the client device 210. For example, the client device 210 may be a set-top box, and the display 215 may be part of a television connected to the set-top box.

[0121] Compared with existing solutions with graphics rendering, video compression, video encoding, etc., the above-mentioned server-based user interface is a significant improvement, which will save a lot of server computing (e.g., CPU, RAM, cache, etc.) resources (e.g., a ratio of 1 / 50). For example, in a video-based system, the server system 201 does not need to process the video (e.g., compression / encoding, etc.) and send the video to the client device 210. In addition, instead of the communication of the generated user interface, the elements of the user interface, or both of the media content, based on the communication of the interface model for rendering the user interface, the bandwidth amount of the communication between the server system 201 and the client device 210 can be minimized. Another advantage is that the use of the server-based system described herein will greatly improve the startup speed of the application. For example, when a user starts an application on a client device, since the application has been started and run on the server system 201, the application can be loaded in real time. For illustration, the application on the server system 201 (e.g., an electronic program guide (EPG) or other application) will be shared between many concurrent client devices and their users. When a client device starts an application that is not yet running on the server system 201, the application is loaded by the server system 201 and can be added to the list of applications running on the server system 201. For example, when a client device starts an application, the server system 201 can determine that the application is triggered by the client device. The server system 201 can determine that the application is not running on the server system 201, and in response to determining that the application is not running on the server system 201, the server system 201 can add the application to the list of applications running on the server system 201. The list of applications can be accessed by multiple client devices using the server computer. For example, when a new client device (and a new user of a client device) begins to interact with an application, the application (e.g., EPG) may have been started and is running by the server system 201 for access by other client devices and users, so there is no startup time to load the application or load the application data (e.g., EPG data) for the new client device and user. In the application user interface engine 206 (e.g., sandbox), applications share common data for all users (e.g., the EPG database can be completely in RAM), and there is little need for user context (e.g., one or more states, such as the current DOM and connections). In existing solutions where the application runtime engine is on the client device, the application on startup must be loaded and initialized, which increases latency.

[0122] Another benefit of the server-based user interface described in this article is related to security. For example, a side effect of a server-based user interface is that the information sent to the client device by the server system (e.g., server system 201) is information related to graphics-related information (e.g., state information of one or more graphics objects of the user interface), in which case there is no exposure to the form of an application programming interface (API). Therefore, such a scheme provides a secure way to implement a user interface. This is an advantage over existing server-based schemes, in which the client device calls (e.g., sends a request) to the server via an API to obtain content (e.g., information such as movies, search results, etc.), leaving data in such an API call opens a security vulnerability (e.g., an unauthorized third party can access keys, sensitive data, and / or other information included in the API call).

[0123] Figure 4 is a diagram showing an example of a document object model (DOM) 400 that defines state information of a user interface having three menu items, which are labeled Item 1, Item 2, and Item 3. The three menu items include text graphic elements. Figure 5 4 is a diagram showing an example of a DOM 500 that defines different state information of a user interface based on received events. For example, the application user interface engine 206 can generate a DOM 400 of a user interface in which the first menu item (for item 1) is highlighted in red text and the other two menu items (for item 2 and item 3) are in black text. In response to an input event (e.g., in response to an input event such as<ARROW DOWN> key is pressed), the application user interface engine 206 can generate a new DOM 500, in which the second menu item (for item 2) is highlighted in red text, and the other two menus (for item 1 and item 3) are black text.

[0124] exist Figure 4 and Figure 5 In the example of , the only changes between DOM 400 and DOM 500 are the color properties of the first item and the second item. Figure 6 is an example of a delta DOM 600 that includes only the differences between DOM 400 and DOM 500. For example, delta DOM 600 includes only an indication that the color of item 1 has changed to black and an indication that the color of item 2 has changed to gray. Figure 7 6 is a diagram showing an example of a change in the user interface based on the incremental DOM 600. For example, based on the indication that the item 1 menu item has changed from gray to black Figure 6 Incremental DOM 600 shown in FIG. 600 , the "Open" menu item (corresponding to Figure 6 The "Item 1" in the first screen of the user interface) changes from gray in the first screen of the user interface to black in the second screen of the user interface. In addition, based on the delta DOM 600 indicating that the Item 2 menu item has changed from black to gray, the "Close" menu item (corresponding to Figure 6 In this case, objects that have not changed may not be sent to the client device 210. For objects that have changed, only the properties that have changed are sent.

[0125] In some examples, since the server-based system described herein (e.g., server system 101 and / or server system 201) maintains and manages the user interface of one or more applications, remote client control of the user interface of the application displayed by the client device can be efficiently performed, making it possible for the server-based system or another client device to control the client device (e.g., in the case of a remote desktop). For example, since the application may have been started and run on the server system 201, the computer of the server system 201 can display the same content of the application displayed by the client device, and the client device receives the interface model (or incremental interface model) for the application from the server system 201. In some cases, remote client control can enable content to be replayed, thereby enabling problems in presentation to be discovered and / or providing detection of problems. In another example, a first client device can remotely control the user interface of an application displayed by a second client device. For example, the same interface model or incremental interface model (e.g., DOM or incremental DOM) can be sent to the first device and the second device. Using the server-based system described herein can make it easier to perform client control or remote desktop operations in such examples, because the user interface of the application is rendered by the server system and can be sent by the server system to both the first client device and the second client device. In some cases, the first client device and the second client device can both provide input to the server system for the instance of the application, and the server system can modify the user interface based on the input from both the first client device and the second client device. In this case, users of the first client device and the second client device can collaborate and interact with the same user interface of the application instance.

[0126] In some cases, when using certain client devices (e.g., desktop computers with mice or other input devices, mobile phones, tablet computers, wearable devices, or other devices with touch screens, gesture-based and / or other interfaces, etc.), a large number of input events can be sent from the client devices to the server system (e.g., server system 201). In one illustrative example, 100 events can be generated per second when the mouse moves from one point to another. Touch events on the touch screen can also generate a large number of events. From a resource perspective (e.g., bandwidth, computing, memory, etc.), such a large number of events may be too cumbersome for the server system. For example, the number of raw events may quickly become too large for real-time transmission and processing by the server system.

[0127] In some examples, events from client devices can be throttled. For example, a new way to process certain types of input (e.g., swipe and / or other gestures including repeated movement of input) is provided herein. For example, a swipe or other gesture or input on a client device (e.g., on a mobile phone, tablet computer, etc.) can be used to scroll a list of items. When a user quickly slides on a touch screen using a mouse, etc., some form of automatic repetition occurs, causing the items to continue to scroll, after which the menu items begin to gradually slow down. Then, the user will need to slide again and again to scroll the long list, resulting in the generation of many input events. By detecting that the user has kept a finger, a fixed-point device, a mouse or other input tool at the end of the input (e.g., at the end of the sliding movement), and continuing the scrolling operation until the input is released, the input event (e.g., an event based on sliding and / or other gesture-based events) can be throttled. In an illustrative example, the server system can detect that the user has kept a finger, a fixed-point device or other input tool on the touch interface at the end of the sliding movement (e.g., the user slides a finger and keeps the finger at the position at the end of the sliding). In response to detecting that the user maintains the input tool (e.g., finger, etc.) at the position at the end of the sliding movement, the server system will continue the scrolling in the same direction until the user removes their finger. In some cases, the server system can detect the speed of the input and can perform continuous scrolling operations based on the speed (e.g., faster scrolling can be performed in response to faster input). This scheme for throttling input reduces the number of input events generated and allows the user to perform fewer input gestures to scroll the list of items.

[0128] In some cases, about keyboard events, client devices can send key press (keyboard button is pressed) and key release (keyboard button is released) events.For keyboard events, all event processing is performed by server system (e.g., server system 201).In this case, automatic repeat management can be realized, wherein repeated release events are not sent to server system.In some cases, server system can provide (e.g., send or transmit) updates to client more than the updates that client or network can handle.In some examples, for example, in order to ensure that server system does not provide too many repeated updates, client device can send a message (referred to as "ready message") indicating that client is ready when client determines that it is ready to accept more repeated updates.Server system can wait until it receives ready message from client device and then send update to client device.In some examples, client device can provide (e.g., send or transmit) message to server system to indicate that server system pauses or stops sending update.This message can be referred to as "pause x message", wherein "x" can be a delay indicator of how long server system will wait before providing further update to client device. In some cases, a client device may send a PAUSE x message when the client gets more updates than it can handle (e.g., the number of updates exceeds a threshold number of updates such as 10 updates, 50 updates, or other appropriate number, or other threshold indicating that the client device has received too many updates).

[0129] In some examples, an input event (e.g., mouse event, touch event, etc.) handler may be implemented. For example, in HTML and / or JavaScript (JS), functions may be declaratively associated to handle events. An illustrative example is as follows:

[0130] <button onclick＝”javascript function here()”> Click Me< / button>

[0131] The difficulty is that the interface model (e.g., DOM) is sent to and processed by the client device, while the function is executed by the server computer of the server system (e.g., server system 201). The server system (e.g., server system 201) can use an input event handler to convert a function reference (e.g., HTML and / or JS function reference) into a unique function identifier (ID). For example, a first function can be mapped or associated with a first function ID, a second function can be mapped or associated with a second function ID, a third function can be mapped or associated with a third function ID, and so on. The server side can send the function ID to the client device. In response to an event detected based on user input, an application-level event, or a system-level event, the client device can associate a function with the event, locate the ID associated with the function, and send the ID to the server system. The server system 201 can then call the function and execute the function to determine how to modify the user interface. In some examples, the client device can send a list of N function-ID pairs or pairs (e.g., including a target object, a function ID) generated by the event. The server system can call these N functions, passing information about the event (e.g., to the application user interface engine 206 and / or the application domain engine 208). By using such input event handlers, the server system can execute functions without requiring the application user interface to change the way it operates (e.g., the application can continue to declaratively associate functions to handle specific events).

[0132] In some examples, a server system (e.g., server system 201) can maintain a remote system object for each connection with a client device. For example, a remote system object can include data represented as an instantiation of a data object structure that stores information related to the system (including connection information). Remote system objects can persist so that multiple applications can rely on existing connections to improve efficiency and speed, such as for communication of client events across different applications. Each connected remote system object can be shared by some or all applications running for a client device and / or a user of the client device. The effect of maintaining a remote system object for each connection is that, from the perspective of the client device, a multi-tasking system is created. A user can switch back and forth between a first application and a second application (e.g., from a game to an EPG), and execution can be resumed in the second application in the state in which the user previously switched away from the second application.

[0133] In some implementations, a server system (e.g., server system 201) can adapt to the capabilities of different client devices. For example, a filter can be added to an interface model (e.g., to a user interface tree of a DOM or deltaDOM) to define certain restrictions imposed on what can be included in the interface model for a particular client device. In an illustrative example, if a client has known restrictions on the handling of certain pixel sizes or positions, such as being able to render and display high-definition resolution or lower resolution video images, a filter or restriction can be placed on the interface model for that client device to indicate a maximum video resolution of 1280×720 pixels (or other high-definition resolution) that can be displayed by the client device.

[0134] Fig. 9 1 is a flowchart showing an example of a process 900 for generating user interface content using the techniques described herein. At box 902, process 900 includes receiving (e.g., by a server computer or other device) a user interface event. In the case where process 1000 is performed by a server computer, the server computer can be a part of server system 101 or server system 201. The user interface event corresponds to the user interface of the client device. For example, the user interface event can be based on user input received by the user interface. In another example, the user interface event can be a system-based event related to the user interface. The client device may include any appropriate device. In some examples, the client device includes a media device. The user interface may include any type of user interface, such as a graphical user interface, a media content guide (e.g., an electronic program guide (EPG)), an interface of a game application, an interface of a mobile application, an interface of a smart TV application, or any other appropriate user interface.

[0135] At block 904, process 900 includes determining an application associated with the user interface event. For example, the application associated with the user interface event can be determined based on a connection between a server computer (or other device) and a client device. The connection can include a socket or any other suitable connection.

[0136] At box 906, process 900 includes generating an interface model using an application associated with the user interface event. The interface model defines state information for one or more graphical objects of the user interface. The state information is obtained from the user interface event. In some cases, process 900 may include determining the state information based at least on the user interface event. In some examples, the generated interface model may include a portion of a larger interface model. For example, generating the interface model includes generating a portion of the interface model. The portion of the interface model is associated with a portion of the user interface. For example, the portion of the interface model may define state information for a subset of the user interface (e.g., for a page or a portion of a page of the user interface).

[0137] In some examples, process 900 may include compressing the interface model. In such examples, the compressed interface model is sent to the client device. In some cases, the interface model may be considered a first interface model. The interface model may be compressed by determining a difference between state information of one or more graphic objects and previous state information of the one or more graphic objects, and by generating a second interface model that includes only data of the one or more graphics having state information different from the previous state information. The second interface model may also be referred to as an incremental interface model.

[0138] In some examples, the interface model may include a hierarchical data structure having one or more graphic objects. For example, the interface model and the incremental interface model may include a document object model (DOM). Each of the one or more graphic objects may have one or more attributes defined by state information. In some cases, the one or more graphic objects include at least one of a frame, a line, an image, an icon, a text, a content item to be displayed in the user interface, or any combination thereof. Each of the one or more graphic objects may include one or more attributes defined by state information. The one or more attributes of the one or more graphic objects may include at least one of the position, shape, color of the graphic object, any combination thereof and / or other appropriate attributes.

[0139] At block 908, process 900 includes sending the interface model (or the second interface model) to the client device (e.g., via a server computer or other device). In some cases, the interface model may be sent to another server computer (e.g., one or more server computers 140 or another device other than a server computer). The interface model enables the client device (or another server computer or other device) to render a user interface.

[0140] In some examples, process 900 may include determining that an application is triggered by a client device and determining that the application is not running on a server computer. Process 900 may include adding the application to a list of applications accessible to multiple client devices using the server computer. For example, in response to determining that the application is not running on the server computer, process 900 may add the application to a list of applications accessible to multiple client devices using the server computer. As described above, when a new client device begins to interact with an application in the application list, the application may have already been started and running by the server system, so that there is no startup time for the client device to load the application or load application data.

[0141] Fig.10 1 is a flow chart showing an example of a process 1000 for generating user interface content using the techniques described herein. At box 1002, process 1000 includes receiving (e.g., by a server computer or other device) a user interface event. In the case where process 1000 is performed by a server computer, the server computer can be a part of server system 101 or server system 201. The user interface event corresponds to the user interface of the client device. For example, the user interface event can be based on user input received by the user interface. In another example, the user interface event can be a system-based event related to the user interface. The client device may include any appropriate device. In some examples, the client device includes a media device. The user interface may include any type of user interface, such as a graphical user interface, a media content guide (e.g., an electronic program guide (EPG)), an interface of a game application, an interface of a mobile application, an interface of a smart TV application, or any other appropriate user interface.

[0142] At block 1004, process 1000 includes determining an application associated with the user interface event. For example, the application associated with the user interface event can be determined based on a connection between a server computer (or other device) and a client device. The connection can include a socket or any other suitable connection.

[0143] At block 1006, process 1000 includes generating an interface model using an application associated with the user interface event. The interface model defines state information for one or more graphical objects of the user interface. The state information is obtained from the user interface event. In some cases, process 1000 may determine the state information based at least on the user interface event.

[0144] At block 1008, process 1000 includes determining a difference between state information of one or more graphical objects and previous state information of the one or more graphical objects. At block 1010, process 1000 includes generating an updated interface model that includes data of the one or more graphical objects having state information different from the previous state information. For example, the updated interface model may include only data of the one or more graphical objects having state information different from the previous state information (in which case data of other graphical objects having state information that is not different from the previous state information will not be included in the updated interface model). The updated interface model may also be referred to as an incremental interface model or a second model (or second interface model).

[0145] In some examples, the interface model may include a hierarchical data structure having one or more graphic objects. For example, the interface model and the incremental interface model may include a document object model (DOM). Each of the one or more graphic objects may have one or more attributes defined by state information. In some cases, the one or more graphic objects include at least one of a frame, a line, an image, an icon, a text, a content item to be displayed in the user interface, or any combination thereof. Each of the one or more graphic objects may include one or more attributes defined by state information. The one or more attributes of the one or more graphic objects may include at least one of the position, shape, color of the graphic object, any combination thereof and / or other appropriate attributes.

[0146] At block 1012, process 1000 includes sending the updated interface model to the client device (e.g., by a server computer or other device). In some cases, the updated interface model may be sent to another server computer (e.g., one or more server computers 140 or another device other than a server computer). The updated interface model enables the client device (or another server computer or other device) to render a user interface.

[0147] In some examples, processes 800, 900, and 1000 may be performed by a computing device or apparatus. For example, processes 800, 900, and 1000 may be performed by Figure 2 The server system 201 shown (e.g., a server computer or multiple server computers of the server system 201) is executed. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more output devices and / or other components of a computing device (e.g., a server computer or other device), which are configured to perform the steps of processes 800, 900, 1000 and / or other processes described herein. The computing device may include a memory configured to store data (e.g., interface models, incremental interface models, graphic elements and other user interface content and / or any other appropriate data for one or more applications) and one or more processors configured to process the data. The computing device may also include one or more network interfaces configured to transmit data. The network interface may be configured to transmit network-based data (e.g., data based on the Internet Protocol (IP) or other appropriate network data). In some implementations, the computing device may also include a display.

[0148] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include: one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The computing device may further include a display (as an example of an output device or as a supplement to an output device), a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive data based on an Internet Protocol (IP) or other types of data.

[0149] Processes 800, 900, and 1000 are illustrated as flow charts or logic flow charts, the operations of which represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media, which performs the operation when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement processing.

[0150] In addition, processes 800, 900, and 1000 can be performed under the control of one or more computer systems configured with executable instructions, and can be implemented as a common code (e.g., executable instructions, one or more computer programs, or one or more applications) on one or more processors, implemented by hardware, or a combination thereof. As described above, the code can be stored on a computer-readable or machine-readable storage medium, such as in the form of a computer program including multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0151] Fig.11 The architecture of a computing system 1100 is shown, wherein the components of the system 1100 communicate with each other electrically using a system connection 1105 such as a bus. The example system 1100 includes a processing unit (CPU or processor) 1110 and a system connection 1105, which couples various system components including a system memory 1115 such as a read-only memory (ROM) 1120 and a random access memory (RAM) 1125 to the processor 1110. The system 1100 may include a cache of high-speed memory that is directly connected to, adjacent to, or integrated as part of the processor 1110. The system 1100 may copy data from the memory 1115 and / or storage device 1130 to the cache 1112 for quick access by the processor 1110. In this way, the cache may provide a performance boost, avoiding delays in the processor 1110 while waiting for data. These and other modules may control or be configured to control the processor 1110 to perform various actions. Other system memory 1115 may also be used. Memory 1115 may include a variety of different types of memory with different performance characteristics. Processor 1110 may include any general purpose processor and hardware or software services, such as service 11132, service 2 1134, and service 3 1136 stored in storage device 1130, which are configured to control processor 1110 as well as special purpose processors, where the software instructions are incorporated into the actual processor design. Processor 1110 may be a completely independent computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0152] To enable a user to interact with the system 1100, the input device 1145 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 1135 can also be one or more of many output mechanisms known to those skilled in the art. In some cases, a multimodal system can enable a user to provide multiple types of input to communicate with the system 1100. The communication interface 1140 can generally govern and manage user input and system output. There is no restriction on the operation on any particular hardware arrangement, so when developing them, the basic features here can be easily replaced for improved hardware or firmware arrangements.

[0153] The storage device 1130 is a non-volatile memory and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 1125, a read-only memory (ROM) 1120, and a mixture thereof.

[0154] Storage device 1130 may include services 1132, 1134, 1136 for controlling processor 1110. Other hardware or software modules are contemplated. Storage device 1130 may be connected to system connection 1105. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium connected to the necessary hardware components (such as processor 1110, connection 1105, output device 1135, etc.) to implement the function.

[0155] For clarity of explanation, in some cases, the present technology may be presented as including various functional blocks, including functional blocks including steps or routines in methods, devices, and device components implemented in software or a combination of hardware and software.

[0156] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0157] The method according to the above example can be implemented using computer executable instructions stored in or available from a computer readable medium. Such instructions may include, for example, instructions and data that cause or configure a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a certain function or group of functions. Some of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions (such as assembly language), firmware, or source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the example include disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0158] Various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits and steps are generally described above with respect to their functions. Whether this function is implemented as hardware or software depends on specific application and the design constraints imposed on the total system. The technician can implement the described function in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the application.

[0159] Devices implementing methods or processes according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of forms. Typical examples of such form factors include laptop computers, smart phones, personal computers in small form factors, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-on cards. By way of further example, such functions may also be implemented on circuit boards among different chips or on different processes performed in separate devices.

[0160] The technology described in this article can also be implemented in electronic hardware, computer software, firmware or any combination thereof. Such technology can be implemented in any of a variety of devices, for example. A general-purpose computer, a wireless communication device mobile phone, or an integrated circuit device with multiple uses, the uses of which include applications in wireless communication devices and other devices. Any features described as modules or components can be implemented together in an integrated logic device, or can be implemented separately as discrete but interoperable logic devices. If implemented in software, the technology can be implemented at least in part by a computer-readable data storage medium containing a program code, the program code including instructions for performing one or more of the above methods when executed. The computer-readable data storage medium can form a part of a computer program product, which can include packaging materials. The computer-readable medium can include a memory or a data storage medium, for example, a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, the technology may be implemented at least in part through a computer-readable communication medium, such as a propagated signal or wave, which carries or communicates program code in the form of instructions or data structures and can be accessed, read and / or executed by a computer.

[0161] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor can be a microprocessor; however, in an alternative solution, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" used in this article can refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described in this article.

[0162] Instructions, media for communicating such instructions, computing resources for executing them, and other structures for supporting such computing resources are the means for providing the functionality described in these disclosures.

[0163] Although various examples and other information are used to explain various aspects within the scope of the appended claims, limitations on the claims should not be implied based on specific features or arrangements in such examples, as those of ordinary skill in the art will be able to use these examples to derive a variety of implementations. In addition, although some topics may have been described in language specific to examples of structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or actions. For example, such functionality may be distributed differently or performed in components other than those identified herein. Specifically, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

[0164] Declarative language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the statement. For example, declarative language stating "at least one of A and B" means A, B, or A and B. In another example, declarative language stating "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. Language stating "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, declarative language stating "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0165] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operation, by programming electronic circuits (e.g., a microprocessor or other appropriate electronic circuitry) to perform the operation, or any combination thereof.

[0166] One of ordinary skill will recognize that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols without departing from the scope of the present description.

Claims

1. A method for processing user interface content, the method comprising: Receiving, by the server computer, a first user interface event, the first user interface event corresponding to a user interface of a client device; determining a first application associated with the first user interface event; dispatching the first user interface event to the first application for the client device; generating an interface model using the first application associated with the first user interface event, wherein the interface model defines first state information of one or more graphical objects of the user interface, the first state information being obtained from the first user interface event; sending the interface model to the client device by the server computer, the interface model enabling the client device to render the user interface; receiving, by the server computer, a second user interface event different from the first user interface event; determining a second application associated with the second user interface event; dispatching the second user interface event to the second application for the client device; generating a second interface model using the second application associated with the second user interface event, wherein the second interface model defines second state information of one or more graphical objects of the user interface, the second state information being obtained from the second user interface event; as well as The server computer sends the second interface model to the client device. 2 . The method of claim 1 , wherein the interface model comprises a hierarchical data structure having the one or more graphic objects, wherein a graphic object in the one or more graphic objects has one or more properties defined by the state information.

3. The method according to any one of claims 1 or 2, wherein the one or more graphical objects include at least one of a box, a line, an image, an icon, text, and a content item to be displayed in the user interface. 4 . The method of claim 2 , wherein the one or more properties of the one or more graphical objects include at least one of a position, a shape, and a color of a graphical object.

5. The method according to any one of claims 1 or 2, further comprising: The state information is determined based at least on the first user interface event.

6. The method according to any one of claims 1 or 2, further comprising: The interface model is compressed, wherein the interface model sent to the client device is a compressed interface model.

7. The method of claim 6, wherein the interface model is a first interface model, and wherein compressing the interface model comprises: determining a difference between the state information of the one or more graphics objects and previous state information of the one or more graphics objects; as well as A second interface model is generated, the second interface model including data of the one or more graphic objects having state information different from the previous state information.

8. The method of any one of claims 1 or 2, wherein the first application associated with the first user interface event is determined based on a connection between the server computer and the client device.

9. The method of any one of claims 1 or 2, wherein the client device comprises a media device, and wherein the user interface comprises a media content guide.

10. The method according to any one of claims 1 or 2, wherein generating the interface model comprises generating a portion of the interface model, the portion of the interface model being associated with a portion of the user interface.

11. The method according to any one of claims 1 or 2, further comprising: Determining that the first application is triggered by the client device; determining that the first application is not running on the server computer; as well as The first application is added to a list of applications accessible to a plurality of client devices using the server computer.

12. A system for processing user interface content, the system comprising: One or more processors; as well as A memory accessible to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a first user interface event, the first user interface event corresponding to a user interface of a client device; determining a first application associated with the first user interface event; dispatching the first user interface event to the first application for the client device; generating an interface model using the first application associated with the first user interface event, wherein the interface model defines state information of one or more graphical objects of the user interface, the state information being obtained from the first user interface event; sending the interface model to the client device, the interface model enabling the client device to render the user interface; receiving a second user interface event different from the first user interface event; determining a second application associated with the second user interface event; dispatching the second user interface event to the second application for the client device; generating a second interface model using the second application associated with the second user interface event, wherein the second interface model defines second state information of one or more graphical objects of the user interface, the second state information being obtained from the second user interface event; as well as The second interface model is sent to the client device.

13. The system of claim 12, wherein the interface model comprises a hierarchical data structure having the one or more graphical objects, each of the one or more graphical objects having one or more properties defined by the state information.

14. The system of any one of claims 12 or 13, wherein the one or more graphical objects include at least one of a box, a line, an image, an icon, text, and a content item to be displayed in the user interface.

15. A system according to any one of claims 12 or 13, wherein each of the one or more graphical objects includes one or more attributes defined by the state information, wherein the one or more attributes of the one or more graphical objects include at least one of a position, a shape, and a color of the graphical object.

16. The system of any one of claims 12 or 13, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to: The state information is determined based at least on the first user interface event.

17. The system of any one of claims 12 or 13, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to: The interface model is compressed, wherein the interface model sent to the client device is a compressed interface model.

18. The system of claim 17, wherein the interface model is a first interface model, and wherein compressing the interface model comprises: determining a difference between the state information of the one or more graphics objects and previous state information of the one or more graphics objects; as well as A second interface model is generated, the second interface model including data of the one or more graphic objects having state information different from the previous state information.

19. The system of any one of claims 12 or 13, wherein the first application associated with the first user interface event is determined based on a connection between the system and the client device.

20. The system of any of claims 12 or 13, wherein the client device comprises a media device, and wherein the user interface comprises a media content guide.

21. The system of any one of claims 12 or 13, wherein generating the interface model comprises generating a portion of the interface model, the portion of the interface model being associated with a portion of the user interface.

22. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to: receiving a first user interface event, the first user interface event corresponding to a user interface of a client device; determining a first application associated with the first user interface event; dispatching the first user interface event to the first application for the client device; generating an interface model using the first application associated with the first user interface event, wherein the interface model defines state information of one or more graphical objects of the user interface, the state information being obtained from the first user interface event; sending the interface model to the client device, the interface model enabling the client device to render the user interface; receiving a second user interface event different from the first user interface event; determining a second application associated with the second user interface event; dispatching the second user interface event to the second application for the client device; generating a second interface model using the second application associated with the second user interface event, wherein the second interface model defines second state information of one or more graphical objects of the user interface, the second state information being obtained from the second user interface event; as well as The second interface model is sent to the client device.

23. The non-transitory computer readable medium of claim 22, wherein the interface model is a first interface model, and wherein compressing the interface model comprises: determining a difference between the state information of the one or more graphics objects and previous state information of the one or more graphics objects; as well as A second interface model is generated, the second interface model including data of the one or more graphic objects having state information different from the previous state information.

24. The non-transitory computer-readable medium of any one of claims 22 or 23, wherein generating the interface model comprises generating a portion of the interface model, the portion of the interface model being associated with a portion of the user interface.

Images