Cloud operating system and communication method

Abstract

The invention discloses a communication method based on a cloud operating system. The communication method comprises the following steps: establishing network association between remote computing equipment and a plurality of local terminals, and taking the plurality of local terminals as an operating end of at least one user; providing a communication service station at the remote computing equipment and enabling the communication service station to access the distributed database, and establishing access between the plurality of local terminals and the communication service station so as to respectively present a communication page and / or cache communication data at each local terminal; determining that at least two local terminals have communication association; and receiving the communication data of the local terminals based on the communication service station or based on the user data in the distributed database, updating a communication page and / or communication data corresponding to each local terminal, and simultaneously transmitting the updated communication page and / or communication data to the local terminals associated with the communication.

Description

Technical Field

[0001] The present invention relates to an operating system based on cloud computing and a method thereof, and belongs to the technical field of software and communication. Background Art

[0002] Smart terminals, such as smartphones, car central control devices, smart wearable devices, smart home appliances, etc., have become network service terminals that people cannot live without. The operating systems of these smart devices are not only of different types, but also have large differences in versions and are severely fragmented, which leads to the need to take into account the requirements and habits of various types and versions of operating system platforms when developing, installing, upgrading and using user applications, which brings more and more troubles to program developers and users. On the other hand, the hardware configuration and replacement cycle of various smart terminals are quite different, making it difficult to improve the user experience synchronously. For example, due to the increasingly bloated APP, the entire smartphone needs to be replaced more frequently to obtain more local storage capacity, but the upgrade of other hardware of the mobile phone has excess performance, resulting in waste. For another example, the replacement frequency of car central control or smart home appliances is low, and the upgrade frequency and experience of user applications that require human-computer interaction are far behind similar programs on smartphones.

[0003] Therefore, there is a need for a cross-platform network-based operating system that is compatible with various smart terminals, provides a unified user application ecosystem, and offers rich user experience and easy program development. In addition, thanks to the continuous development of the new generation of mobile communication technology and the comprehensive coverage of network base station facilities, the data interconnection between smart terminals and external servers is faster, less delayed and more stable, providing a technical foundation and operational guarantee for network-based operating systems. Summary of the invention

[0004] The present invention provides a cross-platform cloud operating system and related methods, browsers, storage media, terminals, servers, etc., aiming to solve at least one of the technical problems existing in the prior art.

[0005] One aspect of the present invention relates to a cloud operating system, which is set in a remote computing device but not deployed in a local terminal, the remote computing device is associated with the local terminal through a network and the local terminal is used as a user operation terminal. The cloud operating system includes: a basic kernel, which is used to call the physical and / or virtualized hardware resources of the remote computing device according to a request; a runtime library, which is based on the basic kernel to provide a runtime environment of at least one operating system platform; an interface service module, which corresponds to the runtime library to provide an application interface; and an application service module, which is used to provide an application interface and service docking to the local terminal.

[0006] Furthermore, the application service module includes: a device monitoring unit, used to obtain the hardware parameters and real-time operation status data of the local terminal; an interface generation unit, used to generate an operation interface of the user application according to the hardware parameters of the local terminal; a cloud control docking unit, used to maintain control interaction with the local terminal; a driver docking unit, used to handle the redirection between the hardware driver request required by the application in the remote computing device and the hardware driver request between the local terminal.

[0007] Furthermore, the local terminal has a display screen, network communication hardware and a local operating system, and the remote computing device includes any one or more of a cloud server, a virtual machine, a container and a personal computer, wherein at least a portion of the cloud operating system is deployed in a remote server, virtual machine or container.

[0008] Furthermore, the local terminal includes a cloud control desktop, which runs based on a browser framework in a local operating system of the local terminal, and the cloud control desktop includes: an interface control or plug-in, which provides a user operation interface and function extensions associated with the interface based on the browser framework; a local access unit, which is connected to the interface control or plug-in and is used to capture the user's real-time control data and information and hardware information of local applications accessing the local terminal; one or more service units, which are connected to the interface control or plug-in and are used for interaction of interface images, videos and audios between local and remote; one or more client units, which are respectively connected to the service unit and the local access unit, and are used to communicate and transmit with the proxy service program in the remote computing device.

[0009] Furthermore, at least one user application and at least one background service program are remotely run in an environment based on the application service module, wherein: after the authentication connection between the local terminal and the remote computing device, the device monitoring unit obtains the hardware data of the local terminal including the screen and the input device, and receives the data of the local application of the local terminal; the interface generation unit generates an interface environment for presenting the user application according to the interface style and preset operating habits corresponding to the user account and based on the hardware configuration parameters obtained by the device monitoring unit, wherein the running interface of the user application is intercepted by the background service program and the interface image is transmitted to the cloud control desktop of the local terminal; the cloud control docking unit is called by the background service program to receive the user's control data from the local terminal in real time and convert it into the control data of the remote computing device; the driving docking unit responds to the hardware resource request of the user application to redirect the source data for hardware execution of the remote computing device to the hardware of the local terminal.

[0010] Furthermore, at least a part of the interface generation unit is provided by an interface rendering engine of a browser frame of a remote computing device, so as to generate a running interface of the user application for a web page file received by the user application from a WEB server.

[0011] Furthermore, in response to the user application initiating a hardware resource request through the application program interface, the driver docking unit captures source data from the memory area of ​​the remote computing device, where the source data is the data that the remote computing device hardware is to receive from the memory area when the driver program triggers it, and then forwards the source data to the local terminal; in response to the instruction of the driver docking unit, the service unit in the background of the local terminal initiates hardware resource docking through the application program interface to receive the source data, and triggers the local hardware to execute the source data through the local driver program.

[0012] Furthermore, the remote computing device includes a cloud server and at least one virtual machine generated in the cloud server, in which at least a runtime library, an interface service module and an application service module of the cloud operating system are deployed; the user database of the cloud server and the system scheduling between multiple virtual machines are performed through the general agent service center to allocate corresponding virtual machines and cloud operating system data to the user account belonging to the local terminal.

[0013] Further, the basic kernel is a Linux kernel, and the runtime library includes any one or more of a Linux runtime component, an Android runtime component, a Windows runtime component, and an iOS runtime component. The interface service module includes an interpreter or engine for interpreting and executing the user application code.

[0014] Another aspect of the present invention relates to a remote interface generation method for the cloud operating system. The remote interface generation method includes the following steps: establishing a network association between a remote computing device and a local terminal and using the local terminal as a user operation terminal; obtaining a user account and hardware configuration parameters from the local terminal, wherein the hardware configuration parameters include the local operating system type, screen size, screen resolution, and input device type; obtaining the interface style and operation habits corresponding to the user account or the local operating system type from a user database associated with the remote computing device, generating a browser window in the browser frame of the remote computing device with the obtained screen size or screen resolution of the local terminal, configuring a renderer to generate an interface layout based on the interface style, and determining the operation gesture of the browser window with the operation habit, thereby serving as an interface environment for a user application running based on the browser frame; capturing an interface image of the browser window and transmitting it to the local terminal for display as an interface image.

[0015] Further, for the remote interface generation method, wherein: the renderer of the browser frame of the remote computing device is provided with an HTML interpreter and a CSS interpreter, and a JS engine is used to generate a running interface of the user application for the web page file and JS script received by the user application from the WEB server; the running interface is converted into an interface image through a plug-in running in the browser frame of the remote computing device; and a browser frame is provided in the local terminal to allow the browser control to receive the interface image for display.

[0016] Furthermore, the remote interface generation method includes the following steps: pre-adjusting the interface image to be transmitted by the remote computing device to the local terminal according to the current network transmission status, application scenario status, user instructions, control status or display resolution of the local terminal, so as to transmit at least a part of the dynamically generated interface image, and the pre-adjustment includes cutting, compressing or contrasting at least a part of the image.

[0017] Another aspect of the present invention relates to an interface image transmission method, which can be used in the above-mentioned cloud operating system. The interface image transmission method includes the following steps: establishing a control data interactive connection between a local terminal and a remote computing device, wherein the change of at least a part of the interface of the remote computing device is caused by the control data sent by the interactively connected local terminal; maintaining the transmission connection of the interface image of the remote computing device to the local terminal for at least one period of time; dynamically adjusting the interface image to be transmitted from the remote computing device to the local terminal according to the transmission interval of the interface image transmitted from the remote computing device to the local terminal, wherein the dynamic adjustment includes: after determining that an image transmission below an interval threshold occurs in a period of time, reducing the number of image transmissions and / or reducing the quality of at least a part of the interface image in the next period of time.

[0018] Furthermore, for the interface image transmission method, the step of dynamically adjusting the interface image to be transmitted from the remote computing device to the local terminal includes: real-time detection of the image transmission interval of the interface image in the same interface range from the remote computing device to the local terminal; if it is determined that the current image transmission interval is lower than a first time threshold, then the corresponding interface image to be transmitted is abandoned; if it is determined that the current image transmission interval is lower than a second time threshold, then the quality of the corresponding interface image to be transmitted is reduced; wherein, the first time threshold is less than the second time threshold.

[0019] Furthermore, for the interface image transmission method, the step of dynamically adjusting the interface image to be transmitted from the remote computing device to the local terminal includes: detecting the current transmission interval of the interface image in the same interface range from the remote computing device to the local terminal; if it is determined that the current transmission interval is higher than a third time threshold, then restoring the quality of the interface image to be transmitted, or collecting an interface image with higher image quality from the remote computing device for transmission to the local terminal.

[0020] Furthermore, the interface image transmission method also includes the following steps: enabling the local terminal to receive the interface image from the remote computing device; dynamically filtering the interface image from the remote computing device according to the operation interval monitored in real time on the local terminal, so as to display at least a portion of the filtered interface image on the local terminal.

[0021] Furthermore, the interface image transmission method may also include the following steps: determining that the control interval of the local terminal is less than a fourth time threshold within at least one time period; discarding at least a portion of the interface image received from the remote computing device for display on the local terminal; and reducing the frequency of control data sent by the local terminal to the computing device.

[0022] Further, for the interface image transmission method, the step of dynamically screening the interface images from the remote computing device includes: within at least one time period, if it is determined that the manipulation interval of the local terminal is greater than a fifth time threshold, then all the interface images for display on the local terminal are retained from the received multiple interface images; within at least one time period, if it is determined that the manipulation interval of the local terminal is greater than a sixth time threshold, then at least intermittently select interface images from the received multiple interface images to be displayed on the local terminal until the next manipulation of the local terminal occurs; wherein the sixth time threshold is greater than the fifth time threshold.

[0023] Furthermore, the interface image transmission method may also include the following steps: within at least one time period, if it is determined that the manipulation interval of the local terminal is greater than a seventh time threshold, the display frame rate of the local terminal is reduced until the next manipulation of the local terminal occurs.

[0024] Furthermore, the interface image transmission method may include the following steps: determining that the movement amplitude of the control point of the local terminal exceeds a preset amplitude threshold; retaining the interface images associated with the starting action and the ending action of the control point among the multiple interface images transmitted to the local terminal by the remote computing device caused by the control data generated by the control point of the local terminal, and reducing the remaining interface images for display on the local terminal; and periodically synchronizing the interface images between the local terminal and the remote computing device.

[0025] Another aspect of the present invention relates to a remote browser running on the above-mentioned cloud operating system. The remote browser comprises: a client unit, which is used for the remote browser to communicate with other applications running on the cloud operating system and communicate with a local terminal to obtain data from the local terminal and the remote computing device; an interface image acquisition unit, which is used to obtain at least a part of the interface image from the user interface of the web-based user application running on the remote browser; wherein the client unit and the interface image acquisition unit are respectively connected to an image interaction service program in the cloud operating system, and the image interaction service program implements the above-mentioned remote interface generation method.

[0026] Another aspect of the present invention relates to a local browser running on a local operating system. The local browser comprises: a client unit, which is used for the local browser to communicate with other applications running on the local operating system and to communicate with a remote computing device to obtain data from a local terminal and the remote computing device; a display component, which is used to display an interface image from the remote computing device; and an image interaction service unit connected to the display component, which is used to implement the above-mentioned interface image transmission method.

[0027] Another aspect of the present invention relates to a data transmission method for a cloud operating system, wherein the cloud operating system is deployed in a remote computing device, the remote computing device is associated with a local terminal through a network and the local terminal is used as a user operation terminal and a system interface display terminal of the cloud operating system, the local terminal and the remote computing device are respectively provided with one or more client units, and the remote computing device is provided with an agent unit that runs in the background. The data transmission method comprises the following steps: through a subscription protocol transmission channel, a client unit set in the local terminal or the remote computing device publishes one or more data message packets through the agent unit, so as to allow one or more other client units set in the local terminal or the remote computing device that have pre-subscribed to the data message packets to pull the data message packets to complete the consumption of all data message packets.

[0028] Further, for the data transmission method, wherein: user control data is transmitted from the local terminal to the remote computing device through a control data direct connection channel, and the control data direct connection channel has a first transmission priority; an image data direct connection channel is established based on a long-chain network communication protocol to transmit an interface image from the remote computing device to the local terminal, and the image data direct connection channel has a second transmission priority; the subscription protocol transmission channel has a third transmission priority, wherein the first transmission priority is higher than the second transmission priority or the third transmission priority.

[0029] Further, for the data transmission method, wherein: the remote computing device includes a cloud server and at least one virtual machine generated in the cloud server, and the user database of the cloud server is used to perform system scheduling between multiple virtual machines through the general agent service center, so as to allocate corresponding cloud operating system data and at least one virtual machine to the user account to which the local terminal belongs; the process running through the browser frame in the allocated virtual machine provides an interface environment for the user application, and is used to access the user database; the local terminal includes a cloud control desktop running based on a local browser frame, so as to receive an interface image generated by the browser frame of the virtual machine; the local terminal includes a first local client unit and a second local client unit respectively arranged inside and outside the cloud control desktop, and the virtual machine includes the agent unit and a first remote client unit and a second remote client unit arranged inside and outside the browser frame; and wherein the method includes the following steps: allowing the first local client unit or the second local client unit to allow the first remote client unit or the second remote client unit to pull the first data message packet through the first data message packet published by the agent unit; or allowing the first remote client unit or the second remote client unit to allow the first local client unit or the second local client unit to pull the second data message packet through the second data message packet published by the agent unit.

[0030] Furthermore, the data transmission method includes the following steps: enabling the second local client unit to directly obtain local data, wherein the local data includes any one or more of local storage data, local application data, local hardware data and local user feature data; enabling the first local client unit to directly communicate with the second local client unit to obtain the local data, and generating the first data message packet using at least a portion of the obtained data.

[0031] Furthermore, the data transmission method includes the following steps: enabling the first remote client unit or the second remote client unit of the virtual machine of the logged-in user account to obtain remote data, the remote data including virtual machine storage data, virtual machine hardware data or data associated with the user account in the user database of the cloud server; generating the second data message packet with at least a part of the obtained remote data.

[0032] Furthermore, the data transmission method includes the following steps: ensuring that the agent unit of each of the multiple virtual machines associated with the user account maintains a communication connection with the general agent service center of the cloud server; enabling the agent units of the multiple virtual machines to exchange data transmission packets, wherein the data transmission packets include user application data and one or more data message packets; and determining that the data message packets in the agent unit have been consumed by the client units in the multiple virtual machines associated with the user account.

[0033] Another aspect of the present invention relates to a data redirection method based on the above-mentioned cloud operating system, comprising the following steps: optionally based on a long link network protocol, establishing a communication connection between a local terminal and a remote computing device, obtaining a first communication docking address connected to the local terminal, and sending a local terminal identifier to the remote computing device; using the local terminal identifier to establish a communication connection between the remote computing device and a third-party server, and obtaining a second communication docking address connected to the third-party server; modifying the first communication docking address and the second communication docking address to be consistent, so that at least a part of the data transmission between the remote computing device and the third-party server is redirected to data transmission between the local terminal and the third-party server.

[0034] Further, for the data redirection method, multimedia data is intercepted from the data sent by the third-party server to the remote computing device, so that the multimedia data transmission between the remote computing device and the third-party server is redirected to the streaming data transmission between the local terminal and the third-party server. Another aspect of the present invention relates to a data redirection method, based on the above-mentioned cloud operating system, wherein the first local terminal and the second local terminal are respectively used as the operation terminals of at least one user of the first remote computing device and the second remote computing device. The data redirection method includes the following steps: according to the communication connection established between the first remote computing device and the second remote computing device, determining the event of the direct communication connection initiated by the first local terminal to the second local terminal; obtaining the first communication docking address connected to the first local terminal from the data connection between the first local terminal and the first remote computing device; obtaining the second communication docking address connected to the second local terminal from the data connection between the second local terminal and the second remote computing device; modifying the first communication docking address and the second communication docking address to be consistent, so that at least a part of the data transmission between the first remote computing device and the second remote computing device is redirected to the data transmission between the first local terminal and the second local terminal. Further, the at least part of the data includes real-time call data of the user.

[0035] Another aspect of the present invention relates to a data buffering method for the above-mentioned cloud operating system. The data buffering method comprises the following steps: establishing one or more data buffers between a plurality of nodes transmitted through a network in a local terminal and a remote computing device; completing and generating continuous data in the data buffer according to the historical received data of the node, and the node for reading data from the data buffer continuously performs data reading. Furthermore, the data buffer comprises any one or more of an interface image data buffer, a multimedia data buffer, and a positioning data buffer.

[0036] Another aspect of the present invention relates to a proxy service method for the above-mentioned cloud operating system, wherein the remote computing device includes a cloud server and at least one virtual machine generated in the cloud server. The proxy service method includes the following steps: according to the user account to which the local user terminal belongs, obtaining the cloud operating system data and application storage data corresponding to the user account from the user database in the cloud server, allocating at least one virtual machine with the cloud operating system data and application storage data corresponding to the user account, and providing system scheduling between multiple virtual machines; providing each virtual machine with a proxy unit for the local terminal to communicate with the virtual machine, and exchanging data transmission packets between the proxy units of the multiple virtual machines. Furthermore, the system scheduling between the multiple virtual machines includes: system migration, computer resource scheduling, cross-virtual machine application scheduling execution, or user data synchronization storage.

[0037] Another aspect of the present invention relates to a method for generating a user application based on the above-mentioned cloud operating system, the method comprising the following steps: providing an application store client in a remote computing device through the application service module, the application store client connecting to the application store server to obtain the icon and installation link of the user application; receiving a new request for a new user application triggered by an operation of the local terminal user, and obtaining the icon and installation link through the application store server; obtaining the storage data of the new user application associated with the user account from a user database; and loading the new user application with the installation link in the remote computing device based on the application service module, and importing the storage data.

[0038] Furthermore, the method for generating the user application includes the following steps: providing the operating environment of the new user application, connecting to local hardware resources and interface images, and accessing the user database through the application service module in the browser frame of the remote computing device; transmitting the interface image of the new user application to the local browser frame of the local terminal to run the cloud control desktop, and allocating local resources to the new user application according to the permission request table configured by the user.

[0039] Furthermore, the method for generating a user application includes the following steps: obtaining listing request information of the user application, the listing request information including an icon file, a WEB address, a permission request table and a source code; determining that the listing request information complies with application review rules, and then packaging and listing the icon file, WEB address and source code to the application store server, and allocating an installation link.

[0040] Furthermore, with respect to the method for generating the user application, wherein the listing request information includes the type of the required browser, the method includes the following steps: when the type of the required browser does not match the kernel type of the browser frame provided by the local terminal or the remote computing device, causing the browser frame provided by the remote computing device to switch the kernel to match the type of the required browser, or overloading the function in the original code to make the installed user application match the kernel of the browser frame; when running the user application, causing the page provided by the browser frame to switch to the page theme and operation mode of the type of browser required by the user application.

[0041] Another aspect of the present invention relates to an input method based on a cloud operating system, wherein the cloud operating system is set in a remote computing device, the remote computing device is associated with a local terminal through a network to transmit an interface image to the local terminal, and the local terminal is used as a user operation terminal. The input method includes the following steps: collecting user input operations of the user at the local terminal, the user input operations include user voice or operation points on the local interface; in the remote computing device, converting the user input operations into voice recognition words or operation instructions of the running interface; triggering an input panel based on the voice recognition words or the operation instructions to generate first language data; retrieving second language data associated with the first language data from a remote word library to present the first language data and / or second language data on the running interface; wherein the remote word library receives local input data from the local word library of the local terminal, and the remote word library is synchronized with the user input data stored in the user database.

[0042] Further, with respect to the input method, the remote vocabulary receives a personalized cloud vocabulary associated with a user account from a third-party WEB server.

[0043] Furthermore, the input method includes the following steps: calling an AI function through an application programming interface of an AI server to convert first language data into third language data to present the third language data on the remote computing device, wherein the AI ​​server deploys a large-scale pre-trained language model.

[0044] Another aspect of the present invention relates to a method for opening a user application, which is used in the above-mentioned cloud operating system. The method for opening the user application includes the following steps: capturing a user input operation of the user in the local terminal, wherein the user input operation includes a user voice command or an operation point on the local interface; in the remote computing device, determining the operation object of the user input operation in the current running interface of the first user application; determining that the operation object is associated with a second user application, and after capturing a message allowing the second user application to start, generating the running interface of the second user application, allowing at least a part of the interface of the first user application to be superimposed, and allowing the first user application to keep running.

[0045] Furthermore, the method for opening the user application includes the following steps: pre-configuring an association list of the operation objects and the user applications to determine that each operation object is opened by the corresponding user application, the operation objects including file types, hardware resources or website addresses; or, after determining that the permissions required by the user application to be opened are consistent with the permission type assigned by the local terminal, generating a message allowing the user application to be started.

[0046] Furthermore, the method for opening the user application includes the following steps: when it is determined in the remote computing device that the operation object is associated with a third user application or a system bar, calling a user application or process of the same type in the local terminal, the type of the third user application includes a camera application, a phone application, a text message application, an address book application or other basic applications based on local functions, and the system bar includes a remote computing device status bar or a hardware control center bar.

[0047] Another aspect of the present invention relates to a communication method based on the above-mentioned cloud operating system. The communication method comprises the following steps: establishing a network association between a remote computing device and multiple local terminals and using the multiple local terminals as the operating terminals of at least one user; providing a communication service station on the remote computing device and enabling it to access a distributed database, establishing access between the multiple local terminals and the communication service station, so as to present communication pages and / or cache communication data respectively on each local terminal; determining that at least two local terminals have a communication association; based on the communication service station receiving the communication data of the local terminal or based on the user data in the distributed database, updating the communication page and / or communication data corresponding to each local terminal, and simultaneously transmitting the updated communication page and / or communication data to the communication-associated local terminal.

[0048] Furthermore, the communication method includes the following steps: determining that at least two local terminals have an instant messaging association; assigning network addresses within the same local area network to the at least two local terminals, and establishing a point-to-point communication connection between the local terminals; generating an instant messaging page in the cloud control desktop of the local terminal, so that the local terminals can directly transmit communication data and update the instant messaging page without passing through the communication service station. Among them, the at least two local terminals with a communication association include a first communication terminal and a second communication terminal with an instant messaging association, and the remote computing device and the distributed database share the same physical device resources. The communication method may also include the following steps: enabling the communication service station to receive communication data from one or more local terminals and store it in the distributed database; determining through the communication service station that a user operation instruction from the first communication terminal is received, the user operation instruction instructs to transmit specified communication data to the second communication terminal; obtaining the specified communication data from the distributed database and sending it to the remote computing device associated with the second communication terminal.

[0049] Another aspect of the present invention relates to a method for generating a lightweight application in a user application, based on the above-mentioned cloud operating system. The lightweight application generation method comprises the following steps: providing a lightweight application store client in a remote computing device, the lightweight application store client connecting to an application store server to obtain icons and navigation links of lightweight applications; receiving a request for a new lightweight application triggered by an operation of a local terminal user, and obtaining icons and navigation links through the lightweight application store server; obtaining storage data of the new lightweight user application associated with a user account from a user database; based on a page generation mechanism of a browser framework, loading a page of the new lightweight user application with the navigation link in the user application, and importing the storage data.

[0050] Another aspect of the present invention relates to a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement any of the above methods when executed by a processor.

[0051] Another aspect of the present invention relates to a mobile computing terminal, comprising: a network communication module connected to a cloud server; an image processor for processing interface image data sent by a remote computing device; a screen for displaying the interface image data; and the computer-readable storage medium.

[0052] Another aspect of the present invention relates to a cloud computing server, comprising: at least one remote computing device, the remote computing device including one or more virtual machines; a general agent service center connected to the one or more virtual machines, for performing system scheduling between multiple virtual machines and cloud operating system data allocation; and the computer-readable storage medium.

[0053] Therefore, the present invention provides a complete set of cross-platform cloud operating system solutions, which can be compatible with various smart terminals, provide a unified user application ecosystem, and provide rich user experience and simple program development. The present invention also provides various methods based on the cloud operating system to achieve stable and efficient interface interaction, data transmission, instant messaging and other rich smart terminal application services between local and remote. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram showing the architecture of a cloud operating system according to the present invention in some embodiments.

[0055] Figure 2 The diagram shows the association between the interface generation unit of the cloud operating system and various local terminals in the embodiment.

[0056] Figure 3 The interaction between the interface generation unit of the cloud operating system and the local terminal in the embodiment is shown.

[0057] Figure 4 The components of the cloud control desktop of the local terminal and the image interaction association of the cloud operating system in the embodiment are shown.

[0058] Figure 5 The cloud operating system in the embodiment shows the scheme of generating interfaces and laying out interfaces corresponding to various types of terminals.

[0059] Figure 6 The cloud-based operating system deployed on the cloud server in the illustrated embodiment generates various interfaces corresponding to various local terminals.

[0060] Figure 7 The generic driver redirection of the cloud operating system in an embodiment is shown.

[0061] Figure 8 An example of drive redirection of a cloud operating system in an embodiment is shown.

[0062] Fig. 9 The transmission method between the local terminal and the cloud server in the embodiment is shown.

[0063] Fig.10 The details of the transmission method between the local terminal and the cloud server in the embodiment are shown.

[0064] Fig.11 The details of the method for transmitting positioning data between the local terminal and the cloud server in the embodiment are shown.

[0065] Fig.12 A method for transmitting user application data between a local terminal and a cloud server in an embodiment is shown.

[0066] Fig.13 A method for redirecting transmission of multimedia data between a local terminal and a cloud server in an embodiment is shown.

[0067] Fig.14 The following illustrates a process of putting a user application on the shelves in an embodiment.

[0068] Fig.15 An example of listing a user application in the embodiment is shown.

[0069] Fig.16 The flowchart of the method for putting up and installing a user application in an embodiment is shown.

[0070] Fig.17 A method for integrating functions of user applications in an embodiment is described.

[0071] Fig.18 The remote input method between the local terminal and the cloud server in the embodiment is shown.

[0072] Fig.19 The interface switching of the user application based on the cloud operating system in the embodiment is shown.

[0073] Fig. 20 The opening and switching between multiple user applications based on the cloud operating system in an embodiment are shown.

[0074] Fig.21 The embodiment shown is a remote user application based on a cloud operating system triggering a call to a local application.

[0075] Fig. 22 The control center interaction and data synchronization between the local terminal and the cloud server in the embodiment are shown.

[0076] Fig.23 Some other embodiments of the method for installing user applications include the installation of communication applications.

[0077] Fig.24 Shows the service data interaction of the communication application between the local terminal and the cloud server in some other embodiments.

[0078] Fig.25 A communication method between multiple terminals based on a cloud operating system in an embodiment is shown.

[0079] Fig.26 A data interaction method of an application providing a communication service in an embodiment is shown.

[0080] Fig. 27 The following describes a method for implementing a lightweight application in a user application in an embodiment.

[0081] Fig.28 An instant messaging application scenario between multiple terminals based on a cloud operating system in an embodiment is shown. DETAILED DESCRIPTION

[0082] The following combination Figures 1 to 28 , the terminals, devices, systems, architectures, methods, processes and various software and hardware parts involved in the entire technical solution of the present invention are described in detail in multiple exemplary embodiments.

[0083] 1. Deployment and architecture of cloud operating system

[0084] Reference Figure 1 The cloud operating system according to the present invention is not deployed on a local terminal but is set on a remote computing device. The remote computing device is associated with the local terminal through a network and uses the local terminal as a user operation terminal.

[0085] 1.1 Local Terminal and Remote Computing Device

[0086] In some embodiments, the local terminal has a display screen, network communication hardware and a local operating system. Specifically, the local terminal may include a smartphone, tablet computer, portable computer, game console, car central control device, smart home appliance or other networkable computing terminal with a display screen across operating systems (e.g., Android, IOS, Windows, Linux) platforms. In some embodiments, the remote computing device may include any one or more of a cloud computing server (referred to as a cloud server), a virtual machine, a container and a personal computer. Preferably, the cloud operating system according to the present invention is deployed on a cloud server, and the cloud server generates at least one virtual machine for interacting with the user's local terminal. In addition, in some embodiments, at least a portion of the cloud operating system may be deployed in a virtual machine or a container. The following mainly describes the technical solution of the present invention in the form of a remote computing device implemented as a cloud server and a virtual machine. It should be understood that within the technical solution system of the present invention, the virtual machine can be replaced by a container or a physical computing device at any time to implement the solution of the present invention.

[0087] Cloud servers can include infrastructure, computing servers, schedulers, automation components, network managers, etc. based on the cloud cluster architecture. The cloud cluster architecture can support multiple servers with distributed redundancy, so cloud servers can also be integrated by integrating various hardware resources in the cloud to provide a distributed physical operating environment for multiple cloud operating systems.

[0088] Specifically, the infrastructure is the hardware and software infrastructure of the cloud server, which includes computer hardware (such as CPU, memory, hard disk, graphics card, switch, etc.), cloud computing underlying operating system, database or other software components. The computing server integrates resources through virtualization and cluster technology, and also generates relevant host resources on demand through the cloud control platform. Specifically, through virtualization technology, computing resources and storage resources are decomposed into small, independent virtual machines so that they can be independently operated and managed. The computing server is generally installed with cloud computer software, and preferably a general agent service center is deployed. The automation component is used to implement automated repetitive tasks, such as load balancing, backup and recovery. The network manager provides cloud computing network services, network channels connecting cloud servers and local terminals, and can also be mobilized by the general agent service center to manage all computing servers, shared storage, and networks of the entire cloud cluster architecture. Preferably, the cloud server provides a stable, high-security user database with a backup synchronization mechanism, which divides data storage space for each user account and can be mobilized by the general agent service center. Therefore, no matter how the virtual machine of the cloud server is uninstalled or regenerated, all user data can be directly retrieved from the user database, making the virtual machine as lightweight as possible to improve the working efficiency of the cloud operating system.

[0089] The cloud server and the local terminal are networked through a communication device, and the networking method adopted can be a wired connection, a wireless connection, or any combination thereof. Specifically, a mobile communication (preferably a mobile communication using 4G, 5G or next generation technology), a wide area network or an Ethernet or other communication network can be used between the local terminal and the cloud server. In one embodiment, the technical solution of the present invention may relate to a method for network switching for a cloud operating system, including: triggering a local terminal to switch a communication network or a communication address of a cloud server according to signal strength and / or communication delay. In one embodiment, the local terminal can be triggered to monitor the signal strength of the currently connected first communication network (such as a 5G network) by running a scenario, a user instruction or a timing method, and the communication delay of the first network address of the cloud server can also be obtained by running a Ping command in the background of the local terminal; when the signal strength is lower than a signal strength threshold (the threshold value is between -100dbm and -75dbm) or the communication delay is greater than a delay threshold (the threshold value is between 100ms and 200ms), it can be switched to a second communication network (such as a 4G network) with a lower network frequency, or switched to the second network address of the cloud server.

[0090] 1.2 Cloud Control Desktop

[0091] In some embodiments, the local terminal may include and run various types of local operating systems (OS) (e.g., Android, iOS, Windows, Linux, etc.). The local operating system can switch between basic mode and cloud computing mode. In the basic mode of the operating system, only basic communication functions (e.g., phone calls, sending and receiving text messages) and local basic applications (e.g., browsers, cameras, audio and video players) can be run; in cloud mode, through a high-speed network, the local terminal can interact with the cloud operating system deployed in the cloud server based on the cloud operating desktop to experience rich software and hardware resources and user applications (User APP, referred to as UAPP in this article).

[0092] In some embodiments, the cloud operation desktop includes: an interface control or plug-in, which provides a user operation interface and function extensions associated with the interface based on a browser unit; a local access unit, which is connected to the interface control or plug-in and is used to capture the user's real-time manipulation data and information and hardware information of local applications accessing the local terminal; one or more service units, which are connected to the interface control or plug-in and are used for various interactive functions and method flows between local and remote; one or more client units, which are respectively connected to the service unit and the local access unit, and are used to communicate and transmit with the proxy service program in the remote computing device. The service unit and the client unit can be implemented in the form of subroutines, program modules, etc.

[0093] The implementation forms of cloud-controlled desktops include independent APPs (such as native APPs, system-level built-in APPs, or independent APPs that need to be installed), Web APPs (such as JavaScript (JS) applications based on the HTML5 framework), or plug-ins in applications based on the local OS (such as browsers), etc.

[0094] Reference Figure 2 and Figure 3 In one embodiment, the cloud control desktop is implemented as a native APP or a hybrid APP, and the cloud control desktop may include a browser unit, a local access unit, one or more service subroutines, and one or more client subroutines. The browser unit may be a (web) browser framework or a browser subroutine integrated in the cloud control desktop APP, in which various browser controls and plug-ins are provided. The service subroutine runs an optimized local-remote interaction algorithm, such as an image interaction subroutine in multiple service subroutines docking with the interface controls in the browser unit to realize interface image interaction, and is used for filtering, slicing, compressing, comparing, integrating, or parallel processing of images transmitted between the local terminal and the cloud server virtual machine. It should be understood that in this embodiment, other service subroutines are also used to implement various local and remote control interactions, interface image transmission and display, data redirection, data dynamic adjustment, and other methods. The local access unit can be implemented as a subroutine for calling a local application program interface (Local API) based on the local OS. The local access unit can also be connected to the local management unit outside the cloud control desktop to obtain local data, wherein the local management unit can be implemented as a background service program of the local terminal, which keeps transmitting local data to the device monitoring program running in the background of the cloud server. The client subroutine is connected to the local access unit to read the transmission data of the hardware of the local terminal. The communication method implemented in the client subroutine allows the cloud control desktop to establish network communication with the service program of the cloud server, and then reads the communication data received from the cloud server by the network hardware through the local access control.

[0095] Reference Figure 4In one embodiment, the cloud control desktop is implemented as a Web APP, plug-in program, etc. running in a browser (framework) based on a local terminal. The cloud control desktop in the browser framework also includes interface controls for image interaction with the remote interface of the virtual machine. In addition, the cloud control desktop may include an image interaction service subroutine for slicing, compressing, comparing, integrating or parallel processing of interface images transmitted between the local terminal and the cloud server virtual machine. In addition, in another embodiment, other service subroutines can be inserted into the browser application through other plug-ins or components to connect with the cloud control desktop in the browser application, wherein the script in the service subroutine is interpreted by the JS engine (e.g., V8, WebKit, etc.) in the local browser framework to call the Local API or the Web API from the WEB server, thereby implementing various control interactions between local and remote, interface image transmission and display, data redirection, data dynamic adjustment and other methods.

[0096] It will be appreciated that the APIs mentioned above and below may define parameters passed between a calling application and other software code (e.g., an operating system, a stock program, a function) that provides a service, provides data, or performs an operation or calculation. The API may be implemented as one or more calls in program code that send or receive one or more parameters through a parameter list or other structure based on a calling convention defined in an API specification document. A parameter may be a constant, a key, a data structure, a target, a target class, a variable, a data type, a pointer, an array, a list, or another call. API calls and parameters may be implemented in any programming language. A programming language may define the vocabulary and calling conventions that a programmer will use to access the functionality that supports the API.

[0097] 1.3 Basic Architecture of Cloud Operating System

[0098] Reference Figure 1 In some embodiments, the cloud operating system according to the present invention includes a basic kernel, a runtime library, an interface service module and an application service module.

[0099] The basic kernel includes process management, drivers, memory management, virtual file systems, etc., to call the physical and / or virtualized hardware resources of the cloud server according to the request. The basic kernel of the cloud operating system is composed of the kernel of the underlying operating system of the cloud server, preferably the Linux kernel.

[0100] The runtime library is based on the basic kernel to provide a runtime environment for applications on various operating system platforms. In one embodiment, the runtime library is a computer program library that can be used by compilers of programming languages ​​of various operating system platforms to implement built-in functions of the programming language to provide runtime (or execution) support for programs in the programming language. Preferably, in addition to the Linux runtime library provided by default in the cloud operating system according to the present invention, runtime libraries of Android, Windows, or macOS can also be installed and provided, so that applications of Android, Windows, or macOS can be run in the cloud operating system, especially browser frameworks of various kernels of various platforms (e.g., Safari, Edge, Firefox, Chrome, open source Chromium, etc.).

[0101] The interface service module corresponds to the runtime library to provide an application program interface (API). The interface service module may also include an interpreter, engine (e.g., V8, WebKit, etc.) or runtime module (e.g., Node.js or a customized simplified JS runtime) for interpreting and executing the UAPP code (e.g., JS script code). In addition, the interface service module may also provide a WebAPI for obtaining network services, resources, or functions from a Web server, such as Fig.12 shown.

[0102] The application service module provides control interface services and UAPP services to the local terminal, and provides local and remote docking of these services. The application service module includes: communication unit, device monitoring unit, interface generation unit, cloud control docking unit and driver docking unit, where these units can be implemented in the form of background service programs, subroutines, program modules, etc.

[0103] The communication unit is used for communication between the local terminal and the virtual machine of the cloud server, and between multiple devices (or virtual machines) associated with the local terminal in the cloud server. The data transmission method will be described in detail in Section 4 below with multiple embodiments.

[0104] The device monitoring unit obtains the hardware parameters and real-time operation status data of the local terminal. Figure 3In one embodiment, the device monitoring unit can be implemented as a device monitoring program of the cloud server background service, and after the authentication connection between the local terminal and the remote computing device, the hardware configuration parameters (network type, memory size, storage capacity, processor type, multimedia device type, etc.) of the local terminal, the local OS type (e.g., iOS, Android, Windows, Linux, lightweight IoT operating system, etc.) and version, screen size, screen resolution, display refresh rate, input device type (e.g., touch, mouse, keyboard, recording device, operator, etc.) are obtained. The device monitoring unit also receives the current status and data of local applications (including local OS built-in programs and local applications), such as network communication quality, local performance, local application action data, geolocation data, user instruction data, etc.

[0105] Reference Figure 3 , Figure 5 and Figure 6 , the interface generation unit generates the running interface of the UAPP according to the hardware parameters of the local terminal. In some embodiments, the interface generation unit generates an interface environment for presenting the UAPP based on the interface style and preset operating habits corresponding to the user account and based on the hardware configuration parameters obtained by the device monitoring unit, wherein the running interface of the UAPP is intercepted by the virtual machine background service program of the cloud server and then transmitted to the cloud control desktop of the local terminal. The interface generation unit can generate the running interface of the UAPP based on the interface manager in the runtime library. Preferably, the interface generation unit runs in a browser framework, and generates the running interface of the UAPP through the renderer and interpreter (or engine) of the browser. The interface generation method will be described in detail in Part 3 below with multiple embodiments.

[0106] The cloud control docking unit is called by the background service program to receive the user's control data (such as the operation data of the touch screen, mouse, keyboard and other devices) from the local terminal in real time, and convert it into the control data of the virtual machine in the cloud server.

[0107] The driver docking unit responds to the hardware resource request of the UAPP to redirect the source data for hardware execution of the remote computing device to the hardware of the local terminal. Figure 7In some embodiments, the present invention relates to a driver redirection method, comprising the following steps: responding to a hardware resource request initiated by UAPP through an API, calling a first data conversion program to generate source data to be cached in a memory area, driving a docking unit to capture the source data from the memory area of ​​a remote computing device, the source data being the data to be received from the memory area by the driver triggering the remote computing device hardware (e.g., a bitmap that a graphics card driver needs to read, a standard digital audio file (wav) that a sound card driver needs to read), and then forwarding the source data to a local terminal through a remote-side data forwarding unit (compressing and); responding to an instruction of the driver docking unit, a service unit in the background of the local terminal initiates hardware resource docking through an API, calling a second data conversion program, receiving (and decompressing) the source data through a local-side data forwarding unit, replacing the output data of the second data conversion program, and triggering the local hardware to execute the source data through a local driver. Preferably, the first and second data conversion programs are programs of the same type and function. Similarly, the method for redirecting a driver from a local terminal to a remote computing device may include the following steps: responding to a hardware resource request initiated by UAPP through an API, causing the driver to trigger the remote computing device hardware to collect blank source data, calling a third data conversion program to generate a blank file from the source data to be cached in a memory area; responding to an instruction from a driver docking unit, causing the driver of the local terminal to synchronously trigger the local hardware to collect source data, calling a fourth data conversion program to generate a target file from the source data to be forwarded to the remote computing device by a data forwarding unit, and replacing the blank file with the target file; causing UAPP to obtain the target file. Preferably, the third and fourth data conversion programs are programs of the same type and function.

[0108] Reference Figure 8 In a specific embodiment, an audio application (Voice APP) runs in the virtual machine of the cloud server, and the local terminal displays the interface of the Voice APP to allow the user to remotely operate the Voice APP. When the user of the local terminal interactively triggers the Voice APP to play the audio file (such as an mp3 file) in the storage unit, the Voice APP calls the API to trigger the audio decoder to decode the obtained wav audio stream and temporarily stores it in the memory area. The driver originally used for the virtual machine calls the virtual machine hardware to play, which will actually be forwarded to the local terminal. Because the driver docking unit synchronously sends instructions to call the service unit in the background of the local terminal to trigger the local audio decoder to work through the API, the local audio decoder does not actually decode the wav audio stream, but redirects the wav audio stream data from the remote side to the output result of the local audio decoder through the data forwarding unit on the local side, so that the driver of the local OS can trigger the hardware for playback. Continue to refer to Figure 8In another specific embodiment, when the user triggers the Voice APP to perform local recording in an interactive manner,

[0109] The Voice APP initiates a hardware resource request through the API, causing the driver to trigger the hardware of the virtual machine to record (in fact, the recording obtains a blank audio stream), and calls the encoder in the virtual machine to generate a blank file from the source data to be cached in the memory area; in response to the instruction of the driver docking unit, the driver of the local terminal synchronously triggers the local Mic hardware recording through the local API to obtain a wav audio stream, and calls the encoder to generate a compressed audio file (such as an mp3 file) to be forwarded to the remote computing device by the data forwarding unit, and replace the blank file; then the Voice APP obtains the target audio file to complete the storage of the recording file.

[0110] In some embodiments, driver docking can also be achieved through interaction between local and remote browser frames, which will be described in detail in part 2 below with examples.

[0111] It can be understood that based on the cross-platform system environment built on the network between the local terminal and the remote computing device, the cloud operating system of the present invention simplifies and adjusts the application framework layer in the traditional operating system architecture into an interface service module, and customizes the application service module, thereby realizing the control, hardware resources and interface interaction between the local terminal and the remote computing device, improving the program operation efficiency and ensuring the user's remote operation experience.

[0112] In these embodiments, the cloud operating system of the present invention is characterized in that at least a portion of the cloud operating system is deployed in any one or more of the cloud server, virtual machine and container. For example, the basic kernel of the cloud operating system is provided by the operating system kernel of the cloud server and virtual machine, the interface service module and the runtime library can be installed and provided in the virtual machine, and the application service template can be deployed in the virtual machine and / or container. Containers (or cloud service containers) and virtual machines are both based on virtualization technology, which is convenient for generation and migration. Virtual machines create a complete native operating system and hardware environment by virtualizing hardware, while containers create an isolated operating environment on a computing device (which can also be a virtual machine). It does not require simulated hardware, is more lightweight, has a faster startup speed, and occupies less resources, so it is more suitable for deploying lightweight applications. Therefore, the cloud operating system of the present invention can rely on huge server computing power to provide each user with a cross-platform multiple device environment to run UAPP, and make the development process of UAPP more flexible, efficient and unified.

[0113] In these embodiments, the cloud operating system of the present invention is also characterized in that the interface image is transmitted from the remote computing device to the local terminal for the user to control the UAPP on the local terminal. Compared with the transmission of the interface screen recording video, the solution of transmitting the interface image saves network bandwidth, and does not require the local terminal to consume computing power for video decoding, and the compression algorithm of high-definition images is mature, so that when the user uses the cloud operating system, the user can obtain a clear interface visual sense and maintain a smooth control feedback experience.

[0114] 2. Browser Framework

[0115] Reference Fig. 9 , based on the cloud operating system, a remote browser framework (or web browser framework) can be provided for each virtual machine generated in the cloud server. Accordingly, the local OS in the local terminal can come with a local browser framework to interact with the remote browser framework. Whether it is a remote or local browser framework, its infrastructure can include a browser interface, a browser kernel, and a back-end service layer. The browser interface is used to display pages or windows. The browser kernel may include a browser engine and a rendering engine for presenting pages, wherein the browser engine may be V8, WebKit, etc., and the rendering engine may include an HTML interpreter, a CSS interpreter, and a layout. Generally, the kernel and rendering engine of the Safari browser are WebKit; while the kernel of the Chrome / Chromium / Edge browser is Chromium, and the rendering engine is Blink (a branch of WebKit). The back-end service layer is used to connect to the network, data storage, or call native services through an API.

[0116] Preferably, refer to Fig.10 The backend service layer may be provided with a backend runtime module (e.g., JS runtime) that supports JS running outside the browser. For example, the V8 engine may be used to run JS scripts outside the browser to implement functions such as file systems, modules, system APIs, network communications, and conventional applications. The backend runtime module may integrate existing JS runtime environment tools (e.g., Node.js) or integrate a customized lightweight JS runtime that retains specific functions. In one embodiment, the backend runtime module may include a JS engine, a local module, and a standard library, and may directly call a Native API.

[0117] 2.1 Local Browser

[0118] Generally, the browser built into the local terminal of most intelligent platforms can be configured to cooperate with the cloud operating system. If the local browser does not have a suitable browser engine, an additional toolkit can be installed to implement a local operating environment based on a specific browser engine, thereby configuring a local browser that cooperates with the cloud operating system of the present invention.

[0119] In some embodiments, the local browser that cooperates with the cloud operating system of the present invention may include: a client unit, which is used for the local browser to communicate with other applications running on the local operating system, and to communicate with the remote computing device to obtain data from the local terminal and the remote computing device; a display component, which is used to display the interface image from the remote computing device; and an image interaction service unit connected to the display component, which is used to implement the interface image transmission and interaction method. Specifically, the display component may include a WebView control that is compatible with local systems and browsers such as Linux, Android, Pinggu and Google. The local browser framework provides a backend runtime module that supports the operation of the client unit and the image interaction service unit. In addition, it can also run various subroutines, program modules, etc. written in JS scripts of the cloud control desktop.

[0120] 2.2 Remote Browser

[0121] In some embodiments, since the cloud operating system of the present invention can provide multiple runtime libraries (such as Android, Mac), various browsers (such as Chrome / Chromium, Safari) can be built into the virtual machine generated in the cloud server to match the kernel type of the local browser (such as Chromium, WebKit) to obtain better compatibility and UAPP operation efficiency. In addition, since the Chromium kernel and the WebKit kernel are homologous, a Linux version of the browser based on the Chromium kernel can be built into the virtual machine generated by the cloud server by default, so that the browser (framework) configured in the virtual machine can match the Linux base kernel and default runtime library of the cloud operating system.

[0122] In some embodiments, the remote browser may include: a client unit, which is used for the remote browser to communicate with other applications running on the cloud operating system and communicate with the local terminal to obtain data from the local terminal and the remote computing device; an interface image acquisition unit, which is used to obtain at least a portion of the interface image from the user interface of the web-based UAPP running on the remote browser; wherein the client unit and the interface image acquisition unit are respectively connected to the image interaction service program in the cloud operating system, and the image interaction service program implements the interface image transmission and interaction method. The remote browser framework provides a backend runtime module to support the operation of the client unit, the interface image acquisition unit, and the image interaction program.

[0123] 2.3 Application service connection based on browser framework

[0124] In a preferred embodiment, when the kernels of the remote and local browser frameworks are consistent or matched, and the same backend runtime module is configured and used, the application service docking and driver redirection of the cloud operating system become flexible and simple. For example, the script that calls the API in the UAPP running in the remote browser framework can also reuse the code in the local browser to achieve synchronous calling of the local API of the same type or function, thereby providing great convenience for the remotely running UAPP to call the resources of the local terminal.

[0125] Reference Figure 7 and Figure 8 , taking the working process of UAPP calling the hardware resources of the local terminal as an example, the method of implementing API and driver redirection through remote and local browser frameworks is described. First, the UAPP running in the remote browser framework of the virtual machine generates a page by the rendering engine to be displayed on the browser interface. The page is captured by the interface image acquisition unit as the UAPP running interface and sent to the local browser and displayed as the local interface; then the user transmits the control data based on the local interface to the virtual machine to operate the UAPP in a remote interactive manner; because the remote browser is configured with a back-end runtime module, when responding to the instruction script of the UAPP, the hardware resource request of the virtual machine outside the browser is initiated through the API of the virtual machine (for example, audio and video decoding and playback, opening the camera, etc.), the driver docking unit triggers the back-end runtime module of the local browser with the same instruction script, and initiates the local hardware resource request with the same API; then the data originally returned by the virtual machine hardware resources is replaced by the data generated by the local hardware resources, and then returned to the UAPP, thereby achieving the effect of UAPP calling the hardware resources of the local terminal.

[0126] 3. Interface Methods

[0127] 3.1 Interface generation and layout

[0128] Reference Figure 2 In some instances, the remote interface generation method may include: establishing a network association between the cloud server and each local terminal; obtaining a user account and hardware configuration parameters from each local terminal, the hardware configuration parameters including the local operating system type, screen size, screen resolution, and input device type, etc.; generating an interface window in the browser in the cloud server with the obtained screen size or screen resolution of each local terminal, thereby serving as the interface environment of the UAPP; capturing the interface image of the interface window and transmitting it to the local terminal as an interface image display. In this embodiment, each virtual machine of the cloud server can generate an interface corresponding to the local terminal of each user account. In addition, referring to Figure 2 and Figure 6The corresponding operating system or runtime virtual machine can be allocated according to the type of the local terminal. For example, the local terminal corresponding to the Android system is allocated with the Android or Linux operating system or runtime virtual machine, so that the interface generation unit generates the interface style of the local terminal by default.

[0129] Reference Figure 3 In some instances, the local hardware configuration parameters can be obtained through a local management unit connected to the cloud control desktop, where the local management unit can be implemented as a background service program of the local terminal, which keeps transmitting the local hardware configuration parameters to the device monitoring program running in the background of the cloud server. In addition, the renderer of the browser frame of the remote computing device is provided with an HTML interpreter and a CSS interpreter, and the JS engine is used to interpret the web page script files (such as the web page script files) of the UAPP cached locally or received from the WEB server. Fig.12 and Fig.13 As shown), generate the running interface of UAPP. Specifically, the aspect ratio of the interface window presented by the display control is consistent with the screen aspect ratio of the local terminal, and the DPI value of the interface window is greater than or equal to the screen DPI of the local terminal, so as to allow the cloud server's higher computing power to generate high-definition images or decode and play high-definition videos for full display in the interface window. Then, the running interface can be converted into an interface image (further compressed according to network transmission requirements) by a screenshot plug-in or subroutine running in the browser framework of the remote computing device. The local terminal can provide a browser framework to allow the browser display control (such as WebView) to receive the interface image for display on the screen of the local terminal. Preferably, a video playback control can also be provided. When it is detected that the interface of the virtual machine of the cloud server has a video playback window, the interface image to be transmitted by the cloud server is adjusted to retain only the image of the video playback window, and the video stream can be redirected to the video control window of the local interface for playback later. Therefore, after the remote interface is generated, the local terminal can receive the size and position data of the video playback window, so that the local video control is arranged and covered on the received interface image according to the size and position.

[0130] Reference Figure 5 and Figure 6, the remote interface generation method may include: obtaining the interface style and operating habits corresponding to the user account or the local operating system type from the user database of the cloud server, generating a browser window in the browser frame of the cloud server virtual machine with the obtained screen size or screen resolution of the local terminal, configuring the renderer to generate an interface layout based on the interface style, and determining the operating gestures of the browser window based on the operating habits, thereby serving as the interface environment of the UAPP running based on the browser frame. In a specific embodiment, a user may have multiple local terminals, such as smart phones, smart cars, wearable devices, and portable PCs, and multiple virtual machines may generate browser windows with hardware configuration parameters corresponding to each local terminal of the user, wherein the same set of CSS standard layout interfaces are used at the same time, so that only the script code of the same UAPP can present various local terminal style interfaces in a separate virtual machine running environment. For example, for the same map APP, a small-size vertical screen interface that meets the needs of a mobile phone can be generated in one virtual machine, and a large-size interface that meets the needs of in-vehicle navigation can be generated in another virtual machine. In addition, referring to Figure 6 , you can also use the same virtual machine (such as Figure 6 Multiple windows of the browser frame of VM2 in the example generate local terminal style interfaces (such as Figure 6 GUI2 and GUI3 in the virtual machine) to allow the same user account to quickly access the required local terminal style interface after logging into the virtual machine. This brings an excellent user experience for scenarios where different terminals need to be switched while the same APP is used. For example, the user uses a wearable device to receive the interface GUI3 in advance, and operates a map APP (actually running on the virtual machine VM2), and sets the navigation destination in advance by voice. At this time, the map APP switches to the map interface GUI2 of the car navigation and sends it to the smart car logged in by the user, so that when the user enters the smart car, the car's display screen directly displays the interface information of the map APP's navigation to the destination. Therefore, when developing UAPP, the page code based on the CSS standard that implements the UAPP interface generation method is separated from the script code that implements the UAPP function.

[0131] 3.2 Interface Image Transmission

[0132] Reference Figure 4 In some embodiments, the interface image transmission method for a cloud operating system may include the following steps:

[0133] S001. Establishing a control data interactive connection between the local terminal and the virtual machine of the cloud server;

[0134] S002. Maintaining the transmission connection of the interface image of the virtual machine to the local terminal for at least one period of time;

[0135] S003. Dynamically adjust the interface image to be transmitted by the cloud server according to the transmission interval of the interface image transmitted by the cloud server to the local terminal.

[0136] For step S001, the control data sent by the interactively connected local terminal will cause changes in at least a portion of the interface of the cloud server. Such control data can be data generated by the local terminal's input device (such as a touch screen, mouse, keyboard, etc.) causing local click, move, zoom and other control actions. After the local control data is transmitted to the cloud server, it is distributed to the virtual machine that has established a connection with the local terminal, thereby establishing an interface interaction between the local terminal control action and the virtual machine. The control of the local terminal can cause changes in the entire interface of the virtual machine, or it can cause changes in part of the interface, or cause changes in the background activities of the interface.

[0137] For step S002, based on the transmission connection between the local terminal and the virtual machine of the cloud server, when the control data generated by the local terminal is sent to the virtual machine each time, if it causes the interface of the virtual machine to change, the cloud server will transmit the interface image of that time to the local terminal. In other words, the higher the frequency of the control data received by the virtual machine, the more images are generally transmitted to the local terminal. Preferably, the transmission of the interface image between the local terminal and the cloud server can be based on a long link network protocol to be compatible with high-frequency image transmission.

[0138] For step S003, the dynamic adjustment of the interface image to be transmitted by the cloud server includes: within a certain time period, if the interval between the cloud server transmitting images to the local terminal is too short (or the image transmission is too frequent), in the next time period, implementing any one or more of the following sub-steps: reducing the number of image transmissions to the local terminal; reducing the quality of the entire or part of the interface image by image compression; transmitting the dynamically changing part of the interface image.

[0139] For step S003, in a specific embodiment, when it is detected that the interface of the virtual machine has a video playback window, the interface image to be transmitted by the cloud server is adjusted to retain only the image of the video playback window. This is because the video stream can be redirected to the video control window of the local interface for playback later. Preferably, if the interface of the virtual machine is a full-screen video playback, the transmission of the interface image to the local terminal is paused, and the video stream is directly transmitted to the local terminal for playback.

[0140] The following describes the method flow and interaction steps involved in the interface image transmission of the local and remote sides of the image transmission method for the cloud operating system through more detailed embodiments.

[0141] 3.2.1 Local side method flow

[0142] The image transmission method for a cloud operating system may include the following steps S110 to S119 on the local terminal side.

[0143] S110, start the cloud control desktop program of the local terminal, and activate the local control process at the same time, so as to capture the control data of the input device in real time and transmit it to the remote computer device (the virtual machine of the cloud server). Preferably, the user information database of the cloud server can be accessed through the cloud control desktop program to verify the user's login information (such as account number, password, fingerprint data, face recognition data, etc.), which is used to log in to the virtual machine of the user in the cloud server and establish a secure encrypted control data transmission from the local to the remote.

[0144] S120. In the background program of the local terminal, run the data transmission process and activate the dedicated channel for image data transmission to allow the reception of the interface image data transmitted by the cloud server. The image data transmission channel and the control data transmission channel are independent of each other to allow the transmission of image data without being disturbed by the network connection when the control data is continuously transmitted. Preferably, the transmission frequency configured for the image data transmission channel is lower than or equal to the transmission frequency of the control data transmission channel.

[0145] S130. Continuously receive the interface image of the virtual machine from the cloud server during a preset period of time through the long link between the local terminal and the cloud server.

[0146] S140, determining whether the frequency of received images is too fast (for example, the time interval Δtr between two received image data is less than a preset threshold Tr), if yes, executing step S150, otherwise jumping to executing step S160.

[0147] S150. According to the state of the local terminal, filter and receive a part of the interface images from the cloud server or filter the received interface images for the interface update and display of the local terminal. Preferably, the number of screened interface images received can also be adjusted based on the power, network status, application scenario and other status of the local terminal. For example, when the local terminal is running in energy-saving mode or the workload of the baseband chip exceeds the preset value, the transmission frequency of the image data transmission channel is reduced, thereby reducing power consumption and heat generation. For example, when the network bandwidth connected to the local terminal is low or in the standby scene, a part of the interface images are selected at intervals for display at longer time intervals. In other instances, when the sending frequency of the local control data is monitored to be lower than the threshold ft, the received interface image is selected at intervals; if it is monitored that there is no local control data sent, but the frequency of the received image is too fast, the received interface image is selected at a longer interval for the interface update of the local terminal.

[0148] S160, updating the display interface of the local terminal with the selected interface image. Preferably, the local terminal display interface can also be locally or globally updated by any of the following methods: timing; switching of the application scenario of the local terminal (e.g., web browsing, office work, game mode, etc.); change of the system status of the local terminal; receiving a user-specific refresh operation instruction.

[0149] S170, determining whether the local terminal generates control data for the remote virtual machine (for example, whether the user's control of the cloud-controlled desktop has ended), if so, executing the next step S180, otherwise returning to executing step S130.

[0150] S180, determining whether the image data transmission between the local terminal and the remote server is completed (for example, whether the transmission channel is closed), if yes, executing the next step S190, otherwise returning to executing step S130.

[0151] S190, synchronizing the current control data to the virtual machine of the cloud server, and synchronizing the global interface image of the virtual machine to the interface of the local terminal. Finally, this process ends.

[0152] 3.2.2 Remote side method flow

[0153] The image transmission method for a cloud operating system may include the following steps S210 to S290 on the remote cloud server side.

[0154] S210, through the transmission process of the cloud server, obtain the user login data of the local terminal, start the remote computer device of the user (the virtual machine generated in the cloud server), so that the virtual machine can be connected to the control data of the local terminal. Preferably, there can be multiple virtual machines allocated to the user in the cloud server, and the control data transmission channel of each currently used virtual machine is associated with the specific control data transmission process of the cloud control desktop of the local terminal, so that the cloud control desktop can switch to control each virtual machine of the user in the cloud server.

[0155] S220: Run an image interaction service program in a cloud server or in a virtual machine, and activate a dedicated channel for image data transmission to allow transmission of interface image data of the virtual machine to a local terminal.

[0156] S230: Calling a screen capture process of the virtual machine to continuously obtain an interface image of the virtual machine.

[0157] S240 , determining whether the frequency of the images sent is too fast (for example, the time interval Δts between two received image data is less than a preset threshold Ts), if yes, executing step S250 , otherwise jumping to executing step S260 .

[0158] S250: Dynamically adjust the quality of the acquired interface image of the virtual machine.

[0159] For steps S240 to S250, in some embodiments, dynamically adjusting the interface image to be transmitted from the cloud server to the local terminal includes the following steps: detecting the current transmission interval Δts of the interface image in the same interface range from the cloud server to the local terminal; if it is determined that the current transmission interval Δts is less than the first time threshold Ts1, the interface image to be transmitted is abandoned at the current transmission; if it is determined that the current transmission interval is lower than the second time threshold Ts2, the quality of the interface image to be transmitted is reduced (for example, the quality of at least a part of the interface image is reduced by further image compression, and the resolution of the interface image can also be reduced); wherein, the first time threshold Ts1 is less than the second time threshold Ts2.

[0160] In some examples, the first time threshold Ts1 can be a smaller value, such as 20ms, and the second time threshold Ts2 can be configured as multiple larger thresholds (e.g., 40ms, 60ms, 80ms) to step-by-step reduce the interface image quality (or, increase the compression rate / reduce the resolution). For example, when the transmission interval is less than 40ms, the image quality is compressed to only 30% of the original file volume, when the transmission interval is less than 60ms, the image quality is compressed to only 60% of the original file volume, and when the transmission interval is less than 80ms, the image quality is compressed to only 80% of the original file volume.

[0161] In other embodiments, dynamically adjusting the interface image to be transmitted by the cloud server may also include the following steps: detecting the current transmission interval of the interface image of the same interface range from the cloud server to the local terminal; determining that the current transmission interval Δts is higher than the third time threshold Ts3, then restoring the quality of the interface image to be transmitted (such as reducing the image compression rate, or converting it into a lossless compressed image), or collecting an interface image with higher image quality from the cloud server for transmission to the terminal. The third time threshold Ts3 is generally greater than the second time threshold Ts2. Preferably, the third time threshold Ts3 can be 100ms or a greater value.

[0162] In other embodiments, dynamic adjustment of the interface image to be transmitted by the cloud server may also include the following steps: detecting the current transmission interval Δts of the overall interface image from the cloud server to the terminal; determining that the current transmission interval Δts is lower than a preset threshold Ts0, then reducing the resolution of the interface image to be transmitted or adjusting it to a grayscale image.

[0163] Preferably, the interface image transmitted by the cloud server to the local terminal can be pre-adjusted according to the current network transmission status between the local terminal and the cloud server, application scenarios (such as web browsing, office work, game mode, etc.) or user instructions, and the pre-adjustment includes image resolution adjustment, color channel adjustment, adjustment to grayscale image, etc. For example, when the network transmission speed or bandwidth of the cloud server to the local terminal decreases significantly, the above pre-adjustment step is directly executed, and then step S250 is executed.

[0164] S260, through the long link between the local terminal and the cloud server, the screened interface image is continuously transmitted to the local terminal during a preset period of time. Preferably, a time mark can be established between the control data and the interface image, so that each time the interface image of the cloud server virtual machine triggered (screenshot) by the control from the local terminal at a certain time node can be attached with the time node mark, so that the interface image received by the local terminal is also attached with the time node mark, so that the control action of the local terminal corresponds to the displayed interface image.

[0165] S270, determining whether the remote virtual machine has received the control data of the local terminal, and whether the control process caused by the control data of the local terminal in the virtual machine is completed, if yes, executing the next step S280, otherwise returning to executing step S230.

[0166] S280, determining whether the image data transmission between the local terminal and the remote server is completed (for example, whether the transmission channel is closed), if yes, executing the next step S290, otherwise returning to executing step S230.

[0167] S290, synchronizing the current control data with the local terminal, and synchronizing the global interface image of the virtual machine to the interface of the local terminal. Finally, this process ends.

[0168] 3.2.3 Manipulating data and image screening

[0169] The control data of the local terminal affects the generation and transmission of the interface image of the virtual machine. In some embodiments, the method may also include the following basic steps related to control: A. According to the type of local control, when specific control, too fast control, abnormal control, etc. occur, the control data after screening or adjustment is sent to the cloud server again to avoid the cloud server from generating unnecessary interface images; or, B. According to the sending interval of the control data of the local terminal, the multiple interface images received by the local terminal are dynamically screened to display the screened interface image or at least a part of the image on the terminal. Basic step A directly affects the interface change of the virtual machine of the cloud server by screening the local control data, and can also directly affect the frequency of image transmission from the cloud server to the local, so basic step A belongs to the "active" step. Basic step B is to screen the received images, which belongs to the "passive" step. It can be understood that in the images received locally and the interface images sent remotely, the image receiving module and the image transmission module can work independently to screen the images, and cooperate with the above-mentioned active step A and passive step B. This is the flexibility of the image transmission method scheme of the present invention, which can be compatible with image transmission optimization in various control-based application scenarios.

[0170] In some embodiments, basic step A can be combined with the manipulation process in step S110 of the above embodiment. In these embodiments, it can be implemented as the following steps S111 to S116.

[0171] S111 , activating a local control process to capture control data of an input device of a local terminal in real time.

[0172] S112: Capture and classify manipulation data during the manipulation triggering period. In some embodiments, the manipulation events may be classified into click, double click, single finger sliding, multi-finger sliding, fingerprint recognition action, body physical button action, etc.

[0173] S113: Determine whether a specific type of manipulation occurs, or whether the manipulation is too fast or abnormal. If yes, execute the next step S114; otherwise, jump to step S115. In some embodiments, for each category, the specific type of manipulation may be a long-term continuous click, a long-distance (for example, more than half the length of the screen) slide, etc. Abnormal manipulation may be, for example, frequent back-and-forth touch dragging, pressing multiple buttons at the same time, etc.

[0174] S114, filter or adjust the control data of the local terminal. In one embodiment, for the control events of frequent back-and-forth touch dragging, the generated control data contains more coordinates of the path of the back-and-forth dragging. Therefore, the control data can be filtered to retain the coordinates of the starting touch point and the ending touch point in the touch dragging event to avoid repeated control data causing repeated generation of virtual machine interface images. In another embodiment, for long-distance sliding control events, only the coordinates of the sliding starting point and the sliding ending point can be retained and recorded in the control data. In addition, for abnormal control events, the control data generated by them can be directly shielded to avoid sending them to the remote server.

[0175] S115. Sending control data to the virtual machine of the remote cloud server.

[0176] S116, waiting for a certain period of time to determine whether the next control event occurs, if yes, return to step S112, otherwise end this process.

[0177] In some embodiments, basic step B may be combined with the manipulation process in step S150 of the above embodiment. Dynamically screening the received multiple interface images according to the manipulation interval of the local terminal may include the following steps:

[0178] S151. Determine, within at least one time period, that the control interval Δtc of the local terminal is less than a fourth time threshold Tc4 (preferably, the fourth time threshold Tc4 may be in the range of 10 ms to 20 ms according to the control type);

[0179] S152. From the interface images received from the remote computing device, discard at least a portion of the interface images used for display on the local terminal (such as intermediate redundant images repeatedly generated due to frequent control events of the local terminal), and then reduce the frequency of control data sent by the local terminal to the cloud server virtual machine as needed.

[0180] In some embodiments, dynamically screening the received multiple interface images according to the manipulation interval of the local terminal may also include the following steps: within at least one time period, it is determined that the manipulation interval Δtc of the local terminal is greater than the fifth time threshold Tc5 (corresponding to the frequency of manipulation of the local terminal by the user under normal circumstances), then all the interface images for display on the local terminal are retained from the received multiple interface images; within at least one time period, it is determined that the manipulation interval Δtc of the local terminal is greater than the sixth time threshold Tc6 (corresponding to the situation where the user does not manipulate the local terminal for a long time), then the interface images for display on the local terminal are selected from the received multiple interface images at intervals until the next manipulation of the local terminal occurs, so that the effect of local standby can be achieved and the power consumption of the local terminal can be reduced. Among them, the sixth time threshold Tc6 is greater than the fifth time threshold Tc5, and the fifth time threshold Tc5 is greater than the fourth time threshold Tc4. The value of the fifth time threshold Tc5 can be greater than 200ms, and the value of the sixth time threshold Tc6 can be greater than several seconds.

[0181] In one embodiment, if it is determined that the manipulation interval Δtc of the local terminal is greater than the seventh time threshold Tc7 within at least one time period, the display frame rate of the local terminal is reduced until the next manipulation occurs, wherein the seventh time threshold Tc7 is greater than the fifth time threshold Tc5. This embodiment can be applied to reducing the display power consumption of the local terminal when the local terminal has not been manipulated for a long time (such as in a reading scenario).

[0182] In some embodiments, the method may further include the following steps: determining that the movement amplitude (or distance) of the control point of the local terminal exceeds a preset amplitude threshold; retaining the interface images associated with the starting action and ending action of the control point among the multiple interface images transmitted to the local terminal by the cloud server due to the control data generated by the control point of the local terminal, and reducing the remaining interface images for display on the local terminal.

[0183] In addition, the user's manipulation frequency of the human-computer interaction device can also be monitored through the local subroutine of the local terminal. In a preferred embodiment: when the manipulation frequency is slow (for example, below a certain threshold), the image quality transmitted from the cloud server to the local terminal can be improved and / or the refresh rate of the display (or display interface) of the local terminal can be reduced; when the manipulation frequency is fast (for example, above a certain threshold), the image quality transmitted from the cloud server to the local terminal can be reduced and / or the refresh rate of the display (or display interface) of the local terminal can be increased; when the manipulation frequency is too fast (for example, above a certain threshold), the interface image of the last frame of the virtual machine in the too-fast manipulation process can be transmitted to the local terminal, and then the remaining interface image data in the too-fast manipulation process of the local terminal is not transmitted, or these images are compressed (even lossy compression) at a high compression rate.

[0184] 3.2.4 Interaction between remote virtual machine and local terminal

[0185] Steps S110, S120, S130, S170, S180 and S190 on the local terminal side have two-way data transmission and data exchange with steps S210, S220, S260, S270, S280 and S290 on the cloud server side respectively.

[0186] In particular, there is a timed synchronization interaction between steps S160 and S260 to synchronize the interface image between the local terminal and the cloud server at a timed interval, so as to avoid the interface display deviation of both parties caused by the cloud server transmitting the filtered interface image to the local terminal.

[0187] In some embodiments, the hardware monitoring module running in the cloud server can be used to determine the operation status of the cloud server hardware (including the virtual hardware of the virtual machine) caused by the local terminal (e.g., network transmission quality, processor load, memory occupancy, etc.). For example, when it is monitored that the network transmission speed decreases or the network delay increases beyond a threshold, the compression rate of the image transmitted from the cloud server to the local terminal can be increased, or the image quality can be reduced.

[0188] 3.2.5 Other methods involving interface image transmission

[0189] In other embodiments, the method involving interface image transmission may further include the following steps to be implemented in a local terminal or a cloud server.

[0190] S351. Before the cloud server transmits the interface image to the local terminal, synchronize the interface resolution of the local terminal with the interface display resolution of the cloud server. Preferably, a data buffer for transmitting at least a portion of the interface image is set between the cloud server and the local terminal; based on the historical transmission data, the data of at least a portion of the interface image is continuously generated in the data buffer, and the application program for reading data from the data buffer continuously reads the interface image data.

[0191] S352: Determine whether a network problem occurs (for example, when the network transmission speed is detected to decrease or the network delay increases to exceed a threshold) or switch to high-speed image interaction mode, if yes, execute the next step S353.

[0192] S353: Determine whether an interface resolution adjustment instruction is received. If yes, execute step S354; otherwise, execute step S355.

[0193] S354. Reduce the interface display resolution.

[0194] S355: Reduce the resolution of the transmitted interface image.

[0195] 4. Data Transmission

[0196] In addition to the dedicated transmission channels for transmitting interface images and control data, the data transmission mechanism of the cloud operating system of the present invention also needs to cope with the transmission of user files, local application data, call data, etc., which is described in detail in some embodiments below.

[0197] 4.1 Basic transmission method

[0198] The framework of the application layer communication protocol for direct transmission between the local terminal and the cloud server may include but is not limited to HTTP, Socket, Webtransport, etc. Preferably, the communication protocols such as WebSocket and HTTP3.0 are used between the local terminal and the cloud server. The transmission between the local terminal and the cloud server can reuse the handshake channel of the HTTP protocol. In addition, it can also be expanded to allow multiplexing, such as Figure 4 The image interaction program in the program can be a channel based on the WebSocket protocol for image data transmission. In an example of image transmission based on the WebSocket protocol, the Socket is used as the main thread in the virtual machine to process image source upload requests and manage WebSocket sessions. When the Socket main thread receives an image upload request, the server automatically creates a WebSocket service to send image data. When the WebSocket subservice receives an image browsing request, it starts to send image data to the cloud control desktop of the local terminal.

[0199] In a preferred embodiment, the data transmission method of the cloud operating system of the present invention may include the following steps: transmitting user control data from a local terminal to a remote computing device through a control data direct connection channel, and the control data direct connection channel has a first transmission priority; an image data direct connection channel established based on a long-chain network communication protocol to transmit an interface image from a remote computing device to a local terminal, and the image data direct connection channel has a second transmission priority; a subscription protocol transmission channel has a third transmission priority, wherein the first transmission priority is higher than the second transmission priority or the third transmission priority.

[0200] 4.2 Data transmission method based on subscription

[0201] Reference Figures 9 to 11In some embodiments, a subscription-based communication protocol (e.g., MQTT) is used between the local terminal and the cloud server. The subscription-based communication method makes data exchange between the local terminal and the cloud server, the virtual machine inside the cloud server, and even the browser simpler and more efficient, and also allows the server to actively push data to the client. In addition, the subscription-based communication method can provide real-time and reliable message services for connecting remote computing devices with very little code and limited bandwidth, and with asynchronous communication. In these embodiments, the local terminal and the remote computing device (e.g., cloud server, virtual machine or container) are respectively provided with one or more client units, and the remote computing device is provided with a proxy unit that runs resident in the background. Through the subscription protocol transmission channel, the client unit set in the local terminal or remote computing device can publish one or more data message packets through the proxy unit to allow one or more other client units set in the local terminal or remote computing device that have pre-subscribed to the data message packet to pull the data message packet to complete the consumption of all data message packets. It should be understood that in the above embodiments, the proxy unit can be set and run in the virtual machine of the cloud server, and can also be set and run in the cloud computing underlying operating system of the cloud server as a background service program of the cloud server. Specifically, refer to Fig. 9 ,The general agent service center deployed by the cloud server can also be used to ,uniformly manage multiple agent units.

[0202] Continue to refer to Fig. 9 In some embodiments, the local terminal includes a first local client unit respectively arranged in the cloud control desktop and a second local client unit outside the cloud control desktop, and the virtual machine includes an agent unit and a first remote client unit arranged in the browser frame and a second remote client unit outside the browser frame.

[0203] The first local client unit or the second local client unit can allow the first remote client unit or the second remote client unit to pull the first data message packet through the first data message packet published by the proxy unit. Therefore, it is possible to achieve that the browser data of the local terminal or the data of the local application is obtained by the browser of the virtual machine or other programs running in the virtual machine (consumption of the data message packet).

[0204] Similarly, the first remote client unit or the second remote client unit can publish the second data message packet through the proxy unit to allow the first local client unit or the second local client unit to pull the second data message packet. Therefore, the browser data (including UAPP running data) of the virtual machine or the data of the application running in the virtual machine can be obtained by the browser of the local terminal or other programs running in the local terminal (consumption of data message packets).

[0205] In addition, the second local client unit can also directly obtain local data, the local data includes any one or more of local storage data, local application data, local hardware data and local user feature data; and the first local client unit and the second local client unit can directly communicate to obtain local data, and generate a first data message packet from at least a part of the obtained data. The first data message packet here can be used by the subroutine running in the local browser frame, and can also be published to the proxy unit for remote client units to pull or consume.

[0206] For example, refer to Fig.10 In one embodiment, the second local client unit is integrated into a part of the background service of the local terminal, and can call the local basic application to obtain local data such as photos, text messages, phone records, decoded data, etc., and then generate a data message packet and publish it to the proxy unit for the remote client unit to pull or consume. Fig.10 In another embodiment, the local terminal background service obtains local data of control and operation status (such as network status, power, CPU occupancy, etc.) by calling the local system program, which can be transmitted to the first local client unit, and then a data message packet is generated and published to the proxy unit for remote client units to pull or consume. The difference of this embodiment is that various method processes (such as the methods in Sections 3.2.1 and 3.2.3 above) can be implemented according to local data through subroutines running in the local browser framework (or cloud control desktop).

[0207] Reference Fig.11In one embodiment, a location request (with a request ID) sent by a map APP running in a remote browser framework is captured by a local docking unit associated with the positioning application service layer of the cloud operating system, triggering the local client unit to publish a positioning data message packet, and then the positioning data obtained by the local terminal background service (e.g., Lacation Manager) with a local positioning program or process is transmitted to the first local client unit, and then a data message packet is generated and published to the proxy unit. The remote client unit pre-subscribes to the positioning data message of the local terminal from the proxy unit according to the corresponding request ID. When the proxy unit has the message, it can consume the message and obtain the positioning data. Among them, the positioning data obtained by the first local client unit can be used by the subroutine running in the local browser framework to generate a positioning point on the map interface received locally using the positioning data under specific circumstances. For example, when using a map APP for in-vehicle navigation, if the network connection between the local terminal and the cloud server is limited or disconnected, the subroutine running in the local browser framework can continue to display the navigation map interface using the interface image cached on the network, while using the current positioning data of the local terminal to display the positioning point on the interface, thus maintaining the navigation experience for a period of time; due to the asynchronous communication method of the subscription protocol, when the local terminal and the cloud server resume network connection, the local client unit resumes sending the positioning data during the network disconnection period to the proxy unit, thereby being able to push positioning data messages for all navigation periods to the remote client unit for the map APP to obtain complete positioning data.

[0208] In addition, in some embodiments, the data transmission method based on the subscription mode may also include the following steps: enabling the first or second remote client unit of the virtual machine logged in to the user account to obtain remote data, the remote data including virtual machine storage data, virtual machine hardware data or data associated with the user account in the user database of the cloud server; generating a second data message packet with at least a portion of the obtained remote data. Therefore, it can be achieved that the UAPP running in the virtual machine can cross programs or cross browser frames to obtain the user database outside the virtual machine, or the hardware data in the virtual machine.

[0209] In addition, in some embodiments, the data transmission method based on the subscription method may also include the following steps: enabling the proxy unit of each of the multiple virtual machines associated with the user account to maintain a communication connection with the general agent service center of the cloud server; enabling the proxy units of the multiple virtual machines to exchange data transmission packets, wherein the data transmission packets include UAPP data and one or more data message packets; and determining that the data message packets in the proxy unit have been consumed by the client units of the multiple virtual machines associated with the user account. Therefore, data transmission, synchronization, migration, etc. between multiple virtual machines in the cloud operating system environment of the present invention can be realized.

[0210] 4.3 Data Buffering Method

[0211] Since the application service module of the cloud operating system of the present invention often needs to connect to the local terminal and the remote computing device via the Internet, or needs to call services and resources from a third-party server, an optimized data buffering method can be used on both sides of the network transmission.

[0212] Reference Fig.12 In some embodiments, a web page file cache is provided in the browser frame of the virtual machine, which is used to cache the page files obtained by UAPP from a third-party server (e.g., a web server); and an interface image cache is provided in the browser frame of the local terminal, which is used to cache the interface images of UAPP. It can be understood that only a large number of more fragmented page data (which may also contain confidential information of the user) is cached in the cloud service / virtual machine, and the local terminal only needs to cache the image data, so it will not put pressure on the storage of the local terminal, and it is also convenient to clean up the cache. In addition, this solution is safer and can completely prevent the possibility of user data leakage from the local cache.

[0213] In other embodiments, the data buffering method according to the present invention basically includes the following steps: establishing a data buffer between applications (such as local programs and cloud programs) associated with data transmission in a local terminal and a virtual machine of a cloud server; completing and generating continuous data in the data buffer according to historical received data, and the application for reading data from the data buffer continuously performs data reading. The data buffer includes an interface image data buffer, a multimedia data buffer, a positioning data buffer, etc.

[0214] In one embodiment, the data transmitted by the local program to the cloud program is first temporarily stored in the data buffer of the local terminal, and then transmitted to the buffer of the cloud server, and then read by the cloud program. Timestamps (set to t_n to t_n+4) can be set for each data packet (set to d_n to d_n+4, etc.) in the local and cloud server buffers. If the data that the local program should have transmitted to the cloud server buffer at a time node (such as t_n+3) is interrupted or lost due to network interruption, the supplementary data (d_n+2') corresponding to the time node (such as t_n+3) is written in the data buffer. The supplementary data (d_n+2') can be equal to the data (d_n+2) of the previous time node (such as t_n+2), or it can be the linear prediction value of the most recent multiple data. When the local program resumes the normal data docking with the buffer of the cloud server, normal data (d_n+4) can be written under the new timestamp (such as t_n+4) of the buffer of the cloud server. Therefore, data continuity can be maintained in the data buffer, so that the cloud program can have continuous data reading.

[0215] In one example, the local program takes the positioning APP of the local terminal as an example, and the UAPP in the virtual machine can be a map APP, in which the positioning APP sends positioning data to the cloud program map APP so that the map APP can normally locate the geographical location of the local terminal. By adopting the data buffering method of this embodiment, the map APP of the virtual machine can continuously obtain positioning data, at least when the local terminal is disconnected from the cloud service, there will still be no map positioning drift. Similarly, when the cloud program transmits data to the local program, continuous data can also be generated in the buffer of the local terminal to avoid drift in the data received by the local program.

[0216] 4.4 Data Redirection Method

[0217] In some embodiments, a data redirection method for a cloud operating system may generally include the following steps: establishing a communication connection between a local terminal and a remote computing device, obtaining a first communication docking address connected to the local terminal, and sending a local terminal identifier to the remote computing device; using the local terminal identifier to establish a communication connection between the remote computing device and a third-party server, and obtaining a second communication docking address connected to the third-party server; disconnecting the communication connection between the remote computing device and the third-party server, and modifying the first communication docking address and the second communication docking address to be consistent, so that at least a portion of the data transmission between the remote computing device and the third-party server is redirected to data transmission between the local terminal and the third-party server.

[0218] In some embodiments, multimedia data (e.g., video and / or audio data) can be intercepted from data sent by a third-party server to a remote computing device, so that the multimedia data transmission between the remote computing device and the third-party server is redirected to stream data transmission between the local terminal and the third-party server. Fig.13 In one example, UAPP, taking the news APP as an example, calls services and obtains data files from the portal WEB server, which include news content data, web page layout, multimedia stream playback address, etc. When the user remotely browses to the video page of the news APP, the video stream playback address of the third-party server is obtained from the video page through the multimedia interception unit. At the same time, the playback address of the video control of the local terminal is redirected to the video stream playback address of the third-party server, and then the local video control is arranged and covered on the local interface according to the size and position of the video window of the video page of the news APP. Preferably, if the UAPP running on the cloud server plays the video in full screen, the video stream is directly transmitted to the local terminal for full screen playback, and the transmission of the interface image to the local terminal is paused.

[0219] In other embodiments, when the first local terminal and the second local terminal serve as the operation terminals of at least one user of the first remote computing device and the second remote computing device respectively, the data redirection method for the cloud operating system may include the following steps: determining an event of a direct communication connection initiated by the first local terminal to the second local terminal according to the communication connection established between the first remote computing device and the second remote computing device; obtaining a first communication docking address connected to the first local terminal from the data connection between the first local terminal and the first remote computing device; obtaining a second communication docking address connected to the second local terminal from the data connection between the second local terminal and the second remote computing device; modifying the first communication docking address and the second communication docking address to be consistent, so that at least a part of the data transmission between the first remote computing device and the second remote computing device is redirected to the data transmission between the first local terminal and the second local terminal. Preferably, at least a part of the data includes real-time call data, so that a direct call between the first local terminal and the second local terminal (assuming that the first and second terminals have a call function) can be realized without being forwarded by the remote computing device.

[0220] 5. User database and general agent service center

[0221] The user database deployed by the cloud server has the characteristics of backup synchronization mechanism and high security level, and it divides the data storage space corresponding to each user account. The user database can store data corresponding to each user account, including: user terminal list, UAPP list, UAPP data, user personal data and files, user operation habit records, user key, etc.

[0222] In some embodiments, Fig. 22 As shown, data synchronization is performed between the local terminal and the distributed user database (preferably the user relationship database) on the server side, so that the user's personal data and files such as photos, address books, and text messages stored in the local terminal can be updated synchronously with the content of the user database. The hardware data corresponding to each local terminal, such as the location data of the terminal itself, the battery power value, the local communication signal status, etc., are also stored in the user relationship database according to the ID of different local terminals and the data is kept synchronized to facilitate the authorized UAPP to retrieve data.

[0223] In addition, the user database of the cloud server and the general agent service center can be used to perform system scheduling among multiple virtual machines to allocate corresponding cloud operating system data and at least one virtual machine to the user account belonging to the local terminal.

[0224] In some embodiments, the proxy service method for a cloud operating system may include the following steps: according to the user account to which the local user terminal belongs, obtaining the cloud operating system data and application storage data corresponding to the user account from the user database in the cloud server, allocating at least one virtual machine having the cloud operating system data and application storage data corresponding to the user account, and providing system scheduling between multiple virtual machines; providing each virtual machine with a proxy unit for the local terminal to communicate with the virtual machine, and exchanging data transmission packets (such as 10 packets) between the proxy units of the multiple virtual machines. Fig. 9 The system scheduling among multiple virtual machines includes: system migration, computer resource scheduling, cross-virtual machine application scheduling and execution, or user data synchronization storage.

[0225] 6. User Application (UAPP)

[0226] In the following embodiments, a remote computing device is associated with a local terminal via a network and the local terminal serves as a user operation terminal and a user application interface receiving terminal.

[0227] The following first explains the working mode or method of UAPP. Fig.13 , UAPP, taking news APP as an example, generally includes interface layout code and program function code. When the UAPP runs in the WEB browser framework, the interface layout code is generated by the rendering engine. The control docking unit in the browser framework transmits the control instructions of the local terminal to operate the UAPP, and the program function code can call the local API and Web API to obtain user feature data, user personality data, user requests, etc. from the local terminal to send to the Web server of the news portal to obtain news content, media streams and other data associated with the user account, which are used to present user-customized news content in the interface of the news APP. Among them, a transmission channel is established between the local client unit and the remote client unit so that the user files of the local terminal (such as photos taken locally) are transmitted to the UAPP. Referring to the embodiment of the browser framework described in Part 2 above, the UAPP is preferably written based on JS scripts, and can adopt Native APP or a development method particularly suitable for Web APP, and can be used on multiple platforms (such as Windows, Linux / Android, Mac / iOS) to unify the development code with the Electron framework (a cross-platform desktop GUI application development framework based on Web front-end technology), which greatly facilitates the development of UAPP. The following describes the UAPP solution of the cloud operating system based on the present invention with multiple embodiments.

[0228] 6.1 Generation Method

[0229] Reference Figures 14 to 16, the method for generating a UAPP based on a cloud operating system includes the following steps: providing an application store client in a remote computing device through an application service module, the application store client connects to the application store server to obtain the icon and installation link of the UAPP; receiving a new UAPP addition request triggered by a local terminal user operation, and obtaining the icon and installation link through the application store server; obtaining the storage data of the new UAPP associated with the user account from the user database; loading the new UAPP in the remote computing device with the installation link based on the application service module, and importing the storage data. Among them, the application store server can be set in a cloud server or in a server of another operator.

[0230] The method for generating a UAPP based on a cloud operating system also includes the following steps: providing a new UAPP operating environment, connecting to local hardware resources and interface images, and accessing a user database through an application service module using a browser framework of a remote computing device; transmitting the interface image of the new UAPP to a local browser framework of a local terminal to run a cloud control desktop, and allocating local resources to the new UAPP according to a permission request table configured by the user. The permission request table will be described in detail in the following embodiments.

[0231] 6.2 Shelf and Installation

[0232] Reference Figures 14 to 16 , combined with Fig.16 In the steps S411-S416 of the UAPP listing method, the UAPP listing request data includes the following steps: S411, obtaining the listing request data of the UAPP, the listing request data includes the icon file, WEB address, permission request table, the type of the required browser kernel and the original code; S412, determining the browser kernel type; S413, determining that the listing request data complies with the application review rules, and then packaging the icon file, WEB address and original code and listing them to the application store server (which can be a server website), and assigning an installation link. If it fails to pass the review, the listing fails.

[0233] Reference Fig.14 In some embodiments, the listing request information may also include: various keys of the application, which are used as credentials to enable program functions or access the Web server; page style and control habits, such as Android or iOS interface style and gesture habits; the type of plug-ins required to be installed for the browser framework; and copyright information certification information. Fig.15In some embodiments, the permission request table is to ensure the function realization and safe and stable operation of UAPP, and needs to apply for or use the relevant permissions of the cloud operating system, such as SMS and contact information, device location information, user mobile phone number, read and write local storage, notification, etc. Then, when UAPP is installed, after the user confirms the authorized permissions, a target permission table is generated (such as S416).

[0234] In a preferred embodiment, (such as S414) when the required browser type does not match the kernel type of the browser frame provided by the local terminal or the remote computing device, the browser frame provided by the remote computing device switches the kernel to match the required browser type, or reloads the function in the original code to make the installed UAPP match the kernel of the browser frame. (such as S415) It is necessary to configure the theme and operation mode of the UAPP so that when the UAPP is running, the page provided by the browser frame is switched to the page theme and operation mode of the type of the required browser of the UAPP.

[0235] The UAPP listing review can include the following processes: (1) Content review - the app store server will review the content of the app to ensure that the content of the app meets the requirements of the app store server. For example, the app store server prohibits apps that do not comply with local regulations. Therefore, the review process will carefully check the content keywords of the app to ensure that the content of the app will not cause any harm to users. (2) Functional review - the app store server will review the functions of the app to ensure that the functions of the app meet the requirements of the app store server. For example, the app store server may require that the application must not contain any malicious code, viruses, Trojans, etc. The review process will carefully check the functions of the app to ensure that the functions of the app will not cause any harm to the user's device. (3) User experience review - the app store server will review the user experience of the app to ensure that the user experience of the app meets the requirements of the app store server. For example, the app store server may require that the interface design of the application is simple and easy to use, and the functions of the application must also be easy to use. The review process will carefully check the user experience of the app to ensure that the user experience of the app meets the requirements of the app store server. (4) Copyright review - The App Store server will review the copyright of the app to ensure that the copyright of the app meets the requirements of the App Store server. For example, the App Store server may require that the application must not infringe on the intellectual property rights of any third party. The review process will carefully check the copyright materials of the application to ensure that the copyright of the application meets the requirements of the App Store server. (5) Security review - The App Store server will review the security of the application to ensure that the security of the application meets the requirements of the App Store server. For example, the App Store server may require the network communication protocol and third-party server used by the application to be tested and screened to ensure the privacy of users.

[0236] In some embodiments, reference Fig.23 , the icon files, WEB addresses and original codes of the UAPPs allowed to be listed can be packaged into a suite to be stored in the APP management and scheduling center, which is deployed and run in the physical machine of the cloud server. The icons of the UAPPs allowed to be listed are displayed in the UAPP mall webpage of the application mall service end (Web server). In some embodiments, the Web server that provides application mall services is a third-party server. Furthermore, one or more containers can be generated in the cloud server to obtain the UAPP suite from the APP management and scheduling center and deploy the public service station of the UAPP.

[0237] In a specific example, refer to Fig.23, when a user who interacts with the interface of a remote computing device (such as a VM in a cloud server) clicks the UAPP mall icon, the VM interacting with the user navigates to the website page of the UAPP mall. When the user clicks the icon of the APP to be installed to confirm the installation of the APP, the application mall Web server sends the user ID and the identification code of the APP to be installed (referred to as InAPP) to the APP management and scheduling center. The APP management and scheduling center allows the user ID and the identification code (APP ID) of the InAPP to establish an APP association tag, and then the VM that interacts with the user determines from the APP management and scheduling center which APPs are associated with the user, and then presents the InAPP icon on the APP homepage of the running interface, and assigns a navigation link to the icon to connect to the container that already hosts the InAPP, thereby completing the installation of the UAPP. In other embodiments, a service station for a third-party Web APP can be deployed in the container of the cloud server, and the WEB address in the original Web APP can be redirected to the address of the container, so as to provide APP services in the cloud server instead of the third-party Web server. Fig.23 As shown, the service station of the news APP is carried in the container Container1, and the original news APP is as follows Fig.13 The ones shown are served by a web server.

[0238] 6.3 Application Content Integration

[0239] In some embodiments, Userkey can be used to obtain resources and functions from the Web server to be integrated into the same UAPP. Fig.17 The local terminal used by the user stores the user's personal characteristic information (for example, fingerprints, facial data, personal security passwords, mobile phone number verification codes, cloud server user accounts, etc.), which requires a higher level of security and confidentiality, so it is necessary to convert or additionally extract the Userkey associated with the personal characteristic information (for example, secondary user account passwords, service request keys, etc.). Based on the fact that the user has registered an account through one or more service providers and obtained the Userkey, the registered Userkey can be obtained by allowing the user to authorize through the local user terminal, and then the proxy sends the Userkey to the service provider's Web server to obtain the user's personalized resources and integrate them into the same UAPP for presentation. Therefore, the Userkey-based solution brings greater flexibility and richness to the development and ecological chain of the UAPP of the cloud operating system of the present invention.

[0240] 6.4 Cloud Input Method

[0241] Reference Fig.18In some embodiments, a cloud input method for a cloud operating system may include the following steps: collecting user input operations of a user at a local terminal, the user input operations including user voice or operation points on a local interface; in a remote computing device, converting the user input operations into speech recognition words or operation instructions of a running interface; triggering an input panel based on the speech recognition words or operation instructions to generate first language data; retrieving second language data associated with the first language data from a remote word library to present the first language data and / or second language data on the running interface; wherein the remote word library receives local input data from a local word library of a local terminal, and the remote word library is synchronized with the user input data stored in a user database. The remote word library receives a personalized cloud word library associated with a user account from a third-party WEB server.

[0242] Preferably, the cloud input method for a cloud operating system may further include the following steps: calling an AI function through an application programming interface of an AI server to convert first language data into third language data, so as to present the third language data on a remote computing device, wherein the AI ​​server deploys a large-scale pre-trained language model (e.g., GPT, BERT, BLOOM, XLNet, ERNIE Bot, etc.).

[0243] Preferably, in addition to the input method APP, the application program interface of the AI ​​server can also be provided in other UAPPs, so that other UAPPs can access the AI ​​server to obtain services.

[0244] 6.5 Open other apps from within an app

[0245] Reference Fig.19 In some embodiments, the opening method of a UAPP for a cloud operating system may include the following steps: capturing a user input operation on a local terminal, where the user input operation includes a user voice command or an operation point on a local interface; in a remote computing device, determining an operation object of the user input operation in the current running interface of the first UAPP; determining that the operation object is associated with a second UAPP, and after capturing a message allowing the second UAPP to start, generating a running interface of the second UAPP (which may be a page of web page nature), allowing at least a portion of the interface of the first UAPP to be overlaid, and allowing the first UAPP to keep running.

[0246] Reference Fig. 20, the opening method of UAPP for cloud operating system may also include the following steps: pre-configuring an association list of operation objects and UAPPs to determine that each operation object is opened by the corresponding UAPP, and the operation object includes a file type, a hardware resource or a website address; or, after determining that the permission required by the UAPP to be opened is consistent with the permission type assigned by the local terminal, generating a message allowing the UAPP to start. Specifically, the association list of operation objects and UAPPs can be a file opening method table, for example, the default opening application for MP3 files is configured as a local player, and when a file that is not configured to open an application is clicked, an inquiry window will be issued to prompt the user whether to download an application that can open the file from the application store. Specifically, when the UAPP or link to be opened requires operating system permissions, the target permission table of the UAPP is called or the user is asked whether he agrees to authorize the permissions required for the opening event. After confirming that the permissions are enjoyed, the opening operation is allowed.

[0247] Reference Fig.21 In some embodiments, when it is determined in the remote computing device that the operation object is associated with the third UAPP or the system bar, the same type of UAPP or process is called in the local terminal. The types of the third UAPP include camera applications, phone applications, SMS applications, address book applications or other basic applications based on local functions, and the system bar includes the remote computing device status bar or the hardware control center bar. For example, a plurality of third UAPP icons and operation interfaces are provided in the running interface of the cloud server virtual machine, and the APP ID corresponding to each third UAPP is associated with the local basic APP or process (for example, camera, address book, SMS, positioning); when the user operates the icon of the third UAPP (for example, clicks on the phone), the operation interface (such as the dial pad) pops up, and then obtains the user operation instruction (such as dialing a mobile phone number) through the operation interface, and then returns to the local terminal to trigger the local program or process (local mobile phone dialer) to work. In one embodiment, the user operation instruction can be used through the JS program in the local browser framework to use the backend runtime module (for example, JS runtime) to trigger the local basic APP or process to work. Therefore, it can meet the switching requirements of the cloud UAPP and the application that requires local hardware.

[0248] Reference Fig. 22In some embodiments, the page of the control center bar in the remote computing device provides at least one icon for the user to directly operate the hardware functions of the local device, such as Bluetooth, WiFi, mobile network, ringing, location positioning, etc. When the monitoring feedback unit on the server side (such as the virtual machine in the remote computing device) detects that the user operates the icon of the control center (for example, starts positioning), an operation instruction (for example, starts the location positioning function of the local terminal) is issued to the local application calling unit of the local terminal, and the local application calling unit triggers the local basic program or process (local GPS system program) to start the local program or change the local system settings (for example, turn on GPS positioning); then the local terminal forwards the local data (for example, location data) triggered by the operation instruction to the user database of the remote computing device, and the data can be kept synchronized. Preferably, the control center bar can also be implemented as a page-type application that comes with the cloud operating system, and can also be as follows Fig.19 The method shown is superimposed on other UAPPs.

[0249] 7. Communication scheme for local terminals

[0250] Generally, multiple users can log in and associate at least one local terminal respectively. The following embodiments cover both communication between multiple terminals under the same user and communication between different terminals associated with multiple users. Based on the above-mentioned cloud operating system embodiments, the remote computing device may include a cloud server, which runs a virtual machine (VM), an APP management and scheduling center, and a container. The local terminal is connected to the VM in a one-to-one correspondence to perform interface interaction and communication data transmission after the user logs in. The APP management and scheduling center allocates APP services to VMs with users. The container is a series of processes isolated from the VM based on the Docker engine technology, which shares the kernel of the cloud operating system according to the present invention, and also encapsulates the application and its dependent environment in an independent runtime environment, so the container can be used to carry computer software programs, website programs, service processes, public APPs, etc. Preferably, the container carries an application service station, such as a public service website that provides APP services based on a page. As a result, the cloud server that interacts with multiple local terminals makes the local terminal a user operation terminal and an APP operation interface receiving terminal, and realizes instant communication and data interaction between multiple local terminals. The following will describe the communication-related technical solutions in detail through multiple embodiments.

[0251] 7.1 Communication Application and Installation

[0252] The applications involved in the communication function include basic programs or plug-ins installed on the local terminal, and user applications (UAPP) deployed on the server side. The local terminal can install native basic communication applications, or install basic communication application components as plug-ins in the browser framework of the cloud control desktop, which can also realize the basic communication functions on the local terminal, such as local system function calls, communication interface display, communication data file reception / forwarding, etc. The communication UAPP can be implemented in the form of a web page, so that the communication service website can be flexibly deployed in the service station of the cloud server or a third-party web server. The website provides multiple users with pages with communication interaction functions that can be accessed by logging in with different accounts.

[0253] Fig.23 The process of user installation of communication UAPP is described. After user 1 clicks the UAPP mall icon on the interface of the local terminal, VM1 interacting with user 1 navigates to the website page of the UAPP mall and transmits it to the local terminal for display (such as Fig.23 The second page on the left). Then user 1 clicks the installation button of the communication UAPP icon, and the App Store Web server sends user 1's ID1 and the identification code of the communication UAPP to the APP management and scheduling center. The APP management and scheduling center associates ID1 with the identification code of the communication UAPP, and can also add ID1 to the installation user ID field of the suite table of the communication UAPP to confirm that user 1 has installed the communication UAPP. Then the APP homepage of the running interface of VM 1 presents the icon of the communication UAPP (such as Fig.23 ), and assign a navigation link to the icon to connect to Container 2, which already hosts the communication service station, to complete the installation of the communication UAPP.

[0254] 7.2 Basic communication modes and methods

[0255] Reference Fig.24In some embodiments, the communication method according to the present invention basically includes the following steps: establishing a network association between a cloud server and multiple local terminals and using the multiple local terminals as the operating terminals of at least one user; providing a communication service station and a distributed database on the cloud server, and enabling the communication service station to access the distributed database; establishing access between the multiple local terminals and the communication service station, so as to present communication pages and / or store (or cache) communication data or files (for example, text, pictures, recordings, videos, etc.) on each local terminal; when there is a communication association between at least two local terminals, based on the communication service station receiving the communication data of the local terminal or based on the user data in the distributed database, the communication page and / or communication data is updated for each local terminal, and the updated communication page and / or communication data are simultaneously transmitted to the local terminals with communication association.

[0256] In a specific embodiment, the communication service station (website) is accessed through a browser logged in by a user in the cloud server, and after logging in to the user's account, the user's communication service webpage is entered. This page is taken as a communication UAPP running interface by an interface image acquisition unit and sent to a local browser and displayed as a local communication function interface.

[0257] The user's control data of the local interface is transmitted to the cloud server through the data forwarding unit, and the remote browser is operated in a remote interactive manner. At the same time, the text data of the user's input method can also be transmitted to the remote browser. The communication data files transmitted between user terminals during the communication process can be accessed in the distributed database through the service station. Files can also be accessed from the distributed database through the remote data forwarding unit to be transmitted with the local terminal. The remote browser can be configured with a back-end runtime module. When responding to the user's control instruction script, a remote hardware resource request (for example, audio and video decoding and playback, camera startup, etc.) is initiated through the local API where the remote browser is located, the local and remote data forwarding units (or driver docking units) trigger the back-end runtime module of the local browser with the same instruction script, and initiate a local hardware resource request with the same API; then the data originally returned by the virtual machine hardware resources is replaced by the data generated by the local hardware resources, and then returned to the remote browser, thereby achieving the effect of the remote browser calling the hardware resources of the local terminal. It can be understood that during the communication process of the user's local terminal, although the remote browser is operated to play audio in a remote interactive manner, it is actually the local hardware that makes the sound.

[0258] In another embodiment, a local terminal (such as Fig.25The local terminal 2 shown in the figure can also directly install a native communication UAPP, which includes a browser module to directly access the web interface locally. In this embodiment, the local terminal 2 directly accesses the communication service station to browse the communication service web page, and the native communication UAPP can directly call local hardware resources for audio and video playback. Compared with the solution of cloud-controlled desktop controlling the remote browser, the native communication UAPP in this embodiment is larger in size and less lightweight, but it can be directly installed on local terminals that do not have deep interaction with the cloud server, so that Fig.25 The local terminal 1 based on the cloud server virtual machine shown can also communicate with the local terminal 2 with a separate integrated communication UAPP.

[0259] 7.3 Communication interface and communication data interaction between multiple terminals

[0260] In the following embodiments, the communication interface and communication data interaction between the terminals are described by taking the two local terminals 1 and 2 that establish an instant messaging association as an example. It should be understood that it can be applicable to multiple local terminals. In some embodiments, the communication service station receives the communication data of the local terminals 1 and 2 (such as chat information, pictures, voice clips, etc.), which are generally cached in the storage unit of the communication service station, and can also be stored in a distributed database. In other embodiments, the communication service station receives the communication data of the remote computing device (such as a virtual machine) associated with the local terminal, which can also be cached in the storage unit of the communication service station, and can also be stored in a distributed database. Preferably, a subscription-based communication protocol (such as MQTT) is used to transmit data between the local terminal and the communication service station, such as Fig.28 described.

[0261] When the communication service station receives a user operation instruction from the local terminal 1, which instructs to transmit the specified communication data to the local terminal 2, the specified communication data is obtained from the communication service station and / or the distributed database to the local terminal 2. If the local terminal 2 interacts with the virtual machine of the cloud server like the local terminal 1, the specified communication data obtained from the communication service station and / or the distributed database is transmitted to the virtual machine interacting with the local terminal 2.

[0262] Reference Fig.25In the embodiment shown, the local browser framework or native APP of the local terminals 1 and 2 includes WebView and WebRTC components, wherein the WebView component is used to render and display the communication page, and the WebRTC component is used to implement the real-time transmission of video streams, audio streams or any other communication data. The WebRTC components of the local terminals 1 and 2 are connected to the signaling server deployed in the cloud server to establish peer-to-peer instant communication between the WebRTC components of the local terminals 1 and 2. The timing and triggering process of peer-to-peer instant communication between local terminals are described in detail in Section 7.5 below.

[0263] Reference Fig.25 ,The distributed databases accessed by the communication service stations may include relational ,user databases (e.g., SQL) and non-relational databases (e.g., No-SQL).

[0264] In some embodiments, data synchronization is performed between the relational user database and the user's local terminal, so that the user's personal data and files such as photos, address books, and text messages stored in the local data storage units of one or more local terminals can be synchronized and updated in the user database. In addition, the device data corresponding to each local terminal, such as location data, battery power value, local communication signal status, etc., are also stored in the user relational database according to the ID of different local terminals and the data is kept synchronized. In addition, each user's communication history data can also be stored in the user database. Fig.26 In one example, the relational database table stores the data data_U1 sent by the local terminal U1 of user 1 in the directory DIR1 and uses the MD5 code as a check, wherein the directory DIR1 and the data data_U1 are associated with the local device identifier U1_ID of user 1, so that the communication UAPP logged in by the user 1 account can quickly retrieve the device data of the corresponding local terminal U1.

[0265] In some embodiments, the non-relational database can be used to store document files, picture files, audio files, video files, APP suites, etc. shared by multiple users or multiple communication terminals. Fig.26 As shown, the files stored in the relational database table are equipped with MD5 codes as indexes and verification. When the MD5 codes of the files to be obtained by the local terminals of one or more users or their associated remote computing devices are retrieved in the non-relational database, the database file access path corresponding to the MD5 code can be directly allocated to the local terminals or their associated remote computing devices, which can improve file access efficiency and save database storage space.

[0266] Continue to refer to Fig.26In one example, users 1 and 2 log in to the communication service station through their local terminals respectively to display the communication pages of users 1 and 2 on their local terminals respectively, wherein the communication service station pre-accesses the device data, historical communication data, etc. related to users 1 and 2 in the user database to present the historical communication records on the communication pages of users 1 and 2.

[0267] When user 1 Fig.26 The dotted finger in the figure long presses the voice recording and sending button to record a voice and send it to user 2. This voice data_U1 is cached in the communication service station and also stored in the user database directory DIR1. Preferably, the user data storage directory can be constructed based on factors such as user ID, local terminal name, UAPP name, generation time, file type, etc. For example, the value of the directory DIR1 here can be "user1001:U1\IM app\202407031830\voice\". Then, the communication service station presents the voice notification icon on the communication page of user 2, and forwards the voice data_U1 to user 2 via the data forwarding unit. In addition, this voice data can also be synchronously stored in the user database directory of user 2 to facilitate users 1 and 2 to retrieve the voice history data between them when they reopen the communication UAPP page later.

[0268] When user 2 Fig.26 The solid finger in the figure selects the image transmission, and operates the local terminal to take a picture and send it to user 1. At this time, the local monitoring unit captures the camera photo event of the local terminal of user 2, and transmits the just-taken picture stored in the local terminal to the communication service station through the data forwarding unit. Similar to the above example of voice transmission, the picture data data_U2 is cached in the communication service station and is also synchronously stored in the user database directory of users 1 and 2. Then the picture data data_U2 is forwarded to user 1 via the data forwarding unit.

[0269] Preferably, when user 1 sends a public file to user 2 (for example, a shared document of a communication group to which users 1 and 2 belong), the communication service station preferentially obtains its MD5 code when receiving the public file. When it is determined that there is a corresponding file for the MD5 code in a non-relational database, the corresponding file is directly retrieved from the non-relational database and transmitted to user 2.

[0270] 7.4 Lightweight Applications

[0271] A lightweight application (Mini APP) can be provided inside the communication UAPP. In some embodiments, the method for generating a Mini APP in a user application may include the following steps: providing a lightweight application store client in a remote computing device, the lightweight application store client connects to the application store server to obtain the icon and navigation link of the lightweight application; receiving a request for a new lightweight application triggered by the operation of the local terminal user, and obtaining the icon and navigation link through the lightweight application store server; obtaining the storage data of the new lightweight user application associated with the user account from the user database; based on the page generation mechanism of the browser framework, loading the page of the new lightweight user application with the navigation link in the user application, and importing the storage data.

[0272] Reference Fig. 27 In one specific embodiment, the user enters the Mini APP page by sliding on the communication page. The Mini APP is also similar to Figures 14 to 17 as well as Fig.23 The generation, listing, installation, etc. of the UAPP shown are not described again here. Preferably, the Mini APP can be a web application that is lighter than the UAPP, and the web page path of the Web service station of each Mini APP can be directly provided at the communication service station, thereby providing a light application service.

[0273] 7.5 Audio and Video Instant Messaging

[0274] In some embodiments, a cloud operating system-based instant messaging method between multiple local terminals may include the following steps: determining that instant messaging has been established between multiple local terminals; assigning IP addresses within the same local area network to at least two of the local terminals, and establishing a point-to-point communication connection between the local terminals; generating an instant messaging page in the cloud control desktop of the local terminal, so that communication data (for example, voice or video call data) can be directly transmitted between the local terminals without passing through a communication service station and the instant messaging page can be updated.

[0275] Reference Fig.28 In a specific embodiment, the local terminals of users 1 and 2 may respectively have WebRTC components for implementing real-time transmission of video streams, audio streams or any other communication data. Preferably, each local terminal communicates with the communication service station through a subscription protocol, or multiple local terminals that establish communication associations may communicate through a subscription protocol. In addition, the WebRTC component of the local terminal may also be connected to the signaling server deployed in the cloud server through a subscription communication method. In addition, the signaling server may also subscribe to data from the communication service station.

[0276] When user 1 clicks the instant video communication button on the ordinary chat page, the monitoring unit of the local terminal of user 1 captures the action event and calls the camera and audio hardware of the local terminal; and the communication service station pushes the video communication notification and button to user 2 and displays it in its chat page. When the monitoring unit of the local terminal of user 2 captures the instruction of user 2 clicking the button to agree to the instant video communication, the camera and audio hardware of the local terminal are called, and the handshake data of the instant communication (agreeing between users 1 and 2) is sent to the communication service station. The signaling server can subscribe to the handshake data of agreeing to the instant communication in advance at the communication service station, the local terminal, or the remote computing device interacting with the local terminal, thereby receiving the instruction of agreeing to the instant video communication issued by user 2, and then triggering the console to allocate the LAN IP address and port (for example, 10.0.8.2:3478 and 10.0.8.3:3478) to the local terminals of users 1 and 2 through NAT routing, which is used to establish point-to-point instant communication between the WebRTC components (WebRTC_clt1 and WebRTC_clt2) of the local terminals 1 and 2. Therefore, users 1 and 2 can experience high-speed, low-latency, instant video communication without having to relay communication data through other network devices.

[0277] 8. Others

[0278] The architecture, framework, module, kit, component, unit, control, plug-in, method steps, etc. in the embodiments of the present invention may implement its features in digital electronic circuits or in computer hardware, firmware, software, or in a combination thereof. The features may be presented in a computer program product that is tangibly embodied in an information carrier (e.g., in a machine-readable storage device) so as to be executed by a programmable processor; and the method steps may be executed by a programmable processor that executes an instruction program to perform a specific implemented function by operating on input data and generating output.

[0279] The described features can be advantageously implemented in one or more computer programs that can be executed on a programmable system, the programmable system comprising at least one input device, at least one output device and at least one programmable processor coupled to receive data and instructions from a data storage system and to transfer data and instructions to the data storage system. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform a certain activity or produce a certain result. A computer program can be written in any form of programming language (e.g., C++, Java) including compiled and interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a program segment, subroutine or other unit suitable for use in a computing environment. For this purpose, a computer program can run on a programmed ASIC.

[0280] For example, suitable processors for executing a program of instructions include both general-purpose microprocessors and special-purpose microprocessors, and one or the only processor of multiple processors or cores of any type of computer. Generally speaking, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage media for storing instructions and data. Generally speaking, a computer can communicate with mass storage devices to store data files. These mass storage devices may include disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. The processor and memory may be supplemented by, or incorporated into, a dedicated integrated circuit.

[0281] Although various aspects of the specific implementation are described above, it should be apparent that the various features of the above-mentioned specific implementation can be embodied in various forms, and any specific structure and / or function described above is only exemplary. Based on the present disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or the method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced. It will also be understood that although the terms "first", "second", etc. may be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first node can be referred to as the second node. The terms used herein are only for describing a specific specific implementation and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one" and "the" are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. As used herein, the term "if" may be interpreted to mean "when the antecedent condition is true" or "when the antecedent condition is true" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent condition is true, depending on the context.

Claims

1. A cloud operating system, arranged in a remote computing device, the remote computing device is associated with a local terminal via a network and the local terminal is used as a user operation terminal and a user application interface receiving terminal, the local terminal includes a cloud control desktop, It is characterized in that The cloud operating system includes: A basic kernel, configured to call physical or virtualized hardware resources of the remote computing device according to a request; A runtime library, based on the basic kernel, to provide a runtime environment for at least one operating system platform; An interface service module, the interface service module corresponds to the runtime library to provide an application program interface; The application service module is used to provide an application program interface and service connection to the cloud control desktop of the local terminal.

2. The cloud operating system according to claim 1, wherein: The cloud control desktop includes: Interface controls or plug-ins to provide a user operation interface and functional extensions associated with the interface based on the browser framework; A local access unit, which is connected to the interface control or plug-in and is used to capture the user's real-time manipulation data and information of the local application program accessing the local terminal and hardware information; One or more service units, the service units are connected to the interface controls or plug-ins, and are used for interaction of interface images, videos and audios between local and remote; One or more client units, each of which is connected to the service unit and the local access unit, and is used for communicating and transmitting with the proxy service program in the remote computing device.

3. A remote interface generation method, wherein: The method comprises the following steps: Establishing a network association between a remote computing device and a local terminal and using the local terminal as a user operation terminal; Acquire a user account and hardware configuration parameters from the local terminal, the hardware configuration parameters including a local operating system type, a screen size, a screen resolution, and an input device type; Acquire the interface style and operation habits corresponding to the user account or the local operating system type from a user database associated with the remote computing device, generate a browser window in the browser frame of the remote computing device with the acquired screen size or screen resolution of the local terminal, configure a renderer to generate an interface layout based on the interface style, and determine the operation gestures of the browser window with the operation habits, thereby serving as an interface environment for a user application running based on the browser frame; The interface image of the browser window is captured and transmitted to the local terminal for display as an interface image.

4. A data transmission method, for a cloud operating system, wherein the cloud operating system is deployed in a remote computing device, the remote computing device is associated with a local terminal via a network and the local terminal is used as a user operation terminal and a system interface display terminal of the cloud operating system, the local terminal and the remote computing device are respectively provided with one or more client units, and the remote computing device is provided with an agent unit that is resident and runs in the background, in, The method comprises the following steps: Through the subscription protocol transmission channel, the client unit set in the local terminal or the remote computing device publishes one or more data message packets through the proxy unit to allow one or more other client units set in the local terminal or the remote computing device that have pre-subscribed to the data message packets to pull the data message packets to complete the consumption of all data message packets.

5. A communication method, based on the cloud operating system according to any one of claims 1 to 2, wherein the method comprises the following steps: Establishing a network association between a remote computing device and a plurality of local terminals and using the plurality of local terminals as operating terminals for at least one user; Providing a communication service station on the remote computing device and enabling it to access a distributed database, establishing access between the plurality of local terminals and the communication service station, so as to present a communication page and / or cache communication data respectively on each local terminal; Determining that at least two local terminals have a communication association; Based on the communication service station receiving the communication data of the local terminal or based on the user data in the distributed database, the communication page and / or communication data is updated for each local terminal, and the updated communication page and / or communication data is simultaneously transmitted to the communication-associated local terminals.

6. The method according to claim 5, wherein: The method comprises the following steps: Determining that at least two local terminals have an instant messaging association; Allocating network addresses in the same local area network to the at least two local terminals, and establishing a point-to-point communication connection between the local terminals; Generate an instant messaging page in the cloud control desktop of the local terminal, The local terminals can directly transmit communication data and update the instant messaging page without passing through the communication service station.

7. The method according to claim 5 or 6, wherein: At least two local terminals with communication association include a first communication terminal and a second communication terminal with instant communication association, the remote computing device and the distributed database share the same physical device resources, The method comprises the following steps: enabling the communication service station to receive communication data from one or more local terminals and store the data in the distributed database; determining, by the communication service station, that a user operation instruction from the first communication terminal is received, the user operation instruction instructing to transmit specified communication data to the second communication terminal; The designated communication data is obtained from the distributed database and sent to a remote computing device associated with the second communication terminal.

8. A data redirection method, based on the cloud operating system according to any one of claims 1 to 2, wherein the method comprises the following steps: Establishing a communication connection between the local terminal and the remote computing device, obtaining a first communication docking address connected to the local terminal, and sending a local terminal identifier to the remote computing device; Establishing a communication connection between the remote computing device and a third-party server using the local terminal identifier, and acquiring a second communication connection address connected to the third-party server; The first communication docking address and the second communication docking address are modified to be consistent, so that at least a part of the data transmission between the remote computing device and the third-party server is redirected to the data transmission between the local terminal and the third-party server.

9. A method for generating a lightweight application in a user application, based on the cloud operating system according to any one of claims 1 to 2, wherein the method comprises the following steps: Providing a lightweight application store client in a remote computing device, wherein the lightweight application store client is connected to an application store server to obtain icons and navigation links of lightweight applications; Receiving a request for adding a new lightweight application triggered by the local terminal user operation, and obtaining an icon and a navigation link through the lightweight application store server; Obtaining storage data of the new lightweight user application associated with the user account from a user database; Based on the page generation mechanism of the browser frame, the page of the new lightweight user application is loaded in the user application with the navigation link, and the stored data is imported.

10. A computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 3 to 9.

Images