Cross-platform cloud operating system and method thereof

Abstract

A cloud operating system arranged in a remote computing device, the remote computing device is associated with a local terminal through a network and takes the local terminal as a user operating end, the cloud operating system comprises a basic kernel used for calling physical and / or virtualized hardware resources of the remote computing device according to a request; the runtime library is used for providing at least one runtime environment of the operating system platform based on the basic kernel; the interface service module corresponds to the runtime library so as to provide an application program interface; and the application service module is used for providing interface and service docking of the application program for the local terminal.

Description

Cross-platform cloud operating system and method thereof Technical Field

[0001] The present invention relates to an operating system based on cloud computing and a method thereof, and belongs to the technical field of software and communication. Background Art

[0002] Smart devices, such as smartphones, car central control units, smart wearables, and smart home appliances, have become indispensable internet service terminals for people. The operating systems of these smart devices vary not only in type but also in version, resulting in significant fragmentation. This means that the development, installation, upgrade, and use of user applications must accommodate the requirements and practices of each type and version of the operating system, creating increasing challenges for both developers and users. Furthermore, the hardware configurations and replacement cycles of various smart devices vary significantly, making it difficult to simultaneously improve the user experience. For example, the increasing size of apps necessitates more frequent replacements of smartphones to increase local storage capacity, while other hardware upgrades are often oversupplied, resulting in wasteful performance. Alternatively, the replacement frequency of car central control units or smart home appliances is low, while the upgrade frequency and user experience of user applications requiring human-computer interaction within them lag far behind those of similar smartphone applications.

[0003] Therefore, there is a need for a cross-platform, network-based operating system that is compatible with a variety of smart terminals, provides a unified user application ecosystem, and offers a rich user experience and simplified program development. Furthermore, thanks to the continued development of next-generation mobile communications technology and the comprehensive coverage of network base station infrastructure, data connections between smart terminals and external servers are faster, less-latent, and more stable, providing the technical foundation and operational support for network-based operating systems. Summary of the Invention

[0004] The present invention provides a cross-platform cloud operating system and related methods, browsers, storage media, terminals, servers, etc., aiming to solve at least one of the technical problems existing in the prior art.

[0005] One aspect of the present invention relates to a cloud operating system, which is provided in a remote computing device but not deployed on a local terminal. The remote computing device is associated with the local terminal via a network, and the local terminal serves as a user operation terminal. The cloud operating system includes: a base kernel for invoking the physical and / or virtualized hardware resources of the remote computing device upon request; a runtime library based on the base kernel to provide a runtime environment for at least one operating system platform; an interface service module corresponding to the runtime library to provide an application program interface; and an application service module for providing an application program interface and service connection to the local terminal.

[0006] Another aspect of the present invention relates to a remote interface generation method for the cloud operating system. The remote interface generation method includes the following steps: establishing a network association between a remote computing device and a local terminal and using the local terminal as a user operation terminal; obtaining a user account and hardware configuration parameters from the local terminal, the hardware configuration parameters including the local operating system type, screen size, screen resolution, and input device type; obtaining the interface style and operating habits corresponding to the user account or local operating system type from a user database associated with the remote computing device; generating a browser window in a browser frame of the remote computing device using the obtained screen size or screen resolution of the local terminal; configuring a renderer to generate an interface layout based on the interface style, and determining operating gestures for the browser window based on the operating habits, thereby serving as an interface environment for a user application running based on the browser frame; and capturing an interface image of the browser window and transmitting it to the local terminal for display as an interface image.

[0007] Another aspect of the present invention relates to an interface image transmission method that can be used in the above-mentioned cloud operating system. The interface image transmission method includes the following steps: establishing a control data interactive connection between a local terminal and a remote computing device, wherein changes in at least a portion of the interface of the remote computing device are caused by control data sent by the interactively connected local terminal; maintaining the transmission connection of the interface image of the remote computing device to the local terminal for at least one period of time; and dynamically adjusting the interface image to be transmitted from the remote computing device to the local terminal based on the transmission interval of the interface image transmitted from the remote computing device to the local terminal, the dynamic adjustment including: after determining that image transmission below an interval threshold occurs in a period of time, reducing the number of image transmissions and / or reducing the quality of at least a portion of the interface image in the next period of time.

[0008] Another aspect of the present invention relates to a remote browser running on the aforementioned cloud operating system. The remote browser includes: a client unit, configured to communicate with other applications running on the cloud operating system and with a local terminal to obtain data from the local terminal and the remote computing device; and an interface image acquisition unit, configured to acquire at least a portion of an interface image from the user interface of a web-based user application running on the remote browser. The client unit and the interface image acquisition unit are each connected to an image interaction service program in the cloud operating system, which implements the aforementioned remote interface generation method.

[0009] Another aspect of the present invention relates to a local browser running on a local operating system. The local browser includes: a client unit, which is used for the local browser to communicate with other applications running on the local operating system and with a remote computing device to obtain data from the local terminal and the remote computing device; a display component, which is used to display interface images from the remote computing device; and an image interaction service unit connected to the display component, which is used to implement the above-mentioned interface image transmission method.

[0010] Another aspect of the present invention relates to a data transmission method for a cloud operating system. The cloud operating system is deployed on a remote computing device, which is associated with a local terminal via a network, with the local terminal serving as the user operation terminal and system interface display terminal of the cloud operating system. The local terminal and the remote computing device are each equipped with one or more client units, and the remote computing device is equipped with a proxy unit that runs resident in the background. The data transmission method comprises the following steps: via a subscription protocol transmission channel, causing a client unit located on the local terminal or the remote computing device to publish one or more data message packets through the proxy unit, allowing one or more additional client units located on the local terminal or the remote computing device that have pre-subscribed to the data message packets to pull the data message packets to complete the consumption of all data message packets.

[0011] Another aspect of the present invention relates to a data redirection method based on the above-mentioned cloud operating system, comprising the following steps: optionally based on a long link network protocol, establishing a communication connection between a local terminal and a remote computing device, obtaining a first communication docking address connected to the local terminal, and sending a local terminal identifier to the remote computing device; using the local terminal identifier to establish a communication connection between the remote computing device and a third-party server, and obtaining a second communication docking address connected to the third-party server; modifying the first communication docking address and the second communication docking address to be consistent, so that at least part of the data transmission between the remote computing device and the third-party server is redirected to data transmission between the local terminal and the third-party server.

[0012] Another aspect of the present invention relates to a data redirection method, based on the above-mentioned cloud operating system, wherein a first local terminal and a second local terminal serve as operation terminals for at least one user of a first remote computing device and a second remote computing device, respectively. The data redirection method comprises the following steps: determining an event of a direct communication connection initiated by the first local terminal to the second local terminal based on a communication connection established between the first remote computing device and the second remote computing device; obtaining a first communication docking address connected to the first local terminal from a data connection between the first local terminal and the first remote computing device; obtaining a second communication docking address connected to the second local terminal from a data connection between the second local terminal and the second remote computing device; and modifying the first communication docking address and the second communication docking address to be consistent, so that at least a portion of the data transmission between the first remote computing device and the second remote computing device is redirected to data transmission between the first local terminal and the second local terminal.

[0013] Another aspect of the present invention relates to a data buffering method for use in the aforementioned cloud operating system. The data buffering method comprises the following steps: establishing one or more data buffers between a local terminal and multiple nodes in a remote computing device transmitted via a network; and generating continuous data in the data buffers based on historically received data from the nodes, such that nodes reading data from the data buffers continuously read data.

[0014] Another aspect of the present invention relates to a proxy service method for the aforementioned cloud operating system, wherein the remote computing device includes a cloud server and at least one virtual machine generated in the cloud server. The proxy service method comprises the following steps: based on the user account of a local user terminal, obtaining cloud operating system data and application storage data corresponding to the user account from a user database in the cloud server; allocating at least one virtual machine corresponding to the cloud operating system data and application storage data to the user account; and providing system scheduling between multiple virtual machines; providing a proxy unit for each virtual machine to enable the local terminal to communicate with the virtual machine, and exchanging data transmission packets between the proxy units of the multiple virtual machines.

[0015] Another aspect of the present invention relates to a method for generating a user application based on the above-mentioned cloud operating system, the method comprising the following steps: providing an application store client in a remote computing device through the application service module, the application store client connecting to the application store server to obtain the icon and installation link of the user application; receiving a new request for a new user application triggered by an operation of a local terminal user, and obtaining the icon and installation link through the application store server; obtaining the storage data of the new user application associated with the user account from a user database; loading the new user application with the installation link in the remote computing device based on the application service module, and importing the stored data.

[0016] Another aspect of the present invention relates to an input method based on a cloud operating system, wherein the cloud operating system is set in a remote computing device, the remote computing device is associated with a local terminal through a network to transmit an interface image to the local terminal, and the local terminal is used as a user operation terminal. The input method includes the following steps: collecting user input operations of the user on the local terminal, the user input operations including user voice or operation points on the local interface; in the remote computing device, converting the user input operations into voice recognition words or operation instructions of the running interface; triggering an input panel based on the voice recognition words or the operation instructions to generate first language data; retrieving second language data associated with the first language data from a remote vocabulary to present the first language data and / or second language data on the running interface; wherein the remote vocabulary receives local input data from the local vocabulary of the local terminal, and the remote vocabulary is synchronized with the user input data stored in the user database.

[0017] Another aspect of the present invention relates to a method for opening a user application, which is used in the above-mentioned cloud operating system. The method for opening a user application includes the following steps: capturing a user input operation on the local terminal, wherein the user input operation includes a user voice command or an operation point on the local interface; determining, in the remote computing device, an operation object of the user input operation in the current running interface of a first user application; determining that the operation object is associated with a second user application, and after capturing a message allowing the second user application to be launched, generating a running interface for the second user application, allowing it to be superimposed on at least a portion of the interface of the first user application, and allowing the first user application to continue running.

[0018] Another aspect of the present invention relates to a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement any of the above methods when executed by a processor.

[0019] Another aspect of the present invention relates to a mobile computing terminal, comprising: a network communication module connected to a cloud server; an image processor for processing interface image data sent by a remote computing device; a screen for displaying the interface image data; and the computer-readable storage medium.

[0020] Another aspect of the present invention relates to a cloud computing server, comprising: at least one remote computing device, the remote computing device including one or more virtual machines; a general agent service center connected to the one or more virtual machines, used for performing system scheduling between multiple virtual machines and cloud operating system data distribution; and the computer-readable storage medium.

[0021] Therefore, the present invention provides a complete cross-platform cloud operating system solution that is compatible with various smart terminals, provides a unified user application ecosystem, and offers a rich user experience and simplified program development. The present invention also provides various cloud operating system-based methods to achieve stable and efficient interface interaction and data transmission between local and remote users. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 shows a schematic diagram of the architecture of a cloud operating system according to the present invention in some embodiments.

[0023] FIG2 illustrates the association between the interface generation unit of the cloud operating system and various local terminals in an embodiment.

[0024] FIG3 illustrates the interaction between the interface generation unit of the cloud operating system and the local terminal in an embodiment.

[0025] FIG4 shows components of a cloud control desktop of a local terminal and image interaction associations of a cloud operating system in an embodiment.

[0026] FIG5 shows a solution for generating and laying out interfaces of a cloud operating system corresponding to various types of terminals in an embodiment.

[0027] FIG6 shows that a cloud-based operating system deployed on a cloud server generates various interfaces corresponding to various local terminals in an embodiment.

[0028] FIG7 illustrates universal driver redirection of a cloud operating system in an embodiment.

[0029] FIG8 shows an example of driver redirection of a cloud operating system in an embodiment.

[0030] FIG9 illustrates a transmission method between a local terminal and a cloud server in an embodiment.

[0031] FIG10 shows details of the transmission method between the local terminal and the cloud server in an embodiment.

[0032] FIG11 shows details of a method for transmitting positioning data between a local terminal and a cloud server in an embodiment.

[0033] FIG12 illustrates a method for transmitting user application data between a local terminal and a cloud server in an embodiment.

[0034] FIG13 illustrates a method for redirecting transmission of multimedia data between a local terminal and a cloud server in an embodiment.

[0035] FIG. 14 illustrates a process of listing a user application in an embodiment.

[0036] FIG. 15 shows an example of listing a user application in the embodiment.

[0037] FIG16 shows the process of a method for listing and installing a user application in an embodiment.

[0038] FIG17 illustrates a method for integrating functions of user applications in an embodiment.

[0039] FIG18 illustrates a remote input method between a local terminal and a cloud server in an embodiment.

[0040] FIG19 illustrates the interface switching of a user application based on a cloud operating system in an embodiment.

[0041] FIG. 20 illustrates opening and switching between multiple user applications based on a cloud operating system in an embodiment.

[0042] FIG21 illustrates a remote user application triggering a call to a local application based on a cloud operating system in an embodiment. Implementation Method

[0043] The following, in conjunction with Figures 1 to 21, describes in detail the terminals, devices, systems, architectures, methods, processes and various software and hardware parts involved in the entire technical solution of the present invention in multiple exemplary embodiments.

[0044] 1. Cloud operating system deployment and architecture

[0045] 1 , the cloud operating system according to the present invention is not deployed on a local terminal but is set up on a remote computing device. The remote computing device is associated with the local terminal via a network and uses the local terminal as a user operation terminal.

[0046] 1.1 Local Terminal and Remote Computing Device

[0047] In some embodiments, the local terminal has a display screen, network communication hardware, and a local operating system. Specifically, the local terminal may include a smartphone, tablet computer, laptop computer, game console, car control unit, smart home appliance, or other internet-connected computing terminal with a display screen, running a cross-platform operating system (e.g., Android, iOS, Windows, Linux). In some embodiments, the remote computing device may include any one or more of a cloud computing server (referred to as a cloud server), a virtual machine, a container, and a personal computer. Preferably, the cloud operating system according to the present invention is deployed on a cloud server, which generates at least one virtual machine for interacting with the user's local terminal. Furthermore, in some embodiments, at least a portion of the cloud operating system may be deployed in a virtual machine or container. The following description of the technical solution of the present invention primarily assumes that the remote computing device is implemented as a cloud server and a virtual machine. It should be understood that within the technical solution of the present invention, the virtual machine can be replaced at any time by a container or a physical computing device to implement the solution of the present invention.

[0048] Cloud servers, based on cloud cluster architecture, can include infrastructure, computing servers, schedulers, automation components, network managers, and more. Cloud cluster architecture can support multiple servers with distributed redundancy. Therefore, cloud servers can be integrated from various cloud hardware resources to provide a distributed physical operating environment for multiple cloud operating systems.

[0049] Specifically, infrastructure refers to the hardware and software infrastructure of cloud servers, including computer hardware (e.g., CPU, memory, hard drives, graphics cards, switches, etc.), the underlying cloud computing operating system, databases, and other software components. Computing servers integrate resources through virtualization and clustering technologies, and the cloud control platform generates relevant host resources on demand. Specifically, virtualization technology decomposes computing and storage resources into small, independent virtual machines, allowing them to operate and be managed independently. Computing servers typically have cloud computing software installed and are preferably also deployed with a general agent service center. Automation components automate repetitive tasks such as load balancing, backup, and recovery. The network manager provides cloud computing network services, connecting cloud servers to local terminals and can be mobilized by the general agent service center to manage all computing servers, shared storage, and networks within the entire cloud cluster architecture. Preferably, the cloud server provides a stable, highly secure user database with backup and synchronization mechanisms. This database allocates data storage space for each user account and can be mobilized by the general agent service center.

[0050] The cloud server and the local terminal are connected to a network via a communication device. The networking method used can be a wired connection, a wireless connection, or any combination thereof. Specifically, a communication network such as mobile communication (preferably using 4G, 5G, or next-generation mobile communication technologies), a wide area network, or Ethernet can be used between the local terminal and the cloud server. In one embodiment, the technical solution of the present invention may relate to a network switching method for a cloud operating system, comprising: triggering a local terminal to switch a communication network or a communication address of a cloud server based on signal strength and / or communication delay. In one embodiment, the local terminal can be triggered to monitor the signal strength of a currently connected first communication network (such as a 5G network) through a running scenario, user instructions, or a timer. The communication delay of the first network address of the cloud server can also be obtained by running a ping command in the background of the local terminal. When the signal strength falls below a signal strength threshold (the threshold value is between -100dBm and -75dBm) or the communication delay exceeds a delay threshold (the threshold value is between 100ms and 200ms), the local terminal can switch to a second communication network with a lower network frequency (such as a 4G network) or switch to the second network address of the cloud server.

[0051] 1.2 Cloud Control Desktop

[0052] In some embodiments, the local terminal can include and run various types of local operating systems (OSs) (e.g., Android, iOS, Windows, Linux, etc.). The local OS can switch between basic mode and cloud computing mode. In basic mode, only basic communication functions (e.g., phone calls, text messaging) and local basic applications (e.g., browsers, cameras, audio and video players) can be run. In cloud mode, the local terminal can interact with the cloud OS deployed in the cloud server via a high-speed network based on a cloud operating desktop, allowing users to experience rich software and hardware resources and user applications (UAPPs).

[0053] In some embodiments, the cloud operation desktop includes: an interface control or plug-in that provides a user operation interface and functional extensions associated with the interface based on a browser unit; a local access unit that is connected to the interface control or plug-in and is used to capture the user's real-time operation data and information and hardware information of local applications accessing the local terminal; one or more service units that are connected to the interface control or plug-in and are used for various interactive functions and method flows between the local and remote terminals; one or more client units that are respectively connected to the service unit and the local access unit and are used to communicate and transmit with the proxy service program in the remote computing device. The service unit and the client unit can be implemented in the form of subroutines, program modules, etc.

[0054] The implementation forms of cloud-controlled desktops include independent APPs (such as native APPs, system-level built-in APPs, or independent APPs that need to be installed), Web APPs (such as JavaScript (JS) applications based on the HTML5 framework), or plug-ins in applications based on the local OS (such as browsers), etc.

[0055] Referring to Figures 2 and 3, in one embodiment, the cloud control desktop is implemented as a native app or a hybrid app. The cloud control desktop may include a browser unit, a local access unit, one or more service subroutines, and one or more client subroutines. The browser unit may be a (web) browser framework or browser subroutine integrated into the cloud control desktop app, which includes various browser controls and plug-ins. The service subroutines run optimized local-remote interaction algorithms. For example, the image interaction subroutines within multiple service subroutines interface with the interface controls within the browser unit to enable interface image interaction. This is used to filter, slice, compress, compare, integrate, or perform parallel processing on images transmitted between the local terminal and the cloud server virtual machine. It should be understood that in this embodiment, other service subroutines are also used to implement various local-remote control interactions, interface image transmission and display, data redirection, dynamic data adjustment, and other methods. The local access unit may be implemented as a subroutine for invoking the local application programming interface (API) based on the local operating system. The local access unit can also connect to a local management unit outside the cloud control desktop to obtain local data. The local management unit can be implemented as a background service program on the local terminal, which continuously transmits local data to the device monitoring program running in the background of the cloud server. The client subroutine connects to the local access unit to read the transmitted data from the local terminal hardware. The communication method implemented in the client subroutine allows the cloud control desktop to establish network communication with the service program of the cloud server, and then reads the communication data received from the network hardware to the cloud server through the local access control.

[0056] Referring to Figure 4 , in one embodiment, the cloud control desktop is implemented as a web app, plug-in, or the like running in a browser (framework) based on a local terminal. The cloud control desktop in the browser framework also includes interface controls for graphical interaction with the remote interface of the virtual machine. Furthermore, the cloud control desktop may include an image interaction service subroutine for slicing, compressing, comparing, integrating, or parallel processing interface images transmitted between the local terminal and the cloud server virtual machine. Furthermore, in another embodiment, other service subroutines can be inserted into the browser application via other plug-ins to interface with the cloud control desktop in the browser application. The script in the service subroutine is interpreted by a JS engine (e.g., V8, WebKit, etc.) in the local browser framework to call local APIs or web APIs from a web server, thereby implementing various local and remote control interactions, interface image transmission and display, data redirection, dynamic data adjustment, and other methods.

[0057] It will be understood that the APIs mentioned above and below may define parameters passed between a calling application and other software code (e.g., an operating system, a library program, a function) that provides a service, provides data, or performs an operation or calculation. The API may be implemented as one or more calls in program code that send or receive one or more parameters through a parameter list or other structure based on a calling convention defined in the API specification document. A parameter may be a constant, a key, a data structure, an object, an object class, a variable, a data type, a pointer, an array, a list, or another call. API calls and parameters may be implemented in any programming language. A programming language may define the vocabulary and calling convention that a programmer will use to access functionality that supports the API.

[0058] 1.3 Basic Architecture of Cloud Operating System

[0059] 1 , in some embodiments, a cloud operating system according to the present invention includes a basic kernel, a runtime library, an interface service module, and an application service module.

[0060] The basic kernel includes process management, drivers, memory management, virtual file systems, etc., to call the physical and / or virtualized hardware resources of the cloud server according to the request. The basic kernel of the cloud operating system is composed of the kernel of the underlying operating system of the cloud server, preferably the Linux kernel.

[0061] The runtime library is based on the basic kernel to provide a runtime environment for applications on various operating system platforms. In one embodiment, the runtime library is a computer program library that can be used by compilers of programming languages ​​on various operating system platforms to implement built-in functions of the programming language to provide runtime (or execution) support for programs in the programming language. Preferably, in addition to the default Linux runtime library, the cloud operating system according to the present invention can also install and provide runtime libraries for Android, Windows, or macOS, so that Android, Windows, or macOS applications can be run in the cloud operating system, especially browser frameworks of various platforms and kernels (such as Safari, Edge, Firefox, Chrome, open source Chromium, etc.).

[0062] The interface service module corresponds to the runtime library and provides an application programming interface (API). The interface service module may also include an interpreter, engine (such as V8, WebKit, etc.), or runtime module (such as Node.js or a customized, streamlined JS runtime) for interpreting and executing UAPP code (such as JS script code). Furthermore, the interface service module may also provide a Web API for accessing network services, resources, or functions from a web server, as shown in Figure 12.

[0063] The application service module provides control interface services and UAPP services to local terminals, and facilitates the connection between these services locally and remotely. The application service module includes: a communication unit, a device monitoring unit, an interface generation unit, a cloud control connection unit, and a driver connection unit. These units can be implemented as background service programs, subroutines, program modules, etc.

[0064] The communication unit is used for communication between the local terminal and the virtual machine of the cloud server, as well as between multiple devices (or virtual machines) associated with the local terminal in the cloud server. The data transmission method will be described in detail in Section 4 below using multiple embodiments.

[0065] The device monitoring unit obtains the hardware parameters and real-time operating status data of the local terminal. Referring to Figure 3 , in one embodiment, the device monitoring unit can be implemented as a device monitoring program serving as a backend service on a cloud server. After establishing an authenticated connection between the local terminal and the remote computing device, the device monitoring unit obtains the local terminal's hardware configuration parameters (network type, memory size, storage capacity, processor type, multimedia device type, etc.), local OS type (e.g., iOS, Android, Windows, Linux, lightweight IoT operating system, etc.) and version, screen size, screen resolution, display refresh rate, and input device type (e.g., touch, mouse, keyboard, recording device, operator, etc.). The device monitoring unit also receives the current status and data of local applications (including local OS built-in programs and local applications), such as network communication quality, local performance, local application action data, geolocation data, user command data, and so on.

[0066] Referring to Figures 3, 5, and 6, the interface generation unit generates a UAPP's operating interface based on the hardware parameters of the local terminal. In some embodiments, the interface generation unit generates an interface environment for presenting the UAPP based on the interface style and preset operating habits corresponding to the user account, and based on the hardware configuration parameters obtained by the device monitoring unit. The UAPP's operating interface is captured by the cloud server's virtual machine background service program and then transmitted to the local terminal's cloud control desktop. The interface generation unit can generate the UAPP's operating interface based on the interface manager in the runtime library. Preferably, the interface generation unit runs in a browser framework and generates the UAPP's operating interface through the browser's renderer and interpreter (or engine). The interface generation method will be described in detail in Section 3 below using multiple embodiments.

[0067] The cloud control docking unit is called by the background service program to receive user control data (such as operation data of touch screen, mouse, keyboard and other devices) from the local terminal in real time, and convert it into control data of the virtual machine in the cloud server.

[0068] The driver docking unit responds to a hardware resource request from a UAPP, redirecting source data for hardware execution from a remote computing device to the hardware of a local terminal. Referring to FIG7 , in some embodiments, the present invention relates to a driver redirection method, comprising the following steps: responding to a hardware resource request initiated by a UAPP via an API, invoking a first data conversion program to generate source data for caching in a memory area; the driver docking unit captures the source data from the memory area of ​​the remote computing device, the source data being the data that the driver triggers the remote computing device hardware to receive from the memory area (e.g., a bitmap to be read by a graphics card driver, a standard digital audio file (WAV) to be read by a sound card driver); and then forwarding the source data (compressing and decompressing it) to the local terminal via a remote-side data forwarding unit; responding to instructions from the driver docking unit, a service unit in the backend of the local terminal initiates hardware resource docking via an API, invoking a second data conversion program, receiving (and decompressing) the source data via the local-side data forwarding unit, replacing it with output data of the second data conversion program, and triggering the local hardware to execute the source data via the local driver. Preferably, the first and second data conversion programs are programs of the same type and function. Similarly, a method for redirecting a driver from a local terminal to a remote computing device may include the following steps: responding to a hardware resource request initiated by a UAPP via an API, causing the driver to trigger the remote computing device hardware to collect blank source data, calling a third data conversion program to generate a blank file from the source data for caching in a memory area; responding to an instruction from a driver docking unit, causing the driver of the local terminal to synchronously trigger the local hardware to collect source data, calling a fourth data conversion program to generate a target file from the source data, to be forwarded to the remote computing device by a data forwarding unit, and replacing the blank file with the target file; causing the UAPP to obtain the target file. Preferably, the third and fourth data conversion programs are programs of the same type and function.

[0069] Referring to Figure 8, in a specific embodiment, an audio application (Voice APP) is run in the virtual machine of the cloud server, and the local terminal displays the interface of the Voice APP to allow the user to remotely operate the Voice APP. When the user of the local terminal interactively triggers the Voice APP to play the audio file (such as an mp3 file) in the storage unit, the Voice APP calls the API to trigger the audio decoder to decode the obtained wav audio stream and temporarily stores it in the memory area. The driver originally used for the virtual machine calls the virtual machine hardware to play, which will actually be forwarded to the local terminal. Because the driver docking unit synchronously sends instructions to call the service unit in the background of the local terminal to trigger the local audio decoder to work through the API, the local audio decoder does not actually decode the wav audio stream, but redirects the remote wav audio stream data to the output result of the local audio decoder through the data forwarding unit on the local side, so that the driver of the local OS triggers the hardware to play. Continuing to refer to Figure 8, in another specific embodiment, when the user interactively triggers the Voice APP to perform local recording,

[0070] The Voice APP initiates a hardware resource request through the API, causing the driver to trigger the virtual machine's hardware to record (in fact, the recording obtains a blank audio stream), and calls the encoder in the virtual machine to generate a blank file from the source data to be cached in the memory area; in response to the instruction of the driver docking unit, the local terminal's driver synchronously triggers the local Mic hardware recording through the local API to obtain a wav audio stream, calls the encoder to generate a compressed audio file (such as an mp3 file), which is forwarded to the remote computing device by the data forwarding unit and replaces the blank file; then the Voice APP obtains the target audio file and completes the storage of the recording file.

[0071] In some embodiments, driver docking can also be achieved through interaction between local and remote browser frames, which will be described in detail in part 2 below using an embodiment.

[0072] It can be understood that based on the cross-platform system environment built on the network between the local terminal and the remote computing device, the cloud operating system of the present invention simplifies and adjusts the application framework layer in the traditional operating system architecture into an interface service module, and customizes the application service module, thereby realizing the interaction of control, hardware resources and interfaces between the local terminal and the remote computing device, improving program operation efficiency and ensuring the user's remote operation experience.

[0073] In these embodiments, the cloud operating system of the present invention is characterized by deploying at least a portion of the cloud operating system on one or more of a cloud server, a virtual machine, and a container. For example, the cloud operating system's basic kernel is provided by the operating system kernels of the cloud server and virtual machine, interface service modules and runtime libraries can be installed and provided in the virtual machine, and application service templates can be deployed in the virtual machine and / or container. Containers (or cloud service containers) and virtual machines are both based on virtualization technology, making them easy to create and migrate. Virtual machines create a complete native operating system and hardware environment by virtualizing hardware, while containers create an isolated operating environment on a computing device (which can also be a virtual machine). Without the need for hardware emulation, containers are more lightweight, have faster startup times, and consume fewer resources, making them more suitable for deploying lightweight applications. Therefore, the cloud operating system of the present invention can leverage the vast computing power of servers to provide each user with a cross-platform, multi-device environment to run UAPPs, making the UAPP development process more flexible, efficient, and unified.

[0074] In these embodiments, the cloud operating system of the present invention is further characterized by transmitting interface images from a remote computing device to a local terminal for the user to control the UAPP on the local terminal. Compared to transmitting interface screen recordings, transmitting interface images saves network bandwidth and does not require the local terminal to expend computing power for video decoding. In addition, the high-definition image compression algorithm is mature, so when using the cloud operating system, users can obtain a clear interface visual experience while maintaining a smooth control feedback experience.

[0075] 2. Browser Framework

[0076] Referring to Figure 9 , based on the cloud operating system, a remote browser framework (or web browser framework) can be provided for each virtual machine generated in the cloud server. Accordingly, the local OS on the local terminal can come with its own local browser framework to interact with the remote browser framework. Whether remote or local, the browser framework's infrastructure can include a browser interface, a browser kernel, and a backend service layer. The browser interface is used to display pages or windows. The browser kernel can include a browser engine and a rendering engine for presenting pages. The browser engine can be V8, WebKit, etc., and the rendering engine can include an HTML interpreter, a CSS interpreter, and a layout. Generally, the kernel and rendering engine of the Safari browser are WebKit; while the kernel of the Chrome / Chromium / Edge browser is Chromium, and the rendering engine is Blink (a branch of WebKit). The backend service layer is used to connect to the network, data storage, or call native services through APIs.

[0077] Preferably, referring to Figure 10 , the backend service layer can include a backend runtime module (e.g., a JS runtime) that supports running JS outside of the browser. For example, the V8 engine can be used to run JS scripts outside the browser to implement functions such as the file system, modules, system APIs, network communications, and regular applications. The backend runtime module can integrate with existing JS runtime environment tools (e.g., Node.js) or integrate a customized lightweight JS runtime that retains specific functionality. In one embodiment, the backend runtime module can include a JS engine, native modules, and a standard library, and can directly call native APIs.

[0078] 2.1 Local Browser

[0079] Generally, the browser built into the local terminal of most intelligent platforms can be configured to cooperate with the cloud operating system. If the local browser does not have a suitable browser engine, an additional toolkit can be installed to implement a local operating environment based on a specific browser engine, thereby configuring a local browser that cooperates with the cloud operating system of the present invention.

[0080] In some embodiments, the local browser that cooperates with the cloud operating system of the present invention may include: a client unit, which is used for the local browser to communicate with other applications running on the local operating system and to communicate with the remote computing device to obtain data from the local terminal and the remote computing device; a display component, which is used to display the interface image from the remote computing device; and an image interaction service unit connected to the display component, which is used to implement the interface image transmission and interaction method. Specifically, the display component may include a WebView control that is compatible with local systems and browsers such as Linux, Android, Pinggu and Google. The local browser framework provides a back-end runtime module that supports the operation of the client unit and the image interaction service unit. In addition, it can also run various subroutines and program modules written in JS scripts of the cloud control desktop.

[0081] 2.2 Remote Browser

[0082] In some embodiments, because the cloud operating system of the present invention can provide multiple runtime libraries (such as Android and Mac), various browsers (such as Chrome / Chromium and Safari) can be built into the virtual machine generated by the cloud server to match the kernel type of the local browser (such as Chromium and WebKit), thereby achieving better compatibility and UAPP operation efficiency. In addition, because the Chromium kernel and the WebKit kernel are homologous, a Linux version of the browser based on the Chromium kernel can be built into the virtual machine generated by the cloud server by default, so that the browser (framework) configured in the virtual machine can match the Linux base kernel and default runtime library of the cloud operating system.

[0083] In some embodiments, the remote browser may include: a client unit, configured to communicate with other applications running on the cloud operating system and with a local terminal to obtain data from the local terminal and the remote computing device; an interface image acquisition unit, configured to acquire at least a portion of an interface image from the user interface of a web-based UAPP running on the remote browser; wherein the client unit and the interface image acquisition unit are respectively connected to an image interaction service program in the cloud operating system, and the image interaction service program implements the interface image transmission and interaction method. The remote browser framework provides a backend runtime module to support the operation of the client unit, the interface image acquisition unit, and the image interaction program.

[0084] 2.3 Application Service Interconnection Based on Browser Framework

[0085] In a preferred embodiment, when the kernels of the remote and local browser frameworks are identical or matching, and the same backend runtime modules are configured and used, the cloud operating system's application service docking and driver redirection become flexible and simple. For example, a script that calls an API in a UAPP running in the remote browser framework can also reuse code in the local browser to synchronously call local APIs of the same type or function, greatly facilitating the remote UAPP's access to local terminal resources.

[0086] Referring to Figures 7 and 8, the method for implementing API and driver redirection through remote and local browser frameworks is described, using the process of a UAPP calling a local terminal's hardware resources as an example. First, the UAPP running in the virtual machine's remote browser framework generates a page for display on the browser interface by its rendering engine. This page, as the UAPP's running interface, is captured by the interface image acquisition unit and sent to the local browser, where it is displayed as a local interface. The user then transmits control data based on the local interface to the virtual machine, operating the UAPP in a remote interactive manner. Because the remote browser is configured with a backend runtime module, when responding to the UAPP's command script and initiating a request for hardware resources outside the browser (for example, audio and video decoding and playback, camera activation, etc.) through the virtual machine's local API, the driver docking unit triggers the local browser's backend runtime module with the same command script, initiating a local hardware resource request using the same API. The data originally returned by the virtual machine's hardware resources is then replaced by data generated by the local hardware resources and returned to the UAPP, thereby achieving the effect of the UAPP calling the local terminal's hardware resources.

[0087] 3. Interface Methods

[0088] 3.1 Interface Generation and Layout

[0089] 2 , in some instances, the remote interface generation method may include: establishing a network association between the cloud server and each local terminal; obtaining a user account and hardware configuration parameters from each local terminal, the hardware configuration parameters including the local operating system type, screen size, screen resolution, and input device type, etc.; generating an interface window in the browser in the cloud server with the obtained screen size or screen resolution of each local terminal, thereby serving as the interface environment of the UAPP; capturing the interface image of the interface window and transmitting it to the local terminal as an interface image display. In this embodiment, each virtual machine of the cloud server can generate an interface for the local terminal corresponding to each user account. In addition, referring to FIG2 and FIG6 , the corresponding operating system or runtime virtual machine can be assigned according to the type of the local terminal. For example, the local terminal corresponding to the Android system can be assigned an Android or Linux operating system or runtime virtual machine, so that the interface generation unit generates the interface style of the local terminal by default.

[0090] Referring to Figure 3 , in some instances, local hardware configuration parameters can be obtained via a local management unit connected to the cloud control desktop. The local management unit can be implemented as a background service program on the local terminal that continuously transmits the local hardware configuration parameters to a device monitoring program running in the background of the cloud server. Furthermore, the renderer in the browser frame of the remote computing device is equipped with an HTML interpreter and a CSS interpreter. Using a JS engine, the UAPP generates the running interface from a web script file (as shown in Figures 12 and 13) cached locally or received from a web server. Specifically, the aspect ratio of the interface window presented by the display control matches the aspect ratio of the local terminal's screen, while the DPI value of the interface window is greater than or equal to the DPI of the local terminal's screen, allowing the cloud server's higher computing power to generate high-definition images or decode and play high-definition videos for full display in the interface window. A screenshot plug-in or subroutine running in the browser frame of the remote computing device can then convert the running interface into an interface image (which may be further compressed based on network transmission requirements). The local terminal can provide a browser frame to allow a browser display control (e.g., WebView) to receive the interface image for display on the local terminal's screen. Preferably, a video playback control can also be provided. When the interface of the cloud server's virtual machine is detected to have a video playback window, the interface image to be transmitted from the cloud server is adjusted to retain only the image of the video playback window, and the video stream can be redirected to the video control window of the local interface for playback. Therefore, after the remote interface is generated, the local terminal can receive the size and position data of the video playback window, so that the local video control can be arranged and overlaid on the received interface image according to the size and position.

[0091] 5 and 6 , the remote interface generation method may include: obtaining the interface style and operating habits corresponding to the user account or local operating system type from the user database of the cloud server, generating a browser window in the browser frame of the cloud server virtual machine using the obtained screen size or screen resolution of the local terminal, configuring a renderer to generate an interface layout based on the interface style, and determining the operating gestures of the browser window based on the operating habits, thereby serving as the interface environment of the UAPP running based on the browser frame. In a specific embodiment, a user may have multiple local terminals, such as smartphones, smart cars, wearable devices, and portable PCs, and multiple virtual machines may be used to generate browser windows using the hardware configuration parameters corresponding to each local terminal of the user, while simultaneously using the same set of CSS standard layout interfaces to facilitate the presentation of various local terminal style interfaces in a separate virtual machine running environment using only the script code of the same UAPP. For example, for the same map app, a small-sized interface that meets the vertical screen of a mobile phone can be generated in one virtual machine, while a large-sized interface that meets the needs of in-car navigation can be generated in another virtual machine. Furthermore, referring to Figure 6 , local terminal-style interfaces (GUI2 and GUI3 in Figure 6 ) can be generated in multiple windows of the browser frame of the same virtual machine (such as VM2 in Figure 6 ). This allows the same user account to quickly access the desired local terminal-style interface after logging into the virtual machine. This provides an excellent user experience for scenarios where users need to switch between different terminals while still using the same app. For example, a user can use a wearable device to receive GUI3 and operate a map app (actually running in VM2), setting a navigation destination in advance via voice. The map app then switches to the in-car navigation map interface GUI2 and sends it to the user's logged-in smart car. This allows the car's display to directly display the map app's navigation interface information for the destination when the user enters the smart car. Therefore, when developing a UAPP, the CSS-based page code that implements the UAPP interface generation method is separated from the script code that implements the UAPP functionality.

[0092] 3.2 Interface Image Transmission

[0093] 4 , in some embodiments, the interface image transmission method for a cloud operating system may include the following steps:

[0094] S001. Establishing a control data interaction connection between the local terminal and the virtual machine of the cloud server;

[0095] S002. Maintaining the transmission connection of the interface image of the virtual machine to the local terminal for at least one period of time;

[0096] S003. Dynamically adjust the interface image to be transmitted by the cloud server according to the transmission interval of the interface image transmitted by the cloud server to the local terminal.

[0097] In step S001, control data transmitted by the interactively connected local terminal causes changes to at least a portion of the cloud server's interface. This control data can be data generated by local terminal input devices (e.g., touchscreen, mouse, keyboard, etc.) causing local control actions such as clicks, moves, and zooms. After the local control data is transmitted to the cloud server, it is distributed to the virtual machine connected to the local terminal, thereby establishing interface interaction between the local terminal's control actions and the virtual machine. Controls performed by the local terminal can cause changes to the entire virtual machine interface, partial interfaces, or background activities within the interface.

[0098] Regarding step S002, based on the transmission connection between the local terminal and the virtual machine of the cloud server, each time the control data generated by the local terminal is sent to the virtual machine, if it causes the interface of the virtual machine to change, the cloud server will transmit the interface image of that time to the local terminal. In other words, the more frequently the virtual machine receives control data, the more images are generally transmitted to the local terminal. Preferably, the transmission of interface images between the local terminal and the cloud server can be based on a long link network protocol to be compatible with high-frequency image transmission.

[0099] For step S003, the dynamic adjustment of the interface image to be transmitted by the cloud server includes: within a certain period of time, if the interval between the cloud server transmitting the image to the local terminal is too short (or the image transmission is too frequent), in the next period of time, implementing any one or more of the following sub-steps: reducing the number of image transmissions to the local terminal; reducing the quality of the entire or part of the interface image by image compression; transmitting the dynamically changing part of the interface image.

[0100] Regarding step S003, in a specific embodiment, when it is detected that the virtual machine's interface has a video playback window, the interface image to be transmitted to the cloud server is adjusted to retain only the image of the video playback window. This is because the video stream can be redirected to the video control window of the local interface for playback. Preferably, if the virtual machine's interface is playing a full-screen video, the transmission of the interface image to the local terminal is suspended, and the video stream is directly transmitted to the local terminal for playback.

[0101] The following describes the method flow and interactive steps involved in the interface image transmission of the local and remote sides of the image transmission method for the cloud operating system through more detailed embodiments.

[0102] 3.2.1 Local side method flow

[0103] The image transmission method for a cloud operating system may include the following steps S110 to S119 on the local terminal side.

[0104] S110: Launching a cloud control desktop program on the local terminal and simultaneously activating the local control process to capture control data from the input device in real time and transmit it to the remote computer device (the virtual machine on the cloud server). Preferably, the cloud control desktop program can access the user information database on the cloud server, verify the user's login information (e.g., account number, password, fingerprint data, facial recognition data, etc.), log in to the user's virtual machine on the cloud server, and establish secure, encrypted control data transmission from the local terminal to the remote terminal.

[0105] S120. In the background program of the local terminal, run a data transmission process and activate a dedicated channel for image data transmission to allow reception of interface image data transmitted by the cloud server. The image data transmission channel and the control data transmission channel are independent of each other to ensure that the image data transmission is not interrupted by the network connection while the control data is continuously transmitted. Preferably, the transmission frequency of the image data transmission channel is configured to be lower than or equal to the transmission frequency of the control data transmission channel.

[0106] S130 : Continuously receiving the interface image of the virtual machine from the cloud server during a preset period of time through the long link between the local terminal and the cloud server.

[0107] S140 , determining whether the frequency of received images is too fast (for example, whether the time interval Δtr between two received image data is less than a preset threshold Tr), if so, executing step S150 , otherwise, jumping to executing step S160 .

[0108] S150. According to the status of the local terminal, a portion of the interface images received from the cloud server is filtered or the received interface images are filtered for use in the interface update and display of the local terminal. Preferably, the number of filtered interface images received can also be adjusted based on the status of the local terminal, such as the power, network status, and application scenario. For example, when the local terminal is running in energy-saving mode or the workload of the baseband chip exceeds a preset value, the transmission frequency of the image data transmission channel is reduced, thereby reducing power consumption and heat generation. For example, when the network bandwidth connected to the local terminal is low or in a standby scenario, a portion of the interface images are selected at intervals for display at longer time intervals. In other instances, when the sending frequency of the local control data is monitored to be lower than the threshold ft, the received interface images are selected at intervals; if it is monitored that no local control data is sent, but the frequency of the received images is too fast, the received interface images are selected at longer intervals for the interface update of the local terminal.

[0109] S160: Update the display interface of the local terminal with the selected interface image. Preferably, the local terminal display interface can be updated locally or globally by any of the following methods: timing; switching of the local terminal application scenario (e.g., web browsing, office work, gaming mode, etc.); change of the local terminal system status; or receipt of a user-specified refresh operation instruction.

[0110] S170 , determining whether the local terminal generates control data for the remote virtual machine (for example, whether the user's control of the cloud-controlled desktop has ended). If so, executing the next step S180 ; otherwise, returning to executing step S130 .

[0111] S180 , determining whether the image data transmission between the local terminal and the remote server is completed (for example, whether the transmission channel is closed); if so, executing the next step S190 ; otherwise, returning to executing step S130 .

[0112] S190: Synchronize the current control data to the virtual machine of the cloud server, and synchronize the global interface image of the virtual machine to the interface of the local terminal. Finally, end this process.

[0113] 3.2.2 Remote side method flow

[0114] The image transmission method for a cloud operating system may include the following steps S210 to S290 on the remote cloud server side.

[0115] S210: Obtain the user login data of the local terminal through the cloud server's transmission process, activate the user's remote computer device (a virtual machine generated in the cloud server), and connect the virtual machine to the local terminal's control data. Preferably, the cloud server may allocate multiple virtual machines to the user, and the control data transmission channel of each currently used virtual machine is associated with a specific control data transmission process of the local terminal's cloud control desktop, allowing the cloud control desktop to switch and control various virtual machines of the user in the cloud server.

[0116] S220: Run an image interaction service program in the cloud server or in the virtual machine, and activate a dedicated channel for image data transmission to allow transmission of the interface image data of the virtual machine to the local terminal.

[0117] S230: Calling a screen capture process of the virtual machine to continuously obtain an interface image of the virtual machine.

[0118] S240 , determining whether the frequency of the images sent is too fast (for example, if the time interval Δts between two received image data is less than a preset threshold Ts), if so, executing step S250 , otherwise, jumping to executing step S260 .

[0119] S250: Dynamically adjust the quality of the obtained interface image of the virtual machine.

[0120] For steps S240 to S250, in some embodiments, dynamically adjusting the interface image to be transmitted from the cloud server to the local terminal includes the following steps: detecting the current transmission interval Δts of the interface image in the same interface range from the cloud server to the local terminal; if it is determined that the current transmission interval Δts is less than the first time threshold Ts1, the interface image to be transmitted is abandoned at that time; if it is determined that the current transmission interval is lower than the second time threshold Ts2, the quality of the interface image to be transmitted is reduced (for example, by further compressing the image to reduce the quality of at least a part of the interface image, or the resolution of the interface image can be reduced); wherein, the first time threshold Ts1 is less than the second time threshold Ts2.

[0121] In some instances, the first time threshold Ts1 can be a relatively small value, such as 20ms, while the second time threshold Ts2 can be configured to multiple larger thresholds (e.g., 40ms, 60ms, and 80ms) to gradually reduce the interface image quality (or increase the compression rate / reduce the resolution). For example, when the transmission interval is less than 40ms, the image quality is compressed to only 30% of the original file size; when the transmission interval is less than 60ms, the image quality is compressed to only 60% of the original file size; and when the transmission interval is less than 80ms, the image quality is compressed to only 80% of the original file size.

[0122] In other embodiments, dynamically adjusting the interface images to be transmitted from the cloud server may further include the following steps: detecting the current transmission interval of interface images within the same interface range from the cloud server to the local terminal; and if the current transmission interval Δts is determined to be greater than a third time threshold Ts3, restoring the quality of the interface images to be transmitted (e.g., reducing the image compression rate or converting them into lossless compressed images), or acquiring interface images of higher image quality from the cloud server for transmission to the terminal. The third time threshold Ts3 is generally greater than the second time threshold Ts2. Preferably, the third time threshold Ts3 may be 100 ms or greater.

[0123] In other embodiments, dynamic adjustment of the interface image to be transmitted by the cloud server may also include the following steps: detecting the current transmission interval Δts of the overall interface image from the cloud server to the terminal; determining that the current transmission interval Δts is lower than a preset threshold Ts0, then reducing the resolution of the interface image to be transmitted or adjusting it to a grayscale image.

[0124] Preferably, the interface image transmitted from the cloud server to the local terminal can be pre-adjusted based on the current network transmission status between the local terminal and the cloud server, the application scenario (e.g., web browsing, office work, gaming mode, etc.), or user instructions. Such pre-adjustment may include adjusting image resolution, color channels, or grayscale images. For example, if the network transmission speed or bandwidth of the cloud server to the local terminal decreases significantly, the pre-adjustment step is performed directly, followed by step S250.

[0125] S260: Continuously transmit the screened interface images to the local terminal over a preset period of time via a persistent link between the local terminal and the cloud server. Preferably, a time stamp can be established between the manipulation data and the interface images, such that each time a manipulation from the local terminal triggers (captures) an interface image of the cloud server virtual machine (VM) at a certain time, the time stamp can be attached to the interface image received by the local terminal. This time stamp is then attached to the interface image received by the local terminal, ensuring that the manipulation action performed by the local terminal corresponds to the displayed interface image.

[0126] S270. Determine whether the remote virtual machine has received the control data from the local terminal, and whether the control process caused by the control data from the local terminal in the virtual machine is completed. If yes, execute the next step S280; otherwise, return to execute step S230.

[0127] S280 , determining whether the image data transmission between the local terminal and the remote server is completed (for example, whether the transmission channel is closed); if so, executing the next step S290 ; otherwise, returning to executing step S230 .

[0128] S290: Synchronize the current control data with the local terminal and synchronize the global interface image of the virtual machine to the interface of the local terminal. Finally, end this process.

[0129] 3.2.3 Manipulating Data and Image Screening

[0130] The control data from the local terminal affects the generation and transmission of the virtual machine's interface image. In some embodiments, the method may also include the following basic control-related steps: A. Based on the type of local control, when specific control, excessively fast control, or abnormal control occur, filtered or adjusted control data is retransmitted to the cloud server to prevent the cloud server from generating unnecessary interface images; or B. Based on the transmission interval of the local terminal's control data, multiple interface images received by the local terminal are dynamically filtered to display the filtered interface image or at least a portion of the image on the terminal. Basic step A directly affects the interface changes of the cloud server's virtual machine by filtering local control data, and can also directly affect the frequency of image transmission from the cloud server to the local terminal. Therefore, basic step A is an "active" step. Basic step B, which filters the received images, is a "passive" step. It can be understood that the image receiving module and the image transmission module can independently operate to filter images in both locally received images and remotely transmitted interface images, while coordinating the above-mentioned active steps A and B. This is the flexibility of the image transmission method of the present invention, which is compatible with image transmission optimization in various control-based application scenarios.

[0131] In some embodiments, basic step A can be combined with the manipulation process in step S110 of the above embodiment. In these embodiments, the basic step A can be implemented as the following steps S111 to S116.

[0132] S111: Activate a local control process to capture control data of an input device of a local terminal in real time.

[0133] S112: Capture and classify manipulation data during the manipulation triggering period. In some embodiments, the manipulation events may be classified into click, double click, single finger slide, multi-finger slide, fingerprint recognition action, and body physical button action.

[0134] S113: Determine whether a specific type of manipulation occurs, or whether excessively fast or abnormal manipulation occurs. If so, proceed to step S114; otherwise, proceed to step S115. In some embodiments, for each category, a specific type of manipulation may include a long, continuous click, a long-distance swipe (e.g., exceeding half the screen length), etc. Abnormal manipulation may include frequent back-and-forth touch dragging, simultaneous pressing of multiple buttons, etc.

[0135] S114. Filter or adjust the control data of the local terminal. In one embodiment, for the control events of frequent back-and-forth touch drag (as indicated by the dotted arrow in FIG7 ), the generated control data contains more coordinates of the path of the back-and-forth drag. Therefore, the control data can be filtered to retain the coordinates of the starting touch point and the ending touch point in the touch drag event to avoid repeated control data causing repeated generation of virtual machine interface images. In another embodiment, for the control events of long-distance sliding, only the coordinates of the sliding starting point and the sliding ending point can be retained and recorded in the control data. In addition, for abnormal control events, the control data generated by them can be directly shielded to avoid being sent to the remote server.

[0136] S115: Send control data to the virtual machine of the remote cloud server.

[0137] S116: Wait for a certain period of time to determine whether the next control event occurs. If so, return to step S112; otherwise, end this process.

[0138] In some embodiments, basic step B may be combined with the manipulation process in step S150 of the above embodiment. Dynamically screening the received multiple interface images according to the manipulation interval of the local terminal may include the following steps:

[0139] S151. Determine, within at least one time period, that a manipulation interval Δtc of the local terminal is less than a fourth time threshold Tc4 (preferably, the fourth time threshold Tc4 may be in a range of 10 ms to 20 ms depending on the manipulation type);

[0140] S152. Discard at least a portion of the interface images received from the remote computing device that are used to be displayed on the local terminal (the intermediate redundant images repeatedly generated due to frequent manipulation events of the local terminal, as shown in FIG7 ). Then, as needed, reduce the frequency of manipulation data sent by the local terminal to the cloud server virtual machine.

[0141] In some embodiments, dynamically filtering the received multiple interface images based on the local terminal's manipulation interval may further include the following steps: if, within at least one time period, the local terminal's manipulation interval Δtc is determined to be greater than a fifth time threshold Tc5 (corresponding to the frequency of normal user manipulation of the local terminal), then retaining all interface images for display on the local terminal from the received multiple interface images; and if, within at least one time period, the local terminal's manipulation interval Δtc is determined to be greater than a sixth time threshold Tc6 (corresponding to a situation where the user has not manipulated the local terminal for a long period of time), then intermittently selecting interface images for display on the local terminal from the received multiple interface images until the next manipulation of the local terminal occurs. This can achieve a local standby effect and reduce power consumption of the local terminal. The sixth time threshold Tc6 is greater than the fifth time threshold Tc5, and the fifth time threshold Tc5 is greater than the fourth time threshold Tc4. The fifth time threshold Tc5 can be greater than 200 ms, and the sixth time threshold Tc6 can be greater than several seconds.

[0142] In one embodiment, if it is determined that the operation interval Δtc of the local terminal is greater than a seventh time threshold Tc7 within at least one time period, the display frame rate of the local terminal is reduced until the next operation occurs, wherein the seventh time threshold Tc7 is greater than the fifth time threshold Tc5. This embodiment can be applied to reduce the display power consumption of the local terminal when the local terminal has not been operated for a long period of time (for example, in a reading scenario).

[0143] In some embodiments, the method may further include the following steps: determining that the movement amplitude (or distance) of the control point of the local terminal exceeds a preset amplitude threshold; retaining the interface images associated with the starting action and ending action of the control point among the multiple interface images transmitted to the local terminal by the cloud server due to the control data generated by the control point of the local terminal, and reducing the remaining interface images for display on the local terminal.

[0144] In addition, the user's manipulation frequency of the human-computer interaction device can also be monitored through the local subroutine of the local terminal. In a preferred embodiment: when the manipulation frequency is slow (for example, below a certain threshold), the image quality transmitted from the cloud server to the local terminal can be improved and / or the refresh rate of the local terminal's display (or display interface) can be reduced; when the manipulation frequency is fast (for example, above a certain threshold), the image quality transmitted from the cloud server to the local terminal can be reduced and / or the refresh rate of the local terminal's display (or display interface) can be increased; when the manipulation frequency is too fast (for example, above a certain threshold), the interface image of the last frame of the virtual machine in the excessively fast manipulation process can be transmitted to the local terminal, and then the remaining interface image data in the excessively fast manipulation process of the local terminal will not be transmitted, or these images will be compressed at a high compression rate (even lossy compression).

[0145] 3.2.4 Interaction between the remote virtual machine and the local terminal

[0146] Steps S110, S120, S130, S170, S180 and S190 on the local terminal side respectively have two-way data transmission and data exchange with steps S210, S220, S260, S270, S280 and S290 on the cloud server side.

[0147] In particular, there is a timed synchronization interaction between steps S160 and S260 to synchronize the interface images between the local terminal and the cloud server. This can avoid the interface display deviation between the two parties caused by the cloud server transmitting the filtered interface images to the local terminal.

[0148] In some embodiments, a hardware monitoring module running in the cloud server can determine the operating conditions of the cloud server's hardware (including the virtual hardware of the virtual machine) caused by the local terminal (e.g., network transmission quality, processor load, memory usage, etc.). For example, if the network transmission speed is detected to have decreased or the network latency has increased beyond a threshold, the compression rate of the image transmitted from the cloud server to the local terminal can be increased, or the image quality can be reduced.

[0149] 3.2.5 Other methods involving interface image transmission

[0150] In other embodiments, the method involving interface image transmission may further include the following steps to be implemented in a local terminal or a cloud server.

[0151] S351. Before the cloud server transmits the interface image to the local terminal, the interface resolution of the local terminal is synchronized with the interface display resolution of the cloud server. Preferably, a data buffer for transmitting at least a portion of the interface image is provided between the cloud server and the local terminal; based on historical transmission data, continuous data of at least a portion of the interface image is generated in the data buffer, and an application program that reads data from the data buffer continuously reads the interface image data.

[0152] S352: Determine whether a network problem occurs (for example, when the network transmission speed is detected to be reduced or the network delay is increased to exceed a threshold) or switch to high-speed image interaction mode. If yes, execute the next step S353.

[0153] S353: Determine whether an interface resolution adjustment instruction has been received. If yes, execute step S354; otherwise, execute step S355.

[0154] S354. Reduce the interface display resolution.

[0155] S355: Reduce the resolution of the transmitted interface image.

[0156] 4. Data Transfer

[0157] In addition to dedicated transmission channels for transmitting interface images and control data, the data transmission mechanism of the cloud operating system of the present invention also needs to handle the transmission of user files, local application data, call data, etc., which is described in detail in some embodiments below.

[0158] 4.1 Basic transmission method

[0159] The framework of the application layer communication protocol for direct transmission between the local terminal and the cloud server may include but is not limited to HTTP, Socket, Webtransport, etc. Preferably, communication protocols such as WebSocket and HTTP3.0 are used between the local terminal and the cloud server. The transmission between the local terminal and the cloud server can reuse the handshake channel of the HTTP protocol. In addition, multiplexing can be allowed in an extensible manner. For example, the image interaction program in Figure 4 can be a channel based on the WebSocket protocol for image data transmission. In an image transmission example based on the WebSocket protocol, Socket is used as the main thread in the virtual machine to process image source upload requests and manage WebSocket sessions. When the Socket main thread receives an image upload request, the server automatically creates a WebSocket service for sending image data. When the WebSocket sub-service receives an image browsing request, it starts sending image data to the cloud control desktop of the local terminal.

[0160] In a preferred embodiment, the data transmission method for the cloud operating system of the present invention may include the following steps: transmitting user control data from a local terminal to a remote computing device through a control data direct connection channel, and the control data direct connection channel has a first transmission priority; an image data direct connection channel established based on a long-chain network communication protocol to transmit interface images from a remote computing device to a local terminal, and the image data direct connection channel has a second transmission priority; the subscription agreement transmission channel has a third transmission priority, wherein the first transmission priority is higher than the second transmission priority or the third transmission priority.

[0161] 4.2 Subscription-based data transmission method

[0162] Referring to Figures 9 to 11 , in some embodiments, a subscription-based communication protocol (e.g., MQTT) is employed between the local terminal and the cloud server. This subscription-based communication approach simplifies and improves data exchange between the local terminal and the cloud server, within virtual machines within the cloud server, and even between browsers. It also allows the server to proactively push data to the client. Furthermore, this subscription-based communication approach can provide real-time, reliable messaging services for remote computing devices with minimal code, limited bandwidth, and asynchronous communication. In these embodiments, the local terminal and the remote computing device (e.g., a cloud server, virtual machine, or container) are each equipped with one or more client units, and the remote computing device is equipped with a resident proxy unit running in the background. Through a subscription protocol transmission channel, a client unit located in the local terminal or remote computing device can publish one or more data message packets through the proxy unit, allowing one or more other client units located in the local terminal or remote computing device that have pre-subscribed to the data message packets to pull the data message packets, thereby completing the consumption of all data message packets. It should be understood that in the above embodiments, the proxy unit can be located and run in the virtual machine of the cloud server, or more specifically, in the underlying cloud computing operating system of the cloud server, serving as a background service program for the cloud server. Specifically, referring to FIG9 , the general agent service center deployed by the cloud server can also be used to uniformly manage multiple agent units.

[0163] Continuing with reference to Figure 9, in some embodiments, the local terminal includes a first local client unit respectively set in the cloud control desktop and a second local client unit outside the cloud control desktop, and the virtual machine includes an agent unit and a first remote client unit set in the browser frame and a second remote client unit outside the browser frame.

[0164] The first local client unit or the second local client unit can allow the first remote client unit or the second remote client unit to retrieve the first data message packet published by the proxy unit. Thus, browser data or local application data of the local terminal can be obtained by the browser of the virtual machine or other programs running in the virtual machine (consumption of the data message packet).

[0165] Similarly, the first remote client unit or the second remote client unit can publish a second data message packet through the proxy unit, allowing the first local client unit or the second local client unit to pull the second data message packet. Thus, the browser data (including UAPP execution data) of the virtual machine or the data of the application running in the virtual machine can be obtained by the browser of the local terminal or other programs running in the local terminal (consuming the data message packet).

[0166] Furthermore, the second local client unit can be enabled to directly obtain local data, including any one or more of locally stored data, local application data, local hardware data, and local user profile data. Furthermore, the first local client unit can communicate directly with the second local client unit to obtain the local data and generate a first data message packet from at least a portion of the obtained data. The first data message packet can be used by a subroutine running in the local browser frame or published to the proxy unit for retrieval or consumption by a remote client unit.

[0167] For example, referring to FIG10 , in one embodiment, the aforementioned second local client unit is integrated into a portion of the background service of the local terminal, and can call a local basic application to obtain local data such as photos, text messages, call logs, and decoded data, and then generate a data message packet and publish it to the proxy unit for the remote client unit to pull or consume. Referring to FIG10 , in another embodiment, the local terminal background service obtains local data of control and operating status (e.g., network status, power, CPU occupancy, etc.) by calling a local system program, which can be transmitted to the first local client unit, and then generate a data message packet and publish it to the proxy unit for the remote client unit to pull or consume. The difference of this embodiment is that various method processes (such as the methods in Sections 3.2.1 and 3.2.3 above) can be implemented based on local data through subroutines running in the local browser framework (or cloud control desktop).

[0168] Referring to Figure 11, in one embodiment, a location request (with a request ID) sent by a map app running in a remote browser framework is captured by a local docking unit associated with the cloud operating system's location application service layer, triggering the local client unit to publish a location data message packet. The location data then is acquired by a local location program or process in a local terminal backend service (e.g., Location Manager) and transmitted to the first local client unit. This data message packet is then generated and published to the proxy unit. The remote client unit pre-subscribes to the local terminal's location data messages from the proxy unit according to the request ID. When the proxy unit receives the message, it consumes it and obtains the location data. The location data acquired by the first local client unit can be used by a subroutine running in the local browser framework to, under specific circumstances, generate a location point on the locally received map interface using the location data. For example, when using a map APP for in-vehicle navigation, if the network connection between the local terminal and the cloud server is limited or disconnected, the subroutine running in the local browser frame can continue to display the navigation map interface using the interface image cached on the network, while using the current positioning data of the local terminal to display the positioning point on the interface, thereby maintaining the navigation experience for a period of time; due to the asynchronous communication method of the subscription protocol, when the local terminal and the cloud server restore the network, the local client unit resumes sending the positioning data during the disconnection period to the proxy unit, thereby being able to push positioning data messages for all navigation periods to the remote client unit for the map APP to obtain complete positioning data.

[0169] In some embodiments, the subscription-based data transmission method may further include the following steps: enabling the first or second remote client unit of the virtual machine logged into the user account to obtain remote data, wherein the remote data includes virtual machine storage data, virtual machine hardware data, or data associated with the user account in the user database of the cloud server; and generating a second data message packet from at least a portion of the obtained remote data. This enables a UAPP running in the virtual machine to access user databases outside the virtual machine or hardware data within the virtual machine across programs or browser frameworks.

[0170] In some embodiments, the subscription-based data transmission method may further include the following steps: ensuring that the proxy unit of each of the multiple virtual machines associated with the user account maintains a communication connection with the general agent service center of the cloud server; enabling the proxy units of the multiple virtual machines to exchange data transmission packets, wherein the data transmission packets include UAPP data and one or more data message packets; and determining that the data message packets in the proxy units have been consumed by the client units of the multiple virtual machines associated with the user account. Thus, the present invention can implement data transmission, synchronization, migration, etc. between multiple virtual machines in the cloud operating system environment.

[0171] 4.3 Data Buffering Method

[0172] Since the application service modules of the cloud operating system of the present invention often need to connect to local terminals and remote computing devices via the Internet, or need to call services and resources from third-party servers, an optimized data buffering method can be used on both sides of the network transmission.

[0173] Referring to Figure 12 , in some embodiments, the virtual machine's browser framework includes a web page file cache for caching page files retrieved by the UAPP from a third-party server (e.g., a web server); while the local terminal's browser framework includes an interface image cache for caching the UAPP's interface images. It is understood that only the larger and more fragmented page data (which may also contain confidential user information) is cached in the cloud service / virtual machine, while the local terminal only needs to cache image data. This reduces storage pressure on the local terminal and facilitates cache cleanup. Furthermore, this solution is more secure, completely preventing the possibility of user data leaking from the local cache.

[0174] In other embodiments, the data buffering method according to the present invention generally includes the following steps: establishing a data buffer between applications (e.g., a local application and a cloud application) associated with data transmission in a local terminal and a virtual machine on a cloud server; and generating continuous data in the data buffer based on historically received data, so that the application reading data from the data buffer continuously reads data. The data buffer includes an interface image data buffer, a multimedia data buffer, a positioning data buffer, and the like.

[0175] In one embodiment, data transmitted by the local program to the cloud program is first temporarily stored in the local terminal's data buffer before being transferred to the cloud server's buffer and subsequently read by the cloud program. Both the local and cloud server buffers can set a timestamp (set to t_n to t_n+4) for each data packet (set to d_n to d_n+4, etc.). If a network interruption causes data that the local program should have transmitted to the cloud server buffer at a time point (e.g., t_n+3) to be interrupted or lost, supplementary data (d_n+2') corresponding to that time point (e.g., t_n+3) is written to the data buffer. This supplementary data (d_n+2') can be equal to the data (d_n+2) at the previous time point (e.g., t_n+2) or a linear prediction of multiple recent data points. When the local program resumes normal data connection with the cloud server's buffer, it can write normal data (d_n+4) to the cloud server's buffer at the new timestamp (e.g., t_n+4). This maintains data continuity in the data buffer, allowing the cloud program to read data continuously.

[0176] In one example, the local program is a positioning app of a local terminal, and the UAPP in the virtual machine can be a map app. The positioning app sends positioning data to the cloud program map app so that the map app can normally locate the geographic location of the local terminal. By adopting the data buffering method of this embodiment, the map app of the virtual machine can continuously obtain positioning data, at least when the local terminal is disconnected from the cloud service, there will still be no map positioning drift. Similarly, when the cloud program transmits data to the local program, continuous data can also be generated in the buffer of the local terminal to avoid drift in the data received by the local program.

[0177] 4.4 Data Redirection Method

[0178] In some embodiments, a data redirection method for a cloud operating system may generally include the following steps: establishing a communication connection between a local terminal and a remote computing device, obtaining a first communication docking address connected to the local terminal, and sending a local terminal identifier to the remote computing device; using the local terminal identifier to establish a communication connection between the remote computing device and a third-party server, and obtaining a second communication docking address connected to the third-party server; disconnecting the communication connection between the remote computing device and the third-party server, and modifying the first communication docking address and the second communication docking address to be consistent, so that at least a portion of the data transmission between the remote computing device and the third-party server is redirected to data transmission between the local terminal and the third-party server.

[0179] In some embodiments, multimedia data (e.g., video and / or audio data) can be intercepted from data sent by a third-party server to a remote computing device, so that the multimedia data transmission between the remote computing device and the third-party server is redirected to stream data transmission between the local terminal and the third-party server. For example, referring to FIG13 , in one instance, a UAPP, taking a news APP as an example, calls a service and obtains a data file from a portal WEB server, which contains news content data, web page layout, multimedia stream playback address, etc. When a user remotely browses to the video page of the news APP, the video stream playback address of the third-party server is obtained from the video page through the multimedia interception unit. At the same time, the playback address of the video control of the local terminal is redirected to the video stream playback address of the third-party server, and then the local video control is arranged and overlaid on the local interface according to the size and position of the video window of the video page of the news APP. Preferably, if the UAPP running on the cloud server plays the video in full screen, the video stream is directly transmitted to the local terminal for full screen playback, and the transmission of the interface image to the local terminal is paused.

[0180] In other embodiments, when a first local terminal and a second local terminal serve as operating terminals for at least one user of a first remote computing device and a second remote computing device, respectively, a data redirection method for a cloud operating system may include the following steps: determining, based on a communication connection established between the first remote computing device and the second remote computing device, an event in which the first local terminal initiates a direct communication connection to the second local terminal; obtaining, from the data connection between the first local terminal and the first remote computing device, a first communication docking address connected to the first local terminal; obtaining, from the data connection between the second local terminal and the second remote computing device, a second communication docking address connected to the second local terminal; and modifying the first communication docking address and the second communication docking address to be consistent, thereby redirecting at least a portion of data transmission between the first remote computing device and the second remote computing device to data transmission between the first local terminal and the second local terminal. Preferably, at least a portion of the data includes real-time call data, thereby enabling a direct call between the first local terminal and the second local terminal (assuming the first and second terminals have call functionality) without forwarding through the remote computing device.

[0181] 5. User database and general agent service center

[0182] The user database features a backup and synchronization mechanism and a high level of security. It allocates data storage space for each user account. The user database can store data for each user account, including: user terminal lists, UAPP lists, UAPP data, user personal files, user operation habit records, user keys, etc.

[0183] The user database of the cloud server and the general agent service center can be used to perform system scheduling between multiple virtual machines to allocate corresponding cloud operating system data and at least one virtual machine to the user account belonging to the local terminal.

[0184] In some embodiments, a proxy service method for a cloud operating system may include the following steps: based on the user account to which a local user terminal belongs, obtaining the cloud operating system data and application storage data corresponding to the user account from a user database in a cloud server, allocating at least one virtual machine corresponding to the cloud operating system data and application storage data to the user account, and providing system scheduling between multiple virtual machines; providing a proxy unit for each virtual machine to communicate between the local terminal and the virtual machine, and exchanging data transmission packets between the proxy units of the multiple virtual machines (as shown in FIG9 ). System scheduling between multiple virtual machines includes system migration, computer resource scheduling, cross-virtual machine application scheduling and execution, or user data synchronization and storage.

[0185] 6. User Application (UAPP)

[0186] First, the working mode or method of UAPP is explained. Referring to Figure 13, the UAPP, taking the news APP as an example, generally includes an interface layout code and a program function code. When the UAPP runs in the WEB browser framework, the interface layout code is generated by the rendering engine to generate the interface layout. The control docking unit in the browser framework transmits the control instructions of the local terminal to operate the UAPP, and the program function code can call the local API and Web API to obtain user feature data, user personality data, user requests, etc. from the local terminal to send to the Web server of the news portal to obtain news content, media streams and other data associated with the user account, which are used to present user-customized news content in the interface of the news APP. Among them, a transmission channel is established between the local client unit and the remote client unit to enable user files of the local terminal (such as locally taken photos) to be transferred to the UAPP. Referring to the browser framework embodiment described in Section 2 above, UAPPs are preferably written using JavaScript scripts and can adopt Native App or Web App development methods. They can also be developed using the Electron framework (a cross-platform desktop GUI application development framework based on web front-end technology) across multiple platforms (such as Windows, Linux / Android, and Mac / iOS), significantly facilitating UAPP development. The following describes a UAPP solution based on the cloud operating system of the present invention using multiple embodiments.

[0187] 6.1 Generation Method

[0188] Referring to Figures 14 to 16, a method for generating a UAPP running on a cloud operating system includes the following steps: providing an application store client on a remote computing device through an application service module, wherein the application store client connects to the application store server to obtain the UAPP icon and installation link; receiving a new UAPP addition request triggered by a local terminal user operation, and obtaining the icon and installation link through the application store server; obtaining the storage data of the new UAPP associated with the user account from the user database; and loading the new UAPP on the remote computing device using the installation link based on the application service module and importing the storage data. The application store server can be set up in a cloud server or in a server of another operator.

[0189] The method for generating a UAPP running on a cloud operating system also includes the following steps: providing the new UAPP's operating environment, connecting to local hardware resources and interface images, and accessing a user database through the browser framework of a remote computing device through an application service module; transmitting the new UAPP's interface image to the local browser framework of a local terminal to run the cloud control desktop, and allocating local resources to the new UAPP according to a user-configured permission request table. The permission request table will be described in detail in the embodiments below.

[0190] 6.2 Shelf Installation

[0191] Referring to Figures 14 to 16, and in conjunction with steps S411-S416 in Figure 16, the UAPP listing method includes the following steps: S411. Obtaining the UAPP listing request information, which includes an icon file, a web address, a permission request table, the required browser kernel type, and the source code; S412. Determining the browser kernel type; S413. Determining whether the listing request information complies with application review rules, then packaging the icon file, web address, and source code, listing them to the app store server, and assigning an installation link. If the listing fails to pass review, the listing fails.

[0192] Referring to Figure 14 , in some embodiments, the listing request information may also include: various application keys, which serve as credentials for enabling application functions or accessing the web server; page style and control preferences, such as Android or iOS interface style and gesture preferences; required browser plugin types; and copyright information. Referring to Figure 15 , in some embodiments, the permission request table, to ensure the functionality and secure and stable operation of the UAPP, requires application for or use of cloud operating system permissions, such as SMS and contact information, device location information, user phone number, access to local storage, notifications, and more. Then, during UAPP installation, after the user confirms the authorized permissions, a target permissions table is generated (e.g., S416 ).

[0193] In a preferred embodiment, (e.g., S414) when the required browser type does not match the kernel type of the browser framework provided by the local terminal or remote computing device, the browser framework provided by the remote computing device switches its kernel to match the required browser type, or reloads functions in the source code to match the installed UAPP with the browser framework's kernel. (e.g., S415) The UAPP's theme and operating mode need to be configured so that when the UAPP is running, the pages provided by the browser framework switch to the page theme and operating mode of the browser type required by the UAPP.

[0194] The UAPP listing review can include the following processes: (1) Content review - the app store server will review the content of the app to ensure that the content of the app meets the requirements of the app store server. For example, the app store server prohibits apps that do not comply with local regulations. Therefore, the review process will carefully check the content keywords of the app to ensure that the content of the app will not cause any harm to users. (2) Functional review - the app store server will review the functions of the app to ensure that the functions of the app meet the requirements of the app store server. For example, the app store server may require that the app must not contain any malicious code, viruses, Trojans, etc. The review process will carefully check the functions of the app to ensure that the functions of the app will not cause any harm to the user's device. (3) User experience review - the app store server will review the user experience of the app to ensure that the user experience of the app meets the requirements of the app store server. For example, the app store server may require that the interface design of the app be simple, clear, and easy to use, and the functions of the app must also be easy to use. The review process will carefully check the user experience of the app to ensure that the user experience of the app meets the requirements of the app store server. (4) Copyright review - The App Store server will review the copyright of the app to ensure that the copyright of the app meets the requirements of the App Store server. For example, the App Store server may require that the app must not infringe on the intellectual property rights of any third party. The review process will carefully check the copyright materials of the app to ensure that the copyright of the app meets the requirements of the App Store server. (5) Security review - The App Store server will review the security of the app to ensure that the security of the app meets the requirements of the App Store server. For example, the App Store server may require that the network communication protocols and third-party servers used by the app be tested and screened to ensure the privacy of users.

[0195] 6.3 Application Content Integration

[0196] In some embodiments, Userkey can be used to obtain resources and functions from the Web server to integrate them into the same UAPP. For example, referring to Figure 17, the local terminal used by the user stores the user's personal characteristic information (such as fingerprints, facial data, personal security passwords, mobile phone number verification codes, cloud server user accounts, etc.), which requires a higher level of security and confidentiality, so it is necessary to convert or additionally extract the Userkey associated with the personal characteristic information (such as secondary user account passwords, service request keys, etc.). Based on the fact that the user has registered an account through one or more service providers and obtained the Userkey, the registered Userkey can be obtained by the user's authorization through the local user terminal, and then the proxy sends the Userkey to the service provider's Web server to obtain the user's personalized resources and integrate them into the same UAPP for presentation. Therefore, the solution based on Userkey brings greater flexibility and richness to the development and ecological chain of the UAPP of the cloud operating system of the present invention.

[0197] 6.4 Cloud Input Method

[0198] Referring to FIG. 18 , in some embodiments, a cloud input method for a cloud operating system may include the following steps: collecting user input operations on a local terminal, the user input operations including user voice or operation points on a local interface; converting the user input operations into speech recognition phrases or operation instructions for an operating interface on a remote computing device; triggering an input panel based on the speech recognition phrases or operation instructions to generate first language data; retrieving second language data associated with the first language data from a remote vocabulary to present the first language data and / or second language data on the operating interface; wherein the remote vocabulary receives the local input data from the local vocabulary of the local terminal, and the remote vocabulary is synchronized with the user input data stored in a user database. The remote vocabulary receives a personalized cloud vocabulary associated with the user account from a third-party web server.

[0199] Preferably, the cloud input method for a cloud operating system may further include the following steps: calling an AI function through an application programming interface of an AI server to convert first language data into third language data, so as to present the third language data on a remote computing device, wherein the AI ​​server deploys a large-scale pre-trained language model (e.g., GPT, BERT, BLOOM, XLNet, ERNIE Bot, etc.).

[0200] 6.5 Opening other apps within an app

[0201] 19 , in some embodiments, a method for opening a UAPP for a cloud operating system may include the following steps: capturing a user input operation on a local terminal, where the user input operation includes a user voice command or an operation point on a local interface; determining, in a remote computing device, an operation object of the user input operation in the current running interface of a first UAPP; determining that the operation object is associated with a second UAPP, and after capturing a message allowing the second UAPP to start, generating a running interface of the second UAPP, allowing it to overlay at least a portion of the interface of the first UAPP, and allowing the first UAPP to continue running.

[0202] 20 , the method for opening a UAPP for a cloud operating system may further include the following steps: pre-configuring an association list of operation objects and UAPPs to ensure that each operation object is opened by the corresponding UAPP, the operation objects including file types, hardware resources, or website addresses; or, after determining that the permissions required by the UAPP to be opened are consistent with the permission type assigned by the local terminal, generating a message allowing the UAPP to be started. Specifically, the association list of operation objects and UAPPs may be a file opening method table. For example, in the configuration in which the default application for opening MP3 files is a local player, when clicking on a file that is not configured with an application to open, a query window will be issued, prompting the user whether to download an application that can open the file from the application store. Specifically, when the UAPP or link to be opened requires operating system permissions, the target permission table of the UAPP is called or the user is asked whether he agrees to authorize the permissions required for the opening event. After confirming that the permissions are enjoyed, the opening operation is allowed.

[0203] Referring to Figure 21 , in some embodiments, when an operation object is determined to be associated with a third UAPP or system bar on a remote computing device, a UAPP or process of the same type is invoked on the local terminal. The third UAPP types include camera applications, phone applications, SMS applications, address book applications, or other basic applications based on local functionality. The system bar includes the remote computing device status bar or the hardware control center bar. For example, the cloud server virtual machine's runtime interface includes multiple third UAPP icons and operation interfaces. Each third UAPP's corresponding APP ID is associated with a local basic app or process (e.g., camera, address book, SMS, location). When a user operates a third UAPP icon (e.g., clicking on a phone), an operation interface (e.g., a dial pad) pops up. The user's operation instruction (e.g., dialing a mobile phone number) is then received through the operation interface. The interface then returns to the local terminal to trigger the operation of the local program or process (e.g., a local mobile phone dialer). In one embodiment, the user's operation instruction can be triggered by a backend runtime module (e.g., a JS program in a local browser framework) via a JS program. This allows the local basic app or process to be triggered. This allows the switching requirements between cloud-based UAPPs and applications requiring local hardware to be met.

[0204] 7. Others

[0205] The architecture, framework, modules, components, units, controls, plug-ins, method steps, and the like in the embodiments of the present invention may implement their features in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. Such features may be embodied in a computer program product tangibly embodied in an information carrier (e.g., in a machine-readable storage device) for execution by a programmable processor; and the method steps may be performed by a programmable processor that executes a program of instructions to perform the specifically implemented functions by operating on input data and generating output.

[0206] The described features can be advantageously implemented in one or more computer programs that can be executed on a programmable system, the programmable system comprising at least one input device, at least one output device, and at least one programmable processor coupled to receive data and instructions from a data storage system and to transfer the data and instructions to the data storage system. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform a certain activity or produce a certain result. A computer program can be written in any form of programming language (e.g., C++, Java) including compiled and interpreted languages, and can be deployed in any form, including as a stand-alone program or as a program segment, subroutine, or other unit suitable for use in a computing environment. For this purpose, a computer program can be run on a programmed application specific integrated circuit.

[0207] For example, suitable processors for executing a program of instructions include both general-purpose and special-purpose microprocessors, as well as one or the sole processor of multiple processors or cores of any type of computer. Generally speaking, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more storage media for storing instructions and data. Generally speaking, a computer can communicate with mass storage devices to store data files. These mass storage devices may include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. The processor and memory may be supplemented by, or incorporated into, application-specific integrated circuits.

[0208] While various aspects of a specific implementation have been described above, it should be apparent that the various features of the above-described specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, those skilled in the art will appreciate that the aspects described herein can be implemented independently of any other aspects, and that two or more of these aspects can be combined in various ways. For example, a device and / or method can be implemented using any number of the aspects described herein. In addition, other structures and / or functions can be used in addition to or different from one or more of the aspects described herein to implement such a device and / or to implement such a method. It will also be understood that while the terms "first," "second," and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are simply used to distinguish one element from another. For example, a first node can be referred to as a second node. The terms used herein are intended solely to describe a specific implementation and are not intended to limit the claims. As used in the description of this implementation and the appended claims, the singular forms "a," "an," and "the" are intended to encompass the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. As used herein, the term "if" may be interpreted to mean "when the antecedent condition is true" or "when the antecedent condition is true" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent condition is true, depending on the context.

Claims

1. A cloud operating system, arranged in a remote computing device, wherein the remote computing device is associated with a local terminal via a network and the local terminal is used as a user operation terminal and a user application interface receiving terminal, characterized in that: The cloud operating system includes: A basic kernel, configured to call upon physical and / or virtualized hardware resources of the remote computing device according to the request; A runtime library, based on the basic kernel, to provide a runtime environment for at least one operating system platform; An interface service module, the interface service module corresponds to the runtime library to provide an application program interface; The application service module is used to provide the local terminal with an application program interface and service connection.

2. The cloud operating system according to claim 1, wherein: The application service module includes: A device monitoring unit, used to obtain hardware parameters and real-time operation status data of the local terminal; An interface generation unit, used to generate a running interface of a user application according to the hardware parameters of the local terminal; A cloud control docking unit, used to maintain control interaction with the local terminal; The driver docking unit is used to process the redirection between the hardware driver request required by the application program in the remote computing device and the hardware driver request between the local terminal.

3. The cloud operating system according to claim 1 or 2, wherein: The local terminal has a display screen, network communication hardware and a local operating system, and the remote computing device includes any one or more of a cloud server, a virtual machine, a container and a personal computer, wherein at least a portion of the cloud operating system is deployed in a remote server, virtual machine or container.

4. The cloud operating system according to any one of claims 1 to 3, wherein: The local terminal includes a cloud control desktop, which runs based on a browser framework in a local operating system of the local terminal, and includes: Interface controls or plug-ins, which provide a user operation interface and functional extensions associated with the interface based on the browser framework; A local access unit, which is connected to the interface control or plug-in and is used to capture the user's real-time manipulation data and information of the local application program accessing the local terminal and hardware information; One or more service units, the service units are connected to the interface controls or plug-ins, and are used for interaction of interface images, videos and audios between local and remote; One or more client units, each of which is connected to the service unit and the local access unit, and is used for communicating and transmitting with the proxy service program in the remote computing device.

5. The cloud operating system according to any one of claims 2 to 4, wherein: At least one user application and at least one background service program are run remotely in an environment based on the application service module, wherein: The device monitoring unit obtains hardware data of the local terminal including a screen and input devices and receives data of a local application of the local terminal after an authenticated connection is established between the local terminal and the remote computing device; The interface generation unit generates an interface environment for presenting the user application according to the interface style and preset operation habits corresponding to the user account and based on the hardware configuration parameters obtained by the device monitoring unit, wherein the running interface of the user application is intercepted by the background service program and then transmitted to the cloud control desktop of the local terminal; The cloud control docking unit is called by the background service program to receive the user's control data from the local terminal in real time and convert it into the control data of the remote computing device; The driver docking unit responds to the hardware resource request of the user application program to redirect the source data for hardware execution of the remote computing device to the hardware of the local terminal.

6. The cloud operating system according to any one of claims 2 to 5, wherein: At least a part of the interface generation unit is provided by an interface rendering engine of a browser frame of a remote computing device, so as to generate a running interface of the user application for a web page file received by the user application from a WEB server.

7. The cloud operating system according to any one of claims 2 to 6, wherein: In response to the user application program initiating a hardware resource request through the application program interface, the driver docking unit captures source data from the memory area of ​​the remote computing device, the source data being the data that the remote computing device hardware is to receive from the memory area triggered by the driver program, and then forwards the source data to the local terminal; In response to the instruction of the driver docking unit, the service unit in the background of the local terminal initiates hardware resource docking through the application program interface to receive the source data, and triggers the local hardware to execute the source data through the local driver.

8. The cloud operating system according to any one of claims 1 to 7, wherein: The remote computing device includes a cloud server and at least one virtual machine generated in the cloud server, wherein at least a runtime library, an interface service module and an application service module of the cloud operating system are deployed in the virtual machine; The user database of the cloud server is used to perform system scheduling among multiple virtual machines through the general agent service center, so as to allocate corresponding virtual machines and cloud operating system data to the user accounts belonging to the local terminals.

9. The cloud operating system according to any one of claims 1 to 8, wherein: The basic kernel is a Linux kernel, and the runtime library includes any one or more of a Linux runtime component, an Android runtime component, a Windows runtime component, and an iOS runtime component.

10. The cloud operating system according to any one of claims 1 to 9, wherein: The interface service module includes an interpreter or engine for interpreting and executing the user application code.

11. A remote interface generation method, wherein: The method comprises the following steps: Establishing a network association between a remote computing device and a local terminal and using the local terminal as a user operation terminal; Acquire a user account and hardware configuration parameters from the local terminal, the hardware configuration parameters including a local operating system type, a screen size, a screen resolution, and an input device type; Acquire the interface style and operation habits corresponding to the user account or the local operating system type from a user database associated with the remote computing device, generate a browser window in the browser frame of the remote computing device with the acquired screen size or screen resolution of the local terminal, configure a renderer to generate an interface layout based on the interface style, and determine the operation gestures of the browser window with the operation habits, thereby serving as an interface environment for a user application running based on the browser frame; The interface image of the browser window is captured and transmitted to the local terminal for display as an interface image.

12. The method according to claim 11, wherein: The renderer of the browser frame of the remote computing device is provided with an HTML interpreter and a CSS interpreter, and generates a running interface of the user application program for the web page file and the JS script received by the user application program from the WEB server through the JS engine; Converting the running interface into an interface image through a plug-in running in a browser frame of a remote computing device; A browser frame is provided on the local terminal to allow the browser control to receive the interface image for display.

13. The method according to claim 11 or 12, wherein: The following steps are involved: According to the current network transmission status, application scenario status, user instructions, control status or display resolution of the local terminal, The interface image to be transmitted from the remote computing device to the local terminal is pre-adjusted to transmit at least a portion of the dynamically generated interface image, wherein the pre-adjustment includes cutting, compressing or contrasting at least a portion of the image.

14. An interface image transmission method, used in the cloud operating system according to any one of claims 1 to 10, wherein: The method comprises the following steps: Establishing a control data interactive connection between a local terminal and a remote computing device, wherein a change in at least a portion of an interface of the remote computing device is caused by control data sent by the interactively connected local terminal; Maintaining a transmission connection of the interface image of the remote computing device to the local terminal for at least one period of time; According to a transmission interval of the interface image transmitted by the remote computing device to the local terminal, dynamically adjusting the interface image to be transmitted by the remote computing device to the local terminal, the dynamic adjustment comprising: After determining that image transmissions below an interval threshold occur in a time period, the number of image transmissions is reduced and / or the quality of at least a portion of the interface image is reduced in the next time period.

15. The method according to claim 14, wherein: The step of dynamically adjusting the interface image to be transmitted from the remote computing device to the local terminal includes: Real-time detection of the image transmission interval of the interface image in the same interface range from the remote computing device to the local terminal; Determining that the current image transmission interval is lower than the first time threshold, then giving up the current transmission of the interface image to be transmitted; Determining that the current image transmission interval is lower than a second time threshold, then reducing the quality of the interface image to be transmitted; The first time threshold is smaller than the second time threshold.

16. The method according to claim 14 or 15, wherein: The step of dynamically adjusting the interface image to be transmitted from the remote computing device to the local terminal includes: Detecting a current transmission interval of an interface image of a same interface range from the remote computing device to the local terminal; If it is determined that the current transmission interval is higher than the third time threshold, the quality of the interface image to be transmitted is restored, or an interface image with higher image quality is collected from the remote computing device for transmission to the local terminal.

17. The method according to claim 14, comprising the steps of: enabling the local terminal to receive an interface image from the remote computing device; According to the manipulation interval monitored in real time on the local terminal, the interface image from the remote computing device is dynamically filtered to display at least a portion of the filtered interface image on the local terminal.

18. The method according to claim 17, comprising the steps of: Determining that a manipulation interval of the local terminal is less than a fourth time threshold within at least one time period; From the received interface image of the remote computing device, discard at least a portion of the interface image for display on the local terminal; The frequency of the manipulation data sent by the local terminal to the computing device is reduced.

19. The method according to claim 17, wherein: The step of dynamically screening the interface image from the remote computing device includes: In at least one time period, if it is determined that the manipulation interval of the local terminal is greater than a fifth time threshold, all interface images for display on the local terminal are retained from the received multiple interface images; In at least one time period, if it is determined that the manipulation interval of the local terminal is greater than a sixth time threshold, at least one interface image is selected from the received multiple interface images at intervals to be displayed on the local terminal until the next manipulation of the local terminal occurs; Among them, the sixth time threshold is greater than the fifth time threshold.

20. The method according to any one of claims 14 to 19, wherein: The following steps are involved: If it is determined that the manipulation interval of the local terminal is greater than a seventh time threshold within at least one time period, the display frame rate of the local terminal is reduced until the next manipulation of the local terminal occurs.

21. The method according to any one of claims 14 to 20, wherein: The method comprises the following steps: Determining that the movement amplitude of the control point of the local terminal exceeds a preset amplitude threshold; Among a plurality of interface images transmitted to the local terminal by a remote computing device caused by the manipulation data generated by the manipulation point of the local terminal, retain the interface images associated with the start action and the end action of the manipulation point, and reduce the remaining interface images for display on the local terminal; The interface image is synchronized between the local terminal and the remote computing device at a regular interval.

22. A remote browser, running on a cloud operating system, wherein: The remote browser comprises: The client unit is used for the remote browser to communicate with other applications running on the cloud operating system and to communicate with the local terminal to obtain data from the local terminal and the remote computing device; An interface image acquisition unit, used to acquire at least a portion of the interface image from a user interface of a web-based user application program running on a remote browser; The client unit and the interface image acquisition unit are respectively connected to an image interaction service program in the cloud operating system, and the image interaction service program implements the method according to any one of claims 14 to 16.

23. A local browser, running on a local operating system, wherein: The local browser includes: A client unit, used for the local browser to communicate with other applications running on the local operating system and to communicate with the remote computing device to obtain data from the local terminal and the remote computing device; A display component, used to display the interface image from the remote computing device; The image interaction service unit connected to the display component is used to implement the method according to any one of claims 17 to 21.

24. A data transmission method for a cloud operating system, wherein the cloud operating system is deployed in a remote computing device, the remote computing device is associated with a local terminal via a network and the local terminal is used as a user operation terminal and a system interface display terminal of the cloud operating system, the local terminal and the remote computing device are respectively provided with one or more client units, and the remote computing device is provided with an agent unit that is resident and runs in the background, in, The method comprises the following steps: Through the subscription protocol transmission channel, the client unit set in the local terminal or the remote computing device publishes one or more data message packets through the proxy unit to allow one or more other client units set in the local terminal or the remote computing device that have pre-subscribed to the data message packets to pull the data message packets to complete the consumption of all data message packets.

25. The method of claim 24, wherein: Transmitting user manipulation data from the local terminal to the remote computing device via a manipulation data direct connection channel, wherein the manipulation data direct connection channel has a first transmission priority; An image data direct connection channel established based on a long-chain network communication protocol to transmit an interface image from the remote computing device to the local terminal, the image data direct connection channel having a second transmission priority; The subscription protocol transmission channel has a third transmission priority, wherein the first transmission priority is higher than the second transmission priority or the third transmission priority.

26. The method according to claim 24 or 25, wherein: The remote computing device includes a cloud server and at least one virtual machine generated in the cloud server, and uses a user database of the cloud server and a general agent service center to perform system scheduling between multiple virtual machines, so as to allocate corresponding cloud operating system data and at least one virtual machine to a user account belonging to the local terminal; The process running through the browser frame in the allocated virtual machine provides an interface environment for the user application and is used to access the user database; the local terminal includes a cloud control desktop running based on the local browser frame, which is used to receive the interface image generated by the browser frame of the virtual machine; The local terminal includes a first local client unit and a second local client unit respectively arranged in and outside the cloud control desktop, and the virtual machine includes the proxy unit and a first remote client unit and a second remote client unit arranged in and outside the browser frame; And wherein the method comprises the following steps: enabling the first local client unit or the second local client unit to publish a first data message packet through the proxy unit to allow the first remote client unit or the second remote client unit to pull the first data message packet; or The first remote client unit or the second remote client unit publishes a second data message packet through the proxy unit to allow the first local client unit or the second local client unit to pull the second data message packet.

27. The method according to claim 26, comprising the steps of: enabling the second local client unit to directly obtain local data, wherein the local data includes any one or more of local storage data, local application data, local hardware data and local user feature data; The first local client unit is enabled to communicate directly with the second local client unit to obtain the local data, and the first data message packet is generated by using at least a portion of the obtained data.

28. The method according to claim 26 or 27, The following steps are involved: Enable a first remote client unit or a second remote client unit of a virtual machine logged in to a user account to obtain remote data, wherein the remote data includes virtual machine storage data, virtual machine hardware data, or data associated with the user account in a user database of the cloud server; The second data message packet is generated by using at least a portion of the acquired remote data.

29. A method according to any one of claims 26 to 28, comprising the steps of: The agent unit of each virtual machine in the plurality of virtual machines associated with the user account maintains a communication connection with the general agent service center of the cloud server; enabling the proxy units of the plurality of virtual machines to exchange data transmission packets, wherein the data transmission packets include user application data and one or more data message packets; It is determined that the data message packet in the proxy unit has been consumed by the client units in the plurality of virtual machines associated with the user account.

30. A data redirection method, based on the cloud operating system according to any one of claims 1 to 10, wherein the method comprises the following steps: Establishing a communication connection between the local terminal and the remote computing device, obtaining a first communication docking address connected to the local terminal, and sending a local terminal identifier to the remote computing device; Establishing a communication connection between the remote computing device and a third-party server using the local terminal identifier, and acquiring a second communication connection address connected to the third-party server; The first communication docking address and the second communication docking address are modified to be consistent, so that at least a part of the data transmission between the remote computing device and the third-party server is redirected to the data transmission between the local terminal and the third-party server.

31. The method according to claim 30, wherein: Multimedia data is intercepted from data sent by the third-party server to the remote computing device, so that the multimedia data transmission between the remote computing device and the third-party server is redirected into streaming data transmission between the local terminal and the third-party server.

32. A data redirection method, based on the cloud operating system according to any one of claims 1 to 10, wherein a first local terminal and a second local terminal serve as operating terminals of at least one user of a first remote computing device and a second remote computing device, respectively, wherein the method comprises the following steps: Determining, based on the communication connection established between the first remote computing device and the second remote computing device, an event of a direct communication connection initiated by the first local terminal to the second local terminal; Acquire a first communication docking address connected to the first local terminal from a data connection between the first local terminal and the first remote computing device; acquiring, from a data connection between the second local terminal and the second remote computing device, a second communication docking address connected to the second local terminal; The first communication docking address and the second communication docking address are modified to be consistent, so that at least a portion of data transmission between the first remote computing device and the second remote computing device is redirected to data transmission between the first local terminal and the second local terminal.

33. The method of claim 32, wherein: The at least part of the data includes real-time call data of the user.

34. A data buffering method, used in the cloud operating system according to any one of claims 1 to 10, wherein the method comprises the following steps: establishing one or more data buffers between a plurality of nodes in a local terminal and a remote computing device for transmission over a network; According to the historical received data of the node, continuous data is complemented and generated in the data buffer, so that the node that reads data from the data buffer continuously performs data reading.

35. The method of claim 34, wherein: The data buffer includes any one or more of an interface image data buffer, a multimedia data buffer, and a positioning data buffer.

36. A proxy service method, used in the cloud operating system according to any one of claims 1 to 10, wherein: The remote computing device includes a cloud server and at least one virtual machine generated in the cloud server, The proxy service method comprises the following steps: According to the user account to which the local user terminal belongs, cloud operating system data and application storage data corresponding to the user account are obtained from the user database in the cloud server, so as to allocate at least one virtual machine having the cloud operating system data and application storage data corresponding to the user account, and provide system scheduling between multiple virtual machines; A proxy unit is provided for each virtual machine for the local terminal to communicate with the virtual machine, and data transmission packets are exchanged between the proxy units of the multiple virtual machines.

37. The method of claim 36, wherein: The system scheduling between the multiple virtual machines includes: system migration, computer resource scheduling, cross-virtual machine application scheduling execution, or user data synchronization storage.

38. A method for generating a user application, based on the cloud operating system according to any one of claims 1 to 10, wherein the method comprises the following steps: Providing an application store client in a remote computing device through the application service module, wherein the application store client connects with the application store server to obtain icons and installation links of user applications; Receiving a request for adding a new user application triggered by the local terminal user operation, and obtaining an icon and an installation link through the application store server; Obtaining stored data of the new user application associated with the user account from a user database; Based on the application service module, the new user application is loaded into the remote computing device using the installation link, and the storage data is imported.

39. The method of claim 38, wherein: The following steps are involved: Providing the operating environment of the new user application program, connecting to local hardware resources and interface images, and accessing the user database through the browser framework of the remote computing device through the application service module; The interface image of the new user application is transmitted to the local browser frame of the local terminal to run the cloud control desktop, and local resources are allocated to the new user application according to the permission request table configured by the user.

40. The method according to claim 38 or 39, comprising the steps of: Obtain the user application listing request information, which includes icon file, WEB address, permission request table and original code; Determine whether the listing request information complies with the application review rules, then package the icon file, WEB address and original code and list them on the application store server, and allocate an installation link.

41. The method according to claim 40, wherein the listing request information includes the type of browser required, the method comprising the following steps: When the required browser type does not match the kernel type of the browser frame provided by the local terminal or remote computing device, Causes the browser frame provided by the remote computing device to switch kernels to match the type of browser required, or Reloading the functions in the original code to make the installed user application program match the kernel of the browser framework; When the user application is running, the page provided by the browser frame is switched to the page theme and operation mode of the type of browser required by the user application.

42. An input method based on a cloud operating system, wherein the cloud operating system is set in a remote computing device, the remote computing device is associated with a local terminal through a network to transmit an interface image to the local terminal, and the local terminal is used as a user operation terminal, the method comprising the following steps: Collecting a user input operation of the user at the local terminal, wherein the user input operation includes a user voice or an operation point on the local interface; In the remote computing device, converting the user input operation into a speech recognition phrase or an operation instruction of a running interface; triggering an input panel based on the voice recognition words or the operation instructions to generate first language data; Retrieving second language data associated with the first language data from a remote word library to present the first language data and / or the second language data on the operation interface; The remote word library receives local input data from the local word library of the local terminal, and the remote word library is synchronized with the user input data stored in the user database.

43. The method of claim 42, wherein: The remote word library receives a personalized cloud word library associated with a user account from a third-party WEB server.

44. The method according to claim 42 or 43, wherein: The following steps are involved: The AI ​​function is called through the application program interface of the AI ​​server to convert the first language data into the third language data so as to present the third language data on the remote computing device, wherein the AI ​​server deploys a large-scale pre-trained language model.

45. A method for opening a user application, used in the cloud operating system according to any one of claims 1 to 10, comprising the following steps: Capturing a user input operation of the user at the local terminal, wherein the user input operation includes a user voice command or an operation point on a local interface; In the remote computing device, determining an operation object of the user input operation in the current running interface of the first user application; Determine that the operation object is associated with a second user application, and after capturing a message allowing the second user application to start, generate a running interface for the second user application, allow at least a portion of the interface of the first user application to be overlaid, and allow the first user application to keep running.

46. ​​The method according to claim 45, comprising the steps of: Pre-configuring an association list of the operation objects and user applications to determine that each operation object is opened by a corresponding user application, the operation objects including file types, hardware resources or website addresses; or, After determining that the permission required by the user application to be opened is consistent with the permission type assigned by the local terminal, a message allowing the user application to be started is generated.

47. The method according to claim 45 or 46, comprising the steps of: When it is determined in the remote computing device that the operation object is associated with a third user application or a system bar, a user application or process of the same type is called in the local terminal. The type of the third user application includes a camera application, a phone application, a text message application, an address book application or other basic applications based on local functions. The system bar includes a remote computing device status bar or a hardware control center bar.

48. A computer-readable storage medium having program instructions stored thereon, wherein when the program instructions are executed by a processor, the method according to any one of claims 4 to 16 and 24 to 47 is implemented.

49. A mobile computing terminal, characterized in that: include: A network communication module connected to a cloud server; An image processor for processing interface image data sent by a remote computing device; A screen for displaying the interface image data; The computer readable storage medium of claim 48.

50. A cloud computing server, characterized in that: include: at least one remote computing device, the remote computing device comprising one or more virtual machines; A general agent service center connected to the one or more virtual machines, used for system scheduling among the multiple virtual machines and cloud operating system data allocation; The computer readable storage medium of claim 48.

Images