Augmented reality cooperation system of cloud desktop

Abstract

The invention provides an augmented reality cooperation system for a cloud desktop, and the system comprises a cloud service module which generates a virtual desktop image, and carries out the resource monitoring and data storage; the AR equipment is used for collecting data and interacting with virtual reality; the collaborative rendering module is used for fusing AR rendering and virtual reality; the interaction module is used for a multi-modal interaction mode and personalized recommendation; the intelligent scheduling module is used for dynamic resource allocation, resource prediction and optimal management; the network optimization module is used for designing a layered edge architecture and introducing a QUIC protocol to carry out optimized transmission of data streams; the security module is used for carrying out encryption transmission and verifying the identity and access of the user; and the compatible module is used for defining standardized interfaces and protocols and realizing multi-platform support. Functions and resources can be deeply fused, the problems of network dependence, interaction precision, resource allocation and the like are solved, optimal AR presentation and experience effects are provided, and various application scenes are supported.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud desktops, and particularly to an augmented reality collaboration system for cloud desktops. Background Art

[0002] A cloud desktop is a virtual desktop infrastructure based on cloud computing technology, which hosts a user's desktop environment (including operating system, applications, files, etc.) on a cloud server. The user remotely accesses and uses these desktop environments through terminal devices (such as PCs, tablets, mobile phones, etc.). The intelligent augmented reality system of cloud desktops is a system that combines cloud desktop technology and augmented reality (AR) technology, aiming to provide users with a fusion experience of virtual desktops and the real environment, and is an important direction for future digital office and interaction. However, with the increase in application fields, user requirements have become diversified. Under diversified requirements, the existing technologies have the following problems: 1) In high-concurrency scenarios, there is a problem of low efficiency in cloud resource allocation; 2) Strong network dependence, network latency or insufficient bandwidth will lead to a decline in the user experience; 3) Low interaction accuracy, which affects the user's stable immersive experience and personalized services, etc. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide an augmented reality collaboration system for cloud desktops, which can deeply integrate functions and resources, solve problems such as network dependence, interaction accuracy, and resource allocation, provide the best AR presentation and experience effects, and support various application scenarios.

[0004] To achieve the above purpose, the present invention provides the following solution: An augmented reality collaboration system for cloud desktops, comprising:

[0005] A cloud service module, which is used to run a virtual desktop infrastructure and cloud computing resources in the cloud, generate a virtual desktop image, and perform real-time resource monitoring and data storage;

[0006] An AR device, which is used to collect gesture data, voice data, eye movement data, and environmental data, and realize the interaction between the user and the virtual desktop;

[0007] A collaborative rendering module, which is used to perform AR rendering on the image of the cloud desktop in the cloud, and then synthesize the rendered image into the real scene through the AR device;

[0008] An interaction module, which is used to perform gesture recognition, voice control, eye movement tracking, and environmental perception, realize multi-modal interaction methods, and perform personalized recommendations for interaction methods according to the user's operation mode;

[0009] The intelligent scheduling module is used to evaluate the user's resource requirements, dynamically allocate computing resources, storage resources, and network resources according to the evaluation results, analyze the user's historical operation data, perform resource demand prediction and resource pre-allocation, and build a resource pool to perform resource optimization management in high-concurrency scenarios;

[0010] The network optimization module is used to design a hierarchical edge architecture, perform task allocation, and introduce the QUIC protocol to optimize the transmission of data streams;

[0011] The security module is used to encrypt and transmit data by combining forward security and end-to-end encryption technologies, and verify the user's identity and access;

[0012] The compatibility module is used to define standardized interfaces and protocols, develop client applications adapted to different operating systems, provide SDKs and development documents, and achieve multi-platform support;

[0013] Among them, the cloud service module, the AR device, the collaborative rendering module, the interaction module, the intelligent scheduling module, the network optimization module, the security module, and the compatibility module are interconnected with each other.

[0014] Optionally, the collaborative rendering module includes:

[0015] The cloud desktop rendering adaptation unit is used to capture the virtual desktop image, encode and compress the virtual desktop image according to the network bandwidth, adjust the frame rate of the virtual desktop image according to the refresh rate of the AR device, and then scale, crop, optimize the edge, and correct the color of the virtual desktop image according to the display resolution and display characteristics of the AR device; among them, during the frame rate adjustment process, when the virtual desktop image is of low frame rate, frame rate interpolation technology is used to generate intermediate frames to improve visual fluency;

[0016] The AR-side construction unit is used to deploy a client application supporting the cloud desktop protocol on the AR device, introduce multi-threading technology to perform image decoding and rendering tasks based on the client application, integrate the AR SDK toolkit to perform spatial positioning of the virtual desktop image, use SLAM technology to fix the virtual desktop image at a specified position in the real scene, and then project the decoded virtual desktop image into the real scene;

[0017] The cloud communication interface unit is used to real-time monitor the network bandwidth, latency, and packet loss rate between the client application and the cloud to obtain the network status, adjust the bit rate and resolution of the virtual desktop image according to the network status, and perform multi-channel data transmission using the UDP transmission protocol and forward error correction technology; among them, the multi-channel data transmission includes an image data transmission channel, an interaction data transmission channel, and a control data transmission channel;

[0018] A collaborative rendering unit, which is used to combine with a GPU cluster in the cloud, utilize distributed rendering technology to decompose complex graphics computing tasks into multiple GPU nodes for parallel processing, complete graphics computing and preliminary rendering, generate low-resolution preliminary image frames, transmit the preliminary image frames to an edge computing device for frame optimization, and then transmit the optimized preliminary image frames to the client application for final rendering and display to obtain final image frames, and project the final image frames into the real scene to complete spatial positioning and projection.

[0019] Optionally, the interaction module includes:

[0020] A gesture recognition unit, which is used to deploy a depth camera and sensors in the AR device, capture the user's hand movements, collect hand depth information, bone structure features and movement trajectories, generate hand data, and combine a convolutional neural network and a recurrent neural network to build a gesture recognition model, use the gesture recognition model to recognize the user's gesture actions in the hand data, and then classify and operation map the user's gesture actions;

[0021] A voice control unit, which is used to integrate a speech recognition engine, utilize a pre-trained language model to parse the user's voice commands, extract semantic information, convert the user's speech into text according to the semantic information, and design an extensible voice command library for automatic operation mapping;

[0022] An eye movement tracking unit, which is used to deploy an eye movement sensor in the AR device, capture the user's line of sight trajectory, calculate the line of sight focus in the user's visual trajectory by using optical reflection technology, geometric models and deep learning algorithms, and map the line of sight focus to a specific position on the virtual desktop;

[0023] A personalized recommendation unit, which is used to analyze the user's operation mode, and according to the analysis results, introduce matrix factorization technology into the collaborative filtering algorithm based on user behavior to recommend operations or applications similar to the user's habits;

[0024] A scene perception unit, which is used to utilize the GPS, light sensor and microphone in the AR device to collect the user's surrounding environment data, analyze the user's surrounding environment data, extract scene features, and automatically adjust the brightness, contrast and interaction mode of the virtual desktop according to the scene features.

[0025] Optionally, the expression of the line of sight focus is:

[0026]

[0027]

[0028]

[0029] where θ x and θ y are the rotation angles in the horizontal and vertical directions of the eyeball respectively, is the line-of-sight direction vector, F is the line-of-sight focus coordinate, R x and R y are the horizontal coordinate and vertical coordinate of the corneal reflection point respectively, P x and P y are the horizontal coordinate and vertical coordinate of the pupil center respectively, f is the focal length from the eyeball to the sensor, C is the eyeball center coordinate, and t is the intersection parameter of the line of sight and the plane.

[0030] Optionally, the intelligent scheduling module includes:

[0031] A resource monitoring unit, which is used to store resource usage-related data in a time series database, preset monitoring metrics, analyze user operation behaviors according to the monitoring metrics, and evaluate user resource requirements; the monitoring metrics include CPU / GPU utilization rate, memory occupancy rate, storage IO, network bandwidth, and network latency;

[0032] A resource dynamic allocation unit, which is used to introduce a reinforcement learning algorithm, learn resource allocation under different loads, realize dynamic adjustment of the resource allocation strategy, dynamically adjust the allocation of CPU / GPU according to the complexity and priority of user tasks, and use a distributed load balancing algorithm to allocate user tasks to different cloud nodes;

[0033] An exception handling unit, which is used to use an anomaly detection algorithm to identify resource leakage anomalies and resource overload anomalies. When an anomaly is detected, it automatically triggers resource recovery or task migration, and performs regular recovery processing on unused or low-utilization resources;

[0034] A resource pre-allocation unit, which is used to collect user historical operation data, extract features from the user historical operation data, construct a user behavior feature vector, predict changes in user resource requirements according to the user behavior feature vector using a time series prediction model, and perform pre-allocation and storage of resources according to the changes in user resource requirements;

[0035] A high-concurrency scenario optimization unit, which is used to divide tasks into critical tasks and non-critical tasks, and construct a resource pool to group and manage computing resources, storage resources, and network resources.

[0036] Optionally, the network optimization module includes:

[0037] A hierarchical edge computing unit is used to build local edge computing nodes deployed in the surrounding area of users and regional edge computing nodes deployed in the regional data center, assign real-time tasks to the local edge computing nodes for processing, assign intensive tasks to the regional edge computing nodes for processing, and use a distributed load balancing algorithm to allocate tasks among multiple edge computing nodes;

[0038] A data caching unit is used to cache the common data of the virtual desktop in the edge computing node to reduce the cloud access frequency;

[0039] An adaptive transmission unit is used to simultaneously transmit multiple data streams using the QUIC protocol, adjust the transmission window size according to network changes, and adjust the retransmission interval and number of times according to the packet loss rate.

[0040] Optionally, the security module includes:

[0041] An encryption unit is used to perform end-to-end encryption among the cloud, the client, and the edge computing node in combination with forward secrecy;

[0042] A security authentication unit is used to authenticate users in combination with passwords, biometrics, and dynamic tokens, and use a zero-trust security model to dynamically verify and adjust permissions for each access.

[0043] Optionally, the compatibility module includes:

[0044] An interactive transmission unit is used to define an interactive protocol and a transmission protocol through open APIs and communication protocols; the interactive protocol includes an input event format and an output data format, and the transmission protocol includes data priority and an error recovery mechanism;

[0045] A cross-platform framework unit is used to select the Flutter framework or the Qt framework for the development of client applications according to the client type

[0046] An application building unit is used to provide standard API interfaces and tool support using multi-platform SDKs, and provide API descriptions and example codes using development documents.

[0047] The present invention discloses the following technical effects by providing an augmented reality collaboration system for a cloud desktop:

[0048] 1. Deep integration of virtual reality: The present invention breaks the separation mode between traditional cloud desktops and AR systems, coordinates the remote end, the user client (device end), and edge computing, and realizes the deep integration of functions and resources. At the same time, the image data generated by the cloud desktop is integrated with the rendering pipeline of the AR device to ensure synchronous display of the picture and realize the seamless integration of the virtual desktop and the real scene.

[0049] 2. Low network dependence: The present invention develops an efficient real-time transmission protocol to support low-latency data exchange between cloud desktops and AR devices, ensuring the real-time operation of users and the smoothness of the screen. At the same time, edge computing nodes are used to process part of the AR rendering and interaction logic, reduce the load on the cloud and device ends, improve the response speed and stability of the system, and avoid the decline in user experience caused by network delays or insufficient bandwidth.

[0050] 3. High interaction accuracy: Through gesture recognition, voice control, eye tracking and other processes, multimodal interaction is achieved, and personalized recommendations can be made based on user behavior. According to environmental perception, the brightness and contrast of the virtual desktop can be automatically adjusted, and the interaction mode can be dynamically switched (such as from voice to gesture).

[0051] 4. Intelligent resource allocation: By real-time monitoring of cloud resource usage and user needs, dynamic resource allocation and predictive scheduling can be achieved, resource utilization can be optimized, user experience can be improved, and system stability and priority execution of key tasks can be guaranteed in high-concurrency scenarios.

[0052] 5. Optimize network transmission: By designing a layered edge computing architecture, it undertakes some computing and data forwarding tasks, reduces the number of network hops between the cloud and the device, reduces latency, and improves user experience. Through adaptive transmission based on the QUIC protocol, it optimizes transmission efficiency and reliability. By dynamically adjusting transmission parameters (such as window size, retransmission mechanism, etc.), it adapts to different network conditions and ensures low latency and high reliability.

[0053] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 A schematic diagram of a system framework provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of a process for generating a virtual desktop provided by an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the process of intelligent resource scheduling provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] As Figure 1 shown, the present invention provides an augmented reality collaboration system for a cloud desktop, including a cloud service module, an AR device, a collaborative rendering module, an interaction module, an intelligent scheduling module, a network optimization module, a security module, and a compatibility module that are interconnected.

[0061] 1. Cloud service module

[0062] It is used to run the virtual desktop infrastructure and cloud computing resources in the cloud, generate virtual desktop images, and perform real-time resource monitoring and data storage.

[0063] 2. AR device

[0064] It is used to collect gesture data, voice data, eye movement data, and environmental data, and realize the interaction between the user and the virtual desktop.

[0065] 3. Collaborative rendering module

[0066] As Figure 2 shown, it is used to perform AR rendering on the image of the cloud desktop in the cloud, and then, through the AR device, synthesize the rendered image into the real scene. The collaborative rendering module includes:

[0067] 3.1 Cloud desktop rendering adaptation unit

[0068] It is used to capture the virtual desktop image, encode and compress the virtual desktop image according to the network bandwidth, adjust the frame rate of the virtual desktop image according to the refresh rate of the AR device (such as 60Hz or 90Hz), and then, according to the display resolution (such as 1920x1080 or 2560x1440) and display characteristics (such as color gamut, brightness, contrast) of the AR device, perform scaling, cropping, edge optimization, and color correction on the virtual desktop image; wherein, during the frame rate adjustment process, when the virtual desktop image has a low frame rate, frame interpolation technology is used to generate intermediate frames to improve visual fluency.

[0069] 3.2 AR-side construction unit

[0070] For deploying a client application that supports cloud desktop protocols (such as RDP, PCoIP, HDX) on the AR device, introducing multi-threading technology to parallelly process image decoding and rendering tasks based on the client application, integrating an AR SDK toolkit (such as ARCore, ARKit) for spatial positioning of the virtual desktop image, using SLAM (Simultaneous Localization and Mapping) technology to fix the virtual desktop image at a specified position in the real scene, and then projecting the decoded virtual desktop image into the real scene; supporting the user to adjust the size, position, and angle of the virtual screen through gestures or voice.

[0071] 3.3 Cloud Communication Interface Unit

[0072] For real-time monitoring of the network bandwidth, latency, and packet loss rate between the client application and the cloud to obtain the network status, adjusting the bitrate and resolution of the virtual desktop image according to the network status (giving priority to ensuring the transmission of interactive data when the network bandwidth is insufficient), and using the UDP transmission protocol (to reduce transmission latency) and forward error correction technology (adding redundant information to data frames to support packet loss recovery) for multi-channel data transmission; wherein, the multi-channel data transmission includes an image data transmission channel, an interactive data transmission channel (prioritizing the transmission of interactive data to ensure the real-time nature of user operations), and a control data transmission channel.

[0073] 3.4 Collaborative Rendering Unit

[0074] For combining a GPU cluster in the cloud (such as NVIDIA Tesla or AMD Instinct to improve rendering efficiency), using distributed rendering technology to decompose complex graphics computing tasks into multiple GPU nodes for parallel processing, completing graphics computing and preliminary rendering, generating low-resolution preliminary image frames, transmitting the preliminary image frames to an edge computing device (such as NVIDIA Jetson or Intel Movidius) for frame optimization (such as anti-aliasing, light and shadow adjustment), then transmitting the optimized preliminary image frames to the client application for final rendering and display to obtain the final image frames, and projecting the final image frames into the real scene to complete spatial positioning and projection, ensuring synchronization with the user's perspective.

[0075] 4. Interaction Module

[0076] Such as Figure 2 As shown, it is used for gesture recognition, voice control, eye movement tracking, and environmental perception to achieve multi-modal interaction methods, and for personalized recommendation of interaction methods according to the user's operation mode. The interaction module includes:

[0077] 4.1 Gesture Recognition Unit

[0078] It is used to deploy a depth camera and sensors (such as ToF sensors or binocular cameras) in the AR device to capture the user's hand movements, collect hand depth information, bone structure features, and movement trajectories, generate high-quality hand data, and combine convolutional neural networks and recurrent neural networks to build a gesture recognition model. The gesture recognition model is used to recognize the user's gesture actions (such as clicking, dragging, zooming in and out, swiping, etc.) in the hand data, and then classify the user's gesture actions and map the operations to specific operations on the cloud desktop (such as opening an application, dragging a window, resizing a window, etc.).

[0079] 4.2 Voice Control Unit

[0080] It is used to integrate a voice recognition engine (such as Google Speech-to-Text or iFlytek speech recognition), use a pre-trained language model (such as BERT) to parse the user's voice commands, extract semantic information, convert the user's voice into text according to the semantic information, and design an extensible voice command library for automatic operation mapping; the extensible voice command library supports multiple expression methods (such as both "open browser" and "start Chrome" are mapped to the same operation).

[0081] 4.3 Eye Tracking Unit

[0082] It is used to deploy an eye movement sensor (such as an infrared camera) in the AR device to capture the user's line-of-sight trajectory, calculate the line-of-sight focus in the user's visual trajectory using optical reflection technology (Pupil Center Corneal Reflection, PCCR), geometric models, and deep learning algorithms, and map the line-of-sight focus to a specific position on the virtual desktop. For example, the user makes a line-of-sight selection by gazing at a certain interface element (such as a window, button), sets a gaze time threshold (such as 500ms) to avoid misoperations, and completes a confirmation operation by gazing at a certain area and combining gestures or voice, supporting combinations of line-of-sight and other interaction methods (such as "gaze + voice" or "gaze + gesture").

[0083] The expression of the line-of-sight focus is as follows:

[0084]

[0085]

[0086]

[0087] where, θ x and θ y are the rotation angles of the eyeball in the horizontal and vertical directions respectively, is the line-of-sight direction vector, F is the line-of-sight focus coordinate, R x and Ry are the horizontal and vertical coordinates of the corneal reflection point, respectively, P x and P y are the horizontal and vertical coordinates of the pupil center, respectively, f is the focal length from the eyeball to the sensor, C is the eyeball center coordinate, and t is the intersection parameter of the line of sight and the plane.

[0088] t satisfies the plane equation ax + by + cz + d = 0, that is:

[0089] a(C x + t·d x ) + b(C y + t·d y ) + c(C z + t·d z ) + d = 0

[0090] Solve for:

[0091]

[0092] Substitute t into to obtain the coordinates of the line of sight focus.

[0093] The role of deep learning algorithms in the calculation of the line of sight focus is mainly reflected in: 1) Pupil and reflection point detection, using convolutional neural networks to accurately detect the pupil center and corneal reflection point from the images captured by the eye movement sensor. For example, models based on YOLO or UNet are used for real-time detection. 2) Line of sight direction correction, analyzing the user's historical line of sight trajectory through a deep learning model to correct the jitter and errors in the line of sight direction. 3) Line of sight focus prediction, in complex three-dimensional scenes, the deep learning model can combine the line of sight direction and scene depth information to predict the user's line of sight focus.

[0094] Calculation process: Conduct eyeball modeling, calculate the initial line of sight direction vector, use the corneal reflection point and pupil center to calculate the rotation angle of the eyeball, and based on the rotation angle, calculate the real-time line of sight direction vector. Then perform intersection calculation between the real-time line of sight direction vector and the geometric model of the display plane or three-dimensional scene to obtain the line of sight focus.

[0095] 4.4 Personalized Recommendation Unit

[0096] Used to analyze the user's operation mode, and based on the analysis results, introduce matrix factorization technology (such as SVD) into the collaborative filtering algorithm based on user behavior to recommend operations or applications similar to the user's habits.

[0097] 4.5 Scene Perception Unit

[0098] It is used to collect the ambient data of the user by using the GPS, light sensor and microphone in the AR device, analyze the ambient data of the user, extract scene features (such as location, light intensity, noise level, etc.), and automatically adjust the brightness, contrast and interaction mode of the virtual desktop (such as switching from voice to gesture) according to the scene features.

[0099] 5. Intelligent Scheduling Module

[0100] Such as Figure 3 As shown, it is used to evaluate the user's resource requirements, dynamically allocate computing resources, storage resources and network resources according to the evaluation results, analyze the user's historical operation data, perform resource requirement prediction and resource pre-allocation, and build a resource pool to perform resource optimization management in high-concurrency scenarios. The intelligent scheduling module includes:

[0101] 5.1 Resource Monitoring Unit

[0102] It is used to store the resource usage-related data into a time series database (such as InfluxDB), preset monitoring metrics, analyze the user's operation behavior (such as opened applications, used functions, interaction frequency, etc.) according to the monitoring metrics, and evaluate the user's resource requirements; the monitoring metrics include CPU / GPU utilization rate, memory occupancy rate, storage IO, network bandwidth, network latency, etc.;

[0103] 5.2 Resource Dynamic Allocation Unit

[0104] It is used to introduce a reinforcement learning algorithm (such as Deep Q-Learning), learn how to allocate resources under different load conditions to achieve optimal performance, realize the dynamic adjustment of the resource allocation strategy, dynamically adjust the allocation of CPU / GPU according to the complexity and priority of the user's tasks (for example, allocate more GPU computing power for high-priority tasks), and use a distributed load balancing algorithm (such as Consistent Hashing) to allocate the user's tasks to different cloud nodes to avoid single-node overload.

[0105] 5.3 Exception Handling Unit

[0106] It is used to use an anomaly detection algorithm (such as Isolation Forest) to identify resource leakage anomalies and resource overload anomalies. When an anomaly is detected, it automatically triggers resource recovery or task migration, and performs regular recovery processing on unused or under-utilized resources to release computing, storage and network bandwidth.

[0107] 5.4 Resource Pre-allocation Unit

[0108] For collecting historical operation data of users (such as used applications, operation frequencies, interaction patterns, etc.), extracting key features (such as task types, resource consumption patterns, time periods, etc.) from the historical operation data of users, constructing user behavior feature vectors, predicting changes in user resource requirements using a time series prediction model (such as LSTM or Transformer) based on the user behavior feature vectors, and making early allocation and storage of resources according to the changes in user resource requirements.

[0109] 5.5 High-concurrency Scenario Optimization Unit

[0110] For classifying tasks into critical tasks (such as real-time rendering, interactive operations) and non-critical tasks (such as background data synchronization), and constructing a resource pool to group and manage computing resources, storage resources, and network resources. In a high-concurrency scenario, dynamically adjust the allocation ratio of the resource pool to ensure sufficient resources for critical tasks.

[0111] 6. Network Optimization Module

[0112] For designing a hierarchical edge architecture, performing task allocation, and introducing the QUIC protocol for optimizing the transmission of data streams. The network optimization module includes:

[0113] 6.1 Hierarchical Edge Computing Unit

[0114] For constructing local edge computing nodes deployed in the surrounding area of users and regional edge computing nodes deployed in the regional data center, allocating real-time tasks (such as gesture recognition, eye movement tracking) to the local edge computing nodes for processing, allocating intensive tasks (such as complex rendering, data aggregation) to the regional edge computing nodes for processing, and using a distributed load balancing algorithm to perform task allocation among multiple edge computing nodes to avoid single-node overload.

[0115] 6.2 Data Caching Unit

[0116] For caching common data of virtual desktops (such as application programs, user configurations) in edge computing nodes to reduce the frequency of cloud access.

[0117] 6.3 Adaptive Transmission Unit

[0118] For simultaneously transmitting multiple data streams using the QUIC protocol, adjusting the transmission window size according to network changes to avoid network congestion, and adjusting the retransmission interval and number of times according to the packet loss rate. For example, in a high packet loss rate scenario, give priority to transmitting critical data (such as interactive data) and delay the retransmission of non-critical data.

[0119] QUIC is a UDP-based transport protocol with the characteristics of low latency, fast connection establishment, and multiplexing. Using the multiplexing feature of QUIC, multiple data streams (such as image data and interactive data) can be transmitted simultaneously in a single connection. When a packet loss occurs in a certain data stream, it does not affect the transmission of other data streams, thus improving the overall transmission efficiency. Utilizing the 0-RTT connection establishment feature of QUIC reduces the time overhead for connection establishment.

[0120] 7. Security Module

[0121] It is used to encrypt and transmit data by combining forward secrecy and end-to-end encryption technology, and to verify user identities and access. The security module includes:

[0122] 7.1 Encryption Unit

[0123] It is used to perform end-to-end encryption among the cloud, client, and edge computing nodes by combining forward secrecy.

[0124] The encryption protocol can select the latest TLS1.3 protocol to achieve end-to-end encryption, ensuring the security of data during transmission. The TLS1.3 protocol has: a faster handshake speed (0-RTT handshake), reducing latency. Stronger encryption algorithms (such as AES-GCM, ChaCha20-Poly1305), enhancing the data encryption strength. Removing insecure algorithms (such as RC4, MD5) to prevent known attacks.

[0125] 7.2 Security Authentication Unit

[0126] It is used to authenticate users by combining passwords, biometrics, and dynamic tokens, and to dynamically verify and adjust permissions for each access using the zero-trust security model.

[0127] Password authentication: The user enters a password as the first layer of authentication.

[0128] Biometric authentication: Fingerprint, iris, or facial recognition is used as the second layer of authentication.

[0129] Dynamic token authentication: One-time dynamic passwords (OTP) or hardware tokens (such as YubiKey) are used as the third layer of authentication.

[0130] Zero-trust security model: It is a brand-new network security strategy. Its core idea is to default to not trusting anyone, anything inside or outside the enterprise network. Verification and minimum privilege granting are performed before authorization. This model breaks the traditional security protection mode based on network boundaries and believes that no internal or external entity, including users, devices, applications, etc., can be trusted in the network environment.

[0131] 8. Compatibility Module

[0132] It is used to define standardized interfaces and protocols, develop client applications adapted to different operating systems, provide SDKs and development documents, and achieve multi-platform support; the compatibility module includes:

[0133] 8.1 Interactive Transmission Unit

[0134] It is used to define an interaction protocol through open APIs and communication protocols (such as OpenXR, WebRTC, etc.) to support multiple input methods (such as gestures, voice, eye movement) and output methods (such as virtual desktops, 3D models), and define a transmission protocol to support efficient and reliable data transmission.

[0135] The interaction protocol includes: 1) Input event format, which defines a unified format (such as JSON or Protobuf) for input events such as gestures, voice, and eye movement. 2) Output data format, which defines the transmission format of virtual desktop rendering data (such as H.264 video stream, 3D model file).

[0136] The transmission protocol includes: 1) Data priority, which defines the priority of different types of data (such as real-time rendering data prior to background synchronization data). 2) Error recovery mechanism, which defines a retransmission mechanism in case of data loss to ensure transmission reliability.

[0137] OpenXR: It is an open standard developed by the Khronos Group, aiming to unify the development interfaces of virtual reality (VR) and augmented reality (AR) devices. It supports cross-device and cross-platform XR application development, avoiding repeated development by developers for different devices.

[0138] WebRTC: It is an open-source real-time communication protocol that supports low-latency transmission of audio and video streams and data streams. It is suitable for real-time interaction between cloud desktops and terminal devices.

[0139] 8.2 Cross-Platform Framework Unit

[0140] It is used to select the Flutter framework or the Qt framework for the development of client applications according to the client type.

[0141] Flutter: It supports high-performance UI rendering and is suitable for Android and iOS client development.

[0142] Qt: It supports C++ development and is suitable for Windows and Linux client development.

[0143] 8.3 Application Building Unit

[0144] Multi-platform SDK:

[0145] API Wrapper: Provide standard API interfaces to simplify the calling process for developers.

[0146] Tool Support: Provide debugging tools and performance analysis tools to help developers optimize the application.

[0147] Development Documentation:

[0148] API Reference: Describe in detail the functions, parameters, and return values of each API.

[0149] Sample Code: Provide sample code for multiple platforms to help developers get started quickly.

[0150] Therefore, by providing an augmented reality collaboration system for cloud desktops, the present invention can deeply integrate functions and resources, solve problems such as network dependence, interaction accuracy, and resource allocation, provide the best AR presentation and experience effects, and support various application scenarios.

[0151] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0152] Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An augmented reality collaboration system for a cloud desktop, characterized in that: include: The cloud service module is used to run the virtual desktop infrastructure and cloud computing resources in the cloud, generate virtual desktop images, and perform real-time resource monitoring and data storage; AR devices, used to collect gesture data, voice data, eye movement data, and environmental data, and to enable interaction between users and virtual desktops; A collaborative rendering module, used to perform AR rendering on the cloud desktop image in the cloud, and then synthesize the rendered image into the real scene through the AR device; The interactive module is used for gesture recognition, voice control, eye tracking and environmental perception to achieve multi-modal interaction and make personalized recommendations based on user operation modes. Intelligent scheduling module, which is used to evaluate user resource requirements, dynamically allocate computing resources, storage resources, and network resources based on the evaluation results, analyze user historical operation data, predict resource requirements and pre-allocate resources, and build resource pools to optimize resource management in high-concurrency scenarios; The network optimization module is used to design a layered edge architecture, perform task allocation, and introduce the QUIC protocol to optimize the transmission of data streams; Security module, which is used to encrypt data transmission and verify user identity and access by combining forward security and end-to-end encryption technology; Compatibility module, which is used to define standardized interfaces and protocols, develop client applications that adapt to different operating systems, provide SDK and development documents, and achieve multi-platform support; Among them, the cloud service module, the AR device, the collaborative rendering module, the interaction module, the intelligent scheduling module, the network optimization module, the security module, and the compatibility module are interconnected.

2. The augmented reality collaboration system of cloud desktop according to claim 1, characterized in that: The collaborative rendering module includes: A cloud desktop rendering adaptation unit is used to capture the virtual desktop image, encode and compress the virtual desktop image according to the network bandwidth, adjust the frame rate of the virtual desktop image according to the refresh rate of the AR device, and then scale, crop, optimize edges and perform color correction on the virtual desktop image according to the display resolution and display characteristics of the AR device; wherein, during the frame rate adjustment process, when the virtual desktop image has a low frame rate, an intermediate frame is generated using a frame rate interpolation technology to improve visual fluency; An AR end construction unit is used to deploy a client application supporting the cloud desktop protocol on the AR device, introduce multi-threading technology to perform image decoding and rendering tasks based on the client application, integrate the AR SDK toolkit to perform spatial positioning of the virtual desktop image, use SLAM technology to fix the virtual desktop image at a specified position in the real scene, and then project the decoded virtual desktop image into the real scene; A cloud communication interface unit is used to monitor the network bandwidth, delay and packet loss rate of the client application and the cloud in real time, obtain the network status, adjust the bit rate and resolution of the virtual desktop image according to the network status, and use the UDP transmission protocol and forward error correction technology to perform multi-channel data transmission; wherein the multi-channel data transmission includes an image data transmission channel, an interactive data transmission channel and a control data transmission channel; The collaborative rendering unit is used to combine with the GPU cluster in the cloud, use distributed rendering technology to decompose complex graphics computing tasks into multiple GPU nodes for parallel processing, complete graphics computing and preliminary rendering, generate low-resolution preliminary image frames, transmit the preliminary image frames to the edge computing device for frame optimization, and then transmit the optimized preliminary image frames to the client application for final rendering and display to obtain the final image frames, and project the final image frames into the real scene to complete spatial positioning and projection.

3. The augmented reality collaboration system of cloud desktop according to claim 2, characterized in that: The interaction module includes: A gesture recognition unit is used to deploy a depth camera and a sensor in the AR device to capture the user's hand movements, collect hand depth information, bone structure features and motion trajectories, generate hand data, and build a gesture recognition model in combination with a convolutional neural network and a recurrent neural network. The gesture recognition model is used to recognize the user's gesture movements in the hand data, and then the user's gesture movements are gesture classified and mapped; A voice control unit, which is used to integrate a voice recognition engine, parse the user's voice commands using a pre-trained language model, extract semantic information, convert the user's voice into text based on the semantic information, and design an extensible voice command library for automatic operation mapping; An eye tracking unit, configured to deploy an eye sensor in the AR device, capture a user's visual trajectory, calculate a visual focus in the user's visual trajectory using optical reflection technology, a geometric model, and a deep learning algorithm, and map the visual focus to a specific position in the virtual desktop; Personalized recommendation unit, which is used to analyze user operation patterns and, based on the analysis results, introduce matrix decomposition technology into the collaborative filtering algorithm based on user behavior to recommend operations or applications similar to user habits; The scene perception unit is used to use the GPS, light sensor and microphone in the AR device to collect user surrounding environment data, analyze the user surrounding environment data, extract scene features, and automatically adjust the virtual desktop brightness, contrast and interaction mode according to the scene features.

4. The augmented reality collaboration system of cloud desktop according to claim 3, characterized in that: The expression of the sight focus is: Among them, θ x and θ y are the horizontal and vertical rotation angles of the eyeball, is the sight direction vector, F is the sight focus coordinate, R x and R y are the horizontal and vertical coordinates of the corneal reflection point, P x and P y are the horizontal and vertical coordinates of the pupil center, f is the focal length from the eyeball to the sensor, C is the coordinate of the eyeball center, and t is the intersection parameter of the line of sight and the plane.

5. The augmented reality collaboration system of cloud desktop according to claim 4, characterized in that: The intelligent scheduling module includes: A resource monitoring unit is used to store resource usage related data in a time series database and preset monitoring indicators, analyze user operation behaviors according to the monitoring indicators, and evaluate user resource requirements; the monitoring indicators include CPU / GPU utilization, memory occupancy, storage IO, network bandwidth, and network latency; The resource dynamic allocation unit is used to introduce reinforcement learning algorithms, learn resource allocation under different load conditions, realize dynamic adjustment of resource allocation strategies, dynamically adjust CPU / GPU allocation according to the complexity and priority of user tasks, and use distributed load balancing algorithms to allocate user tasks to different cloud nodes; The exception handling unit is used to use the exception detection algorithm to identify resource leakage anomalies and resource overload anomalies. When an anomaly is detected, it automatically triggers resource recycling or task migration, and regularly recycles unused or low-utilization resources. A resource pre-allocation unit, used to collect user historical operation data, extract features from the user historical operation data, construct a user behavior feature vector, use a time series prediction model to predict user resource demand changes based on the user behavior feature vector, and pre-allocate and store resources based on the user resource demand changes; The high-concurrency scenario optimization unit is used to divide tasks into critical tasks and non-critical tasks, build resource pools, and manage computing resources, storage resources, and network resources in groups.

6. The augmented reality collaboration system of cloud desktop according to claim 5, characterized in that: The network optimization module includes: A hierarchical edge computing unit is used to construct local edge computing nodes deployed in the area around the user and regional edge computing nodes deployed in the regional data center, allocate real-time tasks to the local edge computing nodes for processing, allocate intensive tasks to the regional edge computing nodes for processing, and use a distributed load balancing algorithm to distribute tasks among multiple edge computing nodes; A data cache unit is used to cache common data of virtual desktops in edge computing nodes to reduce the frequency of cloud access; Adaptive transmission unit, used to transmit multiple data streams simultaneously using the QUIC protocol, adjust the transmission window size according to network changes, and adjust the retransmission interval and number according to the packet loss rate.

7. The augmented reality collaboration system of cloud desktop according to claim 6, characterized in that: The security module comprises: Cryptographic unit for end-to-end encryption between the cloud, client, and edge computing nodes in combination with forward secrecy; The security authentication unit is used to combine passwords, biometrics, and dynamic tokens to authenticate users, and uses a zero-trust security model to dynamically verify and adjust permissions for each access.

8. The cloud desktop augmented reality collaboration system according to claim 7, characterized in that: The compatible modules include: An interactive transmission unit, used to define an interactive protocol and a transmission protocol through an open API and a communication protocol; the interactive protocol includes an input event format and an output data format, and the transmission protocol includes a data priority and an error recovery mechanism; Cross-platform framework unit, used to select Flutter framework or Qt framework for client application development according to the client type The application building unit is used to provide standard API interfaces and tool support using multi-platform SDKs, and to provide API descriptions and sample codes using development documentation.

Images