Cloud mobile phone interaction method and related equipment

Abstract

The invention discloses a cloud mobile phone interaction method and related equipment, and relates to the technical field of cloud computing, and the method comprises the steps: collecting and determining operation data based on a multi-mode sensor of terminal equipment; performing context compression coding on the operation data based on a preset semantic communication model to generate a compression instruction stream; transmitting the compressed instruction stream to an edge computing node based on a 6G network, and determining a priority marking result based on an intention awareness protocol stack; based on the user operation historical data and the priority marking result, path prediction and pre-rendering processing are carried out on the edge computing node, and a multi-version pre-rendering frame sequence is generated; determining an optimal rendering frame from the multi-version pre-rendering frame sequence based on physical simulation calculation of a cloud server and a dynamic buffering mechanism; and performing time delay compensation on the optimal rendering frame based on the network state monitoring data to generate a target interaction picture.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a cloud phone interaction method and related equipment. Background Art

[0002] With the rapid development of cloud computing and mobile communication technologies, the demand for the application of cloud phone systems in scenarios such as extended reality and industrial Internet of Things is increasing. However, existing technologies face challenges in actual deployment: the traditional centralized cloud rendering architecture has a long data transmission path and multiple processing levels, which leads to a prominent problem of time delay superposition between operation instructions and screen feedback. Especially in interactive scenarios that require high real-time performance, the end-to-end delay often exceeds the user's acceptable range. In addition, the hardware differences of heterogeneous terminal devices lead to unstable encoding and decoding efficiency, and the network transmission delay and jitter problems further aggravate the freeze and distortion of the interactive screen, making it difficult to meet the stringent requirements of time-sensitive services such as the tactile Internet for millisecond-level response. Therefore, there is an urgent need for a cloud phone interaction method to solve the above-mentioned technical problems. Summary of the Invention

[0003] The Summary of the Invention introduces a series of simplified concepts that will be further described in the Detailed Description of the Invention. The Summary of the Invention of this application is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0004] In a first aspect, the present application provides a cloud phone interaction method, the method comprising:

[0005] Determine operation data based on the multimodal sensor collection of the terminal device, wherein the operation data includes touch coordinate changes, pressure parameters and inertial measurement data;

[0006] Perform context compression encoding on the operation data based on a preset semantic communication model to generate a compressed instruction stream;

[0007] Transmit the compressed instruction stream to the edge computing node based on the 6G network, and determine the priority marking results based on the intent-aware protocol stack;

[0008] Based on user operation history data and priority marking results, path prediction and pre-rendering processing are performed at the edge computing node to generate multi-version pre-rendered frame sequences;

[0009] Determine the optimal rendering frame from multiple versions of pre-rendered frame sequences based on physical simulation calculations and dynamic buffering mechanisms on cloud servers;

[0010] Based on network status monitoring data, the optimal rendering frame is delayed and compensated to generate the target interactive screen.

[0011] In some embodiments, determining operational data based on multimodal sensor acquisition of a terminal device includes:

[0012] collecting touch coordinate changes and pressure parameters based on a touch pressure sensor, wherein a sampling rate of the touch pressure sensor is greater than or equal to a first preset threshold;

[0013] Collect inertial measurement data based on an inertial measurement unit, wherein the inertial measurement data includes spatial attitude parameters and motion acceleration of the terminal device;

[0014] Based on the preset time synchronization protocol, the touch coordinate change, pressure parameters and inertial measurement data are time-aligned to determine the synchronous operation data;

[0015] The synchronous operation data is denoised and fused based on a preset filtering algorithm to generate operation data.

[0016] In some embodiments, contextual compression encoding is performed on the operation data based on a preset semantic communication model to generate a compressed instruction stream, including:

[0017] Perform semantic association analysis on touch coordinate changes based on the knowledge graph to determine the operation intention characteristics;

[0018] Perform joint encoding processing based on operation intention features and pressure parameters to generate differential compression instructions;

[0019] Performing spatial attitude compression processing on inertial measurement data based on the Lie group SE (3) coding model to determine spatial attitude compression data;

[0020] Based on the preset data integration rules, the differential compression instructions and the spatial posture compression data are stream-integrated to generate a compressed instruction stream.

[0021] In some embodiments, transmitting the compressed instruction stream to the edge computing node based on the 6G network and determining the priority marking result based on the intent-aware protocol stack includes:

[0022] Based on the 6G network transmission protocol, the compressed instruction stream is processed for transmission path selection to determine the target edge computing node;

[0023] Perform emergency instruction detection and processing on the compressed instruction stream based on the intent-aware protocol stack to determine the instruction type label;

[0024] Based on the preset service quality policy, the instruction type label is prioritized and mapped to determine the priority marking result.

[0025] In some embodiments, based on user operation history data and priority marking results, path prediction and pre-rendering processing are performed at the edge computing node to generate a multi-version pre-rendered frame sequence, including:

[0026] Perform time series analysis on the operation trajectory of the terminal device based on the user's operation history data to determine the historical trajectory characteristics;

[0027] Based on the long short-term memory neural network model, the path prediction calculation of the historical trajectory features is performed to determine the predicted operation path;

[0028] Perform trajectory fitting processing on the predicted operation path based on the Bezier curve generator to generate a predicted trajectory curve;

[0029] Based on the priority marking results, the predicted trajectory curve is assigned rendering priority to determine the target rendering area;

[0030] Perform multi-precision rendering processing on the target rendering area based on the LOD rendering algorithm to generate pre-rendered frames at different LOD levels;

[0031] Based on the preset frame sequence rules, version integration processing is performed on pre-rendered frames of different LOD levels to determine a multi-version pre-rendered frame sequence.

[0032] In some embodiments, determining the optimal rendered frame from multiple versions of pre-rendered frame sequences based on physical simulation calculations and a dynamic buffering mechanism on a cloud server includes:

[0033] Determine physical simulation requirements based on rendering parameters of multiple versions of pre-rendered frame sequences;

[0034] Based on physical simulation requirements, collision detection and ray tracing calculations are performed on multiple versions of pre-rendered frame sequences to generate physical simulation parameters;

[0035] Determine real-time network delay and jitter indicators based on a dynamic buffering mechanism;

[0036] Based on real-time network latency and jitter indicators, frame screening strategy matching is performed on multiple versions of pre-rendered frame sequences to determine the candidate frame set;

[0037] The optimal rendering frame is determined based on the physical simulation parameters and the rendering quality evaluation results of the candidate frame set.

[0038] In some embodiments, performing delay compensation processing on the optimal rendering frame based on network status monitoring data to generate a target interactive screen includes:

[0039] Perform real-time analysis and processing of current network delay and jitter parameters based on network status monitoring data to determine delay compensation requirements;

[0040] Based on the delay compensation requirements, the optimal rendering frame is processed to generate the optical flow transition frame and determine the transition frame sequence;

[0041] Perform inverse kinematic correction calculations on the user's operating posture based on the inertial measurement data of the terminal device to determine the posture correction parameters;

[0042] Perform dynamic image fusion processing on the transition frame sequence based on the posture correction parameters to determine the corrected rendering frame;

[0043] The corrected rendering frame is subjected to picture quality verification based on a preset rendering quality threshold to determine the target interactive picture.

[0044] In a second aspect, the present application provides a cloud phone interaction device, comprising:

[0045] an operation data determining unit, which collects and determines operation data based on a multimodal sensor of the terminal device, wherein the operation data includes a touch coordinate change, a pressure parameter, and an inertial measurement data;

[0046] A compression instruction generation unit performs context compression encoding on the operation data based on a preset semantic communication model to generate a compressed instruction stream;

[0047] The marking result determination unit transmits the compressed instruction stream to the edge computing node based on the 6G network and determines the priority marking result based on the intent-aware protocol stack;

[0048] The pre-rendered frame generation unit performs path prediction and pre-rendering processing at the edge computing node based on user operation history data and priority marking results, generating multi-version pre-rendered frame sequences;

[0049] The optimal rendering frame determination unit determines the optimal rendering frame from multiple versions of pre-rendered frame sequences based on the physical simulation calculation and dynamic buffering mechanism of the cloud server;

[0050] The target screen generation unit performs delay compensation processing on the optimal rendering frame based on the network status monitoring data to generate the target interactive screen.

[0051] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the cloud phone interaction method of any one of the first aspects when executing the computer program stored in the memory.

[0052] In a fourth aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, which implements the cloud phone interaction method of any one of the first aspects when the computer program is executed by a processor.

[0053] In summary, this application optimizes the interactive delay and picture quality through multimodal sensor data fusion acquisition, semantic communication model compression coding, 6G network efficient transmission, and edge and cloud collaborative computing. Specifically, based on context compression coding and priority scheduling of the intent-aware protocol stack, data transmission redundancy is reduced and network resource utilization is improved; through path prediction and multi-version pre-rendering technology combined with a dynamic buffering mechanism, accurate prediction and real-time adjustment of rendering tasks are achieved; at the same time, physical simulation calculations and delay compensation processing further ensure high fidelity and low latency of picture output. The synergistic effect of the above technologies effectively solves the problems of high end-to-end delay and insufficient picture smoothness in traditional solutions, providing users with an immersive interactive experience with millisecond-level response. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present description. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0055] Figure 1 A schematic diagram of a cloud phone interaction method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of the structure of a cloud phone interaction device provided in an embodiment of the present application;

[0057] Figure 3 A structural diagram of a cloud phone interactive electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments.

[0059] See also Figure 1 , which is a flow chart of a cloud phone interaction method provided in an embodiment of the present application, which may specifically include:

[0060] S110, collecting and determining operation data based on a multimodal sensor of the terminal device, wherein the operation data includes touch coordinate change, pressure parameters, and inertial measurement data;

[0061] For example, during the interaction with a cloud phone, the terminal device uses multimodal sensors to collaboratively collect the user's operation data, providing basic input for subsequent interactive processing. The touch pressure sensor captures the user's touch coordinate changes and pressure parameters in real time, such as the displacement difference and pressure gradient of the finger sliding trajectory; the inertial measurement unit (IMU) synchronously collects the spatial posture parameters and motion acceleration of the terminal device to accurately reflect the user's operation dynamics in three-dimensional space. Through the parallel operation of multimodal sensors, the system can fully perceive the user's interaction intentions in the dimensions of touch, pressure, and spatial motion, laying a data foundation for building a low-latency, high-precision interactive closed loop.

[0062] The collection and preliminary processing of the above-mentioned operation data are aimed at achieving efficient digital representation of user interaction behavior. The touch coordinate change represents the user's direct operation trajectory on the interface, the pressure parameter maps the user's operation intensity to support dynamic rendering optimization, and the inertial measurement data captures the user's indirect operation intention (such as device shaking and tilt) through spatial motion characteristics. The collaborative collection of multimodal data not only improves the ability to analyze operation intentions, but also ensures the timeliness of the data through high sampling rates (such as touch sensors ≥1kHz, IMU ≥1.2kHz), providing low-latency, high-fidelity original input for subsequent semantic compression, edge pre-rendering and other links, thereby supporting the optimization of full-link interactive performance.

[0063] S120, performing context compression encoding on the operation data based on a preset semantic communication model to generate a compressed instruction stream;

[0064] For example, the core of the semantic communication model is to extract high-level semantic features from the operation data through knowledge graph and context association analysis to achieve efficient data compression. The model first performs semantic association analysis on the touch coordinate changes, combines the user's operation history with the domain knowledge graph, identifies the core features of the operation intention (such as sliding direction, click mode, etc.), and jointly encodes them with the pressure parameters to generate differential compression instructions; at the same time, the Lie group SE (3) encoding model is used to perform structured compression on the spatial posture information in the inertial measurement data to eliminate the geometric redundancy in the motion parameters. Through semantic-level feature extraction and multimodal data joint encoding, the original high-dimensional operation data is converted into a low-redundancy semantic instruction stream.

[0065] The generation of compressed instruction streams optimizes transmission efficiency through layered semantic integration. Differentially compressed instructions (representing changes in operational intent) and spatial posture compressed data (describing the device's motion state) are stream-encapsulated based on preset data integration rules, forming a contextually linked instruction sequence. This process not only reduces data volume but also preserves the dynamic relevance of operational intent, ensuring that the instruction stream possesses both high semantic density and network adaptability. This provides structured input for efficient subsequent 6G network transmission and intent-based processing at edge nodes, laying the critical data foundation for ultra-low latency interaction.

[0066] S130, transmitting the compressed instruction stream to the edge computing node based on the 6G network, and determining the priority marking result based on the intent-aware protocol stack;

[0067] For example, 6G networks achieve efficient transmission of compressed instruction streams through ultra-high-speed transmission at the physical layer and a dynamic resource scheduling mechanism. Leveraging the high bandwidth characteristics of the millimeter-wave frequency band and the beamforming enhancement technology of the Reconfigurable Smart Surface (RIS), compressed instruction streams are directed to the target edge computing node, reducing air interface latency. At the same time, the network layer uses an intent-aware protocol stack and, through a time-sensitive service priority preemption mechanism, dynamically selects the optimal transmission path, ensuring low-jitter, high-reliability data transmission and providing network support for real-time processing at edge computing nodes.

[0068] The intent-aware protocol stack intelligently identifies the urgency of commands based on the semantic characteristics of the compressed command stream and business scenario requirements. By analyzing command type tags (such as touch operations and physical simulation requests) and combining them with the priority marking results mapped by the preset Quality of Service (QoS) policy, it distinguishes between critical and non-critical commands. This mechanism enables edge nodes to prioritize high-priority tasks (such as urgent interaction commands), optimize computing resource allocation, shorten end-to-end processing latency, and support the closed-loop implementation of millisecond-level real-time interaction.

[0069] S140: Based on the user operation history data and priority marking results, path prediction and pre-rendering processing are performed at the edge computing node to generate a multi-version pre-rendered frame sequence;

[0070] For example, edge computing nodes predict future user operation paths by analyzing the temporal characteristics of historical user operation data and combining it with priority tagging results. They use a long short-term memory (LSTM) neural network model to explore the underlying patterns of historical trajectories, predict possible interaction paths, and generate smooth predicted trajectories through Bezier curve fitting. Furthermore, based on priority tagging results (such as urgent instructions or high-interaction frequency areas), they dynamically allocate rendering resources and determine target rendering areas that require priority processing, providing a decision-making basis for subsequent multi-precision rendering.

[0071] Pre-rendering uses a level-of-detail (LOD) rendering algorithm to generate pre-rendered frame sequences of varying precision based on the predicted path and priority allocation results. By adaptively adjusting the number of polygons and lighting details in the rendering area, multiple versions of candidate frames, ranging from high to low precision, are generated and integrated into a coherent frame sequence according to preset rules. This multi-version pre-rendering mechanism prepares a variety of possible interactive screens in advance, providing flexible input for the cloud to dynamically select the optimal rendering frame, reducing the risk of screen freezes caused by network fluctuations or computational delays.

[0072] S150, determining the optimal rendering frame from multiple versions of pre-rendered frame sequences based on physical simulation calculation and dynamic buffering mechanism of the cloud server;

[0073] For example, the cloud server performs high-precision physical simulation calculations on the pre-rendered frame sequence through a heterogeneous computing cluster (such as FPGA and ray tracing card collaboration), including the generation of key physical interaction parameters such as collision detection and global illumination simulation. Based on the dynamic buffering mechanism, the server monitors the network transmission delay and jitter indicators in real time, and combines the rendering quality of the pre-rendered frame with the physical simulation results to dynamically select the optimal rendering frame that meets the current network status and terminal performance. For example, when the network is highly loaded, frames with low bit rates but complete physical simulation are given priority to ensure a balance between picture continuity and physical reality.

[0074] The coordinated optimization of physical simulation and dynamic buffering achieves a global trade-off between rendering quality and transmission efficiency. By verifying the interactive rationality of pre-rendered frames using physical simulation parameters and combining them with the adaptive buffering mechanism, the system can quickly match the optimal frame sequence even in fluctuating network environments. This mechanism not only ensures the generation of high-fidelity interactive images, but also reduces transmission pressure from the cloud to the terminal through elastic buffering, providing a high-quality input foundation for terminal latency compensation.

[0075] S160: Compensate for the delay of the optimal rendering frame based on the network status monitoring data to generate a target interactive screen.

[0076] For example, the terminal device performs latency compensation on the received optimal rendering frame based on real-time network status monitoring data to offset interaction deviations introduced by network transmission. The optical flow engine analyzes the motion vectors between adjacent frames and dynamically generates transitional frame sequences to fill in the image gaps caused by latency. Simultaneously, the device uses inertial measurement data to reversely calculate the real-time correction value of the user's operating posture and perform spatial posture calibration on the rendered frame to ensure spatiotemporal consistency between the image and the user's operating intention.

[0077] This compensation mechanism achieves smooth transitions and precise synchronization of interactive screens through multi-layer collaborative control. A dynamic buffer pool adaptively adjusts the frame buffering strategy based on network latency and jitter parameters. Combined with the sub-millisecond response of the haptic feedback module, it ultimately generates a target interactive screen that matches the user's actions in real time. This end-to-end latency compensation closed loop effectively reduces user-perceivable deviations in the interactive screen and ensures an immersive experience in highly real-time interactive scenarios.

[0078] In summary, in the embodiment of the present application, multimodal sensors are used to collaboratively collect touch, pressure and inertial measurement data, and the semantic communication model is used to contextually compress and encode the operation intention, thereby reducing the data redundancy of the instruction stream; relying on the high-speed transmission of the 6G network and the dynamic priority marking of the intent-aware protocol stack, key instructions are ensured to reach the edge node with low latency; based on the path prediction and multi-version pre-rendering technology of the user's historical operation mode, candidate frame sequences of different accuracy levels are generated in advance, effectively alleviating the real-time computing pressure on the cloud; cloud-based physical simulation and dynamic buffering mechanism are collaboratively optimized, combined with the network state adaptive frame screening strategy, while ensuring high-fidelity picture quality and balancing transmission efficiency; the terminal side uses optical flow transition frame generation and inverse kinematics correction to accurately compensate for the interaction deviation caused by network delay, and ultimately achieve millisecond-level end-to-end response. This application solves the problems of high interaction delay, insufficient rendering accuracy, and network fluctuations in traditional cloud phones, and provides a high-real-time and high-consistency immersive interactive experience for scenarios such as extended reality and industrial control.

[0079] In some examples, determining operational data based on multimodal sensor collection of a terminal device includes:

[0080] collecting touch coordinate changes and pressure parameters based on a touch pressure sensor, wherein a sampling rate of the touch pressure sensor is greater than or equal to a first preset threshold;

[0081] Collect inertial measurement data based on an inertial measurement unit, wherein the inertial measurement data includes spatial attitude parameters and motion acceleration of the terminal device;

[0082] Based on the preset time synchronization protocol, the touch coordinate change, pressure parameters and inertial measurement data are time-aligned to determine the synchronous operation data;

[0083] The synchronous operation data is denoised and fused based on a preset filtering algorithm to generate operation data.

[0084] Exemplarily, the terminal device captures the coordinate changes (Δx / Δy) and pressure parameters of the user's touch operation in real time through a high-precision touch pressure sensor. The sampling rate of the touch sensor is not lower than the first preset threshold (for example, ≥1kHz) to ensure accurate capture of operations such as fast sliding and clicking. The touch coordinates are differentially encoded (such as 4 bits to represent the Δx / Δy difference), and the pressure value is quantized by gradient (such as 0-15 levels, 4-bit encoding), which effectively reduces the amount of raw data. The data collected by the sensor directly reflects the user's explicit operation trajectory and interaction strength on the interface, providing basic input for dynamic rendering optimization.

[0085] The terminal device integrates a multi-axis MEMS inertial measurement unit (IMU) to collect the device's spatial attitude parameters (including pitch, roll, and yaw angles) and three-dimensional motion acceleration at a high sampling rate (e.g., ≥1.2kHz). The IMU data is structured and compressed using the Lie group SE(3) encoding model, compressing the 9-axis raw data (e.g., 2.4KB / frame) to a low-dimensional representation (e.g., 0.8KB / frame), eliminating geometric redundancy in the motion parameters. This data can accurately represent the user's indirect operational intentions in three-dimensional space (e.g., device shaking and tilting), enhancing the multi-dimensional analysis capability of operational intentions.

[0086] To address the heterogeneous timing characteristics of touch, pressure, and inertial measurement data, a preset time synchronization protocol (such as the IEEE1588v2 Precision Clock Synchronization Protocol) is used to time-align multi-source data. Through hardware timestamps and software interpolation compensation, touch coordinate changes, pressure gradients, and spatial posture parameters are synchronized within a microsecond time window (deviation <10μs). This time synchronization effectively avoids intent parsing errors caused by timing misalignment, providing a consistent input foundation for subsequent data fusion and semantic analysis.

[0087] The synchronized operation data is denoised using a preset filtering algorithm (such as Kalman filtering or low-pass filtering) to eliminate sensor noise and environmental interference. For example, high-frequency vibration noise in the inertial measurement data is filtered in the frequency domain, and touch coordinate jitter is smoothed. Subsequently, a weighted fusion strategy is used to multi-dimensionally correlate the touch trajectory, pressure gradient, and spatial motion features to generate a low-noise, highly consistent set of operation data. The fused data contains both the precise coordinates of the planar operation instructions and the implicit intention of the pressure intensity and the device's motion state, providing a high signal-to-noise ratio input for semantic compression.

[0088] Through the above process, the system achieves efficient collection and preprocessing of multimodal sensor data. The high sampling rate ensures the real-time nature of the data, time synchronization and denoising fusion improve data quality, and compression encoding reduces the transmission load. The resulting set of operation data (including touch Δx / Δy, pressure, posture angle, and acceleration) can fully characterize the user's interaction intentions in plane operation, force control, and spatial motion dimensions, providing an accurate, low-redundancy input source for the efficient compression of subsequent semantic communication models, path prediction of edge nodes, and cloud-based physical simulation, laying the underlying data foundation for ultra-low latency interaction.

[0089] In some examples, contextual compression encoding is performed on the operation data based on a preset semantic communication model to generate a compressed instruction stream, including:

[0090] Perform semantic association analysis on touch coordinate changes based on the knowledge graph to determine the operation intention characteristics;

[0091] Perform joint encoding processing based on operation intention features and pressure parameters to generate differential compression instructions;

[0092] Performing spatial attitude compression processing on inertial measurement data based on the Lie group SE (3) coding model to determine spatial attitude compression data;

[0093] Based on the preset data integration rules, the differential compression instructions and the spatial posture compression data are stream-integrated to generate a compressed instruction stream.

[0094] Exemplarily, the semantic communication model extracts semantic-level features of touch coordinate changes (Δx / Δy) through a pre-built domain knowledge graph (such as an industrial design ontology library or a human-computer interaction behavior model). A mapping is established between the entity relationships in the knowledge graph and the operation mode library (such as typical interaction behaviors such as sliding, clicking, and zooming). Combined with the user's historical operation data, the potential intention features of the current touch operation (such as "swiping left to turn the page" and "long pressing to trigger the menu") are identified. For example, when the continuous negative offset of Δx exceeds the threshold, the "page return" intention label is associated. This analysis converts the low-dimensional coordinate sequence into a high-semantic-density operation intention feature vector, reducing data redundancy.

[0095] The operation intention features obtained from the semantic analysis are jointly encoded with the touch pressure parameters. Differential coding technology is used to transmit only the change in the intention feature (such as the intention switch flag, direction increment), rather than the complete feature vector; the pressure value is non-uniformly quantized (4-bit representation) according to the preset gradient (such as 0-15 levels), and the fine resolution corresponding to high-intensity pressing is retained first. Differential compression instructions are generated through joint coding to achieve dynamic associative expression of intention features and operation strength. For example, when a high pressure value is combined with the "drag" intention, a higher priority rendering instruction is triggered.

[0096] The 6-DOF spatial posture data (translation vector + rotation matrix) collected by the IMU is structured and compressed using the Lie group SE(3) encoding model. Specifically, the instantaneous posture of the terminal device is represented as an SE(3) group element (containing the rotation matrix R∈SO(3) and the translation vector ), the continuous motion trajectory is parameterized into coefficients (e.g., 6-dimensional vectors) of the Lie algebra se(3) through exponential mapping. After scalar quantization and entropy coding, these coefficients generate compressed spatial attitude data (e.g., 0.8KB / frame). Compared to the original IMU data (e.g., 2.4KB / frame), the compression rate is improved by 66.7%, while retaining motion continuity characteristics (e.g., angular velocity smoothness), providing high-precision input for path prediction of edge nodes.

[0097] Differential compression instructions (representing operation intentions and pressure changes) and spatial posture compression data (describing the device's motion state) are stream-encapsulated according to preset data integration rules. The rules include priority sorting, context association, and metadata embedding. Priority sorting is to prioritize the encapsulation of urgent instructions (such as DSCP=46 mark) into the instruction stream header; context association is to bind the touch intention, pressure gradient, and spatial posture data within the same time window to form a spatiotemporal data block; metadata embedding is to add timestamps, data checksums, and compression parameter identifiers (such as SE(3) encoding version numbers). The integrated compressed instruction stream has both semantic integrity and transmission adaptability, supporting efficient transmission of 6G networks and fast parsing of edge nodes.

[0098] Through the synergy of semantic association analysis, joint encoding, and structured compression, the original multimodal operation data is converted into a compressed instruction stream with low redundancy and high semantic density. The knowledge graph-driven touch instruction stream is compressed from the original 1.2MB to 320KB, and Lie group SE(3) encoding compresses the IMU data to 33% of its original size. The end-to-end transmission delay is reduced from 8.2ms to 2.3ms, and joint encoding further eliminates cross-modal redundancy. The generated instruction stream ensures timing consistency through streaming encapsulation and provides structured input for priority marking of the intent-aware protocol stack and path prediction of edge nodes. In addition, structured encapsulation supports cross-layer optimization, ensuring the transmission priority of urgent instructions, ultimately achieving the technical goal of millisecond-level interactive response, and laying the core data foundation for ultra-low latency interaction.

[0099] In some instances, the compressed instruction stream is transmitted to the edge computing node based on the 6G network, and the priority marking results are determined based on the intent-aware protocol stack, including:

[0100] Based on the 6G network transmission protocol, the compressed instruction stream is processed for transmission path selection to determine the target edge computing node;

[0101] Perform emergency instruction detection and processing on the compressed instruction stream based on the intent-aware protocol stack to determine the instruction type label;

[0102] Based on the preset service quality policy, the instruction type label is prioritized and mapped to determine the priority marking result.

[0103] Exemplarily, the compressed instruction stream is transmitted through the millimeter wave frequency band of the 6G network (such as a 28GHz carrier), and the physical layer adopts orthogonal time-frequency-space (OTFS) modulation technology to combat the Doppler effect and multipath fading of high-frequency band signals. The network layer dynamically determines the target edge computing node through the transmission path selection algorithm based on real-time network status monitoring data (such as delay heat map and node load rate). For example, when it is detected that the air interface delay of path A is 1.2ms and the bandwidth is sufficient, it is allocated to this path first; if the delay jitter of path B is greater than 3ms due to congestion, it automatically switches to the low-load slice channel. During the transmission process, the reconfigurable intelligent surface (RIS) array dynamically adjusts the beam direction to compensate for the path loss (such as achieving 23dB coverage gain) to ensure high-reliability transmission of the instruction stream.

[0104] The intent-aware protocol stack performs semantic-level feature analysis on the compressed instruction stream to identify the type of emergency instructions. By analyzing the metadata tags in the instruction stream (such as operation intention characteristics and pressure gradient levels) and context-related information (such as the acceleration trend of continuous sliding instructions), combined with predefined business scenario rules (such as the time-sensitive business needs of the tactile Internet), the command types that need to be processed first (such as "gesture zoom" and "emergency stop") are detected. The detection results are mapped to standardized command type tags (such as DSCP=46 marks emergency commands) and embedded in the instruction stream header to provide a semantic basis for subsequent priority scheduling.

[0105] The QoS policy library defines the priority levels and resource allocation rules for different instruction type tags. Policy rules include latency-sensitive instructions, high-fidelity rendering instructions, and regular interaction instructions. Latency-sensitive instructions (such as DSCP=46) are mapped to the highest priority (Level 0), triggering the network layer priority preemption mechanism and allocating dedicated transmission channels and computing resources; high-fidelity rendering instructions (such as LOD3 pre-rendered frame requests) are mapped to the second-highest priority (Level 1) to ensure bandwidth and computing latency < 2ms; regular interaction instructions (such as page sliding) are mapped to standard priority (Level 2) and dynamically allocated according to remaining resources. The priority marking results are synchronized to the edge nodes and cloud servers through control plane signaling to achieve cross-layer resource collaborative scheduling.

[0106] Through 6G network transmission path optimization, emergency command recognition in the intent-aware protocol stack, and priority mapping of QoS policies, end-to-end latency is reduced and service reliability is improved. This mechanism ensures that end-to-end latency is strictly controlled within millisecond thresholds, meeting the real-time requirements of time-sensitive services.

[0107] In some instances, based on user operation history data and priority marking results, path prediction and pre-rendering processing are performed at the edge computing node to generate multiple versions of pre-rendered frame sequences, including:

[0108] Perform time series analysis on the operation trajectory of the terminal device based on the user's operation history data to determine the historical trajectory characteristics;

[0109] Based on the long short-term memory neural network model, the path prediction calculation of the historical trajectory features is performed to determine the predicted operation path;

[0110] Perform trajectory fitting processing on the predicted operation path based on the Bezier curve generator to generate a predicted trajectory curve;

[0111] Based on the priority marking results, the predicted trajectory curve is assigned rendering priority to determine the target rendering area;

[0112] Perform multi-precision rendering processing on the target rendering area based on the LOD rendering algorithm to generate pre-rendered frames at different LOD levels;

[0113] Based on the preset frame sequence rules, version integration processing is performed on pre-rendered frames of different LOD levels to determine a multi-version pre-rendered frame sequence.

[0114] Exemplarily, the edge computing node extracts historical trajectory features based on user operation history data (such as touch coordinate sequence and inertial measurement data) through time series analysis algorithms (such as autoregressive integrated moving average model, ARIMA). Specifically, the operation data is divided into multiple segments according to the time window (such as 300ms), and the statistical characteristics of each segment (such as sliding speed mean, acceleration variance, contact distribution density) are calculated, and a spatiotemporal feature matrix is constructed. The matrix is reduced in dimension through principal component analysis to generate a low-dimensional feature vector, which characterizes the periodicity, directional preference and interaction intensity of the user operation, and provides input for path prediction.

[0115] The long short-term memory (LSTM) neural network model receives historical trajectory feature vectors as input. Its hidden layer (e.g., 64 nodes) learns the temporal dependencies of the operation trajectory through a gating mechanism (forget gate, input gate, and output gate). The model outputs a probability distribution of the user's future operation path (e.g., a predicted sequence of touch point coordinates within the next 200ms). A softmax layer generates a prediction confidence score (e.g., >90% triggers pre-rendering). During training, the mean squared error (MSE) loss function is used to optimize network parameters, and the backpropagation algorithm is used to dynamically adjust weights to ensure that the deviation between the predicted path and the actual operation trajectory is less than 1.2mm.

[0116] The predicted operation path is smoothly fitted using a Bezier curve generator. Specifically, the discrete coordinate points of the predicted path are converted into control points, and a continuous trajectory curve is generated using the cubic Bezier curve equation. During the curve fitting process, the number of control points is adaptively adjusted (for example, one control point is inserted every 5ms) to ensure that the trajectory curvature is consistent with the smoothness of the user's historical operations. The fitted trajectory curve is output as a parameterized equation that supports real-time interpolation calculation of the predicted touch point coordinates at any time, providing a geometric basis for rendering priority allocation.

[0117] Based on the priority marking results (such as emergency instructions with DSCP=46), the edge node assigns rendering priority to the interface area covered by the predicted trajectory curve. The priority allocation algorithm dynamically divides the priority areas according to the instruction type (such as tactile feedback, interface update) and trajectory space coverage (such as the percentage of the trajectory coverage area in the screen area). For example, the trajectory area corresponding to the emergency touch instruction is marked as high priority and allocated 80% of the rendering resources; high-frequency interaction areas (such as areas with dense predicted trajectories) are marked as medium priority and allocated 15% of the resources; background static areas are marked as low priority and allocated 5% of the resources. At the same time, the regional rendering granularity is adjusted based on the trajectory speed parameters to ensure the continuity of the picture.

[0118] The level of detail (LOD) rendering algorithm dynamically adjusts the rendering accuracy based on the priority of the target area and the touch pressure parameters. For high-priority areas, the LOD3 high-precision rendering strategy is adopted (such as the number of polygons N = 500 × (1 + log2 (p + 1), where p is the pressure value) to increase model details and lighting complexity; for medium-priority areas, the LOD2 medium-precision strategy is adopted (when 3≤p<10, N = 1200) to simplify light and shadow calculations; for low-priority areas, the LOD1 low-precision strategy is adopted (when p<3, N = 500) to reduce rendering accuracy and use simplified textures and low-resolution shading. The rendering engine (such as the adaptive mesh subdivision unit) processes areas of different LOD levels in parallel to generate pre-rendered frames containing geometric details, texture mapping, and lighting parameters.

[0119] Pre-rendered frames at different LOD levels (e.g., LOD1 high precision, LOD2 medium precision, LOD3 low precision) are generated through a parallel rendering pipeline. Each frame contains complete geometry, material, and lighting information, and is embedded with metadata (e.g., LOD level and rendering time). The rendering engine uses an asynchronous computing mode, prioritizing the generation of high-priority frames while pre-calculating low-priority frames as backup. After frame generation, qualified frames are screened through a quality assessment module, and distorted or timed-out frames are removed to ensure the availability and consistency of the pre-rendered frame sequence.

[0120] Different versions of pre-rendered frames are integrated according to preset frame sequence rules (such as timestamp alignment and LOD level gradient). Specifically, frame sequences are grouped by time windows, each group contains candidate frames of 3 LOD levels, and indexes are established through header fields. The integrated multi-version frame sequence is stored in the DDR5 video memory pool (such as 8GB capacity) of the edge node, and the least recently used cache replacement strategy is used to manage frame data. When the network status changes, the candidate frame version is dynamically switched to ensure that the terminal device can quickly obtain the optimal rendering frame adapted to the current network conditions.

[0121] Through the synergy of path prediction, priority allocation and multi-precision rendering, the system reduces the computing latency of edge nodes (from 22.4ms to 5.1ms) and improves resource utilization. The dynamic generation mechanism of pre-rendered frame sequences (such as switching LOD levels according to network status) effectively responds to latency fluctuations, ensuring automatic downgrade to LOD1 frames when the end-to-end latency is greater than 5ms, and the screen freeze rate is reduced by 77.2%. This application provides high-quality, multi-granularity input sources for cloud-based physical simulation and terminal latency compensation, supporting the realization of millisecond-level immersive interactive experience.

[0122] In some instances, the optimal rendering frame is determined from multiple versions of pre-rendered frame sequences based on physical simulation calculations and a dynamic buffering mechanism on a cloud server, including:

[0123] Determine physical simulation requirements based on rendering parameters of multiple versions of pre-rendered frame sequences;

[0124] Perform collision detection and ray tracing calculations on multiple versions of pre-rendered frame sequences based on physical simulation requirements to generate physical simulation parameters;

[0125] Determine real-time network delay and jitter indicators based on a dynamic buffering mechanism;

[0126] Based on real-time network latency and jitter indicators, frame screening strategy matching is performed on multiple versions of pre-rendered frame sequences to determine the candidate frame set;

[0127] The optimal rendering frame is determined based on the physical simulation parameters and the rendering quality evaluation results of the candidate frame set.

[0128] For example, after receiving multiple versions of pre-rendered frame sequences, the cloud server parses the physical simulation requirements based on the rendering parameters of each frame (such as the number of polygons, the complexity of the lighting model, and the resolution of the material map). For high-precision frames (such as LOD3 level), global illumination calculations and high-density collision detection need to be performed; for medium and low-precision frames (such as LOD2 / LOD1), the physical interaction parameters need to be simplified. For example, the LOD3 frame needs to call the FPGA-accelerated Neural Radiation Field (NeRF) technology to generate radiation field data at a ray tracing speed of 2.1G Rays / s, while the LOD1 frame only needs to perform basic collision detection based on a simplified rigid body dynamics model. This step uses a hierarchical simulation strategy to accurately match rendering accuracy and computing resource requirements to avoid excessive consumption of cloud computing power.

[0129] Cloud-based heterogeneous computing clusters (e.g., FPGAs + RTX A6000 ray tracing cards) perform parallel physics simulation calculations on pre-rendered frame sequences. The collision detection module uses the Bullet physics engine's hardware logic unit to perform real-time collision detection on 50,000 polygonal models (taking 1.2ms / frame), outputting collision point coordinates, normal vectors, and energy loss parameters. The ray tracing module uses CUDA cores to calculate global illumination in parallel, generating physical simulation data (e.g., radiation field metadata) containing diffuse reflection, specular reflection, and shadow parameters. Simulation results are compressed and transmitted using SM4 encryption to ensure data security and transmission efficiency.

[0130] The dynamic buffering mechanism collects network delay and jitter parameters in real time through the timestamp marking unit deployed on the network interface card (synchronization error <10ns), including air interface transmission delay (1.2ms±0.3ms), edge and cloud backhaul delay (1.9ms±0.6ms) and jitter standard deviation (<0.5ms). The buffer controller dynamically adjusts the video stream GOP structure (such as the I / P / B frame ratio) based on the H.266 standard and configures an 8GB DDR5 video memory pool to store multiple versions of pre-rendered frame sequences. When network jitter >3ms is detected, it automatically switches to low-bitrate candidate frames (LOD1 level) and triggers the FPGA resource reallocation strategy to prioritize delay-sensitive frames.

[0131] Based on real-time network indicators (latency, packet loss rate, bandwidth fluctuation) and pre-rendered frame properties (bitrate, resolution, and physical simulation completeness), the cloud implements a multi-dimensional frame screening strategy: low-latency mode (latency < 5ms): selects LOD3 high-fidelity frames to retain complete physical interaction details; high-jitter mode (jitter > 2ms): switches to LOD1 low-bitrate frames to ensure picture continuity; balanced mode: uses LOD2 frames, taking into account both fidelity and transmission stability. The screening algorithm combines a weighted scoring model (such as 70% weight allocated to physical simulation parameters and 30% to network adaptability) to generate a candidate frame set from multiple versions of frame sequences. The filtered candidate frame set eliminates distorted or timed frames to ensure that each frame in the set meets the preset latency and quality constraints.

[0132] The candidate frames are then evaluated by the rendering quality assessment module using multiple metrics. These include physical fidelity (verifying that collision detection results and ray tracing parameters meet preset thresholds (e.g., shadow accuracy error <0.1mm); visual fidelity (calculating peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to ensure PSNR ≥ 40dB and SSIM ≥ 0.92); and latency compliance (verifying that the frame generation timestamp matches the network transmission latency (deviation <1ms). After normalization, the frame with the highest overall score is marked as the optimal rendering frame and preferentially transmitted to the terminal via the 6G network slice channel.

[0133] Through the coordinated optimization of physical simulation computing and dynamic buffering mechanisms, end-to-end latency from the cloud to the terminal is reduced (from 48.6ms to 7.9ms), while simultaneously improving image quality (PSNR increased by 4.2dB). The elastic frame filtering strategy reduces the lag rate by 77.2% in scenarios with fluctuating network conditions, and FPGA-accelerated physical simulation increases computing density to 5.3 times that of traditional GPU solutions. This mechanism provides highly reliable input for terminal latency compensation, ensuring millisecond-level response and a highly immersive interactive experience for users, meeting the demanding requirements of time-sensitive services such as extended reality and industrial control.

[0134] In some instances, delay compensation is performed on the optimal rendering frame based on network status monitoring data to generate the target interactive screen, including:

[0135] Perform real-time analysis and processing of current network delay and jitter parameters based on network status monitoring data to determine delay compensation requirements;

[0136] Based on the delay compensation requirements, the optimal rendering frame is processed to generate the optical flow transition frame and determine the transition frame sequence;

[0137] Perform inverse kinematic correction calculations on the user's operating posture based on the inertial measurement data of the terminal device to determine the posture correction parameters;

[0138] Perform dynamic image fusion processing on the transition frame sequence based on the posture correction parameters to determine the corrected rendering frame;

[0139] The corrected rendering frame is subjected to picture quality verification based on a preset rendering quality threshold to determine the target interactive picture.

[0140] For example, the terminal device monitors the network status in real time through the delay heat map generated by Spirent TestCenter, and collects the current network delay (such as 1.2ms±0.3ms) and jitter parameters (such as jitter standard deviation <0.6ms). Based on the preset threshold rules (for example, delay >3ms or jitter >2ms triggers compensation), the delay compensation requirement level is dynamically determined. For example, when the delay deviation is detected to be >5ms, the highest level compensation mode is activated, and the joint compensation strategy of optical flow transition frame generation and inverse kinematics correction is activated to ensure that the user-perceivable operation screen deviation is less than 1.3mm.

[0141] To address image discontinuities caused by latency, the terminal optical flow engine (such as that implemented in OpenGL ES 3.2) performs motion vector analysis on the optimal rendered frame. The Lucas-Kanade algorithm is used to calculate the pixel displacement field between adjacent frames, and combined with temporal interpolation, a transitional frame sequence is generated (e.g., a single frame processing time of 0.8ms). During interpolation, the motion continuity of the visual focus area (ROI annotated based on eye tracking data) is prioritized, while low-resolution interpolation (1 / 4 pixel sampling) is used for non-focus areas, ensuring image smoothness while reducing the computational load.

[0142] Based on the 9-axis attitude data (acceleration, angular velocity, magnetometer) collected by the terminal IMU, the Lie Group SE(3) model is used to inversely infer the real-time correction of the user's operating attitude. The device posture deviation (such as the rotation matrix error ΔR∈SO(3)) is calculated using the Newton-Euler dynamics equation and mapped to the displacement compensation (Δx', Δy') of the screen coordinate system. For example, when the device tilt angle deviation is detected to be greater than 0.5°, the inverse kinematic correction is triggered to generate the attitude correction parameters (Δx'=2px, Δy'=-1px) to eliminate the screen misalignment caused by device shaking.

[0143] The optical flow transition frame sequence and pose correction parameters are fed into the dynamic image fusion module. A weighted blending algorithm dynamically adjusts the transparency of the transition frames and the weight of the correction parameters based on the latency level. For example, with a 5ms latency deviation, the transition frame weight is set to 70% and the correction parameter weight is 30%. With a 10ms latency deviation, the transition frame weight increases to 90%. The fused corrected rendered frame uses bilinear interpolation to optimize edge jaggedness and ensure geometric consistency.

[0144] The corrected rendered frames must pass preset quality verification rules, including visual fidelity, physical rationality, and latency compliance. Among them, visual fidelity is to calculate PSNR ≥ 40dB and SSIM ≥ 0.92 to ensure the accuracy of light and shadow details and color; physical rationality is to verify the matching degree between the collision detection results and the corrected motion trajectory (error < 0.2mm); latency compliance is to check the synchronization error between the frame timestamp and the network transmission timestamp (< 1ms). Frames that do not meet the standards are marked as invalid, triggering the edge node to re-push the pre-rendered frame or downgrade to a low-precision version (LOD1). The target interactive screen that passes the verification is stored in the dynamic buffer pool (8GB DDR5 video memory) and supports high refresh rate (≥ 120Hz) output.

[0145] Through the closed-loop compensation mechanism of optical flow interpolation, inverse correction and dynamic fusion, the system reduces the end-to-end latency from 48.6ms to 7.9ms, and the user operation screen response deviation is less than 1.3mm. The piezoelectric tactile feedback module (response time 0.4ms) synchronously generates a force feedback signal (such as a pressing force error of ±0.05N) to achieve multimodal synchronization of vision and touch. The interactive screen finally generated achieves a balance in the three dimensions of latency, fidelity and physical rationality, providing an immersive experience without perceived delay for time-sensitive scenarios such as the metaverse and industrial control, verifying the technical advantages and practical value of this application in the field of ultra-low latency interaction.

[0146] See also Figure 2 , is a schematic diagram of the structure of a cloud phone interaction device provided in an embodiment of the present application, including:

[0147] An operation data determining unit 21 collects and determines operation data based on a multimodal sensor of the terminal device, wherein the operation data includes touch coordinate changes, pressure parameters, and inertial measurement data;

[0148] The compression instruction generation unit 22 performs context compression encoding on the operation data based on a preset semantic communication model to generate a compressed instruction stream;

[0149] The marking result determining unit 23 transmits the compressed instruction stream to the edge computing node based on the 6G network, and determines the priority marking result based on the intent-aware protocol stack;

[0150] The pre-rendered frame generation unit 24 performs path prediction and pre-rendering processing at the edge computing node based on the user operation history data and priority marking results to generate a multi-version pre-rendered frame sequence;

[0151] The optimal rendering frame determining unit 25 determines the optimal rendering frame from the multiple versions of pre-rendered frame sequences based on the physical simulation calculation and dynamic buffering mechanism of the cloud server;

[0152] The target screen generation unit 26 performs delay compensation processing on the optimal rendering frame based on the network status monitoring data to generate a target interactive screen.

[0153] See also Figure 3 An embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored on the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, the steps of any method of cloud phone interaction are implemented.

[0154] Since the electronic device introduced in this embodiment is a device used to implement a cloud phone interaction device in the embodiment of this application, based on the method introduced in the embodiment of this application, technical personnel in this field can understand the specific implementation method of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of this application will not be introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of this application falls within the scope of protection of this application.

[0155] During the specific implementation process, when the computer program 311 is executed by the processor, any implementation method of the embodiments corresponding to the first aspect can be implemented.

[0156] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0157] Those skilled in the art will appreciate that embodiments of the present application may provide methods, systems, or computer program products. Thus, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-readable storage media containing computer-readable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] The present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes Figure 1 The process of a cloud phone interaction method in the corresponding embodiment.

[0162] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium can be a magnetic medium, an optical medium or a semiconductor medium, etc.

[0163] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0165] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware and / or software functional units.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device to execute all or part of the steps of the various embodiments of the method of the present application.

[0168] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0169] Although the preferred embodiments of this specification have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this specification.

[0170] Obviously, those skilled in the art may make various changes and modifications to this specification without departing from the spirit and scope of this specification. Thus, if such changes and modifications fall within the scope of the claims of this specification and their equivalents, this specification is intended to include such changes and modifications.

Claims

1. A cloud phone interaction method, characterized in that: include: Determining operation data based on the multimodal sensor of the terminal device, wherein the operation data includes touch coordinate changes, pressure parameters, and inertial measurement data; Performing context compression encoding on the operation data based on a preset semantic communication model to generate a compressed instruction stream; Transmitting the compressed instruction stream to an edge computing node based on a 6G network, and determining a priority marking result based on an intent-aware protocol stack; Based on the user operation history data and the priority marking result, path prediction and pre-rendering processing are performed at the edge computing node to generate a multi-version pre-rendered frame sequence; Determining the optimal rendering frame from the multiple versions of pre-rendered frame sequences based on physical simulation calculations and a dynamic buffering mechanism on a cloud server; The optimal rendering frame is subjected to delay compensation processing based on network status monitoring data to generate a target interactive screen.

2. The method according to claim 1, characterized in that The determining of the operation data by collecting the multimodal sensor of the terminal device includes: collecting touch coordinate changes and pressure parameters based on a touch pressure sensor, wherein a sampling rate of the touch pressure sensor is greater than or equal to a first preset threshold; Collecting inertial measurement data based on an inertial measurement unit, wherein the inertial measurement data includes spatial attitude parameters and motion acceleration of the terminal device; Performing time alignment processing on the touch coordinate change, the pressure parameter, and the inertial measurement data based on a preset time synchronization protocol to determine synchronization operation data; The synchronous operation data is subjected to denoising and fusion processing based on a preset filtering algorithm to generate operation data.

3. The method according to claim 1, characterized in that The performing context compression encoding on the operation data based on a preset semantic communication model to generate a compressed instruction stream includes: Perform semantic association analysis on touch coordinate changes based on the knowledge graph to determine the operation intention characteristics; Performing joint encoding processing based on the operation intention feature and the pressure parameter to generate a differential compression instruction; Performing spatial attitude compression processing on the inertial measurement data based on a Lie group SE (3) encoding model to determine spatial attitude compression data; The differential compression instruction and the spatial posture compression data are subjected to stream integration processing based on a preset data integration rule to generate a compression instruction stream.

4. The method according to claim 1, wherein The method of transmitting the compressed instruction stream to the edge computing node based on the 6G network and determining the priority marking result based on the intent-aware protocol stack includes: Performing transmission path selection processing on the compressed instruction stream based on the 6G network transmission protocol to determine the target edge computing node; Performing emergency instruction detection processing on the compressed instruction stream based on the intent-aware protocol stack to determine an instruction type tag; Priority mapping is performed on the instruction type label based on a preset quality of service policy to determine a priority marking result.

5. The method according to claim 1, wherein Based on the user operation history data and the priority marking result, path prediction and pre-rendering processing are performed at the edge computing node to generate a multi-version pre-rendered frame sequence, including: Perform time series analysis on the operation trajectory of the terminal device based on the user's operation history data to determine the historical trajectory characteristics; Performing path prediction calculation on the historical trajectory features based on a long short-term memory neural network model to determine a predicted operation path; Performing trajectory fitting processing on the predicted operation path based on a Bezier curve generator to generate a predicted trajectory curve; Performing rendering priority allocation processing on the predicted trajectory curve based on the priority marking result to determine a target rendering area; Perform multi-precision rendering processing on the target rendering area based on the LOD rendering algorithm to generate pre-rendered frames at different LOD levels; The pre-rendered frames at different LOD levels are subjected to version integration processing based on a preset frame sequence rule to determine a multi-version pre-rendered frame sequence.

6. The method according to claim 1, characterized in that The cloud server-based physical simulation calculation and dynamic buffering mechanism determines the optimal rendering frame from the multiple versions of the pre-rendered frame sequence, including: Determining physical simulation requirements based on rendering parameters of the multiple versions of pre-rendered frame sequences; Performing collision detection and ray tracing calculations on the multiple versions of pre-rendered frame sequences based on the physical simulation requirements to generate physical simulation parameters; Determine real-time network delay and jitter indicators based on a dynamic buffering mechanism; Performing frame screening strategy matching processing on the multi-version pre-rendered frame sequence based on the real-time network delay and the jitter index to determine a candidate frame set; An optimal rendering frame is determined based on the physical simulation parameters and a rendering quality evaluation result of the candidate frame set.

7. The method according to claim 1, characterized in that The performing delay compensation processing on the optimal rendering frame based on the network status monitoring data to generate a target interactive screen includes: Perform real-time analysis and processing of current network delay and jitter parameters based on network status monitoring data to determine delay compensation requirements; Performing optical flow transition frame generation processing on the optimal rendering frame based on the delay compensation requirement to determine a transition frame sequence; Perform inverse kinematic correction calculations on the user's operating posture based on the inertial measurement data of the terminal device to determine the posture correction parameters; Performing dynamic image fusion processing on the transition frame sequence based on the posture correction parameters to determine a corrected rendering frame; A picture quality verification process is performed on the modified rendering frame based on a preset rendering quality threshold to determine a target interactive picture.

8. A cloud phone interactive device, characterized in that: include: an operation data determining unit, configured to collect and determine operation data based on a multimodal sensor of the terminal device, wherein the operation data includes a touch coordinate change, a pressure parameter, and an inertial measurement data; A compression instruction generating unit, performing context compression encoding on the operation data based on a preset semantic communication model to generate a compression instruction stream; a marking result determining unit, which transmits the compressed instruction stream to the edge computing node based on the 6G network and determines a priority marking result based on the intent-aware protocol stack; A pre-rendered frame generation unit performs path prediction and pre-rendering processing at an edge computing node based on user operation history data and the priority marking result to generate a multi-version pre-rendered frame sequence; An optimal rendering frame determining unit, which determines an optimal rendering frame from the multiple versions of pre-rendered frame sequences based on physical simulation calculations and a dynamic buffering mechanism of a cloud server; The target screen generation unit performs delay compensation processing on the optimal rendering frame based on the network status monitoring data to generate a target interactive screen.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that the processor is used to implement the steps of the cloud phone interaction method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cloud phone interaction method according to any one of claims 1 to 7 is implemented.

Images