Cloud rendering message communication interaction method for connecting multi-terminal digital twin business scenarios

Abstract

The application discloses a cloud rendering message communication interaction method for connecting multi-end digital twin business scenarios. The method is realized based on a three-layer architecture system of a client layer, a middleware layer and a server layer. The client layer collects the parameters of the three-dimensional camera's view cone in real time; the middleware layer uses the parameters, combines spatial semantic index and distance attenuation law, calculates the dynamic scheduling weight of massive concurrent messages, and accurately distributes them to high-optimal transmission, smooth flow limiting or aggregation degradation channels; at the same time, the server layer reuses the same set of view cone parameters to implement cascade rendering degradation and physical simulation control on twin objects outside the field of view and in the buffer zone. The application not only effectively eliminates the head-of-line blocking effect at the network transmission level, but also realizes the spatio-temporal coupling linkage of message scheduling and computing power distribution based on unified spatial perception, and supports seamless switching of heterogeneous cloud rendering platforms.

Description

Technical Field

[0001] This invention relates to the field of digital twin and cloud rendering technology, and in particular to a cloud rendering message communication and interaction method that connects multiple digital twin business scenarios. It is applicable to application scenarios that require real-time interaction and cloud rendering collaboration of 3D digital twin scenes on multiple terminal devices such as web browsers, mobile terminals or desktop applications. Background Technology

[0002] In recent years, the rapid development of information technology has driven increased demand for visualization-based decision support systems across various industries. As business scenarios become more complex, users are placing increasingly stringent requirements on 3D visualization effects, real-time interactive responses, and multi-terminal collaboration capabilities. Simultaneously, the trend towards centralized management of computing resources and hardware has spurred the widespread application of cloud rendering technology. This technology lowers the performance threshold for terminal devices by transferring high-load graphics computing tasks to cloud servers, providing a technological foundation for the efficient presentation of large-scale 3D scenes.

[0003] Traditional solutions typically involve developing business pages directly using UI controls built into game engines like 3D engines or embedded browser components, then packaging them with the 3D scene and releasing them uniformly via cloud rendering technology. However, this approach suffers from significant drawbacks, including limited visualization effects, high coupling of interactive logic, and complex cross-platform adaptation. To overcome these bottlenecks, the industry has begun to explore separating the front-end business pages from the 3D scene, utilizing a message communication development kit provided by cloud rendering software to achieve dynamic interaction between the two. While this improvement enhances system flexibility and scalability to some extent, it still faces numerous challenges in practical engineering applications. The interplay of factors such as varying scales across different business scenarios, compatibility with heterogeneous operating system environments, efficiency of dynamic hardware resource scheduling, and project budget constraints makes it difficult for a single cloud rendering architecture to simultaneously meet performance, cost, and maintainability requirements. Especially in scenarios involving multi-user concurrency, high-precision model rendering, and low-latency interaction, traditional solutions are prone to resource allocation imbalances and decreased communication efficiency.

[0004] Specifically, the first shortcoming of the existing solution lies in the tight coupling between business code and the underlying communication protocol. When developing digital twin applications, business personnel need to first select a specific third-party cloud rendering platform, and then use the corresponding message communication development package of the platform for business interaction development. Once the cloud rendering platform needs to be changed due to cost, performance, or maintenance reasons, a large amount of business interaction code has to be re-modified. This tight coupling design severely restricts the system's portability and platform flexibility.

[0005] The second drawback is the cumbersome debugging process. Existing solutions require the 3D engine program to be packaged and released before the digital twin scene can be linked to the front-end page for debugging using the message communication development kit. If scene content needs to be edited or modified, such as adding data layers or adjusting scene configuration parameters, the entire packaging and uploading process must be re-executed. This prevents rapid iterative debugging within the engine editor environment, increasing development time and debugging costs.

[0006] The third shortcoming lies in the complexity of command transmission and the lack of built-in object management. Existing cloud rendering platforms typically only provide basic message sending interfaces. Developers need to define complex JSON data structures and front-end / back-end parsing rules for different categories of objects involved in digital twin applications, such as scenes, layers, tools, and effects. Because the platform itself lacks effective built-in support for managing these objects, developers also need to maintain the object model themselves when calling commands, making the development and maintenance process quite cumbersome.

[0007] The fourth shortcoming lies in a severe bottleneck at the message transmission level. Existing cloud rendering communication middleware generally adopts a blind full synchronization strategy, lacking the ability to perceive the three-dimensional spatial field of view and the priority of business semantics. Digital twin scenarios typically contain a large number of dynamic objects, such as tens of thousands of IoT monitoring sensors in water conservancy projects. The full synchronization strategy leads to redundant messages from a large number of non-critical objects that are out of the user's view or obscured, crowding out limited network bandwidth and creating a head-of-line congestion effect, causing critical alarm signals (such as water level warnings) to fail to arrive in real time. At the same time, in high-concurrency scenarios, frequent JSON serialization and large data packet transmission can easily trigger the 16KB payload limit of communication protocol layers such as WebRTC, further exacerbating network congestion, front-end parsing lag, and the loss of critical data.

[0008] The fifth shortcoming lies in the overly coarse resource scheduling strategy. Existing cloud rendering engines suffer from double waste in resource management, both during runtime and idle periods. In the runtime spatial dimension, the engine typically performs indiscriminate full-load rendering and physical simulation of the entire scene. Even when numerous high-resolution models (such as fluid particles and dynamic vegetation) are outside the user's camera's view frustum, the system continues to consume expensive GPU computing power, resulting in extremely low resource utilization. In the idle temporal dimension, when users stop operating or are waiting for a connection during pre-startup, the system often lacks automatic connection detection and sleep mechanisms, causing the program to idle at high frame rates for extended periods. This not only accelerates the wear and tear on hardware such as graphics cards but also significantly increases the operating costs of cloud services.

[0009] Therefore, there is an urgent need to propose a cloud rendering message communication and interaction method that connects multiple digital twin business scenarios, so as to systematically solve the above problems and meet the actual needs of the growing digital twin business scenarios for cloud rendering applications. Summary of the Invention

[0010] The purpose of this invention is to overcome the technical shortcomings of existing digital twin cloud rendering systems, such as tight code coupling, debugging difficulties, unstable message transmission, and resource waste, and to provide a cloud rendering message communication interaction method that connects multiple digital twin business scenarios. This method achieves unified access to heterogeneous cloud rendering platforms, flexible debugging and deployment of multi-terminal applications, adaptive message scheduling based on spatial vision awareness, and dynamic optimization of computing resources based on spatiotemporal coupling by constructing a generalized, configurable, and highly robust message communication middleware architecture.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios is characterized by being based on a distributed three-layer architecture comprising a client layer, a middleware layer, and a server layer. The client layer is configured with a front-end development standard API library and a viewpoint synchronization daemon; the middleware layer has a built-in message processing engine, a spatial semantic indexer, and a connection state manager; and the server layer loads a server-side message plugin. The method includes the following interconnected and coordinated steps:

[0013] Step 1, Initialization Phase: The client layer reads the global cloud rendering configuration parameters. When the parameters point to the cloud rendering mode, a communication connection is established with the specified cloud rendering engine type through the middleware layer.

[0014] Step 2, Message Communication Phase: The front-end development standard API library encapsulates business interaction operations into standard message bodies and reports them to the middleware layer. Simultaneously, the viewpoint synchronization daemon process collects the view frustum parameters of the 3D camera at a preset frequency and sends them to the middleware layer. The message processing engine uses the spatial semantic indexer to determine the spatial intersection relationship between the message source object and the current view frustum, and calculates a dynamic scheduling weight for each message based on the business semantic importance and distance decay law. Based on the dynamic scheduling weight, the message processing engine routes the messages to the high-priority pass-through channel, the smooth flow-limiting channel, or the aggregation and degradation channel for traffic control.

[0015] Step 3, Resource Optimization Phase: The middleware layer forwards the received view frustum parameters in parallel to the server layer. Based on these view frustum parameters, the server-side message plugin performs a physical simulation pause or multi-level detail (MLD) downgrading rendering on twin objects determined to be outside the view frustum, thereby achieving spatiotemporal linkage between message communication scheduling and cloud computing power control based on the same spatial awareness data. Furthermore, when the connection state manager detects network bandwidth congestion, it dynamically reduces the proportion of the extended view frustum to reduce high-priority message transmission and cloud rendering load. The extended view frustum is an extended buffer zone constructed by expanding the field of view (FOV) of the standard view frustum outwards by a set proportion.

[0016] This invention defines a three-layer architecture and a three-stage collaborative process (initialization, message communication, and resource optimization) for cloud rendering message communication interaction methods that connect multi-terminal digital twin business scenarios. Its advantages are: by separating the responsibilities of the client layer, middleware layer, and server layer, a closed-loop linkage between message communication and computing power scheduling is achieved; in particular, by using view frustum parameters simultaneously for message filtering and cloud rendering control, a spatiotemporally coupled dynamic optimization mechanism is formed, effectively solving the problem of the separation between message transmission and resource management in traditional solutions.

[0017] Preferably, the dynamic scheduling weight in step two... The calculation formula is:

[0018]

[0019] In the formula A semantic importance constant preset for business functions; Spatial visibility coefficient; is the distance decay function; d is the real-time distance from the object to the camera focus; α, β, and γ are normalized weighting coefficients.

[0020] This invention defines dynamic scheduling weights. The specific calculation formula integrates three dimensions: semantic importance, spatial visibility, and distance attenuation. Its advantages are: it breaks through the limitations of traditional static priority or blind full synchronization, enabling the system to finely determine the transmission priority of each message based on the business importance of the object, whether it is within the user's field of vision, and its distance, thereby prioritizing the real-time arrival of critical signaling under limited bandwidth.

[0021] Preferably, the spatial visibility coefficient The value is determined based on the spatial intersection detection results: 1.0 when the object is completely contained within the view frustum, 0.6 when the object intersects with the extended buffer zone, and 0.1 when the object is outside the view frustum; the extended buffer zone is constructed by extending the FOV of the standard view frustum outward by a set ratio.

[0022] This invention further defines the spatial visibility coefficient V. frustum The rules for determining the value of the buffer were established, and the concept of an extended buffer was introduced. Its advantages are: it not only distinguishes between the two extreme states of being completely visible and completely invisible, but also adds the intermediate state of intersection, providing a pre-loading buffer for objects about to enter the field of view; the extended buffer effectively avoids message loss or rendering stuttering when the camera moves quickly.

[0023] Preferably, the distance decay function adopts an exponential decay model:

[0024] Where k is the distance attenuation coefficient, which is adjusted according to the scene scale, and λ is the preset exponential attenuation rate.

[0025] This invention specifies that the distance decay function adopts an exponential decay model. Its advantages are: the model is simple to calculate, easy to implement in real time, and can smoothly reduce the scheduling weight of objects far from the camera, allowing the system to naturally focus transmission and computing resources on the core area currently of user interest, which conforms to the visual attention patterns of human-computer interaction.

[0026] Preferably, the high-priority pass-through channel is suitable for messages with a weight W greater than or equal to the high-priority threshold. For messages whose length exceeds a preset threshold, a fragmentation mechanism is initiated and metadata such as sequence ID, total number of fragments, timestamp, and fragmentation index is added. The smooth rate limiting channel is suitable for messages with a weight W between the low-priority threshold and the high-priority threshold. A token bucket algorithm is used for smooth rate limiting. The aggregation and degradation channel is suitable for messages with a weight W less than the low-priority threshold. A time window aggregation mechanism is initiated to merge multiple state updates into a single final state message or to discard them directly.

[0027] This invention defines a three-tiered channel traffic splitting mechanism based on weighted thresholds (high-priority pass-through, smooth rate limiting, and aggregation / degradation). Its advantages are: it abandons the traditional one-size-fits-all approach to message processing, supports intelligent fragmentation for high-priority messages to ensure reliable transmission, applies rate limiting to medium-weighted messages to avoid congestion, and aggregates or discards low-weighted messages to save bandwidth. This hierarchical processing strategy improves the overall system throughput and the real-time performance of critical messages.

[0028] Preferably, the middleware layer maintains a lightweight octree spatial index for dynamic digital twin objects. When the server generates a state update downlink command, it extracts the spatial bounding box of the object and performs intersection calculation with the extended view frustum, outputting three states: fully contained, intersecting, or invisible.

[0029] This invention defines a lightweight octree spatial index maintained in the middleware layer for fast spatial intersection detection. Its advantages are: when dealing with massive dynamic objects (such as tens of thousands of IoT sensors in water conservancy projects), the octree index can efficiently complete spatial queries of object bounding boxes and extended view frustums, avoiding the performance bottleneck caused by traversing all objects one by one, and providing a supportive computational foundation for real-time message scheduling.

[0030] Preferably, the rendering protocol adapter is implemented based on a plug-in architecture, with each cloud rendering platform corresponding to an independent adapter module. It interacts with the message processing engine through a unified interface. When a new cloud rendering platform is connected, only the adapter module corresponding to that cloud rendering platform needs to be loaded in the middleware layer.

[0031] This invention defines a plug-in architecture for the rendering protocol adapter, with each cloud rendering platform having its own independent adapter module. Its advantages include: decoupling business logic from the underlying cloud rendering communication protocol; when switching to a third-party cloud rendering platform, only the corresponding adapter module needs to be loaded, without modifying any business layer code; this improves system portability and platform flexibility, and reduces the risk of vendor lock-in.

[0032] Preferably, the rendering cascade optimization based on the view frustum in step three specifically involves: the server-side message plugin calling the engine API to pause the physical simulation calculation and particle effect rendering of twin objects determined to be outside the field of view, or forcibly downgrading them to static meshes, and reducing the tick frequency of objects within the buffer band.

[0033] This invention defines specific operations for rendering cascade optimization based on view frustum: pausing physical simulation or downgrading to static meshes for objects outside the view frustum, and reducing the tick frequency for objects within the buffer zone. Its advantages are: achieving spatiotemporal linkage between message communication scheduling and cloud computing power control—the same view frustum parameters are used both for message filtering and to drive the dynamic allocation of rendering resources, avoiding indiscriminate full-load rendering across the entire scene and reducing GPU computing power consumption.

[0034] Preferably, the connection state manager detects network connectivity through a heartbeat mechanism, detects network latency by measuring message round-trip time, and detects network stability by statistically analyzing message loss rate. When the network latency exceeds a threshold, a rendering mode rollback strategy is triggered. When a network interruption is detected, a local caching mechanism is started to automatically synchronize the state after the network is restored, i.e., reconnection.

[0035] This invention defines a multi-dimensional network detection and adaptive recovery mechanism (heartbeat, RTT measurement, packet loss statistics) for the connection state manager. Its advantages include: the system can perceive changes in network quality in real time and automatically revert to rendering mode when network latency increases, activate local caching when the network is interrupted, and automatically synchronize the state after network recovery. This robust adaptive capability enables applications to maintain a stable interactive experience in complex public network environments.

[0036] Preferably, the middleware layer further includes a rendering mode switcher, which switches between local rendering mode and cloud rendering mode according to global configuration parameters. In local rendering mode, it communicates directly with the engine editor through an embedded browser plugin to support real-time debugging. In cloud rendering mode, it communicates with the cloud engine instance through the rendering protocol adapter.

[0037] This invention defines the design of a rendering mode switcher, supporting seamless switching between local and cloud rendering modes. Its advantages include: during development, the local rendering mode can communicate directly with the engine editor for real-time debugging, eliminating the need for repeated packaging and uploading; after system deployment, switching to cloud rendering mode only requires changing global parameters. This mechanism shortens the development cycle and reduces debugging costs, while balancing development efficiency and deployment flexibility.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] First, code decoupling and cross-platform compatibility. By introducing a plugin architecture with a rendering protocol adapter, business logic is completely decoupled from the underlying cloud rendering communication protocol. Business users can develop multi-terminal digital twin applications without needing to know the specific implementation of the underlying cloud rendering software. When changing cloud rendering platforms, only the adapter module needs to be adjusted, without modifying the business logic, thus improving system portability and development efficiency.

[0040] Secondly, real-time debugging and rapid switching are achieved. A rendering mode switcher enables seamless switching between local and cloud rendering, allowing real-time debugging via an embedded browser page based on a WebUI plugin during the digital twin scenario building and business function development phases. After the system is packaged and released, only the global parameters of the cloud rendering type need to be changed to quickly adapt to the cloud rendering environment through message communication switching. This seamless switching mechanism reduces development cycles and deployment costs.

[0041] Third, front-end development is simplified. The standard front-end API library fully encapsulates functionalities such as scenes, layers, tools, and effects in digital twin scenarios, unifies the handling of various response events and callback functions, and manages all types of objects within the standard API library. Developers no longer need to manually construct complex JSON objects or maintain front-end and back-end parsing rules themselves, significantly reducing the complexity of front-end development and allowing them to focus on implementing business logic.

[0042] Fourth, highly reliable transmission of critical signaling and elimination of head-of-line congestion. Abandoning the inefficient traditional indiscriminate fragmentation model, a weighted intelligent fragmentation mechanism is used. Fragmentation transmission is initiated only for critical messages in high-priority queues that exceed length limits (such as emergency disaster warnings), while low-weight long messages are aggregated or intercepted. This strategy effectively avoids massive amounts of low-value data consuming fragmentation and reassembly resources, ensuring the stability and rapid response of core commands in complex business scenarios.

[0043] Fifth, high bandwidth efficiency in high-concurrency scenarios with massive objects. A dual message scheduling algorithm based on view frustum and semantic weights is introduced. Through joint calculation of spatial intersection detection and business importance, it prioritizes the real-time arrival of messages for core objects within the field of view, while aggregating and downgrading messages for objects outside the field of view or at greater distances. This significantly reduces the network transmission of invalid information without sacrificing the user's visual experience, effectively addressing the challenges of concurrent congestion of massive IoT sensor data in scenarios such as smart water conservancy.

[0044] Sixth, fine-grained scheduling of computational resources coupled with spatiotemporal dynamics. This not only automatically pauses physical simulation and special effects rendering of objects outside the view frustum or performs LOD degradation during user operations, but also triggers global idle freeze when prolonged inactivity is detected. This two-dimensional coupling strategy of spatial degradation and temporal freeze minimizes the high load on cloud CPUs / GPUs, extends hardware lifespan, and reduces operating costs. Attached Figure Description

[0046] Figure 1 This is the overall system architecture diagram of the present invention, which shows the data flow and interaction relationship between the client layer, the middleware layer containing the spatial semantic indexer and the view frustum weight calculator, and the server layer.

[0047] Figure 2 This is a schematic diagram of the spatial region determination and extended buffer zone of the view frustum of the present invention, which shows the geometric positional relationship between the standard view frustum and the extended buffer zone, as well as the determination logic of twin objects in three spatial states: complete containment, buffer zone intersection, and externally invisible.

[0048] Figure 3This invention presents a message scheduling flowchart based on spatial semantic weights, illustrating the complete decision-making logic from message generation to transmission, including strong control instruction determination, dynamic weight formula calculation, and hierarchical processing of high-priority fragmentation, smooth rate limiting, and aggregation interception based on weight thresholds.

[0049] Figure 4 This invention presents a cloud resource dynamic optimization state transition diagram, which illustrates the control logic that automatically switches between three modes—full-speed rendering, spatial degradation, and global sleep—based on changes in user viewpoint and idle time. Detailed Implementation

[0051] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The following examples are used to illustrate the present invention, but are not intended to limit the scope of protection of the present invention. The advantages of the present invention will become clearer and easier to understand through this description.

[0052] This invention presents a cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios. The method is implemented based on a distributed three-tier architecture, including a client layer, a middleware layer, and a server layer. Each layer interacts through standardized interfaces, ensuring good scalability and flexibility. The client establishes a persistent connection with the middleware via the WebSocket protocol, and the middleware interacts with the server via a RESTful API or a custom protocol, achieving efficient bidirectional message transmission. Figure 1 As shown.

[0053] The client-side layer supports access from multiple terminal formats, including web browsers, mobile terminals, and desktop applications, and integrates a lightweight front-end development standard API library. This standard API library is built upon the API standard library provided by the middleware layer, encapsulating object management operations involved in digital twin scenarios, such as scene management, layer control, tool invocation, and special effects playback, into standardized high-level semantic interfaces. Developers do not need to concern themselves with the underlying communication protocols or message formats to interact with the 3D scene; all operations are completed through high-level semantic calls.

[0054] In terms of specific technical implementation, the front-end development standard API library listens for user input events (such as mouse clicks, double-clicks, and drags) and scene state change events (such as layer visibility changes and camera pose changes) through an event registration mechanism. When an event is triggered, it automatically encapsulates the operation command into a standard message body carrying metadata and sends it to the middleware. When developing business functions based on this front-end development API, the HTML5 + JavaScript technology stack can be used on the web side. The front-end development standard API library provides standardized event listening and command sending interfaces through the browser environment. Taking the large-screen application in the flood control consultation room as an example, when a user clicks the "Start Flood Simulation" button, the front-end development API automatically obtains the user's operation function, determines the message format (including fields such as command, msgType, priority, layer, and intensity) based on the operation type, and adds necessary information such as timestamps and source identifiers, ultimately forming a complete standard message body. Developers do not need to manually construct complex JSON structures to complete the message encapsulation for business operations. The client-side front-end development API also provides interfaces for layer visibility control, special effects playback, and camera movement. Developers only need to call these high-level semantic interfaces to achieve interaction with the digital twin scene.

[0055] The standard API library has a built-in viewpoint synchronization daemon that can automatically collect the current 3D camera's frustum parameters at a preset low frequency (e.g., 10Hz), including the field of view (FOV), camera spatial coordinates, and line-of-sight vector, and continuously send these parameters to the middleware layer to provide a real-time viewpoint data foundation for the backend spatial intelligent scheduling algorithm.

[0056] The client layer also includes a global configuration module for setting runtime parameters such as rendering mode (local or cloud), message fragmentation threshold, and connection timeout. These parameters can be dynamically adjusted, allowing the system to automatically optimize performance under different network environments and hardware conditions. For example, on a large-screen display in a conference room equipped with high-performance hardware, it can be set to local rendering mode for the best interactive experience, while on ordinary office terminals, it can be set to cloud rendering mode to fully utilize the graphics processing power of the cloud server.

[0057] The middleware layer, as the core layer of this invention, is deployed on a cloud or edge server and includes five key modules: an API standard library, a message processing engine, a connection state manager, a rendering protocol adapter, and a rendering mode switcher. The middleware layer is responsible for uniformly converting messages sent by the client into a format recognizable by the server, and implementing functions such as long string segmentation, message queue buffering, and anti-blocking mechanisms.

[0058] The API standard library is responsible for defining unified message interfaces and parameter specifications. This library categorizes messages from different business scenarios into three types: control, data, and status, and defines standard field structures and value ranges for each type. Control messages include commandType, commandId, parameters, and timestamp fields; data messages include dataType, dataId, payload, and source fields; and status messages include statusType, statusValue, target, and timestamp fields. The API standard library also defines message priority levels, divided into four levels: urgent, important, normal, and low priority, for subsequent hierarchical message queue management.

[0059] The message processing engine is the core of the intelligent scheduling in this invention. It incorporates a spatial semantic indexer and a view frustum weight calculator, completely changing the traditional middleware's blind pass-through working mode. This engine performs a preprocessing flow based on three-dimensional spatial semantics on all downlink messages. For example... Figure 2 As shown, the engine first uses a spatial semantic indexer to maintain a lightweight octree spatial index for dynamic digital twin objects. When the server generates a state update downlink command, it extracts the object's spatial bounding box and performs intersection calculations with the extended view frustum, outputting three visibility states: fully contained, intersecting, or invisible. Subsequently, the engine combines business semantic importance, spatial visibility coefficient, and distance decay function to calculate the scheduling weight for each message using a dynamic weight calculation formula. Ultimately, messages are distributed to different processing channels based on their weight values. Only messages deemed high-priority will trigger length checks and automatic fragmentation mechanisms, adding metadata such as sequence IDs and total fragment counts; low-priority messages are automatically aggregated within a time window or discarded. The engine also implements anti-blocking control, dynamically adjusting the sending rate.

[0060] like Figure 3 As shown, the message processing engine introduces an adaptive scheduling mechanism based on view frustums, and the specific execution steps are as follows:

[0061] Step A: Viewpoint Synchronization and Region Construction. The client's standard API library sends the current 3D camera's state information (coordinates, view direction vector, and field of view (FOV)) to the middleware at a preset frequency. Upon receiving the parameters, the middleware expands the standard FOV outward by a set percentage (e.g., 10%), constructing an extended view frustum buffer. This buffer provides a preloaded buffer area for objects about to enter the user's field of view, preventing message breaks during rapid camera movement.

[0062] Step B, Spatial Intersection Detection. The message processing engine uses an octree spatial index to perform intersection detection calculations on the spatial bounding box of the message source object and the extended view frustum, outputting three visibility states: Inside (the object is completely inside the view frustum), Intersecting (the object intersects with the extended buffer), and Outside (the object is completely outside the view frustum). This octree index structure can efficiently support fast querying and visibility determination of large-scale spatial objects, avoiding the performance bottleneck caused by traversing all objects one by one.

[0063] Step C, Dynamic Weight Calculation. The message processing engine calculates a dynamic scheduling weight for each message, breaking through the static priority setting of traditional solutions. The middleware divides messages into two categories: strong control instructions and weak state updates. Strong control instructions, such as startup deduction, are assigned the highest semantic weight by default. To ensure delivery, subsequent status update messages (such as high-frequency data from massive monitoring sensors within water conservancy projects) are strictly filtered according to a dynamic weighting calculation formula. This formula is:

[0064]

[0065] in A semantic importance constant preset for business purposes. The spatial visibility coefficient is 1.0 when completely contained, 0.6 when intersecting, and 0.1 when invisible. The real-time Euclidean distance from the object to the camera focus. This is the distance decay function. , , These are the normalized weighting coefficients. The distance decay function uses an exponential decay model, whose simplified approximation is:

[0066]

[0067] in This is the distance attenuation coefficient, which is adjusted according to the scene scale. This is the preset exponential decay rate.

[0068] Step D, hierarchical route distribution. Based on the calculated weights... The messages are processed in a tiered manner and adapted to the physical layer, specifically through three channels:

[0069] (1) High-performance transparent transmission channel ( This applies to messages with a weight value greater than or equal to the high-priority threshold. Messages targeting core objects within the field of view (such as key layers in flood control simulations) directly enter the sending queue. At this point, the engine triggers a physical layer length check. If the message length exceeds a preset threshold (e.g., 16KB), a fragmentation mechanism is initiated only for that high-priority message. Metadata such as sequenceId, totalFragments, timestamp, and fragmentIndex are added to each fragment before it is pushed into the queue. Since a large number of low-weight messages outside the field of view have been filtered out, fragmented transmission here is less likely to cause network congestion.

[0070] (2) Smooth flow limiting channel ( This applies to messages with weight values ​​between the low and high priority thresholds. For messages containing objects within the buffer band, a token bucket algorithm is used for smooth rate limiting, prohibiting the generation of large data packets exceeding the MTU (Maximum Transmission Unit), forcing data simplification at the business level (such as sending only key coordinates instead of complete model data), and avoiding the consumption of fragment reassembly resources.

[0071] (3) Aggregated Degradation Channel ( This applies to messages with a weight value less than or equal to the low priority threshold. The middleware intercepts these messages, preventing them from entering the physical sending queue, and initiates a time window aggregation mechanism (such as a 5-second time window). This merges all state updates generated within the time window into a single final state message or directly discards intermediate state data. These messages do not trigger fragmentation logic, thus significantly reducing bandwidth consumption.

[0072] The rendering protocol adapter converts standard messages into the proprietary protocol required by the target platform, based on the currently configured cloud rendering platform. For example, for the first cloud rendering platform, the adapter converts messages into Raystreaming format instructions; for the second cloud rendering platform or a cloud rendering platform adapted to different protocols, the adapter converts messages into binary messages in WebRTC data channel format. The converted messages are sent to the server via the appropriate cloud platform protocol (such as WebSocket, WebRTC, or HTTP). The server-side message plugin receives the messages and reassembles them. If the message is fragmented, it reassembles the fragments into a complete message based on the sequenceId and timestamp metadata. After reassembly, the plugin parses the message content, extracts key information such as commandType, commandId, and parameters, and converts it into operations that the engine can understand.

[0073] The message fragmentation and reassembly mechanism also includes fault tolerance. When the receiving end detects a missing fragment, it requests retransmission from the middleware using the sequenceId and timestamp. The number of retries is configurable; after a preset number is exceeded, retransmission is abandoned to avoid resource waste caused by infinite retries. Furthermore, the middleware implements a time window verification mechanism to check message timeliness. If the difference between the message's reception time and transmission time exceeds a preset threshold (e.g., 500ms), the expired message is discarded to prevent inconsistencies in scenario states due to network latency.

[0074] The connection state manager is responsible for real-time monitoring of network connection status. This is implemented through three layers of detection mechanisms: detecting network connectivity via heartbeats (e.g., sending a lightweight probe message every 30 seconds), detecting network latency by measuring round-trip time (RTT), and assessing network stability by statistically analyzing message loss rate. When network latency exceeds a threshold (e.g., 200ms), a rendering mode rollback strategy is triggered, converting high-precision rendering to low-precision rendering to reduce bandwidth consumption. When a network interruption is detected, a local caching mechanism is activated to maintain basic scene interactivity, automatically synchronizing the state once the network is restored. When user interaction is idle (e.g., no operation within 10 minutes), a frame rate freeze command is sent to the server to reduce computational power consumption.

[0075] The rendering mode switcher dynamically verifies the current rendering mode and performs logical synchronization control on messages under different modes by monitoring changes in global rendering mode parameters. In local rendering mode, messages are directly passed to the local engine editor via an embedded browser plugin, supporting real-time debugging by developers. In cloud rendering mode, messages are converted by the rendering protocol adapter and then sent to the cloud engine instance. The implementation consists of three parts: first, monitoring changes in global rendering mode parameters; second, adjusting the message processing strategy based on parameter changes; and finally, ensuring that the message processing logic remains consistent across different rendering modes to avoid differences in execution effects caused by mode switching.

[0076] The server-side layer runs on a cloud server cluster and includes at least one instance of a 3D engine (such as UE5 or Unity 3D) that supports cloud streaming rendering, and loads the server-side message plugin that accompanies this invention. This plugin listens for standardized instructions from the middleware, converts and executes engine operations through these standardized instructions, and simultaneously controls the engine's rendering frame rate.

[0077] The core functionality of the server-side messaging plugin comprises three stages: command listening, command parsing, and command execution. Upon receiving a command from the middleware, the plugin first reassembles the message: if the message is transmitted in fragments, it reassembles the fragments into a complete message based on metadata such as sequenceId and timestamp. After reassembly, the plugin parses the message content, extracts key information such as commandType, commandId, and parameters, and converts it into native operations that the engine can understand and execute. For example, when receiving a layer visibility control command, the UE5 plugin calls the framework's API to set the visibility of the specified layer, while the Unity plugin calls the corresponding engine API to achieve the same functionality. This design allows the plugin to adapt to the characteristics of different engines while maintaining the consistency of the message interface.

[0078] The server-side messaging plugin also implements frame rate control functionality to respond to resource optimization commands from the middleware layer. Upon receiving a frame rate freeze command, the plugin limits the engine's rendering frame rate to an extremely low value (e.g., 5 FPS), thereby reducing the computational load on the CPU and GPU and extending hardware lifespan. The server-side layer also implements a bidirectional communication mechanism with the middleware to transmit rendering status and user operation feedback. During normal scene operation and rendering, the plugin continuously sends rendering status messages to the client via the middleware, including information such as the current frame rate and CPU / GPU utilization. When a user performs an operation in the 3D scene (e.g., clicking on an object), the plugin also sends operation feedback messages to the client via the middleware, triggering corresponding business logic. This bidirectional communication mechanism ensures close integration between the digital twin scene and business functions.

[0079] After developing system functions using the method of this invention and deploying them using cloud rendering, the specific execution flow is as follows:

[0080] Step 1, Initialization Phase

[0081] Upon system startup, the client reads global cloud rendering configuration parameters. If the parameters are set to local rendering, the system directly calls the local rendering engine and establishes a communication connection with the UE editor through an embedded browser plugin, supporting developers in quickly iterating and debugging business functions within the editor environment. If the parameters are set to cloud rendering, the client establishes a connection with the specified cloud rendering platform type through middleware, and the middleware's rendering protocol adapter automatically loads the corresponding platform's communication protocol module. The cloud rendering platform can seamlessly switch between platforms by extending the type of rendering protocol adapter without modifying the upper-layer business logic.

[0082] Step 2, Message Communication Phase

[0083] The client develops various business function applications based on the front-end development standard API library supporting this invention. During the operation of the system, the client synchronizes the frustum parameters to the middleware in real time through the built-in view point synchronization daemon process, providing a data basis for the backend spatial calculation. When the user triggers a business operation or the scene undergoes an automatic update, the system conducts message communication according to the complete process of message encapsulation, intelligent scheduling, hierarchical processing, and transformation execution.

[0084] In the message encapsulation stage, the front-end development API converts the business operation into a standard message body carrying metadata and sends it to the middleware. In the intelligent scheduling stage, the message processing engine first identifies the message type. For strong control instructions, the highest weight is directly assigned to ensure delivery. For status update messages, the dynamic weight is calculated by substituting the frustum parameters received in real time into the weight calculation formula (combining the comprehensive semantic importance, spatial visibility coefficient, and distance attenuation function). In the hierarchical processing stage, the messages are shunted to high-priority, rate-limiting, or aggregation queues according to the weight values. Specifically, only when the length of the message entering the high-priority queue exceeds the threshold, the sharding mechanism is started to split it into multiple packet groups, and the remaining messages are processed according to the established strategies (rate-limiting or aggregation). In the transformation execution stage, the valid messages after screening and processing are converted into the corresponding cloud platform protocol through the adapter and sent to the server. The server message plug-in receives and analyzes (reassembling the shards if necessary), and finally drives the engine to execute the operation.

[0085] In addition, the middleware also implements time window verification to detect the reception time of the messages. If the difference between the reception time and the sending time of the message exceeds the preset threshold (such as 500 ms), the message is discarded to avoid inconsistent scene states caused by network latency. For example, if the user operates again after a long time, the middleware will discard all expired instructions remaining during this period to ensure the consistency of the scene state.

[0086] Step 3, resource optimization phase

[0087] The resource optimization strategy of this invention is no longer limited to traditional idle detection, but realizes dynamic computing power scheduling with two-dimensional coupling of space and time, as Figure 4 shown.

[0088] In the spatial dimension, this invention implements a rendering cascade optimization based on the view frustum. After receiving the viewpoint parameters forwarded by the middleware, the server-side message plugin not only uses them for message filtering but also directly controls the engine's rendering logic. For twin objects determined to be outside the field of view, the plugin calls engine APIs (such as UE's SetComponentTickEnabled or LOD interface) to pause their physical simulation calculations and particle effect rendering, or forces them to be downgraded to static mesh display. For objects within the buffer zone, their tick frequency is reduced (e.g., from 60Hz to 10Hz). This allows cloud computing resources to focus on high-value areas currently visible to the user, rather than performing indiscriminate full-load rendering across the entire scene.

[0089] In terms of time, this invention implements an adaptive idle sleep strategy. The middleware layer's connection state manager has built-in idle detection logic, continuously listening for user interaction events (including mouse movements, keyboard operations, and business function API calls), and setting a configurable idle timer (e.g., 10 minutes). When a user interaction event is detected, the timer is reset. When the timer expires and the heartbeat packet is received normally, it is determined to be in an idle state, triggering a frame rate freeze command to lock the engine frame rate to an extremely low value (e.g., 5 FPS), significantly reducing computing power consumption. The heartbeat packet is a lightweight probe message periodically sent by the middleware layer to detect network connectivity. If the heartbeat packet does not respond for a certain period of time (e.g., 30 seconds), it is determined that the network is interrupted, the connection is disconnected, and computing resources are released.

[0090] In terms of the linkage control of frame rate and bandwidth, when the middleware detects network bandwidth congestion (i.e., a sudden increase in RTT), it actively sends instructions to reduce the range ratio of the extended view frustum and artificially expand the judgment range of the area outside the field of view, thereby forcibly reducing the amount of message transmission and cloud rendering computation, and achieving adaptive steady-state maintenance under network fluctuation conditions.

[0091] In terms of dynamic recovery, once a new user interaction command or viewpoint movement to the relevant area is detected, the system immediately increases the scheduling weight of the corresponding object and removes the frame rate limit, restoring full-performance rendering. The entire recovery process is transparent to the user and has no noticeable lag.

[0092] Example

[0093] To verify the technical effect of the present invention, the construction of a flood control four-prevention digital twin application system of a city's water conservancy department is used as an example for illustration.

[0094] Scenario Description: This system serves flood control scheduling consultations and emergency drills, involving high-frequency concurrent data reporting from over 10,000 IoT monitoring sensors within the river basin (including water level gauges, rain gauges, flow meters, displacement gauges, deformation monitors, seepage meters, and piezometers), as well as real-time rendering of dynamic water flow particles in major river sections. The system needs to support simultaneous access from multiple terminals including web, mobile, and desktop, and effectively address network congestion and rendering lag issues under massive data concurrency conditions.

[0095] System Architecture: The system is built using the three-tier architecture of this invention. The client layer is based on the front-end development standard API library of this invention, using the Vue framework to develop business pages, and integrates the viewpoint synchronization daemon built into the standard API library to continuously send the current camera's FOV and position parameters to the middleware at a frequency of 10Hz. The middleware layer is deployed on a cloud server, and its core integrates the spatial semantic indexer and view frustum weight calculator modules of this invention, and is configured with the Rayvision 3DCAT rendering protocol adapter. The server layer deploys a UE5 application and integrates the server-side messaging plugin of this invention, used to receive degradation instructions and control the tick frequency of the physics simulation.

[0096] Functional Application: When using this system on the large screen in the flood control consultation room, since the consultation room is equipped with high-performance computers, the global parameters are set to local rendering mode to obtain the best 3D display effect. When using it on ordinary office computers of business personnel, the global parameters are switched to cloud rendering mode to utilize the hardware resources on the server side for rendering. During subsequent operation and maintenance, when scene editing or function debugging is required on local devices, the global parameters can be switched back to local rendering mode at any time.

[0097] Resource optimization results: In the high-concurrency scenario of flood control simulation, the system implemented dynamic scheduling and spatiotemporal coupling optimization based on view frustum. The weight allocation coefficient was set as semantic importance. Spatial visibility Distance attenuation Distance attenuation coefficient .

[0098] Taking the water level gauge in the core area of ​​the dam (denoted as object A) as an example, this object is a strongly business-related object with semantic importance. And it is located within the user's current view frustum, with a spatial visibility coefficient. It is relatively close to the camera. Its overall weight is calculated using the weighting formula. Above the high-quality threshold The middleware places its data into a high-optimal pass-through queue, enabling millisecond-level real-time updates and maintaining full-lighting, high-precision rendering on the server side.

[0099] Taking the rainfall sensor in the downstream urban area (denoted as object B) as an example, this object belongs to ordinary business data, and its semantic importance is... Located outside the visual cone, spatial visibility coefficient The overall weight is calculated using the weighting formula. Less than the low-optimal threshold The middleware intercepts its real-time reports, which are sent 60 times per second, and automatically initiates a time window aggregation mechanism, downgrading them to a heartbeat packet sent every 5 seconds. Simultaneously, upon receiving the weight reduction notification, the server-side plugin automatically suspends the water physics simulation of the downstream urban area, retaining only the static mesh display.

[0100] Regarding idle time optimization, when user inactivity is detected to have exceeded 10 minutes, the middleware layer sends a global freeze command, locking the engine frame rate at 5 FPS, and reducing GPU utilization from 65% to 10%. In terms of dynamic recovery, once the user turns the camera view towards the downstream urban area, object B enters the view frustum, and the spatial visibility coefficient... Change to version 1.0, overall weight. When the value is increased to 0.8 or higher, the middleware layer immediately resumes real-time pass-through and physical rendering of the area, with no noticeable lag throughout the process.

[0101] Performance Comparison

[0102] Actual testing and verification have shown that, compared with traditional solutions, this invention achieves improvements in the following key indicators:

[0103]

[0104] Results Analysis: The digital twin application system constructed using the method of this invention not only achieves seamless switching between local and cloud rendering and low-cost multi-terminal adaptation, but more importantly, it successfully reduces network bandwidth usage by more than 30% in scenarios involving tens of thousands of concurrent sensor data through a view frustum-based spatial semantic scheduling algorithm, and concentrates cloud computing power on the core areas of user concern, effectively solving the contradiction between high concurrency, low bandwidth and high experience in large-scale digital twin scenarios.

[0105] All other unspecified parts belong to the prior art.

Claims

1. A cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios, characterized in that, This method is implemented based on a distributed three-tier architecture comprising a client layer, a middleware layer, and a server layer. The client layer is configured with a front-end development API library and a viewpoint synchronization daemon. The middleware layer has a built-in message processing engine, a spatial semantic indexer, and a connection state manager. The server layer loads a server-side message plugin. The method includes the following interconnected and coordinated steps: Step 1, Initialization Phase: The client layer reads the global cloud rendering configuration parameters. When the parameters point to the cloud rendering mode, a communication connection is established with the specified cloud rendering engine type through the middleware layer. Step 2, Message Communication Phase: The front-end development API library encapsulates business interaction operations into standard message bodies and reports them to the middleware layer. Simultaneously, the viewpoint synchronization daemon process collects the view frustum parameters of the 3D camera associated with the client layer at a preset frequency and sends them to the middleware layer. The message processing engine uses the spatial semantic indexer to determine the spatial intersection relationship between the message source object and the current view frustum, and calculates a dynamic scheduling weight for each message based on the importance of business semantics and the distance decay law. Based on the dynamic scheduling weight, the message processing engine routes messages to the high-quality transparent transmission channel, the smooth rate limiting channel, or the aggregation and degradation channel for traffic control. Step 3, Resource Optimization Phase: The middleware layer forwards the received view frustum parameters to the server layer in parallel; The server-side message plugin, based on the view frustum parameters, performs a physical simulation pause operation or multi-level detail downgrading rendering on twin objects determined to be outside the view frustum, thereby realizing spatiotemporal linkage between message communication scheduling and cloud computing power control based on the same spatial perception data; in addition, when the connection status manager detects network bandwidth congestion, it dynamically reduces the proportion range of the expanded view frustum to reduce the amount of high-priority message transmission and cloud rendering load.

2. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 1, characterized in that, In step two, the dynamic scheduling weights The calculation formula is: ； In the formula A semantic importance constant preset for business functions; Spatial visibility coefficient; is the distance decay function; d is the real-time distance from the object to the camera focus; α, β, and γ are normalized weighting coefficients.

3. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 2, characterized in that, The spatial visibility coefficient The value is determined based on the spatial intersection detection results: 1.0 when the object is completely contained within the view frustum, 0.6 when the object intersects with the extended buffer zone, and 0.1 when the object is outside the view frustum; the extended buffer zone is constructed by extending the FOV of the standard view frustum outward by a set ratio.

4. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 2, characterized in that, The distance decay function adopts an exponential decay model: ； Where k is the distance attenuation coefficient, which is adjusted according to the scene scale, and λ is the preset exponential attenuation rate.

5. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 1, characterized in that, The high-quality pass-through channel is suitable for messages with a weight W greater than or equal to the high-quality threshold. For messages whose length exceeds the preset threshold, a fragmentation mechanism is initiated and metadata such as sequence ID, total number of fragments, timestamp, and fragmentation index are added. The smooth rate limiting channel is suitable for messages with a weight W between the low-quality threshold and the high-quality threshold. The token bucket algorithm is used for smooth rate limiting. The aggregation and degradation channel is applicable to messages with a weight W less than the low-optimal threshold. The time window aggregation mechanism is activated to merge multiple state updates into a final state message or discard them directly.

6. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 1, characterized in that, The middleware layer maintains a lightweight octree spatial index for dynamic digital twin objects. When the server generates a state update downlink command, it extracts the spatial bounding box of the object and performs intersection calculation with the extended view frustum, outputting three states: fully contained, intersecting, or invisible.

7. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 1, characterized in that, The middleware layer also has a built-in rendering protocol adapter, which is implemented based on a plug-in architecture. Each cloud rendering platform corresponds to an independent adapter module, which interacts with the message processing engine through a unified interface. When a new cloud rendering platform is connected, only the adapter module corresponding to that cloud rendering platform needs to be loaded in the middleware layer.

8. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 1, characterized in that, The rendering cascade optimization based on the view frustum in step three specifically involves the server-side message plugin calling the engine API to pause the physical simulation calculation and particle effect rendering of twin objects determined to be outside the field of view, or forcibly downgrading them to static meshes, and reducing the tick frequency of objects within the buffer band.

9. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 1, characterized in that, The connection state manager detects network connectivity through a heartbeat mechanism, detects network latency by measuring message round-trip time, and detects network stability by statistically analyzing message loss rate. When network latency exceeds a threshold, a rendering mode rollback strategy is triggered. When a network interruption is detected, a local caching mechanism is started to automatically synchronize the state after the network is restored.

10. The cloud rendering message communication and interaction method for connecting multi-terminal digital twin business scenarios according to claim 1 or 6, characterized in that, The middleware layer also includes a built-in rendering mode switcher, which switches between local rendering mode and cloud rendering mode based on global configuration parameters. In local rendering mode, it communicates directly with the engine editor through an embedded browser plugin to support real-time debugging. In cloud rendering mode, it communicates with the cloud engine instance through the rendering protocol adapter.

Images