Cloud-native distributed real-time rendering framework, rendering method, and system

Abstract

This cloud-native distributed real-time rendering framework includes a network topology structure link establishment module, a conventional interactive real-time 3D framework adaptation layer, an ECS-based distributed development framework, a network synchronization layer, a data transmission layer, and a network transmission layer. The multiple domains of the conventional interactive real-time 3D framework adaptation layer are divided into multiple service modules, each corresponding to one type of distributed node. The network topology structure link establishment module establishes TCP, UDP, or RDMA type links to multiple types of distributed nodes based on differences in data processing flows and data transmission types for multiple types of distributed nodes. The ECS-based distributed development framework automatically completes data synchronization in accordance with the development system. The network synchronization layer synchronizes input and output data of multiple types of distributed nodes of the cloud-native distributed real-time rendering framework based on computation task rules and frame rates. The network transmission layer encapsulates TCP, UDP, or RDMA link schemes as selected by the upper layers according to the requirements for data characteristics between different systems and reachability between systems.

Claims

1. A cloud-native distributed real-time rendering framework used to process real-time streaming unstructured data, comprising a network topology structure link establishment module, a conventional interactive real-time three-dimensional (3D) framework adaptation layer, a distributed development framework based on an entity component system (ECS), a network synchronization layer, a data transmission layer, and a network transmission layer, The aforementioned cloud-native distributed real-time rendering framework provides various distributed nodes, including synthesis nodes, rendering nodes, logic nodes, and compute nodes. The aforementioned conventional interactive real-time 3D framework adaptive layer's multiple domains are divided into multiple service modules, each service module corresponding to one type of distributed node. The network topology structure link establishment module is used to establish links of the Communication Control Protocol (TCP), User Datagram Protocol (UDP), or Remote Direct Memory Access (RDMA) type to the various types of distributed nodes, depending on the data processing flow and the differences in data transmission types of the various types of distributed nodes. The aforementioned distributed development framework based on ECS is used to automatically complete data synchronization in accordance with the developed system. The network synchronization layer is used to synchronize input and output data of multiple types of distributed nodes of the cloud-native distributed real-time rendering framework based on computation task rules and frame rates. The aforementioned data transmission layer is used to provide a service for sending and receiving multiple types of data packets. The network transmission layer is used to encapsulate TCP, UDP, or RDMA link schemes so that the network topology structure link establishment module of the upper layer can select one in response to requirements for data characteristics between different systems and reachability between systems. The aforementioned distributed development framework based on ECS is used to further provide a development framework for business logic, and the conventional interactive real-time 3D framework adaptive layer, which runs in parallel with the aforementioned distributed development framework based on ECS, is used to further provide the adaptive capability of Unity functional modules within the aforementioned distributed development framework based on ECS. When processing rendering tasks that cannot be completed by a single rendering node, the logic node divides the rendering task and assigns the divided rendering tasks to multiple rendering nodes, so that the multiple rendering nodes can jointly complete the divided rendering tasks using a cooperative rendering method. A cloud-native, distributed, real-time rendering framework.

2. Regarding the coordination of rendering nodes within a group, if the rendering time per frame of the rendering nodes within the group exceeds a preset threshold, the target update frequency within the group is calculated based on the actual update frequency and historical data of multiple nodes within the group. A cloud-native distributed real-time rendering framework according to claim 1.

3. For rendering nodes and servers, the frequency at which service module nodes transmit data status information for each group over the network is dynamically adjusted based on target frequencies calculated for different groups. A cloud-native distributed real-time rendering framework according to claim 1.

4. The dynamic transmission frequency is calculated based on the group's target frequency and the server's own frequency parameters to determine whether the current frame will transmit data status to the group. A cloud-native distributed real-time rendering framework according to claim 1.

5. The rendering task partitioning scheme includes at least one of spatial dimension partitioning and temporal dimension partitioning. A cloud-native distributed real-time rendering framework according to claim 1.

6. The aforementioned spatial dimension division supports the tile-base method, dividing the rendering task by the dimensions of the final screen, and the spatial dimension division can be changed by altering the complexity of the rendering task. The cloud-native distributed real-time rendering framework according to claim 5.

7. The aforementioned division of the time dimension means that in a distributed real-time rendering system, the rendering task by the logic node transmits point data representing the amount of change in the virtual world at an update frequency of the logic frame rate of the virtual world, and the current architectural design allows for the rendering node and logic node to operate at different frame rates. The cloud-native distributed real-time rendering framework according to claim 5.

8. In the aforementioned cloud-native distributed real-time rendering framework, a predictive model is added to the rendering node for processing the point sequence data of the logic result. A cloud-native distributed real-time rendering framework according to claim 1.

9. The logic node among the distributed nodes is responsible for executing the logic code of real-time interaction content, responding to external inputs from the entire system, generating computation and rendering tasks, scheduling computation and rendering tasks, receiving the results of computation tasks, and maintaining the logic status of the virtual world. A cloud-native distributed real-time rendering framework according to claim 1.

10. The compute node among the distributed nodes operates a system that executes compute tasks, receives compute tasks sent from the logic node, performs compute locally, and then returns the results to the logic node. A cloud-native distributed real-time rendering framework according to claim 1.

11. The compute node is a stateless node, which only receives tasks and returns results, does not store the status of its content, has no contextual relationships, and if the stateless node fails, it is replaced by another node. A cloud-native distributed real-time rendering framework according to claim 1.

12. The logic node is responsible for generating computation and rendering tasks in response to user input, the rendering node is responsible for processing rendering tasks that cannot be handled by a single node, and the computational load is divided among them, the compositing node is responsible for compositing the screen, and the total rendering time is the sum of the game node usage time, the maximum usage time of a single computed rendering node, and the compositing node usage time. A cloud-native distributed real-time rendering framework according to claim 1.

13. The data transmission layer comprises an ECS data transmission module, a rendering data transmission module, and other data transmission modules, wherein the ECS data transmission module is used to process component data of corresponding entities in the ECS, the rendering data transmission module is used to process rendering result data of rendering nodes, and the other data transmission modules are used to process serialized data blocks of other data structures, and in the event of performance jitter, the data transmission modules use a BERT structure deep learning model as a backup module. A cloud-native distributed real-time rendering framework according to any one of claims 1 to 12.

14. The computation task rules in the aforementioned network synchronization layer are divided into asynchronous update mode or frame rate-based update mode. A cloud-native distributed real-time rendering framework according to any one of claims 1 to 12.

15. In response to user input, the logic node processes the logic to generate a computation task, This involves distributing computation tasks to computation nodes, and having the computation nodes process the computation tasks for the current frame, The calculation results required for the current frame are returned to the logic node, generating a rendering task, and Distributing rendering tasks and having rendering nodes process rendering tasks, The compositing node handles rendering and compositing tasks, This includes performing a stream push of the rendered screen and the user receiving the stream push screen. A cloud-native, distributed, real-time rendering method.

16. This includes providing a logic frame prediction model as a pre-trained model for training on the status of a specific virtual world, and further performing end-to-end training for the logic data structure of each virtual world. The cloud-native distributed real-time rendering method according to claim 15.

17. The training of the aforementioned pre-trained model is performed as follows: After the development of the real-time interaction system is complete, the inputs and outputs of each system are determined, and the input / output data sequences are acquired during the operation of the real-time interaction system development. The cloud-native distributed real-time rendering method according to claim 16.

18. Further training of the predictive model, The training of the aforementioned predictive model is as follows: The process involves acquiring frame sequence data and randomly replacing one of three data points—the input to the simulation system by the logic system, the output of the simulation system, and the output to the rendering system after the logic system updates the simulation results—with a single special character. The process involves acquiring the substituted data, outputting it as an output sequence to the embedded layer of a single data component, and encoding the output features. This includes obtaining encoded features and processing them through a single standard transformer structure, with the expectation that the resulting output will be able to predict data that has been replaced with special symbols by the feature encoding. The cloud-native distributed real-time rendering method according to claim 16.

19. The trained model further includes, if the logic system or simulation system cannot complete a computational task at a given frame rate, using data sequences that have been replaced with special symbols representing data from other systems that can complete the computation and data from systems that cannot complete the computation, and inputting these into the predictive model to predict the output of the system. The cloud-native distributed real-time rendering method according to claim 18.

20. The cloud-native distributed real-time rendering method is executed by the cloud-native distributed real-time rendering framework described in any one of claims 1 to 14. A cloud-native distributed real-time rendering method according to any one of claims 15 to 19.

21. Memory configured to store software applications, A processor configured to execute the software application, wherein the program portion of the software application is a processor that executes the cloud-native distributed real-time rendering method described in any one of claims 15 to 19. A cloud-native, distributed, real-time rendering device.

22. When executed by the processor, a computer program is stored which performs the cloud-native distributed real-time rendering method according to any one of claims 15 to 19. Computer-readable storage medium.

23. The system includes a logic node service module, a compute node service module, a rendering node processing module, a compositing node service module, and a real-time communication RTC or real-time system RTS service module, which respond to user input operations. The logic node service module is configured to process logic to generate compute tasks and distribute the compute tasks to the compute node service module. The compute node service module is configured to execute compute tasks and process compute tasks and asynchronous compute tasks for the current frame. After processing the compute tasks for the current frame, the compute node service module is configured to return the compute results required for the current frame to the logic node service module. The logic node service module is configured to generate rendering tasks and distribute the rendering tasks to the rendering node processing module. The rendering node processing module is configured to process rendering tasks. The compositing node service module is configured to process rendering and compositing tasks and perform screen compositing. The RTC or RTS service module performs stream pushing of rendered screens and is configured for the user to receive stream-pushed screens. A cloud-native, distributed, real-time rendering system.

24. When processing rendering tasks that cannot be completed by a single node, the logic node service module divides the rendering task, and the divided rendering tasks are assigned to multiple rendering node processing modules to distribute the computational load. The multiple rendering node processing modules then divide the computational load and jointly complete the divided rendering tasks using a collaborative rendering method. The cloud-native distributed real-time rendering system according to claim 23.

25. The aforementioned compute node service module is a stateless node. A cloud-native distributed real-time rendering system according to claim 23 or 24.

Images