Cloud native distributed real-time rendering framework, rendering method and device

By using a cloud-native distributed real-time rendering framework and leveraging multi-node collaborative rendering and logical frame prediction models, the low latency and computing power limitations of real-time 3D rendering in existing technologies are solved, enabling an ultra-low latency interactive real-time 3D experience that efficiently processes unstructured data.

WO2024239876A1PCT designated stage expired Publication Date: 2024-11-28YOU SAN DI TECHNOLOGY (SHANGHAI) CO LTD

Patent Information

Application Number
PCT/CN2024/089050
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-04-22
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing cloud rendering technologies cannot meet the low latency requirements of real-time 3D rendering, and traditional distributed rendering frameworks cannot achieve unlimited elastic scaling computing power and cannot handle unstructured data.

Method used

It adopts a cloud-native distributed real-time rendering framework, including a network topology link establishment module, a traditional interactive real-time 3D framework adaptation layer, an ECS-based distributed development framework, a network synchronization layer, a data transmission layer, and a network transport layer. Through collaborative rendering of multiple distributed nodes, it uses a logical frame prediction model and a collaborative rendering mechanism to achieve data synchronization and task allocation.

Benefits of technology

It achieves efficient processing of real-time streaming unstructured data, provides an interactive real-time 3D experience with ultra-low latency, breaks through the computing power limitations of a single device, realizes elastic scaling of cloud computing power, and improves the success rate of rendering and compositing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089050_28112024_PF_FP_ABST
    Figure CN2024089050_28112024_PF_FP_ABST
Patent Text Reader

Abstract

A cloud native distributed real-time rendering framework, comprising a network topology structure link establishment module, a traditional interactive real-time 3D framework adaptation layer, an ECS-based distributed development framework, a network synchronization layer, a data transport layer and a network transport layer; a plurality of domains of the traditional interactive real-time 3D framework adaptation layer are split into a plurality of service modules, each type of service modules corresponding to one type of distributed nodes; according to a data processing flow and the differences of data transport types of multiple types of distributed nodes, the network topology structure link establishment module establishes a link in a TCP or UDP or RDMA type for the multiple types of distributed nodes; the ECS-based distributed development framework is compatible with a development system to automatically execute data synchronization; according to a computing task rule and a frame rate, the network synchronization layer synchronizes input and output data of the multiple types of distributed nodes of the cloud native distributed real-time rendering framework; according to data characteristics of different systems and requirements of the systems for reachability, the network transport layer encapsulates the TCP or UDP or RDMA link mode for an upper layer to select.
Need to check novelty before this filing date? Find Prior Art

Description

Cloud-native distributed real-time rendering framework, rendering methods, and devices

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 23, 2023, with application number 202310584110.6, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to a cloud-native distributed real-time rendering framework, rendering method, and device. Background Art

[0003] The main technologies and products in the field of cloud rendering focus on the development of technologies such as cloud rendering and real-time communication (RTC) / real-time system (RTS) streaming. In essence, it is still to install the original client application on a single cloud computing instance to achieve a remote desktop experience.

[0004] Common real-time streaming big data processing frameworks, such as Spark Streaming, Storm, and Apache Flink, are mostly distributed. However, they process structured data rather than the unstructured data required in real-time three-dimensional (3D) environments. Therefore, the processed data is often linked to a database. Furthermore, the frameworks require coordination with the database, resulting in overall data processing delays ranging from hundreds to thousands of milliseconds, making it impossible to achieve the 60 frames per second (FPS) or even higher 16 milliseconds required for real-time 3D rendering.

[0005] In addition, traditional distributed rendering is mostly used in the form of render farms (both cloud-based and privately deployed) for computer graphics (CG) and traditional film and television industries, using distributed clusters for non-real-time rendering tasks.

[0006] Among traditional interactive real-time 3D rendering frameworks, the most representative game engine frameworks, such as Unity, Unreal, and Cocos, can meet the needs of real-time interactive rendering and process unstructured data. However, they lack cloud-native distributed frameworks and cannot support elastic scalability and unlimited computing power. Traditional interactive real-time 3D rendering frameworks can only use the computing power of a single device. The computing power of a single device cannot be elastically scaled like cloud-native distributed systems, so they can only use the computing power that the framework can dispatch on a single device. Therefore, traditional interactive real-time 3D rendering frameworks are limited by the computing power limit of the device. Big data real-time streaming processing frameworks such as Spark Streaming, Storm, and Apache Flink, although based on cloud-native distributed frameworks and capable of elastic scalability and unlimited computing power, cannot meet the needs of real-time interactive rendering and process unstructured data.

[0007] Summary of the Invention

[0008] The present application provides a cloud-native distributed real-time rendering framework, which is used to process real-time streaming unstructured data, including a network topology link establishment module, a traditional interactive real-time 3D framework adaptation layer, a distributed development framework based on an entity-component system (ECS), a network synchronization layer, a data transmission layer, and a network transmission layer. The cloud-native distributed real-time rendering framework provides a variety of distributed nodes, including synthesis nodes, rendering nodes, logical nodes, and computing nodes; the multiple domains of the traditional interactive real-time 3D framework adaptation layer are divided into multiple types of service modules, each type of service module corresponds to a distributed node; the network topology link establishment module is used to establish a communication control protocol (TCP) or a user datagram protocol (UDP) or a remote direct memory access (RDM) for the multiple distributed nodes according to the data processing flow and the data transmission type of the multiple distributed nodes. The ECS-based distributed development framework is used to automatically complete data synchronization with compatible development systems; the network synchronization layer is used to synchronize the input and output data of various distributed nodes of the cloud-native distributed real-time rendering framework according to computing task rules and frame rates; the data transmission layer is used to provide a variety of data packet sending and receiving services; the network transmission layer is used to encapsulate TCP, UDP or RDMA link modes for selection by the upper-layer network topology structure link establishment module based on the data characteristics and accessibility requirements between different systems; the ECS-based distributed development framework is also used to provide a business logic development framework, and the traditional interactive real-time 3D framework adaptation layer parallel to it is also used to provide the adaptability of the Unity functional module on the ECS-based distributed development framework; when processing a rendering task that cannot be completed by a single rendering node, the logic node splits the rendering task and assigns the split rendering task to multiple rendering nodes, so that the multiple rendering nodes can jointly complete the split rendering task through collaborative rendering.

[0009] This application also provides a cloud-native distributed real-time rendering method, including:

[0010] In response to user input operations, the logic node logic processing generates computing tasks; publishes computing tasks to computing nodes, and computing nodes process computing tasks for the current frame; returns the required computing results for the current frame to the logic node, generating rendering tasks; publishes rendering tasks, and rendering nodes process rendering tasks; synthesis nodes process rendering and synthesis tasks; and executes rendering screen streaming, and users receive the streaming screen.

[0011] This application also provides a cloud-native distributed real-time rendering device, including:

[0012] The memory is configured to store a software application, and the processor is configured to execute the software application, wherein the program portion of the software application correspondingly executes the cloud-native distributed real-time rendering method.

[0013] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the cloud-native distributed real-time rendering method.

[0014] The present application also provides a cloud-native distributed real-time rendering system, which includes a logic node service module, a computing node service module, a rendering node processing module, a synthesis node service module and an RTC or RTS service module; in response to user input operations, the logic node service module is configured to perform logic processing to generate computing tasks and publish the computing tasks to the computing node service module; the computing node service module is configured to execute computing tasks, process current frame computing tasks and process asynchronous computing tasks, the computing node service module is configured to process current frame computing tasks and return the required computing results for the current frame to the logic node service module, the logic node service module is configured to generate rendering tasks and publish rendering tasks to the rendering node processing module, the rendering node processing module is configured to process rendering tasks; the synthesis node service module is configured to process rendering and synthesis tasks and perform screen synthesis; the RTC or RTS service module is configured to execute rendering screen streaming, and the user receives the streamed screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a schematic diagram of the structure of a cloud-native distributed real-time rendering framework provided by an embodiment of the present application;

[0016] FIG2 is a schematic diagram of an interface of a cloud-native distributed real-time rendering framework provided in an embodiment of the present application;

[0017] FIG3 is a network topology result link diagram of a cloud-native distributed real-time rendering framework provided by an embodiment of the present application;

[0018] FIG4 is a sequence-to-sequence prediction model provided by an embodiment of the present application;

[0019] FIG5 is a system runtime diagram of a cloud-native distributed real-time rendering framework provided by an embodiment of the present application;

[0020] FIG6 is a schematic diagram of the overall system latency of a cloud-native distributed real-time rendering framework provided by an embodiment of the present application;

[0021] FIG7 is a diagram of a real-machine project test scenario using a cloud-native distributed real-time rendering framework provided in an embodiment of the present application;

[0022] Figure 8 is a pre-trained model diagram of a cloud-native distributed real-time rendering framework provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following description is intended to disclose the present application so that those skilled in the art can implement the present application. The embodiments described below are for illustrative purposes only.

[0024] The terms “a” and “an” should be understood as “at least one” or “one or more”, that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be plural.

[0025] The present application relates to computer programs. As shown in Figure 1, Figure 1 is a schematic diagram of the structure of a cloud-native distributed real-time rendering framework provided by an embodiment of the present application. The cloud-native distributed real-time rendering framework can give full play to the elastic scalability and high-availability computing power characteristics of the cloud network, and is specifically designed for processing real-time streaming unstructured data. It can be applied to the ultra-low latency requirements of the interactive real-time 3D field, making the interactive real-time 3D experience more realistic and immersive, breaking through the virtual experience of a single computing device.

[0026] As shown in Figure 1, the cloud-native distributed real-time rendering framework includes a network topology link establishment module, a traditional interactive real-time 3D framework adaptation layer, an ECS-based distributed development framework, a network synchronization layer, an ECS data transmission module, a rendering data transmission module, other data transmission modules, and a network transmission layer. The network topology structure link establishment module is used to establish TCP, UDP or RDMA type links for multiple distributed nodes according to the data processing flow (for example, the logical world affects the logical input to generate simulation computing tasks, the simulation computing results are returned to affect the logical world, the logical world updates the rendering world to initiate rendering tasks from different viewpoints, the rendering node completes the rendering task and outputs the rendering results to the synthesis node to complete cloud rendering streaming). The data transmission layer is used to provide various data packet sending and receiving services. The ECS data transmission module is used for data transmission between logical nodes and simulation nodes, and the rendering data transmission module is used for data transmission between rendering nodes and synthesis nodes. Other data transmission modules are used to supplement special data communication requirements of non-ECS data types between multiple nodes. The ECS-based distributed development framework is used to provide a business logic development framework, and the traditional interactive real-time 3D framework adaptation layer parallel to it provides the adaptation capability of the Unity functional module on the ECS-based distributed development framework.

[0027] In this embodiment of the present application, the traditional interactive real-time 3D framework adaptation layer is implemented as a Unity game engine framework. The multiple domains of the traditional interactive real-time 3D framework adaptation layer are split into multiple types of service modules, and each type of service module corresponds to a type of distributed node. It is defined as a System in the ECS framework, and the input and output data of each module are defined as the component data (Entity-Component) of the corresponding entity in the ECS. In this embodiment of the present application, Unity's rendering module is defined as a rendering service system, Unity's post-processing and integrated cloud rendering RTC push flow module is defined as a synthesis service system, Unity's logic processing module is defined as a logic service system, and the self-developed real-time cloth solution module is defined as a computing service system. The real-time cloth solution module here serves as the task processing system of the computing node, and the computing node can use any user-defined computing system as a computing service system. Using Unity with the development template component provided by this application installed, you can directly compile executable binary files that run on the above four types of service modules (distributed nodes).

[0028] The cloud-native distributed real-time rendering framework of this application supports compiling binary files for Linux and Windows, as shown in Figure 2. Among them, the Client compilation parameters generate binary files of synthesis nodes and rendering nodes.

[0029] Compositing nodes are essentially special render nodes that can independently complete rendering tasks while a single render node can render a scene. Therefore, they share the same compilation parameters and produce the same binary file, differing only in the startup parameters. Remote nodes generate binaries for compute nodes, while Server nodes generate binaries for logical nodes. The term "Server" is a concept in network architecture and is the only stateful part of a distributed system, thus being defined as a service module. These are uniformly triggered by the Build menu.

[0030] The ECS-based distributed development framework described in this application is compatible with the Unity ECS development process, allowing developers to develop interactive real-time 3D rendering experiences within their familiar development environment. Developers who need to develop custom computing systems simply use the distributed computing application programming interface (API) and continue to define their computing systems, input and output data, and task execution rules according to the ECS software design method. The ECS-based distributed development framework automatically synchronizes data, such as in a fabric simulation system.

[0031] The network synchronization layer of the present application synchronizes the input and output data of various nodes according to the computing task rules and frame rates. Among them, the computing task rules are divided into asynchronous update mode or frame rate update mode.

[0032] The data transmission module of the present application processes three types of data: (1) component data of the corresponding entity of the ECS, which is sent directly to the transport layer; (2) rendering result data of the rendering node, which is encoded using the codec function supported by the hardware, sent to the transport layer, and decoded by the receiving end hardware, such as supporting the use of NVENC codec acceleration on the NV graphics card; and (3) serialized data blocks of other complex data structures, which are sent directly to the transport layer. The data transmission module uses a deep learning model based on the Bidirectional Encoder Representations from Transformers (BERT) structure as a backup module when the system module has performance jitter, increasing the stability of system performance.

[0033] The network transport layer of this application is optimized for cloud-network computer room network environments. Based on the data characteristics and inter-system accessibility requirements between different systems, the network transport layer encapsulates three possible link methods available in cloud-network environments for selection by the upper-layer network topology link establishment module: TCP, UDP, and RDMA. In the embodiments of this application, UDP links are used between compute nodes and logic nodes, TCP links are used between logic nodes and rendering nodes, and TCP links are used between rendering nodes and compositing nodes.

[0034] The network topology link establishment module of the present application links the central processing unit (CPU) or graphics processing unit (GPU) nodes allocated by the cloud platform Infrastructure as a Service (IaaS) layer according to the network topology structure defined by the framework according to the network topology result shown in Figure 3. Among them, the synthesis node and rendering node use GPU nodes, and other nodes use CPU nodes.

[0035] Distributed nodes include synthesis nodes, rendering nodes, logical nodes and computing nodes. When processing a rendering task that a single node cannot complete, the rendering node will split the rendering task by the logical node and assign the split rendering task to N rendering nodes, so that the N rendering nodes can jointly complete the split rendering task through collaborative rendering. The rendering node is a stateless node and can be quickly replaced. The cloud native distributed real-time rendering framework of the present application provides collaborative rendering. The tile-based distributed rendering is a process in which multiple rendering nodes share the rendering task. However, the simulation speed between multiple rendering nodes and the service module (server) node does not match and the server fluctuates, resulting in inconsistent simulation speeds between the nodes participating in the rendering task. These problems will lead to a low success rate of picture synthesis. Therefore, it is necessary to use a collaborative rendering mechanism based on a central service module node to manage and coordinate the workloads of multiple rendering nodes, and make temporary compensation in the interpolation and frame filling module of the deep learning prediction network based on the Transformer structure, so as to improve the success rate of rendering synthesis.

[0036] For the coordination of rendering nodes within a group, when the rendering time per frame of a rendering node within the group is greater than the preset threshold, it is considered that the rendering nodes within the group fluctuate greatly. The actual update frequency and historical data of multiple nodes within the group are comprehensively considered to calculate the appropriate target update frequency within the group.

[0037] Due to varying workloads on different render nodes, the rendering time for each frame fluctuates. For example, if render node 1 has a heavier workload and renders a frame in 10 milliseconds, while render node 2 has a lighter workload and renders a frame in 5 milliseconds, the resulting large fluctuations in the rendering time between the two render nodes will result in different output times. Without any action, these large fluctuations in rendering time can cause screen tearing.

[0038] In one embodiment, the target frame rates are set to 30 frames and 60 frames, corresponding to maximum rendering times of 33 milliseconds and 16 milliseconds, respectively. If the rendering node cannot complete rendering within these two time windows, the final image output will be unstable. Therefore, the preset thresholds can be set to 33 milliseconds and 16 milliseconds.

[0039] For rendering nodes and servers, the network sending frequency of the service module node to the data status of the group is dynamically adjusted according to the target frequency calculated by different groups.

[0040] For the calculation of dynamic sending frequency, it is calculated based on the target frequency of the group and the frequency parameters of the server itself to determine whether the current frame is in the sending data state for the group.

[0041] Rendering tasks can be split in two ways: spatial and temporal. These two splitting methods can be used individually or in combination to achieve high performance and a high frame rate experience.

[0042] Spatial dimensional splitting supports a tile-based approach, splitting rendering tasks based on the dimensions of the final image. This tile splitting method can adjust the complexity of the rendering task and change the configuration, such as splitting into four, three, or two. This method was originally used to split rendering tasks within the GPU's internal parallel processors, and is commonly used on mobile GPUs in conjunction with caches. This application combines the tile-based splitting method with a GPU-driven rendering pipeline. Relying on the GPU-driven rendering pipeline, tile-based splitting of rendering tasks does not rely on cache coordination, making it suitable for distributed network structures and can be achieved by synchronizing small amounts of data.

[0043] Splitting of the time dimension In a distributed real-time rendering system, the rendering tasks generated by the logic nodes will send dotting data that express the amount of change in the world at the update frequency of the logical frame rate of the virtual world. The current architectural design allows rendering nodes and logic nodes to run at different frame rates. In the cloud-native distributed real-time rendering framework of the present application, a prediction model for processing logic result dotting sequence data is added to the rendering node, so that a higher and more stable frame rate can be obtained with more GPU computing power. For example, in an embodiment, when the prediction model can predict the logical dotting data of 2 frames within a given delay, the two rendering nodes can be responsible for rendering odd frames and even frames respectively. In this way, the computational task splitting of the entire system result in the time dimension is achieved. Frame-based (rendering tasks are split in a frame-by-frame manner, for example, one rendering node for odd frames and one rendering node for even frames) distributed rendering technology is a technology in the field of offline rendering, and is mostly used in video production-related fields, such as film and television, advertising, etc. Because offline rendering does not have real-time interaction, all behaviors in the virtual world are deterministic, so it can be easily processed in parallel by frame; however, in the field of real-time rendering, because there is real-time interaction, and the entire system responds to real-time interaction on the client, there is neither future deterministic behavior nor redundant computing power to support future scene rendering, so this technology cannot be used in the field of real-time rendering; the cloud-native distributed real-time rendering framework described in this application, by splitting the time dimension and adding a prediction model for processing logic results and dot sequence data to the rendering node, can well solve the problem of combining the two.

[0044] The cloud-native distributed real-time rendering framework of the present application provides a logical frame prediction model, which is a pre-trained model trained for a specific virtual world state. As shown in Figure 4, a deep learning module with transformer as the basic interface is used as a sequence-to-sequence prediction model for the output data of multiple computing nodes in the backbone network. It serves as a burst compensation mechanism for the inability of a single computing node to complete the computing task at the expected efficiency due to sudden performance jitter. After the real-time interactive development is completed, it is very easy to obtain a large amount of logical frame dot data, and the model is trained using mask modeling. End-to-end training is performed for the logical data structure of each virtual world. During runtime, only a decoder is needed to accurately predict the logical dot data.

[0045] The training method of the pre-trained model is: after the real-time interactive system is developed, the input and output of each system are determined, and a large number of input and output data sequences can be obtained during the development and operation of the real-time interactive system.

[0046] In one embodiment, as shown in Figure 8, which illustrates the training method for the pre-trained model, a frame of the real-time interactive system contains four types of data: user-generated input to the logic system, input to the simulation system generated by the logic system, output from the simulation system, and output from the logic system to the rendering system after updating the simulation results. Figure 8 shows a data sequence with the [FS] feature symbol as a frame delimiter and the [SP] feature symbol as a system data delimiter.

[0047] Using Mask Modeling and a bidirectional transformer structure similar to BERT, a prediction model for a given system input and output can be trained. The training steps are:

[0048] S10: Collect the above frame sequence data, and randomly use a special symbol [Mask] to replace one of the following three data with a probability of 15%: the input of the simulation system generated by the logic system, the output of the simulation system, and the output of the logic system to the rendering system after updating the simulation results.

[0049] S20: Output the data of step S10 as an output sequence to an embedding layer of a 30000×H (256) data component to encode the output features.

[0050] S30: The feature encoding of step S20 is passed through a standard transformer structure with H being 256 and L being 8, and it is expected that the output obtained can predict the data replaced by the special symbol of [Mask] through feature encoding.

[0051] After training, the model is used as a data sequence that uses data from other systems that can complete the calculation and data from systems that cannot complete the calculation replaced by [Mask] when the logic system or simulation system cannot complete the calculation task at the specified frame rate. The data is input into the prediction model to predict the system output. By using additional computing power, the performance stability of the entire system is greatly improved.

[0052] The logical nodes in the distributed nodes run the main logic code of real-time interactive content. They are mainly responsible for responding to external inputs of the entire system (such as user input or other sensor signals), generating computing and rendering tasks, scheduling computing and rendering tasks, receiving computing task results, and maintaining the logical state of the virtual world.

[0053] Compute nodes within the distributed system run systems that perform computational tasks, such as fabric simulation, animation simulation, and time-consuming general-purpose tasks. Compute nodes receive computational tasks from logic nodes, perform local calculations, and return the results to the logic nodes. Compute nodes are stateless and can be quickly replaced.

[0054] The stateless nodes in the cloud-native distributed real-time rendering framework described in this application only receive tasks and return results. They do not store any content state or context. If a stateless node fails, it can be quickly replaced with another node. This architecture ensures high availability in distributed rendering and computing.

[0055] Figure 5 is a system runtime diagram of a cloud-native distributed real-time rendering framework provided by an embodiment of this application. This distributed architecture allows for the ability to split modules as needed, assigning dedicated nodes to specific tasks and distributing performance bottlenecks. This eliminates the overlap between computational and rendering time seen in traditional cloud content.

[0056] Figure 6 is a schematic diagram of the overall system latency of a cloud-native distributed real-time rendering framework provided in an embodiment of the present application. The logical node (service module) is responsible for responding to user operations and generating computing and rendering tasks. The rendering node is responsible for processing rendering tasks that cannot be handled by a single machine and is split into 1 / N computing tasks. The synthesis node (Postprocess) is responsible for the synthesis of the picture.

[0057] Total rendering time = game node time + maximum single rendering (compute) node time + compositing node time. This improves the rendering architecture and provides technical support for low latency.

[0058] Figure 7 is a diagram of a real-machine project test scenario using a cloud-native distributed real-time rendering framework provided by an embodiment of the present application. A scene with an overall triangle count of approximately 10 billion faces, but an average viewing angle of approximately 200 million faces, was used to test data in a cloud network environment. The comparative test results are shown in the following table:

[0059] Test data shows that in such a complex scenario, a single 3060 can no longer provide a 60FPS non-stuttering interactive experience. However, through distributed rendering technology, relying on two 3060 rendering nodes, users can still experience a 60FPS or above non-stuttering interactive experience. Four 3060 rendering nodes can correspond to a higher frame rate. If combined with the usual rendering engine's frame rate limit mode, the overall system frame rate can be limited to 60FPS. In the case of redundant computing power, the frame rate can be very stable, but the experimental environment needs to test the upper limit of rendering performance, so the rendering engine's frame rate limit mode was turned off during the test.

[0060] Therefore, the cloud-native distributed real-time rendering framework of this application can provide a cloud-native distributed framework suitable for interactive real-time 3D rendering tasks, and adapt a real-time 3D rendering engine Unity widely used in the industry to the framework. There is real project experimental data to prove that the framework can indeed schedule multi-node computing power to complete the rendering and computing power tasks generated by the Unity engine, breaking through the computing power limit to obtain better overall system performance.

[0061] Corresponding to the embodiment of the cloud-native distributed real-time rendering framework described in this application, according to another aspect of this application, a cloud-native distributed real-time rendering method is also provided, which describes a solution for solving the problems raised in this application, based on a computer program processing flow, by executing a computer program compiled according to the flow to control or process external or internal objects of the computer. The cloud-native distributed real-time rendering method includes the following steps:

[0062] In response to user input operations, the logic node logic processing generates computing tasks; publishes computing tasks to computing nodes, and computing nodes process computing tasks for the current frame; returns the required calculation results for the current frame to the logic node, generating rendering tasks; publishes rendering tasks, and rendering nodes process rendering tasks; synthesis nodes process rendering and synthesis tasks; executes rendering screen streaming, and users receive the streaming screen.

[0063] The cloud-native distributed real-time rendering method further includes the step of: computing nodes processing asynchronous computing tasks.

[0064] The cloud-native distributed real-time rendering method also includes the steps of: synchronizing input and output data of multiple nodes according to computing task rules and frame rates; and dividing the computing task rules into an asynchronous update mode or a frame rate update mode.

[0065] The cloud-native distributed real-time rendering method also includes executing a data transmission module processing step: obtaining component data of the corresponding entity of the ECS and sending it directly to the transport layer; obtaining rendering result data of the rendering node, encoding it using the codec function supported by the hardware, sending it to the transport layer, and decoding it by the receiving end hardware; and obtaining serialized data blocks and sending them directly to the transport layer.

[0066] The cloud-native distributed real-time rendering method further includes the following steps:

[0067] Based on the data characteristics and accessibility requirements between different systems, TCP, UDP, or RDMA link modes are provided for the upper layer to choose. UDP links are used between computing nodes and logical nodes, TCP links are used between logical nodes and rendering nodes, and TCP links are used between rendering nodes and synthesis nodes.

[0068] The cloud-native distributed real-time rendering method further includes the steps of: when a rendering task that cannot be completed by a single node is detected, splitting the rendering task by a logic node and assigning the split rendering tasks to multiple rendering nodes, so that the multiple rendering nodes jointly complete the split rendering tasks through collaborative rendering.

[0069] The cloud-native distributed real-time rendering method further includes the following steps: for rendering node coordination within a group, when the rendering node fluctuations within the group are large, comprehensively considering the actual update frequencies and historical data of multiple nodes within the group, and calculating the appropriate target update frequency within the group; for rendering nodes and servers, dynamically adjusting the network transmission frequency of the service module node for the group's data status based on the target frequencies calculated for different groups; for dynamic transmission frequency calculation, determining whether the current frame sends data status to the group based on the group's target frequency and the server's own frequency parameters.

[0070] The cloud-native distributed real-time rendering method further includes the steps of: performing spatial dimension splitting; and performing temporal dimension splitting. In the spatial dimension splitting step, a tile-based splitting approach is combined with the rendering pipeline of a GPU-driven rendering task. In the temporal dimension splitting step, the rendering task generated by the logic node sends dot data representing world changes at an update frequency consistent with the virtual world's logical frame rate, allowing rendering nodes and logic nodes to run at different frame rates. A prediction model for processing the dot sequence data of the logical results is added to the rendering node.

[0071] The cloud-native distributed real-time rendering method also includes the steps of providing a logical frame prediction model as a pre-trained model for training specific virtual world states, using mask modeling to train the model and performing end-to-end training for each virtual world's logical data structure. The pre-trained model training steps include: after the real-time interactive system is developed, the input and output of each system are determined, and a large number of input and output data sequences can be obtained during the development and operation of the real-time interactive system; and using mask modeling to train a prediction model for the given system's input and output using a BERT-like bidirectional transformer structure.

[0072] The cloud-native distributed real-time rendering method also includes the steps of: obtaining input data of a logic system generated by a user; obtaining input data of a simulation system generated by the logic system, obtaining output data of the simulation system, and obtaining output data of the logic system to the rendering system after updating the simulation results; and assigning a data sequence with an "FS" feature symbol as a frame separator and an "SP" feature symbol as a system data separator.

[0073] The cloud-native distributed real-time rendering method further includes a prediction model training step:

[0074] Obtain frame sequence data, and with a probability of 15%, randomly use a special symbol "Mask" to replace one of the following three data: the input of the simulation system generated by the logic system, the output of the simulation system, and the output of the logic system to the rendering system after updating the simulation results; obtain the data in the previous step as the output sequence and output it to the embedding layer of a 30000X H data component to encode the output features; and obtain the feature encoding of the previous step through a standard transformer structure with H of 256 and L of 8, hoping that the output obtained can predict the data replaced by the special symbol "Mask" through feature encoding.

[0075] The cloud-native distributed real-time rendering method also includes the following steps: when the logic system or simulation system cannot complete the calculation task at the specified frame rate, the trained model is used as a data sequence in which data from other systems that can complete the calculation and data from systems that cannot complete the calculation are replaced with a "Mask", and the data is input into the prediction model to predict the output of the system.

[0076] It will be appreciated by those skilled in the art that the embodiments of the present application may be provided in the form of a method, system, or computer program product. Therefore, the present application may take the form of an all-hardware embodiment, an all-software embodiment, or a combination of software and hardware. The method of the present application may be implemented by hardware, software, or a combination of software and hardware. The present application may be implemented in a centralized manner in at least one computer system, or in a decentralized manner by different parts distributed across several interconnected computer systems. Any computer system or other device that can implement the method is applicable. A common combination of hardware and software may be a general-purpose computer system with a computer program installed, which controls the computer system by installing and executing the program so that it operates according to the method.

[0077] The present application may be embedded in a computer program product, which includes all the features that enable the method described herein to be implemented. The computer program product is contained in one or more computer-readable storage media, and the computer-readable storage medium has a computer-readable program code contained therein. According to another aspect of the present application, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method of the present application are capable of executing. A computer storage medium is a medium for storing a discontinuous physical quantity in a computer memory. Computer storage media include but are not limited to semiconductors, disk memories, magnetic cores, magnetic drums, magnetic tapes, laser discs, non-transitory storage media, etc. Computer storage media are not limited to the aforementioned examples, and the aforementioned examples are merely examples.

[0078] According to another aspect of the present application, a cloud-native distributed real-time rendering device is provided, comprising: a software application, a memory configured to store the software application, and a processor configured to execute the software application. The software application program is capable of correspondingly executing the steps of the cloud-native distributed real-time rendering method described herein.

[0079] Corresponding to the embodiments of the cloud-native distributed real-time rendering framework and method described in the present application, according to another aspect of the present application, a cloud-native distributed real-time rendering system is also provided, and the cloud-native distributed real-time rendering system includes a logic node service module, a computing node service module, a rendering node processing module, a synthesis node service module, and an RTC or RTS service module.

[0080] In response to user input operations, the logical node service module is configured to logically process and generate computing tasks, and publish the computing tasks to the computing node service module; the computing node service module is configured to execute computing tasks, process the current frame computing tasks and process asynchronous computing tasks, and the computing node service module is configured to return the required computing results of the current frame to the logical node service module after processing the current frame computing tasks, and the logical node service module is configured to generate rendering tasks and publish the rendering tasks to the rendering node processing module, and the rendering node processing module is configured to process rendering tasks; the synthesis node service module is configured to process rendering and synthesis tasks, and perform screen synthesis; the RTC or RTS service module is configured to execute the rendering screen push, and the user receives the pushed screen.

[0081] The logical node service module runs the primary logic code for real-time interactive content. In response to user input or sensor signals from outside the system, it generates and schedules computation and rendering tasks, receives computational task results, and maintains the logical state of the virtual world. The cloud-native distributed real-time rendering system provides time-based decomposition. The rendering tasks generated by the logical node service module transmit dot data representing world changes at the virtual world's logical frame rate.

[0082] The computing node service module is a stateless node and can be quickly replaced.

[0083] A TCP connection is used between the logical node service module and the rendering node processing module, a UDP connection is used between the computing node service module and the logical node service module, and a TCP connection is used between the rendering node processing module and the synthesis node service module.

[0084] The cloud-native distributed real-time rendering system of this application, when processing rendering tasks that cannot be completed by a single node, splits the rendering tasks by the logical node service module and assigns the computational load to multiple rendering node processing modules, which are then completed together through collaborative rendering. Total rendering time = game node time + maximum time of a single rendering (computing) node + synthesis node time. This method provides technical support for low latency.

[0085] In the cloud-native distributed real-time rendering system of the present application, the rendering node processing module and the logic node service module operate at different frame rates. A prediction model for processing the logical result dot sequence data of the logic node service module is added to the rendering node processing module, thereby achieving a higher and more stable frame rate with more GPU computing power.

[0086] The stateless nodes of the cloud-native distributed real-time rendering system of this application only receive tasks and return results, and do not save the status of any content. There is no contextual relationship. If a stateless node fails, it can be quickly replaced with other nodes, thereby ensuring high availability of distributed rendering and computing. The cloud-native distributed real-time rendering system of this application can split modules as much as possible according to demand, let dedicated nodes be responsible for dedicated tasks, and disperse performance bottlenecks. Unlike traditional cloud content, there will be no overlap of computing time and rendering time.

[0087] The cloud-native distributed real-time rendering system also includes a data transmission service module, which is configured to process and send component data of the corresponding entity of the ECS, rendering node rendering result data, and serialized data blocks. The data transmission service module uses a deep learning model with a BERT structure as a backup module when performance jitter occurs in the system module, thereby increasing the stability of system performance.

[0088] Multiple embodiments of the cloud-native distributed real-time rendering framework and method of the present application are applicable to the cloud-native distributed real-time rendering system of the present application.

[0089] This application provides a cloud-native distributed real-time rendering framework, rendering method and device, which can use the computing framework of cloud-based elastic scaling computing power, and can schedule computing power tasks to use multiple computing instances to complete computing tasks.

[0090] This application provides a cloud-native distributed real-time rendering framework, rendering method and equipment, which can perform real-time streaming processing on unstructured data that needs to be processed in the real-time 3D field, such as meshes, animations, special effects, lights, scenes, etc.

[0091] This application provides a cloud-native distributed real-time rendering framework, rendering method, and device that can process data structures processed by real-time rendering engines.

[0092] This application provides a cloud-native distributed real-time rendering framework, rendering method and equipment, providing interactive real-time rendering tasks, which not only handles rendering problems, but also business logic, user input, interactive component logic in the virtual world, animation system, physical simulation system, etc.

[0093] The present application has been described with reference to the flowcharts and / or block diagrams of the methods, systems, and computer program products according to the present application. Each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions (the instructions are passed through the processor of the computer or other programmable data processing device) generate means for implementing the functions specified in one or more blocks of the flowcharts and / or block diagrams.

Claims

1. A cloud-native distributed real-time rendering framework for processing real-time streaming unstructured data, including a network topology link establishment module, a traditional interactive real-time three-dimensional 3D framework adaptation layer, a distributed development framework based on entity-component system ECS, a network synchronization layer, a data transmission layer and a network transmission layer, wherein: The cloud-native distributed real-time rendering framework provides a variety of distributed nodes, including synthesis nodes, rendering nodes, logic nodes, and computing nodes; The multiple domains of the traditional interactive real-time 3D framework adaptation layer are divided into multiple types of service modules, each type of service module corresponds to a distributed node; The network topology structure link establishment module is used to establish a communication control protocol TCP or user data packet protocol UDP or remote direct memory access RDMA type link for the multiple distributed nodes according to the data processing flow and the difference in data transmission types of the multiple distributed nodes; The ECS-based distributed development framework is used to automatically complete data synchronization with compatible development systems; The network synchronization layer is used to synchronize input and output data of various distributed nodes of the cloud-native distributed real-time rendering framework according to computing task rules and frame rates; The data transmission layer is used to provide a variety of data packet sending and receiving services; The network transport layer is used to encapsulate TCP or UDP or RDMA link modes for selection by the network topology link establishment module of the upper layer according to the data characteristics between different systems and the accessibility requirements between the systems; The ECS-based distributed development framework is also used to provide a business logic development framework, and the traditional interactive real-time 3D framework adaptation layer parallel to the ECS-based distributed development framework is also used to provide the adaptation capability of the Unity functional module on the ECS-based distributed development framework; When processing a rendering task that cannot be completed by a single rendering node, the logic node splits the rendering task and distributes the split rendering tasks to multiple rendering nodes, so that the multiple rendering nodes jointly complete the split rendering tasks through collaborative rendering.

2. The cloud-native distributed real-time rendering framework according to claim 1, wherein: For rendering node coordination within a group, when the rendering time per frame of a rendering node within the group is greater than a preset threshold, a target update frequency within the group is calculated according to actual update frequencies and historical data of multiple nodes within the group.

3. The cloud-native distributed real-time rendering framework according to claim 1, wherein: For rendering nodes and servers, the network sending frequency of the server node for each group's data status is dynamically adjusted according to the target frequency calculated by different groups.

4. The cloud-native distributed real-time rendering framework according to claim 1, wherein: The calculation of the dynamic sending frequency is calculated based on the target frequency of the group and the frequency parameters of the server itself to determine whether the current frame is in the sending data state for the group.

5. The cloud-native distributed real-time rendering framework according to claim 1, wherein: The rendering task splitting method includes at least one of splitting in a spatial dimension and splitting in a temporal dimension.

6. The cloud-native distributed real-time rendering framework according to claim 5, wherein: The splitting of the spatial dimension supports a tile-based approach, splitting the rendering task according to the dimension of the final picture, and the splitting of the spatial dimension changes the configuration by changing the complexity of the rendering task.

7. The cloud-native distributed real-time rendering framework according to claim 5, wherein: In the distributed real-time rendering system, the rendering tasks generated by the logic nodes will send dot data expressing the amount of world changes at the update frequency of the logic frame rate of the virtual world. The current architecture design allows rendering nodes and logic nodes to run at different frame rates.

8. The cloud-native distributed real-time rendering framework according to claim 1, wherein: In the cloud-native distributed real-time rendering framework, a prediction model for dotting sequence data of processing logic results is added to the rendering node.

9. The cloud-native distributed real-time rendering framework according to claim 1, wherein: The logic node in the distributed node runs the logic code of the real-time interactive content, and is responsible for responding to the external input of the entire system, generating computing and rendering tasks, scheduling computing and rendering tasks, receiving computing task results, and maintaining the logical state of the virtual world.

10. The cloud-native distributed real-time rendering framework according to claim 1, wherein: The computing node in the distributed nodes runs a system for executing computing tasks, receives computing tasks sent by the logical node, performs calculations locally, and returns the results to the logical node.

11. The cloud-native distributed real-time rendering framework according to claim 1, wherein: The computing node is a stateless node. The stateless node only receives tasks and returns results, does not save the status of the content, and has no contextual relationship. In the event of a failure of the stateless node, other nodes are used to replace the stateless node.

12. The cloud-native distributed real-time rendering framework according to claim 1, wherein: The logic node is responsible for responding to user operations and generating computing and rendering tasks. The rendering node is responsible for processing rendering tasks that cannot be processed by a single machine, and the computing workload is split. The synthesis node is responsible for screen synthesis, wherein the total rendering time is the sum of the game node time, the maximum time of a single rendering computing node, and the synthesis node time.

13. The cloud-native distributed real-time rendering framework according to any one of claims 1 to 12, wherein: The data transmission layer includes an ECS data transmission module, a rendering data transmission module and other data transmission modules, wherein the ECS data transmission module is used to process the entity component data of the ECS, the rendering data transmission module is used to process the rendering result data of the rendering node, and the other data transmission modules are used to process the serialized data blocks of other data structures; in the event of performance jitter, the data transmission module uses a deep learning model with a BERT structure as a backup module.

14. The cloud-native distributed real-time rendering framework according to any one of claims 1 to 12, wherein: The computing task rules in the network synchronization layer are divided into an asynchronous update mode or a frame rate update mode.

15. A cloud-native distributed real-time rendering method, comprising: In response to user input operations, the logic node logic processes and generates computing tasks; Publish computing tasks to computing nodes, and computing nodes process the computing tasks of the current frame; Returns the calculation result required for the frame to the logic node and generates a rendering task; Release rendering tasks, and the rendering nodes process the rendering tasks; Compositing nodes handle rendering and compositing tasks; as well as The rendered image is pushed and the user receives the pushed image.

16. The cloud-native distributed real-time rendering method according to claim 15, further comprising: A logical frame prediction model is provided as a pre-trained model for training a specific virtual world state, and end-to-end training is performed for the logical data structure of each virtual world.

17. The cloud-native distributed real-time rendering method according to claim 16, wherein: The training of the pre-trained model includes: After the real-time interactive system is developed, the input and output of each system are determined, and the input and output data sequences are obtained during the development and operation of the real-time interactive system.

18. The cloud-native distributed real-time rendering method according to claim 16, further comprising prediction model training; The prediction model training includes: Get the frame sequence data, and randomly use a special symbol to replace one of the following three data: the input of the simulation system generated by the logic system, the output of the simulation system, and the output of the logic system to the rendering system after updating the simulation results; Obtain the replaced data as an output sequence and output it to an embedding layer of a data component to encode the output features; and The encoded features are obtained through a standard transformer structure, and the output is expected to be able to predict the data replaced by special symbols through feature encoding.

19. The cloud-native distributed real-time rendering method according to claim 18, further comprising: The trained model is used as a resource for using data from other systems that can complete the calculation and data from other systems that cannot complete the calculation when the logic system or simulation system cannot complete the calculation at the specified frame rate. Based on the data sequence replaced by special symbols, the input prediction model predicts the output of the system.

20. The cloud-native distributed real-time rendering method according to any one of claims 15 to 19, wherein: The cloud-native distributed real-time rendering method is executed by the cloud-native distributed real-time rendering framework described in any one of claims 1 to 14.

21. A cloud-native distributed real-time rendering device, comprising: a memory configured to store software applications; A processor is configured to execute the software application, wherein the program portion of the software application correspondingly executes the cloud-native distributed real-time rendering method as described in any one of claims 15 to 19.

22. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the cloud-native distributed real-time rendering method according to any one of claims 15 to 19 is executed.

23. A cloud-native distributed real-time rendering system, comprising a logic node service module, a computing node service module, a rendering node processing module, a synthesis node service module, and a real-time communication RTC or real-time system RTS service module; In response to a user input operation, the logical node service module is configured to generate a computing task for logical processing and publish the computing task to the computing node service module; The computing node service module is configured to execute computing tasks, process current frame computing tasks and process asynchronous computing tasks. The computing node service module is configured to return the current frame required computing results to the logic node service module after processing the current frame computing tasks. The logic node service module is configured to generate rendering tasks and publish rendering tasks to the rendering node processing module. The rendering node processing module is configured to process rendering tasks. The synthesis node service module is configured to process rendering and synthesis tasks and perform picture synthesis; the RTC or RTS service module is configured to perform streaming of the rendered picture, and the user receives the streaming picture.

24. The cloud-native distributed real-time rendering system according to claim 23, wherein: In the case of processing a rendering task that cannot be completed by a single node, the rendering task is split by the logical node service module and the split rendering task is assigned to multiple rendering node processing modules to split the calculation amount, so that the multiple rendering node processing modules split the calculation amount and jointly complete the split rendering task through collaborative rendering.

25. The cloud-native distributed real-time rendering system according to claim 23 or 24, wherein: The computing node service module is a stateless node.

Citation Information

Patent Citations

  • Cloud native distributed real-time rendering framework, rendering method and device

    CN116893878A

  • Cloud native 3D scene game method and system

    CN111951366A

  • Mobile Web augmented reality 3D model data service method based on cooperative computing

    CN112989251A

  • Distributed mobile WEB 3D rendering system and method

    CN115550335A

  • Variable distributed rendering system for editing video based on cloud and method thereof

    KR102081364B1

Cited By

  • Task processing method and system based on distributed voice gateway, and storage medium

    CN119718674A

  • Distributed big data real-time stream processing method and system

    CN120069501A

  • High-fidelity cloud rendering cluster scheduling method and system

    CN120371468A

  • Metacosmic digital twin electronic dart system and method based on wearing-free somatosensory motion capture technology

    CN120643897A

  • Rendering node video stream low-delay synthesis output system

    CN120935379A