A multi-engine adaptive rendering system and method for digital twinning

Abstract

This invention discloses a multi-engine adaptive rendering system and method for digital twins. The adaptive rendering system includes a component layer, an adaptation layer, and an engine layer. The adaptive rendering method includes: setting GPU monitoring indicators and setting engine switching conditions based on the monitoring indicators; preprocessing general 3D object data and converting the data format to the data format of the corresponding object in the UE engine or WebGL engine, and distributing the corresponding data to different engines; utilizing the pixel streaming function provided by UE and the designed data streaming scheme, streaming different types of data from the Web client to the UE server according to different schemes, then the UE engine performs real-time cloud rendering and streams the rendering results to the Web client for real-time display in the browser; and monitoring the GPU in real time and selecting the rendering engine based on the monitoring data. This solves the problems of existing rendering engines being single, resulting in poor rendering effects and GPU performance limitations.

Description

Technical Field

[0001] This invention belongs to the field of VR image generation technology, specifically relating to a multi-engine adaptive rendering system and method for digital twins. Background Technology

[0002] Digital twins were first applied in large-scale manufacturing to dynamically simulate physical objects. In recent years, with the development of new-generation information technologies such as big data, cloud computing, 5G, and artificial intelligence, the application scenarios of digital twin technology have continued to expand and deepen across various industries. It has been widely used in smart cities, industrial manufacturing, healthcare, agricultural technology, and many other fields, while also placing higher demands on rendering technology.

[0003] Rendering technology is crucial for the visual presentation and interactive experience of digital twins. In a digital twin system, the rendering engine is responsible for transforming complex 3D models, physical data, and interactive information into intuitive and realistic visual images, enabling users to observe and analyze the state and behavior of physical objects in real time through a virtual interface. To ensure that the digital twin model accurately recreates the scene, the rendering engine needs to have excellent rendering capabilities, support advanced graphics technologies such as ray tracing, shadow generation, and material detail processing; it also needs to support advanced interactive functions and overcome the limitations of local device performance, achieving good rendering results on any device.

[0004] There are three main existing digital twin rendering engine solutions, as follows:

[0005] 1) WebGL Engine: WebGL is a JavaScript API used to render high-performance 3D graphics in web browsers. However, the WebGL engine relies on the performance of the local device and its rendering capabilities are not excellent. It is only suitable for small-scale business scenarios. If the WebGL engine is used to render large-scale scenes, it cannot make the virtual model accurately reflect the state of its physical counterpart.

[0006] 2) UE Engine: The UE engine is a game engine with powerful real-time rendering, physical simulation, efficient data processing, and optimization technologies, capable of rendering high-quality digital twin scenes. Cloud rendering is a rendering technology provided by UE that distributes rendering tasks to cloud servers for processing and uses pixel streaming technology to provide real-time feedback of rendering results to the web client. UE engines often combine with cloud rendering technology to improve rendering efficiency, but current technologies only achieve one-way data streaming between the UE engine and the web client, failing to achieve bidirectional data streaming and command interaction between the UE engine and the web client.

[0007] 3) Multi-engine: This approach involves developing applications on both the WebGL and UE engines simultaneously, and deploying both rendering environments. Users can choose the appropriate engine version for rendering as needed. Existing multi-engine solutions require developing multiple applications, significantly increasing development workload. Furthermore, they cannot dynamically switch engines based on real-time GPU load, leading to server denial-of-service issues when the server GPU load is too high. Therefore, in engine rendering solutions for digital twins, improving rendering capabilities to meet diverse business needs while avoiding GPU performance limitations is a pressing issue. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, the present invention aims to provide a multi-engine adaptive rendering system and method for digital twins, addressing issues such as the reliance on a single rendering engine, unsatisfactory rendering results, and GPU performance limitations. To achieve this objective, the specific technical solution of the present invention is as follows:

[0009] On one hand, this invention provides a multi-engine adaptive rendering system for digital twins, including a component layer, an adaptation layer, and an engine layer.

[0010] The component layer is used to define and describe common functions, data, and effects. The rendering system manages the stored common data through the component layer.

[0011] The adaptation layer is used to preprocess the data. The code in the adaptation layer converts general 3D object data into the data format required for rendering by different engines, and then calls the rendering engine to be used.

[0012] The engine layer integrates the WebGL engine and the UE engine, selecting either the WebGL engine or the UE engine for rendering depending on the situation, and displaying the rendered results on the system. The WebGL engine renders the 3D scene in the browser via the JavaScript API, while the UE engine performs rendering on the cloud server and uses pixel streaming technology to display the remote rendering results in real time.

[0013] On the other hand, the present invention also provides a rendering method for a multi-engine adaptive rendering system for digital twins, comprising the following steps:

[0014] S1. Set GPU monitoring metrics and set engine switching conditions based on the GPU monitoring metrics; the GPU monitoring metrics include GPU 3D rendering rate, GPU utilization, GPU video encoding rate, and GPU video decoding rate;

[0015] S2. Using the designed 3D universal rendering pipeline, preprocess the universal 3D object data, convert the data format of the universal 3D object into the data format of the corresponding object in the UE engine or WebGL engine, and distribute the corresponding data to different engines.

[0016] S3. Utilize the pixel streaming function and data streaming scheme provided by UE to stream different types of data from the Web end to the UE server according to different schemes. Then, the UE engine performs real-time cloud rendering and streams the rendering results to the Web end to display the rendering results in the browser in real time.

[0017] S4. Monitor the GPU in real time. When the monitoring data meets the engine switching conditions, switch the UE engine to the WebGL engine and use the WebGL engine for rendering.

[0018] Furthermore, setting the engine switching conditions in step S1 includes:

[0019] Each GPU monitoring metric is assigned a different weight. The weights of GPU 3D rendering rate, GPU utilization, GPU video encoding rate, and GPU video decoding rate are w1, w2, w3, and w4, respectively. Then, the weighted sum W is calculated, and W > a ​​is used as the engine switching condition, where a represents the GPU load threshold.

[0020] Furthermore, the preprocessing of the general 3D object data in step S2 includes:

[0021] S21. Extract and abstract the general functions, data, and effects of each engine for rendering common 3D objects; the common 3D objects include models, cameras, lighting, and scenes;

[0022] S22. Preprocess general 3D object data, transform the general 3D object data to adapt to the data format of the corresponding object in different engines, and obtain different data required for rendering by different engines;

[0023] S23. Real-time rendering of objects is achieved according to different engines; the WebGL engine loads data locally through tools that simplify WebGL development and JavaScript API, and performs rendering on the Web client; the UE engine performs cloud rendering on the server and displays the rendering results in real time on the Web client through pixel streaming technology.

[0024] Furthermore, the data stream transmission scheme in step S3 specifically includes:

[0025] Web event feedback mechanism: Web events are streamed in the order of triggering. The UE engine responds in sequence, and then the Web client processes the response results, thus solving the problem of asynchronous events.

[0026] Data packet periodic sending mechanism: Non-frame loop control messages of 3D objects from the Web client are packaged into a certain periodic set and sent to the UE engine;

[0027] Set a message blocking lock: When the web client sends frame-looping messages, the lock is used to block these frequently sent messages, and the lock is released after the current frame loop is completed.

[0028] Command parsing mechanism: Commands sent from the Web client are divided into event commands, control commands, and frame loop commands. When the UE engine receives a command, it first parses the command and then executes the corresponding operation. Event commands trigger the UE's event response, control commands call the UE's 3D object API to implement control, and frame loop commands control the UE's frame loop state.

[0029] Furthermore, the specific process for real-time monitoring of the GPU in step S4 is as follows:

[0030] S41. Obtain GPU monitoring indicator data through the operating system interface, and respond to the GPU monitoring indicators according to the set engine switching conditions;

[0031] S42. If the GPU monitoring metrics do not meet the engine switching conditions, continue to use the UE engine for rendering;

[0032] S43. If the GPU monitoring metrics data meet the engine switching conditions, then stop using the UE engine and use the WebGL engine for rendering.

[0033] Furthermore, the specific operations for rendering using the WebGL engine are as follows: obtain data that is adapted to the object data format of the WebGL engine, load the data into the WebGL engine, render it using the user's local computer resources, and finally display the real-time rendering result on the Web client.

[0034] Compared with existing technologies, this invention has the following advantages: The multi-engine adaptive rendering system and method proposed in this invention, by integrating multiple rendering engines and tools, can not only flexibly meet the diverse needs of different projects in terms of performance, effects, and compatibility, but also monitor the GPU in real time and switch engines according to the GPU's real-time load, avoiding GPU performance limitations. This architecture not only improves the overall expressiveness and interactivity of digital twin projects, but also enhances the scalability and maintainability of the projects; this method plays an important role in the field of digital twins, providing more solid technical support and guarantee for the digital transformation of various industries. Attached Figure Description

[0035] Figure 1 This invention provides a block diagram of a multi-engine adaptive rendering system for digital twins;

[0036] Figure 2 A flowchart of a multi-engine adaptive rendering method for digital twins provided by the present invention;

[0037] Figure 3 This is a schematic diagram illustrating the real-time rendering stream between the Web client and the UE server of this invention;

[0038] Figure 4 This is a schematic diagram illustrating the engine switching of the present invention. Detailed Implementation

[0039] The following description, in conjunction with the accompanying drawings, further illustrates a multi-engine adaptive rendering system and method for digital twins according to the present invention.

[0040] Example 1

[0041] like Figure 1 As shown, a multi-engine adaptive rendering system for digital twins is provided. The multi-engine adaptive rendering method for digital twins proposed in Embodiment 1 can be implemented based on this system. The multi-engine adaptive rendering system for digital twins includes: a component layer, an adaptation layer, and an engine layer.

[0042] The component layer defines and describes the general functions, data, and effects of common objects such as models, cameras, lighting, and scenes. The system manages the stored general data through the component layer.

[0043] The adaptation layer preprocesses the data, transforming general 3D object data into the data format required for rendering by different engines through the adaptation layer code, and then calling the rendering engine to be used.

[0044] The engine layer integrates the WebGL engine and the UE engine, and selects to use either the WebGL engine or the UE engine for rendering depending on the situation, and then displays the rendered results on the system; WebGL renders 3D scenes in the browser through the JavaScript API; the UE engine renders on the cloud server and uses pixel streaming technology to display the results of remote rendering in real time.

[0045] Example 2

[0046] like Figure 2 As shown, a multi-engine adaptive rendering method for digital twins includes the following steps:

[0047] (1) Set GPU monitoring metrics and set engine switching conditions based on the GPU monitoring metrics.

[0048] The GPU monitoring metrics include GPU 3D rendering rate, GPU utilization, GPU video encoding rate, and GPU video decoding rate. Each GPU monitoring metric is assigned a different weight. The weights for GPU 3D rendering rate, GPU utilization, GPU video encoding rate, and GPU video decoding rate are w1, w2, w3, and w4, respectively. Then, a weighted sum W is calculated, and W > a ​​is used as the engine switching condition, where a represents the GPU load threshold.

[0049] In this embodiment, the weights of GPU3D rendering rate, GPU utilization, GPU video encoding rate, and GPU video decoding rate are set to 0.3, 0.2, 0.4, and 0.1, respectively. Then, the weighted sum is calculated, and the event that the weighted sum is greater than 0.9 is used as the engine switching condition.

[0050] (2) Using the designed 3D general rendering pipeline, preprocess the general 3D object data, convert the data format of the general 3D object into the data format of the corresponding object in the UE engine or WebGL engine, and distribute the corresponding data to different engines.

[0051] The specific process for preprocessing general 3D object data is as follows:

[0052] ① Extract and abstract the general functions, data, and effects of rendering common 3D objects from various engines; rendering common 3D objects includes models, cameras, lighting, and scenes.

[0053] ② Preprocess general 3D object data, convert the general 3D object data to adapt to the data format of corresponding objects in different engines, and obtain different data required for rendering by different engines.

[0054] ③ Real-time rendering of objects is achieved according to different engines; the WebGL engine uses the Three.js framework, which simplifies WebGL development by using tools and APIs to load data locally and render it on the Web; the UE engine uses UE5 to achieve cloud rendering on the server and displays the rendering results in real time on the Web through pixel streaming technology.

[0055] In this embodiment, general 3D object data is preprocessed, converted into a data format adapted to the corresponding object in the UE engine, and then distributed to the UE engine.

[0056] (3) Using the pixel streaming function and data streaming scheme provided by UE, different types of data are streamed from the Web end to the UE server according to different schemes. Then the UE engine performs real-time cloud rendering and streams the rendering results to the Web end, and displays the rendering results in the browser in real time.

[0057] Data streaming strategies include Web event feedback mechanisms, periodic data packet sending mechanisms, message blocking mechanisms, and command parsing mechanisms.

[0058] Web event feedback mechanism: Web events are streamed in the order they are triggered. The UE engine responds in sequence, and then the Web client processes the response results, thus solving the problem of asynchronous events.

[0059] Periodic packet sending mechanism: Non-frame loop control messages for 3D objects from the Web client are packaged into sets according to a certain period and sent to the UE engine to improve message sending efficiency. The periodic packet sending mechanism uses MT ≤ RT to determine whether to send Web client messages, where MT represents the message sending period and RT represents the cumulative time of currently collected messages. The default value of RT is 0, and it is accumulated according to the time. Once MT is reached, a batch of packaged messages is sent, and RT is reset to zero after each message is sent.

[0060] Setting a message masking lock: When the web client sends frame-looping messages, the lock is used to block these frequently sent messages. The lock is then released after the current frame loop finishes executing. The message masking lock has two states: Lock and Unlock. When the web client sends frame-looping messages, the message masking lock is set to the Lock state. After the current frame loop finishes executing, the message masking lock is set to the Unlock state.

[0061] In a preferred embodiment, the web-based message is based on the JSON structure {"ID", "HT", "HP"}, where ID represents the unique ID of the 3D object; HT represents the operation type for each 3D object, including create, set, event, animate, etc.; HP represents the operation parameters required for the operation, where the parameters for create are the initial construction parameters, the parameters for set are the attribute name and attribute value, the parameters for event are the event type and response operation, and the parameters for animate are the animation type, control attribute and value.

[0062] Command parsing mechanism: Commands sent from the Web client are divided into event commands, control commands, and frame loop commands. When the UE engine receives a command, it first parses the command and then executes the corresponding operation. Event commands trigger the UE's event response, control commands call the UE's 3D object API to implement control, and frame loop commands control the UE's frame loop state.

[0063] The command parsing mechanism categorizes message types into Event, Control, and Animation, and sets up an operation type lookup table and a UE 3D object lookup table. The operation type lookup table is used to look up the message type corresponding to different HTs, and the UE 3D object lookup table is used to look up the UO corresponding to the 3D object ID, where UO represents the UE's 3D object. When the UE receives a message from the Web client, it looks up the corresponding message type in the operation type lookup table based on the message's HT, and simultaneously looks up the corresponding UO in the UE 3D object lookup table based on the message's ID. Then, it performs corresponding message processing on the UO based on the message type, HT, and HP.

[0064] Specifically, message processing includes: Event message processing: The UE executes the event command and sends an event response message to the Web client, which then performs the response operation upon receiving it; Control message processing: The UE performs the control operations corresponding to the HT and HP of the message on the UO; Animation message processing: The UE hands over the control of the UO animation to the animation system.

[0065] like Figure 3 The diagram shown illustrates the real-time rendering stream between the web client and the UE server. Figure 3 In this process, the web client sends data streams to the UE server, where the UE engine performs cloud rendering. Using the pixel streaming technology provided by UE, the Pixel Streaming Plugin is first run in UE to establish connections with the signaling and web servers. Then, a connection is established between the UE engine and the system client. Finally, UE streams the rendering results to the client in real time, displaying the real-time rendering results in the browser.

[0066] (4) Monitor the GPU in real time. When the monitoring index data meets the engine switching conditions, switch the UE engine to the WebGL engine and use the WebGL engine for rendering.

[0067] The specific process for real-time monitoring of the GPU is as follows:

[0068] ① Obtain GPU monitoring metrics data through the operating system interface, and adjust the GPU monitoring metrics according to the set engine switching conditions, such as... Figure 4 The response shown.

[0069] ②If the GPU monitoring metrics do not meet the engine switching conditions, continue to use the UE engine for rendering.

[0070] ③ If the GPU monitoring metrics meet the engine switching conditions, stop using the UE engine and use the WebGL engine for rendering. Rendering using the WebGL engine includes: obtaining data adapted to the object data format corresponding to the WebGL engine, loading the data into the WebGL engine, rendering using the user's local computer resources, and finally displaying the real-time rendering results on the web interface.

[0071] In this embodiment, the GPU 3D rendering rate was monitored to be 0.91, the GPU utilization rate was 0.92, the GPU video encoding rate was 0.89, and the GPU video decoding rate was 0.88. The weighted sum was calculated to be 0.901, which met the engine switching conditions. Therefore, the UE engine was stopped and the WebGL engine was switched to for rendering.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-engine adaptive rendering method for digital twins, characterized in that, The rendering system that executes the rendering method includes a component layer, an adaptation layer, and an engine layer. The component layer is used to define and describe common functions, data, and effects, and the rendering system manages the stored common data through the component layer. The adaptation layer is used to preprocess the data, converting common 3D object data into data formats required by different rendering engines through the code in the adaptation layer, and then calling the rendering engine to be used. The engine layer is used to integrate the WebGL engine and the UE engine, selecting the WebGL engine or the UE engine for rendering according to different situations, and displaying the rendered results on the system. The WebGL engine renders 3D scenes in the browser via a JavaScript API; The UE engine performs rendering on a cloud server and uses pixel streaming technology to display the results of remote rendering in real time. The rendering method includes the following steps: S1. Set GPU monitoring metrics and set engine switching conditions based on the GPU monitoring metrics; the GPU monitoring metrics include GPU 3D rendering rate, GPU utilization, GPU video encoding rate, and GPU video decoding rate; S2. Using the designed 3D universal rendering pipeline, preprocess the universal 3D object data, convert the data format of the universal 3D object into the data format of the corresponding object in the UE engine or WebGL engine, and distribute the corresponding data to different engines. S3. Utilize the pixel streaming function and data streaming scheme provided by UE to stream different types of data from the Web end to the UE server according to different schemes. Then, the UE engine performs real-time cloud rendering and streams the rendering results to the Web end to display the rendering results in the browser in real time. S4. Monitor the GPU in real time. When the monitoring data meets the engine switching conditions, switch the UE engine to the WebGL engine and use the WebGL engine for rendering.

2. The multi-engine adaptive rendering method for digital twins according to claim 1, characterized in that, The engine switching conditions set in step S1 include: Different weights are assigned to each GPU monitoring metric, with the following weights for GPU 3D rendering rate, GPU utilization, GPU video encoding rate, and GPU video decoding rate: , , , Then calculate the weighted sum. ,Will As a condition for engine switching This indicates the GPU load threshold.

3. The multi-engine adaptive rendering method for digital twins according to claim 1, characterized in that, The preprocessing of the general 3D object data in step S2 includes: S21. Extract and abstract the general functions, data, and effects of each engine for rendering common 3D objects; the common 3D objects include models, cameras, lighting, and scenes; S22. Preprocess general 3D object data, transform the general 3D object data to adapt to the data format of the corresponding object in different engines, and obtain different data required for rendering by different engines; S23. Real-time rendering of objects is achieved according to different engines; the WebGL engine loads data locally through tools that simplify WebGL development and JavaScript API, and performs rendering on the Web client; the UE engine performs cloud rendering on the server and displays the rendering results in real time on the Web client through pixel streaming technology.

4. The multi-engine adaptive rendering method for digital twins according to claim 1, characterized in that, The data transmission scheme in step S3 specifically includes: Web event feedback mechanism: Web events are streamed in the order of triggering. The UE engine responds in sequence, and then the Web client processes the response results, thus solving the problem of asynchronous events. Data packet periodic sending mechanism: Non-frame loop control messages of 3D objects from the Web client are packaged into a certain periodic set and sent to the UE engine; Set a message blocking lock: When the web client sends frame-looping messages, the lock is used to block these frequently sent messages, and the lock is released after the current frame loop is completed. Command parsing mechanism: Commands sent from the Web client are divided into event commands, control commands, and frame loop commands. When the UE engine receives a command, it first parses the command and then executes the corresponding operation. Event commands trigger the UE's event response, control commands call the UE's 3D object API to implement control, and frame loop commands control the UE's frame loop state.

5. The multi-engine adaptive rendering method for digital twins according to claim 1, characterized in that, The specific process for real-time monitoring of the GPU in step S4 is as follows: S41. Obtain GPU monitoring indicator data through the operating system interface, and respond to the GPU monitoring indicators according to the set engine switching conditions; S42. If the GPU monitoring metrics do not meet the engine switching conditions, continue to use the UE engine for rendering; S43. If the GPU monitoring metrics data meet the engine switching conditions, then stop using the UE engine and use the WebGL engine for rendering.

6. The multi-engine adaptive rendering method for digital twins according to claim 5, characterized in that, The specific steps for rendering using the WebGL engine are as follows: obtain data that is adapted to the object data format of the WebGL engine, load the data into the WebGL engine, render it using the user's local computer resources, and finally display the real-time rendering result on the web.

Images