Digital twinning-oriented multi-engine adaptive rendering system and method

Abstract

The invention discloses a digital twinning-oriented multi-engine adaptive rendering system and a digital twinning-oriented multi-engine adaptive rendering method. The adaptive rendering system comprises a component layer, an adaptation layer and an engine layer, the adaptive rendering method comprises the following steps: setting a GPU monitoring index and setting an engine switching condition according to the monitoring index; preprocessing the universal 3D object data, converting a data format into a data format of a corresponding object in a UE engine or a WebGL engine, and distributing the corresponding data to different engines; the method comprises the following steps: streaming different types of data from a Web side to a UE server side according to different schemes by utilizing a pixel streaming function provided by UE and a designed data streaming scheme, then carrying out real-time cloud rendering by a UE engine, streaming a rendering result to the Web side, and displaying the rendering result in real time on a browser; and monitoring the GPU in real time, and selecting a rendering engine according to monitoring data. The problems that an existing rendering engine is single, the rendering effect is not excellent, and GPU performance is limited are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of VR image generation, and specifically relates to a multi-engine adaptive rendering system and method for digital twins. Background Art

[0002] Digital twins were first used in large-scale manufacturing industries for dynamic simulation of physical objects. In recent years, with the development of new-generation information technologies such as big data, cloud computing, 5G, and artificial intelligence, the application scenarios of digital twin technology in various industries have continued to expand and deepen. It has been widely used in smart cities, industrial manufacturing, medical health, agricultural technology and other fields. At the same time, it has also put forward higher requirements for rendering technology.

[0003] Rendering technology is the key to achieving visual presentation and interactive experience for digital twins. In the digital twin system, the rendering engine is responsible for converting complex three-dimensional models, physical data, and interactive information into intuitive and realistic visual images, allowing users to observe and analyze the status and behavior of physical objects in real time through a virtual interface. In order to make the digital twin model restore the scene as realistically as possible, the rendering engine needs to have good rendering effects, support advanced graphics technology, and realize details such as ray tracing, shadow generation, and material detail processing; it also needs to support advanced interactive functions and be able to break through the limitations of local device performance to achieve good rendering effects on any device.

[0004] There are three main existing digital twin rendering engine solutions, as follows:

[0005] 1) WebGL engine: WebGL is a JavaScript API used to render high-performance 3D graphics in web browsers. However, the WebGL engine relies on local device performance and its rendering capabilities are not good enough. It is only suitable for small-scale business scenarios. If the WebGL engine is used to render large-scale scenes, the virtual model cannot accurately reflect the state of its physical counterpart.

[0006] 2) UE Engine: UE Engine is a game engine that has powerful real-time rendering, physical simulation, efficient data processing and optimization technologies, and can render high-quality digital twin scenes. Cloud rendering is a rendering technology provided by UE. It assigns rendering tasks to cloud servers for processing, and uses pixel streaming technology to feed back rendering results to the Web side in real time. UE engines usually combine cloud rendering technology to improve rendering efficiency, but the existing technology only realizes one-way data streaming from the UE engine to the Web side, and fails to achieve two-way data streaming and command interaction between the UE engine and the Web side.

[0007] 3) Multi-engine: Develop applications on both the WebGL engine and the UE engine, and deploy two engine rendering environments, the WebGL engine and the UE engine. Users can choose applications with different engine versions for rendering as needed. The existing multi-engine solution requires the development of multiple sets of applications, which greatly increases the development workload. In addition, it is impossible to dynamically switch engines according to the real-time GPU load. When the server GPU load is too high, there is a problem of server denial of service. Therefore, in the engine rendering solution for digital twins, how to improve rendering capabilities, meet different business needs, and avoid GPU performance limitations is an urgent problem to be solved. Summary of the invention

[0008] In order to make up for the shortcomings of the prior art, the purpose of the present invention is to provide a multi-engine adaptive rendering system and method for digital twins to solve the problems of the existing single rendering engine, poor rendering effect, GPU performance limitation, etc. To achieve this purpose, the specific technical solution of the present invention is as follows:

[0009] On the one hand, the present invention provides a multi-engine adaptive rendering system for digital twins, including a component layer, an adaptation layer, and an engine layer.

[0010] The component layer is used to define and describe common functions, data, and effects, and the rendering system manages the stored common data through the component layer;

[0011] The adaptation layer is used to pre-process the data, convert the general 3D object data into the data format required by different rendering engines through the adaptation layer code, and then call the rendering engine to be used downward;

[0012] The engine layer is used to integrate the WebGL engine and the UE engine, select the WebGL engine or the UE engine for rendering according to different situations, and display the rendered results on the system; the WebGL engine renders the three-dimensional scene on the browser through the JavaScript API; the UE engine renders on the cloud server and uses pixel streaming technology to display the remote rendering results in real time.

[0013] On the other hand, the present invention also provides a rendering method for a multi-engine adaptive rendering system for digital twins, comprising the following steps:

[0014] S1. Set GPU monitoring indicators and set engine switching conditions according to GPU monitoring indicators; the GPU monitoring indicators include GPU 3D rendering rate, GPU utilization rate, GPU video encoding rate, and GPU video decoding rate;

[0015] S2. Use the designed 3D universal rendering pipeline to pre-process the universal 3D object data, convert the data format of the universal 3D object into the data format of the corresponding object in the UE engine or the WebGL engine, and distribute the corresponding data to different engines;

[0016] S3. Using the pixel streaming function provided by UE and the designed data streaming scheme, different types of data are streamed from the Web end to the UE server end according to different schemes. Then the UE engine performs real-time cloud rendering and streams the rendering results to the Web end, and displays the rendering results in real time on the browser;

[0017] S4. Monitor the GPU in real time. When the monitoring indicator data meets the engine switching condition, switch the UE engine to the WebGL engine and use the WebGL engine for rendering.

[0018] Furthermore, setting the engine switching condition in step S1 includes:

[0019] Different weights are assigned to each GPU monitoring indicator, among which the weights of GPU 3D rendering rate, GPU utilization rate, GPU video encoding rate, and GPU video decoding rate are w1, w2, w3, and w4 respectively. Then the weighted sum W is calculated, and W>a is used as the engine switching condition, where a represents the GPU load threshold.

[0020] Furthermore, the preprocessing of the general 3D object data in step S2 includes:

[0021] S21. Extract and abstract the definitions of general functions, data and effects of rendering general 3D objects by various engines; the general 3D objects include models, cameras, lighting and scenes;

[0022] S22. Preprocessing the general 3D object data, converting the general 3D object data to adapt to the data format of the corresponding object in different engines, and obtaining different data required for rendering by different engines;

[0023] S23. Real-time rendering of objects is achieved according to different engines; the WebGL engine loads data locally and achieves rendering on the Web side by simplifying the tools and JavaScript APIs developed by WebGL; the UE engine achieves cloud rendering in the server and displays the rendering results in real time on the Web side through pixel streaming technology.

[0024] Furthermore, the data streaming scheme in step S3 specifically includes:

[0025] Web event feedback mechanism: Web events are streamed in the order of triggering, and the UE engine responds in order and then the Web end processes the response results to solve the event asynchrony problem;

[0026] Data packet periodic sending mechanism: the non-frame cycle control message of the Web end to the 3D object is packaged according to a certain periodic set and sent to the UE engine;

[0027] Set message shielding lock: When the Web client sends frame cycle messages, this type of frequently sent messages can be shielded by locking it, and the lock will be opened when the frame cycle is finished.

[0028] Command parsing mechanism: The commands sent by the Web end are divided into event commands, control commands, and frame loop commands. When the UE engine receives the command, it first parses the command and then executes the corresponding operation. Among them, the event command triggers the UE's event response, the control command calls the UE's 3D object API to achieve control, and the frame loop command controls the state of the UE frame loop.

[0029] Furthermore, the specific process of real-time monitoring of the GPU in step S4 is as follows:

[0030] S41. Obtain GPU monitoring index data through the operating system interface, and respond to the GPU monitoring index according to the set engine switching conditions;

[0031] S42. If the GPU monitoring indicator data does not meet the engine switching conditions, continue to use the UE engine for rendering;

[0032] S43. If the GPU monitoring indicator data meets the engine switching condition, stop using the UE engine and use the WebGL engine for rendering.

[0033] Furthermore, the specific operation of using the WebGL engine for rendering is: obtaining data that is adapted to the object data format corresponding to the WebGL engine, then loading the data into the WebGL engine, using the user's local computer resources for rendering, and finally displaying the real-time rendering results on the Web side.

[0034] Compared with the prior art, the present invention has the following advantages: the multi-engine adaptive rendering system and method proposed in the present invention, by integrating multiple rendering engines and tools, can not only flexibly respond to the diverse requirements of different projects in terms of performance, effect, compatibility, etc., but also monitor the GPU in real time, and switch engines according to the real-time load of the GPU to avoid the limitation of GPU performance. This architecture not only improves the overall expressiveness and interactivity of the digital twin project, but also enhances the scalability and maintainability of the project; this method plays an important role in the field of digital twins and can provide more solid technical support and guarantee for the digital transformation of all walks of life. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A block diagram of a multi-engine adaptive rendering system for digital twins provided by the present invention;

[0036] Figure 2 A flow chart of a multi-engine adaptive rendering method for digital twins provided by the present invention;

[0037] Figure 3 A schematic diagram of real-time rendering and streaming of a Web terminal and a UE server terminal of the present invention;

[0038] Figure 4 It is a schematic diagram of engine switching of the present invention. DETAILED DESCRIPTION

[0039] The following is a further description of a multi-engine adaptive rendering system and method for digital twins of the present invention in conjunction with the accompanying drawings.

[0040] Example 1

[0041] like Figure 1 As shown, a multi-engine adaptive rendering system for digital twins. A multi-engine adaptive rendering method for digital twins proposed in Example 1 can be implemented based on this system. The multi-engine adaptive rendering system for digital twins includes: a component layer, an adaptation layer, and an engine layer.

[0042] The component layer defines and describes general functions, data, and effects for common objects such as models, cameras, lighting, and scenes. The system manages the stored general data through the component layer.

[0043] The adaptation layer preprocesses the data, converts the general 3D object data into the data format required by different rendering engines through the adaptation layer code, and then calls down the rendering engine to be used.

[0044] The engine layer integrates the WebGL engine and the UE engine, and chooses to use the WebGL engine or the UE engine for rendering according to different situations, and then displays the rendered results on the system; WebGL renders three-dimensional scenes in the browser through the JavaScript API; the UE engine renders on the cloud server and uses pixel streaming technology to display the results of remote rendering in real time.

[0045] Example 2

[0046] like Figure 2 As shown, a multi-engine adaptive rendering method for digital twins includes the following specific steps:

[0047] (1) Setting GPU monitoring indicators, and setting engine switching conditions according to the GPU monitoring indicators.

[0048] Among them, GPU monitoring indicators include GPU 3D rendering rate, GPU utilization rate, GPU video encoding rate, and GPU video decoding rate. Each GPU monitoring indicator is assigned a different weight, where the weights of GPU 3D rendering rate, GPU utilization rate, GPU video encoding rate, and GPU video decoding rate are w1, w2, w3, and w4 respectively. Then the weighted sum W is calculated, and W>a is used as the engine switching condition, where a represents the GPU load threshold.

[0049] In this embodiment, the weights of GPU 3D rendering rate, GPU utilization rate, GPU video encoding rate, and GPU video decoding rate are set to 0.3, 0.2, 0.4, and 0.1 respectively, and then the weighted sum is calculated, and the event that the weighted sum is greater than 0.9 is used as the engine switching condition.

[0050] (2) Using the designed 3D universal rendering pipeline, the universal 3D object data is preprocessed, the data format of the universal 3D object is converted into the data format of the corresponding object in the UE engine or the WebGL engine, and the corresponding data is distributed to different engines.

[0051] The specific process of preprocessing general 3D object data is as follows:

[0052] ① Extract and abstract the definitions of general functions, data, and effects of each engine for rendering general 3D objects; rendering general 3D objects includes models, cameras, lighting, and scenes.

[0053] ② Preprocess the general 3D object data and convert the general 3D object data to adapt to the data format of the corresponding objects in different engines, so as to obtain the different data required for rendering by different engines.

[0054] ③ Realize real-time rendering of objects according to different engines; the WebGL engine uses the Three.js framework, simplifies the tools and APIs developed by WebGL, loads data locally, and implements rendering on the Web side; the UE engine uses UE5 to implement cloud rendering on the server, and displays the rendering results in real time on the Web side through pixel streaming technology.

[0055] In this embodiment, the general 3D object data is pre-processed, converted into a data format adapted to the corresponding object of the UE engine, and distributed to the UE engine.

[0056] (3) Utilizing the pixel streaming function provided by the UE and the designed data streaming scheme, different types of data are streamed from the Web end to the UE server end according to different schemes. The UE engine then performs real-time cloud rendering and streams the rendering results to the Web end, where they are displayed in real time on the browser.

[0057] The data streaming strategy includes Web event feedback mechanism, data packet periodic sending mechanism, message shielding mechanism and command parsing mechanism.

[0058] Web event feedback mechanism: Web events are streamed in the order of triggering. The UE engine responds in sequence and the Web side processes the response results to solve the event asynchrony problem.

[0059] Data packet periodic sending mechanism: The non-frame cyclic control messages of the Web side for 3D objects are packaged according to a certain periodic set and sent to the UE engine to improve the efficiency of message sending. The data packet periodic sending mechanism uses MT≤RT to determine whether to send the Web side message, where MT represents the message sending period, and RT represents the cumulative time of the currently collected messages. The default value of RT is 0, and according to the time accumulation, a packaged batch message is sent once when MT is reached, and RT is set to zero after each message is sent.

[0060] Set the message shielding lock: When the Web client sends a frame cycle message, this type of frequently sent message is shielded by locking it, and the lock is opened after the execution of this frame cycle is completed. The message shielding lock includes two states: Lock and UnLock. When the Web client sends a frame cycle message, the message shielding lock is set to the Lock state, and when the execution of this frame cycle is completed, the message shielding lock is set to the UnLock state.

[0061] In a preferred embodiment, the Web-side message is based on the JSON structure {"ID", "HT", "HP"}, where ID represents the unique ID of the 3D object; HT represents the operation type for each 3D object, including create, set, event, animate, etc.; HP represents the operation parameters required for the operation, the parameters of the create type are the initial construction parameters, the parameters of the set type are the attribute name and attribute value, the parameters of the event type are the event type and response operation, and the parameters of the animate type are the animation type, control attributes and values.

[0062] Command parsing mechanism: The commands sent by the Web end are divided into event commands, control commands, and frame loop commands. When the UE engine receives the command, it first parses the command and then executes the corresponding operation. Among them, the event command triggers the UE's event response, the control command calls the UE's 3D object API to achieve control, and the frame loop command controls the state of the UE frame loop.

[0063] The command parsing mechanism divides message types into Event, Control, and Animation, and sets an operation type query table and a UE 3D object query table. The operation type query table is used to query the message types corresponding to different HTs, and the UE 3D object query table is used to query the UO corresponding to the 3D object ID, where UO represents the 3D object of the UE. When the UE receives a message from the Web side, it queries the corresponding message type in the operation type query table according to the HT of the message, and queries the corresponding UO in the UE 3D object query table according to the ID of the message, and then performs corresponding message processing on the UO according to the message type, HT, and HP.

[0064] Specifically, message processing includes: Event message processing: the UE executes the event command and sends an event response message to the Web, which performs a response operation after receiving it; Control message processing: the UE performs the control operations corresponding to the HT and HP of the message on the UO; Animation message processing: the UE hands over the control to the animation system to control the UO animation.

[0065] like Figure 3 The figure shows a schematic diagram of real-time rendering streaming on the Web side and the UE server side. Figure 3 In the process, the Web end sends the data stream to the UE server, and performs cloud rendering operations through the UE engine. Using the pixel streaming technology provided by the UE, first run the Pixel Streaming Plugin in the UE to establish a connection with the signaling and Web server, then establish a connection between the UE engine and the system client, and finally the UE sends the rendering results to the client in real time, and displays the real-time rendering results in the browser.

[0066] (4) Monitor the GPU in real time. When the monitoring indicator data meets the engine switching condition, switch the UE engine to the WebGL engine and use the WebGL engine for rendering.

[0067] The specific process of real-time monitoring of GPU is as follows:

[0068] ① Obtain GPU monitoring indicator data through the operating system interface, and perform the following operations on the GPU monitoring indicators according to the set engine switching conditions: Figure 4 Response shown.

[0069] ② If the GPU monitoring indicator data does not meet the engine switching conditions, continue to use the UE engine for rendering.

[0070] ③ If the GPU monitoring indicator data meets the engine switching conditions, stop using the UE engine and use the WebGL engine for rendering. Using the WebGL engine for rendering includes: obtaining data that is adapted to the object data format corresponding to the WebGL engine, then loading the data into the WebGL engine, using the user's local computer resources for rendering, and finally displaying the real-time rendering results on the Web end.

[0071] In this embodiment, the GPU 3D rendering rate is monitored to be 0.91, the GPU utilization rate is 0.92, the GPU video encoding rate is 0.89, and the GPU video decoding rate is 0.88. The weighted sum is calculated to be 0.901, which meets the engine switching condition. The UE engine is stopped and switched to the WebGL engine, and the WebGL engine is used for rendering.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-engine adaptive rendering system for digital twins, characterized in that: Including component layer, adaptation layer, and engine layer, The component layer is used to define and describe common functions, data, and effects, and the rendering system manages the stored common data through the component layer; The adaptation layer is used to pre-process the data, convert the general 3D object data into the data format required by different rendering engines through the adaptation layer code, and then call the rendering engine to be used downward; The engine layer is used to integrate the WebGL engine and the UE engine, select the WebGL engine or the UE engine for rendering according to different situations, and display the rendered results on the system; The WebGL engine renders the three-dimensional scene in the browser through the JavaScript API; the UE engine renders on the cloud server and uses pixel streaming technology to display the remote rendering results in real time.

2. The rendering method of a multi-engine adaptive rendering system for digital twins according to claim 1, characterized in that: The following steps are involved: S1. Set GPU monitoring indicators and set engine switching conditions according to GPU monitoring indicators; the GPU monitoring indicators include GPU 3D rendering rate, GPU utilization rate, GPU video encoding rate, and GPU video decoding rate; S2. Use the designed 3D universal rendering pipeline to pre-process the universal 3D object data, convert the data format of the universal 3D object into the data format of the corresponding object in the UE engine or the WebGL engine, and distribute the corresponding data to different engines; S3. Using the pixel streaming function provided by UE and the designed data streaming scheme, different types of data are streamed from the Web end to the UE server end according to different schemes. Then the UE engine performs real-time cloud rendering and streams the rendering results to the Web end, and displays the rendering results in real time on the browser; S4. Monitor the GPU in real time. When the monitoring indicator data meets the engine switching condition, switch the UE engine to the WebGL engine and use the WebGL engine for rendering.

3. The rendering method according to claim 2, characterized in that: Setting the engine switching condition in step S1 includes: Different weights are assigned to each GPU monitoring indicator, among which the weights of GPU 3D rendering rate, GPU utilization rate, GPU video encoding rate, and GPU video decoding rate are w1, w2, w3, and w4 respectively. Then the weighted sum W is calculated, and W>a is used as the engine switching condition, where a represents the GPU load threshold.

4. The rendering method according to claim 2, characterized in that: The preprocessing of the general 3D object data in step S2 includes: S21. Extract and abstract the definitions of general functions, data and effects of rendering general 3D objects by various engines; the general 3D objects include models, cameras, lighting and scenes; S22. Preprocessing the general 3D object data, converting the general 3D object data to adapt to the data format of the corresponding object in different engines, and obtaining different data required for rendering by different engines; S23. Real-time rendering of objects is achieved according to different engines; the WebGL engine loads data locally and achieves rendering on the Web side by simplifying the tools and JavaScript APIs developed by WebGL; the UE engine achieves cloud rendering in the server and displays the rendering results in real time on the Web side through pixel streaming technology.

5. The rendering method according to claim 2, characterized in that: The data streaming scheme in step S3 specifically includes: Web event feedback mechanism: Web events are streamed in the order of triggering, and the UE engine responds in order and then the Web end processes the response results to solve the event asynchrony problem; Data packet periodic sending mechanism: the non-frame cycle control message of the Web end to the 3D object is packaged according to a certain periodic set and sent to the UE engine; Set message shielding lock: When the Web client sends frame cycle messages, this type of frequently sent messages can be shielded by locking it, and the lock will be opened when the frame cycle is finished. Command parsing mechanism: The commands sent by the Web end are divided into event commands, control commands, and frame loop commands. When the UE engine receives the command, it first parses the command and then executes the corresponding operation. Among them, the event command triggers the UE's event response, the control command calls the UE's 3D object API to achieve control, and the frame loop command controls the state of the UE frame loop.

6. The rendering method according to claim 2, characterized in that: The specific process of real-time monitoring of the GPU in step S4 is as follows: S41. Obtain GPU monitoring index data through the operating system interface, and respond to the GPU monitoring index according to the set engine switching conditions; S42. If the GPU monitoring indicator data does not meet the engine switching conditions, continue to use the UE engine for rendering; S43. If the GPU monitoring indicator data meets the engine switching condition, stop using the UE engine and use the WebGL engine for rendering.

7. The rendering method according to claim 6, characterized in that: The specific operation of using the WebGL engine for rendering is: obtaining data that is adapted to the object data format corresponding to the WebGL engine, then loading the data into the WebGL engine, using the user's local computer resources for rendering, and finally displaying the real-time rendering results on the Web side.

Images