A Digital Twin Method Based on Real-Time Rendering

Abstract

The present invention relates to the technical field of three-dimensional scene display, and particularly to a digital twin method based on real-time rendering, including: the control center builds a target scene according to the model file and the requirements for building the target scene, and synchronizes it to the rendering device; when the control center receives a rendering request, it schedules any rendering device to connect to the user terminal; the rendering device obtains the to-be-rendered picture according to the received picture request, and obtains the 2D image corresponding to the to-be-rendered picture based on the model elements of the 3D model, and sends it to the user terminal; during the rendering device provides services for the user terminal, the control center detects the user picture frame rate of the rendering device in real time, and judges whether the device state of the rendering device is healthy; when it is unhealthy, the control center switches another rendering device to connect to the user terminal. The beneficial effect is that it can automatically process coordinate conversion based on the model file, and while building the target scene, ensure the stable output of the picture during the real-time rendering process.

Claims

1. A digital twin method based on real-time rendering, characterized in that: The method is implemented based on a pre-set control center and at least two rendering devices connected to the control center, and the method includes: S11, the control center builds the target scene according to the preset model file, the real geographical location information of the target scene and the virtual coordinate information of the scene reference point, and sends the target scene and the model elements corresponding to each 3D model in the target scene to all rendering devices; the model file includes at least one 3D model and the model elements corresponding to each 3D model; S12, when the control center receives a rendering request sent by the user terminal, based on a preset exclusive scheduling algorithm, schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal; S13, after any rendering device is connected to the user end, it receives the screen request of the user end in real time, and obtains the corresponding to-be-rendered screen according to the screen request, passes the to-be-rendered screen to the octree container object, obtains the transparent rendering object and the opaque rendering object, and stores the transparent rendering object and the opaque rendering object in the rendering queue; based on the model elements corresponding to each 3D model in the target scene, obtains the MVP matrix corresponding to each rendering object in the rendering queue; based on the model elements corresponding to each 3D model in the target scene, determines the specific rendering parameters, material name, material state value, texture map and macro definition value of the rendering object, and uploads the specific rendering parameters of the rendering object to the GPU at one time; determines whether the rendering pipeline has been created according to the Cache, if there is no Cache, creates the rendering pipeline, calls the rendering pipeline to perform real-time rendering of the to-be-rendered screen, obtains the 2D image corresponding to the to-be-rendered screen, and sends it to the user end; Creating a rendering pipeline involves: Generate Shader Module CacheKey according to the name, macro variables and specific Shader code of the Shader code, and check whether there is a compiled Shader Module. If not, create a Shader Module. According to the Shader code, construct the slot information parameters of the texture of the sampler required by BindGroupLayout, and generate BindGroupLayout; Generate a Cache Key according to the material state value, macro definition value, and material name to obtain whether there is a reusable pipeline; if not, call the preset underlying interface to create a rendering pipeline, pass the created BindGroupLayout, the rendering state contained in the material, and the Shader Module, and output the rendering pipeline; According to the pre-set sorting algorithm, the rendering queue is sorted, the sorted rendering queue is obtained, and the rendering object is rendered according to the sorting; The pre-set sorting algorithm includes: Render opaque objects first, then transparent objects; For transparent rendering objects, the larger the renderOrder attribute, the earlier they are rendered. By default, if the value is 0, they are sorted according to the order in which they are added to the rendering queue. For opaque rendering objects, contrary to transparent rendering objects, the larger the value, the closer it is to the front, and the smaller the value, the closer it is to the back; S14, when any rendering device provides rendering service for the user terminal, the control center detects in real time the user picture frame rate corresponding to the user picture provided by the rendering device, and determines whether the device status of the rendering device is healthy based on a preset picture health threshold; When it is unhealthy, the control center switches another rendering device to connect to the user end based on a pre-set exclusive scheduling algorithm to provide rendering services for the user end.

2. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S11 and before also include: S10, the control center establishes an object management pool according to the received at least one first 3D model; Then the S11 includes: S11-1. When the control center receives a target scene construction requirement sent by a user, the control center obtains at least one second 3D model sent by the user and a model element corresponding to each second 3D model, and according to the model requirement for building the target scene, obtains at least one first 3D model required for building the target scene from the object management pool, and configures the model element corresponding to each first 3D model; The target scene construction requirements include the real geographical location information corresponding to the target scene, the virtual coordinate information of the scene reference point and the model requirements for building the target scene; Each 3D model is a first 3D model or a second 3D model; S11-2, the control center obtains the real coordinate information corresponding to the scene reference point according to the virtual coordinate information of the scene reference point and the real geographical location information corresponding to the target scene; S11-3, the control center establishes a first coordinate transformation relationship according to the virtual coordinate information and the real coordinate information of the scene reference point; the first coordinate transformation relationship is a transformation relationship between the virtual coordinate and the real coordinate; S11-4, the control center configures the virtual coordinates of each first 3D model and the second 3D model according to the first coordinate conversion relationship and the real geographical location information of the target scene, and adjusts all the 3D models to corresponding positions; S11-5, the control center performs information diagnosis on each 3D model according to the pre-set repair tool and corrects the error part; S11-6, the control center configures the scene light source and the scene camera, completes the construction of the target scene, and obtains the model elements corresponding to each 3D model at this time; S11-7, the control center sends the built target scene and the model elements corresponding to each 3D model in the target scene to all rendering devices.

3. The digital twin method based on real-time rendering according to claim 2, characterized in that: The S11-1 also includes: The control center configures the spatiotemporal element information corresponding to each 3D model based on the real geographic location information corresponding to the target scene.

4. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S13 includes: S13-1, after any rendering device is connected to the user end, it receives the picture request from the user end in real time, and constructs the camera viewing cone of the target scene according to the picture request, that is, the picture to be rendered corresponding to the picture request; The picture request includes camera position information and viewpoint information in the target scene; S13-2, the rendering device transfers the camera view frustum to the octree container object according to the octree object corresponding to each 3D model in the target scene, performs step-by-step traversal and category classification, obtains the corresponding transparent rendering object and opaque rendering object in the to-be-rendered picture, and stores the transparent rendering object and opaque rendering object in the rendering queue; The octree object corresponding to the 3D model is the object generated after the octree division of the 3D model; S13-3, the rendering device obtains the global data of the rendering queue and the model matrix of each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the MVP matrix corresponding to each rendering object in the rendering queue according to the global data of the rendering queue and the model matrix of each rendering object in the rendering queue; S13-4, the rendering device traverses each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the material of each rendering object in the rendering queue; and determines the rendering parameters, material name, material state value, texture map and macro definition value corresponding to each rendering object in the rendering queue according to the material of each rendering object in the rendering queue; S13-5, the rendering device determines whether the rendering pipeline has been created according to the Cache. If the Cache exists, the creation is completed and the rendering pipeline is directly called for rendering. If the Cache does not exist, the rendering pipeline is created and called to render each rendering object in the rendering queue to obtain a 2D image corresponding to the picture to be rendered. The Cache is set when creating the rendering pipeline; The model elements corresponding to each 3D model include materials and global data; The global data specifically includes the camera's projection matrix, the camera's view matrix, plane resolution, mouse position, timestamp, and exposure.

5. The digital twin method based on real-time rendering according to claim 4 is characterized in that: The S13-2 includes: Traverse the octree container object step by step, starting from the root node, and check layer by layer whether the bounding box of the octree container object node intersects with the camera's viewing frustum; If they intersect, check whether the bounding box of the 3D object in the node intersects with the camera's viewing frustum; if they intersect, mark the rendered object and continue traversing; if they do not intersect, continue traversing downwards; If the bounding box of the octree container object node does not intersect with the camera viewing frustum, the node is directly skipped and the traversal continues downward; According to the renderOrder attribute of each rendering object in the rendering queue, the rendering object is divided into an opaque rendering object and a transparent rendering object; The renderOrder property of a transparent render object is greater than the set value, and the renderOrder property of an opaque render object is less than the set value.

6. The digital twin method based on real-time rendering according to claim 4, characterized in that: The global data also includes basic structure information; The basic structure information includes vertex information, texture coordinate information, transformation information and hierarchical relationship information; After S11 and before S13-3, the following are included: The rendering device asynchronously loads the WASM memory module and dynamically adds scripts to construct the basic structure information of each rendering object in the rendering queue into an initial matrix, and stores the constructed initial matrix into the matrix buffer of the WASM memory module; The S13-3 includes: All initial matrices in the matrix buffer are translated, rotated, scaled, and their parent-child relationships are confirmed to obtain the updated model matrix for each initial matrix; Perform MVP matrix calculation on the model matrix corresponding to each rendering object in the rendering queue to obtain the MVP matrix of each rendering object in the rendering queue; The matrix buffer comprises: a first matrix buffer for storing matrix data, a second matrix buffer for storing scaling, rotation and translation data of each matrix, a third matrix buffer for storing continuous transformation data of the matrix and a fourth matrix buffer for storing state information of each matrix; The state information of each matrix includes one-time transformation state information, continuous transformation state information, parent matrix index information, and depth order information.

7. The digital twin method based on real-time rendering according to claim 6, characterized in that: The S13-3, translating, rotating, scaling, and confirming the parent-child relationship of all the initial matrices in the matrix buffer, to obtain the model matrix corresponding to each initial matrix, includes: Determine the translation vector, rotation axis, rotation angle and scaling factor when the matrix is ​​updated, and construct a translation matrix, a rotation matrix and a scaling matrix according to the translation vector, rotation axis, rotation angle and scaling factor; According to the translation matrix, rotation matrix and scaling matrix, and the preset formula 1, a model matrix is ​​obtained for updating the initial matrix; the formula 1 is: Model matrix = translation matrix × rotation matrix × scaling matrix.

8. The digital twin method based on real-time rendering according to claim 6, characterized in that: The S13-3 also includes: Determine the parent-child relationship of different model matrices according to the preset SetParent strategy; When there is a parent-child relationship, the updated model matrix is: Updated parent matrix = child matrix × model matrix; Updated child matrix = parent matrix -1 × model matrix.

9. The digital twin method based on real-time rendering according to claim 6, characterized in that: The step S13-3, performing MVP matrix calculation on the model matrix corresponding to each rendering object in the rendering queue to obtain the MVP matrix of each rendering object in the rendering queue, includes: According to the observation point position, direction and field of view angle of the camera's viewing cone corresponding to the image to be rendered, the view matrix corresponding to each rendering object in the rendering queue is dynamically constructed using the preset lookAt function; According to the field of view angle, aspect ratio, near clipping plane distance and far clipping plane distance of the camera viewing cone corresponding to the image to be rendered, the projection matrix corresponding to each rendering object in the rendering queue is constructed through a pre-set tool library; According to the model matrix, view matrix and projection matrix corresponding to each rendering object in the rendering queue and the preset formula 2, the MVP matrix corresponding to each rendering object in the rendering queue is obtained; the formula 2 is: MVP matrix = view matrix × projection matrix × updated model matrix.

10. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S12 includes: When the control center receives a rendering request sent by a client, it detects whether there is any rendering device connected to the client to provide rendering services for the client; If it does not exist, the control center schedules any rendering device to connect to the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm.

11. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S14 includes: S14-1. After any rendering device is connected to the user terminal, the control center detects the user screen frame rate of the rendering device connected to the user terminal in real time during the rendering process, and determines whether the user screen frame rate of the current rendering device during the rendering process is abnormal based on a preset frame rate threshold, and records the number of abnormal user screen frame rates; During this period, when the user screen frame rate detected by the control center is less than the frame rate threshold, the number of frame rate anomalies increases by one; When the user screen frame rate detected by the control center is greater than or equal to the frame rate threshold, the number of frame rate anomalies decreases by one; S14-2, the control center detects the number of abnormal frame rates of the user screen in real time, and determines whether the screen rendered by the current rendering device is healthy based on the frame rate abnormality number threshold; When the number of abnormal frame rates of the user screen is greater than or equal to the frame rate abnormal number threshold, it is determined that the screen rendered by the current rendering device is unhealthy; S14-3, when it is unhealthy, the control center detects whether there is a rendering device that is not connected to the user terminal to provide rendering services among all the rendering devices connected to it; S14-4, if it exists, the control center switches another rendering device to connect with the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm, and repeats steps S14-1 to S14-4; If it does not exist, the control center schedules the protection and monitoring of the current rendering device based on the pre-set daemon; The user screen frame rate is the number of 2D images played by the user terminal per unit time during the period when the rendering device provides rendering services to the user terminal; The image health threshold includes the frame rate threshold and the frame rate abnormality threshold; The daemon is used to restart the rendering service when the rendering device crashes during the process of providing rendering service to the client.

12. The digital twin method based on real-time rendering according to claim 11, characterized in that: The method further comprises: When the control center schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal, the total running times preset by the rendering device are updated, that is, the total running times of the rendering device are increased by one; When the control center determines that the image rendered by any rendering device is unhealthy, the crash count preset by the rendering device is updated, that is, the crash count of the rendering device is increased by one; When the current rendering device provides rendering services to the user end, the control center records the user screen frame rate of the rendering device in the rendering process in real time; When another rendering device is switched to connect with the client to provide rendering services for the client, the control center records the service duration of the rendering device before the switching to provide rendering services for the client, and stores the service duration as the historical service duration in a preset historical service duration database; Then the step S14-4 also includes: S14-5. When the control center determines that the images rendered by all the connected rendering devices are unhealthy, the control center obtains the frame rate stability of each rendering device based on the frame rates of all user images when each rendering device provides rendering services for the client, the total number of operations corresponding to each rendering device, the number of crashes corresponding to each rendering device, and all historical service durations corresponding to each rendering device in the historical service duration database, as well as the preset formula three; the formula three is: ； Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, and the average frame rate and frame rate standard deviation are both obtained based on the frame rates of all user screens when the rendering device provides rendering services to the user terminal. is a preset minimum constant, HAD is the historical average service duration corresponding to the rendering device, the historical average service duration is obtained based on all historical service durations corresponding to the rendering device, MAX_DUR is the preset maximum operating duration of the rendering device, k is a preset weight coefficient, C is the number of crashes of the rendering device, and T is the total number of operations of the rendering device; Based on the frame rate stability corresponding to each rendering device, the control center switches to the rendering device with the highest frame rate stability to connect with the user terminal and provide rendering services to the user terminal.

13. The digital twin method based on real-time rendering according to claim 11, characterized in that: The S14-2 also includes: Acquire the device status of the current rendering device in real time, and adjust the picture health threshold in real time based on a preset dynamic threshold adjustment strategy and a preset picture health threshold; the picture health threshold includes a frame rate threshold; The device status includes: GPU temperature, GPU usage, video memory usage, CPU usage, memory usage and disk usage; The dynamic threshold adjustment strategy includes: When the GPU temperature is greater than 80 degrees and less than 100 degrees, the first frame rate threshold increases by 1 for every 5 degrees increase; when the GPU temperature is greater than 100 degrees, the first frame rate threshold increases by 10 for every 5 degrees increase; When the GPU usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the video memory usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the CPU usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the memory usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the disk usage is greater than 80%, the disk usage enters an alarm state. For every 5% increase, the first frame rate threshold increases by 0.

5.

14. The digital twin method based on real-time rendering according to claim 11, characterized in that: The S14-2 also includes: The control center monitors and synchronizes the updated data of the rendering device connected to the user end in real time during the rendering process; The S14-4 also includes: When another rendering device is switched to connect with the client to provide rendering service for the client, the control center synchronizes the latest updated data of the rendering device before switching to the rendering device after switching, so as to synchronize the data of the rendering device after switching.

15. The digital twin method based on real-time rendering according to claim 14, characterized in that: The S14-4 also includes: when switching another rendering device to connect to the user terminal to provide rendering services for the user terminal, the control center uses the most recent cut-off frequency of the rendering device before the switching as the picture provided to the user terminal until the switching is completed, and hides the cut-off frequency.

Images