Digital twinning method based on real-time rendering

Abstract

The invention relates to the technical field of three-dimensional scene display, in particular to a digital twinning method based on real-time rendering, which comprises the following steps: a control center builds a target scene and synchronizes the target scene to rendering equipment according to a model file and a target scene building demand; when the control center receives a rendering request, any rendering device is scheduled to be connected with the user side; the rendering device obtains a to-be-rendered picture according to the received picture request, obtains a 2D image corresponding to the to-be-rendered picture based on the model elements of the 3D model, and sends the 2D image to the user side; when the rendering equipment provides service for the user side, the control center detects the user picture frame rate of the rendering equipment in real time and judges whether the equipment state of the rendering equipment is healthy or not; and when the user is not healthy, the control center switches another rendering device to be connected with the user side. The method has the beneficial effects that coordinate conversion can be automatically processed based on the model file, and stable output of pictures in a real-time rendering process is ensured while a target scene is established.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scene display technology, and in particular to a digital twin method based on real-time rendering. Background Art

[0002] Digital twin technology is an emerging technology concept that refers to the use of digital technology to digitize physical entities or systems, establish corresponding virtual models, and achieve real-time data synchronization, interaction, simulation and analysis between physical and virtual models. Digital twins are usually used in industrial production, urban planning, architectural design, the Internet of Things and other fields. They aim to improve system operating efficiency, optimize design solutions, reduce costs and risks, and increase reliability through the interaction between virtual simulation and actual systems.

[0003] Existing digital twin technology is usually targeted at specific scenarios in specific fields. Targeted construction for specific scenarios in specific fields requires a lot of time and resources for customized development. For general users or enterprises, customization is difficult, and when the demand or scenario changes, it needs to be rebuilt from scratch, which lacks good flexibility.

[0004] Moreover, the current digital twin technology based on real-time rendering cannot guarantee the smoothness of the user's screen. Once an abnormality occurs during the rendering process, that is, the screen freezes or the service crashes, it will affect the stable output of the rendered screen and affect the user experience.

[0005] To sum up, at this stage, there is an urgent need for a real-time rendering digital twin technology that can be flexibly applied to various scenarios and improve the flexibility of user screens during real-time rendering. Summary of the invention

[0006] 1. Technical issues to be resolved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a digital twin method based on real-time rendering, which solves the technical problems that the existing digital twin technology is a targeted construction in a specific field and a specific scenario, lacks good flexibility, and cannot guarantee the smoothness of the user screen during real-time rendering, affecting the user experience.

[0007] (II) Technical solution In order to achieve the above object, the main technical solutions adopted by the present invention include: In a first aspect, an embodiment of the present invention provides a digital twin method based on real-time rendering, the method is implemented based on a pre-set control center and at least two rendering devices connected to the control center, and the method includes: S11, the control center builds the target scene according to the preset model file, the real geographical location information of the target scene and the virtual coordinate information of the scene reference point, and sends the target scene and the model elements corresponding to each 3D model in the target scene to all rendering devices; the model file includes at least one 3D model and the model elements corresponding to each 3D model; S12, when the control center receives a rendering request sent by the user terminal, based on a preset exclusive scheduling algorithm, schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal; S13, after any rendering device is connected to the user terminal, it receives the picture request of the user terminal in real time, obtains the corresponding picture to be rendered according to the picture request, and renders the picture to be rendered in real time based on the model elements corresponding to each 3D model in the target scene, obtains the 2D image corresponding to the picture to be rendered, and sends it to the user terminal; S14, when any rendering device provides rendering service for the user terminal, the control center detects in real time the user picture frame rate corresponding to the user picture provided by the rendering device, and determines whether the device status of the rendering device is healthy based on a preset picture health threshold; When it is unhealthy, the control center switches another rendering device to connect to the user end based on a pre-set exclusive scheduling algorithm to provide rendering services for the user end.

[0008] Optionally, before S11, the step further includes: S10, the control center establishes an object management pool according to the received at least one first 3D model; Then the S11 includes: S11-1. When the control center receives a target scene construction requirement sent by a user, the control center obtains at least one second 3D model sent by the user and a model element corresponding to each second 3D model, and according to the model requirement for building the target scene, obtains at least one first 3D model required for building the target scene from the object management pool, and configures the model element corresponding to each first 3D model; The target scene construction requirements include the real geographical location information corresponding to the target scene, the virtual coordinate information of the scene reference point and the model requirements for building the target scene; Each 3D model is a first 3D model or a second 3D model; S11-2, the control center obtains the real coordinate information corresponding to the scene reference point according to the virtual coordinate information of the scene reference point and the real geographical location information corresponding to the target scene; S11-3, the control center establishes a first coordinate transformation relationship according to the virtual coordinate information and the real coordinate information of the scene reference point; the first coordinate transformation relationship is a transformation relationship between the virtual coordinate and the real coordinate; S11-4, the control center configures the virtual coordinates of each first 3D model and the second 3D model according to the first coordinate conversion relationship and the real geographical location information of the target scene, and adjusts all the 3D models to corresponding positions; S11-5, the control center performs information diagnosis on each 3D model according to the pre-set repair tool and corrects the error part; S11-6, the control center configures the scene light source and the scene camera, completes the construction of the target scene, and obtains the model elements corresponding to each 3D model at this time; S11-7, the control center sends the built target scene and the model elements corresponding to each 3D model in the target scene to all rendering devices.

[0009] Optionally, the S11-1 further includes: The control center configures the spatiotemporal element information corresponding to each 3D model based on the real geographic location information corresponding to the target scene.

[0010] Optionally, the S13 includes: S13-1, after any rendering device is connected to the user end, it receives the picture request from the user end in real time, and constructs the camera viewing cone of the target scene according to the picture request, that is, the picture to be rendered corresponding to the picture request; The picture request includes camera position information and viewpoint information in the target scene; S13-2, the rendering device transfers the camera view frustum to the octree container object according to the octree object corresponding to each 3D model in the target scene, performs step-by-step traversal and category classification, obtains the corresponding transparent rendering object and opaque rendering object in the to-be-rendered picture, and stores the transparent rendering object and opaque rendering object in the rendering queue; The octree object corresponding to the 3D model is the object generated after the octree division of the 3D model; S13-3, the rendering device obtains the global data of the rendering queue and the model matrix of each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the MVP matrix corresponding to each rendering object in the rendering queue according to the global data of the rendering queue and the model matrix of each rendering object in the rendering queue; S13-4, the rendering device traverses each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the material of each rendering object in the rendering queue; and determines the rendering parameters, material name, material state value, texture map and macro definition value corresponding to each rendering object in the rendering queue according to the material of each rendering object in the rendering queue; S13-5, the rendering device determines whether the rendering pipeline has been created according to the Cache. If the Cache exists, the creation is completed and the rendering pipeline is directly called for rendering. If the Cache does not exist, the rendering pipeline is created and called to render each rendering object in the rendering queue to obtain a 2D image corresponding to the picture to be rendered. The Cache is set when creating the rendering pipeline; The model elements corresponding to each 3D model include materials and global data; The global data specifically includes the camera's projection matrix, the camera's view matrix, plane resolution, mouse position, timestamp, and exposure.

[0011] Optionally, the S13-2 includes: Traverse the octree container object step by step, starting from the root node, and check layer by layer whether the bounding box of the octree container object node intersects with the camera's viewing frustum; If they intersect, check whether the bounding box of the 3D object in the node intersects with the camera's viewing frustum; if they intersect, mark the rendered object and continue traversing; if they do not intersect, continue traversing downwards; If the bounding box of the octree container object node does not intersect with the camera viewing frustum, the node is directly skipped and the traversal continues downward; According to the renderOrder attribute of each rendering object in the rendering queue, the rendering object is divided into an opaque rendering object and a transparent rendering object; The renderOrder property of a transparent render object is greater than the set value, and the renderOrder property of an opaque render object is less than the set value.

[0012] Optionally, in S13-5, creating a rendering pipeline includes: Generate Shader ModuleCache Key according to the name, macro variables and specific Shader code of the Shader code, and check whether there is a compiled Shader Module. If not, create a Shader Module. According to the Shader code, construct the slot information parameters of the texture of the sampler required by BindGroupLayout, and generate BindGroupLayout; Generate a Cache Key according to the material state value, macro definition value, and material name to obtain whether there is a reusable pipeline; if not, call the preset underlying interface to create a rendering pipeline, pass the created BindGroupLayout, the rendering state contained in the material, and the Shader Module, and output the rendering pipeline; According to the pre-set sorting algorithm, the rendering queue is sorted, the sorted rendering queue is obtained, and the rendering object is rendered according to the sorting; The pre-set sorting algorithm includes: Render opaque objects first, then transparent objects; For transparent rendering objects, the larger the renderOrder attribute, the earlier they are rendered. By default, if the value is 0, they are sorted according to the order in which they are added to the rendering queue. For opaque rendering objects, contrary to transparent rendering objects, larger values ​​are rendered in the front, and smaller values ​​are rendered in the back.

[0013] Optionally, the global data also includes basic structure information; The basic structure information includes vertex information, texture coordinate information, transformation information and hierarchical relationship information; After S11 and before S13-3, the following are included: The rendering device asynchronously loads the WASM memory module and dynamically adds scripts to construct the basic structure information of each rendering object in the rendering queue into an initial matrix, and stores the constructed initial matrix into the matrix buffer of the WASM memory module; The S13-3 includes: All initial matrices in the matrix buffer are translated, rotated, scaled, and their parent-child relationships are confirmed to obtain the updated model matrix for each initial matrix; Perform MVP matrix calculation on the model matrix corresponding to each rendering object in the rendering queue to obtain the MVP matrix of each rendering object in the rendering queue; The matrix buffer comprises: a first matrix buffer for storing matrix data, a second matrix buffer for storing scaling, rotation and translation data of each matrix, a third matrix buffer for storing continuous transformation data of the matrix and a fourth matrix buffer for storing state information of each matrix; The state information of each matrix includes one-time transformation state information, continuous transformation state information, parent matrix index information, and depth order information.

[0014] Optionally, the S13-3, translating, rotating, scaling, and confirming the parent-child relationship of all initial matrices in the matrix buffer to obtain a model matrix corresponding to each initial matrix, includes: Determine the translation vector, rotation axis, rotation angle and scaling factor when the matrix is ​​updated, and construct a translation matrix, a rotation matrix and a scaling matrix according to the translation vector, rotation axis, rotation angle and scaling factor; According to the translation matrix, rotation matrix and scaling matrix, and the preset formula 1, a model matrix is ​​obtained for updating the initial matrix; the formula 1 is: Model matrix = translation matrix × rotation matrix × scaling matrix.

[0015] Optionally, the S13-3 further includes: Determine the parent-child relationship of different model matrices according to the preset SetParent strategy; When there is a parent-child relationship, the updated model matrix is: Updated parent matrix = child matrix × model matrix; Updated child matrix = parent matrix -1 × model matrix.

[0016] Optionally, the S13-3, performing MVP matrix calculation on the model matrix corresponding to each rendering object in the rendering queue to obtain the MVP matrix of each rendering object in the rendering queue, includes: According to the observation point position, direction and field of view angle of the camera's viewing cone corresponding to the image to be rendered, the view matrix corresponding to each rendering object in the rendering queue is dynamically constructed using the preset lookAt function; According to the field of view angle, aspect ratio, near clipping plane distance and far clipping plane distance of the camera viewing cone corresponding to the image to be rendered, the projection matrix corresponding to each rendering object in the rendering queue is constructed through a pre-set tool library; According to the model matrix, view matrix and projection matrix corresponding to each rendering object in the rendering queue and the preset formula 2, the MVP matrix corresponding to each rendering object in the rendering queue is obtained; the formula 2 is: MVP matrix = view matrix × projection matrix × updated model matrix.

[0017] Optionally, the S12 includes: When the control center receives a rendering request sent by a client, it detects whether there is any rendering device connected to the client to provide rendering services for the client; If it does not exist, the control center schedules any rendering device to connect to the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm.

[0018] Optionally, the S14 includes: S14-1. After any rendering device is connected to the user terminal, the control center detects the user screen frame rate of the rendering device connected to the user terminal in real time during the rendering process, and determines whether the user screen frame rate of the current rendering device during the rendering process is abnormal based on a preset frame rate threshold, and records the number of abnormal user screen frame rates; During this period, when the user screen frame rate detected by the control center is less than the frame rate threshold, the number of frame rate anomalies increases by one; When the user screen frame rate detected by the control center is greater than or equal to the frame rate threshold, the number of frame rate anomalies decreases by one; S14-2, the control center detects the number of abnormal frame rates of the user screen in real time, and determines whether the screen rendered by the current rendering device is healthy based on the frame rate abnormality number threshold; When the number of abnormal frame rates of the user screen is greater than or equal to the frame rate abnormal number threshold, it is determined that the screen rendered by the current rendering device is unhealthy; S14-3, when it is unhealthy, the control center detects whether there is a rendering device that is not connected to the user terminal to provide rendering services among all the rendering devices connected to it; S14-4, if it exists, the control center switches another rendering device to connect with the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm, and repeats steps S14-1 to S14-4; If it does not exist, the control center schedules the protection and monitoring of the current rendering device based on the pre-set daemon; The user screen frame rate is the number of 2D images played by the user terminal per unit time during the period when the rendering device provides rendering services to the user terminal; The picture health threshold includes a frame rate threshold and a frame rate abnormality times threshold; The daemon is used to restart the rendering service when the rendering device crashes during the process of providing rendering service to the client.

[0019] Optionally, the method further comprises: When the control center schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal, the total running times preset by the rendering device are updated, that is, the total running times of the rendering device are increased by one; When the control center determines that the image rendered by any rendering device is unhealthy, the crash count preset by the rendering device is updated, that is, the crash count of the rendering device is increased by one; When the current rendering device provides rendering services to the user end, the control center records the user screen frame rate of the rendering device in the rendering process in real time; When another rendering device is switched to connect with the client to provide rendering services for the client, the control center records the service duration of the rendering device before the switching to provide rendering services for the client, and stores the service duration as the historical service duration in a preset historical service duration database; Then the step S14-4 also includes: S14-5. When the control center determines that the images rendered by all connected rendering devices are unhealthy, the control center obtains the frame rate stability of each rendering device based on the frame rates of all user images when each rendering device provides rendering services for the client, the total number of operations corresponding to each rendering device, the number of crashes corresponding to each rendering device, and all historical service durations corresponding to each rendering device in the historical service duration database, as well as the preset formula three; the formula three is: ; Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, and the average frame rate and frame rate standard deviation are both obtained based on the frame rates of all user screens when the rendering device provides rendering services to the user terminal. is a preset minimum constant, HAD is the historical average service duration corresponding to the rendering device, the historical average service duration is obtained based on all historical service durations corresponding to the rendering device, MAX_DUR is the preset maximum operating duration of the rendering device, k is a preset weight coefficient, C is the number of crashes of the rendering device, and T is the total number of operations of the rendering device; Based on the frame rate stability corresponding to each rendering device, the control center switches to the rendering device with the highest frame rate stability to connect with the user terminal and provide rendering services to the user terminal.

[0020] Optionally, the S14-2 further includes: Acquire the device status of the current rendering device in real time, and adjust the picture health threshold in real time based on a preset dynamic threshold adjustment strategy and a preset picture health threshold; the picture health threshold includes a frame rate threshold; The device status includes: GPU temperature, GPU usage, video memory usage, CPU usage, memory usage and disk usage; The dynamic threshold adjustment strategy includes: When the GPU temperature is greater than 80 degrees and less than 100 degrees, the first frame rate threshold increases by 1 for every 5 degrees increase; when the GPU temperature is greater than 100 degrees, the first frame rate threshold increases by 10 for every 5 degrees increase; When the GPU usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the video memory usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the CPU usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the memory usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the disk usage is greater than 80%, the disk usage enters an alarm state. For every 5% increase, the first frame rate threshold increases by 0.5.

[0021] Optionally, the S14-2 further includes: The control center monitors and synchronizes the updated data of the rendering device connected to the user end in real time during the rendering process; The S14-4 also includes: When another rendering device is switched to connect with the client to provide rendering service for the client, the control center synchronizes the latest updated data of the rendering device before switching to the rendering device after switching, so as to synchronize the data of the rendering device after switching.

[0022] Optionally, S14-4 also includes: when switching another rendering device to connect to the user terminal to provide rendering services to the user terminal, the control center uses the most recent cut-off frequency of the rendering device before the switching as the picture provided to the user terminal until the switching is completed, and hides the cut-off frequency.

[0023] (III) Beneficial effects The beneficial effect of the present invention is that a digital twin method based on real-time rendering of the present invention can automatically process coordinate conversion based on model files, build target scenes and other tasks, while ensuring stable output of images during real-time rendering, and reducing the impact of image freezes or server crashes on user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flow chart of a digital twin method based on real-time rendering provided in Example 1 of the present invention; Figure 2 A flowchart of a method for generating a real-time rendering pipeline provided in Embodiment 2 of the present invention; Figure 3 A flowchart of initializing a rendering pipeline provided by Embodiment 2 of the present invention; Figure 4 The whole engine rendering flow chart provided by Embodiment 2 of the present invention; Figure 5 A schematic diagram of an octree structure provided in Example 2 of the present invention; Figure 6 A schematic diagram of the structure of a camera viewing cone provided in Embodiment 2 of the present invention; Figure 7 A schematic diagram of a scene view provided in Embodiment 2 of the present invention; Figure 8 A schematic diagram of memory sharing of a WASM-based real-time rendering method provided in Example 3 of the present invention; Fig. 9A schematic diagram of memory allocation in a matrix buffer area taking 100 matrices as an example provided in Embodiment 3 of the present invention; Fig.10 A flowchart of a WASM-based real-time rendering method provided in Embodiment 3 of the present invention; Fig.11 A flowchart of a method for optimizing a real-time rendering image provided by Embodiment 4 of the present invention; Fig.12 A shot synchronization flow chart of a method for optimizing real-time rendering of images provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0025] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0026] Cache Key: Cache keyword. In the engine, some reused functional modules are hit by this keyword.

[0027] RenderOrder: Rendering order number.

[0028] Shader: A language running on the GPU, which is the WGSL language in the embodiment of the present invention.

[0029] Shader Moudle: The compiled module generated after the Shader code is compiled.

[0030] ShaderModule Cache Key: Cache keyword of Shader Moudle.

[0031] BindGroupLayout: A piece of configuration code passed to the GPU to obtain the data specification format of the cache data inside the GPU.

[0032] A digital twin method based on real-time rendering proposed in an embodiment of the present invention can automatically process coordinate conversion based on model files, build target scenes and other tasks, while ensuring stable output of images during real-time rendering and reducing the impact of image freezes or server crashes on user experience.

[0033] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0034] Example 1

[0035] The embodiment of the present invention provides a digital twin method based on real-time rendering. The method is implemented based on a pre-set control center and at least two rendering devices connected to the control center. The method flow is as follows: Figure 1 As shown, including: S11, the control center builds the target scene according to the preset model file, the real geographical location information of the target scene and the virtual coordinate information of the scene reference point, and sends the target scene and the model elements corresponding to each 3D model in the target scene to all rendering devices; The model file includes at least one 3D model and model elements corresponding to each 3D model; S12, when the control center receives a rendering request sent by the user terminal, based on a preset exclusive scheduling algorithm, schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal; S13, after any rendering device is connected to the user terminal, it receives the picture request of the user terminal in real time, obtains the corresponding picture to be rendered according to the picture request, and renders the picture to be rendered in real time based on the model elements corresponding to each 3D model in the target scene, obtains the 2D image corresponding to the picture to be rendered, and sends it to the user terminal; S14, when any rendering device provides rendering service for the user terminal, the control center detects in real time the user picture frame rate corresponding to the user picture provided by the rendering device, and determines whether the device status of the rendering device is healthy based on a preset picture health threshold; When it is unhealthy, the control center switches another rendering device to connect to the user end based on a pre-set exclusive scheduling algorithm to provide rendering services for the user end.

[0036] This embodiment provides a digital twin method based on real-time rendering, which can automatically process coordinate conversion based on model files, build target scenes and other tasks, while ensuring stable output of images during real-time rendering and reducing the impact of image freezes or server crashes on user experience.

[0037] Example 2

[0038] This embodiment provides a method for generating a real-time rendering pipeline, such as Figure 2 As shown, including: Step 1: Get the camera frustum of the object to be tested (3D model), pass the camera frustum to the octree container object, perform step-by-step traversal and category differentiation, get transparent rendering objects and opaque rendering objects, and store the transparent rendering objects and opaque rendering objects in the rendering queue; specifically include: Traverse the octree container object step by step, starting from the root node, and check whether the bounding box of the octree container object node intersects with the camera frustum layer by layer; if it intersects, check whether the bounding box of the three-dimensional object in the node intersects with the camera frustum; if it intersects, mark the rendering object and continue traversing; if it does not intersect, continue to traverse downward; if the bounding box of the octree container object node does not intersect with the camera frustum, skip the node directly and continue to traverse downward. According to the renderOrder attribute of each rendering object in the rendering queue, the rendering object is divided into opaque rendering objects and transparent rendering objects; the renderOrder attribute of transparent objects is greater than the set value, and the renderOrder attribute of opaque objects is less than the set value. The set value is set according to the actual situation. For example, the set value can be 3000. At this time, the renderOrder attribute greater than 3000 is a transparent object, and the renderOrder attribute less than 3000 is an opaque object.

[0039] Among them, the viewing frustum: The viewing frustum is a core concept in computer graphics, especially in 3D rendering and geometric culling technology. The viewing frustum vividly describes the 3D space area that can be seen from the observer (such as the camera or the player's perspective). The shape of this area is similar to an incomplete cuboid, with its two ends gradually shrinking, similar to the top of a four-sided pyramid being truncated, such as Figure 6 shown.

[0040] Octree, such as Figure 5 As shown in Figure 1, an octree is a tree-shaped data structure used to organize three-dimensional spatial data. It recursively divides the space into eight equal sub-regions (similar to a quadtree in two-dimensional space), and each sub-region can continue to be divided until a certain stopping condition is met (such as reaching a certain depth limit or each sub-region contains only one or zero data points). Each node of the octree represents a spatial region, and leaf nodes usually represent the region that actually contains data, while non-leaf nodes are used to assist in spatial division and search.

[0041] The screening principle of the octree is: Objects that are not within the scope of the octree are stored in the root node of the octree (level 0); if an object is in level 0 and is contained in multiple levels 1, the object is placed in each level 1 respectively; other spaces larger than level 1, if they are occupied in multiple subsets, are stored in the parent level. For example, if an object exists in both 3-1 space and 3-2 space, it is stored in their parent level 2; changes in the position and bounding box of the object will trigger the recalculation of the octree position and clear the original position.

[0042] Through the above rules, it can be concluded that level 0 is the object that is not in the octree, level 1 may store the objects that can be accommodated and some three-dimensional objects in the space, and levels 2 and above store objects that can be accommodated. The rules use ordinary methods, so that the large number of rendering objects at level 0 mentioned above can be transferred to level 1. 1 level 0 contains 8 levels 1, so that the objects can be divided into eight subspaces. When rendering and filtering, level 1 will be filtered first, so a large number of objects can be filtered out, thereby achieving the goal of high performance.

[0043] The camera frustum is obtained through the viewpoint information provided by the pre-assumed camera; Step 2: Get the global data of the rendering queue and the model matrix of each rendering object in the rendering queue, input the projection matrix, view matrix and model matrix of the rendering object in the global data into the following formula to get the MVP matrix: MVP matrix = projection matrix × view matrix × model matrix. Upload the global data and the MVP matrix of each rendering object to the GPU at one time. The model elements of each 3D model, that is, the model elements of the object to be tested, include global data.

[0044] The model matrix of each rendered object is determined according to its material, and the model matrix represents the position, rotation, and scale of the object in the world coordinate system. Specifically, obtaining the model matrix includes the following steps: initializing a unit matrix; applying a translation transformation according to the position information of the rendered object; applying a rotation transformation according to the position information of the rendered object; applying a scaling transformation according to the scaling information of the rendered object; and multiplying the above transformation matrices in sequence to obtain the final model matrix.

[0045] Step 3: Traverse each object in the rendering queue, obtain its corresponding material, determine the specific rendering parameters, material name, material status value, texture map and macro definition value of the rendering object according to the material, and upload the specific rendering parameters of the rendering object to the GPU at one time.

[0046] Step 4: Determine whether the rendering pipeline has been created based on the Cache. If the Cache exists, the creation is complete and the rendering pipeline is directly called for rendering. If the Cache does not exist, create a rendering pipeline and call the rendering pipeline to render each rendering object in the rendering queue to obtain the 2D image corresponding to the object to be tested. The Cache is set when creating the rendering pipeline. Figure 3 As shown, specifically including: Get the Shader Module cache; specifically, generate the Shader Module Cache Key according to the name of the Shader code, macro variables and specific Shader code, find out whether there is a compiled Shader Module, and if not, create a Shader Module.

[0047] According to the Shader code, the texture slot information parameters of the sampler required by BindGroupLayout are constructed to generate BindGroupLayout; the texture name is used as a Cache Key. If a texture already exists in the cache, the texture can be reused directly without re-creating it; at the same time, the number of references of each texture is recorded. When the number of references is 0, it means that the texture is no longer used by any material, so the texture can be safely destroyed; similarly, the sampler also uses a similar cache mechanism, using the state of the sampler as a Cache Key. If a sampler already exists in the cache, the sampler is directly reused. Through the above-mentioned cache mechanism, the overhead of repeatedly creating textures and samplers is reduced, thereby reducing GPU memory usage, reducing unnecessary resource creation and destruction operations, and improving rendering performance. At the same time, by reusing resources, the addressing time is reduced, further improving rendering efficiency.

[0048] Generate a Cache Key based on the material status value, macro definition value, and material name to determine whether there is a reusable pipeline. If not, call the preset underlying interface to create a rendering pipeline. The underlying interface is generally an interface for modern graphics programs. The created BindGroupLayout, the rendering status contained in the material, and the Shader Module are passed to output the rendering pipeline.

[0049] According to the pre-set sorting algorithm, the rendering queue is sorted, the sorted rendering queue is obtained, and the rendering objects are rendered according to the sorting. The pre-set sorting algorithm includes: rendering opaque objects first, then rendering transparent objects; for transparent rendering objects, the larger the renderOrder attribute, the earlier they are rendered. By default, if the value is 0, they are sorted according to the order of joining the rendering queue; for opaque rendering objects, contrary to transparent rendering objects, the larger the value, the earlier they are rendered, and the smaller the value, the later they are rendered.

[0050] Before step 1 of this embodiment, the following steps are also included: initializing the engine context, initializing the Shader system, initializing the global binding system, initializing the global Render Target, initializing the resource manager, and initializing the input system.

[0051] Specifically, initializing the engine context includes: initializing the Canvas window, creating an Adapter, and requesting a Device (interface device) through the adapter to execute the actual graphics or calculation command object.

[0052] When the Shader system is initialized, a series of Shader codes are preset in the engine, and macro definitions are added to these codes to improve code reusability. For example, different material types can choose which code segments to enable by changing the macro definition value. In addition, the binding points in the Shader code are set to be automatically allocated, which means that the specific location will be determined at compile time.

[0053] To initialize the global binding system, you need to create and initialize a memory space based on the global variables to store global variables such as camera projection matrix, screen size, mouse position, timestamp, etc. These contents will be stored in a Buffer (global buffer); by setting the Buffer, the above frequently used parameters can be efficiently passed to the GPU.

[0054] Initialize the global Render Target. Render Target refers to a buffer for receiving image data output. Specifically, it is a memory area, usually one or a group of textures (Texture) or frame buffers (FrameBuffer), which is used to store pixel data processed by the shader. If the rendered result needs to be post-processed, such as applying some image effects, the rendered result needs to be stored in a defined texture. To this end, initializing the global RenderTarget will set multiple textures as targets, storing color, depth, normal information, and material properties respectively, so as to facilitate secondary use in subsequent post-processing.

[0055] Initialize the resource manager, which is responsible for loading and caching various types of resource files, such as texture maps, 3D models, etc. Initialize the resource manager. Once the resource is loaded and parsed once, it will be saved in the memory environment and can be directly extracted from the cache the next time it is requested. This can significantly reduce the performance overhead caused by repeated loading.

[0056] Furthermore, the input system is initialized, that is, the initialization operation of binding the relevant mouse and keyboard, and the event scheduling of the input device is provided for the entire engine. The engine is the engine preset by the rendering device.

[0057] During the rendering of each frame, if it is detected that the rendering status of part of the 3D object has changed, the macro definition value has changed, the material parameters have changed, or the geometry has changed, it will trigger the re-initialization of the rendering pipeline and regenerate the rendering pipeline. The previously used pipeline will be destroyed if it is not used by other rendering objects.

[0058] If the user sets the material rendering state of a 3D object to the Bundle mode, the rendering pipeline will record a series of information such as the rendering state. The CPU and GPU only need to submit once, and subsequent rendering will maintain this mode, reducing the communication time between the CPU and GPU. For some 3D objects that need to be rendered multiple times, using this mode can greatly improve the rendering performance. The engine integrates the code, and the user only needs to change the material rendering state to achieve efficient rendering in this mode. Figure 4 As shown in the figure, when the rendering state of a 3D object changes and no other rendering objects are in use in the current rendering pipeline, the current rendering pipeline is destroyed in asynchronous mode. The asynchronous mode schedules the destruction in fixed batches frame by frame.

[0059] For example, rendering a cube and a sphere is used to illustrate this embodiment.

[0060] Step 1: Initialize the engine.

[0061] Step 2: Asset Generation and Creation: The user creates multiple views, such as Figure 7As shown in the figure, one of the views contains a camera and a scene. The scene itself contains the background sky. The scene can contain three-dimensional objects. The cube and sphere to be created are the three-dimensional objects to be rendered. Each three-dimensional object to be rendered needs to contain two parts, namely geometry and material. The geometry is used to represent the geometric structure of the object. It is composed of a series of triangular faces. The geometric information of the triangular faces usually requires vertices, normals, UVs, vertex colors, tangents, and indices. This information is generally automatically calculated and generated by the engine, or comes from model files (model files are generally made through some model editing tools such as: 3dMax, C4D, Blender and other tools). Each triangle has Three vertices, each vertex is represented by three numbers [x, y, z]. For a cube, 8 vertices are needed to represent it. For a six-quad face, each quad face contains two triangles, and 12 triangles are needed to represent it. The eight vertices are combined into 12 triangles through indexing. Each triangular face has a corresponding direction. This direction vector is the normal. The normal is also represented by [x, y, z]. UV is the texture coordinate. If we want to put a picture on this plane, we need texture coordinates to correspond to the position of the color point on the picture and the position of the triangular face. Generally, the UV coordinate is between 0-1. Greater or less than this value is equivalent to repeated values.

[0062] Among them, materials can be regarded as a set of parameters to guide the rendering engine on how to generate the rendering pipeline. The material parameters include rendering-related state values, macro definition values, rendering parameters, texture maps, and shader codes. The rendering-related state values ​​indicate the rules used by the GPU to perform internal calculations, such as whether the depth is turned on, whether to perform comparisons, etc. For how to calculate the fusion of translucent objects, macro definitions are not supported in the engine's internal interface. The engine provides macro definition keywords through a custom set of Shader code parsing tools, and reuses related codes similar to C++. These keywords can add pre-compiled logic to remove redundant Shader codes that do not belong to the rendering pipeline, leaving only usable codes.

[0063] For different rendering situations, you only need to change the macro definition. For example, for geometry, if you add vertex color for each vertex in the vertex attributes, you can use vertex coloring. In this way, there will be some functional conflicts between the colors in the material. According to the user's macro settings, you can dynamically use material colors and vertex colors.

[0064] Rendering parameters, including color, roughness, metalness, ambient light reflection intensity, and a series of texture maps. Texture maps can be understood as a picture. The texture is rendered onto the surface of the object through the UV coordinate value of the user geometry. The shader code is a kind of code executed inside the GPU. The engine uses the code with WGSL syntax. The shader code is divided into two parts, one is the vertex shader code, which is used to calculate vertices, and the other is the fragment shader, which is used to color the geometry. In the material, different codes are written according to different materials. The shader code is generally defined inside the engine according to the different materials. Users only need to pay attention to the material status value and rendering parameters.

[0065] This embodiment adopts the octree's level-by-level traversal and intersection judgment optimization to avoid repeated calculations and repeated rendering; at the same time, it adopts a one-time transmission of batch data to reduce the number of uploads and improve the efficiency of data interaction; in addition, by setting different Cache Keys, it allows resource reuse, increases resource utilization, and ensures minimal resource occupancy.

[0066] Example 3

[0067] This embodiment provides a real-time rendering method based on WASM, constructs a WASM memory module, sets a matrix buffer in the WASM memory module, and uniformly processes the buffer object matrix to obtain the MVP matrix. The structural diagram of the method is shown in FIG. Fig.10 As shown, including: Step 1: Obtain model elements of the object to be tested (3D model); model elements include basic structural information. Basic structural information includes: vertex information, texture coordinate information, transformation information and hierarchical relationship information.

[0068] Step 2: Start the engine and initialize the rendering environment. Asynchronously load the pre-set WASM memory module and dynamically add scripts to construct the basic structure information of the 3D object to be tested into an initial matrix. The constructed initial matrix is ​​stored in the matrix buffer of the WASM memory module.

[0069] When the reserved matrix buffer space is insufficient, continue to open up matrix space until the constructed initial matrix can be accommodated.

[0070] Furthermore, when the space of the matrix buffer required by the constructed initial matrix reaches the maximum limit of the browser, an error occurs and the browser issues an error alert.

[0071] The maximum limit of the browser is 2 28 -1 numeric value.

[0072] Step 3: Translate, rotate, scale, and confirm the parent-child relationship of the initial matrix in the matrix buffer to obtain the updated model matrix. Specifically include: Determine the translation vector, rotation axis, rotation angle and scaling factor when updating the matrix, and construct the translation matrix, rotation matrix and scaling matrix according to the translation vector, rotation axis, rotation angle and scaling factor; Enter the translation matrix, rotation matrix, and scaling matrix into the following formula to obtain the model matrix; Model matrix = translation matrix × rotation matrix × scaling matrix.

[0073] In a specific implementation process, the translation matrix is ​​used to change the position of an object, the rotation matrix is ​​used to change the direction of an object, and the scaling matrix is ​​used to change the size of an object.

[0074] The translation matrix, rotation matrix, and scaling matrix determine the specific position and posture of the object in the scene by transforming the local coordinates of the object into the world coordinate system.

[0075] According to the SetParent method, determine the parent-child relationship of different model matrices; When there is a parent-child relationship, the updated model matrix is: Updated parent matrix = child matrix × model matrix; The updated child matrix = parent matrix - 1 × model matrix.

[0076] The SetParent method is used to define the hierarchical relationship between objects. This method allows one object to be set as a child of another object, thereby affecting the position, rotation, and scale of the child object.

[0077] When an object becomes the child of another object, its position, rotation, and scale are calculated relative to its parent. This means that all transformations of the child object are performed based on the parent's world coordinate system.

[0078] That is, the child's local transform (relative to the parent's transform) is applied to the parent's world transform to get the child's world transform.

[0079] Use the updateAllChangedTranslate method to update multiple initial matrices at one time.

[0080] The updateAllChangedTranslate method can iterate over all objects marked as "to be updated" and update their world transformation matrices. In this way, multiple objects can be updated efficiently in batches, ensuring that all objects in the scene maintain the latest transformation state.

[0081] Step 4: Perform MVP matrix calculation on the updated model matrix to obtain the MVP matrix. Specifically, it includes: obtaining the view matrix and projection matrix of the object to be tested, and based on the view matrix and projection matrix of the object to be tested, obtaining the MVP matrix: MVP matrix = view matrix × projection matrix × updated model matrix.

[0082] The view matrix and projection matrix of the object to be measured are obtained in the following way: Determine the position, direction and field of view of the observation point corresponding to the 3D object to be measured, and use the lookAt function to dynamically construct the view matrix based on the position, direction and field of view of the observation point; In the specific implementation process, the basic structural information of the object to be measured is obtained through model editing tools, such as 3dMax, C4D or Blender. The field of view angle, aspect ratio, near clipping plane distance and far clipping plane distance of the observation point corresponding to the 3D object to be measured are obtained, and the projection matrix is ​​constructed using the gl-matrix function (i.e., tool library) according to the field of view angle, aspect ratio, near clipping plane distance and far clipping plane distance of the observation point.

[0083] The matrix buffer consists of: A first matrix buffer, used to store matrix data; The second matrix buffer is used to store the scaling, rotation, and translation data of each matrix; The third matrix buffer is used to store the continuous transformation data of the matrix; The fourth matrix buffer is used to store the status information of each matrix.

[0084] The state information of each matrix includes: one-time transformation state information, continuous transformation state information, parent matrix index information and depth order information.

[0085] See also Fig. 9 , each entry stored in the first matrix buffer is a 4×4×4 floating-point matrix representing the world space transformation of an object, which can be stored in the form of a data or structure array.

[0086] The second matrix buffer stores 3×3×4 scaling, rotation, and translation data.

[0087] The third matrix buffer stores continuous transformation data of a 1×3×4 matrix, such as displacement changes and rotation changes of each frame.

[0088] The one-time transformation state information stored in the fourth matrix buffer refers to whether a one-time transformation operation needs to be performed; the continuous transformation state information refers to whether there is an ongoing continuous transformation, such as animation; the parent matrix index information points to the index of the parent object matrix, which is used to build a hierarchical relationship; the depth order information refers to the specific rendering order, especially the rendering order of transparent objects.

[0089] In the specific implementation process of the method provided in this embodiment, Figure 8 As shown in the figure, the matrix buffer in the WASM memory module is a linear memory block, which is represented as an ArrayBuffer and can be accessed by WASM and other conventional development languages. Memory is allocated in pages, 1 page = 64KB.

[0090] Among them, other conventional development languages ​​directly read or write WASM's matrix buffer through the corresponding array type / pointer; WASM accesses data through memory pointers. Usually, functions in WASM need to pass in pointers to memory. WASM cannot directly process complex data structures, and data needs to be stored in the allocated memory space.

[0091] Through the above operations, WASM and other conventional development languages ​​can directly access and operate data, avoiding frequent data exchange between WASM and other conventional development languages ​​and greatly improving efficiency.

[0092] Example 4

[0093] This embodiment provides a method for optimizing a real-time rendering image. The method is implemented based on a real-time rendering system. The real-time rendering system includes a control center and at least one rendering device connected and scheduled by the control center. Generally, the system includes two modes, namely, an automatic switching mode and a manual switching mode, which can be selected manually. The flowchart of the method is shown in FIG. Fig.11 shown.

[0094] When the automatic switching mode is used and the control center receives a rendering request from the client, it will detect whether the rendering service has started, that is, it will schedule any rendering device to start the rendering service. If it has not started, it will select any rendering device to connect to the client for rendering service. During the service period, the control center will detect in real time whether the rendering service provided by the rendering device to the client is disconnected. Once disconnected, it will immediately switch to the service of another rendering device through the exclusive scheduling algorithm.

[0095] In the process of providing rendering services to the user end, if the rendering service is disconnected, it is certain that the real-time rendering screen cannot be provided. However, a healthy service does not mean that the screen the user sees is stable. It is possible that although the service is healthy, the continuously output screen is very stuck and the frame rate is also very low, which has seriously affected the normal use. At this time, the real-time rendering screen is also considered unstable. Therefore, the control center needs to detect the current user screen frame rate in real time when the rendering device provides rendering services to the user end. And use the pre-set screen health threshold to determine whether the screen rendered by the current rendering device is healthy.

[0096] The image health threshold includes the frame rate threshold and the frame rate abnormality threshold. The frame rate threshold is the lower limit of the unhealthy frame rate value.

[0097] The control center uses polling to detect the user's screen frame rate of the current screen in real time. When the frame rate is less than the frame rate threshold, it is recorded, that is, the number of frame rate anomalies increases by one. When the frame rate is greater than or equal to the frame rate threshold, it is also recorded, that is, the number of frame rate anomalies decreases by one. This is done to eliminate the impact of a single anomaly. When the rendering screen is running, there may indeed be a moment of freeze, which does not mean that the user really wants to switch services.

[0098] Determine whether the number of frame rate anomalies exceeds the frame rate anomaly threshold. If so, the image is considered to be continuously unhealthy and needs to be switched to another rendering device.

[0099] If there are some extreme cases, such as the services of two rendering devices are judged to be unhealthy, it is still necessary to select a relatively more stable one, so it is necessary to judge the frame rate stability of each rendering device.

[0100] To determine the frame rate stability of each Xu Ran device, it is necessary to obtain the average frame rate and frame rate standard deviation during rendering service. The average frame rate is based on the user screen frame rate obtained in real time during the rendering process, which is obtained through the following formula: ; Where N is the total duration of the time window / the amount of user screen frame rate collected, and Fi is the user screen frame rate; The frame rate standard deviation is based on the user screen frame rate obtained in real time during the rendering process and the average power, that is: ; Based on the preset formula three, the frame rate stability corresponding to each rendering device is obtained; the formula three is: ; Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, and the average frame rate and the frame rate standard deviation are both obtained based on the frame rates of all user images when the rendering device provides rendering services to the user terminal; is a preset very small constant (usually set to a positive number, such as 0.001, to avoid the denominator being 0); HAD is the historical average service duration corresponding to the rendering device, and the historical average service duration is obtained based on all historical service durations corresponding to the rendering device; MAX_DUR is the preset maximum operating time of the rendering device; k is a preset weight coefficient, which is used to adjust the impact of the crash ratio on the stability score and can be set according to requirements; C is the number of crashes of the rendering device; T is the total number of operations of the rendering device; The frame rate stability of the video is evaluated in part. The higher the average frame rate and the smaller its standard deviation (fluctuation), the higher the score, indicating that the video is smoother. The part is used to evaluate the duration of the video stream. The closer the ratio is to 1, the more stable the system is, the longer the usage time is, and the higher the score is. The −k×CR part represents the negative impact of each unit crash ratio on stability. The higher the crash ratio is, the lower the score is, because it indicates the instability of the system.

[0101] For the sake of convenience, let's take an example and get a set of collected frame rate data: [25, 30, 28, 27, 26], and calculate the frame rate stability of the rendering device; that is, AFR= (25+30+28+27+26) / 5 = 27.2FPS; ; Count the number of crashes 2 and the total number of runs 10, set k to 5, then -k×CR=-5×2 / 10=-1; The historical average service time is 5 days, and the designed maximum service time is 7 days. ; but, .

[0102] Among them, the crash ratio (CR) is the ratio of the number of crashes that occurred within a period of time to the total number of runs.

[0103] The Historical Average Duration (HAD) indicates the historical average running time of the video stream. A longer HAD generally indicates a more stable system.

[0104] Maximum Possible Duration (MAX_DUR) is the maximum operating duration in this article, which indicates the maximum duration or design duration that the system can run.

[0105] The exclusive scheduling algorithm collects all current service information through the client program deployed on each device. Initially, all services are in the scheduling state. Once a service is selected for scheduling, it will be marked as scheduled. The next time there is a scheduling demand, the service with the scheduling mark will be bypassed. Once the next service is scheduled, the previous service will automatically remove the scheduled mark.

[0106] The frame rate threshold can generally be changed dynamically according to the device status of the rendering device, that is, an initial default value of the frame rate threshold is set; the frame rate threshold is adjusted in real time according to the real-time acquisition of the current device status of the rendering device and the pre-set dynamic threshold adjustment strategy.

[0107] That is, when the GPU temperature is greater than 80 degrees, it starts to reach the alarm state, with 5 degrees as one level, and the threshold is increased by 1 for each increase; when it reaches 100 degrees, the threshold is increased by 10 for each increase (at this time, it is considered that there is a risk of hardware damage and it needs to be switched immediately); GPU usage rate: When the usage rate exceeds 80%, the alarm state begins to be reached. 5% is the first level, and the threshold increases by 1 for each additional level. Video memory usage: When the usage exceeds 80%, the alarm state begins to be reached. 5% is the first level, and the threshold increases by 1 for each additional level. CPU usage: When the CPU usage exceeds 80%, an alarm is triggered, with the threshold value + 0.5. Memory usage: When the disk usage exceeds 80%, an alarm is triggered, with the threshold value +0.5; Disk usage: When the disk usage exceeds 80%, an alarm is triggered, with the threshold value + 0.5.

[0108] For example, the default value of the frame rate threshold is set to 20FPS; At a certain moment, the monitoring data is as follows: GPU temperature 88 degrees, alarm, threshold increased by 2, to 22; GPU usage 90%, alarm, threshold increased by 2 (85%~90%), to 24; video memory usage 75%, no change; CPU usage 70%, no change; memory occupancy 85%, alarm, threshold increased by 0.5, to 24.5; disk occupancy 60%, no change; the final calculated threshold is 24.5 frames.

[0109] The abnormal frame rate threshold is divided into three levels, namely 3, 5 or 7, which can be set based on actual needs.

[0110] The above is the automatic switching mode, which will automatically judge according to the frame rate. However, there is also a certain degree of human subjectivity in whether the picture is stable. It is possible that the frame rate collected by the service just gets stuck near the threshold, sometimes causing stuttering and sometimes being smooth, resulting in the cumulative count of unhealthy frame rates never reaching the set threshold. However, at this time, if it is judged manually that a switch is necessary, a manual forced switching method is also provided.

[0111] Due to the particularity of the rendering service, if the same service runs continuously for 24 hours every day, the risk of instability may increase significantly. Currently, the time period of each day is divided into a high-frequency usage period and a low-frequency usage period. For example, 6:00~24:00 is the high-frequency usage period, and the night from 24:00~6:00 is the low-frequency usage period. During the low-frequency period, in hourly segments, one machine will be automatically selected to shut down the service, and after one hour, it will be automatically restarted. Then, in the next hour, other machines will be轮流 selected to be automatically shut down until all machines have completed restarting (if there are a large number of machines, the segment time can be adjusted to half an hour).

[0112] When there is only one rendering device, it is necessary to rely on the daemon program自带 by the service itself. The daemon program will monitor the process of the rendering program, and when the process crashes, the daemon program will restart the service.

[0113] When switching between multiple rendering devices, in order to ensure the consistency of the displayed content of the picture as much as possible (ensuring the consistency of the camera lens parameters), it is necessary to synchronize the camera lens information. The flow chart of its lens synchronization is as Fig.12 shown.

[0114] The currently actually scheduled service that is being rendered will save the lens parameter information of the camera lens once every 1 second. Once a service switch occurs, the newly scheduled service will read the current lens parameter information to avoid too large a difference in the picture between two different services. In theory, the last frame of the lens of the previous video stream service is the lens of the first frame of the new service after the switch.

[0115] The lens parameter information includes the lens position and the lens rotation parameters. Among them, the lens position is the X, Y, and Z coordinates of the lens, and the lens rendering parameters are the axis rotation parameters of X, Z, and Z. Roll represents the X axis (generally defaulting to zero without modification), Pitch represents the Y axis (-90 < Y < 90), and Yaw represents the Z axis (-180 to 180).

[0116] There will be a short black screen phenomenon between disconnecting the previous connection and completing the next connection and displaying the picture. During this period, to prevent the black screen, the previous screenshot will be used as the background for loading. When the new connection switch is completed, the screenshot will be hidden. Therefore, it is necessary to set the rendering picture to be captured at a frequency of once every 1 second.

[0117] An optimization method for real-time rendering of a screen is provided in an embodiment of the present invention. The method ensures the stability of the rendering screen by switching the rendering device in real time based on the user screen frame rate of the current screen during the rendering process. When all rendering devices are unable to provide high-quality rendering services, the rendering status of each rendering device is scored, and the most stable rendering device under the current conditions is selected to provide rendering services to the user terminal, thereby improving the user experience.

[0118] Example 5

[0119] This embodiment provides a method for digital twinning of urban infrastructure based on real-time rendering, including: the method is implemented based on the real-time rendering system pre-set in Example 3, including: Step 1: The control center builds a digital twin scene based on the 3D model and adds model attributes to the model creation. Specifically: In order to achieve connectivity from modeling software to the digital twin world and meet interactive application scenarios, general engines will directly use model files such as FBX, OBJ, etc. exported from modeling software and manually import them into the engine. However, some general 3D models (i.e., the first 3D model) required for building the scene in this embodiment are directly called from the object management pool and configured with information; while non-general 3D models that are not available in the object management pool still need to be imported from the modeling software.

[0120] After exporting the 3D model and the model elements corresponding to the 3D model from 3DSMax, the information will be aggregated into general model data that can be consumed by the control center through the aggregation program. After receiving this information, the control center will combine it with the 3D model and model elements already in the system, automatically combine the two and pass them to the engine.

[0121] After receiving these 3D models and model elements, the engine automatically processes the data, reads and converts them into model elements that can be consumed by the engine, reads the model files and automatically imports them into the engine, achieving full-link automated import.

[0122] When building a digital twin scene (i.e., target scene), target scene building requirements are needed. The target scene building requirements include the real geographic location information corresponding to the target scene, the virtual coordinate information of the scene reference point, and the model requirements for building the target scene.

[0123] The model requirements for building the target scene have been obtained through the above method. Generally, the model requirements for building the target scene include building models, pipe gallery models, equipment models, vegetation models, etc.

[0124] Place each 3D model at the corresponding position in the target scene according to the real geographic location information (GIS information) and set the attributes of each element to generate a digital twin scene.

[0125] Select the four corners of the target scene as reference points for surveying, and obtain the virtual coordinate information and real coordinate information of the reference points; Based on the virtual coordinate information and the real coordinate information of the reference point, a conversion relationship between the virtual coordinate and the real coordinate of the reference point is obtained, and the conversion relationship is applied to the 3D model to obtain a converted new model.

[0126] Check the geometric errors, duplicate elements, and attribute information of 3D models and model elements, identify whether there are self-intersecting and overlapping geometric errors in the 3D model, and use repair tools to automatically repair the erroneous parts and clear the intersecting and overlapping faces and edges.

[0127] Check the attribute information of the elements and check whether the content of the attribute fields is accurate. The attribute fields include information such as the date of construction, type of equipment, type of vegetation, etc. Check the attributes of other elements of the same category in batches and automatically correct them if errors are found.

[0128] Export 3D model elements such as materials, textures, and position information.

[0129] After importing into the engine, set the light source so that there is lighting in the scene, add a camera to the scene, and determine the viewing angle and observation point.

[0130] Add spatiotemporal element information to the generated target scene and define the mapping rules between the virtual scene and the real scene.

[0131] When a 3D model is clicked in the target scene, the control center sends information including ID, location information and other spatiotemporal elements and renders the click effect. After receiving the request, the control center obtains the corresponding information in the management background and returns it to the engine, which receives the information and renders the information POI page.

[0132] Step one, by setting up an object management pool, a large number of virtual device objects are efficiently managed in the engine. Through the object management pool, virtual devices (i.e., the first 3D model) can be dynamically managed, and some common 3D models (first 3D models) are pre-created, such as conventional sensors, cameras, valves, etc. When in use, they are directly called from the object management pool, thereby significantly improving resource utilization and high scene construction speed.

[0133] This is a mechanism for object management pools that can be widely used in digital twins. The following is an example of how this mechanism can be used to efficiently manage virtual device objects in a digital twin water plant. This mechanism reduces the overhead of frequent object creation and destruction by pre-creating and maintaining a set of reusable object pools, thereby improving the overall performance of digital twin technology. Specifically, it includes the following key components: Virtual device class, which defines the basic properties and usage methods of virtual devices; object management pool class: manages the creation, acquisition, and release of virtual device objects; digital twin environment: simulates device changes in the actual operation of the target scenario.

[0134] For example: Sensors, perform state enumeration, define different states of sensors (such as normal, warning, fault, etc.); provide methods for setting and obtaining sensor states.

[0135] Camera, defines different states of the camera (such as power on, power off, fault, etc.); provides methods for setting and obtaining the camera status.

[0136] Valve, defines different states of the valve (such as open, closed, faulty, etc.); provides methods for setting and obtaining the valve state.

[0137] Object management pool class, defines the object management pool class used to manage the creation, acquisition and release of virtual device objects; pre-create a certain number of objects and store them in the pool. Get an object, get an available object from the object management pool. Release an object, put the unused object back into the pool.

[0138] Specifically, an initial number of objects are created as needed, an available object is obtained from the object management pool, and if there is no available object, a new object is created. Objects that are no longer used are released and put back into the object management pool for subsequent use.

[0139] For example, a digital twin water plant environment is constructed to simulate the device status changes in the actual water plant environment. Device management uses the object management pool to manage virtual device objects. Status update updates the status of virtual devices according to the actual device status changes.

[0140] First, the object management pool is initialized, a certain number of virtual device objects are created as needed, and they are stored in the object management pool. For example, an object management pool with an initial size of 10 is created, and 10 sensor objects are created in advance.

[0141] When a new virtual device object is needed, an available object is obtained from the object management pool. If no object is available, a new object is created. For example, when a new sensor is needed, an available sensor object is obtained from the object management pool.

[0142] Update the state of the virtual device according to the state change of the actual device, for example, update the state of the sensor from "normal" to "warning". The state update is completed by setting the device state method.

[0143] When a virtual device object is no longer needed, it is put back into the object management pool. This avoids frequent creation and destruction of device objects and improves performance and resource utilization.

[0144] Step 2: When the control center receives a rendering request sent by the client, it detects whether there is any rendering device connected to the client to provide rendering services for the client; If it does not exist, the control center schedules any rendering device to connect to the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm.

[0145] Step 3: After any rendering device is connected to the user end, it receives the picture request from the user end in real time, and constructs the camera viewing cone of the target scene according to the picture request, that is, the picture to be rendered corresponding to the picture request; The picture request includes camera position information and viewpoint information in the target scene.

[0146] The rendering device transfers the camera view frustum to the octree container object according to the octree object corresponding to each 3D model in the target scene, performs step-by-step traversal and category distinction, obtains the corresponding transparent rendering object and opaque rendering object in the to-be-rendered picture, and stores the transparent rendering object and opaque rendering object in the rendering queue; The octree object corresponding to the 3D model is an object generated after the octree division of the 3D model.

[0147] The rendering device obtains the global data of the rendering queue and the model matrix of each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the MVP matrix corresponding to each rendering object in the rendering queue according to the global data of the rendering queue and the model matrix of each rendering object in the rendering queue; before this, the rendering device asynchronously loads the WASM memory module and dynamically adds scripts to construct the basic structure information of each rendering object in the rendering queue into an initial matrix, and stores the constructed initial matrix in the matrix buffer of the WASM memory module.

[0148] The rendering device obtains the global data of the rendering queue and the model matrix of each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the MVP matrix corresponding to each rendering object in the rendering queue according to the global data of the rendering queue and the model matrix of each rendering object in the rendering queue, specifically including: translating, rotating, scaling and confirming the parent-child relationship of all initial matrices in the matrix buffer to obtain the updated model matrix of each initial matrix; Perform MVP matrix calculation on the model matrix corresponding to each rendering object in the rendering queue to obtain the MVP matrix of each rendering object in the rendering queue; The matrix buffer comprises: a first matrix buffer for storing matrix data, a second matrix buffer for storing scaling, rotation and translation data of each matrix, a third matrix buffer for storing continuous transformation data of the matrix and a fourth matrix buffer for storing state information of each matrix; The state information of each matrix includes one-time transformation state information, continuous transformation state information, parent matrix index information, and depth order information.

[0149] The rendering device traverses each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the material of each rendering object in the rendering queue; and according to the material of each rendering object in the rendering queue, determines the rendering parameters, material name, material status value, texture map and macro definition value corresponding to each rendering object in the rendering queue.

[0150] The rendering device determines whether the rendering pipeline has been created based on the Cache. If the Cache exists, it is created and the rendering pipeline is directly called for rendering. If the Cache does not exist, the rendering pipeline is created and called to render each rendering object in the rendering queue to obtain the 2D image corresponding to the picture to be rendered. The Cache is set when creating the rendering pipeline; The model elements corresponding to each 3D model include materials and global data; The global data specifically includes the camera's projection matrix, the camera's view matrix, plane resolution, mouse position, timestamp, and exposure.

[0151] Step 4: After any rendering device is connected to the user terminal, the control center detects the user screen frame rate of the rendering device connected to the user terminal in real time during the rendering process, and determines whether the user screen frame rate of the current rendering device during the rendering process is abnormal based on a preset frame rate threshold, and records the number of abnormal user screen frame rates; During this period, when the user screen frame rate detected by the control center is less than the frame rate threshold, the number of frame rate anomalies increases by one; When the user screen frame rate detected by the control center is greater than or equal to the frame rate threshold, the number of frame rate anomalies decreases by one; The control center detects the number of abnormal frame rates of the user's screen in real time, and determines whether the screen rendered by the current rendering device is healthy based on the frame rate abnormality number threshold; When the number of abnormal frame rates of the user screen is greater than or equal to the frame rate abnormal number threshold, it is determined that the screen rendered by the current rendering device is unhealthy; If it is unhealthy, the control center detects whether there is a rendering device that is not connected to the user terminal to provide rendering services among all the rendering devices connected to it; If it exists, the control center switches another rendering device to connect with the user terminal to provide rendering services for the user terminal based on a preset exclusive scheduling algorithm; If it does not exist, the control center schedules the protection and monitoring of the current rendering device based on the pre-set daemon; The user screen frame rate is the number of 2D images played by the user terminal per unit time during the period when the rendering device provides rendering services to the user terminal; The picture health threshold includes a frame rate threshold and a frame rate abnormality times threshold; The daemon is used to restart the rendering service when the rendering device crashes during the process of providing rendering service to the client.

[0152] This embodiment provides a digital twin method based on real-time rendering, which reduces the overhead of frequent creation and destruction of objects by creating and maintaining a reusable object management pool, thereby improving the overall performance of the system; and automatically processes tasks such as coordinate conversion, reducing the time cost of digital twins. And by switching rendering devices to ensure stable output of rendering images, reduce the impact of image freezes or service crashes on user experience.

[0153] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0154] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0155] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0156] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0157] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A digital twin method based on real-time rendering, characterized in that: The method is implemented based on a pre-set control center and at least two rendering devices connected to the control center, and the method includes: S11, the control center builds the target scene according to the preset model file, the real geographical location information of the target scene and the virtual coordinate information of the scene reference point, and sends the target scene and the model elements corresponding to each 3D model in the target scene to all rendering devices; the model file includes at least one 3D model and the model elements corresponding to each 3D model; S12, when the control center receives a rendering request sent by the user terminal, based on a preset exclusive scheduling algorithm, schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal; S13, after any rendering device is connected to the user terminal, it receives the picture request of the user terminal in real time, obtains the corresponding picture to be rendered according to the picture request, and renders the picture to be rendered in real time based on the model elements corresponding to each 3D model in the target scene, obtains the 2D image corresponding to the picture to be rendered, and sends it to the user terminal; S14, when any rendering device provides rendering service for the user terminal, the control center detects in real time the user picture frame rate corresponding to the user picture provided by the rendering device, and determines whether the device status of the rendering device is healthy based on a preset picture health threshold; When it is unhealthy, the control center switches another rendering device to connect to the user end based on a pre-set exclusive scheduling algorithm to provide rendering services for the user end.

2. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S11 and before also include: S10, the control center establishes an object management pool according to the received at least one first 3D model; Then the S11 includes: S11-1. When the control center receives a target scene construction requirement sent by a user, the control center obtains at least one second 3D model sent by the user and a model element corresponding to each second 3D model, and according to the model requirement for building the target scene, obtains at least one first 3D model required for building the target scene from the object management pool, and configures the model element corresponding to each first 3D model; The target scene construction requirements include the real geographical location information corresponding to the target scene, the virtual coordinate information of the scene reference point and the model requirements for building the target scene; Each 3D model is a first 3D model or a second 3D model; S11-2, the control center obtains the real coordinate information corresponding to the scene reference point according to the virtual coordinate information of the scene reference point and the real geographical location information corresponding to the target scene; S11-3, the control center establishes a first coordinate transformation relationship according to the virtual coordinate information and the real coordinate information of the scene reference point; the first coordinate transformation relationship is a transformation relationship between the virtual coordinate and the real coordinate; S11-4, the control center configures the virtual coordinates of each first 3D model and the second 3D model according to the first coordinate conversion relationship and the real geographical location information of the target scene, and adjusts all the 3D models to corresponding positions; S11-5, the control center performs information diagnosis on each 3D model according to the pre-set repair tool and corrects the error part; S11-6, the control center configures the scene light source and the scene camera, completes the construction of the target scene, and obtains the model elements corresponding to each 3D model at this time; S11-7, the control center sends the built target scene and the model elements corresponding to each 3D model in the target scene to all rendering devices.

3. The digital twin method based on real-time rendering according to claim 2, characterized in that: The S11-1 also includes: The control center configures the spatiotemporal element information corresponding to each 3D model based on the real geographic location information corresponding to the target scene.

4. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S13 includes: S13-1, after any rendering device is connected to the user end, it receives the picture request from the user end in real time, and constructs the camera viewing cone of the target scene according to the picture request, that is, the picture to be rendered corresponding to the picture request; The picture request includes camera position information and viewpoint information in the target scene; S13-2, the rendering device transfers the camera view frustum to the octree container object according to the octree object corresponding to each 3D model in the target scene, performs step-by-step traversal and category classification, obtains the corresponding transparent rendering object and opaque rendering object in the to-be-rendered picture, and stores the transparent rendering object and opaque rendering object in the rendering queue; The octree object corresponding to the 3D model is the object generated after the octree division of the 3D model; S13-3, the rendering device obtains the global data of the rendering queue and the model matrix of each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the MVP matrix corresponding to each rendering object in the rendering queue according to the global data of the rendering queue and the model matrix of each rendering object in the rendering queue; S13-4, the rendering device traverses each rendering object in the rendering queue according to the model elements corresponding to each 3D model in the target scene, and obtains the material of each rendering object in the rendering queue; and determines the rendering parameters, material name, material state value, texture map and macro definition value corresponding to each rendering object in the rendering queue according to the material of each rendering object in the rendering queue; S13-5, the rendering device determines whether the rendering pipeline has been created according to the Cache. If the Cache exists, the creation is completed and the rendering pipeline is directly called for rendering. If the Cache does not exist, the rendering pipeline is created and called to render each rendering object in the rendering queue to obtain a 2D image corresponding to the picture to be rendered. The Cache is set when creating the rendering pipeline; The model elements corresponding to each 3D model include materials and global data; The global data specifically includes the camera's projection matrix, the camera's view matrix, plane resolution, mouse position, timestamp, and exposure.

5. The digital twin method based on real-time rendering according to claim 4 is characterized in that: The S13-2 includes: Traverse the octree container object step by step, starting from the root node, and check layer by layer whether the bounding box of the octree container object node intersects with the camera's viewing frustum; If they intersect, check whether the bounding box of the 3D object in the node intersects with the camera's viewing frustum; if they intersect, mark the rendered object and continue traversing; if they do not intersect, continue traversing downwards; If the bounding box of the octree container object node does not intersect with the camera viewing frustum, the node is directly skipped and the traversal continues downward; According to the renderOrder attribute of each rendering object in the rendering queue, the rendering object is divided into an opaque rendering object and a transparent rendering object; The renderOrder property of a transparent render object is greater than the set value, and the renderOrder property of an opaque render object is less than the set value.

6. The digital twin method based on real-time rendering according to claim 4, characterized in that: In S13-5, creating a rendering pipeline includes: Generate Shader Module CacheKey according to the name, macro variables and specific Shader code of the Shader code, and check whether there is a compiled Shader Module. If not, create a Shader Module. According to the Shader code, construct the slot information parameters of the texture of the sampler required by BindGroupLayout, and generate BindGroupLayout; Generate a Cache Key according to the material state value, macro definition value, and material name to obtain whether there is a reusable pipeline; if not, call the preset underlying interface to create a rendering pipeline, pass the created BindGroupLayout, the rendering state contained in the material, and the Shader Module, and output the rendering pipeline; According to the pre-set sorting algorithm, the rendering queue is sorted, the sorted rendering queue is obtained, and the rendering object is rendered according to the sorting; The pre-set sorting algorithm includes: Render opaque objects first, then transparent objects; For transparent rendering objects, the larger the renderOrder attribute, the earlier they are rendered. By default, if the value is 0, they are sorted according to the order in which they are added to the rendering queue. For opaque rendering objects, contrary to transparent rendering objects, larger values ​​are rendered in the front, and smaller values ​​are rendered in the back.

7. The digital twin method based on real-time rendering according to claim 4, characterized in that: The global data also includes basic structure information; The basic structure information includes vertex information, texture coordinate information, transformation information and hierarchical relationship information; After S11 and before S13-3, the following are included: The rendering device asynchronously loads the WASM memory module and dynamically adds scripts to construct the basic structure information of each rendering object in the rendering queue into an initial matrix, and stores the constructed initial matrix into the matrix buffer of the WASM memory module; The S13-3 includes: All initial matrices in the matrix buffer are translated, rotated, scaled, and their parent-child relationships are confirmed to obtain the updated model matrix for each initial matrix; Perform MVP matrix calculation on the model matrix corresponding to each rendering object in the rendering queue to obtain the MVP matrix of each rendering object in the rendering queue; The matrix buffer comprises: a first matrix buffer for storing matrix data, a second matrix buffer for storing scaling, rotation and translation data of each matrix, a third matrix buffer for storing continuous transformation data of the matrix and a fourth matrix buffer for storing state information of each matrix; The state information of each matrix includes one-time transformation state information, continuous transformation state information, parent matrix index information, and depth order information.

8. The digital twin method based on real-time rendering according to claim 7, characterized in that: The S13-3, translating, rotating, scaling, and confirming the parent-child relationship of all the initial matrices in the matrix buffer, to obtain the model matrix corresponding to each initial matrix, includes: Determine the translation vector, rotation axis, rotation angle and scaling factor when the matrix is ​​updated, and construct a translation matrix, a rotation matrix and a scaling matrix according to the translation vector, rotation axis, rotation angle and scaling factor; According to the translation matrix, rotation matrix and scaling matrix, and the preset formula 1, a model matrix is ​​obtained for updating the initial matrix; the formula 1 is: Model matrix = translation matrix × rotation matrix × scaling matrix.

9. The digital twin method based on real-time rendering according to claim 7, characterized in that: The S13-3 also includes: Determine the parent-child relationship of different model matrices according to the preset SetParent strategy; When there is a parent-child relationship, the updated model matrix is: Updated parent matrix = child matrix × model matrix; Updated child matrix = parent matrix -1 × model matrix.

10. The digital twin method based on real-time rendering according to claim 7, characterized in that: The step S13-3, performing MVP matrix calculation on the model matrix corresponding to each rendering object in the rendering queue to obtain the MVP matrix of each rendering object in the rendering queue, includes: According to the observation point position, direction and field of view angle of the camera's viewing cone corresponding to the image to be rendered, the view matrix corresponding to each rendering object in the rendering queue is dynamically constructed using the preset lookAt function; According to the field of view angle, aspect ratio, near clipping plane distance and far clipping plane distance of the camera viewing cone corresponding to the image to be rendered, the projection matrix corresponding to each rendering object in the rendering queue is constructed through a pre-set tool library; According to the model matrix, view matrix and projection matrix corresponding to each rendering object in the rendering queue and the preset formula 2, the MVP matrix corresponding to each rendering object in the rendering queue is obtained; the formula 2 is: MVP matrix = view matrix × projection matrix × updated model matrix.

11. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S12 includes: When the control center receives a rendering request sent by a client, it detects whether there is any rendering device connected to the client to provide rendering services for the client; If it does not exist, the control center schedules any rendering device to connect to the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm.

12. The digital twin method based on real-time rendering according to claim 1, characterized in that: The S14 includes: S14-1. After any rendering device is connected to the user terminal, the control center detects the user screen frame rate of the rendering device connected to the user terminal in real time during the rendering process, and determines whether the user screen frame rate of the current rendering device during the rendering process is abnormal based on a preset frame rate threshold, and records the number of abnormal user screen frame rates; During this period, when the user screen frame rate detected by the control center is less than the frame rate threshold, the number of frame rate anomalies increases by one; When the user screen frame rate detected by the control center is greater than or equal to the frame rate threshold, the number of frame rate anomalies decreases by one; S14-2, the control center detects the number of abnormal frame rates of the user screen in real time, and determines whether the screen rendered by the current rendering device is healthy based on the frame rate abnormality number threshold; When the number of abnormal frame rates of the user screen is greater than or equal to the frame rate abnormal number threshold, it is determined that the screen rendered by the current rendering device is unhealthy; S14-3, when it is unhealthy, the control center detects whether there is a rendering device that is not connected to the user terminal to provide rendering services among all the rendering devices connected to it; S14-4, if it exists, the control center switches another rendering device to connect with the user terminal to provide rendering service for the user terminal based on a preset exclusive scheduling algorithm, and repeats steps S14-1 to S14-4; If it does not exist, the control center schedules the protection and monitoring of the current rendering device based on the pre-set daemon; The user screen frame rate is the number of 2D images played by the user terminal per unit time during the period when the rendering device provides rendering services to the user terminal; The image health threshold includes the frame rate threshold and the frame rate abnormality threshold; The daemon is used to restart the rendering service when the rendering device crashes during the process of providing rendering service to the client.

13. The digital twin method based on real-time rendering according to claim 12, characterized in that: The method further comprises: When the control center schedules any rendering device to connect to the user terminal to provide rendering services for the user terminal, the total running times preset by the rendering device are updated, that is, the total running times of the rendering device are increased by one; When the control center determines that the image rendered by any rendering device is unhealthy, the crash count preset by the rendering device is updated, that is, the crash count of the rendering device is increased by one; When the current rendering device provides rendering services to the user end, the control center records the user screen frame rate of the rendering device in the rendering process in real time; When another rendering device is switched to connect with the client to provide rendering services for the client, the control center records the service duration of the rendering device before the switching to provide rendering services for the client, and stores the service duration as the historical service duration in a preset historical service duration database; Then the step S14-4 also includes: S14-5. When the control center determines that the images rendered by all the connected rendering devices are unhealthy, the control center obtains the frame rate stability of each rendering device based on the frame rates of all user images when each rendering device provides rendering services for the client, the total number of operations corresponding to each rendering device, the number of crashes corresponding to each rendering device, and all historical service durations corresponding to each rendering device in the historical service duration database, as well as the preset formula three; the formula three is: ； Wherein, S is the frame rate stability corresponding to the rendering device, AFR is the average frame rate when the rendering device provides rendering services to the user terminal, SD is the frame rate standard deviation when the rendering device provides rendering services to the user terminal, and the average frame rate and frame rate standard deviation are both obtained based on the frame rates of all user screens when the rendering device provides rendering services to the user terminal. is a preset minimum constant, HAD is the historical average service duration corresponding to the rendering device, the historical average service duration is obtained based on all historical service durations corresponding to the rendering device, MAX_DUR is the preset maximum operating duration of the rendering device, k is a preset weight coefficient, C is the number of crashes of the rendering device, and T is the total number of operations of the rendering device; Based on the frame rate stability corresponding to each rendering device, the control center switches to the rendering device with the highest frame rate stability to connect with the user terminal and provide rendering services to the user terminal.

14. The digital twin method based on real-time rendering according to claim 12, characterized in that: The S14-2 also includes: Acquire the device status of the current rendering device in real time, and adjust the picture health threshold in real time based on a preset dynamic threshold adjustment strategy and a preset picture health threshold; the picture health threshold includes a frame rate threshold; The device status includes: GPU temperature, GPU usage, video memory usage, CPU usage, memory usage and disk usage; The dynamic threshold adjustment strategy includes: When the GPU temperature is greater than 80 degrees and less than 100 degrees, the first frame rate threshold increases by 1 for every 5 degrees increase; when the GPU temperature is greater than 100 degrees, the first frame rate threshold increases by 10 for every 5 degrees increase; When the GPU usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the video memory usage is greater than 80%, the first frame rate threshold increases by 1 for every 5% increase; When the CPU usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the memory usage is greater than 80%, the first frame rate threshold increases by 0.5 for every 5% increase; When the disk usage is greater than 80%, the disk usage enters an alarm state. For every 5% increase, the first frame rate threshold increases by 0.

5.

15. The digital twin method based on real-time rendering according to claim 12, characterized in that: The S14-2 also includes: The control center monitors and synchronizes the updated data of the rendering device connected to the user end in real time during the rendering process; The S14-4 also includes: When another rendering device is switched to connect with the client to provide rendering service for the client, the control center synchronizes the latest updated data of the rendering device before switching to the rendering device after switching, so as to synchronize the data of the rendering device after switching.

16. The digital twin method based on real-time rendering according to claim 15, characterized in that: The S14-4 also includes: when switching another rendering device to connect to the user terminal to provide rendering services for the user terminal, the control center uses the most recent cut-off frequency of the rendering device before the switching as the picture provided to the user terminal until the switching is completed, and hides the cut-off frequency.

Images