Distributed rendering method, device and system for digital twin model in production scenario

Abstract

The application relates to a production scene digital twin model distributed rendering method, device and system, and relates to the technical field of scene rendering. The method comprises the following steps: dividing a screen space according to pictures corresponding to rendering ends; obtaining model data of moving entities in a production workshop; determining entity ranges and entity centers according to the model data and constructing bounding box data; screening moving entities needing synchronization according to the bounding box data; determining projection matrices and observation matrices of the moving entities needing synchronization; constructing view cone information according to the projection matrices and the observation matrices; obtaining motion state information of the moving entities needing synchronization; and rendering the moving entities needing synchronization in corresponding screen spaces according to the view cone information and the motion state information. The application can satisfy real-time and accurate synchronization of dynamic models in a complex production environment, significantly improve the rendering efficiency of a digital twin workshop, and output high-resolution pictures under the condition of ensuring smoothness.

Description

Technical Field

[0001] This invention belongs to the field of scene rendering technology, and in particular relates to a distributed rendering method, device and system for digital twin models in production scenarios. Background Technology

[0002] Digital twins are a key enabling technology for intelligent manufacturing. They create virtual models of physical entities digitally, leveraging technologies such as digital analysis, multi-domain simulation, and comprehensive monitoring to perform multi-dimensional, spatiotemporal image mapping of the entire lifecycle of physical objects. This adds or expands new functions for physical entities, deepening the integration and application of information technology in manufacturing. Digital twin models in complex production scenarios are diverse and numerous, requiring real-time rendering of a vast number of virtual models. These scenarios include various operating equipment responsible for multiple process actions and clamping types, each containing numerous intricate parts. Furthermore, the scenarios need to incorporate production and process logic data, resulting in many models in motion, a large amount of action logic information, and a large number of dynamic models. This leads to enormous rendering and logical computation overhead, placing a significant burden on computer hardware. These issues make it difficult to provide real-time feedback of system calculation results, greatly affecting the rendering effect of the scenario. To address the above issues, this invention proposes a distributed rendering method, device, and system for digital twin models in production scenarios. This method can meet the real-time and accurate synchronization requirements of dynamic models in complex production environments, significantly improve the rendering efficiency of digital twin workshops, and output high-resolution images while ensuring smoothness. Summary of the Invention

[0003] The purpose of this invention is to provide a distributed rendering method, device and system for digital twin models in production scenarios, which can meet the requirements of real-time and accurate synchronization of dynamic models in complex production environments, significantly improve the rendering efficiency of digital twin workshops, and output high-resolution images while ensuring smoothness.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A distributed rendering method for digital twin models in a production scenario includes:

[0006] Divide the screen space according to the image corresponding to the rendering end;

[0007] Obtain model data of moving entities in the production workshop;

[0008] Determine the entity extent and entity center based on the model data;

[0009] Construct bounding box data based on the entity's extent and center;

[0010] Filter the moving entities that need to be synchronized based on the bounding box data;

[0011] Determine the projection matrix and observation matrix of the moving entities that need to be synchronized;

[0012] Construct the view frustum information based on the projection matrix and the observation matrix;

[0013] Obtain the motion state information of the motion entities that need to be synchronized;

[0014] The motion entities that need to be synchronized are rendered in the corresponding screen space based on the view frustum information and motion state information.

[0015] Optionally, after the step of "dividing the screen space according to the image corresponding to the rendering end" and before the step of "obtaining the model data of moving entities in the production workshop", the method further includes:

[0016] Number the moving entities in the production workshop.

[0017] Optional, also includes:

[0018] The relationship between the moving entity and the screen space is calculated based on the projection matrix and the observation matrix.

[0019] Optionally, the step of filtering the moving entities that need to be synchronized based on the bounding box data specifically includes:

[0020] The eight vertices of the bounding box are determined based on the bounding box data;

[0021] The homogeneous coordinates of the corresponding point in screen space are determined based on the eight vertices of the bounding box.

[0022] Determine whether at least one of the homogeneous coordinates satisfies a preset judgment condition;

[0023] If so, the motion entity corresponding to the bounding box data needs to be synchronized.

[0024] A distributed rendering device for digital twin models in a production scenario includes: a control terminal, a transmission terminal, a rendering terminal, and a display terminal;

[0025] The control terminal, rendering terminal, and display terminal are connected in sequence;

[0026] The control terminal is used to determine and distribute rendering tasks, and transmit the rendering tasks to the rendering terminal through the transmission terminal;

[0027] The rendering end is used to parse the rendering task and transmit the rendered image to the display end.

[0028] Optionally, the rendering task is in the format of a data packet, which includes a checksum area, a fixed data area, and a variable data area.

[0029] The verification data area includes a verification identifier and a frame number; the fixed data area includes the viewpoint observation matrix and projection matrix for each frame; and the variable data area includes state data types and state data.

[0030] Optionally, the rendering end is used to parse the rendering task, specifically including:

[0031] Verify that the identification is correct; if incorrect, discard the data packet corresponding to the rendering task.

[0032] Check if the frame number matches; if not, discard the data packet corresponding to the rendering task.

[0033] Optionally, the rendering end is also used to determine whether the state data type is heartbeat data.

[0034] Optionally, if the number of motion entities that need to be synchronized is zero, the data area of ​​the variable data area is empty.

[0035] A distributed rendering system for digital twin models in complex production scenarios includes:

[0036] The screen partitioning module is used to divide the screen space according to the image corresponding to the rendering end;

[0037] The model data acquisition module is used to acquire model data of moving entities in the production workshop;

[0038] The entity extent and entity center determination module is used to determine the entity extent and entity center based on the model data.

[0039] A bounding box data construction module is used to construct bounding box data based on the entity extent and entity center.

[0040] The motion entity filtering module is used to filter motion entities that need to be synchronized based on the bounding box data;

[0041] The projection matrix and observation matrix determination module is used to determine the projection matrix and observation matrix of the moving entity that needs to be synchronized;

[0042] A view frustum construction module is used to construct view frustum information based on the projection matrix and the observation matrix;

[0043] The motion state acquisition module is used to acquire the motion state information of the motion entity that needs to be synchronized;

[0044] The rendering module is used to render the motion entity that needs to be synchronized in the corresponding screen space according to the view frustum information and motion state information.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention divides scene data into logical data and real-time rendering data. The logical data is distributed at the control end, and the rendering data is distributed at the rendering end, realizing multi-machine parallel rendering. Simultaneously, taking the synchronization of dynamic model rendering at the rendering end as a starting point, and combining it with the designed GPU multi-threading mechanism, it achieves synchronized judgment and calculation of logical data and moving entities. Then, it constructs a synchronization data package including workshop state data, observation and projection matrices, etc., and combines it with a master-slave synchronized data transmission and parsing mechanism to achieve fast and accurate synchronization of the motion model. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the distributed rendering method for digital twin models in a production scenario according to the present invention;

[0049] Figure 2 This is a diagram of the view frustum projection transformation of the present invention;

[0050] Figure 3 This is a view space diagram illustrating the single-node rendering specification of the present invention;

[0051] Figure 4 This is a schematic diagram of the view space for the distributed rendering four-channel specification of the present invention;

[0052] Figure 5 This is a schematic diagram of the synchronization data packet format of the present invention;

[0053] Figure 6 This is a flowchart illustrating the process of transferring data to video memory according to the present invention;

[0054] Figure 7 This is a schematic diagram of the multi-threaded fast calculation process of the present invention;

[0055] Figure 8 This is a flowchart of the synchronization judgment process based on bounding boxes in this invention;

[0056] Figure 9 This is a schematic diagram of the synchronous data transmission and parsing process of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The purpose of this invention is to provide a distributed rendering method, device and system for digital twin models in production scenarios, which can meet the requirements of real-time and accurate synchronization of dynamic models in complex production environments, significantly improve the rendering efficiency of digital twin workshops, and output high-resolution images while ensuring smoothness.

[0059] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] The overall method scheme of the present invention is as follows: Figure 1 As shown, the specific solution is as follows:

[0061] Step 1: Building a distributed hardware architecture. Using a master-slave structure, it mainly includes a control terminal, transport layer, rendering terminal, and display terminal. The specific functions of each component are as follows:

[0062] (1) Control end: It holds the logical data in the scene, including dynamic data such as production process, steps and technology of the production line, and is responsible for collision detection and dynamic logic calculation of the scene. At the same time, it performs rendering task data calculation of the observation matrix and its projection matrix of each viewpoint, transmits the logical calculation results and rendering tasks to the rendering end through the network, controls the logic synchronization of the rendering end, and manages and distributes rendering tasks.

[0063] (2) Rendering end: It holds the rendering data in the scene, is responsible for parsing the synchronization data, and completes the rendering of the model through rendering pipeline processes such as view frustum culling, vertex shading, and geometry shading, and transmits the rendered screen to the display end.

[0064] (3) Transport layer: responsible for building the local area network, connecting the control end and the rendering end, and transmitting rendering tasks and logical information.

[0065] (4) Display end: responsible for displaying the splicing and rendering results.

[0066] Step 2: Screen Partitioning at the Rendering End. Analysis of the 3D model rendering pipeline reveals that the 3D model in the scene undergoes a series of coordinate matrix transformations to be drawn onto a 2D plane. During this process, model vertices are transformed from model space to clip space through viewpoint and projection matrices, and then normalized to device coordinates (NDC) and mapped to screen space. In a distributed rendering system, a screen space is divided according to the image corresponding to each rendering end. Multiple rendering ends collaboratively draw the same frame. By scaling, translating, and rotating the projection matrix of the original single-machine system, the rendering task allocation of the distributed rendering system is quickly completed. This paper proposes a multi-screen rendering task allocation method based on matrix transformations of the projection matrix.

[0067] In the rendering engine, NDC is a cube with dimensions ranging from [-1, 1], specifically (-1, -1, -1) to (1, 1, 1). For example... Figure 2 As shown, a point A(x,y,z) in the view frustum's observation space is projected onto the near clipping plane z=N, resulting in a projection point A1(x1,y1,z), where x1=xN / z, y1=yN / z, and z=N. Then, A1 is normalized and mapped to the normal view space, yielding a point A2(x',y',z'), where the lower left corner of the near clipping plane is (L,B,N) and the upper right corner is (R,T,N). Based on the above projection principle, the projection matrix is ​​as follows:

[0068]

[0069] like Figure 3 and Figure 4 As shown, assuming the original single-machine system is numbered Node0, the rendering endpoints are numbered according to their rendering space location. Taking four rendering endpoints as an example, they are numbered Node1, Node2, Node3, and Node4. Node1 renders the model located in the upper left corner of the original screen, Node2 renders the model located in the upper right corner of the original screen, Node3 renders the model located in the lower left corner of the original screen, and Node4 renders the model located in the lower right corner of the original screen. By comparing the view frustum canonical view space of the original single-machine system and the distributed rendering system, it can be seen that the model in the upper left corner of the original screen, after projection transformation, has a view space range of (-1, 0, -1) to (0, 1, 1) in the original system. Only by transforming to NDC coordinates can this rendering task be successfully assigned to Node1. Based on the original p matrix, the projection matrices corresponding to the four nodes are as follows: ① Node1, R = 0, B = 0; ② Node2, B = 0, L = 0; ③ Node3, R = 0, T = 0; ④ Node4, T = 0, L = 0.

[0070] Step 3: Data Structure Design. The data packet format designed in this paper is as follows: Figure 5 As shown, the system mainly consists of a verification data area, a fixed data area, and a variable data area. The verification data area includes a verification identifier and a frame number. The verification identifier is used to test the validity of the data packet, and the frame number corresponds to the frame sequence number of the current scene; all rendering ends must have the same frame sequence number simultaneously. The fixed data area refers to the rendering task data, including the viewpoint observation matrix and projection matrix for each frame, mainly used for synchronizing the rendering tasks of the rendering ends. The variable data area consists of state data types and state data. If the number of moving entities to be synchronized is zero, this data area is empty; if there are twin dynamic entities to be synchronized, the state data type and corresponding state data are added to the variable data area. The control end encapsulates the data into data packets and sends them to multiple rendering ends, which then perform subsequent data parsing and execution.

[0071] Table 1 shows the status data and its corresponding associated events. The status data of the synchronization object is divided into four states: transport state (T) when it is in the buffer conveyor belt position, transition state (S) when it is about to enter a processing unit, processing state (W) when it is in a processing unit, and completion state (D) when it has completed processing and left the processing unit. E represents the related events triggered by different states, as set as follows:

[0072] Table 1 Status Data and Related Events

[0073]

[0074] Where n is the synchronization object number, i represents the processing unit, and j is the number of processes contained in the processing unit. Table 2 provides a detailed semantic explanation of Table 1.

[0075] Table 2 Explanation of Status Data and Related Events

[0076]

[0077]

[0078] Step 4: Data Acquisition and Transmission. All twin motion entities in the scene are numbered, and their model data is acquired. The center and range of each twin motion entity are used to construct a bounding box data structure. Then, the projection matrix P and the observation matrix V are acquired to form the view frustum information. Models with dynamic logical attributes will have their motion trajectories pre-set in the CPU's memory; this part constitutes the motion state information. Table 3 shows the cache data structure. A cache for the above three types of data is pre-allocated in the video memory, and a result buffer is also allocated to store the results of logical calculations. Figure 6The flowchart shown is a process of transferring data to video memory. The model data is preprocessed and variable space is declared in the buffer. Then, logical information such as the bounding box and view frustum is obtained and passed to video memory through the buffer array.

[0079] Table 3 Cache Area Data Structure

[0080]

[0081] Step 5: Entity space computation based on multithreading, such as... Figure 7 The diagram illustrates the multi-threaded fast computation process. The CPU loads model data into memory, acquires bounding box and other logical data, and transfers this logical data to the video memory buffer. The GPU then starts multiple sub-threads that read the logical data from the buffer, perform matrix transformations on moving entities using appropriate matrix operations, and simultaneously determine whether the moving entities require synchronization, filtering out those that do not. Finally, the computation results are stored in the buffer, awaiting delivery to the rendering end, completing the multi-threaded fast computation.

[0082] Figure 8 As shown, synchronization based on bounding boxes is performed within the sub-thread. The eight vertices of the bounding box are calculated using its center point and range. The homogeneous coordinates P of the corresponding screen-space points are then calculated and normalized using state and projection matrices. These homogeneous coordinates are compared with the decision criteria. If at least one vertex of the bounding box satisfies the criteria, the corresponding moving entity needs synchronization. If none of the vertices satisfy the criteria, the moving entity corresponding to that bounding box is meaningless for rendering and does not require synchronization.

[0083] Step 6: Synchronize data transmission and parsing.

[0084] Figure 9 To synchronize data transmission and parsing flowchart, the control end calculates and packages synchronization data packets, writes them to the data buffer, and then sends the data packets through the network layer. The rendering end receives the synchronization data and then performs the following operations: (1) Check if the verification identifier is correct. If incorrect, discard the data packet. (2) Check if the frame number matches. If not, discard the data packet. (3) Read the rendering task and obtain the view frustum viewpoint position and projection matrix. (4) Determine the status data type. If it is heartbeat data, all the data that needs to be synchronized in the current time frame has been sent and received. The rendering end waits for the control end to send the frame buffer exchange instruction. If the data type is the status information of the synchronization object, the rendering end parses the associated events according to Table 1 and updates the status of the synchronization object according to the events. At the same time, it sets the camera projection matrix and viewpoint angle in the scene according to the rendering task data to complete the rendering task of the current frame. Finally, it waits for all rendering ends to complete the drawing, transmits the frame buffer instruction, and makes all rendering ends display the screen synchronously.

[0085] It should be noted that the above-mentioned solution of the present invention is a solution for distributed rendering that uses synchronous model state. Distributed rendering can also be performed at the model rendering pipeline level. For example, during the computer rendering pipeline process, rendering data such as vertices and triangles can be divided and then transmitted to several computers for rendering calculation. The calculation results can then be summarized, which can also provide a reference for distributed rendering.

[0086] Based on the above solution, the present invention also provides a distributed rendering system for digital twin models in complex production scenarios, comprising:

[0087] The screen partitioning module is used to divide the screen space according to the image corresponding to the rendering end.

[0088] The model data acquisition module is used to acquire model data of moving entities in the production workshop.

[0089] The entity extent and entity center determination module is used to determine the entity extent and entity center based on the model data.

[0090] A bounding box data construction module is used to construct bounding box data based on the entity's extent and center.

[0091] The motion entity filtering module is used to filter motion entities that need to be synchronized based on the bounding box data.

[0092] The projection matrix and observation matrix determination module is used to determine the projection matrix and observation matrix of the moving entity that needs to be synchronized.

[0093] A view frustum construction module is used to construct view frustum information based on the projection matrix and the observation matrix.

[0094] The motion state acquisition module is used to acquire the motion state information of the motion entity that needs to be synchronized.

[0095] The rendering module is used to render the motion entity that needs to be synchronized in the corresponding screen space according to the view frustum information and motion state information.

[0096] The present invention also discloses the following technical effects:

[0097] 1. The distributed rendering digital twin workshop of this invention can achieve a seamless splicing effect at the screen splicing points corresponding to multiple nodes, especially when dynamic models with dynamic attributes are located at the screen splicing points, without any screen tearing or distortion.

[0098] 2. In the distributed rendering digital twin workshop of this invention, multiple rendering ends can correctly receive the rendering task of the same frame under the control of the control end.

[0099] 3. The logic of each rendering end in this invention is consistent with that of the control end, and no logical confusion occurs.

[0100] 4. The multi-threaded computing technology of this invention accelerates the main-end logic calculation, improves the data synchronization transmission rate, and the synchronization determination technology reduces the scale of rendering data and speeds up rendering calculation.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0102] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A distributed rendering method for digital twin models in production scenarios, characterized in that, include: Divide the screen space according to the image corresponding to the rendering end; Obtain model data of moving entities in the production workshop; Determine the entity extent and entity center based on the model data; Construct bounding box data based on the entity's extent and center; The bounding box is a cuboid, and the bounding box data is stored in a dedicated cache of type Float. The process of filtering motion entities that need to be synchronized based on the bounding box data includes: determining the 8 vertices of the bounding box based on the bounding box data; determining the homogeneous coordinates of the corresponding points in screen space based on the 8 vertices of the bounding box; determining whether at least one of the homogeneous coordinates satisfies a preset judgment condition; if so, the motion entity corresponding to the bounding box data needs to be synchronized. Determine the projection matrix and observation matrix of the viewpoint for each frame of the moving entity that needs to be synchronized; Based on the screen space division results, the projection matrix is ​​scaled, translated, and rotated to obtain the adjusted projection matrix; Construct the view frustum information based on the adjusted projection matrix and observation matrix; Obtain the motion state information of the motion entities that need to be synchronized; The motion entities that need to be synchronized are rendered in the corresponding screen space based on the view frustum information and motion state information.

2. The distributed rendering method for digital twin models in production scenarios according to claim 1, characterized in that, After the step of "dividing the screen space according to the image corresponding to the rendering terminal" and before the step of "obtaining the model data of moving entities in the production workshop", the following steps are also included: Number the moving entities in the production workshop.

3. The distributed rendering method for digital twin models in production scenarios according to claim 1, characterized in that, Also includes: The relationship between the moving entity and the screen space is calculated based on the projection matrix and the observation matrix.

4. A distributed rendering device for digital twin models in a production scenario, characterized in that, The distributed rendering method for digital twin models in production scenarios as described in any one of claims 1-3 includes: a control terminal, a transmission terminal, a rendering terminal, and a display terminal; The control terminal, rendering terminal, and display terminal are connected in sequence; The control terminal is used to determine and distribute rendering tasks, and transmit the rendering tasks to the rendering terminal through the transmission terminal; The rendering end is used to parse the rendering task and transmit the rendered image to the display end.

5. The distributed rendering device for digital twin models in production scenarios according to claim 4, characterized in that, The rendering task is in the format of a data packet, which includes a checksum area, a fixed data area, and a variable data area. The verification data area includes a verification identifier and a frame number; the fixed data area includes the viewpoint observation matrix and projection matrix for each frame; and the variable data area includes state data types and state data.

6. The distributed rendering device for digital twin models in production scenarios according to claim 5, characterized in that, The rendering end is used to parse the rendering task, specifically including: Verify that the identification is correct; if incorrect, discard the data packet corresponding to the rendering task. Check if the frame number matches; if not, discard the data packet corresponding to the rendering task.

7. The distributed rendering device for digital twin models in production scenarios according to claim 5, characterized in that, The rendering end is also used to determine whether the status data type is heartbeat data.

8. The distributed rendering device for digital twin models in production scenarios according to claim 5, characterized in that, If the number of motion entities that need to be synchronized is zero, then the data area of ​​the variable data area is empty.

9. A distributed rendering system for digital twin models in complex production scenarios, characterized in that, include: The screen partitioning module is used to divide the screen space according to the image corresponding to the rendering end; The model data acquisition module is used to acquire model data of moving entities in the production workshop; The entity extent and entity center determination module is used to determine the entity extent and entity center based on the model data. A bounding box data construction module is used to construct bounding box data based on the entity's extent and center; the bounding box is a cuboid, and the bounding box data is stored in a dedicated cache of type Float. The motion entity filtering module is used to filter motion entities that need to be synchronized based on the bounding box data. Specifically, it includes: determining the 8 vertices of the bounding box based on the bounding box data; determining the homogeneous coordinates of the corresponding points in screen space based on the 8 vertices of the bounding box; determining whether at least one of the homogeneous coordinates satisfies a preset judgment condition; if so, the motion entity corresponding to the bounding box data needs to be synchronized. The projection matrix and observation matrix determination module is used to determine the projection matrix and observation matrix of the viewpoint of the moving entity that needs to be synchronized for each frame; and to adjust the projection matrix by scaling, translation, and rotation according to the screen space division result to obtain the adjusted projection matrix. A view frustum construction module is used to construct view frustum information based on the adjusted projection matrix and the observation matrix; The motion state acquisition module is used to acquire the motion state information of the motion entity that needs to be synchronized; The rendering module is used to render the motion entity that needs to be synchronized in the corresponding screen space according to the view frustum information and motion state information.

Images