A method for joint parameter learning of digital twin model mechanism and appearance

Through the digital twin model mechanism and appearance joint parameter learning method, the mechanism changes of the equipment and the three-dimensional structure are connected, and the problem of frequent modeling of equipment in the existing technology is solved, and the improvement of automated modeling and data analysis is achieved.

CN116467948BActive Publication Date: 2024-11-19JIANGXI KMAX IND CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310438448.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-20
Publication Date
2024-11-19
Estimated Expiration
2043-04-20

AI Technical Summary

Technical Problem

The existing digital twin technology requires frequent three-dimensional modeling under different states of equipment, which is inefficient and has a lot of repetitive work, and lacks the connection between mechanism changes and three-dimensional model structure.

Method used

Using the digital twin model mechanism and appearance joint parameter learning method, through historical data and current running data, multi-layer perceptron and Transformer neural network, we predict the future state and three-dimensional model structure changes of the device virtual model, and connect the mechanism changes with the three-dimensional structure. Avoid frequent modeling.

Benefits of technology

The automated construction of equipment models and the enhanced connection between mechanism data and three-dimensional structures are realized, modeling efficiency is improved, repetitive labor is reduced, and data analysis and prediction capabilities are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116467948B_ABST
    Figure CN116467948B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for learning joint parameters of a digital twin model mechanism and appearance, including: obtaining a real image and reconstructing a three-dimensional scene with it, using a multi-layer perceptron to render the three-dimensional scene to obtain a synthetic image, calculating a three-dimensional representation of the scene when the gap between the synthetic image and the real image is the smallest, and generating a current state three-dimensional grid according to the three-dimensional representation of the scene; obtaining the mechanism data of the target device in the current state and learning its mechanism potential representation to obtain a mechanism data sequence of the target device in the next state; sorting the vertices of the current state three-dimensional grid and inputting them into a neural network model after preprocessing to obtain the vertex distribution of the next state three-dimensional grid; generating the current state grid surface distribution according to the target device mechanism potential representation and the current state three-dimensional grid, sorting the current state grid surface distribution and inputting them into the neural network model after preprocessing to obtain the next state grid surface distribution. The present invention avoids frequent three-dimensional modeling for different states of the device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of digital twins, and in particular to a method for learning joint parameters of a digital twin model mechanism and appearance. Background Art

[0002] Digital twins are virtual models that are designed to accurately reflect physical objects. They obtain data related to physical objects through sensors, and combine data, algorithms, and decision analysis to simulate, that is, virtual mapping of physical objects. Through digital twins, problems can be discovered before they occur, changes in physical objects in virtual models can be monitored, multi-dimensional data complex processing and abnormal analysis based on artificial intelligence can be performed, potential risks can be predicted, and relevant equipment can be planned or maintained reasonably and effectively.

[0003] Although digital twin technology has developed rapidly in recent years, most digital twin scenarios provide CAD design drawings to manually design three-dimensional models. In addition, the same device model also needs to provide one or more three-dimensional models in different states, which requires frequent modeling, low efficiency, and a lot of repetitive work. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a digital twin model mechanism and appearance joint parameter learning method, which is based on machine learning and uses historical data and current operation data to predict the mechanism changes in the future state of the equipment virtual model and the changes in the equipment three-dimensional model structure, thereby avoiding frequent three-dimensional modeling for different states of the equipment, and at the same time connecting the mechanism changes of the model with the three-dimensional model structure.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A method for learning joint parameters of a digital twin model mechanism and appearance includes the following steps:

[0007] S1) obtaining real images of the same scene of a target device under different viewing angles, reconstructing the three-dimensional scene according to the real images, rendering the three-dimensional scene using a multi-layer perceptron to obtain a synthetic image, calculating a three-dimensional representation of the scene when the difference between the synthetic image and the real image is the smallest, and generating a current state three-dimensional grid according to the three-dimensional representation of the scene;

[0008] S2) Acquire the mechanism data of the target device in the current state and learn its potential mechanism representation to obtain the mechanism data sequence of the target device in the next state; sort the vertices of the current state three-dimensional mesh and input them into the Transformer neural network model after preprocessing to obtain the vertex distribution of the next state three-dimensional mesh; generate the current state mesh surface distribution according to the current state three-dimensional mesh of the target device, sort the current state mesh surface distribution and input them into the Transformer neural network model after preprocessing and the potential representation of the mechanism data sequence of the target device in the current state to obtain the next state mesh surface distribution.

[0009] Furthermore, when reconstructing the three-dimensional scene according to the real image in step S1, it includes: generating a ray passing through each pixel in the real image, the origin of the ray is the position of the corresponding pixel in the image plane, and the direction of the ray is the direction of the corresponding pixel toward the camera aperture.

[0010] Furthermore, when obtaining the real images of the same scene of the target device under different viewing angles in step S1, the corresponding camera-to-world matrix is ​​also obtained. The direction expression of the ray is as follows:

[0011]

[0012] Among them, C ex -1 is the camera-to-world matrix corresponding to the real image, R ′ ex It is C ex -1 The upper 3×3 matrix, They are homogeneous representations of points in the camera coordinate frame and the world coordinate frame, respectively.

[0013] Furthermore, when a multilayer perceptron is used to render the three-dimensional scene to obtain a synthetic image in step S1, the process includes: extracting a three-dimensional sample point from each ray, inputting the spatial coordinates and direction of each three-dimensional sample point into the multilayer perceptron to obtain the color and volume density of each three-dimensional sample point, and integrating the color and volume density of each three-dimensional sample point to obtain a synthetic image.

[0014] Furthermore, the three-dimensional sample point expression is as follows:

[0015]

[0016] Among them, U refers to uniform sampling, t f and t n are the farthest and nearest points on the current ray, respectively. N represents the number of parts into which the current ray is divided. i Represents the i-th 3D sample point of the current ray.

[0017] Furthermore, the expression for integrating the color and volume density of each 3D sample point is as follows:

[0018]

[0019] Among them, t f and t n are the farthest and nearest points on the current ray, σ(r(t)) is the volume density of the current ray at the three-dimensional sample point t, c(r(t),) is the color of the current ray at the three-dimensional sample point t, Indicates the degree to which light penetrates the three-dimensional space to reach the three-dimensional sample point.

[0020] Furthermore, when calculating the three-dimensional representation of the scene when the gap between the synthetic image and the real image is minimized in step S1, it includes: minimizing the gap between the synthetic image and the real image by photometric consistency loss to optimize the three-dimensional representation of the scene;

[0021] When generating the current state three-dimensional mesh according to the three-dimensional representation of the scene in step S1, it includes: generating the current state three-dimensional mesh by using the optimized three-dimensional representation of the scene through the Marching Cubes algorithm.

[0022] Furthermore, when obtaining the mechanism data of the target device in the current state and learning its potential mechanism representation in step S2, it includes: inputting the mechanism data sequence of the target device in the current state into an attentive sequence-to-sequence network, and the attentive sequence-to-sequence network predicts and learns the potential representation of the target device mechanism data sequence through the GRU algorithm, for the mechanism data sequence in the current state and the mechanism data sequences of all historical states before the current state, and outputs the mechanism data sequence in the next state of the target device.

[0023] Furthermore, in step S2, the vertices of the three-dimensional mesh in the current state are sorted and pre-processed before being input into the Transformer neural network model to obtain the vertex distribution of the three-dimensional mesh in the next state, including:

[0024] All vertices of the current three-dimensional grid are first sorted from low to high by z coordinate, and for vertices with the same z coordinate value, they are sorted from low to high by y coordinate, and for vertices with the same z coordinate value and y coordinate value, they are sorted from low to high by x coordinate, to obtain a vertex sequence;

[0025] Embed the corresponding coordinate information, belonging point information, value information and mechanism potential representation information for each element in the vertex sequence, and obtain the high-dimensional vector Vector corresponding to each element in the vertex sequence v , and then the high-dimensional vector Vector v Input the Transformer neural network model to generate a high-dimensional vector Vector for the next statev .

[0026] Furthermore, in step S2, the current state grid surface distribution is sorted and preprocessed, and the potential representation of the mechanism data sequence of the target device in the current state is input into the Transformer neural network model to obtain the next state grid surface distribution, including:

[0027] The sequence of mesh faces in the current state is sorted in order from low to high according to the vertex index to obtain the face sequence;

[0028] Embed the corresponding face information, position information in the face, and vertex information for each element in the face sequence, and obtain the high-dimensional vector Vector corresponding to each element in the face sequence f , and then the high-dimensional vector Vector f The potential representation of the mechanism data sequence under the current state of the target device is input into the Transformer neural network model to generate a high-dimensional vector Vector of the next state f .

[0029] Compared with the prior art, the advantages of the present invention are:

[0030] The present invention jointly learns the mechanism and three-dimensional structure of the equipment digital twin model by analyzing historical data and operation data. It can characterize the mechanism data of the equipment model in different states and, as a generative model, has the ability to generate its three-dimensional structure in this state, which helps to automate model construction and strengthen the connection with the equipment mechanism data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of an embodiment of the present invention.

[0032] Figure 2 A schematic diagram of the structure of a multi-layer perceptron.

[0033] Figure 3 Schematic diagram of the mechanism sequence model structure of an embodiment of the present invention.

[0034] Figure 4 Schematic diagram of the vertex sequence model structure of an embodiment of the present invention.

[0035] Figure 5 Schematic diagram of the mesh surface sequence model structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0037] This embodiment proposes a method for learning the joint parameters of the mechanism and appearance of a digital twin model. The method predicts the future state of the virtual model of the device (its mechanism changes and the changes in the structure of the three-dimensional model of the device) through historical data and current operation data. It can solve the problem of frequent three-dimensional modeling of different states of a device and link the mechanism changes of the model with the three-dimensional model structure. Figure 1 As shown, the following steps are included:

[0038] S1) reconstructing a three-dimensional representation of the scene using two-dimensional images of the same scene from multiple angles: obtaining real images of the same scene from different perspectives of the target device, reconstructing the three-dimensional scene based on the real images, rendering the three-dimensional scene using a multi-layer perceptron to obtain a synthetic image, calculating a three-dimensional representation of the scene when the difference between the synthetic image and the real image is the smallest, and generating a current state three-dimensional grid based on the three-dimensional representation of the scene;

[0039] S2) Modeling is performed using the mechanism data in the current state and the three-dimensional mesh of the reconstructed three-dimensional representation: the mechanism data in the current state of the target device is obtained and its potential mechanism representation is learned to obtain a sequence of mechanism data in the next state of the target device; the vertices of the three-dimensional mesh of the current state are sorted and input into the Transformer neural network model after preprocessing to obtain the vertex distribution of the three-dimensional mesh of the next state; the current state mesh surface distribution is generated according to the three-dimensional mesh of the current state of the target device, and the current state mesh surface distribution is sorted and input into the Transformer neural network model after preprocessing and the potential representation of the mechanism data sequence in the current state of the target device to obtain the mesh surface distribution of the next state.

[0040] The target device of this embodiment is a rail transit subway train bogie, and its continuous state data is mechanism data, including structural parameters, performance parameters, and material parameters at a certain moment. The real images of the same scene at different viewing angles are images of the bogie at different viewing angles.

[0041] Step S1 of this embodiment first obtains two-dimensional images under different perspectives. At this time, the above two-dimensional information is not directly analyzed and processed, but the inverse mapping method is innovatively used to restore it to a three-dimensional scene for processing; at the same time, the reconstructed three-dimensional information is sampled using rays. In order to overcome the shortcomings of the traditional method's insufficient information learning ability, the sampling results are sent to the multi-layer perceptron for feature learning. Step S1 improves the traditional method of using only two-dimensional digital models for mechanism and appearance representation, and innovatively adopts the method of 3D model generation to create more reliable synthetic data for downstream generation tasks; similar to the two-dimensional digital model, the three-dimensional digital model also has data performance for data acquisition, data operation, data organization and data analysis, but the spatial information of the three-dimensional model device is more direct, the data analysis function is more powerful, and the huge advantages in visualization can bring better performance in downstream tasks. At the same time, combined with the multi-layer perceptron to optimize the 3D scene representation, this method has greatly improved learning ability compared with traditional linear methods and single-layer perception methods. The optimized 3D scene representation can achieve better simulation prediction effects in downstream data twin tasks. The following is a specific description of step S1 of this embodiment, and step S1 specifically includes:

[0042] S1.1) Obtain images of the current scene of the device at different perspectives and the corresponding camera-to-world matrix. In step S1 of this embodiment, reconstructing the three-dimensional scene according to the real image requires reverse mapping of the images of different perspectives and the corresponding camera-to-world matrix to generate a 3D scene from a 2D image. Therefore, when obtaining the real images of the same scene of the target device at different perspectives, the corresponding camera-to-world matrix C is also obtained. ex -1 .

[0043] S1.2) Reverse mapping, generating a 3D scene from a 2D image. In step S1 of this embodiment, when reconstructing a 3D scene based on the real image, it includes: using ray casting and volume rendering technology in computer graphics to generate rays passing through each pixel in the real image, and these rays are represented as:

[0044] r(t)=o+td (1)

[0045] Among them, o is the origin of the ray, d represents the direction of the ray, and t is a scalar parameter indicating how far one needs to go from the origin o along the direction vector d to reach a specific point on the ray. As t changes, it produces different points on the ray. When t is 0, r(t) is equal to the origin o. When t is positive, r(t) moves along the ray in the direction of d, and when t is negative, r(t) moves in the opposite direction of d. In this embodiment, the origin of the ray is the position of the corresponding pixel on the image plane, and the direction of the ray is the direction of the corresponding pixel toward the camera aperture. Therefore, the direction expression of the ray in step S1 is as follows:

[0046]

[0047] Among them, C ex -1 is the camera-to-world matrix corresponding to the real image, t′ ex represents the origin of the ray, R′ ex It is C ex -1 The upper 3×3 matrix, They are homogeneous representations of points in the camera coordinate frame and the world coordinate frame, respectively.

[0048] Next, when rendering the three-dimensional scene using a multi-layer perceptron to obtain a synthetic image, the process includes:

[0049] S1.3) Sample the rays. Extract three-dimensional sample points from each ray, and the three-dimensional sample point expression is as follows:

[0050]

[0051] Among them, U refers to uniform sampling, t f and t n are the farthest and nearest points on the current ray, respectively. N represents the number of parts into which the current ray is divided. i Represents the i-th 3D sample point of the current ray.

[0052] S1.4) The sampled three-dimensional sample points are sent to the multi-layer perceptron to learn their related mapping. Figure 2 As shown, in this embodiment, the input to the multilayer perceptron is a five-dimensional vector of three-dimensional sample points, including the spatial coordinates (x, y, z) and direction of each three-dimensional sample point. The multi-layer perceptron predicts and outputs the color RGB and volume density σ of each 3D sample point, including:

[0053] Assume that there is a ray sampling point p, which is represented by a three-dimensional coordinate (x, y, z). Encode its coordinates and initialize it to a D-dimensional vector p'. After p' is fully connected through 4 layers, the feature f1 is obtained, which is:

[0054] f1=w 4 (w 3 (w 2 (w 1 p′+b 1 )+b 2 )+b 3 )+b 4 (4)

[0055] The weight parameter w 1 , w2 , w 3 , w 4 They are all randomly initialized and updated as the network is trained;

[0056] Concatenate f1 and p′ and input them into the 3-layer full connection to get feature f2, which is:

[0057]

[0058] The feature f2 is passed through a full connection with a sigmoid activation function to output a density value between 0 and 1, which is:

[0059] σ=sigmoid(w 8 f2+b 8 ) (6)

[0060] Combine feature f2 with the ray direction to predict the RGB4 color value of the ray sampling point p. Assume that the ray direction is Encode it and initialize it as a D-dimensional vector q:

[0061]

[0062] S1.5) Use the color and volume density predicted by the multi-layer perceptron to render the three-dimensional scene. Integrate the color and volume density of each three-dimensional sample point sampled by rays in the three-dimensional scene to obtain a composite image containing the color of the specific point. The expression for integrating the color and volume density of each three-dimensional sample point is as follows:

[0063]

[0064] Among them, t f and t n are the farthest and nearest points on the current ray, σ(r(t)) is the volume density of the current ray at the three-dimensional sample point t, c(r(t),) is the color of the current ray at the three-dimensional sample point t, Indicates the degree to which light penetrates the three-dimensional space to reach the three-dimensional sample point.

[0065] Next, we calculate the 3D representation of the scene where the difference between the synthetic image and the real image is minimized, including

[0066] S1.6) Minimize the gap between the synthetic image and the real image through the photometric consistency loss to optimize the 3D representation of the scene. The photometric consistency loss is used to measure the gap between the synthetic image and the real image, and is usually used in image generation and visual reconstruction tasks. By minimizing the photometric consistency loss, we can make the synthetic image as close to the real image as possible. Assuming we have a synthetic image I_synth and a real image I_real, we can calculate the photometric consistency loss using a pixel-level difference metric (such as mean square error, MSE). It can be expressed as follows:

[0067] L_photometric = 1 / N ∑(I_synth(i) - I_real(i))^2 (9)

[0068] Where L_photometric represents the photometric consistency loss, N represents the total number of pixels in the image, i represents the index of the pixel, I_synth(i) represents the value of the i-th pixel in the synthetic image, and I_real(i) represents the value of the i-th pixel in the real image. This formula calculates the sum of the squares of the difference in each corresponding pixel value between the synthetic image and the real image and normalizes it to the total number of pixels.

[0069] To minimize the gap between synthetic and real images, we need to minimize the photometric consistency loss:

[0070] minimize L_photometric(10)

[0071] By optimizing this loss function, we can make the synthesized image closer to the real image in terms of photometry;

[0072] Generating a current state three-dimensional grid according to the three-dimensional representation of the scene includes: generating a current state three-dimensional grid by using the optimized three-dimensional representation of the scene through the Marching Cubes algorithm. Since it is a conventional method used by those skilled in the art, its specific implementation process will not be repeated.

[0073] Step S2 of this embodiment first models the mechanism data. In order to jointly learn the historical data and current parameters at the same time, an attention module is introduced to represent the mechanism data as a sequence for learning. Secondly, a three-dimensional grid is established by vertex modeling, and the obtained vertex information is flattened and embedded. One vertex information is disassembled into four angles and embedded respectively. The position of the next vertex is predicted by inputting the above embedded information into the Transformer module. Then a mesh surface model is established, and it is also input into the Transformer module for surface position prediction. In terms of mechanism modeling, step S2 introduces an attention mechanism, and the mechanism data of the device is input into the network with an attention mechanism in a sequence to realize the joint learning of current data and historical data. In the mesh modeling of the 3D image, vertex modeling is used, and 3D mesh data is generated in combination with surface modeling to complete the prediction of point and surface information of the next state of the device. The following is a specific description of step S2 of this embodiment. Step S2 specifically includes:

[0074] When obtaining the mechanism data of the current state of the target device and learning its mechanism potential representation, it includes:

[0075] S2.1) Model the mechanism data and establish a sequence-to-sequence model. In this embodiment, in order to model the structural parameters, performance parameters, and material parameters of the bogie in the current state as a sequence model, the structural parameters, performance parameters, and material parameters of the target device in the current state are input as the mechanism data sequence in the current state into the sequence-to-sequence network with attention. The sequence-to-sequence network with attention is as follows: Figure 3 As shown in the figure, through the GRU algorithm, the potential representation C of the target device mechanism data sequence is predicted for the mechanism data sequence in the current state and the mechanism data sequence of all historical states before the current state. Mechanism , output the mechanism data sequence of the target device in the next state, and the calculation and derivation process is as follows:

[0076] A bidirectional GRU is used to process the mechanism data from the forward and reverse directions respectively to extract the feature representation of the mechanism data.

[0077] The formula for the forward GRU is The formula for the reverse GRU is

[0078] The final hidden state is obtained as the feature vector of the potential representation of the mechanism data

[0079] The sequence with attention can not only take the current features as input, but also input the historical information into the prediction network. This method combines historical and current parameters and features, overcoming the problem of inaccurate prediction information caused by traditional methods that can only rely on local information for prediction.

[0080] Next, the vertices of the current state three-dimensional grid are sorted and preprocessed before being input into the Transformer neural network model to obtain the vertex distribution of the next state three-dimensional grid, including:

[0081] S2.2) Establish a vertex model. The vertex model refers to representing the 3D model established in step S1 as a strictly ordered sequence of vertices. This strict order enables them to apply the attention-based sequence modeling method to generate a 3D mesh. In the established three-dimensional model, each vertex of the model corresponds to a unique coordinate representation (x, y, z). In order to facilitate the input of vertex information into the network, the vertex information will be preprocessed as follows: All vertices of the current state three-dimensional mesh are first sorted from low to high by z coordinate, and for vertices with the same z coordinate value, they are sorted from low to high by y coordinate, and for vertices with the same z coordinate value and y coordinate value, they are sorted from low to high by x coordinate, and a flattened vertex sequence is obtained;

[0082] like Figure 4 As shown in the figure, after obtaining the flattened vertex sequence, the node information is encoded into a vector of a specified size through embedding operation, and the corresponding coordinate information, belonging point information, value information and mechanism potential representation information (which is the feature vector C obtained after extracting the mechanism information) are embedded for each element in the vertex sequence. Mechanism ), get the high-dimensional vector Vector corresponding to each element in the vertex sequence v , and then the high-dimensional vector Vector v Input the Transformer neural network model to generate a high-dimensional vector Vector for the next state v , specifically: using the transformer network to maximize the product of conditional vertex distribution, we have: The flattened vertex sequence for the next state is denoted by v seq The elements are represented by v n , n=1,2,...,N v . The high-dimensional vector Vector of the next state v The vertex distribution of the three-dimensional grid in the next state can be obtained.

[0083] Next, include:

[0084] S2.3) Establish a mesh surface model. That is, the vertex distribution of the three-dimensional mesh and the mechanism potential representation C obtained in step S2.1) are used.Mechanism Generates a distribution of mesh surface sequences for the condition.

[0085] First, the current state mesh surface distribution is generated according to the three-dimensional mesh of the current state of the target device. Then, the current state mesh surface distribution is sorted and preprocessed, and the potential representation of the mechanism data sequence of the target device in the current state is input into the Transformer neural network model to obtain the next state mesh surface distribution, including:

[0086] The sequence of mesh faces in the current state is sorted in order from low to high according to the vertex index to obtain the face sequence;

[0087] like Figure 5 As shown, each element in the face sequence is embedded with the corresponding face information, position information in the face, and vertex information, and the high-dimensional vector Vector corresponding to each element in the face sequence is obtained. f , and then the high-dimensional vector Vector f The potential representation C of the mechanism data sequence under the current state of the target device Mechanism Input the Transformer neural network model to generate a high-dimensional vector Vector for the next state f Specifically, the transformer network is used to maximize the product of the conditional grid surface distribution, which is: Where θ represents the weight parameter; v represents the vertex information, which is represented by the vertex high-dimensional vector obtained in the previous step; a three-dimensional mesh is usually composed of a collection of triangles. A polygon is composed of many triangular faces, and different polygons require different numbers of triangular faces. Assume that the i-th polygon is represented as By N f The triangles are composed of a spacer s between each triangle. As with the vertex sequence, we concatenate the faces into a flat sequence and use f seq Indicates. n As f seq The elements in, where subscript n = 1, ..., N f .

[0088] The high-dimensional vector Vector of the next state f The mesh surface distribution of the next state can be obtained.

[0089] In summary, the present invention jointly learns the mechanism and three-dimensional structure of the digital twin device model by analyzing historical data and operation data. It can characterize the mechanism data of the device model in different states and, as a generative model, has the ability to generate its three-dimensional structure in this state, which helps to automate model construction and strengthen the connection with the device mechanism data.

[0090] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for learning joint parameters of digital twin model mechanism and appearance, characterized in that: The following steps are involved: S1) obtaining real images of the same scene of a target device under different viewing angles, reconstructing the three-dimensional scene according to the real images, rendering the three-dimensional scene using a multi-layer perceptron to obtain a synthetic image, calculating a three-dimensional representation of the scene when the difference between the synthetic image and the real image is the smallest, and generating a current state three-dimensional grid according to the three-dimensional representation of the scene; S2) obtaining the mechanism data of the target device in the current state and learning its potential mechanism representation, and obtaining the mechanism data sequence of the target device in the next state. When obtaining the mechanism data of the target device in the current state and learning its potential mechanism representation, the method includes: inputting the mechanism data sequence of the target device in the current state into a sequence-to-sequence network with attention, wherein the sequence-to-sequence network with attention predicts and learns the potential representation of the mechanism data sequence of the target device through a GRU algorithm for the mechanism data sequence in the current state and the mechanism data sequences of all historical states before the current state, and outputs the mechanism data sequence in the next state of the target device; The vertices of the current state three-dimensional grid are sorted and preprocessed before being input into the Transformer neural network model to obtain the vertex distribution of the next state three-dimensional grid, including: All vertices of the current three-dimensional grid are first sorted from low to high by z coordinate, and for vertices with the same z coordinate value, they are sorted from low to high by y coordinate, and for vertices with the same z coordinate value and y coordinate value, they are sorted from low to high by x coordinate, to obtain a vertex sequence; Embed the corresponding coordinate information, belonging point information, value information and mechanism potential representation information for each element in the vertex sequence, and obtain the high-dimensional vector Vector corresponding to each element in the vertex sequence v , and then the high-dimensional vector Vector v Input the Transformer neural network model to generate a high-dimensional vector Vector for the next state v ; The current state mesh surface distribution is generated according to the three-dimensional mesh of the current state of the target device. The current state mesh surface distribution is sorted and preprocessed, and the potential representation of the mechanism data sequence of the target device in the current state is input into the Transformer neural network model to obtain the next state mesh surface distribution, including: The sequence of mesh faces in the current state is sorted in order from low to high according to the vertex index to obtain the face sequence; Embed the corresponding face information, position information in the face, and vertex information for each element in the face sequence, and obtain the high-dimensional vector Vector corresponding to each element in the face sequence f , and then the high-dimensional vector Vector f The potential representation of the mechanism data sequence under the current state of the target device is input into the Transformer neural network model to generate a high-dimensional vector Vector of the next state f .

2. The method for learning joint parameters of digital twin model mechanism and appearance according to claim 1, characterized in that: When reconstructing the three-dimensional scene according to the real image in step S1, it includes: generating a ray passing through each pixel in the real image, the origin of the ray is the position of the corresponding pixel in the image plane, and the direction of the ray is the direction of the corresponding pixel toward the camera aperture.

3. The method for learning the joint parameters of the digital twin model mechanism and appearance according to claim 2 is characterized in that: When obtaining the real images of the same scene of the target device at different viewing angles in step S1, the corresponding camera-to-world matrix is ​​also obtained. The direction expression of the ray is as follows: Among them, C ex -1 is the camera-to-world matrix corresponding to the real image, R ′ ex It is C ex -1 The upper 3×3 matrix, They are homogeneous representations of points in the camera coordinate frame and the world coordinate frame, respectively.

4. The method for learning the joint parameters of the digital twin model mechanism and appearance according to claim 1, characterized in that: When a multilayer perceptron is used to render the three-dimensional scene to obtain a synthetic image in step S1, the process includes: extracting a three-dimensional sample point from each ray, inputting the spatial coordinates and direction of each three-dimensional sample point into the multilayer perceptron, obtaining the color and volume density of each three-dimensional sample point, and integrating the color and volume density of each three-dimensional sample point to obtain a synthetic image.

5. The method for learning the joint parameters of the digital twin model mechanism and appearance according to claim 4 is characterized in that: The three-dimensional sample point expression is as follows: Among them, U refers to uniform sampling, t f and t n are the farthest and nearest points on the current ray, respectively. N represents the number of parts into which the current ray is divided. i Indicates the i-th 3D sample point of the current ray.

6. The method for learning the joint parameters of the digital twin model mechanism and appearance according to claim 4, characterized in that: The expression for integrating the color and volume density of each 3D sample point is as follows: Among them, t f and t n are the farthest and nearest points on the current ray, σ(r(t)) is the volume density of the current ray at the three-dimensional sample point t, c(r(t),) is the color of the current ray at the three-dimensional sample point t, Indicates the degree to which light penetrates the three-dimensional space to reach the three-dimensional sample point.

7. The method for learning joint parameters of digital twin model mechanism and appearance according to claim 1, characterized in that: When calculating the three-dimensional representation of the scene when the gap between the synthetic image and the real image is minimized in step S1, it includes: minimizing the gap between the synthetic image and the real image by photometric consistency loss to optimize the three-dimensional representation of the scene; When generating the current state three-dimensional mesh according to the three-dimensional representation of the scene in step S1, it includes: generating the current state three-dimensional mesh by using the optimized three-dimensional representation of the scene through the Marching Cubes algorithm.