Digital twinborn scene rapid construction method based on cross-modal generation

Abstract

The invention relates to a digital twinborn scene rapid construction method based on cross-modal generation. The method is composed of a cross-modal data generation framework and a digital twinning construction framework. The cross-modal data generation framework comprises a character generation module, an image generation module and a model reconstruction module; the digital twinning construction framework comprises a digital twinning model construction method, a digital twinning model assembly method, a digital twinning model fusion method and a digital twinning model verification and calibration method. According to the method, a rapid virtual modeling strategy in a low-sample application scene is defined, a basis is provided for virtual experiments and virtual-real symbiosis, the construction efficiency of a twinborn scene is improved, and a new possibility is provided for datamation of the scene in a resource-constrained environment.

Description

Technical Field

[0001] The present invention relates to a method for rapidly constructing a digital twin scene based on cross-modal generation, and belongs to the technical field of digital twin modeling. Background Art

[0002] With the continuous development of information and digital technology, digital twin technology has become a hot research topic due to its great potential in equipment monitoring, predictive maintenance and operation optimization. Digital twin was first proposed by American scholar Grieves to describe physical products and virtual products and the relationship between the two. It was first used for the health maintenance and protection of aerospace vehicles. In recent years, it has received widespread attention from scholars at home and abroad and has been widely used in many industrial fields. In 2012, NASA gave a conceptual description of digital twins: Digital twins refer to the integration of multidisciplinary and multi-scale simulation processes by making full use of physical models, sensors, operation history and other data. It creates a virtual copy of a physical entity, which can simulate its performance in real time, predict future states, and provide decision support.

[0003] As an important enabling way to achieve digital transformation and promote intelligent upgrading, digital twins have always attracted attention from all walks of life. They have moved from theoretical research to practical application. Leading companies in many fields have begun to try to use digital twins to discover problems and improve efficiency. In order to realize the potential of digital twins, there is still a lot of work to be done. For example, in the application scenario, each model is built from scratch, and the current research on scenario construction methods and standards is still insufficient. In addition, it is difficult to collect various modal data from a large number of sensors, which has extremely high cost and time requirements. Therefore, in the process of building digital twin scenarios, data problems and modeling challenges are significant. In summary, more research on the construction methods of twin scenarios is urgently needed in the field of digital twins. Summary of the invention

[0004] The present invention aims at the problems existing in the prior art and provides a method for rapidly constructing digital twin scenes based on cross-modal generation, including a cross-modal data generation framework and a digital twin construction framework. The method defines a method for rapidly constructing twin scenes in data-scarce or sample-free engineering environments, speeds up the construction efficiency of twin scenes, and provides new possibilities for object digitization in resource-constrained environments.

[0005] In order to achieve the above purpose, the technical solution of the present invention is as follows: a method for quickly constructing a digital twin scene based on cross-modal generation, which consists of a cross-modal generation framework and a digital twin scene construction framework. The cross-modal generation framework contains three modules, a text generation module, an image generation module, and a model reconstruction module; the digital twin scene construction framework includes four parts: a digital twin model construction method, a digital twin model assembly method, a digital twin model fusion method, and a digital twin model verification and calibration method. The specific construction process and steps are as follows:

[0006] Step 1: According to the twin requirements and current status of the project scenario, the text generation module is used to construct the generation instructions required by the image generation module. Then, in the image generation module, the target image group is generated based on the image generation instructions exported by the text generation module; then, the image group is input into the model reconstruction group to complete the generation of the image from a single perspective to a multi-perspective image, and the 3D model is reconstructed based on the multi-perspective image to obtain a high-quality mesh model and rendering.

[0007] Step 2: Pass the model file and rendering generated in step 1 into the digital twin scene construction framework. Complete the setting of geometric model, physical model, behavior model and rule model based on the digital twin model construction method. Then, adopt the digital twin model assembly method to complete the assembly of the twin model from unit-level model to system-level scene in the spatial dimension according to the knowledge graph and engineering concept. Then, according to the digital twin model fusion method, the multi-domain fusion of the model involved in the project is realized through the multi-domain algorithm library and neural network technology. Finally, after the model is built, assembled and fused, it needs to be verified to ensure correctness and effectiveness. Verify that the model output is consistent with the actual output to ensure accuracy. The unit-level model is verified first, and then the assembled or fused model is verified. It can be applied if it meets the actual project requirements, and it needs to be corrected if it does not meet the requirements. Verification and correction are iterated until the project requirements are met.

[0008] Furthermore, the text generation module, image generation module, and model reconstruction module of step 1 are specifically implemented as follows:

[0009] Step 11: The core of the text generation module is the large language model. The present invention uses a deep learning model based on large-scale data pre-training. The underlying architecture of the model adopts a converter neural network, which consists of an encoder and a decoder and supports a self-attention mechanism. By setting a template and inputting a clearly defined data generation task description, the text generation module can generate appropriate image generation instructions. At the same time, in order to increase the effectiveness of generated text and alleviate the problem of semantic differences, the In-Context Learning method is used in the text generation module to assist in the execution of instruction generation tasks and improve the model's learning ability in few-sample and zero-sample environments.

[0010] Step 12: The core of the image generation module is the stable diffusion model. The model used in the present invention adopts a step-by-step and iterative forward and backward diffusion process, and has strong generalization performance. In the forward diffusion stage, the model gradually adds noise in the latent space to convert the image into a random noise distribution. In the backward diffusion stage, the model estimates the image noise in the latent space through a noise predictor, and gradually removes the noise to restore a clear image. According to the input content of the specific application scenario, the constructed image generation module can mainly receive three forms of modal data, text input, and text image combination input.

[0011] Step 13: Since the image groups outputted in step 12 are all image data of a single perspective, a mechanism for controlling the camera external parameters other than the photographs needs to be added to the model reconstruction group. The core of the model reconstruction group is the camera viewpoint control method and the 3D model reconstruction method.

[0012] 1) The camera viewpoint control method is to fine-tune the pre-trained diffusion model so that the model can obtain a general mechanism for controlling the camera viewpoint. Based on this method, the model's ability to perform novel view synthesis can be released. The camera viewpoint control method can realize the conversion from single-view image data to multi-view image data.

[0013] 2) The 3D model reconstruction method refers to the Score Jacobian Chaining open source framework, which is the main algorithm used in the field of zero-shot development and 3D generation. The 3D model is regarded as a single point and optimized by stochastic gradient descent. In the 3D model reconstruction method, the classifier-free guidance value is set higher than the normal value to improve the fidelity of the reconstruction.

[0014] Furthermore, the digital twin model construction method, digital twin model assembly method, digital twin model fusion method, digital twin model verification and calibration method in step 2 are specifically implemented as follows:

[0015] Step 21: The goal of the digital twin model construction method is to generate mesh data and render to complete the setting of geometric model, physical model, behavior model and rule model based on step 1. The digital twin model construction method completes the setting of the object's "geometry-physics-behavior-rule" features from multi-dimensional and multi-domain aspects. It can be adjusted according to actual needs to build a multi-domain dimensional model of the core part.

[0016] Step 22: The goal of the digital twin model assembly method is to complete the assembly of the twin model from the unit-level model to the system-level scene in the spatial dimension. The method includes the following contents: first, construct the hierarchical relationship within the twin scene and clarify its assembly order; second, add spatial constraints that conform to physical properties during the model assembly process; finally, realize the twin model assembly based on the constructed constraint relationship and assembly order.

[0017] Step 23: The goal of the digital twin model fusion method is to build a multi-disciplinary and multi-domain fusion digital twin model based on the subject areas involved in the modeling object. The digital twin model fusion method includes a multi-domain algorithm library of mechanical-electrical-hydraulic and neural network technology (AI motion control, path optimization algorithm, image vision detection technology, etc.) to achieve multi-domain fusion of the model involved in the project.

[0018] Step 24: After step 21, step 22, and step 23, the currently constructed digital twin model needs to be verified to ensure the correctness and effectiveness of the model. The verification and correction standard is to check whether the output of the model is consistent with the output of the physical object according to the project requirements. Model verification and correction is an iterative process. By selecting model correction parameters and constructing objective functions, the accuracy of the model is continuously optimized, so that the digital twin model construction method in step 2 can better adapt to different application requirements, conditions and scenarios until the project requirements are met.

[0019] This technical solution explores the method of constructing a twin model based on object-oriented thinking, which meets the needs of digital twin modeling in the workshop and can provide guiding suggestions for the digital twin transformation of the production workshop. At the same time, this method defines the modeling mechanism of digital twins under different conditions, clarifies the modeling elements of digital twins in different application scenarios, standardizes the digital twin modeling process, and reduces the efficiency loss caused by unclear application scenarios and unclear assembly methods.

[0020] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) In traditional application scenarios, digital twin scene construction generally builds each model from scratch. It is difficult to collect various modal data from a large number of sensors, and it has extremely high cost and time requirements. The present invention is aimed at the problems existing in the prior art and provides a method for quickly building a digital twin scene based on cross-modal generation, including a cross-modal data generation framework and a digital twin construction framework. The method defines a method for quickly building a twin scene in an engineering environment with scarce data or no samples, speeds up the construction efficiency of the twin scene, and provides new possibilities for object digitization in a resource-constrained environment; (2) The present invention constructs a cross-modal twin model generation framework, which includes three modules, a text generation module, an image generation module and a model reconstruction module. It is possible to construct the generation instructions required for the image generation module through the text generation module in the absence of samples or low samples. Then, the target image group is generated in the image generation module according to the image generation instruction; and the three-dimensional model is reconstructed based on the single-view image in the model reconstruction group to achieve rapid generation of model data without samples or in a low-sample state; (3) The present invention proposes a subsequent digital twin scene construction framework based on cross-modal generated data, which includes four parts: a digital twin model construction method, a digital twin model assembly method, a digital twin model fusion method, and a digital twin model verification and calibration method. According to the knowledge graph and engineering concept, combined with the database and neural network, rapid virtual modeling of low-sample application scenarios is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for rapidly constructing a digital twin scene based on cross-modal generation according to the present invention;

[0022] Figure 2 Construct a flow chart for the twin scenario of a complex product manufacturing unit in the embodiment;

[0023] Figure 3 A flow chart is constructed for the twin scenario of the smart block scenario in the embodiment. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] Example 1: According to the present invention, the twin scene of the complex product manufacturing unit is constructed. Taking the pseudo-satellite complex product manufacturing unit scene as an example, the construction process is as follows: Figure 2 As shown in the figure, firstly, the complex product model is generated within the cross-modal data generation framework: the satellite component image generation instruction (Prompt machine ), input to the image generation module to generate an image group (Image machine) and complete the model reconstruction in the model reconstruction group (3D machine ); then complete the twin scenario construction of the complex product manufacturing unit within the digital twin construction framework: construct the digital twin model based on the complex product model (DT BUILD ), Digital Twin Model Assembly and Fusion (DT ASS ), Digital Twin Model Verification and Calibration Method (DT Testing ). The specific steps include:

[0026] Step 1: According to the twin requirements and current status of the satellite complex product manufacturing unit scenario, the generation instructions required by the image generation module are constructed through the text generation module. machine (prompt1-prompt n ). Then, in the image generation module, the target image group is generated based on the image generation instructions exported by the text generation module. machine (imageData1-imageData n ); Then, the image group is input into the model reconstruction group to complete the generation of the image from a single perspective to a multi-perspective image, and the 3D model is reconstructed based on the multi-perspective image. machine (3Dmodel1-3Dmodel n ) to obtain high-quality mesh models and renderings.

[0027] ①Build image generation command prompt machine (prompt1-prompt n ): The large language model of the text generation module in the instance is based on the generative language model ChatGPT4.0. First, the demonstration and test cases in the instance are input into In-Context Learning to generate a template for image production instructions in the complex product manufacturing unit scenario. Then, a clearly defined data generation task description is input to obtain the corresponding image generation instructions.

[0028] ②Generate target image group Image machine (imageData1-imageData n ): In this example, a single-view image group of a complex product model is generated based on a stable diffusion model. First, select the model detection point, VAE encoder, CLIP encoder and optimization scheme, and then enter the image generation command (prompt1-prompt n ) and sample images (optional) to generate the target image group.

[0029] ③3D model reconstruction machine (3Dmodel1-3Dmodel n): In this example, the camera viewpoint control method and 3D model reconstruction method in the model reconstruction group are used to complete the 3D reconstruction of the complex product model. First, we have the target image group (imageData1-imageData n ) to remove interference and background. Then, seed values ​​and sample steps are set in the camera viewpoint control method and the 3D model reconstruction method to obtain multi-view image data and generate high-quality mesh models and renderings.

[0030] Step 2: Pass the complex product model file and rendering generated in step 1 into the digital twin scene construction framework. Complete the setting of geometric model, behavior model and rule model based on the digital twin model construction method. Then, use the digital twin model assembly method to complete the assembly of product model-manufacturing unit twin scene in the spatial dimension. Next, complete the mechanical / electrical / control model fusion of the manufacturing unit twin scene based on the engineering algorithm library. Finally, verify the accuracy of the model operation output and the actual output of the scene.

[0031] In summary, this embodiment can quickly construct a twin scene of a complex product manufacturing unit, speeding up the construction efficiency of the twin scene. Without large-scale sensor deployment and multimodal data acquisition, the digital twin scene rapid construction method based on cross-modal generation provided by the present invention is adopted, and a cross-modal data generation framework and a digital twin construction framework are used to achieve rapid virtual modeling of low-sample application scenarios.

[0032] Example 2: According to the present invention, a twin scene of a smart block is constructed. Taking a simulated smart block scene as an example, the construction process is as follows: Figure 3 As shown in the figure, firstly, the generation of the main models in the smart block scene (street lights, buildings, bridges, roads, etc.) is completed within the cross-modal data generation framework: the smart block image generation instruction (Prompt street ), input to the image generation module to generate the smart block image (Image street ), and complete the reconstruction of street lights, buildings, bridges, roads and other models in the block in the model reconstruction group (3D street ); then complete the construction of the smart block twin scene within the digital twin construction framework: construct the digital twin model based on the models of each element of the smart block (DT BUILD ), Digital Twin Model Assembly and Fusion (DT ASS ), Digital Twin Model Verification and Calibration Method (DT Testing ). The specific steps include:

[0033] Step 1: Based on the twin requirements and current status of the smart block scene, the text generation module is used to construct the generation instructions required to generate the element image in the block. street(prompt1-prompt n ). Then, in the image generation module, the smart block element image group (street lights, buildings, bridges, roads, etc.) is generated based on the generation instructions. street (imageData1-imageData n ); Then, the image group is input into the model reconstruction group to complete the generation of element images from a single perspective to multi-perspective images, and the 3D model reconstruction of the block elements is completed based on the multi-perspective images. street (3Dmodel1-3Dmodel n ) to obtain high-quality mesh models and renderings.

[0034] ① Build a smart block image generation command prompt street (prompt1-prompt n): The large language model of the text generation module in the instance is based on the generative language model ChatGPT4.0. First, input the demonstration and test cases in the instance into In-Context Learning to generate the image production instruction template of the elements in the smart block scene. Then, input a clear element description to obtain the smart block image generation instruction.

[0035] ② Generate smart block element image group Image street (imageData1-imageData n ): In the example, a single-view image group of smart block elements (street lights, buildings, bridges, roads, etc.) is generated based on the stable diffusion model. After selecting the model detection point, VAE encoder, CLIP encoder and optimization scheme, input the block image generation instruction and sample block image (optional) to realize the generation of block element image group.

[0036] ③3D reconstruction of block elements street (3Dmodel1-3Dmodel n ): In the example, the three-dimensional reconstruction of the core element model of the block is completed by using the camera viewpoint control method and the 3D model reconstruction method in the model reconstruction group. First, we preprocess the smart block element image group to remove interference and background. Then, set parameters in the camera viewpoint control method and the 3D model reconstruction method to obtain multi-view image data of the block elements and generate a high-quality three-dimensional model of the block elements.

[0037] Step 2: Pass the three-dimensional model of the block elements generated in step 1 into the digital twin scene construction framework. Complete the setting of the geometric model, physical model, behavior model and rule model based on the digital twin model construction method. Then, use the digital twin model assembly method to complete the assembly of the block element-smart block twin scene in the spatial dimension. Next, complete the fusion of multi-domain models and functional models (path planning, etc.) of the smart block twin scene based on the engineering algorithm library and artificial intelligence algorithm. Finally, verify the real-time operation of the model and the matching degree of the prediction results with the actual state of the scene.

[0038] In summary, this embodiment can quickly construct the twin scene of the smart block and speed up the construction efficiency of the twin scene. The digital twin scene rapid construction method based on cross-modal generation provided by the present invention is adopted to quickly obtain the model data of the core elements of the smart block (street lights, buildings, bridges, roads, etc.) through the cross-modal data generation framework, and realize the rapid virtual modeling of the twin scene of the smart block within the digital twin construction framework.

[0039] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A method for rapidly constructing a digital twin scene based on cross-modal generation, characterized in that: The method comprises the following steps: Step 1: According to the twin requirements and current status of the project scenario, the text generation module is used to construct the generation instructions required by the image generation module. Then, in the image generation module, the target image group is generated based on the image generation instructions exported by the text generation module. Next, the image group is input into the model reconstruction group to complete the generation of the image from a single perspective to a multi-perspective image, and the 3D model is reconstructed based on the multi-perspective image to obtain a high-quality mesh model and rendering. Step 2: Pass the model file and rendering generated in step 1 into the digital twin scene construction framework, complete the setting of geometric model, physical model, behavior model and rule model based on the digital twin model construction method, and then use the digital twin model assembly method to complete the assembly of the twin model from unit-level model to system-level scene in the spatial dimension according to the knowledge graph and engineering concept. Then, according to the digital twin model fusion method, the multi-domain algorithm library and neural network technology are used to realize the multi-domain fusion of the models involved in the project. Finally, after the model is built, assembled and fused, it is necessary to verify the correctness and effectiveness, verify that the model output is consistent with the actual output, and ensure accuracy. The unit-level model is verified first, and then the assembled or fused model is verified. If it meets the actual project requirements, it can be applied. If it does not meet the requirements, it needs to be corrected. The verification and correction are iterated until the project requirements are met.

2. The method for rapidly constructing a digital twin scene based on cross-modal generation according to claim 1 is characterized in that: The text generation module, image generation module, and model reconstruction module of step 1 are specifically implemented as follows: Step 11: The core of the text generation module is the large language model, which uses a deep learning model based on large-scale data pre-training. The underlying architecture of the model uses a converter neural network, which consists of an encoder and a decoder and supports a self-attention mechanism. By setting a template and inputting a clearly defined data generation task description, the text generation module can generate appropriate image generation instructions. The In-Context Learning method is used in the text generation module to assist in the execution of instruction generation tasks, thereby improving the model's learning ability in few-sample and zero-sample environments. Step 12: The core of the image generation module is the stable diffusion model. The model used adopts a step-by-step and iterative forward and backward diffusion process, and has strong generalization performance. In the forward diffusion stage, the model gradually adds noise in the latent space to convert the image into a random noise distribution. In the backward diffusion stage, the model estimates the image noise in the latent space through the noise predictor, and gradually removes the noise to restore a clear image. According to the input content of the specific application scenario, the constructed image generation module can mainly receive three forms of modal data: text input, text image combination input, Step 13: Since the image groups outputted in step 12 are all image data of a single perspective, a mechanism for controlling camera external parameters other than taking photos needs to be added to the model reconstruction group.

3. The method for rapidly constructing a digital twin scene based on cross-modal generation according to claim 2 is characterized in that: In step 13, the core of the model reconstruction group is the camera viewpoint control method and the 3D model reconstruction method; Among them, the camera viewpoint control method is to fine-tune the pre-trained diffusion model so that the model can obtain a general mechanism for controlling the camera viewpoint. Based on this method, the model's ability to perform novel view synthesis can be released, and the camera viewpoint control method can be used to realize the conversion from single-view image data to multi-view image data. The 3D model reconstruction method refers to the Score Jacobian Chaining open source framework, which is the main algorithm used in the field of zero-shot development and 3D generation work. The 3D model is regarded as a single point and optimized by stochastic gradient descent. The classifier-free guidance value is set higher than the normal value in the 3D model reconstruction method, thereby improving the fidelity of reconstruction.

4. The method for rapidly constructing a digital twin scene based on cross-modal generation according to claim 2 is characterized in that: The digital twin model construction method, digital twin model assembly method, digital twin model fusion method, digital twin model verification and calibration method of step 2 are specifically implemented as follows: Step 21: The goal of the digital twin model construction method is to generate mesh data and render to complete the setting of geometric model, physical model, behavior model and rule model based on step 1. The digital twin model construction method completes the setting of the object's "geometry-physics-behavior-rule" features from multi-dimensional and multi-domain aspects. It can be adjusted according to actual needs to build a multi-domain dimensional model of the core part. Step 22: The goal of the digital twin model assembly method is to complete the assembly of the twin model from the unit-level model to the system-level scene in the spatial dimension. The method includes the following contents: first, construct the hierarchical relationship within the twin scene and clarify its assembly order; second, add spatial constraints that conform to physical properties during the model assembly process; finally, realize the twin model assembly based on the constructed constraint relationship and assembly order. Step 23: The goal of the digital twin model fusion method is to build a multi-disciplinary and multi-domain integrated digital twin model based on the subject areas involved in the modeling object. The digital twin model fusion method includes a multi-domain algorithm library of mechanical-electrical-hydraulic and neural network technology to realize the multi-domain fusion of the model involved in the project. Step 24: After step 21, step 22, and step 23, the currently constructed digital twin model needs to be verified to ensure the correctness and effectiveness of the model. The verification and correction standard is to check whether the output of the model is consistent with the output of the physical object according to the project requirements. Model verification and correction is an iterative process. By selecting model correction parameters and constructing objective functions, the accuracy of the model is continuously optimized, so that the digital twin model construction method in step 2 can better adapt to different application requirements, conditions and scenarios until the project requirements are met.

Images