Digital twin model generation method and computing device

Abstract

This application relates to the field of digital twin technology, and in particular provides a method and computing device for generating a digital twin model. The method includes: extracting a foreground component image and a background image from an image sequence corresponding to a physical entity, wherein the foreground component image refers to a partial image containing a component in the physical entity, and the background image refers to a partial image not containing a component in the physical entity; performing three-dimensional reconstruction on the component in the foreground component image to obtain a three-dimensional visual mesh model; obtaining a target CAD model matching the component in the foreground component image based on the three-dimensional visual mesh model; and generating a digital twin model of the physical entity based on the background image, the target CAD model, and the three-dimensional visual mesh model.

Description

Technical Field

[0001] This application relates to the field of digital twin technology, and in particular to a method and computing device for generating digital twin models. Background Technology

[0002] A digital twin model is a holographic and high-fidelity digital mapping model created for a physical entity, enabling advanced applications such as monitoring, prediction, simulation, and optimization of the physical entity.

[0003] Currently, reconstruction techniques can be used to create digital twin models of physical entities. Specifically, images of the physical entity can be captured by a camera, and then a digital twin model of the physical entity can be created from these images. However, the accuracy and fidelity of digital twin models created using reconstruction techniques are currently insufficient. Summary of the Invention

[0004] This application provides a digital twin model generation method and computing device. For batch inference requests, a scheduling method combining shared prefix and flexible scheduling mode selection is adopted to achieve dual optimization in terms of inference efficiency and resource management.

[0005] According to a first aspect of the embodiments of this application, a method for generating a digital twin model is provided, comprising: Extract foreground component images and background images from the image sequence corresponding to the physical entity. Foreground component images refer to local images containing components in the physical entity, while background images refer to local images that do not contain components in the physical entity. Perform 3D reconstruction of the components in the foreground component image to obtain a 3D visual mesh model; Based on a 3D visual mesh model, obtain a target CAD model that matches the component in the foreground component image; A digital twin model of the physical entity is generated based on the background image, the target CAD model, and the 3D visual mesh model.

[0006] In this embodiment, a foreground component image containing the component and a background image excluding the component are separated from an image sequence of a physical entity. The foreground component image can be used to reconstruct a 3D visual mesh model of the component. The 3D visual mesh model can represent the specific features of the component; therefore, a target CAD model matching the component can be obtained through the 3D visual mesh model. The background image reflects the environmental information of the physical entity, the target CAD model reflects the precise geometric parameters of the physical entity, and the 3D visual mesh model reflects the real morphological features of the physical entity. Therefore, the digital twin model generated by the background image, the target CAD model, and the 3D visual mesh model can completely map the physical entity in terms of geometric structure and environmental association, ensuring the fidelity of the digital twin model, making the texture of the digital twin model more realistic, and significantly improving the integrity, accuracy, and practicality of the digital twin model.

[0007] In one embodiment of the first aspect, obtaining a target CAD model that matches a component in a foreground component image, based on a three-dimensional visual mesh model, includes: Search the CAD model library for the first CAD model that matches the 3D visual mesh model; Based on the 3D visual mesh model, the first CAD model is geometrically corrected to obtain the corrected target CAD model.

[0008] In this embodiment, a first CAD model matching the morphology of the foreground component's 3D visual mesh model is first retrieved from the CAD model library. Then, based on the actual geometric features of the component embodied in the 3D visual mesh model, targeted geometric corrections are performed on the first CAD model, ultimately generating a target CAD model that precisely matches the physical entity component. By quickly locking the CAD model from the CAD model library and correcting the deviation between the theoretical CAD model and the physical object based on the actual morphological data of the 3D visual mesh model, the geometric accuracy of the target CAD model is ensured, making the subsequently generated digital twin model more standardized and consistent with the physical entity.

[0009] In one embodiment of the first aspect, querying a CAD model from a CAD model library that matches a 3D visual mesh model includes: Semantic recognition is performed on the 3D visual mesh model to obtain the semantic labels of the 3D visual mesh model; Based on the semantic tags, query the CAD model library for the first CAD model that matches the semantic tags.

[0010] In this embodiment, semantic recognition is performed on the 3D visual mesh model to obtain semantic tags that can represent the semantics of the components. Then, using these semantic tags as the retrieval target, a first CAD model matching the semantic tags of the 3D visual mesh model is selected from the CAD model library. Semantic retrieval reduces the computational complexity of model retrieval, improves the accuracy and efficiency of retrieval, quickly identifies CAD models whose semantics match those of physical entity components, effectively improves the matching degree between the generated object of the digital twin model and the physical entity, and further enhances the reconstruction accuracy of the digital twin model.

[0011] In one embodiment of the first aspect, geometrically correcting a first CAD model based on a three-dimensional visual mesh model to obtain a corrected target CAD model includes: Vertex matching is performed on the 3D visual mesh model and the first CAD model to obtain the vertex correspondence. The vertex correspondence includes at least one vertex pair, and each vertex pair includes vertices in the 3D visual mesh model and vertices in the first CAD model. Based on the non-rigid iterative nearest point algorithm and vertex correspondence, the first CAD model is subjected to non-rigid deformation to obtain the corrected target CAD model.

[0012] In this embodiment, vertex matching is first performed on the 3D visual mesh model and the first CAD model to construct vertex pairs representing the geometric feature relationship between the two. Then, using the vertex correspondence as a constraint, the NRICP algorithm is used to drive the first CAD model to undergo non-rigid deformation, causing the model vertices to gradually converge to the actual vertex positions of the 3D visual mesh model. Thus, by providing a precise geometric reference for non-rigid deformation through vertex matching, and leveraging the local deformation capability of the NRICP algorithm, geometric deviations between the theoretical CAD model and the physical component are eliminated, avoiding local morphological differences that rigid registration cannot adapt to. Finally, a target CAD model highly consistent with the geometric features of the physical component is generated, providing support for the high-precision construction of digital twin models.

[0013] In one embodiment of the first aspect, a first CAD model is subjected to non-rigid deformation based on a non-rigid iterative nearest-point algorithm and vertex correspondence to obtain a modified target CAD model, including: Based on at least one vertex pair in the vertex correspondence, an energy function is constructed. The energy function E_total(T) is the function formed by multiplying the deformation field T acting on the first CAD model by the smoothing constraint function E_smooth(T) and the balance adjustment parameter, and then adding it to the data fitting function E_data(T). The deformation field T acting on the first CAD model is solved until the minimum value of the energy function is reached, and the target deformation field acting on the first CAD model at the time of stopping iteration is obtained. The first CAD model is deformed according to the target deformation field to obtain the corrected target CAD model.

[0014] In this embodiment, a total energy function E_total(T) is constructed based on vertex correspondence, comprising a data fitting function E_data(T) and a smoothing constraint function E_smooth(T). The target deformation field T, which minimizes the energy function value, is then solved iteratively. This target deformation field is then applied to the first CAD model to complete geometric correction. By precisely matching the vertices of the target CAD model with those of the 3D visual mesh model, geometric deviations between the CAD model and the actual object are eliminated. Furthermore, the target deformation field ensures the continuity and smoothness of the CAD model, avoiding geometric distortions such as local wrinkles and breaks, thus obtaining a target CAD model that combines engineering accuracy with morphological consistency.

[0015] In one embodiment of the first aspect, the smoothing constraint function E_smooth(T) is the sum of the transformation differences corresponding to at least one vertex pair. The transformation difference corresponding to the vertex pair can be the product of the transformation coefficients and the transformation difference measure of the vertex pair. The transformation difference measure is determined by the position difference between the position of vertex j after applying its own transformation and the position of vertex j after applying the transformation of vertex i. Vertex j is a vertex in the first CAD model of the vertex pair.

[0016] In this embodiment, the smoothing constraint function E_smooth(T) is defined as the sum of the transformation differences of each vertex pair. The transformation difference is determined by the product of the transformation coefficients of the vertex pair and the transformation difference metric. The transformation difference metric is obtained by comparing the difference between the transformed position of vertex j in the first CAD model and the transformed position of the matched vertex i. This achieves transformation consistency calculation, effectively reduces the geometric distortion problem in the model deformation process, ensures the continuity and smoothness of the surface of the corrected target CAD model, and finally generates a target CAD model with realistic shape and stable structure.

[0017] In one embodiment of the first aspect, it further includes: The background image is reconstructed to obtain an HDR environment map; Based on the background image, the target CAD model, and the 3D visual mesh model, a digital twin model of the physical entity is generated, including: The HDR environment map corresponding to the background image, the target CAD model, and the 3D visual mesh model are mapped to the same coordinate system of the digital twin scene to obtain a digital twin model of the physical entity.

[0018] In this embodiment, an HDR environment map is generated by reconstructing the background image, accurately restoring the complete texture and environmental details of the scene in which the physical entity is located. Furthermore, the HDR environment map, the target CAD model, and the 3D visual mesh model are mapped to the same digital twin scene coordinate system, achieving deep fusion of geometric, textural, and environmental elements. This enables the digital twin model to possess lighting and shadow rendering effects consistent with the real physical entity, ensuring the spatial position matching accuracy of each element. The resulting digital twin model is a physical entity digital twin model that combines accurate geometric shape, realistic environmental features, and immersive visual effects.

[0019] In one embodiment of the first aspect, environmental reconstruction is performed on a background image to obtain an HDR environment map, including: The spherical mapping method is used to perform 360-degree panoramic mapping on the background image to obtain a panoramic image; The panoramic image is rendered to obtain an HDR environment map.

[0020] In this embodiment, a spherical mapping method is used to stitch the background image into a 360-degree panoramic texture, eliminating seams and geometric distortions in multi-view images and fully restoring the panoramic environmental information of the scene where the physical entity is located. Then, professional image rendering processing is performed on the panoramic image to extract and retain texture details in the scene, generating an HDR environment texture with accurate texture characteristics. This provides highly realistic texture details for the subsequent construction of the digital twin model, significantly improving the consistency between the 3D scene and the real environment.

[0021] In one embodiment of the first aspect, three-dimensional reconstruction of components in a foreground component image is performed to obtain a three-dimensional visual mesh model, including: The Poisson surface reconstruction algorithm is used to reconstruct the three-dimensional components in the foreground component image to obtain a three-dimensional visual mesh model.

[0022] In this embodiment, the components in the foreground component image are reconstructed in three dimensions by Poisson surface reconstruction, so as to extract a complete three-dimensional visual mesh model of the component that can be accurately reconstructed from the foreground component image. This allows the three-dimensional visual mesh model to accurately restore the fine geometric features and complete topological structure of the component, providing a highly reliable three-dimensional morphological benchmark for the subsequent construction of digital twin models.

[0023] According to a second aspect of the embodiments of this application, a schematic diagram of a digital twin model generation apparatus is provided, which may include the following units: The image segmentation unit is used to extract foreground component images and background images from the image sequence corresponding to the physical entity. The foreground component image refers to a local image containing the component in the physical entity, and the background image refers to a local image that does not contain the component in the physical entity. The 3D reconstruction unit is used to reconstruct the parts in the foreground part image in 3D to obtain a 3D visual mesh model; The model matching unit is used to obtain a target CAD model that matches the component in the foreground component image based on the 3D visual mesh model; The model generation unit is used to generate digital twin models of physical entities based on background images, target CAD models, and 3D visual mesh models.

[0024] According to a third aspect of the embodiments of this application, a computing device is provided, including: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement any of the above-described digital twin model generation methods.

[0025] According to a fourth aspect of the embodiments of this application, a communication device is provided, including a transceiver and a processor, wherein the transceiver is used to receive or send data, and the processor is used to execute any of the digital twin model generation methods of the embodiments of this application.

[0026] According to a fifth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements any digital twin model generation method.

[0027] According to a sixth aspect of the embodiments of this application, a computer product is provided, comprising: a computer program that, when executed by a processor, implements the steps of any digital twin model generation method.

[0028] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0029] The above and other objects, features, and advantages of the embodiments of this application will become more apparent from the more detailed description of the embodiments in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the embodiments of this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0030] Figure 1 The figure shows an example diagram of a digital twin model generation system according to an embodiment of this application; Figure 2 The figure shows an application example of a digital twin model generation system according to an embodiment of this application; Figure 3The figure shows another application example of a digital twin model generation system according to an embodiment of this application; Figure 4 The figure shows a flowchart of a digital twin model generation method according to an embodiment of this application; Figure 5 The figure shows an example diagram of a three-dimensional visual mesh model according to an embodiment of this application; Figure 6 The illustration shows an example of the application of geometry correction and texture mapping according to an embodiment of this application; Figure 7 The figure shows another flowchart of a digital twin model generation method according to an embodiment of this application; Figure 8 The figure shows an example diagram of a digital twin model generation method according to an embodiment of this application; Figure 9 The figure shows a schematic diagram of a digital twin model generation device according to an embodiment of this application; Figure 10 The figure shows a hardware block diagram of a computing device according to an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application more apparent, exemplary embodiments according to the embodiments of this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the embodiments of this application, and not all embodiments of the embodiments of this application. It should be understood that the embodiments of this application are not limited to the exemplary embodiments described herein.

[0032] The technical solution of this application embodiment can be applied to the field of CAD (Computer-Aided Design) model reconstruction. By obtaining the background image and foreground image of the physical entity through image segmentation, the components in the physical entity are reconstructed through the foreground image and the CAD model, and the texture of the physical entity is mapped through the background image, thereby obtaining a digital twin model with higher fidelity and more realistic texture.

[0033] In related technologies, reconstruction techniques can be used to create digital twin models of physical entities. Specifically, images of the physical entity can be captured by a camera, and then a digital twin model of the physical entity can be reconstructed from the images. However, the accuracy and fidelity of currently created digital twin models are not high.

[0034] To address the aforementioned issues, in this embodiment, after obtaining the image sequence captured by the surround-view camera, foreground component images and background images corresponding to the physical entity's components are extracted based on the image sequence, thereby acquiring foreground component images and background images that can represent the physical entity. Three-dimensional reconstruction is then performed based on the foreground component images, and combined with the target CAD model corresponding to the component, the three-dimensional reconstruction of the component can be completed. Thus, through the reconstructed CAD model and the environment texture corresponding to the background image, a digital twin model is established, resulting in a digital twin model with higher fidelity and more accurate textures.

[0035] like Figure 1 The diagram shown is a system architecture diagram of a digital twin model generation system provided in an embodiment of this application.

[0036] refer to Figure 1 The digital twin model generation system may include: a physical entity 10, one or more surround-view cameras 20 (two are shown in the figure), a computing device 30, and a display 40.

[0037] The physical entity 10 can refer to the entity that needs to be reconstructed from a CAD model. Examples include a factory workshop or production line, a server room, etc. The surround-view camera 20 can include one or more cameras surrounding the physical entity, each capable of capturing real-world images of the physical entity to obtain image sequence 101.

[0038] The image sequence 101 captured by the surround-view camera 20 is transmitted to the computing device 30. Based on the image sequence 101 captured by the surround-view camera 20, the computing device 30 can execute the digital twin model generation method provided in this application embodiment to complete the establishment of the digital twin model of the physical entity 10.

[0039] In some embodiments, the digital twin model created by the computing device 30 can be output by the display 40. The display 40 can be, for example, a large digital screen, or a display screen of another terminal that has a communication connection with the computing device 30.

[0040] Taking physical entity 10 as a server room as an example, such as Figure 2 The diagram shown is an application example of a digital twin model generation system provided in this application. The server room may include several server racks, such as... Figure 2As shown, the surround-view camera 20 can capture real-world images of the cabinet 50, obtaining an image sequence 102 of the cabinet 50. The surround-view camera 20 can send the image sequence 102 of the cabinet 50 to the computing device 30. Based on the image sequence 102 acquired by the surround-view camera 20, the computing device 30 can execute the digital twin model generation method provided in this application embodiment to complete the establishment of a digital twin model 60 of the cabinet 50.

[0041] Taking physical entity 10 as a factory production line as an example, such as Figure 3 The diagram shown illustrates another application example of a digital twin model generation system provided in this application. A factory may contain several production lines, such as... Figure 3 As shown, the surround-view camera 20 can capture real-world images of the factory production line 70, obtaining an image sequence 103 of the factory production line 70. The surround-view camera 20 can send the image sequence 103 of the factory production line 70 to the computing device 30. Based on the image sequence 102 acquired by the surround-view camera 20, the computing device 30 can execute the digital twin model generation method provided in this application embodiment to complete the establishment of a digital twin model 80 of the factory production line 70.

[0042] like Figure 4 The diagram shown is a flowchart of a digital twin model generation method provided in an embodiment of this application. The digital twin model generation method may include the following steps: S401. Extract foreground component images and background images from the image sequence corresponding to the physical entity. Foreground component images refer to local images containing components in the physical entity, and background images refer to local images not containing components in the physical entity.

[0043] Optionally, S401 may include: acquiring an image sequence of a physical entity captured by a surround-view camera; and extracting foreground component images and background images of the physical entity from the image sequence.

[0044] Furthermore, the image sequence can be input into a pre-trained image segmentation model to obtain the foreground and background images output by the model. The image segmentation model can, for example, refer to a semantic segmentation model. A semantic segmentation model can be, for example, a general image segmentation model (Mask2Former, Mask-attention MaskTransformer). Mask2Former supports three image segmentation tasks: semantic segmentation, instance segmentation, and panoptic segmentation.

[0045] A component can refer to an item that exists in a physical entity. Taking a server room as an example, components can be items such as server racks, servers, switches, and lights within the server room. Taking a factory production line as an example, components can be items such as machines, conveyor belts, workbenches, and parts within the production line.

[0046] The foreground component image can contain a component from a physical entity. Taking a server room as an example, the foreground and background images can be partial images obtained through segmentation that contain a server rack.

[0047] Background images can refer to images that do not contain physical entities, such as images of areas corresponding to walls or the ground. For example, after segmenting a local area of ​​a component from an image sequence, the remaining image area is the background image. The local areas of the component in the background image are blank. For instance, in the field of people tracking, the foreground image in the image can be a full-body image of a person, while the image areas other than the full-body image, such as trees, grass, and buildings, are the background image.

[0048] S402. Perform 3D reconstruction of the components in the foreground component image to obtain a 3D visual mesh model.

[0049] Optionally, S402 may include extracting key points of the component in the foreground component image and connecting adjacent key points to form a three-dimensional visual mesh model. That is, the three-dimensional visual mesh model is a mesh formed by connecting key points of the component in the foreground component image in pairs, such as connecting any three adjacent key points to form a triangle.

[0050] For ease of understanding, the following Figure 5 An example diagram of a three-dimensional visual mesh model is shown.

[0051] refer to Figure 5 Assuming the key points of the table in the image are identified as 501-506, during the 3D reconstruction process, the key points of the components can be interpolated based on the characteristics of the identified components, such as obtaining two new key points 507 and 508 through interpolation. After connecting the pairs of adjacent key points, the 3D visual mesh model of the table 509 is obtained.

[0052] S403. Based on the 3D visual mesh model, obtain the target CAD model that matches the component in the foreground component image.

[0053] As mentioned above, the 3D visual mesh model exists in the form of a mesh, and the stereoscopic display of the model depends on the 3D model. A target CAD model matching the component in the foreground component image can be obtained through S403.

[0054] Optionally, S403 may include: performing geometric correction on the CAD model of the component in the foreground component image based on the three-dimensional visual mesh model to obtain the corrected target CAD model.

[0055] S404. Based on the background image, target CAD model, and 3D visual mesh model, generate a digital twin model of the physical entity.

[0056] Optionally, S404 may include: fusing the target CAD model and the 3D visual mesh model to obtain a fused model. This fusion is necessary because the target CAD model does not contain texture features and needs to be fused with the 3D visual mesh model to obtain texture features. Based on the background image, texture mapping is applied to the background of the fused model to obtain a digital twin model.

[0057] Furthermore, feature anchor points of the component can be determined in the background image, the target CAD model, and the 3D visual mesh model respectively. The 3D visual mesh model can then be aligned to the coordinate system of the target CAD model using the ICP (Iterative Closest Point) algorithm to obtain the fused model.

[0058] Based on the background image, texture mapping is performed on the fused model background to obtain a digital twin model. This can include: adjusting the pixel values ​​of each pixel in the fused model background using the pixel values ​​in the background image, and then performing texture mapping on the fused model background to obtain a digital twin model.

[0059] In this embodiment, a foreground component image containing the component and a background image excluding the component are separated from an image sequence of a physical entity. The foreground component image can be used to reconstruct a 3D visual mesh model of the component. The 3D visual mesh model can represent the specific features of the component; therefore, a target CAD model matching the component can be obtained through the 3D visual mesh model. The background image reflects the environmental information of the physical entity, the target CAD model reflects the precise geometric parameters of the physical entity, and the 3D visual mesh model reflects the real morphological features of the physical entity. Therefore, the digital twin model generated by the background image, the target CAD model, and the 3D visual mesh model can completely map the physical entity in terms of geometric structure and environmental association, ensuring the fidelity of the digital twin model, making the texture of the digital twin model more realistic, and significantly improving the integrity, accuracy, and practicality of the digital twin model.

[0060] In one possible design, a CAD model library is pre-established. This library can include multiple CAD models, each representing a specific component. For example, a CAD model of a table lamp.

[0061] Based on the establishment of the CAD model, the CAD model of the component can be obtained quickly and applied to the 3D reconstruction process of the component.

[0062] Therefore, as another embodiment, S403, based on a three-dimensional visual mesh model, obtains a target CAD model that matches the component in the foreground component image, which may include: A1. Search the CAD model library for the first CAD model that matches the 3D visual mesh model.

[0063] A2. Based on the 3D visual mesh model, perform geometric correction on the first CAD model to obtain the corrected target CAD model.

[0064] Optionally, a CAD model can refer to a pre-built three-dimensional model with a solid structure. CAD models can be created using CAD or other drawing software and stored in a CAD model library.

[0065] Optionally, querying the CAD model library for the first CAD model that matches the 3D visual mesh model may include: calculating the similarity between each CAD model in the 3D feature library and the 3D visual mesh model, and determining the CAD model with the highest similarity as the first CAD model.

[0066] Specifically, the model features corresponding to at least one CAD model in the CAD model library can be determined, the model features of the 3D visual mesh model can be extracted, the similarity between the model features corresponding to at least one CAD model and the model features of the 3D visual mesh model can be calculated, and the similarity between at least one CAD model can be obtained.

[0067] In this embodiment, a first CAD model matching the morphology of the foreground component's 3D visual mesh model is first retrieved from the CAD model library. Then, based on the actual geometric features of the component embodied in the 3D visual mesh model, targeted geometric corrections are performed on the first CAD model, ultimately generating a target CAD model that precisely matches the physical entity component. By quickly locking the CAD model from the CAD model library and correcting the deviation between the theoretical CAD model and the physical object based on the actual morphological data of the 3D visual mesh model, the geometric accuracy of the target CAD model is ensured, making the subsequently generated digital twin model more standardized and consistent with the physical entity.

[0068] Since 3D visual mesh models contain geometric and topological information but lack category labels, semantic recognition is needed to obtain semantic labels for the 3D visual mesh models, and then the model query is completed using these semantic labels. As an example, step A1 above, querying the CAD model library for the first CAD model that matches the 3D visual mesh model, may include: Semantic recognition is performed on the 3D visual mesh model to obtain semantic labels for the 3D visual mesh model; based on the semantic labels, the first CAD model that matches the semantic labels is queried from the CAD model library.

[0069] Optionally, semantic recognition of the 3D visual mesh model to obtain its semantic labels may include: identifying the semantic labels of the 3D visual mesh model based on preset rules of the geometric features (such as size, shape, and position) of the 3D mesh; or identifying the semantic labels of the 3D semantic segmentation model based on a pre-trained 3D semantic segmentation model.

[0070] Optionally, after creating a CAD model using drawing software, semantic tags for the CAD model can be set. By semantically matching the semantic tags of each CAD model in the CAD model library with the semantic tags of the 3D visual mesh model, the first CAD model that matches the semantic tags of the 3D visual mesh model can be determined.

[0071] For example, if the semantic label of a 3D visual mesh model is "table lamp", then the first CAD model with the semantic label "table lamp" can be queried from the CAD model library.

[0072] In this embodiment, semantic recognition is performed on the 3D visual mesh model to obtain semantic tags that can represent the semantics of the components. Then, using these semantic tags as the retrieval target, a first CAD model matching the semantic tags of the 3D visual mesh model is selected from the CAD model library. Semantic retrieval reduces the computational complexity of model retrieval, improves the accuracy and efficiency of retrieval, quickly identifies CAD models whose semantics match those of physical entity components, effectively improves the matching degree between the generated object of the digital twin model and the physical entity, and further enhances the reconstruction accuracy of the digital twin model.

[0073] As an example, step A2 above, based on the 3D visual mesh model, performs geometric correction on the first CAD model to obtain the corrected target CAD model, including: Vertex matching is performed on the 3D visual mesh model and the first CAD model to obtain the vertex correspondence. The vertex correspondence includes at least one vertex pair, and each vertex pair includes vertices in the 3D visual mesh model and vertices in the first CAD model. Based on the Non-rigid Iterative Closest Point (NRICP) algorithm and vertex correspondence, the first CAD model is subjected to non-rigid deformation to obtain the corrected target CAD model.

[0074] Optionally, vertex matching is performed on the 3D visual mesh model and the first CAD model to obtain vertex correspondence, including: extracting vertices from the 3D visual mesh model and the first CAD model, establishing one-to-one corresponding vertex pairs, and determining one or more established vertex pairs as vertex correspondence.

[0075] Further, extracting vertices from the 3D visual mesh model and the first CAD model and establishing one-to-one vertex pairs can include: using the rigid ICP (Iterative Closest Point) algorithm to solve for the rotation matrix and translation vector between the 3D visual mesh model and the first CAD model; rotating and translating the 3D visual mesh model and the first CAD model according to the rotation matrix and translation vector to obtain an aligned 3D visual mesh model and the first CAD model. Based on the aligned 3D visual mesh model and the first CAD model, using the K-nearest neighbor algorithm for each vertex of the 3D visual mesh model, finding the nearest vertex in the first CAD model to obtain one or more vertex pairs. Alternatively, anomaly checks can be performed on the obtained one or more vertex pairs, such as removing vertex pairs whose distance exceeds a threshold, to obtain the remaining one or more vertex pairs as the corresponding vertex correspondences.

[0076] In this embodiment, vertex matching is first performed on the 3D visual mesh model and the first CAD model to construct vertex pairs representing the geometric feature relationship between the two. Then, using the vertex correspondence as a constraint, the NRICP algorithm is used to drive the first CAD model to undergo non-rigid deformation, causing the model vertices to gradually converge to the actual vertex positions of the 3D visual mesh model. Thus, by providing a precise geometric reference for non-rigid deformation through vertex matching, and leveraging the local deformation capability of the NRICP algorithm, geometric deviations between the theoretical CAD model and the physical component are eliminated, avoiding local morphological differences that rigid registration cannot adapt to. Finally, a target CAD model highly consistent with the geometric features of the physical component is generated, providing support for the high-precision construction of digital twin models.

[0077] In one possible design, the first CAD model is subjected to non-rigid deformation based on the non-rigid iterative nearest-point algorithm and vertex correspondence to obtain the corrected target CAD model, including: Based on at least one vertex pair in the vertex correspondence, an energy function is constructed. The energy function E_total(T) is the function formed by multiplying the deformation field T acting on the first CAD model by the smoothing constraint function E_smooth(T) and the balance adjustment parameter, and then adding it to the data fitting function E_data(T). The deformation field T acting on the first CAD model is solved until the minimum value of the energy function is reached, and the target deformation field acting on the first CAD model at the time of stopping iteration is obtained. The first CAD model is deformed according to the target deformation field to obtain the corrected target CAD model.

[0078] The energy function can be expressed as: E_total(T) = E_data(T) + λ E_smooth(T), where λ is the balance adjustment parameter.

[0079] Understandably, the Non-rigid Iterative Closest Point (NAICP) algorithm, while preserving the topology of the model, moves the vertices of the 3D visual mesh model in each vertex pair in the direction of the constraint, fitting them to the vertex position of the first CAD model in that vertex pair.

[0080] Specifically, the energy function is a weighted sum of data fitting and smoothing constraints. Its core purpose is to minimize the total energy, so as to fit the measured vertex without causing model distortion.

[0081] After the first CAD model is deformed according to the target deformation field, the smaller the order deviation between the deformed first CAD model and the 3D visual mesh model, the smaller E_data(T) is, and the more the model combines with the actual data.

[0082] in, N is the number of vertex pairs. Represents Euclidean distance. This represents the target deformation field after the vertices of the first CAD model have been deformed. These are the vertices of the 3D visual mesh model.

[0083] Among them, the smoothing constraint function E represents the set of adjacent edges of the vertices in the first CAD model (such as pairs of vertices sharing the same triangular face in a mesh); As vertex The deformation field gradient; As vertex The deformation field gradient. A smoothing constraint function can constrain the smoothness of the model, preventing excessive local stretching / bending and ensuring the continuity of the deformation field.

[0084] Optionally, λ is a balance adjustment parameter that can be used to adjust the balance between the two constraints. A larger λ value (e.g., 0.1-0.5) prioritizes topology smoothness, while a smaller λ value (e.g., 0.8-0.9) prioritizes accuracy closer to actual conditions. The specific setting can be determined according to usage requirements.

[0085] Understandably, the deformation field T is a deformation function acting on all vertices. The core solution involves iterative optimization, gradually decreasing E_total(T) until convergence. Specifically, the deformation field T0 is initialized, and an iteration termination condition is set, such as E_total(T) < a threshold (this threshold can be pre-set). Starting from the initialization of the deformation field T0, E_total(Tk) generated by each deformation field Tk is calculated. If E_total(Tk) does not meet the iteration termination condition, the iteration stops; otherwise, the gradient descent algorithm is used to update the deformation field Tk, and the calculation of E_total is repeated until E_total(Tk) meets the iteration termination condition. The final obtained deformation field... That is, the target deformation field, at this time, E_total( The minimum value, or the value that satisfies the iteration termination condition.

[0086] For each vertex of the first CAD model ,use ( ) Calculate and obtain the position of the vertex after deformation.

[0087] Optionally, after obtaining the positions of the deformed vertices, the deformed vertex set can be obtained, and the mesh topology can be reconstructed based on the deformed vertex set (e.g., using Open3D's Poisson reconstruction to repair triangular faces). The reconstructed mesh topology can then be optimized to obtain the target CAD model. Specifically, optimizing the reconstructed mesh topology can include performing processes such as removing overlapping surfaces, repairing damaged meshes, and remapping textures.

[0088] In this embodiment, a total energy function E_total(T) is constructed based on vertex correspondence, comprising a data fitting function E_data(T) and a smoothing constraint function E_smooth(T). The target deformation field T, which minimizes the energy function value, is then solved iteratively. This target deformation field is then applied to the first CAD model to complete geometric correction. By precisely matching the vertices of the target CAD model with those of the 3D visual mesh model, geometric deviations between the CAD model and the actual object are eliminated. Furthermore, the target deformation field ensures the continuity and smoothness of the CAD model, avoiding geometric distortions such as local wrinkles and breaks, thus obtaining a target CAD model that combines engineering accuracy with morphological consistency.

[0089] Optionally, the smoothing constraint function E_smooth(T) is the sum of the transformation differences corresponding to at least one vertex pair. The transformation difference corresponding to the vertex pair can be the product of the transformation coefficients and the transformation difference measure of the vertex pair. The transformation difference measure is determined by the difference between the position of vertex j after applying its own transformation and the position of vertex j after applying the transformation of vertex i. Vertex j is the vertex in the first CAD model of the vertex pair.

[0090] In this embodiment, the smoothing constraint function E_smooth(T) is defined as the sum of the transformation differences of each vertex pair. The transformation difference is determined by the product of the transformation coefficients of the vertex pair and the transformation difference metric. The transformation difference metric is obtained by comparing the difference between the transformed position of vertex j in the first CAD model and the transformed position of the matched vertex i. This achieves transformation consistency calculation, effectively reduces the geometric distortion problem in the model deformation process, ensures the continuity and smoothness of the surface of the corrected target CAD model, and finally generates a target CAD model with realistic shape and stable structure.

[0091] The main advantage of employing a smoothing constraint based on vertex pair transformation differences is its ability to more precisely control local details during model deformation, reduce geometric distortion, and maintain the continuity and smoothness of the model surface. Specifically, this constraint method has the following advantages: 1. Precise control of local deformation: Traditional smoothing constraint methods (such as Laplacian operator based on deformation field, gradient, or local stiffness penalty) often smooth the model as a whole, which may ignore some local details. However, smoothing constraints based on the transformation difference of vertex pairs can make fine adjustments for each vertex pair, ensuring the accuracy of local details.

[0092] 2. Reduce geometric distortion: During model deformation, local wrinkles, breaks, and other issues can lead to geometric distortion. By calculating the transformation differences between vertex pairs and using them as smoothing constraint functions, these problems can be avoided more effectively, ensuring that the model maintains a good geometric structure during deformation.

[0093] like Figure 7 The diagram shown is a flowchart of a digital twin model generation method provided in an embodiment of this application. The digital twin model generation method may include the following steps: S701. Extract foreground component images and background images from the image sequence corresponding to the physical entity. The foreground component image refers to a local image containing components in the physical entity, and the background image refers to a local image not containing components in the physical entity.

[0094] Some steps in this embodiment are the same as those in other embodiments, and will not be repeated here.

[0095] S702. Perform 3D reconstruction of the components in the foreground component image to obtain a 3D visual mesh model.

[0096] S703. Based on a 3D visual mesh model, obtain a target CAD model that matches the component in the foreground component image.

[0097] S704. Reconstruct the environment from the background image to obtain an HDR (High Dynamic Range) environment map; S705. Map the HDR environment map corresponding to the background image, the target CAD model, and the 3D visual mesh model to the same coordinate system of the digital twin scene to obtain a digital twin model of the physical entity.

[0098] like Figure 6 The diagram shown is an example of the application of geometry correction and texture mapping provided in an embodiment of this application. (Reference) Figure 6 After obtaining the first CAD model 601 of the desk lamp, geometric corrections and texturing can be performed on the first CAD model 601 to obtain the target CAD model 602. The target CAD model 602 is a model with relatively accurate geometric features such as size and shape.

[0099] like Figure 6 As shown, the first CAD model obtained after geometric correction of the table lamp's CAD model does not contain detailed information such as texture. After step S404, the target CAD model 602 obtained after applying textures to the first CAD model obtained after geometric correction can include detailed information such as texture.

[0100] Optionally, the HDR environment map corresponding to the background image, the target CAD model, and the 3D visual mesh model are mapped to the same coordinate system of the digital twin scene to obtain a digital twin model of the physical entity. This includes: aligning the HDR environment map with the reference coordinate system (e.g., the sky part of the HDR map corresponds to the positive Z-axis direction, and the ground corresponds to the negative Z-axis direction); in the digital twin engine, using the HDR map as an ambient light source and binding it to the origin of the reference coordinate system to obtain a digital twin model of the physical entity.

[0101] The digital twin model includes: an aligned 3D visual mesh model (baseline topology); an aligned target CAD model (refined structure); and an HDR environment map bound to a coordinate system (ambient lighting).

[0102] After obtaining the digital twin model, you can check whether the model's position, size, and lighting are consistent with the physical entity (such as building height, road width, and shadow direction). If they are inconsistent, you can manually adjust them or trigger an update again.

[0103] After obtaining the digital twin model, it can be exported as a digital twin model file (such as GLB (Graphics Library Transmission) / USDZ (Universal Scene Description Zip)), or the scene can be published directly in the engine.

[0104] In this embodiment, an HDR environment map is generated by reconstructing the background image, accurately restoring the complete texture and environmental details of the scene in which the physical entity is located. Furthermore, the HDR environment map, the target CAD model, and the 3D visual mesh model are mapped to the same digital twin scene coordinate system, achieving deep fusion of geometric, textural, and environmental elements. This enables the digital twin model to possess lighting and shadow rendering effects consistent with the real physical entity, ensuring the spatial position matching accuracy of each element. The resulting digital twin model is a physical entity digital twin model that combines accurate geometric shape, realistic environmental features, and immersive visual effects.

[0105] As an example, S704, environment reconstruction is performed on the background image to obtain an HDR environment map, including: A panoramic image is obtained by using a spherical mapping method to apply a panoramic texture to the background image. The panoramic image is rendered to obtain an HDR environment map.

[0106] Optionally, a spherical mapping method is used to perform panoramic mapping on the background image to obtain a panoramic image, which includes: projecting the background image onto a virtual spherical coordinate system and unfolding it into a 360° equirectangular panoramic image to obtain a panoramic image.

[0107] Image rendering processing of panoramic images to obtain HDR environment maps can include: performing dynamic range expansion, radiosity calibration, and lighting optimization on spherical panoramic images, and then converting ordinary LDR panoramic images into HDR environment maps that retain full brightness information.

[0108] In this embodiment, a spherical mapping method is used to stitch the background image into a 360-degree panoramic texture, eliminating seams and geometric distortions in multi-view images and fully restoring the panoramic environmental information of the scene where the physical entity is located. Then, professional image rendering processing is performed on the panoramic image to extract and retain texture details in the scene, generating an HDR environment texture with accurate texture characteristics. This provides highly realistic texture details for the subsequent construction of the digital twin model, significantly improving the consistency between the 3D scene and the real environment.

[0109] Optionally, S402 or S702, performing 3D reconstruction of the components in the foreground component image to obtain a 3D visual mesh model, may include: The Poisson surface reconstruction algorithm is used to reconstruct the three-dimensional components in the foreground component image to obtain a three-dimensional visual mesh model.

[0110] In this embodiment, the components in the foreground component image are reconstructed in three dimensions by Poisson surface reconstruction, so as to extract a complete three-dimensional visual mesh model of the component that can be accurately reconstructed from the foreground component image. This allows the three-dimensional visual mesh model to accurately restore the fine geometric features and complete topological structure of the component, providing a highly reliable three-dimensional morphological benchmark for the subsequent construction of digital twin models.

[0111] like Figure 8 The diagram shown is an example of a digital twin model generation method provided in an embodiment of this application. The digital twin model generation method includes: S801. Acquire the image sequence of physical entities captured by the surround view camera.

[0112] S802, Foreground and Background Segmentation, that is, extracting foreground component images and background images from the image sequence corresponding to the physical entity. Foreground component images refer to local images containing components in the physical entity, and background images refer to local images that do not contain components in the physical entity.

[0113] S803, 3D reconstruction, that is, to perform 3D reconstruction of the components in the foreground component image to obtain a 3D visual mesh model.

[0114] S804, Semantic recognition, that is, performing semantic recognition on the 3D visual mesh model to obtain the semantic label of the 3D visual mesh model.

[0115] S805, CAD model library matching, that is, querying the CAD model library for the first CAD model that matches the semantic tag based on the semantic tag.

[0116] S806, Geometric correction, that is, based on the three-dimensional visual mesh model, the first CAD model is geometrically corrected to obtain the corrected target CAD model.

[0117] S807, Texture Reconstruction, which is to reconstruct the environment of the background image to obtain an HDR environment map.

[0118] S808, model generation, is to map the HDR environment map corresponding to the background image, the target CAD model, and the 3D visual mesh model into the same coordinate system of the digital twin scene to obtain a digital twin model of the physical entity.

[0119] In this embodiment, the system starts with a sequence of physical entity images captured by a surround-view camera. First, foreground and background segmentation is used to accurately separate the entity components from the background image, focusing the core reconstruction object and eliminating environmental interference. Then, 3D reconstruction of the foreground component image is performed to obtain a 3D visual mesh model. Combined with semantic recognition, the model is given structured semantic labels, thus accurately matching the general first CAD model in the CAD model library. Geometric correction ensures the general model conforms to the measured geometric shape of the physical entity, avoiding deviations between the general model and the actual shape of the entity. Simultaneously, the HDR environment map reconstructed from the background image restores the full dynamic range of light and shadow information of the environment in which the entity is located. Finally, the HDR environment map, the corrected target CAD model, and the 3D visual mesh model are unified into the same digital twin scene coordinate system, achieving a full-dimensional accurate replication of the physical entity from geometric structure and semantic attributes to environmental light and shadow. This constructs a highly realistic digital twin model with a unified coordinate system, achieving end-to-end automation from multi-view image acquisition to digital twin model generation. This significantly reduces manual modeling costs and improves the fidelity and practicality of the digital twin scene in reproducing the physical entity.

[0120] like Figure 9 The diagram shown is a structural schematic of a digital twin model generation device provided in an embodiment of this application. The digital twin model generation device 900 may include the following units: Image segmentation unit 901 is used to extract foreground component images and background images from image sequences corresponding to physical entities. Foreground component images refer to local images containing components in the physical entity, and background images refer to local images that do not contain components in the physical entity. The 3D reconstruction unit 902 is used to perform 3D reconstruction of the components in the foreground component image to obtain a 3D visual mesh model. Model matching unit 903 is used to obtain a target CAD model that matches the component in the foreground component image based on a 3D visual mesh model. Model generation unit 904 is used to generate digital twin models of physical entities based on background images, target CAD models, and 3D visual mesh models.

[0121] As one embodiment, the model matching unit 903 includes: The model query module is used to query the first CAD model that matches the 3D visual mesh model from the CAD model library.

[0122] The model deformation module is used to perform geometric corrections on the first CAD model based on the 3D visual mesh model to obtain the corrected target CAD model.

[0123] As another example, the model query module includes: The semantic recognition submodule is used to perform semantic recognition on the 3D visual mesh model and obtain the semantic labels of the 3D visual mesh model.

[0124] The semantic matching submodule is used to query the CAD model library for the first CAD model that matches the semantic tag based on the semantic tag.

[0125] As another embodiment, the model deformation module includes: The vertex matching submodule is used to perform vertex matching between the 3D visual mesh model and the first CAD model to obtain vertex correspondence. The vertex correspondence includes at least one vertex pair, and each vertex pair includes vertices in the 3D visual mesh model and vertices in the first CAD model. The model deformation submodule is used to perform non-rigid deformation on the first CAD model based on the non-rigid iterative nearest point algorithm and vertex correspondence to obtain the corrected target CAD model.

[0126] As yet another example, the model deformation submodule is specifically used for: Based on at least one vertex pair in the vertex correspondence, an energy function is constructed. The energy function E_total(T) is the sum of the deformation field T acting on the first CAD model, which is formed by multiplying the smoothing constraint function E_smooth(T) and the balance adjustment parameter, and then adding it to the data fitting function E_data(T). The deformation field T acting on the first CAD model is solved until the iteration target of minimizing the energy function is reached, thus obtaining the target deformation field acting on the first CAD model when the iteration stops. The first CAD model is deformed according to the target deformation field to obtain the corrected target CAD model.

[0127] As another embodiment, the smoothing constraint function E_smooth(T) is the sum of the transformation differences corresponding to at least one vertex pair. The transformation difference corresponding to the vertex pair can be the product of the transformation coefficients and the transformation difference measure of the vertex pair. The transformation difference measure is determined by the position difference between the position of vertex j after applying its own transformation and the position of vertex j after applying the transformation of vertex i. Vertex j is the vertex in the first CAD model of the vertex pair.

[0128] As yet another embodiment, it also includes: The environment reconstruction unit is used to reconstruct the environment of the background image and obtain an HDR environment map. Model generation unit 904 includes: The model mapping module is used to map the HDR environment map corresponding to the background image, the target CAD model, and the 3D visual mesh model into the same coordinate system of the digital twin scene to obtain a digital twin model of the physical entity.

[0129] As another embodiment, the environment reconstruction unit includes: The spherical mapping module is used to apply a spherical mapping method to the background image to create a 390-degree panoramic texture, thereby obtaining a panoramic image.

[0130] The image rendering module is used to perform image rendering processing on panoramic images to obtain HDR environment maps.

[0131] As another embodiment, the three-dimensional reconstruction unit 902 includes: The 3D reconstruction module is used to perform 3D reconstruction of components in the foreground component image using the Poisson surface reconstruction algorithm to obtain a 3D visual mesh model.

[0132] In the embodiments of this application, Figure 9 The device shown can also be a chip or a chip system, such as a system on chip (SoC) or a baseboard management controller (BMC).

[0133] Figure 10 This is a hardware block diagram of a computing device provided in an embodiment of this application. The computing device 1000 according to an embodiment of this application includes at least a memory 1001 and a processor 1002. The memory 1001 is used to store computer programs. The processor 1002 is used to execute the computer programs to implement the digital twin model generation method of any of the above embodiments.

[0134] In addition, both the memory 1001 and the processor 1002 are electrically connected to the bus 1003.

[0135] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program. When executed by a processor, the computer program implements the digital twin model generation method of any of the preceding embodiments of this application.

[0136] Computer-readable storage media include, but are not limited to, volatile storage media and / or non-volatile storage media. Volatile storage media may include, for example, random access storage media (RAM) and / or cache storage media. Non-volatile storage media may include, for example, read-only storage media (ROM), hard disks, flash memory, optical disks, magnetic disks, etc.

[0137] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the digital twin model generation method of any of the preceding embodiments of this application.

[0138] The basic principles of the embodiments of this application have been described above with reference to specific examples. However, it should be noted that the advantages, benefits, and effects mentioned in the embodiments of this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the embodiments of this application from necessarily employing the aforementioned specific details.

[0139] The block diagrams of devices, apparatuses, devices, and systems involved in the embodiments of this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context explicitly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0140] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0141] It should also be noted that in the systems and methods of this application embodiment, each component or step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the embodiments of this application.

[0142] Various changes, substitutions, and modifications can be made to the technology herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the embodiments of this application is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0143] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use embodiments of this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of embodiments of this application. Therefore, embodiments of this application are not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0144] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for generating a digital twin model, characterized in that, include: Extract foreground component images and background images from the image sequence corresponding to the physical entity. The foreground component image refers to a partial image containing a component in the physical entity, and the background image refers to a partial image not containing a component in the physical entity. The components in the foreground component image are reconstructed in three dimensions to obtain a three-dimensional visual mesh model; Based on the three-dimensional visual mesh model, a target computer-aided design CAD model matching the component in the foreground component image is obtained; A digital twin model of the physical entity is generated based on the background image, the target CAD model, and the 3D visual mesh model.

2. The method according to claim 1, characterized in that, The step of obtaining a target CAD model that matches the component in the foreground component image based on the three-dimensional visual mesh model includes: Query the CAD model library for the first CAD model that matches the 3D visual mesh model; Based on the three-dimensional visual mesh model, the first CAD model is geometrically corrected to obtain the corrected target CAD model.

3. The method according to claim 2, characterized in that, The step of querying the CAD model library for a first CAD model that matches the 3D visual mesh model includes: Semantic recognition is performed on the three-dimensional visual mesh model to obtain semantic labels for the three-dimensional visual mesh model; Based on the semantic tag, query the CAD model library for the first CAD model that matches the semantic tag.

4. The method according to claim 2 or 3, characterized in that, The step of geometrically correcting the first CAD model based on the three-dimensional visual mesh model to obtain the corrected target CAD model includes: Vertex matching is performed on the 3D visual mesh model and the first CAD model to obtain vertex correspondence. The vertex correspondence includes at least one vertex pair, and each vertex pair includes vertices in the 3D visual mesh model and vertices in the first CAD model. Based on the non-rigid iterative nearest point algorithm and the vertex correspondence, the first CAD model is subjected to non-rigid deformation to obtain the corrected target CAD model.

5. The method according to claim 4, characterized in that, The step of performing non-rigid deformation on the first CAD model according to the non-rigid iterative nearest point algorithm and the vertex correspondence to obtain the corrected target CAD model includes: Based on at least one vertex pair in the vertex correspondence, an energy function is constructed. The energy function is the function formed by multiplying the deformation field T acting on the first CAD model by the smoothing constraint function and the balance adjustment parameter, and then adding it to the data fitting function. The deformation field T acting on the first CAD model is solved until the minimum value of the energy function is reached, and the target deformation field acting on the first CAD model at the time of stopping iteration is obtained. The first CAD model is deformed according to the target deformation field to obtain the corrected target CAD model.

6. The method according to claim 5, characterized in that, The smoothing constraint function is the sum of the transformation differences corresponding to the at least one vertex pair. The transformation difference corresponding to the vertex pair can be the product of the transformation coefficients and the transformation difference metric of the vertex pair. The transformation difference metric is determined by the difference between the position of vertex j after applying its own transformation and the position of vertex j after applying the transformation of vertex i. Vertex j is a vertex in the first CAD model of the vertex pair.

7. The method according to any one of claims 1-6, characterized in that, Also includes: The background image is reconstructed to obtain a high dynamic range (HDR) environment map; The process of generating a digital twin model of the physical entity based on the background image, the target CAD model, and the 3D visual mesh model includes: The HDR environment map corresponding to the background image, the target CAD model, and the 3D visual mesh model are mapped into the same coordinate system of the digital twin scene to obtain the digital twin model of the physical entity.

8. The method according to claim 7, characterized in that, The step of reconstructing the environment from the background image to obtain an HDR environment map includes: A panoramic image is obtained by using a spherical mapping method to perform panoramic mapping on the background image; The panoramic image is rendered to obtain the HDR environment map.

9. The method according to any one of claims 1-7, characterized in that, The step of performing 3D reconstruction of the components in the foreground component image to obtain a 3D visual mesh model includes: The components in the foreground component image are reconstructed in three dimensions using the Poisson surface reconstruction algorithm to obtain a three-dimensional visual mesh model.

10. A computing device, characterized in that, include: A processor and a memory, the memory storing a computer program that the processor uses to invoke the digital twin model generation method according to any one of claims 1-9.

Images