Digital twin equipment modeling method, device, equipment, medium and program product
Through multi-view image neural rendering and cross-modal feature alignment network, the problems of low efficiency in building digital twin device models and insufficient dynamic performance reflection are solved, and efficient and accurate digital twin device modeling is achieved.
Patent Information
- Application Number
- CN202510490442.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-18
- Publication Date
- 2025-08-08
AI Technical Summary
The existing digital twin device model construction methods are inefficient and cannot truly reflect the dynamic performance of the device. Manual modeling is prone to errors, and static data acquisition cannot capture dynamic changes of the device.
By acquiring images from multiple perspectives of the target device, neural rendering is performed to generate a rendered point cloud model, and aligning the feature with the existing three-dimensional model library is used to match and generate a digital twin device model.
It realizes rapid and accurate modeling of digital twin equipment models, improves modeling efficiency and quality, and can efficiently reuse existing model resources.
Smart Images
Figure CN120451440A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of model building technology, and in particular to a digital twin device modeling method, device, electronic device, computer-readable storage medium and computer program product. Background Art
[0002] With the construction of smart cities, the continuous development of the industrial Internet and the digital transformation of various industries, digital twin modeling technology is currently widely used to model equipment, such as urban facilities and industrial equipment, to achieve intelligent management of equipment.
[0003] The construction of existing digital twin device models usually adopts manual modeling methods or modeling methods based on static data collection. Among them, the manual modeling method relies on engineers to manually create digital twin device models. The process is cumbersome and time-consuming. This not only requires professional knowledge and experience, but may also cause the digital twin device model to be inaccurate due to human errors, resulting in low modeling efficiency of this method; and the modeling method based on static data collection uses static sensors (such as laser scanners) for data collection, which usually cannot capture the dynamic changes of the device in actual operation, resulting in the final generated digital twin device model may not match the actual operating status, and cannot truly reflect the dynamic performance of the device, affecting the actual application effect of the digital twin device model. Summary of the Invention
[0004] The present invention provides a digital twin device modeling method, apparatus, equipment, medium and program product, which can realize fast and accurate modeling of digital twin device models based on the efficient reuse of existing model resources, significantly improving the efficiency and quality of digital twin device modeling.
[0005] In order to solve the above technical problems, the first aspect of the embodiment of the present invention provides a digital twin device modeling method, including:
[0006] Acquire multi-view captured images of a target device and a three-dimensional model library corresponding to the target device;
[0007] Performing neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device;
[0008] Inputting the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model;
[0009] The first alignment feature vector and the second alignment feature vector are matched to select a digital twin device model corresponding to the target device from the three-dimensional model library.
[0010] As a preferred solution, the step of inputting the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model specifically includes:
[0011] Inputting the rendered point cloud model and each of the three-dimensional device models into the cross-modal feature alignment network, respectively extracting a target device feature vector from the rendered point cloud model and extracting a multi-view image feature vector corresponding to each of the three-dimensional device models;
[0012] Feature alignment is performed on the target device feature vector and the multi-view image feature vector to obtain the first aligned feature vector and the second aligned feature vector.
[0013] As a preferred solution, performing neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device specifically includes:
[0014] Extracting a plurality of three-dimensional sampling points of the target device from the multi-view captured images according to camera pose parameters corresponding to the multi-view captured images;
[0015] Obtaining a directed distance value from each of the three-dimensional sampling points to the surface of the target device through a preset directed distance field network;
[0016] Extracting a plurality of target three-dimensional sampling points with directed distance values of 0 from the plurality of three-dimensional sampling points, and connecting the plurality of target three-dimensional sampling points into a plurality of triangular facets to form a surface mesh of the target device;
[0017] Surface point cloud sampling is performed on the surface mesh to generate a rendering point cloud model corresponding to the target device.
[0018] As a preferred solution, the method specifically extracts the target device feature vector from the rendered point cloud model through the following steps:
[0019] Calculating the distance between each point based on the position information of each point in the rendered point cloud model;
[0020] Generating a point cloud adjacency graph corresponding to the rendered point cloud model according to the distance and a preset adjacency distance threshold;
[0021] A graph convolutional network is used to extract features from the point cloud adjacency graph to obtain the target device feature vector.
[0022] As a preferred solution, the method specifically extracts the multi-view image feature vector corresponding to each 3D device model in the 3D model library through the following steps:
[0023] generating a multi-view image of each of the three-dimensional device models based on different view angles;
[0024] A multi-view convolutional neural network is used to extract features from the multi-view image to obtain a feature vector of the multi-view image.
[0025] As a preferred solution, the method specifically trains the cross-modal feature alignment network through the following steps:
[0026] Performing surface point cloud sampling on each of the three-dimensional device models to obtain a plurality of training point cloud models;
[0027] Inputting the 3D device model and the training point cloud model into a semantic alignment network based on a Siamese network architecture, and utilizing a multi-view convolutional neural network and a graph convolutional network in the semantic alignment network to respectively extract a multi-view image feature training set of the 3D device model and a point cloud feature training set of the training point cloud model;
[0028] Based on the contrast loss function, the semantic alignment network is trained using the multi-view image feature training set and the point cloud feature training set to obtain the cross-modal feature alignment network.
[0029] As a preferred solution, after obtaining the cross-modal feature alignment network, the method further includes:
[0030] The cross-modal feature alignment network is fine-tuned using the multi-view captured images and the rendered point cloud model.
[0031] As a preferred solution, fine-tuning the cross-modal feature alignment network using the multi-view captured images and the rendered point cloud model specifically includes:
[0032] Fixing the convolutional layer parameters of the multi-view convolutional neural network and the graph convolutional network;
[0033] The multi-view captured images and the rendered point cloud model are input into the cross-modal feature alignment network, and the multi-view captured images and the rendered point cloud model are used to fine-tune the fully connected layer parameters of the multi-view convolutional neural network and the graph convolutional network.
[0034] As a preferred solution, matching the first alignment feature vector and the second alignment feature vector to select the digital twin device model corresponding to the target device from the three-dimensional model library specifically includes:
[0035] calculating the Mahalanobis distance between the first alignment feature vector and each of the second alignment feature vectors;
[0036] determining, according to a minimum value of the Mahalanobis distance, a target alignment feature vector that matches the first alignment feature vector from a plurality of second alignment feature vectors;
[0037] A target three-dimensional device model corresponding to the target alignment feature vector is selected from the three-dimensional model library, and the target three-dimensional device model is used as the digital twin device model corresponding to the target device.
[0038] A second aspect of an embodiment of the present invention provides a digital twin device modeling apparatus, comprising:
[0039] A data acquisition module is used to acquire multi-view captured images of a target device and a three-dimensional model library corresponding to the target device;
[0040] A neural rendering module, configured to perform neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device;
[0041] a feature alignment module, configured to input the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model;
[0042] A model reuse module is used to match the first alignment feature vector and the second alignment feature vector to select a digital twin device model corresponding to the target device from the three-dimensional model library.
[0043] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the digital twin device modeling method described in any one of the first aspects is implemented.
[0044] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the digital twin device modeling method described in any one of the first aspects.
[0045] A fifth aspect of an embodiment of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the digital twin device modeling method described in any one of the first aspects.
[0046] Compared with the existing technology, the beneficial effect of the embodiments of the present invention is that by performing neural rendering on the multi-view captured images of the target device, a high-fidelity rendered point cloud model can be generated, and the generated rendered point cloud model is feature aligned with the existing three-dimensional models corresponding to the target device using a cross-modal feature alignment network, and matched based on the obtained alignment feature vector. This can achieve fast and accurate modeling of digital twin device models based on the efficient reuse of existing model resources, significantly improving the efficiency and quality of digital twin device modeling. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is a flow chart of a digital twin device modeling method in an embodiment of the present invention;
[0048] Figure 2 is an architectural diagram of a semantic alignment network based on a twin network architecture in an embodiment of the present invention;
[0049] Figure 3 is a structural diagram of a digital twin device modeling apparatus in an embodiment of the present invention;
[0050] Figure 4 It is a schematic structural diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0052] See Figure 1 A first aspect of an embodiment of the present invention provides a digital twin device modeling method, comprising the following steps S1 to S4:
[0053] Step S1, obtaining multi-view captured images of a target device and a three-dimensional model library corresponding to the target device;
[0054] Step S2, performing neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device;
[0055] Step S3: inputting the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model;
[0056] Step S4: Match the first alignment feature vector and the second alignment feature vector to select a digital twin device model corresponding to the target device from the three-dimensional model library.
[0057] Specifically, this embodiment first obtains multi-perspective images of the target device from different perspectives. For example, this embodiment can use an RGB camera to capture multi-perspective images of the target device from different perspectives, thereby providing surface information of the target device at different angles, and simultaneously recording the camera pose parameters, which include the position and direction of the camera, for mapping the two-dimensional image information to three-dimensional space. Furthermore, this embodiment also obtains a three-dimensional model library corresponding to the target device. It can be understood that the three-dimensional model library in this embodiment is an existing model resource. When obtaining the existing three-dimensional model library, it is necessary to match the device type of the target device. For example, if the target device is a belt conveyor, it is necessary to obtain an existing belt conveyor model library, which contains various different belt conveyor three-dimensional models. This embodiment will not be described in detail here.
[0058] Furthermore, in order to improve the accuracy and efficiency of generating point cloud models of complex industrial equipment and obtain high-fidelity point cloud models, this embodiment performs neural rendering on the acquired multi-view captured images to generate a rendered point cloud model corresponding to the target device. Because the features corresponding to the rendered point cloud model and the features corresponding to the 3D device model are features of two different modalities and cannot be directly matched, this embodiment inputs the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to align the features of the rendered point cloud model with the features of the 3D device model, thereby mapping the features of the two different modalities to the same feature space, ensuring that they can be compared and matched in the same feature space.
[0059] Furthermore, the first alignment feature vector corresponding to the rendered point cloud model output by the cross-modal feature alignment network is matched with the second alignment feature vector corresponding to each three-dimensional device model, so that the model with the highest matching degree can be selected from the existing three-dimensional models as the reuse model, which significantly improves the retrieval and reuse efficiency of digital twin device models, especially in industrial scenarios that require rapid response and efficient modeling.
[0060] Preferably, after selecting the reuse model with the highest matching degree, the selected reuse model can be further subjected to geometric calibration, detail optimization, attribute assignment and dynamic simulation verification to ensure that the reuse model can accurately reflect the actual status and performance of the target device.
[0061] The digital twin device modeling method provided by an embodiment of the present invention can generate a high-fidelity rendered point cloud model by performing neural rendering on multi-view captured images of the target device. It uses a cross-modal feature alignment network to perform feature alignment on the generated rendered point cloud model with the existing three-dimensional models corresponding to the target device, and matches them based on the obtained alignment feature vectors. It can achieve fast and accurate modeling of digital twin device models based on the efficient reuse of existing model resources, significantly improving the efficiency and quality of digital twin device modeling.
[0062] As a preferred solution, the step of inputting the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model specifically includes:
[0063] Inputting the rendered point cloud model and each of the three-dimensional device models into the cross-modal feature alignment network, respectively extracting a target device feature vector from the rendered point cloud model and extracting a multi-view image feature vector corresponding to each of the three-dimensional device models;
[0064] Feature alignment is performed on the target device feature vector and the multi-view image feature vector to obtain the first aligned feature vector and the second aligned feature vector.
[0065] Specifically, this embodiment uses a cross-modal feature alignment network to extract corresponding point cloud features from the rendered point cloud model as target device feature vectors, and extracts multi-view image feature vectors corresponding to each three-dimensional device model, and then performs feature alignment on the extracted target device feature vectors and multi-view image feature vectors to map the target device feature vectors and multi-view image feature vectors of different modalities to the same feature space, thereby obtaining a first aligned feature vector and a second aligned feature vector.
[0066] As a preferred solution, performing neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device specifically includes:
[0067] Extracting a plurality of three-dimensional sampling points of the target device from the multi-view captured images according to camera pose parameters corresponding to the multi-view captured images;
[0068] Obtaining a directed distance value from each of the three-dimensional sampling points to the surface of the target device through a preset directed distance field network;
[0069] Extracting a plurality of target three-dimensional sampling points with directed distance values of 0 from the plurality of three-dimensional sampling points, and connecting the plurality of target three-dimensional sampling points into a plurality of triangular facets to form a surface mesh of the target device;
[0070] Surface point cloud sampling is performed on the surface mesh to generate a rendering point cloud model corresponding to the target device.
[0071] Specifically, this embodiment uses the camera pose parameters corresponding to the multi-view captured images to extract several three-dimensional sampling points on the surface of the target device from the multi-view captured images. These three-dimensional sampling points are represented by three-dimensional space coordinates. express.
[0072] Furthermore, the extracted three-dimensional sampling points are input into a preset signed distance field network (Signed Distance Function Network), so that the signed distance value from each three-dimensional sampling point output by the signed distance field network to the surface of the target device can be obtained. Furthermore, this embodiment uses the NeuS (Neural Implicit Surface Rendering) renderer to output the surface mesh model of the target device. Specifically, the zero level set is first extracted from each signed distance value, and the surface of the target device is constructed by finding three-dimensional sampling points with a signed distance value of zero in the three-dimensional mesh and generating triangular patches based on these three-dimensional sampling points. Once the zero level set is extracted, the surface mesh model of the target device can be generated. The generation of the surface mesh model includes triangulation and post-processing, wherein triangulation is to connect the three-dimensional sampling points of the zero level set into several triangular patches to construct the surface mesh of the target device; post-processing is to perform some post-processing operations on the generated surface mesh, such as smoothing, denoising, etc., to improve the quality of the model.
[0073] Furthermore, surface point cloud sampling is performed on the acquired surface mesh. Exemplarily, this embodiment can use open source three-dimensional graphics and image software such as Blender software tools to convert the surface mesh model into a point cloud model, thereby obtaining a rendered point cloud model of the target device generated based on neural rendering.
[0074] As a preferred solution, the method specifically extracts the target device feature vector from the rendered point cloud model through the following steps:
[0075] Calculating the distance between each point based on the position information of each point in the rendered point cloud model;
[0076] Generating a point cloud adjacency graph corresponding to the rendered point cloud model according to the distance and a preset adjacency distance threshold;
[0077] A graph convolutional network is used to extract features from the point cloud adjacency graph to obtain the target device feature vector.
[0078] Specifically, since the rendered point cloud model is obtained by neural rendering based on several three-dimensional sampling points, and each three-dimensional sampling point has corresponding three-dimensional spatial coordinates, the position information of each point in the rendered point cloud model can be directly obtained, so that the distance between each point can be further calculated. Then, based on the distance between each point and the preset adjacent distance threshold, it is determined whether different points are adjacent points. For example, any two points whose distance is less than or equal to the adjacent distance threshold are regarded as adjacent points, and any two points whose distance is greater than the adjacent distance threshold are regarded as non-adjacent points. Finally, a point cloud adjacency graph corresponding to the rendered point cloud model is generated. It is worth noting that each point in the rendered point cloud model has a certain spatial relationship with the points around it. By constructing a point cloud adjacency graph, the adjacent relationship between these points can be clearly represented. Furthermore, a graph convolutional network (GCN) is used to extract features from the generated point cloud adjacency graph. It can be understood that the graph convolutional network can use the adjacency information in the point cloud adjacency graph to better capture the local geometric structure and semantic information of the point cloud, such as identifying object boundaries, surface normals and other features in the point cloud; in addition, since point cloud data is irregular, it does not have a regular grid structure like image data. The graph convolutional network can directly process this irregular graph structure data without complex preprocessing or data conversion, thereby processing point cloud data more naturally and improving the efficiency and accuracy of feature vector extraction of the target device.
[0079] As a preferred solution, the method specifically extracts the multi-view image feature vector corresponding to each 3D device model in the 3D model library through the following steps:
[0080] generating a multi-view image of each of the three-dimensional device models based on different view angles;
[0081] A multi-view convolutional neural network is used to extract features from the multi-view image to obtain a feature vector of the multi-view image.
[0082] Specifically, this embodiment uses open source 3D graphics and image software such as Blender software tools to generate multi-perspective images of each 3D device model at different perspectives for several 3D device models in a 3D model library. Furthermore, an image from a single perspective may not be able to fully present all the features of an object, while images from different perspectives can provide information from different sides of the object. By extracting features from multi-perspective images, information from multiple perspectives can be integrated to more comprehensively and accurately describe the appearance, shape, structure and other features of each 3D device model, thereby improving the ability to recognize, classify and understand 3D device models. Based on this, this embodiment uses a multi-view convolutional neural network to extract features from the generated multi-perspective images to obtain corresponding multi-perspective image feature vectors.
[0083] As a preferred solution, the method specifically trains the cross-modal feature alignment network through the following steps:
[0084] Performing surface point cloud sampling on each of the three-dimensional device models to obtain a plurality of training point cloud models;
[0085] Inputting the 3D device model and the training point cloud model into a semantic alignment network based on a Siamese network architecture, and utilizing a multi-view convolutional neural network and a graph convolutional network in the semantic alignment network to respectively extract a multi-view image feature training set of the 3D device model and a point cloud feature training set of the training point cloud model;
[0086] Based on the contrast loss function, the semantic alignment network is trained using the multi-view image feature training set and the point cloud feature training set to obtain the cross-modal feature alignment network.
[0087] Specifically, in order to obtain a large amount of point cloud data for training the semantic alignment network based on the twin network architecture, this embodiment performs surface point cloud sampling on each 3D device model, thereby obtaining a training point cloud model corresponding to each 3D device model. Furthermore, the architecture of the semantic alignment network in this embodiment is as follows: Figure 2 As shown, the three-dimensional device model and the training point cloud model are input into the semantic alignment network. In order to ensure the normal progress of feature extraction, this embodiment first needs to process the three-dimensional device model and the training point cloud model. Specifically, multi-view images of each three-dimensional device model are generated based on different viewpoints, and then the corresponding training point cloud adjacency graph is generated based on the position information of each point in the training point cloud model. Then, the parallel multi-view convolutional neural network and graph convolutional network are used to perform feature extraction on the generated multi-view images and the training point cloud adjacency graph, respectively, to obtain a multi-view image feature training set and a point cloud feature training set.
[0088] Furthermore, this embodiment uses the contrast loss function as the loss function of the semantic alignment network. The contrast loss function is specifically as follows:
[0089]
[0090] Where Y represents the label, which is 0 if the aligned multi-view image feature vector and the point cloud feature vector belong to the same device object, and 1 if they belong to different device objects; D represents the Euclidean distance between the multi-view image feature vector and the point cloud feature vector mapped to the same feature space; m is a hyperparameter used to represent the minimum interval between different device objects.
[0091] Specifically, the calculation of the contrastive loss function aims to bring together features that belong to the same device object but express the device in different forms, while pushing features belonging to different device objects further apart. In the semantic alignment process of feature vectors of different modalities for the same device, the contrastive loss function measures the similarity of features based on Euclidean distance. Therefore, this embodiment uses minimizing the contrastive loss function as the training goal, trains the semantic alignment network using a multi-view image feature training set and a point cloud feature training set, and continuously adjusts the model parameters of the multi-view convolutional neural network and the graph convolutional network during the training process, ultimately obtaining a cross-modal feature alignment network.
[0092] As a preferred solution, after obtaining the cross-modal feature alignment network, the method further includes:
[0093] The cross-modal feature alignment network is fine-tuned using the multi-view captured images and the rendered point cloud model.
[0094] It is worth noting that although the point cloud data provided by the three-dimensional device model during the training of the cross-modal feature alignment network is large and comprehensive, there may be slight differences compared to the actual device. The point cloud data generated by neural rendering, although smaller in volume, is more accurate and can well reflect the actual situation of the actual device. Therefore, in order to enable the cross-modal feature alignment network to more accurately capture and align the real features of the target device, this embodiment uses multi-view captured images and rendered point cloud models of the target device to fine-tune the cross-modal feature alignment network.
[0095] As a preferred solution, fine-tuning the cross-modal feature alignment network using the multi-view captured images and the rendered point cloud model specifically includes:
[0096] Fixing the convolutional layer parameters of the multi-view convolutional neural network and the graph convolutional network;
[0097] The multi-view captured images and the rendered point cloud model are input into the cross-modal feature alignment network, and the multi-view captured images and the rendered point cloud model are used to fine-tune the fully connected layer parameters of the multi-view convolutional neural network and the graph convolutional network.
[0098] Specifically, considering that the convolutional layer parameters of the multi-view convolutional neural network and graph convolutional network have already learned stable feature extraction capabilities from training data provided by a large number of 3D device models, this embodiment fixes the convolutional layer parameters of the multi-view convolutional neural network and graph convolutional network to avoid over-adjustment during fine-tuning and maintain the overall feature extraction capability of the cross-modal feature alignment network. Furthermore, the multi-view captured image and rendered point cloud model are input into the cross-modal feature alignment network. First, a corresponding point cloud adjacency graph is generated based on the position information of each point in the rendered point cloud model. Then, the parallel multi-view convolutional neural network and graph convolutional network are used to extract features from the input multi-view captured image and point cloud adjacency graph, respectively. Then, based on the aforementioned contrast loss function, the extracted multi-view captured image features and point cloud features are used to fine-tune the fully connected layer parameters of the multi-view convolutional neural network and graph convolutional network. The fully connected layer parameter adjustment mainly targets the final output feature vector, enabling the cross-modal feature alignment network to more accurately capture and align the real features of the target device.
[0099] As a preferred solution, matching the first alignment feature vector and the second alignment feature vector to select the digital twin device model corresponding to the target device from the three-dimensional model library specifically includes:
[0100] calculating the Mahalanobis distance between the first alignment feature vector and each of the second alignment feature vectors;
[0101] determining, according to a minimum value of the Mahalanobis distance, a target alignment feature vector that matches the first alignment feature vector from a plurality of second alignment feature vectors;
[0102] A target three-dimensional device model corresponding to the target alignment feature vector is selected from the three-dimensional model library, and the target three-dimensional device model is used as the digital twin device model corresponding to the target device.
[0103] Specifically, this embodiment uses the Mahalanobis distance to match the first aligned feature vector and each second aligned feature vector, mapped to the same feature space. It is worth noting that the Mahalanobis distance takes into account the correlation between feature vectors and better reflects the true differences between features than the Euclidean distance, thus ensuring the accuracy of the matching results. Furthermore, the minimum value of the calculated Mahalanobis distance indicates that the current first aligned feature vector is the most closely matched to the second aligned feature vector corresponding to that Mahalanobis distance. This indicates that the 3D device model corresponding to the matched second aligned feature vector is the reused model of the target device and is used as the desired digital twin device model.
[0104] See Figure 3 A second aspect of an embodiment of the present invention provides a digital twin device modeling apparatus, comprising:
[0105] A data acquisition module 101 is used to acquire multi-view captured images of a target device and a three-dimensional model library corresponding to the target device;
[0106] A neural rendering module 102 is configured to perform neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device;
[0107] A feature alignment module 103 is configured to input the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model;
[0108] The model reuse module 104 is configured to match the first alignment feature vector and the second alignment feature vector to select a digital twin device model corresponding to the target device from the three-dimensional model library.
[0109] As a preferred solution, the feature alignment module 103 is used to input the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model, specifically including:
[0110] Inputting the rendered point cloud model and each of the three-dimensional device models into the cross-modal feature alignment network, respectively extracting a target device feature vector from the rendered point cloud model and extracting a multi-view image feature vector corresponding to each of the three-dimensional device models;
[0111] Feature alignment is performed on the target device feature vector and the multi-view image feature vector to obtain the first aligned feature vector and the second aligned feature vector.
[0112] As a preferred solution, the neural rendering module 102 is used to perform neural rendering on the multi-view captured image to generate a rendering point cloud model corresponding to the target device, specifically including:
[0113] Extracting a plurality of three-dimensional sampling points of the target device from the multi-view captured images according to camera pose parameters corresponding to the multi-view captured images;
[0114] Obtaining a directed distance value from each of the three-dimensional sampling points to the surface of the target device through a preset directed distance field network;
[0115] Extracting a plurality of target three-dimensional sampling points with directed distance values of 0 from the plurality of three-dimensional sampling points, and connecting the plurality of target three-dimensional sampling points into a plurality of triangular facets to form a surface mesh of the target device;
[0116] Surface point cloud sampling is performed on the surface mesh to generate a rendering point cloud model corresponding to the target device.
[0117] As a preferred solution, the feature alignment module 103 is used to extract the target device feature vector from the rendered point cloud model:
[0118] Calculating the distance between each point based on the position information of each point in the rendered point cloud model;
[0119] Generating a point cloud adjacency graph corresponding to the rendered point cloud model according to the distance and a preset adjacency distance threshold;
[0120] A graph convolutional network is used to extract features from the point cloud adjacency graph to obtain the target device feature vector.
[0121] As a preferred solution, the feature alignment module 103 is used to extract the multi-view image feature vector corresponding to each 3D device model in the 3D model library:
[0122] generating a multi-view image of each of the three-dimensional device models based on different view angles;
[0123] A multi-view convolutional neural network is used to extract features from the multi-view image to obtain a feature vector of the multi-view image.
[0124] As a preferred solution, the device further includes a model training module for:
[0125] Performing surface point cloud sampling on each of the three-dimensional device models to obtain a plurality of training point cloud models;
[0126] Inputting the 3D device model and the training point cloud model into a semantic alignment network based on a Siamese network architecture, and utilizing a multi-view convolutional neural network and a graph convolutional network in the semantic alignment network to respectively extract a multi-view image feature training set of the 3D device model and a point cloud feature training set of the training point cloud model;
[0127] Based on the contrast loss function, the semantic alignment network is trained using the multi-view image feature training set and the point cloud feature training set to obtain the cross-modal feature alignment network.
[0128] As a preferred solution, the device further includes a model fine-tuning module for:
[0129] The cross-modal feature alignment network is fine-tuned using the multi-view captured images and the rendered point cloud model.
[0130] As a preferred solution, the model fine-tuning module is used to fine-tune the cross-modal feature alignment network using the multi-view captured images and the rendered point cloud model, specifically including:
[0131] Fixing the convolutional layer parameters of the multi-view convolutional neural network and the graph convolutional network;
[0132] The multi-view captured images and the rendered point cloud model are input into the cross-modal feature alignment network, and the multi-view captured images and the rendered point cloud model are used to fine-tune the fully connected layer parameters of the multi-view convolutional neural network and the graph convolutional network.
[0133] As a preferred solution, the model reuse module 104 is used to match the first alignment feature vector and the second alignment feature vector to select the digital twin device model corresponding to the target device from the three-dimensional model library, specifically including:
[0134] calculating the Mahalanobis distance between the first alignment feature vector and each of the second alignment feature vectors;
[0135] determining, according to a minimum value of the Mahalanobis distance, a target alignment feature vector that matches the first alignment feature vector from a plurality of second alignment feature vectors;
[0136] A target three-dimensional device model corresponding to the target alignment feature vector is selected from the three-dimensional model library, and the target three-dimensional device model is used as the digital twin device model corresponding to the target device.
[0137] The digital twin device modeling device provided by an embodiment of the present invention can generate a high-fidelity rendered point cloud model by performing neural rendering on multi-view captured images of the target device. It uses a cross-modal feature alignment network to perform feature alignment on the generated rendered point cloud model with the existing three-dimensional models corresponding to the target device, and matches them based on the obtained alignment feature vectors. It can achieve fast and accurate modeling of digital twin device models based on the efficient reuse of existing model resources, significantly improving the efficiency and quality of digital twin device modeling.
[0138] See Figure 4 The third aspect of an embodiment of the present invention provides an electronic device, including a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201. When the processor 201 executes the computer program, the digital twin device modeling method described in any embodiment of the first aspect is implemented.
[0139] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 202 and executed by the processor 201 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.
[0140] The electronic device may include, but is not limited to, a processor 201 and a memory 202. Those skilled in the art will appreciate that the schematic diagram is merely an example of an electronic device and does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or different components. For example, the electronic device may also include input and output devices, network access devices, buses, and the like.
[0141] The processor 201 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 201 may be any conventional processor 201, etc. The processor 201 is the control center of the electronic device, connecting various parts of the entire electronic device using various interfaces and lines.
[0142] The memory 202 can be used to store the computer programs and / or modules. The processor 201 implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory 202 and calling the data stored in the memory 202. The memory 202 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 202 can include a high-speed random access memory and a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.
[0143] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the digital twin device modeling method described in any embodiment of the first aspect.
[0144] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the digital twin device modeling method described in any embodiment of the first aspect.
[0145] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0146] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A digital twin device modeling method, characterized in that: include: Acquire multi-view captured images of a target device and a three-dimensional model library corresponding to the target device; Performing neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device; Inputting the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model; The first alignment feature vector and the second alignment feature vector are matched to select a digital twin device model corresponding to the target device from the three-dimensional model library.
2. The digital twin device modeling method according to claim 1, wherein: Inputting the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model specifically includes: Inputting the rendered point cloud model and each of the three-dimensional device models into the cross-modal feature alignment network, respectively extracting a target device feature vector from the rendered point cloud model and extracting a multi-view image feature vector corresponding to each of the three-dimensional device models; Feature alignment is performed on the target device feature vector and the multi-view image feature vector to obtain the first aligned feature vector and the second aligned feature vector.
3. The digital twin device modeling method according to claim 1, wherein: The performing neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device specifically includes: Extracting a plurality of three-dimensional sampling points of the target device from the multi-view captured images according to camera pose parameters corresponding to the multi-view captured images; Obtaining a directed distance value from each of the three-dimensional sampling points to the surface of the target device through a preset directed distance field network; Extracting a plurality of target three-dimensional sampling points with directed distance values of 0 from the plurality of three-dimensional sampling points, and connecting the plurality of target three-dimensional sampling points into a plurality of triangular facets to form a surface mesh of the target device; Surface point cloud sampling is performed on the surface mesh to generate a rendering point cloud model corresponding to the target device.
4. The digital twin device modeling method according to claim 2, wherein: The method specifically extracts the target device feature vector from the rendered point cloud model through the following steps: Calculating the distance between each point based on the position information of each point in the rendered point cloud model; Generating a point cloud adjacency graph corresponding to the rendered point cloud model according to the distance and a preset adjacency distance threshold; A graph convolutional network is used to extract features from the point cloud adjacency graph to obtain the target device feature vector.
5. The digital twin device modeling method according to claim 2, wherein: The method specifically extracts the multi-view image feature vector corresponding to each 3D device model in the 3D model library through the following steps: generating a multi-view image of each of the three-dimensional device models based on different view angles; A multi-view convolutional neural network is used to extract features from the multi-view image to obtain a feature vector of the multi-view image.
6. The digital twin device modeling method according to claim 1, wherein: The method specifically trains the cross-modal feature alignment network through the following steps: Performing surface point cloud sampling on each of the three-dimensional device models to obtain a plurality of training point cloud models; Inputting the 3D device model and the training point cloud model into a semantic alignment network based on a Siamese network architecture, and utilizing a multi-view convolutional neural network and a graph convolutional network in the semantic alignment network to respectively extract a multi-view image feature training set of the 3D device model and a point cloud feature training set of the training point cloud model; Based on the contrast loss function, the semantic alignment network is trained using the multi-view image feature training set and the point cloud feature training set to obtain the cross-modal feature alignment network.
7. The digital twin device modeling method according to claim 6, characterized in that: After obtaining the cross-modal feature alignment network, the method further includes: The cross-modal feature alignment network is fine-tuned using the multi-view captured images and the rendered point cloud model.
8. The digital twin device modeling method according to claim 7, characterized in that: The fine-tuning of the cross-modal feature alignment network using the multi-view captured images and the rendered point cloud model specifically includes: Fixing the convolutional layer parameters of the multi-view convolutional neural network and the graph convolutional network; The multi-view captured images and the rendered point cloud model are input into the cross-modal feature alignment network, and the multi-view captured images and the rendered point cloud model are used to fine-tune the fully connected layer parameters of the multi-view convolutional neural network and the graph convolutional network.
9. The digital twin device modeling method according to claim 1, wherein: Matching the first alignment feature vector and the second alignment feature vector to select the digital twin device model corresponding to the target device from the three-dimensional model library specifically includes: calculating the Mahalanobis distance between the first alignment feature vector and each of the second alignment feature vectors; determining, according to a minimum value of the Mahalanobis distance, a target alignment feature vector that matches the first alignment feature vector from a plurality of second alignment feature vectors; A target three-dimensional device model corresponding to the target alignment feature vector is selected from the three-dimensional model library, and the target three-dimensional device model is used as the digital twin device model corresponding to the target device.
10. A digital twin device modeling device, characterized in that: include: A data acquisition module is used to acquire multi-view captured images of a target device and a three-dimensional model library corresponding to the target device; A neural rendering module, configured to perform neural rendering on the multi-view captured images to generate a rendering point cloud model corresponding to the target device; a feature alignment module, configured to input the rendered point cloud model and each 3D device model in the 3D model library into a pre-trained cross-modal feature alignment network to obtain a first alignment feature vector corresponding to the rendered point cloud model and a second alignment feature vector corresponding to each 3D device model; A model reuse module is used to match the first alignment feature vector and the second alignment feature vector to select a digital twin device model corresponding to the target device from the three-dimensional model library.
11. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the digital twin device modeling method according to any one of claims 1 to 9 when executing the computer program.
12. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the digital twin device modeling method according to any one of claims 1 to 9.
13. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the steps of the digital twin device modeling method described in any one of claims 1 to 9.