Industrial equipment digital twin model rapid reconstruction method and system based on laser point cloud

Abstract

The invention discloses a laser point cloud-based industrial equipment digital twin model rapid reconstruction method and system, and the method combines a model matching mode and a point cloud reconstruction mode, and effectively solves the problem of twin model reconstruction in different scenes. According to the first mode, on the basis of an existing three-dimensional model, through shape matching, feature extraction and model retrieval methods, registration of point cloud data and the three-dimensional model is combined, and rapid reconstruction of the equipment twinborn model is successfully achieved. The mode is suitable for equipment with an existing standard three-dimensional model, the repeated utilization rate of the model can be improved, and the reconstruction time and cost are reduced. In the second mode, the reverse grid model construction method based on the point cloud data is provided for equipment which cannot be retrieved through an existing model. Through an improved greedy projection triangulation algorithm, point cloud data is converted into a grid model, and the precision and authenticity of the model are improved in combination with a grid optimization algorithm and a surface texture mapping technology. According to the method, the digital twinborn model of the industrial equipment can be efficiently and accurately reconstructed, the modeling difficulty in an industrial scene is relieved, the actual demand of field modeling is met, and the method has high application value and practicability.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing, and relates to a method for quickly reconstructing a digital twin model of industrial equipment based on laser point cloud. Background Art

[0002] In traditional industrial scenarios, sensors are usually used to monitor parameters such as equipment temperature and vibration. However, this method can only provide limited information on the operating state of the equipment and cannot comprehensively understand the three-dimensional shape and structure of the equipment. When constructing a twin scenario, especially when using manual modeling to construct an industrial twin scenario, it is also necessary to measure and obtain information such as the size of the reconstruction object in the scenario, and then implement it through professional 3D modeling software. This method of constructing a twin scenario has problems such as poor accuracy of equipment size, low reconstruction efficiency, high manual labor intensity, and high cost when facing large amounts of data or complex objects. For this reason, some digital factories combine digital photography or lidar to obtain size information of the reconstruction object when constructing a geometric model. Among them, digital photography combines the elevation information and texture information on the ground in the scenario to generate a real and accurate 3D model. However, when facing industrial equipment with complex structures, the obtained 3D information is not ideal. Although lidar also has some difficulties in data processing, as a high-precision, non-contact three-dimensional data acquisition method, it provides an ideal data basis for the reconstruction of the industrial equipment twin model.

[0003] After accurately obtaining the three-dimensional data of the reconstruction object, the proposal of many automated modeling methods has further improved the reconstruction efficiency of the twin scenario. Although these methods can produce excellent and realistic reconstruction effects for many scenarios or single objects, for environments containing factors such as a large amount of noise and occlusion, the obtained original data has certain defects, and the rapid three-dimensional reconstruction of such scenarios has not yet reached the required quality level. Especially when dealing with the three-dimensional reconstruction of various industrial scenarios, compared with urban scenarios, common indoor or outdoor scenarios, industrial scenarios are usually more complex, with more types of equipment and more complex three-dimensional shape structures.

[0004] The model construction achieved by the current 3D reconstruction methods is mainly aimed at simple scenes with simple equipment shapes and fewer categories. As the complexity and categories of the equipment in the scene increase, the proprietary descriptiveness of these methods gradually weakens. At the same time, considering that when creating industrial scenes of the same or similar nature in different locations, the industrial equipment models to be created may have the same geometric shape structure, if a refined modeling method is used for each equipment object, the scene reconstruction efficiency will be low and the time cost will be high. This has triggered the need to build a 3D model library of commonly used industrial equipment, and to achieve rapid and detailed reconstruction of industrial scenes by making full use of the repeatability of industrial scenes and the similarity between industrial equipment. Therefore, combining the 3D model database with the reconstruction of industrial scenes has become an urgent task to meet the actual modeling needs. In this context, it is particularly important to study the rapid reconstruction technology of industrial equipment twin models based on laser point clouds, which is expected to provide innovative solutions to the challenges of 3D model construction in current complex scenes. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention proposes a method for quickly reconstructing a digital twin model of industrial equipment based on laser point cloud, which solves the problem that the existing three-dimensional model reconstruction algorithm has poor adaptability to industrial equipment and usually adopts the method of manually constructing three-dimensional models of industrial equipment.

[0006] The steps of the method for rapid reconstruction of the digital twin model of industrial equipment based on laser point cloud include:

[0007] 1) Obtain point cloud data corresponding to physical equipment through multiple scans with a laser scanner. The physical equipment mainly includes various machine tools (lathes, milling machines, machining centers, etc.), various transportation tools (conveyor belts, AGVs, RGVs, etc.), various manipulators, automated warehouses and other equipment (detectors, workbenches, etc.) in industrial scenes.

[0008] 2) Perform point cloud registration on the acquired point cloud data, establish the relationship between the obtained points, and segment the point cloud according to the physical entity. This module simplifies and denoises the separated device point cloud data to reduce the difference between the acquired point cloud and the target point cloud.

[0009] 3) Build a model library covering existing point clouds, use PointNet++ combined with Transformer feature extraction method to obtain the similarity between the input point cloud and the 3D model in the existing model library, and if there is a similar model, perform model matching on the point cloud, and quickly reconstruct the point cloud based on the original model in the model library.

[0010] 4) When the input point cloud is difficult to match in the existing model library, use point cloud reverse grid reconstruction to construct. The input point cloud is converted into a three-dimensional grid model through an algorithm, and the holes in the reconstructed grid are repaired and texture mapped to obtain the model reconstruction of the unmatched point cloud data and update the model library.

[0011] The feature extraction of the PointNet++ network adopted in step 3) is carried out layer by layer, and its key component is point set abstraction (SA). SA consists of three parts: sampling, grouping, and feature extraction, and its structure is defined as SA(center point,, lp]), where the center point is multiple key centers obtained by the farthest point sampling method. The device point cloud data is obtained as a 64×1 feature vector after passing through the feature extraction module.

[0012] Furthermore, the feature extraction method steps of the Transformer include:

[0013] Step 1: Obtain the average feature vector p′ of all existing three-dimensional models of the corresponding type in the model library.

[0014] Step 2: After linearly transforming the average feature vector p′, obtain the corresponding query matrix, key matrix, and value matrix. The specific calculation formulas are as follows:

[0015]

[0016] where Q', K', and V' respectively represent the query matrix, key matrix, and value matrix, all of which are matrices of size [N, 64], where N is the number of models in the model library, and is the transpose of the corresponding linear transformation weight matrix.

[0017] Step 3: Calculate the attention weights according to formula (3-2), perform operations on the calculated query matrix and key matrix, and after normalizing the operation results through the Softmax layer, obtain the attention map

[0018]

[0019] Step 4: Use the value matrix V' to weight the attention map

[0020]

[0021] Step 5: Input the updated model feature vector into a fully connected layer, and add the output result to the original feature vector p i . of a single device and perform a normalization transformation to obtain the final attention feature ψ of a single devicei 。

[0022]

[0023] Among them, τ is the normalization transformation made, and is the mapping weight of the fully connected layer.

[0024] The models in the model library are retrieved and evaluated by calculating the cosine similarity between the attention features output by the Transformer module and the feature vectors of the point cloud to be matched, so as to determine whether there are similar replaceable models. The specific calculation method is as follows:

[0025]

[0026] where r i is the similarity value, and its value range is 0 to 1. The larger the value, the higher the similarity between the two; p is the feature vector of the point cloud model to be reconstructed, and p i ' is the attention feature vector of the i-th retrieved model in the model library; |||| represents calculating the L2 norm of the vector.

[0027] Furthermore, the network reconstruction algorithm described in step four adopts an improved greedy projection triangulation algorithm, and the specific steps include:

[0028] Step 1: Establish an octree point cloud tree structure for point cloud retrieval.

[0029] Step 2: Calculate the quadratic parametric surface of the original point cloud, and use it to approximately represent the local base surface. Further, obtain the curvature of each point in the point cloud, and separate the points with larger curvature to avoid affecting the normal vector calculation.

[0030] Step 3: Use the octree instead of the kd-tree for neighborhood search, and calculate the point cloud normal vector according to the curvature by the nearest point search method.

[0031] Step 4: For the point cloud generated in step 3 with normal vector information, reconstruct the point cloud according to the subsequent greedy projection algorithm to obtain the final surface model.

[0032] The rapid reconstruction system of the industrial equipment digital twin model based on point cloud, which is used to integrate the method proposed by the present invention, includes an equipment physical entity layer, an equipment twin model layer, a data preprocessing layer, an equipment model library, a model retrieval and registration layer, and a point cloud grid reconstruction layer.

[0033] (1) Physical entity layer: mainly includes the physical entities of various machine tools (lathes, milling machines, machining centers, etc.), various transportation tools (conveyor belts, AGVs, RGVs, etc.), various manipulators, automated stereoscopic warehouses, and other equipment (detectors, workbenches, etc.) in the industrial scene.

[0034] (2) Device Twin Model Layer: It mainly refers to the twin models corresponding to various device entities included in the Physical Entity Layer, providing model support for the construction of the twin industrial scenario.

[0035] (3) Data Preprocessing Layer: It includes point cloud reduction and point cloud denoising. By downsampling and removing noise from the original device point cloud data collected from the Physical Entity Layer, single-device point cloud data with complete entity features, high precision, and low computational resource occupancy is obtained.

[0036] (4) Device Model Library: It mainly includes existing 3D models of industrial devices and their corresponding model feature values, providing model support for model reconstruction based on shape matching. At the same time, the Device Model Library supports updates and expansions, and the reconstructed mesh models from the Point Cloud Mesh Reconstruction Layer can be stored in the Device Model Library.

[0037] (5) Model Retrieval and Registration Layer: For the single-device point cloud data input from the Data Preprocessing Layer, its feature values are extracted and compared with the feature values of the models in the existing Device Model Library. Similar 3D models are retrieved. After model registration, they can be used as the reconstructed device twin model and output to the Device Twin Model Layer. The point cloud data for which the retrieval fails is input to the Point Cloud Mesh Reconstruction Layer.

[0038] (6) Point Cloud Mesh Reconstruction Layer: It is used to realize the reverse mesh model construction based on point cloud data. By performing mesh reconstruction, hole repair, and texture mapping on the point cloud data input from the Model Retrieval and Registration Layer, a complete device model can be generated, output to the Device Twin Model Layer, and used to update the Device Model Library.

[0039] Beneficial Effects:

[0040] The present invention proposes a method and system for rapid reconstruction of an industrial device digital twin model based on shape matching. Combining laser scanning technology, the idea of model matching, and related technologies of point cloud reverse reconstruction, a rapid reconstruction solution for the twin models of most industrial devices is realized. Through the feature extraction of the collected device point cloud data and existing 3D models, and by combining the retrieval and registration algorithms of point cloud and model, the reconstruction of the industrial device twin model based on shape matching is achieved, reasonably utilizing existing historical model resources and reducing the workload of constructing the twin industrial scenario model. Also, through the research on the method of constructing a reverse network model based on point cloud, the manual labor amount for reconstructing the industrial device twin model is further reduced. Description of the Drawings

[0041] Figure 1 is the system architecture diagram of the present invention;

[0042] Figure 2 is the flowchart of the rapid reconstruction method of the present invention;

[0043] Figure 3 A feature extraction network diagram for the present invention;

[0044] Among them, (a) is the PointNet++ network diagram, and (b) is the Transformer network;

[0045] Figure 4 This is a flow chart of surface mesh reconstruction of the present invention;

[0046] Figure 5 It is the retrieval rate diagram of different methods under the two situations of data missing and noise of the present invention;

[0047] Figure 6 The device point cloud and reconstruction effect diagram of the implementation case of the present invention;

[0048] Among them, (a) is the device point cloud image, and (b) is the reconstruction effect image;

[0049] Figure 7 Point cloud and reconstruction renderings of the machining workshop and assembly workshop of the implementation case of the present invention;

[0050] Among them, (a) is the point cloud map of the machining workshop, (b) is the point cloud map of the assembly workshop, (c) is the reconstruction rendering of the machining workshop, and (d) is the reconstruction rendering of the assembly workshop. DETAILED DESCRIPTION

[0051] In the process of building a digital twin workshop, its core is the construction of a twin model of industrial equipment, and the primary task of building a twin model of industrial equipment is to establish an accurate geometric model of industrial equipment. When building the model, it is necessary to ensure its accuracy, authenticity, and compliance with constraint rules, while considering the consumption of computer resources. In addition, the scalability of the model must also be taken into account to ensure that dynamic behaviors can be added in the future. In the field of model reconstruction, the use of laser point cloud technology for three-dimensional model reconstruction has become an important means, which can provide high-precision and comprehensive equipment geometry information. At the same time, considering that most industrial equipment in actual industrial scenarios is universal, that is, in different enterprises, the same or similar processing types of work workshops are likely to have industrial equipment of the same model and similar appearance, especially industrial equipment such as robotic arms and AGVs, which are more likely to have similar appearances. Reusing the three-dimensional models of these devices can effectively improve the efficiency of model construction and reduce the cost of model reconstruction. Therefore, the present invention proposes a system for building a twin model of industrial equipment based on laser point clouds, such as Figure 1 The proposed system includes device physical entity layer, device twin model layer, data preprocessing layer, device model library, model retrieval and registration layer and point cloud mesh reconstruction layer.

[0052] (1) Physical entity layer: mainly includes various machine tools (lathes, milling machines, machining centers, etc.), various transportation tools (conveyor belts, AGVs, RGVs, etc.), various manipulators, automated stereoscopic warehouses and other equipment (detectors, workbenches, etc.) in the industrial scenario. The corresponding point cloud data of the equipment in the physical entity layer is obtained through a laser scanner.

[0053] (2) Equipment twin model layer: mainly the twin models corresponding to various equipment entities included in the physical entity layer. It provides model support for the construction of the twin industrial scenario.

[0054] (3) Data preprocessing layer: includes point cloud reduction and point cloud denoising. By downsampling and removing noise from the original point cloud data of the equipment collected from the physical entity layer, the point cloud data of a single device with complete entity features, high precision and low computational resource occupancy is obtained.

[0055] (4) Equipment model library: mainly includes existing 3D models of industrial equipment and their corresponding model feature values, providing model support for model reconstruction based on shape matching. At the same time, the equipment model library supports updates and expansions, and the reconstructed mesh model of the point cloud mesh reconstruction layer can be stored in the equipment model library.

[0056] (5) Model retrieval and registration layer: For the point cloud data of a single device input from the data preprocessing layer, its feature values are extracted and compared with the feature values of the models in the existing equipment model library. Similar 3D models are retrieved, and after model registration, they can be used as the reconstructed equipment twin model and output to the equipment twin model layer. The point cloud data for which the retrieval fails is input to the point cloud mesh reconstruction layer.

[0057] (6) Point cloud mesh reconstruction layer: used to realize the reverse mesh model construction based on point cloud data. By performing mesh reconstruction, hole repair, and texture mapping on the point cloud data input from the model retrieval and registration layer, a complete equipment model can be generated, output to the equipment twin model layer and used to update the equipment model library.

[0058] Based on the framework proposed in the previous section, the construction process of the industrial equipment twin model based on laser point cloud is determined. As Figure 2As shown in the figure, this process consists of three modules: a data preprocessing module, a model matching and reconstruction module, and a point cloud reverse reconstruction module. The data preprocessing module is used to provide clean and available point cloud data for the processing module. This module includes three processing steps: point cloud data acquisition, point cloud stitching and registration, and point cloud thinning and denoising. Point cloud is the bridge connecting the physical factory and the virtual factory. Therefore, in this process, point cloud acquisition is the first and most fundamental step. In the present invention, the widely used lidar scanning method is selected to collect the point cloud of the equipment. After the point cloud acquisition is completed, registration and segmentation are required. Since it is difficult to obtain the overall scene of the factory by a single scan, multiple scans are needed to obtain the overall point cloud of the factory. Point cloud registration can establish the relationship between the previously obtained points. At the same time, segmenting the point cloud according to the physical entity can effectively reduce the difficulty of subsequent processing. The registration and separation of the point cloud can be achieved through automatic software or semi-automatic operations with human participation. For the separated single-point cloud data of the equipment, the last step of this module is thinning and denoising, which can reduce the computational amount in the point cloud reconstruction process and reduce the difference between the acquired point cloud and the target point cloud at the same time.

[0059] In thinning and simplification, the commonly used method is the sampling method, that is, sampling the point cloud data according to a certain rule, that is, only retaining some sampling points and ignoring other points. This method is simple and easy to operate, and has high processing efficiency, but it is difficult to retain edge features. Therefore, it is essential to ensure the balance between accuracy and efficiency in point cloud data processing. By removing redundancy and adopting a reasonable thinning and simplification method, the system operation speed and modeling efficiency can be effectively improved, and reasonable thinning of the point cloud data can be achieved on the premise of maintaining a certain accuracy. Corresponding improvements are made to the sampling method, and a three-dimensional voxel grid point cloud thinning algorithm based on the k-neighborhood is designed. The specific algorithm steps are as follows:

[0060] Step 1: First, obtain the extreme coordinate values of the input point cloud data in the directions of its own three coordinate axes, and determine the bounding box so that it can enclose the entire point cloud data.

[0061] Step 2: Obtain the size of the grid through the minimum data volume k in the grid determined in advance.

[0062] Step 3: Roughly calculate the normal vectors of each point using the least squares plane method, and calculate the local least squares plane A of the k-neighborhood where each data point P i is located.

[0063] Step 4: Calculate the normal vector of the plane A using the principal component analysis method, and calculate the centroid of the k-neighborhood points passing through the normal vector of P i . Use the distance from the data point P i to the centroid to construct a covariance matrix, and represent the smallest eigenvector of this matrix as P iThe normal vector of .

[0064] Step 5: Calculate P for each data point i The angle between the normal vector of the point and the normal vector of its neighboring points is calculated. Points with angle values greater than the preset threshold are marked as feature points and retained. At the same time, for non-feature points with smaller angle values, their centroids in the 3D voxel grid are calculated and retained to avoid data loss in smooth areas.

[0065] When using a laser scanner to acquire point cloud data, the acquired data has more or less noise points, considering the differences in reflection characteristics of different material surfaces, the vibration of the equipment itself or the vibration of the external environment, the occlusion caused by obstacles, and the multipath effect encountered by the laser during propagation. According to the distribution law of noise points, they can be divided into dense noise points, drift noise points and mixed noise points. In view of these noise point characteristics, in order to remove as many noise points as possible, only obtain the main point cloud, and reasonably reduce the amount of calculation, the k-domain denoising algorithm has been modified. The conventional k-neighbor denoising algorithm establishes a topological relationship for the initial scattered point cloud through the kd-tree, finds the k nearest neighbor points in the three-dimensional space for each point, obtains the local structure information around it, and uses these local structure information to determine whether it is a noise point through certain judgment criteria (such as distance threshold, normal vector consistency, etc.). The conventional k-neighbor denoising algorithm is only effective in removing drift noise points and mixed noise points, and it is difficult to remove dense noise points. Therefore, the algorithm used in the present invention is improved on the basis of the k-neighbor denoising algorithm, and a layer of search is added to achieve further denoising. The specific algorithm is as follows:

[0066] Step 1: Assume that the three-dimensional coordinates of any point Pi in the initial point cloud with a total number of n points are (x0, y0, z0), i = 1, 2, ..., n, and the coordinates of the points in the k-neighborhood of the point can be expressed as (x i ,y i , z i ), i = 1, 2, ..., k, then P i The average distance D to the midpoint of its k neighbors i It can be expressed as:

[0067]

[0068] Assumption D i It obeys Gaussian distribution, that is Where N(·,·) represents a normal distribution with mean μ1 and standard deviation σ1. Points that are not within the set distance interval R1∈(μ1-a·σ1, μ1+a·σ1) are classified as noise points and removed. Based on experience, when a=1, the removal effect of drift noise points and mixed noise points is better.

[0069] Step 2: Considering that the dense noise points are still at a certain distance from the main body point cloud, that is, the average distance from any one of them to all other points is different from the average distance from any point in the main body point cloud to all other points, so repeat step 1) to adjust the threshold to further remove the dense noise points. For any point P in the currently preliminarily denoised main body point cloud i ′ with coordinates (x0′, y0′, z0′), i = 1, 2, …, m, and the coordinates of the other remaining points are (x i ′, y i ′, z i ′), i = 1, 2, …, m - 1, then the average distance from P i ′ to the remaining m - 1 points is:

[0070]

[0071] Similar to step 1, D i ′ also follows a normal distribution, that is where μ2, σ2 can be expressed as:

[0072]

[0073] For the calculated D i ′, points not within the threshold R2 ∈ (μ1 - a·σ1, μ1 + a·σ1) are determined as noise points, and after deletion, the final main body point cloud is obtained.

[0074] The model matching and reconstruction module is a key module in the reconstruction process of the industrial equipment twin model proposed by the present invention. The main function of this module is to distinguish whether the input point cloud is similar enough to the 3D models in the existing model library. The construction of the model library is the basis for model matching. In addition to simplifying the existing models, the initial equipment model library is further expanded by manually modeling based on the collected point clouds. Finally, there are about 100 models in the constructed 3D lightweight model library of industrial equipment, providing a large amount of resources for subsequent shape-based model matching. Considering that the model library will be further enriched as the number of application scenarios increases, resulting in an increase in the number of files in the model library, and in order to improve the model retrieval efficiency and accuracy to a certain extent, the present invention adopts a management method of storing different types of equipment separately, and creates a 3D model library of industrial equipment according to five major categories (namely: machine tools, transportation tools, manipulators, warehouses, other equipment). When creating the initial 3D model of the equipment in the industrial equipment model library, 3ds Max software is mainly used. Based on the point cloud data, a surface is generated by constructing multiple contour curves, and after operations such as stretching and shearing the surface, a preliminary 3D model of the equipment is obtained, thus realizing the construction of the general outline of the 3D model of the equipment from the contour of the point cloud data. Then, by referring to pictures and other image materials, the disassembly and refined improvement of the equipment structure unit, as well as the formulation of the motion logic related to the equipment, are realized to create a precise 3D model of the industrial equipment. At the same time, in order to make the model look more realistic in the subsequent twin scenarios, the 3D white model of the industrial equipment can be texture-mapped using the collected image data of the model surface, so that the model surface reaches a realistic effect. Mysql is used as the database management system to store and manage the created high-precision 3D models of industrial equipment. This system is used to create and delete model data tables, update and delete existing models, and store new models. The update of the model includes changes in the basic attributes of the model, modification of the file path, and replacement of the model thumbnail, etc. As Figure 3 Some models of the constructed 3D model library are shown.

[0075] To achieve this goal, the present invention adopts a feature extraction network structure that combines PointNet++ and Transformer, as Figure 3 shown. It should be particularly noted that there is a special feature situation during model matching, that is, the model library is empty. At this time, all input point clouds will be determined to have no similar 3D models. In other words, at this time, the model matching and reconstruction module actually cannot play a role, so it is necessary to enter the subsequent point cloud reconstruction module. As the model library is continuously updated and expanded, this module will gradually play a role and participate in the model matching and reconstruction process. First, PointNet++ is used to extract the features of the equipment point cloud, as Figure 3 (a) shown. For the M-dimensional equipment point cloud data with N points obtained, it is denoted as Each point contains M - dimensional information such as three - dimensional coordinates, color, normal vector, etc. In the feature extraction stage, the present invention only considers the three - dimensional coordinate information, that is, M = 3, and p i =(x i , y i , z i ). To further reduce the computational pressure of feature extraction, first, a point cloud P of size n×3 is obtained by randomly sampling the pre - processed data. The feature extraction of the PointNet++ network adopted by the present invention is carried out hierarchically, and its key component is set abstraction (SA). SA consists of three parts: sampling, grouping, and feature extraction, and its structure is defined as SA(center point, radius, [mlp]), where the center point is multiple key centers obtained by the farthest point sampling method. The device point cloud data is obtained as a 64×1 feature vector after passing through the feature extraction module.

[0076] Specifically, the parameter settings of SA in the two stages of the network are (512, 0.2, [64, 64, 128]) and (128, 0.4, [128, 128, 128]) respectively. This means that in the first stage, SA performs feature extraction through 512 center points, a radius of 0.2, and a multi-layer perceptron [64, 64, 128]. In the specific implementation, for the incoming point set, multiple center points are obtained through the farthest point sampling method and used as the centers of ball queries, and then decomposed into multiple local regions containing at most 512 points. Such a design enables the network to gradually focus on local features. Finally, in the processing of the PointNet layer, local features of 128×512 are extracted. In the second stage, SA uses 128 center points, a radius of 0.4, and a multi-layer perceptron structure of [128, 128, 128] to refine the data again. Then, a feature vector p of 64×1 is output via PointNet. For the extraction of the feature vector of the existing device model, point cloud extraction needs to be performed on the model before extraction. Considering that the existing device model may not necessarily be a mesh model, the specific method for feature extraction of the existing model in the present invention is as follows: First, the existing non-Mesh model is tessellated and subdivided, that is, the original model is first represented as an initial mesh, which may be composed of triangles, quadrilaterals, or other polygons. This initial mesh is usually generated by discrete control points and control patches. Through an iterative algorithm implemented by interpolation, weighting, or other geometric transformations, each patch of the initial mesh is decomposed into smaller patches in each iteration. Through multiple iterations, the patches of the initial mesh are gradually decomposed into smaller sub-patches and more vertices and connecting lines are introduced, thereby refining the geometric shape of the model and finally forming a mesh model. For the generated or existing mesh model, considering the performance of the processing device and the accuracy of subsequent feature extraction, mesh subdivision or mesh simplification can be further performed. Finally, a set of points is generated using uniform sampling on the mesh model and used as the extracted point cloud of the existing model. The point cloud is input into the feature extraction network to obtain the feature vector of the existing model.

[0077] After obtaining the feature vector of the device point cloud, it is compared with the feature values of all models of the corresponding type in the model library to find the most similar model. Before performing the retrieval, the feature vectors p of all models of the type to be queried in the model library i are input into the Transformer module to obtain the attention feature ψ corresponding to a single model i . The specific Transformer module is as shown in Figure 3 (b). The specific feature extraction steps are as follows:

[0078] Step 1: Obtain the average feature vector p′ of all existing three-dimensional models of the corresponding type in the model library.

[0079] Step 2: After linearly transforming the average feature vector p', the corresponding query matrix, key matrix, and value matrix are obtained. The specific calculation formulas are as follows:

[0080]

[0081] where Q', K', and V' represent the query matrix, key matrix, and value matrix respectively, all of which are matrices of size [N, 64], where N is the number of models in the model library. and are the transposes of the corresponding linear transformation weight matrices.

[0082] Step 3: Calculate the attention weights. Perform an operation on the calculated query matrix and key matrix, and after normalizing the operation result through the Softmax layer, obtain the attention map.

[0083]

[0084] Step 4: Use the value matrix V' to weight the attention map to obtain the updated feature vector

[0085]

[0086] Step 5: Input the updated model feature vector into a fully connected layer, and add the output result to the original feature vector p of a single device i and perform a normalization transformation to obtain the final attention feature ψ of a single device i :

[0087]

[0088] where τ is the normalization transformation performed, is the mapping weight of the fully connected layer.

[0089] Retrieve and evaluate the models in the model library by calculating the cosine similarity between the attention features output by the Transformer module and the feature vector of the point cloud to be matched, so as to determine whether there are similar replaceable models. The specific calculation method is as follows:

[0090]

[0091] where r i is the similarity value, and its value range is 0 to 1. The larger the value, the higher the similarity between the two; p is the feature vector of the point cloud model to be reconstructed, and p i ' is the attention feature vector of the i-th retrieved model in the model library; || || represents calculating the L2 norm of the vector.

[0092] Specifically, the function of the point cloud reconstruction module is that the point cloud reverse grid reconstruction construction will only be used when the input point cloud cannot be matched in the existing model library. The input point cloud is converted into a three-dimensional grid model through an algorithm. In the present invention, among many existing grid reconstruction algorithms, the surface grid reconstruction algorithm based on greedy projection triangulation with strong adaptability is selected and corresponding improvements are made. At the same time, in order to improve the integrity and authenticity of the reconstructed grid, a grid model repair sub-module and a texture mapping sub-module are proposed. There are certain drawbacks in the traditional algorithms for grid reconstruction: ① The point cloud retrieval and search time is long, which is mainly determined by the data storage and search structure of the kd-tree; ② The reconstruction effect of some data is not accurate enough or even has errors, mainly for sparse or unevenly dense point cloud data; ③ The reconstructed grid is not necessarily complete and is prone to holes, which is mainly reflected in the surface reconstruction of point clouds of complex structure devices. Specifically, due to the unclear corresponding relationship of points after the projection of such point cloud data. Therefore, the present invention makes corresponding improvements on this basis, introduces an octree to replace the kd-tree for retrieval, and at the same time differentiates the point cloud according to curvature and then estimates the normal vector of the point cloud through the nearest point search method. The specific algorithm flow is as Figure 4 shown. The improved algorithm steps are as follows:

[0093] Step 1: Establish an octree point cloud tree structure for point cloud retrieval.

[0094] Step 2: Calculate the quadratic parametric surface of the original point cloud, and approximately express the local base surface with it, further obtain the curvature of each point in the point cloud, and separate the points with larger curvature to avoid affecting the normal vector calculation.

[0095] Step 3: Use the octree to replace the kd-tree for neighborhood search, and calculate the normal vector of the point cloud according to the curvature through the nearest point search method.

[0096] Step 4: For the point cloud with normal vector information generated in Step 3, reconstruct the point cloud according to the subsequent greedy projection algorithm to obtain the final surface model.

[0097] The surface grid reconstruction algorithm based on greedy projection triangulation is efficient for various point cloud data. However, when it processes irregular or unevenly dense surfaces, it will inevitably lead to a large number of hole defects in the generated surface model, which makes surface repair a key step to improve the model quality. The selected repair algorithm for the grid model is the surface repair method based on triangular grids. It can be mainly divided into four steps: detection of hole boundaries, initial repair of holes, further optimization of the initial repair effect based on least squares plane fitting, and combination with radial basis functions to achieve the final grid model repair. The specific repair process is as follows:

[0098] Step 1: Input the initial mesh generated by the algorithm reconstruction, and identify and highlight the hole boundaries through the surface hole recognition algorithm.

[0099] Step 2: Fill the recognized holes with triangular meshes to achieve preliminary hole repair.

[0100] Step 3: Use the least squares plane fitting method to smooth the meshes added during the above repair.

[0101] Step 4: Further optimize the repaired holes using radial basis functions to make them more conform to the initial model features.

[0102] The three-dimensional mesh model after point cloud reconstruction can only display the surface geometric features of the device physical entity. To add information such as color and texture to the model surface and thus enhance the realism of the reconstructed model, texture mapping is an essential task. Briefly speaking, the process of texture mapping is to map the collected two-dimensional image sequence onto the meshed surface of the reconstructed three-dimensional model using the principle of perspective projection imaging. Specifically, this mapping is performed on the triangular patches on the model surface to restore the details and lighting of the object surface during shooting. This invention uses UV mapping to complete the surface texture mapping of the model, and the specific steps are as follows:

[0103] Step 1: UV unwrapping, that is, appropriately parameterize the three-dimensional model, map each point on the model surface to a two-dimensional plane, and generate the associated UV coordinates. By calculating the parameterized coordinates of the model surface, each point has a unique coordinate in the UV space, ensuring that the texture can accurately fit onto the model surface. The mapping function expression for mapping the three-dimensional model to the two-dimensional space is defined as follows:

[0104] (u, v) = f(x, y, z)

[0105] where (u, v) are the coordinates of the model vertices in the UV space, and (x, y, z) are the vertex coordinates of the triangular patches in the three-dimensional model. By determining the corresponding mapping relationship f, the parameterization of the model can be achieved.

[0106] Step 2: Create or select an appropriate texture image. The texture image can be a photo from the real world or artificial two-dimensional image information, etc. The texture mapping images of this invention are all made from the collected real photos through secondary production.

[0107] Step 3: After UV mapping and texture selection are completed, it is necessary to associate the UV coordinates with the pixel values at the corresponding positions in the texture image. This mapping is generally achieved through interpolation techniques to ensure smooth transition of the texture on the model surface between different regions.

[0108] Step 4: After completing the basic texture mapping, some adjustments and optimizations may be required to handle issues such as stretching, distortion, or seamless connection on the model surface. This may include manually editing UV coordinates or using some automatic optimization tools.

[0109] Implementation Case 2:

[0110] The overall 3D point cloud data of the industrial equipment scene is obtained through a 3D laser scanner, and the equipment point cloud data to be reconstructed is obtained after further segmentation. Due to the complexity of the industrial scene, such as mutual occlusion between equipment, the operating state of industrial equipment during scanning, and on-site environment and other factors, the collected equipment point cloud data is not necessarily perfect. To test the superiority of the algorithm retrieval performance, the present invention is compared with two common feature extraction methods, including PCT

[81] , PointNet

[82] , and the test objects are 17 industrial equipment data (including robotic arms, AGVs, machine tools, warehouses, etc.) scanned. Experimental analyses are respectively carried out for these two cases of data missing and noise. For relatively complete equipment point clouds, the algorithms of the present invention, PCT, and PointNet can all achieve accurate model searching. At the same time, the complete equipment point cloud is partially deleted artificially to simulate the situation of data missing, and the model searching is carried out again to test the performance of the algorithm of the present invention and the two comparison algorithms. The results of the method of the present invention and the above two methods in the data set are as Figure 5 shown in (a). It reveals that when the missing point cloud data of the equipment does not exceed 15%, the algorithm of the present invention can ensure a relatively high average accuracy rate and successfully retrieve the corresponding 3D model. This shows that the algorithm of the present invention has good robustness in dealing with data missing and is suitable for accurate model retrieval of industrial equipment point cloud data. Figure 5 (b) shows the influence of different noises on the retrieval performance. It can be observed that the higher the noise intensity, the lower the model searching accuracy rate. When no noise or less noise is added, PCT, PointNet, and the algorithm of the present invention can all maintain a relatively high retrieval accuracy. However, when more Gaussian noise is added, the algorithm of the present invention shows obvious advantages compared with other algorithms. This shows that the algorithm of the present invention has better robustness in dealing with point cloud data with a certain amount of noise. For the twin model reconstruction of industrial equipment in a machining workshop and an assembly workshop, the model reconstruction is carried out according to the method proposed by the present invention. Figure 7(a) and (b) are the overall scene point clouds collected from the machining workshop and the assembly workshop respectively, as well as the images of the main industrial equipment contained therein. Among them, the main industrial equipment in the machining workshop includes two milling machines of the same type, two lathes of the same type, one scanner, two AGVs of the same type, two manipulators of the same type, one stereoscopic warehouse, and multiple workstations, etc. The main industrial equipment in the assembly workshop includes two warehouses of different types, two AGVs of the same type, four manipulators of different types, two man-machine collaborative assembly tables of the same type, one automatic assembly table, and one inspection table. There are a total of 17 types of equipment in the two scenes. First, the above-mentioned equipment is separated from the overall scene to obtain the single equipment point cloud data. Some of the obtained single equipment point clouds are shown in Figure 6 (a). The first row is the point cloud of some equipment in the machining workshop, and the second row is the point cloud of some equipment in the assembly workshop. The processed single equipment point cloud data is sequentially input into the model library retrieval and registration module for model reconstruction based on shape matching. In the construction of the model library of the present invention, three-dimensional models are only constructed for 12 types of equipment among them. After being processed by the corresponding modules in the system of the present invention, the single equipment point clouds of the corresponding 12 types of equipment input have all achieved model reconstruction. For the remaining 5 types of models, they are continued to be input into the point cloud reverse grid reconstruction module of the system, and the corresponding model reconstructions have also been achieved. The reconstruction effects corresponding to the equipment in Figure 6 (a) in the final 17 types of models are shown in Figure 6 (b).

[0111] Table of 3D comparison results of the reconstructed models

[0112]

[0113] From the comparison of the above table results, it can be seen that the deviation values of the models reconstructed by matching from the existing models are relatively small. The maximum upper deviation value basically remains at about 1.3 mm, and the maximum lower deviation value basically remains at about -1.7 mm, and both the overall average deviation and the standard deviation are relatively low. For the models reconstructed based on the point cloud, the maximum upper deviation value remains at about 2.2 mm, and the maximum lower deviation value remains at about -3.1 mm. The final average deviation value and the standard deviation value are slightly higher than those of the matching reconstructed models. Overall, although the error of the point cloud reconstructed model is slightly larger than that of the matching reconstructed model, the errors of the models reconstructed by both methods are within an acceptable range and will not have a greater impact on subsequent development.

[0114] Finally, based on the reconstructed models, the present invention places them in unity according to their relative positions in the real scene, and after simple 3D reconstruction and texture mapping of other objects in the industrial scene, the two industrial twin scenes finally realized are respectively shown in Figure 7 (c) and Figure 7(as shown in (d)). It can be seen from the figure that the reconstructed model of the present invention has a good performance in the overall scene and can restore the original appearance of the equipment in the real scene. The subsequent development of digital twin-related functional applications, such as model action attachment and real information display, can be directly realized based on the equipment object and scene reconstructed by the present invention and has good effects.

Claims

1. A rapid reconstruction method for a digital twin model of industrial equipment based on laser point cloud, characterized in that, The reconstruction method includes the following steps: S1: Obtain the point cloud data corresponding to the physical device through multiple scans by a laser scanner; S2: Perform point cloud registration on the obtained point cloud data, establish the relationship between the obtained points, segment the point cloud according to the physical entity, and for the separated device monomer point cloud data, this module performs reduction and denoising; S3: Construct a model library covering the existing point cloud, adopt the feature extraction method of combining PointNet++ network with Transformer to obtain the similarity between the input point cloud and the 3D models in the existing model library. If there is a similar model, perform model matching on the point cloud, and realize rapid reconstruction of the point cloud based on the original models in the model library; S4: When the input point cloud cannot be matched in the existing model library, use point cloud reverse grid reconstruction construction, convert the input point cloud into a 3D grid model through an improved greedy projection triangulation algorithm, and perform hole repair and texture mapping on the reconstructed 3D grid model to obtain the model reconstruction of the point cloud data that cannot be matched and update the model library.

2. The rapid reconstruction method of the digital twin model of industrial equipment based on laser point cloud according to claim 1, wherein, The physical device includes the entities of various machine tool devices, transportation tools, manipulators, automated stereoscopic warehouses and other devices in the industrial scenario.

3. The rapid reconstruction method of the digital twin model of industrial equipment based on laser point cloud according to claim 1, wherein The PointNet++ network includes a point set abstraction module SA. The point set abstraction module SA consists of three parts: sampling, grouping, and feature extraction. Its structure is defined as SA(center point,, lp]), where the center point is multiple key centers obtained by the farthest point sampling method. After passing through the feature extraction module, the device point cloud data obtains a 64×1 feature vector.

4. The rapid reconstruction method of the digital twin model of industrial equipment based on laser point cloud according to claim 1, characterized in that The specific feature extraction method of the Transformer includes the following steps: Step 31: Obtain the average feature vector p′ of all existing 3D models of the corresponding type in the model library; Step 32: After performing a linear transformation on the average feature vector p′, obtain the corresponding query matrix, key matrix, and value matrix: where Q', K', and V' represent the query matrix, the key matrix, and the value matrix respectively, all of which are matrices of size [N, 64], where N is the number of models in the model library, and is the transpose of the corresponding linear transformation weight matrix; Step 33: Perform an operation on the calculated query matrix and the key matrix, and after normalizing the operation result through the Softmax layer, obtain the attention map Step 34: Using the value matrix V', weight the attention map to obtain the updated feature vector Step 35: Input the updated model feature vector into a fully connected layer and compare the output result with the original feature vector p of a single device. i , after adding and normalizing, we get the final attention feature ψ of a single device i : where τ is the normalization transformation made, is the mapping weight of the fully connected layer; Retrieve and evaluate the models in the model library by calculating the cosine similarity between the attention feature and the feature vector of the point cloud to be matched to determine whether there is a similar replaceable model: where r i is the cosine similarity, whose value ranges from 0 to 1, and the larger the value, the higher the similarity between the two; p is the feature vector of the point cloud model to be reconstructed, and p i ' is the attention feature vector of the i-th retrieval model in the model library; || || represents calculating the L2 norm of the vector.

5. The rapid reconstruction method of the digital twin model of industrial equipment based on laser point cloud according to claim 1, wherein, The specific improved greedy projection triangulation algorithm includes the following steps: Step 41: Establish an octree point cloud tree structure for point cloud retrieval; Step 42: Calculate the quadratic parametric surface of the original point cloud, approximate the local base surface with the quadratic parametric surface, obtain the curvature of each point in the point cloud according to the local base surface, and separate the points with larger curvature to avoid affecting the normal vector calculation; Step 43: Use the octree for neighborhood search, and calculate the point cloud normal vector according to the curvature by the nearest point search method; Step 44: For the point cloud with normal vector information generated in Step 43, reconstruct the point cloud according to the greedy projection algorithm to obtain the final surface model.

6. A rapid reconstruction system for a digital twin model of industrial equipment based on point cloud, characterized in that The reconstruction system includes: Device physical entity layer, device twin model layer, data preprocessing layer, device model library, model retrieval and registration layer, and point cloud grid reconstruction layer; The physical entity layer includes the physical entities of various machine tool devices, transportation tools, manipulators, automated stereoscopic warehouses and other devices in the industrial scenario; The device twin model layer is used to provide model support for the construction of the twin industrial scenario by generating twin models corresponding to various device entities included in the physical entity layer; The data preprocessing layer is used to streamline and denoise point cloud data to obtain single-device point cloud data with complete entity features, high precision, and low computational resource consumption; The device model library includes existing three-dimensional models of industrial devices and corresponding model feature values, providing model support for model reconstruction based on shape matching; the device model library is also used to store the reconstructed mesh models of the point cloud mesh reconstruction layer; The model retrieval and registration layer is used to extract the feature values from the single-device point cloud data input from the data preprocessing layer, compare them with the feature values of the models in the existing device model library, retrieve similar three-dimensional models, and after model registration, output them as the reconstructed device twin models to the device twin model layer; The point cloud data for which the retrieval fails is input to the point cloud mesh reconstruction layer; The point cloud mesh reconstruction layer is used to implement the construction of an inverse mesh model based on the point cloud data.

Images