A computer vision-based point cloud and finite element model generation system and method

Abstract

The application relates to a computer vision-based point cloud and finite element model generation system and method, the system comprising a video acquisition and processing module, a three-dimensional point cloud establishment module and a finite element model generation module, and the method comprising the following steps: acquiring video data of a target structure and processing the same to obtain key pictures and corresponding space-time information; based on the perspective difference between multiple key pictures and the corresponding space-time information, adopting a multi-perspective geometric reconstruction technology based on deep learning to perform three-dimensional point cloud reconstruction on the target object, and preprocessing the point cloud data; extracting effective structure space geometric information from the preprocessed point cloud data, and then adopting a tomographic scanning node mode to cut the point cloud into multiple thin layers, and layer-by-layer establishment is performed to obtain finite element components, and the finite element components are integrated to obtain a finite element model. Compared with the prior art, the application can efficiently and accurately convert video data into a finite element model required for structure analysis, and avoids the tedious operation of manual modeling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional modeling, in particular to a point cloud and finite element model generation system and method based on computer vision. BACKGROUND

[0002] Digital Twin, as one of the core technologies of Industry 4.0 and intelligent development, has been widely applied in civil engineering, intelligent manufacturing and infrastructure management. Digital Twin can realize real-time monitoring, analysis and prediction by establishing dynamic mapping between physical entities and their digital virtual models, thereby optimizing structural performance, prolonging service life and improving operation efficiency.

[0003] In civil engineering, Digital Twin not only provides the geometric shape of the structure, but also integrates environmental data, damage information and mechanical properties, providing scientific basis for structural health monitoring, disaster warning and maintenance decision-making. However, the core challenge of building Digital Twin is how to quickly and accurately generate high-precision three-dimensional models and continuously update their status. Traditional modeling methods rely on expensive equipment such as three-dimensional laser scanners, with high data acquisition costs and complex operations. Secondly, the modeling process is usually time-consuming and highly dependent on technical experience, especially when dealing with complex structures or large-scale projects. For example, Chinese patent CN116611280A proposes a building structure finite element intelligent reverse modeling and analysis system based on three-dimensional computer vision. This system uses DBSCAN clustering algorithm to automatically segment the processed structure point cloud data into each component point cloud data, establishes the local coordinate system of each component to convert the component point cloud into the local coordinate system; and uses a slicing-based method to fit each slice of the component, thereby obtaining the geometric parameters of each component of the existing structure for generating the finite element model of the existing structure. This method needs to obtain the point cloud data of the existing structure through three-dimensional scanning, which has the problems of high data acquisition cost and complex operation.

[0004] In addition, existing methods are difficult to balance accuracy and efficiency, high-precision modeling is often inefficient, and fast modeling may lead to loss of geometric details and difficulty in meeting the needs of dynamic model updating. In addition, converting three-dimensional models to finite element models requires a complex meshing and attribute definition process, which is not only time-consuming but also prone to errors. Therefore, in order to promote the upgrading of the civil engineering industry and promote high-quality development of the civil engineering industry, there is an urgent need for fast and high-quality three-dimensional modeling methods. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a point cloud and finite element model generation system and method based on computer vision, which can efficiently and accurately convert video data into the finite element model required for structural analysis, avoiding the tedious operation of manual modeling.

[0006] The object of the present application can be achieved by the following technical solutions: a point cloud and finite element model generation system based on computer vision, comprising a video acquisition and processing module, a three-dimensional point cloud establishment module and a finite element model generation module connected in sequence, the video acquisition and processing module is used for acquiring video data of a target structure, and representative key frames and corresponding space-time information are obtained by processing the video data;

[0007] The three-dimensional point cloud establishment module is based on the perspective difference between multiple key frames and the corresponding space-time information, adopts a multi-perspective geometric reconstruction technology based on deep learning, performs three-dimensional point cloud reconstruction on the target object, and pre-processes the point cloud data;

[0008] The finite element model generation module is used for extracting structural space geometric information from the pre-processed point cloud data, cutting the point cloud into multiple thin layers based on the tomographic scanning node method, establishing finite element components by layering, and integrating to obtain a finite element model.

[0009] A point cloud and finite element model generation method based on computer vision, comprising the following steps:

[0010] S1, acquiring video data of a target structure, and processing the video data to obtain key pictures and corresponding space-time information;

[0011] S2, based on the perspective difference between multiple key pictures and the corresponding space-time information, adopting a multi-perspective geometric reconstruction technology based on deep learning to perform three-dimensional point cloud reconstruction on the target object, and generating a point cloud model;

[0012] S3, pre-processing the point cloud data of the point cloud model;

[0013] S4, extracting effective structural space geometric information from the pre-processed point cloud data, and cutting the point cloud into multiple thin layers based on the tomographic scanning node method, layering to obtain each finite element component, and integrating each finite element component to obtain a finite element model.

[0014] Further, the step S1 is specifically through a mobile phone, a camera or a drone to acquire video data of a target structure.

[0015] Further, the process of processing the video data in step S1 comprises:

[0016] extracting picture nodes from the video data, applying Fourier transform to the image to obtain a frequency domain representation F(u,v)=f{f(x,y)} of the image, and calculating the proportion of high frequency components ​The image with a ratio value higher than a preset threshold is taken as a key picture, wherein u and v are respectively coordinates of the image in a frequency domain, corresponding to frequency components after Fourier transform; x and y are respectively coordinates of the image in a spatial domain (time domain), corresponding to pixel positions of an original image; R is a radius threshold for dividing a high-frequency region, that is, frequency components higher than the radius are considered as high-frequency components; E total is a total energy of the image, that is, a total sum of energies of all frequency components in the entire frequency spectrum.

[0017] For the key picture, a background node is removed, a U-Net convolutional neural network structure is used, a dividing line between an image foreground and a background is learned, and the background is removed to obtain a region of interest.

[0018] Further, the step S2 is specifically applying an optical flow method to calculate an optical flow of each pixel of adjacent key pictures, to determine a motion of the camera, and jointly with corresponding camera pose information, to perform multi-view three-dimensional point cloud reconstruction in a coordinate system of the first frame image.

[0019] Further, the step S3 of preprocessing point cloud data of the point cloud model includes:

[0020] A median filter algorithm and a RANSAC (Random Sample Consensus) algorithm are adopted to remove noise points;

[0021] A farthest point sampling algorithm is adopted for downsampling to reduce point cloud resolution;

[0022] A DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is used to segment the point cloud to distinguish different structural components.

[0023] Further, the step S4 specifically includes the following steps:

[0024] S41, for the segmented point cloud data, a B-spline surface algorithm is adopted for fitting processing to obtain a fitted surface;

[0025] S42, the fitted surface is cut into multiple layers in a manner of tomographic scanning nodes, and each layer represents a cross section of the structure in a certain height range;

[0026] S43, a Delaunay method is adopted to generate a triangulation mesh for each layer after cutting;

[0027] The mesh of each layer is connected to nodes of the previous layer and the next layer, so as to form a complete three-dimensional finite element mesh.

[0028] Further, the step S41 is specifically as follows:

[0029] The plane normal vector and position are determined by fitting a surface equation for the segmented point cloud data through a B-spline surface, wherein the B-spline surface is represented by a set of control points, basis functions and weights, and the B-spline surface is defined by the following formula:

[0030]

[0031] In the formula, u and v are coordinates in the parameter space; S(u, v) is the position of the surface in the u, v parameter space; P i,j is a control point; N i,p (u) and N j,q (v) are basis functions, which are B-spline basis functions defined in the u and v parameter directions, respectively; p and q are orders of the basis functions.

[0032] The position of the control point P i,j is iteratively calculated by using the least square method, so that the fitted surface is as close as possible to the actual point cloud data, and the optimization objective is

[0033] Further, the basis function is calculated by using the Cox-de Boor recursive formula, and for a given parameter u, the basis function calculation formula is as follows:

[0034]

[0035] In the formula, u i is an element of the parameter vector, and the basis function is recursively calculated.

[0036] Further, the step S42 is specifically as follows: for the surface S(x, y, z) = 0, the object is cut at each fixed z value to obtain a plurality of slices S(x, y, z i ) = 0, i = 1, 2, …, N, wherein z i is the height value of the i-th layer, and N is the total number of layers.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] This invention comprises a video acquisition and processing module, a 3D point cloud creation module, and a finite element model generation module, connected sequentially. The video acquisition and processing module acquires video data of the target structure and processes the data to obtain representative keyframes and corresponding spatiotemporal information. The 3D point cloud creation module, based on the perspective differences and corresponding spatiotemporal information between multiple keyframes, employs deep learning-based multi-view geometric reconstruction technology to reconstruct a 3D point cloud of the target object and preprocesses the point cloud data. The finite element model generation module extracts structural spatial geometric information from the preprocessed point cloud data and, based on tomographic scanning nodes, segments the point cloud into multiple thin layers. Finite element components are then created layer by layer, and the resulting finite element model is obtained. Thus, by integrating video acquisition, 3D reconstruction, point cloud processing, and finite element modeling technologies, the entire process from acquiring video data to creating high-precision finite element components is automated. This efficiently and accurately converts video data into the finite element model required for structural analysis, avoiding the tedious manual modeling process.

[0039] This invention obtains structural spatial information from captured video data, extracts clear keyframes using Fourier transform, and employs a U-Net convolutional neural network to weight the regions of interest in the image. Then, it uses deep learning to reconstruct 3D point clouds using the keyframes. This invention can adapt to the modeling needs of complex structures and irregular shapes, and is particularly suitable for scenarios requiring rapid modeling.

[0040] This invention preprocesses point cloud data by using median filtering and RANSAC algorithms to remove noise points; it also uses farthest point sampling algorithm to downsample and reduce point cloud resolution; and it utilizes DBSCAN clustering algorithm to segment the point cloud and distinguish different structural components, thereby ensuring the accuracy of 3D point cloud reconstruction.

[0041] This invention takes into account the certain geometric features of the segmented point cloud data, such as planar or curved shapes. Therefore, it designs a method based on B-spline surface fitting and tomographic scanning nodes to generate 3D parts from the point cloud data. Through an automated process, the tedious manual modeling operations are avoided, greatly improving modeling efficiency and meeting the need for rapid and continuous updates to the 3D model's state. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0043] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0044] Figure 3 This is a schematic diagram of the application framework for an example.

[0045] Figure 4This is a schematic diagram of a one-dimensional B-spline curve of the B-spline surface in the embodiment;

[0046] Figure 5 This is a schematic diagram of point cloud tomographic scanning, cutting, and layering modeling in the embodiment.

[0047] The markings in the diagram are as follows: 1. Video acquisition and processing module; 2. 3D point cloud creation module; 3. Finite element model generation module. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0049] Example

[0050] like Figure 1 As shown, a point cloud and finite element model generation system based on computer vision includes a video acquisition and processing module 1, a 3D point cloud creation module 2, and a finite element model generation module 3 connected in sequence. The video acquisition and processing module 1 is used to acquire video data of the target structure and process the video data to obtain representative key frames and corresponding spatiotemporal information.

[0051] A 3D point cloud module 2 is established based on the perspective differences and corresponding spatiotemporal information between multiple keyframes. A deep learning-based multi-view geometric reconstruction technique is used to reconstruct the 3D point cloud of the target object and preprocess the point cloud data.

[0052] The finite element model generation module 3 is used to extract structural spatial geometric information from the preprocessed point cloud data. Based on the method of tomographic scanning nodes, the point cloud is cut into multiple thin layers, and finite element components are built by layering and integrated to obtain the finite element model.

[0053] Based on the above system, a method for generating point clouds and finite element models based on computer vision is implemented, such as... Figure 2 As shown, it includes the following steps:

[0054] S1. Collect video data of the target structure and process the video data to obtain key images and corresponding spatiotemporal information;

[0055] S2. Based on the perspective differences and corresponding spatiotemporal information between multiple key images, a deep learning-based multi-view geometric reconstruction technique is used to reconstruct the three-dimensional point cloud of the target object and generate a point cloud model.

[0056] S3. Preprocess the point cloud data for the point cloud model;

[0057] S4, effective structure space geometry information is extracted from the pretreated point cloud data, and the point cloud is cut into multiple thin layers in the form of tomographic nodes, and each finite element component is established by layering, and the finite element model is obtained by integrating each finite element component.

[0058] The embodiment applies the above scheme to build an application framework as shown in Figure 3 The main content includes:

[0059] I. Video acquisition and processing module: video data of the target structure is collected by lightweight devices such as mobile phones, cameras, and drones, representative key frames and corresponding space-time information are obtained by processing the video data, and the key frame pictures are further processed and positioned to extract the interested part;

[0060] Specifically, first, image information of the structure space is obtained by using lightweight devices such as mobile phones, cameras, and drones, Fourier transform is applied to the image to obtain the frequency domain representation F(u,v) = f{f(x,y)}, and the proportion of high-frequency components The proportion of high-frequency components The proportion of high-frequency components

[0061] On this basis, the U-Net convolutional neural network structure is used to learn the boundary between the foreground and background of the image, and the interested region is obtained by removing the background.

[0062] II. Three-dimensional point cloud module: based on the perspective difference and corresponding space-time information between multiple key frames, a multi-perspective geometric reconstruction technology based on deep learning is used to reconstruct a three-dimensional point cloud of the target object in a unified coordinate system, and the point cloud data is preprocessed.

[0063] Specifically, the optical flow method is used to calculate the optical flow of each pixel in the adjacent key frame image, the motion of the camera is inferred, and the multi-perspective three-dimensional point cloud is reconstructed in the coordinate system of the first frame image in combination with the corresponding camera pose information. On this basis, the median filtering algorithm and the RANSAC algorithm are used to remove noise points; the farthest point sampling algorithm is used for downsampling to reduce the point cloud resolution; the DBSCAN clustering algorithm is used to segment the point cloud to distinguish different components of the structure.

[0064] III. Finite element model generation module: effective structure space geometry information is extracted from the segmented point cloud data, and the point cloud is cut into multiple thin layers based on the idea of tomography, and finite element components are established by layering, and finally integrated into a finite element model.

[0065] Specifically, the segmented point cloud data usually has certain geometric characteristics, such as planar and curved surface shapes. This scheme fits the curved surface equation by B-spline surface, and determines the normal vector and position of the curved surface. For example Figure 4As shown, a B-spline surface is represented by a set of control points, basis functions, and weights. A typical two-dimensional B-spline surface is defined by the following formula. u, v are coordinates in the parameter space; S(u, v) is the surface position in the u, v parameter space; P i,j It is a control point, N i,p (u) and N j,q (v) are basis functions, defined as B-spline basis functions in the directions of parameters u and v, respectively; p and q are the orders of the basis functions.

[0066] The basis functions are calculated using the Cox-de-Boor recursive formula. For a given parameter u, the formula for calculating the basis functions is:

[0067]

[0068] Where u i These are the elements of the parameter vector, and the basis functions are obtained by recursive calculation.

[0069] Finally, the least squares method was used to iteratively calculate the control point P. i,j The position of the fitted surface is such that it approximates the actual point cloud data as closely as possible; the optimization objective is...

[0070] like Figure 5 As shown, this scheme is based on the concept of tomography, cutting the fitted curved surface into multiple thin layers (i.e., "tomograms"), each layer representing a cross-section of the structure within a certain height range. These layers can be uniform or non-uniformly cut according to the shape of the structure. Assuming the surface of the object is a curved surface S(x,y,z) = 0, the object can be cut at each fixed z value to obtain multiple slices S(x,y,z). i )=0,i=1,2,…,N, where z i is the height value of the i-th layer, and N is the total number of layers.

[0071] Then, the Delaunay method is used to generate triangular meshes for each cut layer, ensuring reliable mesh element quality and avoiding excessively small angles. For each layer of mesh, the nodal coordinates and shape function of each element are calculated. For a triangular element, its nodal coordinates are (x1, y1), (x2, y2), (x3, y3), and its shape function is: A is the area of ​​the triangular element. Each layer of the mesh connects to the nodes of the layer above and the layer below, thus forming a complete three-dimensional finite element mesh. For each layer z... i and z i+1 Node P(x,y,z) i ) will be connected to node P(x,y,z) i+1 ).

[0072] The finally generated finite element mesh is checked and optimized, further ensuring the quality, accuracy and calculation efficiency of the mesh.

[0073] It should be noted that the present scheme is not only suitable for the modeling of steel structures, but also can be applied to bamboo structures, reinforced concrete and other common structures.

[0074] The present scheme first obtains high-quality image data by shooting a video of the structure and extracting frame images in the video. Then, the point cloud model of the structure is constructed using the extracted images, and the point cloud data is fitted by B-spline surface, and then the finite element model is generated based on the idea of tomography. It can effectively convert video data into the finite element model required for structure analysis, avoid the tedious operation of manual modeling, meet the needs of rapid and continuous updating of three-dimensional model state, and provide a new technical path for structure modeling in engineering practice. The present scheme automatically obtains spatial geometric information of the structure based on video, realizes end-to-end automatic generation of structure finite element model, simplifies the modeling process, greatly improves the modeling efficiency, and is suitable for long-term dynamic modeling and tracking monitoring of structures.

Claims

1. A point cloud and finite element model generation method based on computer vision, implemented using a point cloud and finite element model generation system based on computer vision, characterized in that... The point cloud and finite element model generation system includes a video acquisition and processing module (1), a three-dimensional point cloud creation module (2), and a finite element model generation module (3) connected in sequence. The video acquisition and processing module (1) is used to acquire video data of the target structure and process the video data to obtain representative key frames and corresponding spatiotemporal information. The 3D point cloud module (2) is based on the perspective differences and corresponding spatiotemporal information between multiple keyframes. It uses deep learning-based multi-view geometric reconstruction technology to reconstruct the 3D point cloud of the target object and preprocesses the point cloud data. The finite element model generation module (3) is used to extract structural spatial geometric information from the preprocessed point cloud data. Based on the method of tomographic scanning nodes, the point cloud is cut into multiple thin layers. Finite element components are established by layering and integrated to obtain a finite element model. The point cloud and finite element model generation method includes the following steps: S1. Collect video data of the target structure and process the video data to obtain key images and corresponding spatiotemporal information; S2. Based on the perspective differences and corresponding spatiotemporal information between multiple key images, a deep learning-based multi-view geometric reconstruction technique is used to reconstruct the three-dimensional point cloud of the target object and generate a point cloud model. S3. Preprocess the point cloud data for the point cloud model; S4. Extract effective structural spatial geometric information from the preprocessed point cloud data, and then use the method of tomographic scanning nodes to cut the point cloud into multiple thin layers, build each finite element component layer by layer, and integrate each finite element component to obtain the finite element model. Step S3 involves preprocessing the point cloud data for the point cloud model, including: Median filtering and RANSAC algorithms are used to remove noise points; The farthest point sampling algorithm is used to downsample and reduce the point cloud resolution; The DBSCAN algorithm is used to segment the point cloud to distinguish different structural components; Step S4 specifically includes the following steps: S41. For the segmented point cloud data, the B-spline surface algorithm is used for fitting to obtain the fitted surface; S42. Using a tomographic scanning node approach, the fitted surface is cut into multiple layers, each representing a cross-section of the structure within a certain height range, specifically targeting the surface. In each fixed Cut the object using the value to obtain multiple slices. ,in It is the first The height of the layer, It is the total number of floors; S43. Use the Delaunay method to generate a triangulated mesh for each cut layer; Each layer of mesh is connected to the nodes of the layer above and the layer below, thus forming a complete three-dimensional finite element mesh.

2. The method for generating point clouds and finite element models based on computer vision according to claim 1, characterized in that, Step S1 specifically involves collecting video data of the target structure using a mobile phone, camera, or drone.

3. The method for generating point clouds and finite element models based on computer vision according to claim 1, characterized in that, The process of processing video data in step S1 includes: Image nodes are extracted from video data, and Fourier transform is applied to the images to obtain their frequency domain representation. Calculate high-frequency components proportion Images with a proportion value higher than a preset threshold are selected as key images. These are the coordinates of the image in the frequency domain, corresponding to the frequency components after the Fourier transform; These are the coordinates of the image in the spatial domain, corresponding to the pixel positions in the original image; R This is the radius threshold used to divide the high-frequency region; frequency components above this radius are considered high-frequency components. This represents the total energy of the image, which is the sum of the energy of all frequency components in the entire spectrum. For key images, background nodes are removed using a U-Net convolutional neural network structure to learn the boundary between the foreground and background of the image, thereby removing the background and obtaining the region of interest.

4. The point cloud and finite element model generation method based on computer vision according to claim 1, characterized in that, Specifically, step S2 involves applying optical flow to calculate the optical flow of each pixel in adjacent key images to determine the camera's motion. This motion is then combined with the corresponding camera pose information to perform multi-view 3D point cloud reconstruction in the coordinate system of the first frame image.

5. The point cloud and finite element model generation method based on computer vision according to claim 1, characterized in that, The specific process of step S41 is as follows: Using B-spline surfaces, the surface equation is fitted to the segmented point cloud data to determine the normal vector and position of the plane. The B-spline surface is represented by a set of control points, basis functions, and weights, and is defined by the following formula: , In the formula, These are coordinates in parameter space; Is Surface position in parameter space; It is a control point. and These are basis functions, defined respectively in and B-spline basis functions in the parametric direction; and It is the order of the basis functions; The control points are calculated iteratively using the least squares method. The position of the fitted surface is such that it approximates the actual point cloud data as closely as possible, and the optimization objective is... .

6. The point cloud and finite element model generation method based on computer vision according to claim 5, characterized in that, The basis functions are calculated using the Cox-de-Boor recursive formula, for a given set of parameters. The formula for calculating the basis functions is: ， ， in, These are the elements of the parameter vector, and the basis functions are obtained by recursive calculation.

Images