Adaptive point cloud structured light fusion three-dimensional reconstruction accurate calculation method

Abstract

The invention relates to the technical field of computer vision and three-dimensional reconstruction, and discloses an adaptive point cloud structured light fusion three-dimensional reconstruction accurate calculation method, which comprises the following steps of: acquiring point cloud data of a target object and a corresponding structured light image; preprocessing the point cloud data; extracting features of the point cloud data; optimizing the point cloud features based on geometric analysis; fusing the optimized point cloud features with features extracted from the structured light image; and performing three-dimensional reconstruction calculation by using the fused features. The limitation of a single data source is avoided by obtaining object information from multiple aspects, the data quality is improved through preprocessing, point cloud features are comprehensively described through feature extraction, feature optimization focuses on key information, accurate basis is provided for reconstruction through feature fusion, a high-quality model is generated through three-dimensional reconstruction, and it is ensured that the result is reliable through model optimization and verification. The problem that a single data source cannot comprehensively describe the three-dimensional space form of the target object is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and three-dimensional reconstruction technology, and specifically to a precise calculation method for three-dimensional reconstruction by adaptive point cloud structured light fusion. Background Art

[0002] In the field of computer vision and 3D modeling technology, various industries have an increasing demand for accurately obtaining the three-dimensional spatial form of objects. However, relying on a single data source (point cloud data or structured light images) to obtain and reconstruct the three-dimensional spatial form has limitations. Point cloud data cannot present the complete appearance and details of the object, and structured light images cannot grasp the overall spatial structure. Both cannot provide a comprehensive description and cannot meet the needs of high-precision reconstruction. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the present invention provides an accurate calculation method for three-dimensional reconstruction by adaptive point cloud structured light fusion, which solves the problem that a single data source cannot fully describe the three-dimensional spatial form of the target object.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: an adaptive point cloud structured light fusion three-dimensional reconstruction precision calculation method, comprising the following steps:

[0005] S1, collect point cloud data of the target object and the corresponding structured light image;

[0006] S2. Preprocessing the point cloud data;

[0007] S3, extracting features of the point cloud data;

[0008] S4. Optimizing the point cloud features based on geometric analysis;

[0009] S5. fusing the optimized point cloud features with the features extracted from the structured light image;

[0010] S6. Perform three-dimensional reconstruction calculation using the fused features;

[0011] S7. Optimize and verify the reconstructed three-dimensional model and output the final three-dimensional reconstruction result.

[0012] By adopting the above technical solution, point cloud data and structured light images are integrated to break through the limitations of a single data source. Through multi-step processing, data quality is improved, features are effectively utilized, reconstruction accuracy is increased, and model reliability is enhanced to meet the needs of various application scenarios for high-quality three-dimensional models.

[0013] Preferably, S1 includes: a point cloud acquisition device acquires original point cloud data of the target object, and at the same time utilizes a structured light projection device to project a structured light pattern onto the target object, and uses a camera to acquire a corresponding structured light image, wherein the point cloud data reflects the spatial coordinate information of the target object, and the structured light image includes texture and depth-related information presented on the surface of the target object due to structured light projection.

[0014] Preferably, the step S2 includes normalizing the coordinates of the point cloud data so that each coordinate value is distributed within a preset normalization interval, and calculating the normalization interval by the formula Normalize the x, y, and z coordinates of each point in the point cloud data, where x min ,x max ,y min ,y max ,z min ,z max They are the minimum and maximum values ​​of the point cloud on the x, y, and z axes respectively; the bilateral filtering algorithm is used to remove noise in the point cloud data. For each point p i , its filtered coordinate p i ′Through the formula Calculate, where N(i) is the point p i The neighborhood point set of σ d and σ r are the standard deviations of spatial distance and characteristic distance respectively, and f(p) is the characteristic value of point p.

[0015] Preferably, the S3 includes: constructing a local feature extraction module based on a graph convolutional neural network, finding its k nearest neighbor points for each point through the k-nearest neighbor algorithm to construct a local graph, using the graph convolution layer to perform a convolution operation on the local graph according to the weight matrix, adjacency matrix and activation function to extract local features; fusing the local features based on the attention mechanism to extract global features, first calculating the attention weight of each local feature Where f is a mapping function, and the local features are weighted and summed according to the attention weight to obtain the global features. The local features reflect the geometric structure and spatial relationship information of the point cloud in the local area, and the global features reflect the overall characteristics of the entire point cloud.

[0016] Preferably, the step S4 includes: for each point, estimating the curvature by calculating the covariance matrix of its neighborhood points, and the calculation formula of the covariance matrix C is in is the centroid of the neighborhood points, and the minimum eigenvalue of the covariance matrix is ​​taken as the curvature of the point; feature points are screened according to the set curvature threshold and local feature threshold, and points whose curvature is greater than the curvature threshold and whose local features meet the corresponding threshold requirements are marked as feature points, and the feature points contain the key geometric structure information of the target object.

[0017] Preferably, S5 includes: using feature splicing or weighted fusion to fuse the local features, global features and curvature features contained in the point cloud features with the depth information and texture features extracted from the structured light image, and assigning corresponding weights to each feature according to its importance to the three-dimensional reconstruction, so that the fused features comprehensively include the geometric structure information of the point cloud and the surface and depth-related features of the object reflected by the structured light.

[0018] Preferably, S6 includes: using a voxel-based reconstruction algorithm or a triangulation-based reconstruction algorithm to gradually calculate the three-dimensional shape and structure of the target object according to the spatial position, geometric shape, and depth information contained in the fusion features, and generate a corresponding three-dimensional model.

[0019] Preferably, S7 includes: optimizing the generated three-dimensional model through a mesh smoothing algorithm to repair holes and surface unevenness in the model; comparing and verifying the reconstructed three-dimensional model with a known standard model or partial real information of the target object obtained by other measurement means, evaluating the accuracy of the reconstruction result, and outputting the final three-dimensional reconstruction result after optimization and verification.

[0020] The adaptive point cloud structured light fusion 3D reconstruction precision calculation system is used for the adaptive point cloud structured light fusion 3D reconstruction precision calculation method, including the following modules:

[0021] Point cloud acquisition module, used to collect point cloud data of target objects;

[0022] A structured light projection and acquisition module, used to project a structured light pattern onto a target object and acquire the corresponding structured light image;

[0023] A data preprocessing module, used for preprocessing the point cloud data;

[0024] A point cloud feature extraction module, used to extract features of the point cloud data;

[0025] A feature optimization module, configured to optimize the point cloud features based on geometric analysis;

[0026] A feature fusion module, configured to fuse the optimized point cloud features with the features extracted from the structured light image;

[0027] 3D reconstruction module, used to perform 3D reconstruction calculation using the fused features;

[0028] The result optimization and verification module is used to optimize and verify the reconstructed 3D model and output the final 3D reconstruction result.

[0029] The present invention provides an accurate calculation method for 3D reconstruction by adaptive point cloud structured light fusion. It has the following beneficial effects:

[0030] 1. This invention avoids the limitations of a single data source by acquiring object information from multiple perspectives. Preprocessing improves data quality, feature extraction comprehensively characterizes point cloud features, feature optimization focuses on key information, feature fusion provides an accurate basis for reconstruction, and 3D reconstruction generates a high-quality model. Model optimization and verification ensure the reliability of the results. This solves the problem that a single data source cannot fully describe the 3D spatial morphology of the target object.

[0031] 2. This invention normalizes the coordinates of point cloud data, confining them to a preset range. This results in a uniform spatial scale for the data, facilitating the stable and accurate operation of subsequent algorithms and providing a sound data foundation. Furthermore, by using a bilateral filtering algorithm to remove noise, the point cloud distribution becomes smoother, more accurately reflecting the geometry and surface features of the target object, improving the quality and accuracy of subsequent feature extraction and 3D reconstruction operations. This solves the problem of inconsistent coordinate scales in raw point cloud data, which hinders unified processing.

[0032] 3. By calculating the curvature of each point, this invention can meticulously quantify the degree of curvature of the local area where each point in the point cloud resides, accurately depicting the point cloud's geometry and providing geometrically meaningful reference information for subsequent processing. Furthermore, filtering feature points can streamline and focus on key geometric structural information, reducing data processing volume, improving the efficiency and accuracy of related operations, and helping the final result reflect the object's true shape and structural characteristics. This solves the problem of difficulty in accurately grasping changes in the geometric shape of an object's surface and the lack of an intuitive and quantitative description of it.

[0033] 4. This invention, through voxel- or triangulation-based reconstruction algorithms, can accurately restore the three-dimensional shape and structure of target objects. The generated three-dimensional model can meticulously display features such as contours, concavities and convexities, and internal space. It is highly accurate and complete, meeting the precision and realism requirements of multiple application scenarios and providing a reliable model foundation. This solves the problem of difficulty in accurately reconstructing target objects and poor reconstructed model quality caused by the lack of effective algorithms or underutilization of fusion features. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of the precise calculation method for adaptive point cloud structured light fusion 3D reconstruction proposed in the present invention;

[0035] Figure 2 This is the architecture diagram of the adaptive point cloud structured light fusion three-dimensional reconstruction precision calculation system proposed in this invention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] Please see the attached Figure 1 , an embodiment of the present invention provides an accurate calculation method for adaptive point cloud structured light fusion 3D reconstruction, comprising the following steps:

[0038] S1. Collect point cloud data of the target object and the corresponding structured light image.

[0039] S1 includes: the point cloud acquisition device obtains the original point cloud data of the target object, and uses the structured light projection device to project the structured light pattern to the target object, and uses the camera to collect the corresponding structured light image. The point cloud data reflects the spatial coordinate information of the target object, and the structured light image contains the texture and depth-related information presented on the surface of the target object due to the structured light projection.

[0040] Specifically, point cloud data acquisition: The point cloud acquisition device obtains the original point cloud data of the target object based on a specific measurement principle. For example, if a laser radar is used as a point cloud acquisition device, it will emit a laser beam when it is working. The laser beam will be reflected after it hits the surface of the target object, and the detector in the device will receive the reflected laser signal. By measuring the time difference between laser emission and reception, based on the physical principle that the speed of light is constant (according to the distance formula d=c×t / 2, where d represents the distance between the target object and the laser radar, c is the speed of light, and t is the time difference between laser emission and reception), the distance information between the laser radar and each point on the surface of the target object can be calculated. At the same time, combined with the laser radar's own scanning angle information and the internal coordinate system, the coordinate position of each reflection point in three-dimensional space can be determined, thereby generating point cloud data reflecting the spatial coordinate information of the target object.

[0041] For other types of point cloud acquisition equipment, such as equipment based on the principle of stereo vision, it will use two or more cameras to simultaneously shoot the target object from different angles, and use the principle of triangulation to determine the coordinates of the surface points of the target object in three-dimensional space based on the relative position and focal length between the two cameras and the parallax information of the corresponding points in the image through corresponding geometric calculations (the specific calculation formula is derived based on the geometric relationship of triangulation, and different equipment settings will have corresponding specific forms), thereby forming point cloud data.

[0042] Structured light image acquisition: A structured light projection device first generates a specifically coded structured light pattern. Common structured light patterns include stripes and dot patterns. For example, the light source within the projection device generates stripes of light with varying widths, spacing, and phase variations according to preset coding rules. When these stripes of light are projected onto the surface of a target object, they are deformed by the surface's shape, depth, and texture.

[0043] The camera aims at the target object from a specific angle and captures an image with a structured light pattern. Based on the principles of optical imaging, the camera lens focuses light reflected from the target object's surface onto the image sensor. The photosensitive element in the image sensor (such as a CMOS or CCD) performs photoelectric conversion based on the received light intensity information, converting the light signal into an electrical signal. After a series of processing steps, such as analog-to-digital conversion, a digital image is formed. This is a structured light image that contains information about the texture and depth of the target object's surface caused by the structured light projection.

[0044] By collecting point cloud data of the target object and the corresponding structured light image, comprehensive information about the target object can be obtained from two dimensions: spatial coordinates and surface features. This allows the object's overall spatial morphology to be presented as a set of discrete 3D coordinate points, while also recording the object's surface texture and depth information. This provides a rich, multi-angle data foundation for subsequent 3D reconstruction, helping to improve its accuracy and completeness. This addresses the information limitations inherent in a single data source.

[0045] S2. Preprocess the point cloud data.

[0046] S2 includes: normalizing the coordinates of the point cloud data so that each coordinate value is distributed within a preset normalization interval, and using the formula Normalize the x, y, and z coordinates of each point in the point cloud data, where x min ,x max ,y min ,y max ,z min ,z max They are the minimum and maximum values ​​of the point cloud on the x, y, and z axes respectively; the bilateral filtering algorithm is used to remove noise in the point cloud data. For each point p i , its filtered coordinate p i ′Through the formula Calculate, where N(i) is the point p i The neighborhood point set of σ d and σ r are the standard deviations of spatial distance and characteristic distance respectively, and f(p) is the characteristic value of point p.

[0047] Specifically, coordinate normalization processing: When normalizing the coordinates of point cloud data, a specific mathematical transformation formula is used to adjust the coordinate value range. For each point p in the point cloud data i =(x i ,y i ,z i ), taking the x coordinate as an example, through the formula Perform normalization operation. Here x min It is the minimum value of all the coordinate values ​​of the point cloud on the x-axis. max The maximum value of all the coordinate values ​​of the points on the x-axis. The principle of this formula is to first calculate the point p i The x coordinate of the point cloud is the relative position of the x coordinate range of the entire point cloud, and then it is mapped to the [0,1] interval, and then further mapped to the [-1,1] interval by multiplying it by 2 and subtracting 1. The same reasoning applies to the y coordinate and the z coordinate, which are also based on the y min 、y max and z min 、z max According to the corresponding formula and A similar normalization process is performed. The purpose of this is to unify the coordinates of point clouds of different scales and value ranges into a preset standard range, which facilitates subsequent data processing and algorithm calculations and avoids adverse effects on operations such as feature extraction and fusion due to large differences in coordinate values.

[0048] Bilateral filtering algorithm denoising: Bilateral filtering algorithm removes noise from point cloud data based on the principle of adjusting the coordinates of each point based on the comprehensive consideration of spatial distance and feature distance. i , its filtered coordinate p i ′Through the formula To calculate. Among them, N(i) represents the point p i The neighborhood point set, that is, the distance from point p in space i A set of relatively close points. ij is a weight coefficient, which consists of two parts, namely Here σ d is the standard deviation of spatial distance, which determines the influence of spatial distance factor in weight calculation, ||p i -p j || represents point p i and the neighboring point p j The spatial distance between them is expressed by the exponential function The form of distance point p i The closer the neighboring points are, the greater their weight is when calculating the weight, and vice versa. ris the standard deviation of the characteristic distance, f(p) is the characteristic value of point p, ||f(p i )-f(p j )|| represents point p i and the neighboring point p j The characteristic distance between them is also obtained through the exponential function To adjust the weight according to the difference in feature distance, the neighborhood points with similar features are given more weight when calculating the weight. Finally, by calculating the weight w for the neighborhood points ij Perform weighted averaging to obtain the filtered point p i The coordinates p i ′, so as to achieve the purpose of removing noise and smoothing point cloud data.

[0049] After coordinate normalization of the point cloud data, the coordinate values ​​are confined to a preset range, making the data spatial scale regular and uniform, which facilitates the stable and accurate operation of subsequent algorithms and provides a good data foundation. After using the bilateral filtering algorithm to remove noise, the point cloud distribution is smoother, which can more realistically reflect the geometric shape and surface features of the target object, improving the quality and accuracy of subsequent feature extraction and 3D reconstruction operations. This solves the problem of inconsistent coordinate scales in the original point cloud data, which is not conducive to unified processing.

[0050] S3. Extract features of point cloud data.

[0051] S3 includes: building a local feature extraction module based on graph convolutional neural network, finding its k nearest neighbor points for each point through k-nearest neighbor algorithm to build a local graph, using graph convolution layer to perform convolution operation on the local graph according to weight matrix, adjacency matrix and activation function to extract local features; fusing local features based on attention mechanism to extract global features, first calculating the attention weight of each local feature Where f is a mapping function, and the local features are weighted and summed according to the attention weight to obtain the global features. The local features reflect the geometric structure and spatial relationship information of the point cloud in the local area, and the global features reflect the overall characteristics of the entire point cloud.

[0052] Specifically, local feature extraction based on graph convolutional neural network: for each point in the point cloud data, the k nearest neighbor algorithm is used to determine its k nearest neighbor points. The core principle of the k nearest neighbor algorithm is based on distance measurement (usually using Euclidean distance, etc.), and finds the k points closest to the current point among all the points in the entire point cloud data. For point p in the point cloud data i By calculating the distance between it and all other points, sorting them from small to large, and selecting the first k points as its nearest neighbor points, these nearest neighbor points are related to point p i Together they form a local point set.

[0053] Based on these point sets, a local graph G = (V, E) is constructed, where the vertex set V is the vertex set of point p i and their k nearest neighbor points, and the edge set E is the edges connecting these vertices. The connection relationship of the edges can be determined based on rules such as spatial adjacency. For example, if two points are each other's nearest neighbors in space, there is an edge connecting them.

[0054] The graph convolution layer is used to perform convolution operations to extract local features. When processing this local graph, the graph convolution layer involves key elements such as the weight matrix W, the adjacency matrix A, and the activation function. The adjacency matrix A describes the connection relationship between the vertices in the local graph, and its element A ij Indicates whether there is an edge connecting vertex i and vertex j (if connected, A ij =1, otherwise A ij =0). The weight matrix W is a parameter that needs to be learned during the training process. It determines the fusion weights of different vertex features during the convolution process. When performing graph convolution operations, for vertex p i , its local characteristics The calculation formula is To carry out, where N k (i) represents point p i The set of k nearest neighbor points of is the feature representation of the previous layer (initially, it can be the initial features such as the coordinates of the point), b is the bias term, and σ is the activation function (such as the commonly used ReLU function, which introduces nonlinear factors to enable the network to learn more complex feature representations. When the input value is less than 0, the output is 0, and when the input value is greater than or equal to 0, the output is the input value itself). Through such convolution operations, the features of each vertex in the local graph can be fused and updated, and local features that can reflect the geometric structure and spatial relationship information of the point cloud in the local area can be gradually extracted.

[0055] Global feature extraction based on attention mechanism: After obtaining the local features of each point, in order to obtain the global features that can reflect the overall features of the entire point cloud, the attention mechanism is used. First, the attention weight α of each local feature is calculated. i , and its calculation formula is Here f is a mapping function, which can usually be implemented using a multi-layer perceptron (MLP). The principle is to transform local features As input, it is mapped by the multi-layer perceptron and converted into a new feature space, and then through the exponential function Calculate and get a value related to the local feature, then normalize this value to get the attention weight α iThis weight represents the relative importance of each local feature in constructing the global feature. The larger the value, the greater the contribution of the local feature to the overall feature.

[0056] All local features are weighted and summed according to their corresponding attention weights to obtain the global feature h g ,Right now In this way, it is possible to selectively fuse each local feature according to its importance, highlighting those local features that have a key impact on the overall point cloud features, thereby obtaining a global feature representation that comprehensively reflects the overall features of the entire point cloud.

[0057] Local feature extraction based on graph convolutional neural networks can reveal detailed information such as the geometric structure and spatial relationships within each local area of ​​the point cloud. Global feature extraction based on the attention mechanism can filter and fuse important local features to obtain global features that reflect the overall macroscopic characteristics, making the point cloud feature description more comprehensive and hierarchical, which is conducive to improving the accuracy of subsequent tasks. This solves the problem that traditional feature extraction methods have difficulty in effectively capturing the complex local geometric structure and spatial relationship information of point clouds.

[0058] S4. Optimize point cloud features based on geometric analysis.

[0059] S4 includes: for each point, the curvature is estimated by calculating the covariance matrix of its neighboring points. The calculation formula of the covariance matrix C is in It is the centroid of the neighborhood points, and the minimum eigenvalue of the covariance matrix is ​​taken as the curvature of the point; the feature points are screened according to the set curvature threshold and local feature threshold, and the points whose curvature is greater than the curvature threshold and the local features meet the corresponding threshold requirements are marked as feature points. The feature points contain the key geometric structure information of the target object.

[0060] Specifically, curvature calculation: For each point p in the point cloud data i , we need to calculate the covariance matrix through the information of its neighboring points to estimate the curvature of the point. First, we need to determine the point p i The neighborhood point set N(i) can usually be selected with the help of spatial distance measurement, such as setting a radius range, and the points within this range are considered to be point p i 's neighborhood points.

[0061] Calculate the centroid of neighborhood points The calculation formula is That is, sum the coordinate values ​​of all points in the neighborhood point set N(i) in the x, y, and z dimensions, and then divide it by the number of neighborhood points |N(i)| to get the centroid Coordinates in three-dimensional space.

[0062] Calculate the covariance matrix C, which is calculated as follows: Here Represents the neighborhood point p j Relative to the center of mass The covariance matrix C is obtained by multiplying the coordinate offset of the neighborhood points by their own transpose and summing the values ​​of all neighborhood points. The covariance matrix then divides the covariance matrix by the number of neighborhood points N(i). The covariance matrix mathematically reflects the changes in the coordinates of the neighborhood points in various dimensions and the correlation between them, and can reflect the geometric characteristics of the local area.

[0063] Perform eigenvalue decomposition on the covariance matrix C to obtain multiple eigenvalues ​​(set as λ1, λ2, λ3, etc.), and take the smallest eigenvalue λ min As point p i The principle is that in three-dimensional space, for a local area, if it is approximately a plane, the eigenvalues ​​of the covariance matrix will be relatively close, and if there is a curvature, the minimum eigenvalue will be relatively small and can well reflect the degree of curvature of the local area, so the minimum eigenvalue is selected to represent the point p i The curvature of the local area.

[0064] Feature point screening: pre-set the curvature threshold T k and local feature threshold T h The settings of these two thresholds can be determined according to factors such as the specific application scenario and the characteristics of the target object. For example, for some objects with relatively regular shapes and relatively smooth surfaces, the curvature threshold can be set relatively small, while for objects with complex shapes and drastic surface changes, the threshold needs to be appropriately increased.

[0065] For each point p in the point cloud data i , the curvature λ calculated by min With the curvature threshold T k Compare and compare the local features corresponding to the point (such as the local features extracted by graph convolutional neural network and other methods) ) (e.g., the modulus of the eigenvector ) and local feature threshold T h Compare. If point p i The curvature λ min Greater than the curvature threshold T k , and its local features meet the corresponding threshold requirements, then point p i Marked as feature points. These feature points are often located at key locations where the geometric shape of the object surface changes, and contain key geometric structure information of the target object. For example, points at the edges, corners, and turning points of the object are more likely to be selected as feature points.

[0066] By calculating the curvature of a point, we can meticulously quantify the degree of curvature of the local area where each point in the point cloud resides, accurately depicting the point cloud's geometry and providing geometrically meaningful reference information for subsequent processing. Furthermore, filtering feature points can streamline and focus on key geometric structural information, reducing data processing volume, improving the efficiency and accuracy of related operations, and helping the final result reflect the object's true shape and structural characteristics. This solves the problem of difficulty in accurately grasping changes in the geometric shape of an object's surface and the lack of an intuitive quantitative description of it.

[0067] S5. Fuse the optimized point cloud features with the features extracted from the structured light image.

[0068] S5 includes: using feature splicing or weighted fusion to fuse the local features, global features, and curvature features contained in the point cloud features with the depth information and texture features extracted from the structured light image, and assigning corresponding weights to each feature based on its importance to the three-dimensional reconstruction, so that the fused features comprehensively include the geometric structure information of the point cloud and the surface and depth-related features of the object reflected by the structured light.

[0069] Specifically, feature types and sources are explained: the two types of feature sources involved in the fusion are clarified. On the one hand, there are point cloud features optimized in the previous steps, which include local features. They reflect the detailed geometric structure and spatial relationship information of the point cloud in the local area, and are extracted by constructing a local graph based on a graph convolutional neural network and performing convolution operations; global features reflect the overall characteristics of the entire point cloud, and are obtained by weighted summation of local features with the help of an attention mechanism; and curvature features are obtained by calculating the minimum eigenvalue based on the covariance matrix of the neighborhood points of each point, which is used to describe the degree of curvature of the local area where the point is located. Together, these point cloud features describe the target object from a geometric structural level.

[0070] On the other hand, the features extracted from the structured light image mainly include depth information and texture features. Depth information is based on the deformation of the structured light pattern due to the shape, concave and convex conditions of the surface of the target object after the structured light is projected onto the surface of the target object. Then, a specific algorithm (such as triangulation principle, phase analysis, etc., calculated according to the deformation of the structured light pattern and known projection and shooting parameters) is used to parse the depth value of each point on the surface of the object, thereby reflecting the relative height and concave and convex conditions of different parts of the surface of the object. Texture features are obtained by performing image processing operations on the structured light image (for example, using texture analysis algorithms such as grayscale co-occurrence matrix and local binary pattern to analyze the distribution pattern of pixel grayscale in the image in different directions and distances). It reflects the appearance characteristics of the object surface such as roughness and color change.

[0071] Feature splicing method: If the feature splicing fusion method is adopted, the local feature vectors, global feature vectors and curvature feature values ​​in the point cloud features are arranged in a certain order to form a new joint feature vector. Similarly, the depth information vector and texture feature vector extracted from the structured light image are also spliced ​​together in the corresponding order, and then the two sets of spliced ​​feature vectors are further spliced ​​together to form a fused feature vector with larger dimensions and richer information. For example, assuming that the dimension of the local feature vector is m, the dimension of the global feature vector is n, the curvature feature value is a single number, the dimension of the depth information vector is p, and the dimension of the texture feature vector is q, then the dimension of the spliced ​​fused feature vector is m+n+1+p+q. This method is simple and direct, and can fully retain the original information of each feature, but it may make the dimension of the fused feature vector higher, and the amount of calculation in subsequent processing will increase accordingly.

[0072] Weighted fusion method: For the weighted fusion method, it is necessary to assign corresponding weights to each feature based on its importance to the 3D reconstruction. First, it is necessary to evaluate the importance of different features in the 3D reconstruction process, which can be determined through experimental analysis, prior knowledge, or based on an understanding of the characteristics of the target object. For example, for objects with complex shapes and rich geometric structure variations, the local features and curvature features of the point cloud may be more critical to accurately reconstructing their shape, so a relatively high weight will be assigned to these features; and for situations where the surface texture is rich and the appearance features are important for identifying the object, the weight of the texture features in the structured light image will be appropriately increased.

[0073] Let the local eigenvector of the point cloud be F local , with weight ω local ; The global eigenvector is F global , with weight ω global ; The curvature eigenvalue is k and the weight is ω curυature ; The depth information vector of the structured light image is F depth , with weight ω depth ; Texture feature vector is F texture , with weight ω texture , and satisfies ω local +ω global +ω curυature +ω depth +ω texture =1. Then the weighted fusion feature vector F fusion The calculation formula is: F fusion =ω local F local +ω global F global +ω curυature k+ω depth F depth +ωtexture F texture This weighted fusion approach allows for focused integration of information based on the importance of different features, avoiding the problem of excessive dimensionality caused by simple splicing. It also highlights the role of key features in the fused features, making the fused features more suitable for the actual needs of 3D reconstruction.

[0074] By fusing point cloud features with structured light image features through feature concatenation or weighted fusion, the fused features can comprehensively describe the characteristics of the target object, covering its internal geometry, external surface, depth, and other information. This provides a richer, more comprehensive, and complementary information foundation for 3D reconstruction, improving reconstruction accuracy and quality and generating more realistic 3D models. This solves the problem of incomplete information and the inability to fully and accurately describe the 3D characteristics of the target object when relying solely on point cloud features or structured light image features.

[0075] S6. Use the fused features to perform three-dimensional reconstruction calculation.

[0076] S6 includes: using a voxel-based reconstruction algorithm or a triangulation-based reconstruction algorithm to gradually calculate the three-dimensional shape and structure of the target object according to the spatial position, geometric shape, and depth information contained in the fusion features, and generate a corresponding three-dimensional model.

[0077] Specifically, the voxel-based reconstruction algorithm divides the three-dimensional space into regular small cube units, called voxels. When using the fused features for voxel-based reconstruction, the relationship between each voxel and the target object is determined based on the spatial position, geometry, and depth information contained in the fused features.

[0078] For voxel status judgment, for example, by analyzing the spatial position information in the fused features, the approximate range of the target object in the entire three-dimensional space is determined, thereby defining which voxels may be inside, on the surface, or outside the target object. Combined with geometric morphological information, it is further determined how the curvature, concavity, and other geometric features of the object's surface are reflected at the voxel level. For example, if the fused features of a certain area indicate a curved surface, the voxel distribution and status of the corresponding position will also show characteristics consistent with a curved surface.

[0079] Depth information helps determine the spatial order of different parts of an object, specifically the arrangement of voxels in the depth direction. By continuously analyzing these aspects of the fused features, each voxel is labeled, for example, as occupied by an object, on the surface of an object, or unassociated with the object. As the states of all relevant voxels are determined, those labeled as belonging to the target object are eventually combined to gradually outline the target object's 3D shape and structure, ultimately generating a corresponding 3D model.

[0080] Principle of triangulation-based reconstruction algorithms: Triangulation-based reconstruction algorithms construct triangular facets based on the points in the point cloud data and the topological relationships between them, thereby forming a 3D model. When using the fused features, key information such as the spatial position and geometric shape of each point is first extracted from the fused features.

[0081] The distribution of points in three-dimensional space is determined based on spatial position information. Then, based on geometric features, such as the curvature and planarity of the local area where the points are located, suitable points are found to construct triangular facets. For example, for points in a nearly flat area, adjacent points that can form reasonable planar triangles are prioritized to form triangular facets. For curved areas, appropriate points are selected to form triangular facets that fit the surface based on the curvature of the surface and the relevant geometric information in the fusion features.

[0082] During the construction of triangular facets, depth information is also used to ensure the correct order of the triangles and the overall spatial hierarchy. By continuously searching for appropriate points to combine into triangular facets and ensuring that the connections and spatial relationships between the triangles conform to the actual conditions of the target object, the numerous triangular facets are gradually pieced together to ultimately construct the complete 3D shape and structure of the target object, generating the corresponding 3D model.

[0083] Using voxel- or triangulation-based reconstruction algorithms, the 3D shape and structure of the target object can be accurately restored. The generated 3D model can meticulously display features such as its contours, concavities and convexities, and internal space. With high accuracy and integrity, it can meet the precision and realism requirements of multiple application scenarios and provide a reliable model foundation. This solves the problem of difficulty in accurately reconstructing the target object and poor reconstructed model quality caused by the lack of effective algorithms or the underutilization of fusion features.

[0084] S7. Optimize and verify the reconstructed three-dimensional model and output the final three-dimensional reconstruction result.

[0085] S7 includes: optimizing the generated 3D model through a mesh smoothing algorithm to repair holes, uneven surfaces and other problems in the model; comparing and verifying the reconstructed 3D model with a known standard model or partial real information of the target object obtained by other measurement methods, evaluating the accuracy of the reconstruction result, and outputting the final 3D reconstruction result after optimization and verification.

[0086] Specifically, the optimization principle of the mesh smoothing algorithm is as follows: The mesh smoothing algorithm mainly operates on the mesh structure of a 3D model. A 3D model is usually represented by a mesh composed of numerous triangular facets (or other polygonal facets). These facets are connected by vertices to form the topology of the entire model.

[0087] The core of the mesh smoothing algorithm lies in adjusting the positions of vertices in the model. For example, the common Laplace smoothing algorithm calculates the new position of each vertex based on the neighborhood relationship of the vertex. For any vertex in the model, first determine its neighboring vertices, that is, the vertices directly connected to it. Then, calculate the average value of the coordinates of these neighboring vertices (or calculate the weighted average value according to different weighting strategies, and the weighting coefficient can be set according to factors such as the distance between vertices and the angle of the facet), and use this average value as the estimated value of the new position of the vertex. By continuously iterating this process, the distribution of vertices on the surface of the model becomes more uniform and smooth, thereby repairing problems such as holes and surface unevenness in the model.

[0088] There are also mesh smoothing methods based on partial differential equations. These treat the surface of a 3D model as a continuous curved surface. By establishing and solving the corresponding partial differential equations, they drive the changes in vertex positions, making the geometry of the model surface evolve towards a smoother, more physically accurate form, thereby optimizing the model's surface quality. For example, the heat diffusion equation is used to simulate the transfer of heat across a model's surface. During this process, the positions of vertices are gradually adjusted as the "heat" diffuses, ultimately achieving surface smoothing.

[0089] Comparative Verification Principle: When performing comparative verification of reconstruction results, first obtain a known standard model or some real information about the target object obtained using other measurement methods. The known standard model may be a pre-made model using high-precision modeling tools that accurately reflects the target object's true geometry. Other measurement methods, such as high-precision coordinate measuring machines, can obtain precise dimensions and shape data for key parts of the target object.

[0090] Appropriate metrics are used to compare the differences between the reconstructed 3D model and the reference information. Common metrics include Hausdorff distance, which measures the maximum distance between two point sets. For a 3D model, the vertices of its surface can be considered as point sets. By calculating the Hausdorff distance between the reconstructed model and the standard model (or the point set corresponding to the real information), the degree of difference in the model's overall shape and spatial layout is evaluated. Another metric is mean squared error (MSE), which reflects the deviation of the model from the real situation in local details by calculating the average of the squared distances between corresponding points.

[0091] The calculation results of these metrics can be used to quantitatively judge the accuracy of the reconstruction results and understand in which aspects the reconstructed model differs from the real object, thereby providing a basis for subsequent optimization and improvement or evaluation of the reconstruction results.

[0092] By optimizing 3D models using mesh smoothing algorithms, holes can be filled, surfaces can be smoothed, and overall quality and visual effects can be improved, facilitating subsequent applications and analysis. Comparative verification clearly identifies the differences between the reconstructed model and the real object, objectively assessing accuracy and enhancing the reliability and practicality of 3D reconstruction, ensuring that the output meets requirements. This solves the problem of poor surface quality and imperfections in reconstructed 3D models that hinder their use.

[0093] Please see the attached Figure 2 , Adaptive point cloud structured light fusion 3D reconstruction precise calculation system, used for adaptive point cloud structured light fusion 3D reconstruction precise calculation method, including the following modules:

[0094] Point cloud acquisition module, used to collect point cloud data of target objects;

[0095] A structured light projection and acquisition module, used to project a structured light pattern onto a target object and acquire the corresponding structured light image;

[0096] Data preprocessing module, used to preprocess point cloud data;

[0097] Point cloud feature extraction module, used to extract features of point cloud data;

[0098] Feature optimization module, used to optimize point cloud features based on geometric analysis;

[0099] Feature fusion module, used to fuse the optimized point cloud features with the features extracted from the structured light image;

[0100] 3D reconstruction module, used to perform 3D reconstruction calculation using the fused features;

[0101] The result optimization and verification module is used to optimize and verify the reconstructed 3D model and output the final 3D reconstruction result.

[0102] First, the point cloud data and structured light image of the target object are collected. Then, the cloud data is preprocessed, feature extracted, and feature optimized based on geometric analysis. The optimized point cloud features are fused with the structured light image features. The fused features are then used to perform three-dimensional reconstruction using a voxel or triangulation algorithm. Finally, the reconstructed model is optimized using a mesh smoothing algorithm and the results are output after comparison and verification. Each step is operated according to the corresponding mathematical formulas and algorithm rules.

[0103] By acquiring object information from multiple perspectives, the limitations of a single data source are avoided. Preprocessing improves data quality, feature extraction comprehensively characterizes point cloud features, feature optimization focuses on key information, feature fusion provides an accurate basis for reconstruction, and 3D reconstruction generates high-quality models. Model optimization and verification ensure reliable results that meet the needs of multiple application scenarios. This solves the problem that a single data source cannot fully describe the 3D spatial form of the target object.

[0104] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. Adaptive point cloud structured light fusion 3D reconstruction accurate calculation method, characterized by: The following steps are involved: S1, collect point cloud data of the target object and the corresponding structured light image; S2. Preprocessing the point cloud data; S3, extracting features of the point cloud data; S4. Optimizing the point cloud features based on geometric analysis; S5. fusing the optimized point cloud features with the features extracted from the structured light image; S6. Perform three-dimensional reconstruction calculation using the fused features; S7. Optimize and verify the reconstructed three-dimensional model and output the final three-dimensional reconstruction result.

2. The method for accurate calculation of adaptive point cloud structured light fusion 3D reconstruction according to claim 1, characterized in that: S1 includes: a point cloud acquisition device acquires original point cloud data of a target object, a structured light projection device is used to project a structured light pattern onto the target object, and a camera is used to acquire a corresponding structured light image, wherein the point cloud data reflects the spatial coordinate information of the target object, and the structured light image includes texture and depth-related information presented on the surface of the target object due to the structured light projection.

3. The method for accurate calculation of adaptive point cloud structured light fusion 3D reconstruction according to claim 1, characterized in that: The step S2 includes: normalizing the coordinates of the point cloud data so that each coordinate value is distributed within a preset normalization interval, and performing the normalization process by the formula Normalize the x, y, and z coordinates of each point in the point cloud data, where x min ,x max ,y min ,y max ,z min ,z max They are the minimum and maximum values ​​of the point cloud on the x, y, and z axes respectively; the bilateral filtering algorithm is used to remove noise in the point cloud data. For each point p i , its filtered coordinate p i ′Through the formula Calculate, where N(i) is the point p i The neighborhood point set of σ d and σ r are the standard deviations of spatial distance and characteristic distance respectively, and f(p) is the characteristic value of point p.

4. The method for accurate calculation of adaptive point cloud structured light fusion 3D reconstruction according to claim 1, characterized in that: The S3 includes: constructing a local feature extraction module based on a graph convolutional neural network, finding its k nearest neighbor points for each point through the k-nearest neighbor algorithm to construct a local graph, using the graph convolution layer to perform a convolution operation on the local graph according to the weight matrix, adjacency matrix and activation function to extract local features; fusing the local features based on the attention mechanism to extract global features, first calculating the attention weight of each local feature Where f is a mapping function, and the local features are weighted and summed according to the attention weight to obtain the global features. The local features reflect the geometric structure and spatial relationship information of the point cloud in the local area, and the global features reflect the overall characteristics of the entire point cloud.

5. The method for accurate calculation of adaptive point cloud structured light fusion 3D reconstruction according to claim 1, characterized in that: The step S4 includes: for each point, estimating the curvature by calculating the covariance matrix of its neighborhood points. The calculation formula of the covariance matrix C is: in is the centroid of the neighborhood points, and the minimum eigenvalue of the covariance matrix is ​​taken as the curvature of the point; feature points are screened according to the set curvature threshold and local feature threshold, and points whose curvature is greater than the curvature threshold and whose local features meet the corresponding threshold requirements are marked as feature points, and the feature points contain the key geometric structure information of the target object.

6. The adaptive point cloud structured light fusion 3D reconstruction precision calculation method according to claim 1, characterized in that: The S5 includes: using feature splicing or weighted fusion to fuse the local features, global features and curvature features contained in the point cloud features with the depth information and texture features extracted from the structured light image, and assigning corresponding weights to each feature according to its importance to the three-dimensional reconstruction, so that the fused features comprehensively include the geometric structure information of the point cloud and the surface and depth-related features of the object reflected by the structured light.

7. The method for accurate calculation of adaptive point cloud structured light fusion 3D reconstruction according to claim 1, characterized in that: The S6 includes: using a voxel-based reconstruction algorithm or a triangulation-based reconstruction algorithm to gradually calculate the three-dimensional shape and structure of the target object according to the spatial position, geometric form, and depth information contained in the fusion features, and generate a corresponding three-dimensional model.

8. The method for accurate calculation of adaptive point cloud structured light fusion 3D reconstruction according to claim 1, characterized in that: The S7 includes: optimizing the generated three-dimensional model through a mesh smoothing algorithm to repair holes and surface unevenness in the model; comparing and verifying the reconstructed three-dimensional model with a known standard model or partial real information of the target object obtained by other measurement methods, evaluating the accuracy of the reconstruction result, and outputting the final three-dimensional reconstruction result after optimization and verification.

9. Adaptive point cloud structured light fusion 3D reconstruction precision calculation system, characterized by: The method for accurately calculating the adaptive point cloud structured light fusion three-dimensional reconstruction according to any one of claims 1 to 8 comprises the following modules: Point cloud acquisition module, used to collect point cloud data of target objects; A structured light projection and acquisition module, used to project a structured light pattern onto a target object and acquire the corresponding structured light image; A data preprocessing module, used for preprocessing the point cloud data; A point cloud feature extraction module, used to extract features of the point cloud data; A feature optimization module, configured to optimize the point cloud features based on geometric analysis; A feature fusion module, configured to fuse the optimized point cloud features with the features extracted from the structured light image; 3D reconstruction module, used to perform 3D reconstruction calculation using the fused features; The result optimization and verification module is used to optimize and verify the reconstructed 3D model and output the final 3D reconstruction result.

Images