Handheld 3D scanning system based on visible light

Abstract

The present invention discloses a handheld three-dimensional scanning system based on visible light, which belongs to the technical field of electronic scanning and comprises: a calibration module, an adaptive adjustment module, a three-dimensional scanning module and a storage module; the calibration module is used to calibrate the three-dimensional scanner by using a known reference object; the adaptive adjustment module is used to automatically optimize scanning parameters according to the material and surface characteristics of the target object to adapt to the surface characteristics of different materials; the three-dimensional scanning module acquires multi-angle reflected images of the surface of the target object in real time, identifies geometric information of the surface of the target object in the image, constructs a three-dimensional surface model of the target object, reduces the error between the model and the target object by fusing the multi-angle images, improves the accuracy of the three-dimensional model, captures the texture information of the target object, and enhances the realistic effect of the model; the storage module is used to store data generated during the scanning process, and establishes an efficient data index according to the stored data.

Description

Technical Field

[0001] The invention belongs to the technical field of electronic scanning and relates to a handheld three-dimensional scanning system based on visible light. Background Art

[0002] 3D scanners, as the core application of 3D scanning technology, are high-precision scientific equipment. Their core function is to accurately detect and deeply analyze the shape features and appearance details of various objects in the real world and even complex environments. They have achieved a leap from physical objects to digital signals that can be directly recognized and operated by computers, completely subverting the traditional way of measuring objects and opening a new era of non-contact precision measurement. 3D scanners use advanced technology to collect data, carry out 3D reconstruction calculations, and accurately reproduce the 3D model of real objects in virtual space. The scanner generates point cloud data on the surface of the object, accurately outlining the contours and shapes of the object; the density of the point cloud determines the accuracy of the model, and this process is called "3D reconstruction". If the scanner can capture the color of the object's surface, you can add material maps to the model to make the virtual model more realistic. This process is called "material mapping."

[0003] 3D scanning technology is widely used in various industries that require 3D information. In industry, it helps product design, quality inspection and reverse engineering, improving efficiency and quality; in the cultural field, it becomes the key to cultural relics protection, artwork reproduction and archiving; in medicine, it is used for surgical planning, orthosis production, etc., providing personalized and precise services. In addition, it also plays an important role in the fields of architecture, film and television, games, etc., promoting industry innovation and development.

[0004] The existing Chinese patent with announcement number CN108931205B discloses a 3D scanning system and scanning method, wherein the 3D scanning method includes setting a turntable, selecting a support structure with a set length, a set height, and a set angle, setting a 3D scanning module on the support structure, and aligning it with the turntable; the 3D scanning module determines the distance and angle with the turntable, calculates the internal and external parameters of the image acquisition unit, and selects the corresponding calibration file; the object to be measured is placed on the turntable, and scanning is started, and the 3D scanning module shoots the object to be measured to generate a 3D point cloud; the turntable rotates a fixed angle on the X-axis plane, and the 3D scanning module shoots the object to be measured to generate an X-axis 3D point cloud; the turntable swings a fixed angle on the Y-axis, and the 3D scanning module shoots the object to be measured to generate a Y-axis 3D point cloud; the 3D point cloud photographed on the X-axis plane and the 3D point cloud photographed on the Y-axis are merged to form 3D data of the object to be measured.

[0005] Although the existing technology is more convenient to operate and has no blind spots in scanning, it does not consider the impact of different materials on the scanning effect, the further improvement of scanning accuracy and integrity, and the efficiency of real-time data processing and 3D model generation. Therefore, this application provides a handheld 3D scanning system based on visible light. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a handheld three-dimensional scanning system based on visible light, which improves the system's scanning ability of different materials or colors by adaptively adjusting device parameters and using a camera to capture three-dimensional information of the object's surface. The object is scanned by multiple cameras built into the three-dimensional scanner without the need for additional supporting structures, thereby improving scanning accuracy and application flexibility, and capturing and processing data in real time during the scanning process to generate a three-dimensional model of the object.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A handheld three-dimensional scanning system based on visible light, comprising: a calibration module, an adaptive adjustment module, a three-dimensional scanning module and a storage module;

[0009] The calibration module is used to calibrate the three-dimensional scanner using a known reference object;

[0010] The adaptive adjustment module is used to automatically optimize scanning parameters according to the material and surface characteristics of the target object;

[0011] The three-dimensional scanning module acquires the reflected images of the target object surface from multiple angles in real time, identifies the geometric information of the target object surface in the image, and constructs a three-dimensional surface model of the target object;

[0012] The storage module is used to store the data generated during the scanning process and to establish a data index based on the stored data;

[0013] The three-dimensional scanning module includes a feature extraction unit, a model building unit and a texture positioning unit;

[0014] The feature extraction unit is used to obtain multi-angle reflection images of the surface of the target object, identify geometric information of the surface of the target object in the image, and generate a set of feature points;

[0015] The model building unit is used to match the feature point set in the multi-angle image information of the target object to build a three-dimensional surface model;

[0016] The texture positioning unit is used to obtain a texture image of the target object, extract texture information in the texture image, and map the texture information onto the three-dimensional surface model;

[0017] The specific steps of texture information mapping include:

[0018] Synchronously acquire the texture image of the target object and pre-process the texture image;

[0019] Use the Canny edge detection algorithm and Harris corner detection algorithm to generate a set of feature points of the texture image;

[0020] Acquire a set of feature points of the second image, and calculate the similarity between the texture image and each feature point in the second image;

[0021] Set a texture similarity threshold, filter out texture matching point pairs that meet the conditions, and generate a texture matching set ;in, For the second image pixels, For the texture image pixels;

[0022] Calculate the affine transformation matrix using the least squares method , so that , and based on the 3D point cloud dataset, calculate the texture coordinate set ;

[0023] Calculate any interpolation point in a 3D surface model using bilinear interpolation Texture coordinates ;

[0024] Among them, for any point on the three-dimensional surface model, the corresponding texture coordinates can be calculated through affine transformation. , , , , , , are the parameters of the affine transformation, is the three-dimensional coordinate set in the three-dimensional point cloud dataset;

[0025] Use bilinear interpolation to calculate the texture coordinates of other points in the 3D surface model; determine the four known points , , , , these four known points are vertices on the three-dimensional surface model and already have texture coordinates; get an interpolation point to be calculated ,point Located by , , , Inside or on the surface of a tetrahedron formed by four points, according to the interpolation point Calculate the weight coefficient with the four known points, and use the weight coefficient and the texture coordinates of the four known points to estimate the interpolation point by weighted averaging. The texture coordinates are as follows:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] In the formula, , , , Relative to point , , , The weight coefficient of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of Interpolation point The three-dimensional coordinates of Interpolation point The texture coordinates of

[0032] Using the texture coordinates of each pixel point in the texture image, extracting the pixel color value in the texture image, and applying the pixel color value to the corresponding point of the three-dimensional surface model to generate a three-dimensional surface model with texture information;

[0033] The specific steps for preprocessing texture images include:

[0034] Traverse the pixel points in the texture image and record the coordinates of the pixel points and color value RGB;

[0035] Define a neighborhood window with the pixel to be judged in the texture image as the center, obtain similar pixels in the neighborhood, and calculate the similarity ratio of the pixel to be judged ;

[0036] Convert the pixels in the neighborhood into grayscale values ​​and calculate the gradient amplitude of the pixel to be determined ;

[0037] Set the similarity threshold to , gradient amplitude threshold , determine whether the pixel to be determined is a texture pixel; if and , the pixel to be determined is a texture pixel; otherwise, the pixel to be determined is a non-texture pixel;

[0038] Traversing the pixel points in the texture image to construct a texture set and a non-texture set;

[0039] Acquire a texture image set taken by a 3D scanner from other angles, extract a correction pixel point that matches the non-texture pixel point, and determine whether the correction pixel point is a texture pixel point;

[0040] If the correction pixel is a texture pixel, the color value is corrected by using the correction pixel, and the texture set is updated until the texture image set is traversed;

[0041] Otherwise, the average value of the texture pixels in the neighborhood is calculated, the color value is corrected, and the texture set is updated;

[0042] Converts a collection of textures containing texture pixel information into a texture image.

[0043] Specifically, the specific steps of automatically optimizing scanning parameters include:

[0044] Obtain a standard reflection map in the system and calculate the standard brightness mean of the standard reflection map and standard contrast ;

[0045] Initializing parameters of the three-dimensional scanner;

[0046] Project a light pattern onto the target object, obtain a reflected light pattern, and calculate the brightness average of the reflected light pattern and contrast ;

[0047] Automatically adjust the brightness and contrast of the white light projector. The expression is as follows:

[0048]

[0049]

[0050] In the formula, is the brightness of the white light projector at the current moment, is the brightness of the white light projector after adjustment, is the brightness adjustment coefficient, is the contrast of the white light projector after adjustment, is the contrast of the white light projector at the current moment, is the contrast adjustment factor.

[0051] Specifically, the specific steps of automatically optimizing scanning parameters also include:

[0052] Re-project the light pattern onto the target object and calculate the brightness average of the adjusted reflected light pattern and contrast ;

[0053] Set the brightness difference threshold to , the contrast difference threshold is , judging whether the quality of the adjusted reflected light pattern is qualified;

[0054] like and , the pattern quality is qualified and the automatic adjustment ends;

[0055] like or , the pattern quality is unqualified, continue to adjust the brightness and contrast of the white light projector.

[0056] Specifically, the specific steps of geometric information recognition include:

[0057] Acquire a reflection image of the target object, and preprocess the reflection image;

[0058] Calculate the pixel points in the reflected image The gradient magnitude and direction , and use the Canny edge detection algorithm to generate the edge image of the reflected image; the expression is as follows:

[0059]

[0060]

[0061] In the formula, and The images are and Directional gradient;

[0062] Calculate the pixel points in the reflected image The autocorrelation matrix in the local area , and generate a response function ;in, is an empirical constant;

[0063] Set the response threshold to , determine the pixel Is it a corner point? , pixel is a corner point; if , pixel Not a corner point;

[0064] Based on the Canny edge detection algorithm and the Harris corner detection algorithm, a set of feature points is generated, and the quantity and quality of the extracted geometric features are evaluated.

[0065] Specifically, the specific steps of feature point set matching include:

[0066] Acquire multi-angle reflection images of the target object and a set of feature points of each reflection image;

[0067] Calculate the similarity of each feature point in any two reflected images. The expression for similarity calculation is as follows:

[0068]

[0069] In the formula, , is the descriptor vector of the feature points in any two reflected images, is the dimension of the descriptor vector, , Represents vectors and The Quantity;

[0070] Set a similarity measurement threshold, filter out matching point pairs that meet the conditions, and generate a matching point pair set.

[0071] Specifically, the specific steps of constructing a three-dimensional surface model include:

[0072] For each matching point pair, calculate the horizontal distance difference between the matching point pair in the first image and the third image. , and extract the two-dimensional coordinates of the corresponding matching point pairs in the second image. The expression of the horizontal distance difference is as follows:

[0073]

[0074] In the formula is the pixel in the first image The coordinates in the direction, is the pixel in the third image The coordinates in the direction, the reflected images obtained by the top, middle and bottom cameras are defined as the first image, the second image and the third image respectively;

[0075] Obtaining calibration parameters of the three-dimensional scanner, including internal parameters and external parameters;

[0076] Calculate the depth information of each feature point in the second image , and combine the depth information with the corresponding two-dimensional coordinates to obtain the three-dimensional coordinates of each feature point; the expression of the depth information is as follows:

[0077]

[0078] In the formula, is the focal length of the camera in the three-dimensional scanner, is the baseline distance, which means the distance between the top and bottom cameras;

[0079] Each feature point and its three-dimensional coordinates on the second image plane are used as points in the point cloud to generate a three-dimensional point cloud data set, and the three-dimensional point cloud data set is preprocessed.

[0080] Specifically, the specific steps of constructing the three-dimensional surface model also include:

[0081] Select a point from the point cloud data set as a starting point, find two points closest to the starting point, and form an initial triangle;

[0082] Taking the initial triangle as a starting point, gradually adding the remaining points in the point cloud data set to the triangular mesh, and updating the triangular mesh;

[0083] Set an area threshold to determine whether triangles should be merged;

[0084] If the area of ​​a triangle is smaller than the area threshold, the triangle is defined as a small triangle and a merge operation is performed;

[0085] If the area of ​​the triangle is greater than or equal to the area threshold, no merging operation is performed;

[0086] The Laplace smoothing method is used to adjust the position of the vertices and generate a three-dimensional surface model of the target object. The expression is as follows:

[0087]

[0088] In the formula, is the updated vertex position, is the position of the adjacent vertices, is the number of adjacent vertices.

[0089] Beneficial effects of the present invention:

[0090] 1. The adaptive adjustment module automatically optimizes the scanning parameters according to the material and surface characteristics of the target object, ensuring that high-quality image data can be obtained under different material and surface conditions. This not only improves the imaging quality, but also significantly improves the measurement accuracy, enabling the scanner to cope with a wider range of application scenarios;

[0091] 2. By projecting the stripe pattern through a white light projector and using multiple geometric tracking cameras to capture the deformation pattern from different angles, the scanning blind area is effectively avoided, while improving the accuracy and integrity of the scanning; combined with the 3D reconstruction algorithm, the shape and position of the target object surface can be accurately derived; the independent geometric information provided by multiple cameras verifies and complements each other, effectively reducing errors and improving the accuracy of the 3D model; this multi-angle scanning strategy helps to capture the subtle features and edge contours of the object, making the generated 3D model more realistic and accurate.

[0092] 3. Through the collaborative work of multiple cameras and color cameras, the texture information of the target object's surface is captured by the equipped color camera, and this information is mapped onto the 3D model, which enhances the system's ability to capture surface details of the object, so that even tiny textures and bumps can be accurately recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a structural diagram of a handheld 3D scanning system based on visible light;

[0094] Figure 2 Flowchart for automatically optimizing scanning parameters for visible light-based handheld 3D scanning systems;

[0095] Figure 3 This is a flow chart of geometric information recognition for a handheld 3D scanning system based on visible light;

[0096] Figure 4 Flowchart for constructing a 3D surface model for a visible light based handheld 3D scanning system;

[0097] Figure 5 This is the flow chart of texture information mapping for a handheld 3D scanning system based on visible light. DETAILED DESCRIPTION

[0098] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0099] Example 1

[0100] refer to Figures 1 to 5 As shown, this embodiment introduces a handheld three-dimensional scanning system based on visible light, including: a calibration module, an adaptive adjustment module, a three-dimensional scanning module and a storage module;

[0101] The calibration module is used to calibrate the 3D scanner using known reference objects, such as a user calibration plate, to ensure that the 3D scanner maintains optimal working condition during use, promptly discovers and corrects possible problems, and avoids measurement errors caused by equipment performance degradation, thereby improving the accuracy and stability of the scanning results; this is because changes in the environment will affect the calibration of the 3D scanner. Environmental changes are mostly caused by pressure or temperature differences, and therefore require modification of the mechanical configuration; for example, changes in temperature and humidity may cause mechanical parts to expand or contract, thereby affecting the structural stability of the scanner, changes in light conditions may affect the quality of image acquisition, and electromagnetic interference may affect the performance of electronic equipment.

[0102] The adaptive adjustment module is used to automatically optimize scanning parameters according to the material and surface characteristics of the target object; use the built-in projector of the 3D scanner to project a specific light pattern onto the target object, and use a high-resolution camera to capture the reflected light pattern. Based on the data of the reflected light pattern, the light source information is automatically optimized to adapt to the characteristics of different material surfaces, thereby improving imaging quality and measurement accuracy.

[0103] The specific steps for automatically optimizing scanning parameters include:

[0104] Obtain the reflected light pattern under the optimal state of the system configuration, record it as the standard reflection map, and use image processing software such as MATLAB to calculate the standard brightness mean of the standard reflection map and standard contrast , the expression is as follows:

[0105]

[0106]

[0107] In the formula, is the total number of pixels in the standard reflection map, The standard reflection map The brightness value of each pixel, ;

[0108] Initialize the parameters of the white light projector and the geometry tracking camera, such as the default brightness and contrast of the white light projector and the default shutter speed of the geometry tracking camera;

[0109] Use a white light projector to project a white light stripe pattern onto the target object. At this time, the white light stripe pattern is a series of parallel white lines. Synchronously start the geometric tracking camera to obtain the reflected light pattern and calculate the average brightness of the reflected light pattern. and contrast ;

[0110] The brightness and contrast of the white light projector are automatically adjusted using the successive approximation method. The expressions are as follows:

[0111]

[0112]

[0113] In the formula, is the brightness of the white light projector at the current moment, The brightness after adjustment for the white light projector, is the brightness adjustment coefficient, Adjusted contrast for white light projectors, is the contrast of the white light projector at the current moment, is the contrast adjustment coefficient, and Determined by those skilled in the art according to specific application requirements;

[0114] Use the adjusted white light projector to re-project the white light stripe pattern onto the target object and calculate the brightness mean of the adjusted reflected light pattern. and contrast ; At this time, the adjusted calculated value is used to overwrite the original calculated value;

[0115] Set the brightness difference threshold to , the contrast difference threshold is , determine whether the quality of the adjusted reflected light pattern is qualified; if and , the pattern quality is qualified, and the automatic adjustment ends; if or , the pattern quality is unqualified, continue to adjust the brightness and contrast of the white light projector; among them, and It is determined by those skilled in the art according to specific application requirements.

[0116] The 3D scanning module is used to obtain multi-angle reflection images of the surface of the target object in real time, identify the geometric information of the surface of the target object in the image, and construct a 3D surface model of the target object; however, the geometric information of the surface of some objects is not obvious, and the specific geometric information cannot be identified during the scanning process. It is impossible to construct a complete 3D surface model using only the geometric information of the surface of the object; when the system detects that the geometric information of the surface of some objects is not obvious and cannot be accurately identified, it automatically reminds the staff to paste the positioning target points. These positioning target points are specific marks or features on or around the target object, providing additional positioning information. By combining the geometric information and the information of the positioning target points, the 3D surface model of the target object is reconstructed to improve the accuracy and reliability of the scan; at the same time, the 3D scanner is also equipped with a color camera to capture the texture information of the surface of the target object, including the pattern, color change, and glossiness of the surface of the target object, and map the texture to the 3D model to enhance the realism of the model;

[0117] The three-dimensional scanning module includes a feature extraction unit, a model building unit and a texture positioning unit;

[0118] The feature extraction unit is used to obtain the multi-angle reflection image of the target object surface, and use the edge detection and corner detection algorithms to identify the geometric information of the target object surface in the image and generate a feature point set;

[0119] The model building unit fuses the feature point sets in the multi-angle image information of the target object, accurately matches the feature point sets from different angles through the feature matching algorithm, forms a complete set of matching point pairs, and gradually builds the three-dimensional surface model of the target object based on the spatial position relationship of the feature points using the three-dimensional reconstruction technology;

[0120] The texture positioning unit is used to obtain the texture image of the target object, extract the texture information in the texture image, and map the texture information onto the three-dimensional surface model.

[0121] In this embodiment, the 3D scanner projects white light stripe patterns onto the surface of the target object through a white light projector. These white light stripe patterns are deformed on the surface of the target object, reflecting the 3D shape of the surface of the target object. At the same time, three geometric tracking cameras placed at the top, middle and bottom of the 3D scanner capture the deformation of the pattern from different angles and record these deformed patterns. A 3D reconstruction algorithm is used to deduce the shape and position of the surface of the target object, construct the 3D geometric information of the surface of the object, determine whether the 3D geometric information is obvious and accurate enough for positioning, paste the positioning target point based on the judgment result, and combine the pattern information captured by multiple cameras for fusion and correction. Each camera provides independent geometric information, which verifies and complements each other, thereby reducing errors and improving the accuracy of the 3D model. The geometric information is used to construct a 3D surface model of the target object, and the 3D surface model is post-processed, including smoothing and repairing, to improve the accuracy and realism of the model.

[0122] like Figure 3 As shown, specifically, the specific steps of geometric information recognition include:

[0123] Obtain the reflected image of the target object and preprocess the reflected image, such as using Gaussian filtering to denoise the image, using histogram equalization to enhance the contrast, and using image registration technology to ensure that the optical axes of images taken at different angles are as parallel as possible and the epipolar lines are aligned;

[0124] The Sobel operator is used to calculate the reflected image. and The gradient in the direction is used to obtain the pixel point in the reflected image The gradient magnitude and direction , identify the edges in the reflected image, the expression is as follows:

[0125]

[0126]

[0127] In the formula, is the coordinate of the pixel in the reflected image, indicating the position of the pixel in the image. and The images are and Directional gradient;

[0128] The Canny edge detection algorithm is used to generate an edge image of the reflection image, including non-maximum suppression, dual threshold detection and hysteresis edge tracking; non-maximum suppression refines the edge by comparing the gradient amplitude of the current pixel with the gradient amplitude of its adjacent pixels (in the gradient direction) and retaining the larger value; dual threshold detection determines strong edges, weak edges and non-edge areas by setting high thresholds and low thresholds, where pixels above the high threshold are marked as strong edges, pixels below the low threshold are marked as non-edges, and pixels between the high threshold and the low threshold are marked as weak edges; hysteresis edge tracking starts from the strong edge pixels and tracks the weak edge pixels along the gradient direction until another strong edge pixel is encountered or a certain distance is exceeded, and the weak edge pixels are connected to form a complete edge;

[0129] Calculate the pixel points in the reflected image The autocorrelation matrix in the local area , and define a response function based on the eigenvalues ​​of the autocorrelation matrix , used to determine whether the point is a corner point, the expression is as follows:

[0130]

[0131]

[0132] In the formula, is the coordinate of the pixel in the reflected image, indicating the position of the pixel in the image. is the autocorrelation matrix, which is a 2×2 matrix. is a Gaussian window function, which is used to define the local area considered when calculating the autocorrelation matrix. It means that each pixel in the local area of ​​the window function is summed. Used to describe the gradient structure of a local area in an image. is the determinant of the autocorrelation matrix, is the trace of the autocorrelation matrix, is an empirical constant with a value between 0.04 and 0.06. ;

[0133] Set the response threshold to , determine the pixel Is it a corner point? , pixel is a corner point; if , pixel Not a corner point;

[0134] Based on the Canny edge detection algorithm and the Harris corner detection algorithm, a set of feature points is generated, including edge points and corner points; and the quantity and quality of the extracted geometric features are evaluated; when the number of features is insufficient or the features are not obvious, such as the number of feature points is lower than the preset threshold, or the distribution of feature points is uneven, a reminder mechanism is triggered to require repositioning or adding markers; when the number of features is sufficient and the features are obvious, a three-dimensional surface model is constructed.

[0135] Specifically, the specific steps of feature point set matching include:

[0136] Get three reflection images from different angles taken by the 3D scanner at the same time , and a set of feature points in the reflected image;

[0137] Calculate the similarity of each feature point in any two reflected images. The expression is as follows:

[0138]

[0139] In the formula, , is the descriptor vector of the feature points in any two reflected images, is the dimension of the descriptor vector, , Represents vectors and The The smaller the similarity measure, the more similar the two feature points are.

[0140] Set a similarity measurement threshold to filter out matching point pairs that meet the conditions; if the similarity of two feature points is less than the similarity measurement threshold, the two feature points are defined as a matching point pair; if the similarity of two feature points is greater than the similarity measurement threshold, the two feature points are not a matching point pair;

[0141] Get the matching point pairs that meet the conditions and generate a matching point pair set.

[0142] like Figure 4 As shown, specifically, the specific steps of constructing a three-dimensional surface model include:

[0143] For each matching point pair, calculate the horizontal distance difference between the matching point pair in the first image and the third image. , and extract the coordinates of the matching point pairs in the second image, recorded as the two-dimensional coordinates of the corresponding matching point pairs; the expression of the horizontal distance difference is as follows:

[0144]

[0145] In the formula, is the pixel in the first image The coordinates in the direction, is the pixel in the third image The coordinates in the direction, the reflected images obtained by the top, middle and bottom cameras are defined as the first image, the second image and the third image respectively;

[0146] Obtain the calibration parameters of the 3D scanner, including internal parameters (such as focal length, principal point) and external parameters (such as rotation matrix, translation vector);

[0147] The depth information of each feature point in the second image is calculated using the triangulation principle , combining the depth information with the two-dimensional coordinates of the corresponding matching point pair to obtain the three-dimensional coordinates of each feature point. The expression is as follows:

[0148]

[0149] In the formula, is the focal length of the camera in the 3D scanner, is the baseline distance, which means the distance between the top and bottom cameras;

[0150] Each feature point and its three-dimensional coordinates on the second image plane are used as points in the point cloud to generate a three-dimensional point cloud dataset, which is a set of discrete points in three-dimensional space, and each point contains three-dimensional coordinate information; and the three-dimensional point cloud dataset is preprocessed, including removing noise points, filling missing areas, and smoothing point clouds, which improves the accuracy and efficiency of subsequent triangulation;

[0151] Select a point from the point cloud data set as the starting point, find the two points closest to the starting point, and form an initial triangle;

[0152] Starting from the initial triangle, gradually add the remaining points in the point cloud dataset to the triangular mesh and update the triangular mesh. At this time, the points added to the triangular mesh are recorded as new points of the triangle. For each new point, find the triangle where each new point is located and insert the new triangle into the mesh, and ensure that the newly formed triangle satisfies the Delaunay property. If the new triangle does not satisfy the Delaunay property, make adjustments (such as exchanging edges, splitting triangles) to ensure that the property is satisfied.

[0153] Set an area threshold to determine whether to merge triangles; if the area of ​​a triangle is less than the area threshold, define the triangle as a small triangle and perform a merge operation; if the area of ​​a triangle is greater than or equal to the area threshold, do not perform a merge operation; the merge operation is to merge adjacent small triangles, merge the vertices of adjacent small triangles into a new vertex, and delete the original triangle. The new vertex is the average value of the vertices of adjacent small triangles;

[0154] The Laplace smoothing method is used to adjust the position of the vertices to reduce the roughness of the mesh, making the mesh smoother and more natural, and generating a smooth, continuous and accurate three-dimensional surface model. The position update of the vertex is based on the weighted average of its adjacent vertices. The expression is as follows:

[0155]

[0156] In the formula, is the updated vertex position, is the position of the adjacent vertices, is the number of adjacent vertices.

[0157] like Figure 5 As shown, the specific steps of texture information mapping include:

[0158] When acquiring the surface deformation image of the target object, a color camera is used to synchronously acquire the texture image of the target object, and the texture image is preprocessed, including denoising, contrast enhancement, and color correction, to improve the texture quality and visual effect;

[0159] Use the Canny edge detection algorithm and Harris corner detection algorithm to generate a set of feature points of the texture image;

[0160] Obtaining a set of feature points of the second image, and calculating the similarity of all feature points in the second image and the texture image;

[0161] Set a texture similarity threshold to filter out texture matching point pairs that meet the conditions; if the similarity of two feature points is less than the texture similarity threshold, the two feature points are defined as a texture matching point pair; if the similarity of two feature points is greater than the texture similarity threshold, the two feature points are not a matching point pair;

[0162] Get matching point pairs that meet the conditions and generate a texture matching set ;in, For the second image pixels, For the texture image pixels;

[0163] Solving the affine transformation matrix by least squares method , so that the pixels on the intermediate image correspond to the pixels on the texture image, satisfying , and based on the 3D point cloud dataset, calculate the texture coordinate set ; Among them, for any point on the three-dimensional surface model, the corresponding texture coordinates can be calculated through affine transformation, , , , , , , are the parameters of the affine transformation, is the three-dimensional coordinate set in the three-dimensional point cloud dataset;

[0164] Use bilinear interpolation to calculate the texture coordinates of other points in the 3D surface model; determine the four known points , , , , these four known points are vertices on the three-dimensional surface model and already have texture coordinates; get an interpolation point to be calculated ,point Located by , , , Inside or on the surface of a tetrahedron formed by four points, according to the interpolation point Calculate the weight coefficient with the four known points, and use the weight coefficient and the texture coordinates of the four known points to estimate the interpolation point by weighted averaging. The texture coordinates are as follows:

[0165]

[0166]

[0167]

[0168]

[0169]

[0170] In the formula, , , , Relative to point , , , The weight coefficient of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of Interpolation point The three-dimensional coordinates of Interpolation point The texture coordinates of

[0171] The calculated texture coordinates are combined with the three-dimensional coordinates, and the texture coordinates of each pixel point in the texture image are used to obtain the pixel color value of the corresponding position in the texture image. The pixel color value is applied to the corresponding point of the three-dimensional surface model to achieve the mapping of the texture image to the three-dimensional surface model and generate a three-dimensional surface model with texture information.

[0172] Specifically, when a color camera acquires a texture image, it will inevitably introduce some elements unrelated to the texture, such as debris that accidentally enters the picture during shooting, and non-texture areas caused by light and shadow changes, thereby interfering with the recognition and analysis of the true texture. In addition, when reconstructing a three-dimensional surface model based on the texture image, if a large number of non-texture pixels are mistakenly calculated as texture points, the surface texture of the reconstructed three-dimensional model will be wrong or distorted, and the true appearance of the object cannot be accurately restored. At this time, the texture image is preprocessed to determine whether each pixel in the texture image is a texture pixel.

[0173] The specific steps for preprocessing texture images include:

[0174] Preprocessing the acquired texture image includes: using Gaussian filtering to remove noise in the image, using histogram equalization to enhance contrast, and white balance to correct color, so as to improve the image quality and visual effect, and facilitate subsequent processing;

[0175] For the preprocessed texture image, traverse the pixels in the texture image row by row and column by column, and record the coordinates of each pixel and color value RGB;

[0176] For the pixel to be judged in the texture image, a neighborhood window is defined with the pixel to be judged as the center, and the Euclidean distance between the pixel to be judged and other pixels in the neighborhood is calculated to measure the color similarity between the pixel to be judged and other pixels in the neighborhood. A distance threshold is set, and other pixels whose Euclidean distance is less than the distance threshold are defined as similar pixels. The number of similar pixels in the neighborhood is counted. , calculate the number of similar pixels to the total number of pixels in the neighborhood , expressed as the similarity ratio , the similarity ratio reflects the color consistency of the pixel to be judged in the neighborhood. The higher the similarity ratio, the closer the color of the pixel is to the majority of pixels in the neighborhood. The expression is as follows:

[0177]

[0178] Convert the pixels in the neighborhood into grayscale values, use the Sobel operator to calculate the horizontal and vertical gradients of the pixel to be judged, and calculate the gradient amplitude ,The gradient amplitude reflects the local change rate of the texture image at the pixel to be judged, and the texture area usually has a larger gradient amplitude;

[0179] Set the similarity threshold to , gradient amplitude threshold , determine whether the pixel to be determined is a texture pixel; if and , indicating that the pixel to be judged is not only similar in color to most pixels in the neighborhood, but also has a high spatial correlation, and there is an obvious local change in the neighborhood, which meets the characteristics of the texture area, then the pixel to be judged is determined to be a texture pixel; otherwise, it indicates that the pixel to be judged lacks consistency in color, or the pixel changes in the neighborhood are not obvious, and the pixel to be detected is determined to be a non-texture pixel;

[0180] Traverse the pixel points in the texture image, and according to the judgment result, store the information (coordinates and color values) of the texture pixel points in the constructed texture set, and store the coordinate information of the non-texture pixel points in the constructed non-texture set, so as to perform correction processing later;

[0181] Since the 3D scanner is constantly moving during the scanning process and is affected by the shutter speed of the color camera, the texture image at the next moment contains some pixel information of the texture image at the previous moment. Using this feature, the non-texture pixel points are corrected; for each non-texture pixel point, the texture image set taken by the 3D scanner at adjacent moments from other angles is traversed in turn;

[0182] According to the method of feature point set matching, in each texture image at other angles, a correction pixel point matching the current non-texture pixel point in the original image is extracted, and it is determined whether the extracted correction pixel point is a texture pixel point;

[0183] If a corrected pixel in an image at another angle is found to be a texture pixel, the color value of the corrected pixel is used as the corrected color value of the current non-texture pixel to replace the color value of the non-texture pixel in the original image, and the non-texture pixel is changed to a texture pixel and saved in the texture set until the texture images at other angles in the texture image set are traversed;

[0184] If no texture pixel that can be used for correction is found after traversing texture images of other angles in the texture image set, all texture pixels in the neighborhood of the non-texture pixel are screened out, and the average color value of these valid texture pixels is calculated. The calculated average value is used as the corrected color value to replace the color value of the non-texture pixel in the original image, and the non-texture pixel is changed to a texture pixel and saved in the texture set;

[0185] Convert a texture set containing accurate texture pixel information into a texture image, that is, reassemble each texture pixel into a complete image according to its coordinate position and color value.

[0186] The storage module is used to store the data generated during the scanning process, including original images, 3D point clouds and 3D models, and to perform structured storage of each complete scanning result to facilitate subsequent screening and searching for specific information. An efficient data index is established based on the stored data to achieve data traceability, facilitate quick query and retrieval, and ensure long-term stable operation of the system.

[0187] In summary, the present invention automatically optimizes light source information by capturing reflected light patterns to adapt to the characteristics of surfaces of different materials; after the optimization is completed, the target object is scanned from different angles, the geometric information of the surface of the target object in the image is identified, a feature point set is generated, the feature points in the multi-angle feature point set are matched, a matching point pair set is generated, and three-dimensional reconstruction technology is used to gradually construct a three-dimensional surface model of the target object according to the spatial position relationship of the feature points; and the texture image of the target object is simultaneously captured to map the texture information to the constructed three-dimensional surface model.

[0188] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A handheld 3D scanning system based on visible light, characterized in that: include: Calibration module, adaptive adjustment module, three-dimensional scanning module and storage module; The calibration module is used to calibrate the three-dimensional scanner using a known reference object; The adaptive adjustment module is used to automatically optimize scanning parameters according to the material and surface characteristics of the target object; The three-dimensional scanning module acquires the reflected images of the target object surface from multiple angles in real time, identifies the geometric information of the target object surface in the image, and constructs a three-dimensional surface model of the target object; The storage module is used to store the data generated during the scanning process and to establish a data index based on the stored data; The three-dimensional scanning module includes a feature extraction unit, a model building unit and a texture positioning unit; The feature extraction unit is used to obtain multi-angle reflection images of the surface of the target object, identify geometric information of the surface of the target object in the image, and generate a set of feature points; The model building unit is used to match the feature point set in the multi-angle image information of the target object to build a three-dimensional surface model; The texture positioning unit is used to obtain a texture image of the target object, extract texture information in the texture image, and map the texture information onto the three-dimensional surface model; The specific steps of texture information mapping include: Synchronously acquire the texture image of the target object and pre-process the texture image; Use the Canny edge detection algorithm and Harris corner detection algorithm to generate a set of feature points of the texture image; Acquire a set of feature points of the second image, and calculate the similarity between the texture image and each feature point in the second image; Set a texture similarity threshold, filter out texture matching point pairs that meet the conditions, and generate a texture matching set ;in, For the second image pixels, For the texture image pixels; Calculate the affine transformation matrix using the least squares method , so that , and based on the 3D point cloud dataset, calculate the texture coordinate set ; Calculate any interpolation point in a 3D surface model using bilinear interpolation Texture coordinates ; Among them, for any point on the three-dimensional surface model, the corresponding texture coordinates can be calculated through affine transformation. , , , , , , are the parameters of the affine transformation, is a three-dimensional coordinate set in a three-dimensional point cloud dataset; Use bilinear interpolation to calculate the texture coordinates of other points in the 3D surface model; determine the four known points , , , , these four known points are vertices on the three-dimensional surface model and already have texture coordinates; get an interpolation point to be calculated ,point Located by , , , Inside or on the surface of a tetrahedron formed by four points, according to the interpolation point Calculate the weight coefficient with the four known points, and use the weight coefficient and the texture coordinates of the four known points to estimate the interpolation point by weighted averaging. The texture coordinates are as follows: ； ； ； ； ； In the formula, , , , Relative to point , , , The weight coefficient of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of For point The three-dimensional coordinates of For point The texture coordinates of Interpolation point The three-dimensional coordinates of Interpolation point The texture coordinates of Using the texture coordinates of each pixel point in the texture image, extracting the pixel color value in the texture image, and applying the pixel color value to the corresponding point of the three-dimensional surface model to generate a three-dimensional surface model with texture information; The specific steps for preprocessing texture images include: Traverse the pixel points in the texture image and record the coordinates of the pixel points and color value RGB; Define a neighborhood window with the pixel to be judged in the texture image as the center, obtain similar pixels in the neighborhood, and calculate the similarity ratio of the pixel to be judged ; Convert the pixels in the neighborhood into grayscale values ​​and calculate the gradient amplitude of the pixel to be determined ; Set the similarity threshold to , gradient amplitude threshold , determine whether the pixel to be determined is a texture pixel; if and , the pixel to be determined is a texture pixel; otherwise, the pixel to be determined is a non-texture pixel; Traversing the pixel points in the texture image to construct a texture set and a non-texture set; Acquire a texture image set taken by a 3D scanner from other angles, extract a correction pixel point that matches the non-texture pixel point, and determine whether the correction pixel point is a texture pixel point; If the correction pixel point is a texture pixel point, the color value is corrected by using the correction pixel point, and the texture set is updated until the texture image set is traversed; Otherwise, the average value of the texture pixels in the neighborhood is calculated, the color value is corrected, and the texture set is updated; Converts a collection of textures containing texture pixel information into a texture image.

2. The handheld three-dimensional scanning system based on visible light according to claim 1, characterized in that: The specific steps for automatically optimizing scanning parameters include: Obtain a standard reflection map in the system and calculate the standard brightness mean of the standard reflection map and standard contrast ; Initializing parameters of the three-dimensional scanner; Project a light pattern onto the target object, obtain a reflected light pattern, and calculate the brightness average of the reflected light pattern and contrast ; Automatically adjust the brightness and contrast of the white light projector. The expression is as follows: ； ； In the formula, is the brightness of the white light projector at the current moment, is the brightness of the white light projector after adjustment, is the brightness adjustment coefficient, is the contrast of the white light projector after adjustment, is the contrast of the white light projector at the current moment, is the contrast adjustment factor.

3. The handheld three-dimensional scanning system based on visible light according to claim 2, characterized in that: The specific steps of automatically optimizing scanning parameters also include: Re-project the light pattern onto the target object and calculate the brightness average of the adjusted reflected light pattern and contrast ; Set the brightness difference threshold to , the contrast difference threshold is , judging whether the quality of the adjusted reflected light pattern is qualified; like and , the pattern quality is qualified and the automatic adjustment ends; like or , the pattern quality is unqualified, continue to adjust the brightness and contrast of the white light projector.

4. The handheld three-dimensional scanning system based on visible light according to claim 3, characterized in that: The specific steps of geometric information recognition include: Acquire a reflection image of the target object, and preprocess the reflection image; Calculate the pixel points in the reflected image The gradient magnitude and direction , and use the Canny edge detection algorithm to generate the edge image of the reflected image; the expression is as follows: ； ； In the formula, and The images are and Directional gradient; Calculate the pixel points in the reflected image The autocorrelation matrix in the local area , and generate a response function ;in, is an empirical constant; Set the response threshold to , determine the pixel Is it a corner point? , pixel is a corner point; if , pixel Not a corner point; Based on the Canny edge detection algorithm and the Harris corner detection algorithm, a set of feature points is generated, and the quantity and quality of the extracted geometric features are evaluated.

5. The handheld three-dimensional scanning system based on visible light according to claim 4, characterized in that: The specific steps of feature point set matching include: Acquire multi-angle reflection images of the target object and a set of feature points of each reflection image; Calculate the similarity of each feature point in any two reflected images. The expression for similarity calculation is as follows: ； In the formula, , is the descriptor vector of the feature points in any two reflected images, is the dimension of the descriptor vector, , Represents vectors and The Quantity; Set a similarity measurement threshold, filter out matching point pairs that meet the conditions, and generate a matching point pair set.

6. The handheld three-dimensional scanning system based on visible light according to claim 5, characterized in that: The specific steps of constructing a 3D surface model include: For each matching point pair, calculate the horizontal distance difference between the matching point pair in the first image and the third image. , and extract the two-dimensional coordinates of the corresponding matching point pairs in the second image. The expression of the horizontal distance difference is as follows: ； In the formula, is the pixel in the first image The coordinates in the direction, is the pixel in the third image The coordinates in the direction, the reflected images obtained by the top, middle and bottom cameras are defined as the first image, the second image and the third image respectively; Acquiring calibration parameters of the three-dimensional scanner, including internal parameters and external parameters; Calculate the depth information of each feature point in the second image , and combine the depth information with the corresponding two-dimensional coordinates to obtain the three-dimensional coordinates of each feature point; the expression of the depth information is as follows: ； In the formula, is the focal length of the camera in the three-dimensional scanner, is the baseline distance, which means the distance between the top and bottom cameras; Each feature point and its three-dimensional coordinates on the second image plane are used as points in the point cloud to generate a three-dimensional point cloud data set, and the three-dimensional point cloud data set is preprocessed.

7. The handheld three-dimensional scanning system based on visible light according to claim 6, characterized in that: The specific steps of constructing a three-dimensional surface model also include: Select a point from the point cloud data set as a starting point, find two points closest to the starting point, and form an initial triangle; Taking the initial triangle as a starting point, gradually adding the remaining points in the point cloud data set to the triangular mesh, and updating the triangular mesh; Set an area threshold to determine whether triangles should be merged; If the area of ​​a triangle is smaller than the area threshold, the triangle is defined as a small triangle and a merge operation is performed; If the area of ​​the triangle is greater than or equal to the area threshold, no merging operation is performed; The Laplace smoothing method is used to adjust the position of the vertices and generate a three-dimensional surface model of the target object. The expression is as follows: ； In the formula, is the updated vertex position, is the position of the adjacent vertices, is the number of adjacent vertices.

Images