Three-dimensional model construction method and system for image registration matching

Abstract

The invention relates to the technical field of three-dimensional vision, in particular to a three-dimensional model construction method and system for image registration matching, which can utilize a depth camera to obtain image data and depth data of a target at a high visual angle and a low visual angle, realize space-time alignment and generate point cloud data; performing feature extraction on the grayscale image data, and generating descriptors according to gradient features of areas around the key points; carrying out feature matching on the key points to obtain a plurality of matching point pairs, and carrying out mixed superposition on the color images to generate a superposed image; and calculating the normal vector and curvature of each point cloud surface, converting discrete point cloud data into a continuous three-dimensional surface, and constructing a three-dimensional polygonal mesh model. According to the technical scheme, under the condition that hardware parameters of the depth camera are not changed, information of multiple visual angles is fused, the data fusion problem is effectively solved, the model reconstruction precision and details are improved, the development threshold is lowered, and efficient application of the three-dimensional visual technology in multiple fields is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional vision, in particular to a three-dimensional model construction method and system based on image registration matching. BACKGROUND

[0002] With the rapid development of robot navigation and positioning, industrial automation detection, cultural relic digitization protection and other fields, the demand for high-precision three-dimensional vision technology is increasingly urgent. In the robot navigation scene, it is necessary to quickly construct an environmental three-dimensional model to realize accurate positioning and path planning under the condition of limited hardware; industrial automation detection requires high-precision three-dimensional modeling of products to detect surface defects without changing the depth camera device as much as possible; cultural relic digitization protection needs to restore the appearance of cultural relics completely and finely based on existing hardware.

[0003] It can be seen that the existing three-dimensional vision processing method faces many challenges: there are errors in the space-time synchronization of data collected by different devices, and traditional algorithms are difficult to balance model reconstruction efficiency, accuracy and details under fixed hardware parameters. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a three-dimensional model construction method and system based on image registration matching to solve the problem that the existing technology has errors in the space-time synchronization of data collected by different devices, and traditional algorithms are difficult to balance model reconstruction efficiency, accuracy and details under fixed hardware parameters.

[0005] According to a first aspect of an embodiment of the present application, a three-dimensional model construction method based on image registration matching is provided, comprising:

[0006] Color image data and depth data at each view angle of the target high view angle and low view angle are obtained by using a depth camera, a double-thread asynchronous acquisition mechanism is adopted, and the color image data and depth data are matched by time stamp to realize space-time alignment, thereby generating point cloud data at each view angle;

[0007] The color image data at each view angle is converted into gray image data, and the point cloud data at each view angle is converted from the camera coordinate system to the target coordinate system;

[0008] Feature extraction is performed on the gray image data at each view angle to obtain a plurality of key points of the gray image data, a descriptor is generated according to the gradient features of the surrounding area of the key points, feature matching is performed on the key points of the gray image data of the high view angle and the low view angle according to the similarity of the descriptors, a plurality of matching point pairs are obtained, and the color image data of the high view angle and the low view angle is mixed and superimposed according to all matching pairs to generate a superimposed image;

[0009] According to the point cloud data under the target coordinate system, the normal vector and the curvature of each point cloud surface are calculated; and the discrete point cloud data is converted into continuous three-dimensional surface by using a Poisson reconstruction algorithm and a spherical rotation algorithm according to the normal vector and the curvature of each point cloud surface, so as to construct a three-dimensional polygon mesh model.

[0010] Preferably, the method further comprises:

[0011] The multi-frame median fusion algorithm is used for denoising the collected depth data.

[0012] The voxel downsampling and statistical outlier filtering algorithm is used for redundant compression and outlier elimination of the point cloud data under each view angle, so as to obtain normalized point cloud data.

[0013] Preferably, after the point cloud data under each view angle is converted from the camera coordinate system to the target coordinate system, the method further comprises:

[0014] The point cloud data under the target coordinate system is filtered and downsampled.

[0015] The filtering process adopts a statistical filtering process or a radius filtering process; and the downsampling process adopts a voxel downsampling process or a random downsampling process.

[0016] Preferably, the method further comprises:

[0017] The fast approximate nearest neighbor matcher is used to preliminarily match the key points based on the K nearest neighbor algorithm, so as to obtain the preliminary matching pairs.

[0018] The ratio test threshold is set to perform a first screening on the preliminary matching pairs.

[0019] The angle consistency constraint and random sampling consensus algorithm are used to perform geometric verification and mismatch elimination on the matching pairs after the first screening, so as to obtain the final multiple matching pairs.

[0020] Preferably, the method further comprises:

[0021] The homography matrix describing the perspective transformation relationship of the color images under the high and low view angles is calculated according to the matching pairs.

[0022] The color images under the high view angle are subjected to perspective transformation according to the homography matrix, and the color images under the high and low view angles are superimposed and displayed by using the image fusion algorithm, so as to generate the superimposed image.

[0023] Preferably, the method further comprises:

[0024] According to the superimposed image, the high-view-angle color image and the low-view-angle color image, a mean square error evaluation index, a normalized cross-correlation evaluation index and a structural similarity evaluation index are calculated;

[0025] The registration precision of the superimposed image is quantitatively analyzed according to the mean square error evaluation index, the normalized cross-correlation evaluation index and the structural similarity evaluation index, and an evaluation analysis result is generated.

[0026] Preferably, a normal vector of each point cloud surface is calculated, including:

[0027] A plurality of eigenvalues and eigenvectors of each point cloud surface are solved through a covariance matrix of neighborhood points.

[0028] The eigenvector corresponding to the minimum eigenvalue is taken as the normal vector.

[0029] Preferably, after the plurality of eigenvalues and eigenvectors are obtained, the method further includes:

[0030] The weight of the eigenvalue is set according to the distance of the neighborhood points, and the final eigenvector is obtained as the normal vector according to the eigenvalue and the weight.

[0031] The region with a curvature greater than a preset value is optimized through weighted average of neighborhood normal vectors, so that the normal vector is smoother.

[0032] According to a second aspect of the embodiment of the present application, a three-dimensional model construction system for image registration and matching is provided, including:

[0033] A data acquisition module is configured to acquire color image data and depth data of each view of a target high view angle and low view angle respectively by using a depth camera, and to achieve time-space alignment of the image data and the depth data through timestamp matching by using a double-thread asynchronous acquisition mechanism, so as to generate point cloud data of each view.

[0034] A preliminary processing module is configured to convert the color image data of each view into gray image data, and to convert the point cloud data of each view from a camera coordinate system to a target coordinate system.

[0035] An image processing module is configured to perform feature extraction on the gray image data of each view, to obtain a plurality of key points of the gray image data, to generate a descriptor according to gradient features of a region around the key points, to perform feature matching on the key points of the gray image data of the high view angle and the low view angle according to the similarity of the descriptors, to obtain a plurality of matching point pairs, and to perform hybrid superimposition on the color image data of the high view angle and the low view angle according to all the matching pairs, so as to generate a superimposed image.

[0036] The point cloud processing module is configured to calculate a normal vector and a curvature of each point cloud surface according to point cloud data in a target coordinate system; and convert the discrete point cloud data into a continuous three-dimensional surface by using a Poisson reconstruction algorithm and a spherical rotation algorithm according to the normal vector and the curvature of each point cloud surface, so as to construct a three-dimensional polygon mesh model.

[0037] Preferably, the three-dimensional model construction system further comprises:

[0038] The visualization module is configured to provide an interactive three-dimensional visualization interface to realize rendering and dynamic display of image data, point cloud data and the three-dimensional polygon mesh model in a whole life cycle.

[0039] The performance statistics module is configured to record a calculation time consumption of each processing link; and evaluate a processing efficiency of the system according to the calculation time consumption data of the each processing link.

[0040] The technical scheme provided by the embodiment of the application can have the following beneficial effects:

[0041] It can be understood that the technical scheme provided by the application can obtain image data and depth data of a target high-viewing-angle and low-viewing-angle by using a depth camera, realize space-time alignment, generate point cloud data, extract features from grayscale image data, generate descriptors according to gradient features of a region around a key point, further perform feature matching of the key point, obtain a plurality of matching point pairs, perform mixed superposition on a color image, generate a superimposed image, calculate a normal vector and a curvature of each point cloud surface, convert the discrete point cloud data into a continuous three-dimensional surface, and construct a three-dimensional polygon mesh model. The technical scheme shown in the application effectively solves a data fusion problem by fusing information of a plurality of viewing angles under the condition that hardware parameters of the depth camera are unchanged, improves model reconstruction accuracy and details, reduces a development threshold, and realizes efficient application of three-dimensional vision technology in multiple fields.

[0042] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the application and, together with the description, serve to explain the principles of the application.

[0044] Figure 1 is a three-dimensional model construction method step schematic diagram of image registration matching according to an exemplary embodiment;

[0045] Figure 2 is an image registration flowchart according to an exemplary embodiment;

[0046] Figure 3is a surface reconstruction flow chart according to an exemplary embodiment. DETAILED DESCRIPTION

[0047] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] In an embodiment, Figure 1 is a three-dimensional model construction method step schematic diagram of image registration matching according to an exemplary embodiment, see Figure 1 , provide a three-dimensional model construction method of image registration matching, comprising:

[0049] Step S11, using a depth camera to obtain color image data and depth data of each view of the target high view angle and low view angle respectively, using a double-thread asynchronous acquisition mechanism, the image data and depth data are matched by time stamp to realize space-time alignment, and point cloud data under each view is generated.

[0050] In specific practice, human back three-dimensional modeling is taken as an application scenario, and a multi-view data acquisition strategy is implemented.

[0051] In the low view angle data acquisition stage, the parameter initialization of the depth camera is first completed, including accurately setting the core parameters such as image resolution and acquisition frame rate. After starting the data acquisition link, the strict consistency of the depth frame and the color frame in the space-time dimension is ensured by means of the hardware synchronization mechanism and the time stamp alignment algorithm. Preferably, the user can trigger the acquisition operation through the real-time visual interactive interface.

[0052] The high view angle data acquisition adopts the same data processing flow as the low view angle, and only by adjusting the physical installation angle of the camera, the top structure data of the target object can be obtained. Therefore, the technical solution shown in the embodiment only needs one depth camera. Since the data collected at different view angles have spatial position and attitude deviation, accurate registration and alignment need to be performed in the subsequent processing link, so as to construct complete three-dimensional model data. Through the combination of multi-frame data fusion and adaptive downsampling technology in this stage, not only the original data quality is significantly improved, but also the data redundancy is effectively reduced and the noise interference is reduced.

[0053] Preferably, the step S11 shown in the embodiment further comprises:

[0054] The multi-frame median fusion algorithm is used for denoising the collected depth data; the voxel downsampling and statistical outlier filtering algorithm is used for redundant compression and abnormal value elimination of the point cloud data under each view angle, so as to obtain the normalized point cloud data.

[0055] The technical scheme shown in the embodiment synchronously uses the multi-frame median fusion algorithm for denoising the original data, which can effectively suppress the inherent noise and random error of the sensor. Then, the voxel downsampling and statistical outlier filtering algorithm is used for redundant compression and abnormal value elimination of the original point cloud, so as to finally generate the normalized point cloud data with color information and complete the persistent storage.

[0056] In step S12, the color image data under each view angle is converted into gray image data; and the point cloud data under each view angle is converted from the camera coordinate system to the target coordinate system.

[0057] This step performs the preprocessing of the image and the point cloud.

[0058] Firstly, the image is preprocessed. In the image preprocessing link, for the color images collected at low and high view angles, firstly, the size normalization processing is performed, the image scaling or cropping algorithm is used to unify the image size and resolution, so as to provide standardized input for the subsequent feature extraction operation. At the same time, the color image is converted into a gray image, the color information interference is eliminated, and the stability and extractability of the image features are enhanced.

[0059] In the embodiment, there are two ways for the size normalization processing. By default, the "scaling" strategy is adopted, a fixed image is set, the color image is aligned to the fixed image, the minimum value of the width and height ratio is calculated to calculate the scaling ratio, and the image is scaled according to the ratio to ensure that its size does not exceed the fixed image.

[0060] The "cropping" strategy is to crop the two images to the smallest common size.

[0061] The preprocessing of the point cloud data can convert the point cloud data from the camera coordinate system to the target coordinate system based on the camera calibration parameters and the rigid body transformation theory, using the homogeneous transformation matrix and the homography matrix, so as to realize the unified spatial expression of the multi-view data.

[0062] It should be noted that it also includes: performing filtering processing and downsampling processing on the point cloud data in the target coordinate system; the filtering processing adopts statistical filtering processing or radius filtering processing; and the downsampling processing adopts voxel downsampling processing or random downsampling processing.

[0063] Filtering refers to removing noise points in the point cloud; statistical filtering refers to removing points that deviate too much based on the mean and standard deviation of the neighborhood points. Radius filtering refers to removing isolated points with insufficient number of points in the neighborhood. Downsampling refers to reducing the number of points to improve the efficiency of subsequent processing; voxel downsampling refers to dividing the point cloud into a voxel grid, retaining one representative point for each grid, and retaining the structural features; random downsampling refers to randomly selecting points, which is simple and fast but may lose details.

[0064] The technical solution shown in this embodiment, on the basis of converting the point cloud data from the camera coordinate system to the target coordinate system, further optimizes the quality of the point cloud data, reduces the data dimension, improves the calculation efficiency and processing performance of the subsequent three-dimensional reconstruction algorithm, and effectively solves the problem of inconsistent coordinate systems and uneven data quality of multiple sources.

[0065] Step S13, feature extraction is performed on the gray image data under each view angle to obtain a plurality of key points of the gray image data, a descriptor is generated according to the gradient features of the surrounding area of the key points; according to the similarity of the descriptors, the key points of the gray image data of the high view angle and the low view angle are matched in feature, a plurality of matching point pairs are obtained; according to all the matching pairs, the color image data of the high view angle and the low view angle are superimposed to generate a superimposed image.

[0066] This step is based on image preprocessing, and feature extraction and feature matching are performed on the image, and then a superimposed image is generated by image superposition. The step flow chart is shown in Figure 2 .

[0067] In the image feature extraction stage, the purpose is to extract stable local features (key points) from the gray image and generate descriptors. The feature selection defaults to the SIFT algorithm. The SIFT / SURF algorithm detects key points (including position, scale, direction, etc.), calculates the gradient features of the surrounding area of the key points, and generates high-dimensional descriptors.

[0068] Key points are pixel points in an image that have local uniqueness and stability and can be reliably identified when the image is translated, rotated, scaled, or light changed. Simply put, they are the "most representative" points in the image, such as:

[0069] 1. Corner: a point in an image with a sharp change in brightness (such as the corners of an object, the intersection of edges), for example, the intersection of a checkerboard, the corner of a building.

[0070] 2. Blob: a local area in an image that has a brightness difference from the surrounding area (such as dense patterns in textures, the center of small objects).

[0071] Key point core properties: 1. Stability: can be repeatedly detected when the image is zoomed, rotated, or light changes (for example, a corner point will not become a normal point due to image enlargement). 2. Uniqueness: the local structure of the point is significantly different from the surrounding points, which facilitates the differentiation from the corresponding points in other images. 3. Sparsity: it is not spread throughout the image, but concentrated in the "information-rich" area (avoiding redundant calculation).

[0072] Descriptors are vectors (a string of numbers) that quantify the features of the local area around the key point, used to describe the "unique identity" of the key point. Its role is: when the descriptors of two key points are similar enough, they can be considered as the corresponding points of the same physical structure in different images.

[0073] Core properties of descriptors: 1. Discrimination: the descriptors of different key points are significantly different, and the descriptors of the same / similar key points are highly similar. 2. Invariance: robust to changes in light, rotation, and scale (for example, the same corner point in images with different brightness should have basically the same descriptor). 3. Compactness: represented by a fixed-length vector (such as SIFT is a 128-dimensional vector), which facilitates efficient matching calculation.

[0074] The technical solution of the embodiment adopts the SIFT or SURF feature extraction algorithm under the CPU architecture, realizes image feature extraction through the classical feature detection and description process, completes the feature extraction processing of a single frame image, and meets the image registration requirement.

[0075] After feature extraction, feature matching can be performed. Based on the similarity of the descriptors of the features, the corresponding key points in two images are found.

[0076] It should be noted that the feature matching stage includes:

[0077] A fast approximate nearest neighbor matcher (FLANN) is used to perform preliminary matching of key points based on the K-nearest neighbor algorithm, and a preliminary matching pair is obtained; a ratio test threshold is set to perform a first screening of the preliminary matching pair; an angle consistency constraint and a random sample consensus algorithm are used to perform geometric verification and false match elimination on the matching pair after the first screening, and a plurality of final matching point pairs are obtained.

[0078] The matching strategy of the feature matching stage defaults to FLANN fast matching, and FLANN: through KD tree indexing to accelerate high-dimensional vector matching, suitable for large-scale feature sets. In addition, optional brute force matching (BF): traversing all descriptors to find the nearest neighbor, high accuracy but slow speed.

[0079] The screening strategy is to combine KNN matching and ratio test, and for each descriptor, find 2 nearest neighbor matches (K=2). Keep the matching points with "best matching distance <0.7*second best matching distance" (eliminate false matches caused by noise).

[0080] During the matching point optimization, the matching points are further screened by using the consistency of the key point directions. The logic is that the key points of SIFT / SURF contain direction information, and the direction difference of the real matching key points should be small. In actual processing, the matching points with an angle difference less than a threshold are kept by calculating the angle difference.

[0081] It can be understood that in the feature matching stage, a fast library for approximate nearest neighbors (FLANN) matcher is used, a K nearest neighbor (KNN) algorithm is used for preliminary matching of feature points, a ratio test threshold is set, and reliable matching point pairs are screened. Further introduce angle consistency constraint and random sample consensus (RANSAC) algorithm, geometric verification and false matching elimination are performed on the matching point pairs, the feature matching false matching rate is controlled at a low level through the estimation of homography matrix and re-projection error constraint, which significantly improves the robustness and accuracy of cross-view feature matching, effectively solves the false matching problem of traditional algorithm in complex texture scene.

[0082] Finally, the obtained matching pairs are used to superimpose the images, including:

[0083] According to the matching pairs, a homography matrix describing the perspective transformation relationship of the color images with high and low viewing angles is calculated; according to the homography matrix, the color image with high viewing angle is subjected to perspective transformation, and the color images with high and low viewing angles are superimposed and displayed through an image fusion algorithm, to generate a superimposed image.

[0084] The technical solution shown in this embodiment is based on the estimated homography matrix, and the high-viewing-angle image is subjected to perspective transformation, and the high and low viewing angle images are superimposed and displayed through an image fusion algorithm, to realize intuitive and visual verification of the registration effect.

[0085] It should be noted that the method further includes:

[0086] According to the superimposed image, the color image with high viewing angle and the color image with low viewing angle, a mean square error evaluation index, a normalized cross correlation evaluation index and a structural similarity evaluation index are calculated; the registration accuracy of the superimposed image is quantitatively analyzed according to the mean square error evaluation index, the normalized cross correlation evaluation index and the structural similarity evaluation index, to generate an evaluation analysis result.

[0087] It can be understood that in the quantitative evaluation aspect, multi-dimensional evaluation indexes such as mean square error (MSE), normalized cross correlation (NCC) and structural similarity (SSIM) are used to quantitatively analyze the registration result. By calculating the difference of the image at the pixel level, the accuracy of the registration result is evaluated, and a quantitative reference basis for the image registration effect is provided.

[0088] Step S14, according to the point cloud data under the target coordinate system, the normal vector and the curvature of each point cloud surface are calculated; using Poisson reconstruction algorithm and spherical rotation algorithm, the discrete point cloud data is converted into continuous three-dimensional surface according to the normal vector and the curvature of each point cloud surface, and a three-dimensional polygon mesh model is constructed.

[0089] This step is based on the point cloud data preprocessing, and a three-dimensional polygon mesh model is finally constructed according to the point cloud data under the target coordinate system. The flow chart is shown in Figure 3 .

[0090] It should be noted that the normal vector of each point cloud surface includes: solving a plurality of eigenvalues and eigenvectors of each point cloud surface through the covariance matrix of the neighborhood points; and taking the eigenvector corresponding to the minimum eigenvalue as the normal vector.

[0091] The normal vector is a key feature of surface reconstruction, reflecting the orientation of the point cloud surface. The eigenvalue and eigenvector are solved through the covariance matrix of the neighborhood points, and the vector corresponding to the minimum eigenvalue is the normal vector, which ensures the uniform direction of the Z component (avoids the confusion of the direction of the normal vector).

[0092] It should be noted that after obtaining a plurality of eigenvalues and eigenvectors, it further includes: setting the weight of the eigenvalue according to the distance of the neighborhood points, and obtaining the final eigenvector as the normal vector according to the eigenvalue and the weight; and optimizing the region with curvature greater than a preset value through weighted average of neighborhood normal vectors, so that the normal vector is smoother.

[0093] When calculating the normal vector, distance weight can be added: the closer the neighborhood points, the greater the weight, reducing the interference of distant points. Curvature optimization: the high curvature area (surface changes sharply) is optimized through weighted average of neighborhood normal vectors, so that the normal vector is smoother. When the bending degree of the surface at the point accounts for more than 30% of the overall features, it is considered that smoothing processing is needed.

[0094] In specific practice, the pre-processed point cloud data is converted into a structured data format, based on local neighborhood covariance matrix eigenvalue analysis, a curvature weighted normal vector estimation algorithm is adopted, that is, a distance weight (Gaussian kernel function) and a curvature weight are introduced to optimize the neighborhood of the normal vector in the high curvature area, to improve the accuracy of the normal vector at the edge and detail, and then to accurately calculate the point cloud surface normal vector, to realize the surface detail optimization. In the case of failure of the innovative method to calculate the normal vector, the basic normal vector estimation method is automatically returned, that is, the covariance matrix is calculated for each point, the eigenvector corresponding to the minimum eigenvalue is taken as the normal vector, and the direction is unified. Ensure the robustness and reliability of the algorithm.

[0095] After calculating the normal vector and curvature of each point cloud surface, the surface reconstruction of the three-dimensional model can be carried out. Converting discrete point cloud into continuous three-dimensional surface can be realized by Poisson reconstruction and ball rotation algorithm. Poisson reconstruction refers to generating an implicit surface by solving Poisson equation, which is suitable for generating closed and smooth surfaces, and the parameter depth controls the reconstruction accuracy (the larger the value, the more detailed the details). The ball rotation algorithm refers to generating a triangular mesh by rolling a virtual ball, which is suitable for scenes with uniform point cloud density, and the ball radius is automatically estimated based on the average distance of the point cloud.

[0096] In specific practice, the curvature-aware Poisson reconstruction algorithm (generating a triangular mesh based on the normal vector and point cloud density, controlling the mesh details through the depth parameter, and filtering low-density vertices) and the ball rotation algorithm (BPA, automatically estimating the ball radius, fitting the surface by moving the ball, suitable for closed surface reconstruction) are combined to reconstruct the surface. According to the characteristics of the point cloud data, the Poisson reconstruction algorithm with fixed depth parameter is used to generate an implicit surface, and the ball rotation algorithm (BPA) is used to generate a triangular mesh by rolling a virtual ball. Through the preset threshold, low-density vertices are filtered to optimize the topological structure and geometric accuracy of the reconstructed model. Through coordinate transformation, the point cloud is processed by spatial rotation, and finally a three-dimensional polygon mesh model is generated to realize three-dimensional model reconstruction. In this stage, through the basic grid simplification operation, the main features of the model are retained while reducing the number of model patches, improving the reconstruction efficiency and model basic fidelity.

[0097] In terms of hardware configuration, according to the computational complexity and data throughput requirements of three-dimensional visual processing tasks, a PC computing platform is built to meet the computing power benchmark. The core components include: a central processor with multi-core parallel processing capability, a memory and hard disk system that can adapt to large-scale point cloud data storage and processing requirements, and a graphics processor that can meet real-time rendering and accelerated computing. The depth camera selects a device with high resolution and accurate depth perception, and through the unique serial number of the device, the single camera is accurately identified and controlled, supporting three-dimensional data acquisition based on a single device; a multi-camera expansion interface is reserved, which can be expanded through a serial number list to realize multi-view data acquisition architecture.

[0098] At the software level, based on the mainstream operating system ecosystem, an object-oriented high-level programming language is used as a development tool, and a professional integrated development environment (IDE) is relied on to realize efficient code writing, debugging, and version management. Graphical interaction is based on an open-source vision library to realize lightweight display functions, and real-time image stream window display is completed through OpenCV, and point cloud three-dimensional visualization is realized relying on Open3D to meet the interaction requirements (such as view angle adjustment, parameter preview). At the same time, the point cloud processing library and the computer vision library are deeply integrated, covering core algorithm modules such as point cloud filtering, feature extraction, and image registration, and a complete three-dimensional vision processing software stack is constructed to effectively make up for the short board of traditional open-source tool chains in terms of function integration and performance optimization.

[0099] In another embodiment, a three-dimensional model construction system for image registration and matching is provided, comprising:

[0100] A data acquisition module is configured to acquire color image data and depth data of each view angle of a target high view angle and a target low view angle by using a depth camera, and to realize space-time alignment of the image data and the depth data through timestamp matching by using a double-thread asynchronous acquisition mechanism, thereby generating point cloud data of each view angle.

[0101] A preliminary processing module is configured to convert the color image data of each view angle into grayscale image data, and to convert the point cloud data of each view angle from a camera coordinate system to a target coordinate system.

[0102] An image processing module is configured to perform feature extraction on the grayscale image data of each view angle to obtain a plurality of key points of the grayscale image data, to generate descriptors according to gradient features of a surrounding area of the key points, to perform feature matching on the key points of the grayscale image data of the high view angle and the low view angle according to the similarity of the descriptors, to obtain a plurality of matching point pairs, and to perform mixed superposition on the color image data of the high view angle and the low view angle according to all the matching pairs, thereby generating a superimposed image.

[0103] A point cloud processing module is configured to calculate a normal vector and a curvature of each point cloud surface according to the point cloud data in the target coordinate system, to convert the discrete point cloud data into a continuous three-dimensional surface by using a Poisson reconstruction algorithm and a spherical rotation algorithm according to the normal vector and the curvature of each point cloud surface, and to construct a three-dimensional polygon mesh model.

[0104] Preferably, the three-dimensional model construction system further comprises:

[0105] A visualization module is configured to provide an interactive three-dimensional visualization interface to realize rendering and dynamic display of the image data, the point cloud data, and the three-dimensional polygon mesh model in the whole life cycle.

[0106] A performance statistics module is configured to record the calculation time consumption of each processing link, and evaluate the processing efficiency of the system according to the calculation time consumption data of the processing links.

[0107] The technical solution shown in the embodiment is that the visualization module is based on a cross-platform graphics library, and an interactive three-dimensional visualization interface is developed to realize real-time rendering and dynamic display of point clouds and three-dimensional mesh models. The interface supports multi-view view switching, three-dimensional model scaling, rotation, translation and other interactive operations, integrates distance measurement, angle measurement, volume calculation and other professional measurement tools, and provides model labeling and editing functions. At the same time, the processing result can be saved in common image and video formats to meet the data display and analysis requirements.

[0108] The system has a built-in performance statistics module with processing time statistics function, which can record the calculation time consumption of each key link in the processing flow. The point cloud processing involves filtering, downsampling, normal vector calculation and surface reconstruction, and the image registration covers image reading, preprocessing, feature extraction, matching and registration. These statistical data provide important reference for evaluating the processing efficiency of the system, analyzing the bottlenecks in the processing process and subsequent parameter optimization.

[0109] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0110] It should be noted that in the description of the present application, the terms "first", "second" and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is at least two.

[0111] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions or other processes, and the various embodiments of the present application include additional implementations in which the order of execution is different, in which other code modules are used, in which other structures are used, in which not all code modules are executed, and in which not all of the functions are performed, and in which the functions are performed in different orders, all of which are within the scope of the embodiments of the present application.

[0112] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0114] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0115] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0116] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0117] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A three-dimensional model construction method based on image registration and matching, characterized in that: include: A depth camera is used to acquire color image data and depth data at each of the target's high and low viewing angles. A dual-threaded asynchronous acquisition mechanism is used to achieve spatiotemporal alignment of the image data and depth data through timestamp matching to generate point cloud data at each viewing angle. Convert the color image data at each viewing angle into grayscale image data; convert the point cloud data at each viewing angle from the camera coordinate system to the target coordinate system; Perform feature extraction on the grayscale image data at each viewing angle to obtain multiple key points of the grayscale image data, and generate descriptors based on the gradient features of the area around the key points; Based on the similarity of the descriptors, feature matching is performed on the key points of the grayscale image data of high and low viewing angles to obtain multiple matching point pairs; Based on all matching pairs, the high-viewing angle and low-viewing angle color image data are mixed and superimposed to generate a superimposed image; According to the point cloud data in the target coordinate system, calculate the normal vector and curvature of each point cloud surface; Using the Poisson reconstruction algorithm and the spherical rotation algorithm, the discrete point cloud data is converted into a continuous three-dimensional surface according to the normal vector and curvature of each point cloud surface, and a three-dimensional polygonal mesh model is constructed.

2. The method according to claim 1, characterized in that Also includes: The multi-frame median fusion algorithm is used to reduce the noise of the collected depth data; Voxel downsampling and statistical outlier filtering algorithms are used to perform redundant compression and outlier removal on the point cloud data at each perspective to obtain normalized point cloud data.

3. The method according to claim 1, characterized in that After converting the point cloud data at each viewpoint from the camera coordinate system to the target coordinate system, it also includes: Perform filtering and downsampling on the point cloud data in the target coordinate system; The filtering process adopts statistical filtering process or radius filtering process; the downsampling process adopts voxel downsampling process or random downsampling process.

4. The method according to claim 1, wherein Perform feature matching on the key points of the high-angle and low-angle grayscale image data to obtain multiple matching point pairs, including: A fast approximate nearest neighbor matcher is used to perform preliminary matching of key points based on the K nearest neighbor algorithm to obtain preliminary matching pairs; The preliminary matching pairs were first screened by setting the ratio test threshold; Angle consistency constraint and random sampling consistency algorithm are used to perform geometric verification and false matching elimination on the matching pairs after the first screening to obtain the final multiple matching point pairs.

5. The method according to claim 1, wherein The high-viewing angle and low-viewing angle color image data are mixed and superimposed to generate a superimposed image, including: Calculating a homography matrix describing the perspective transformation relationship between the color image at high viewing angle and low viewing angle based on the matching pairs; According to the homography matrix, perspective transformation is performed on the high-viewing angle color image, and the high-viewing angle and low-viewing angle color images are superimposed and displayed by an image fusion algorithm to generate a superimposed image.

6. The method according to claim 5, characterized in that Also includes: Calculating a mean square error evaluation index, a normalized cross-correlation evaluation index, and a structural similarity evaluation index based on the superimposed image, the high-viewing angle color image, and the low-viewing angle color image; The registration accuracy of the superimposed images is quantitatively analyzed based on the mean square error evaluation index, normalized cross-correlation evaluation index and structural similarity evaluation index to generate evaluation analysis results.

7. The method according to claim 1, characterized in that Calculate the normal vector of each point cloud surface, including: Solve multiple eigenvalues ​​and eigenvectors of each point cloud surface through the covariance matrix of the neighborhood points; The eigenvector corresponding to the minimum eigenvalue is used as the normal vector.

8. The method according to claim 7, characterized in that After obtaining multiple eigenvalues ​​and eigenvectors, it also includes: Setting the weight of the eigenvalue according to the distance of the neighborhood points, and obtaining the final eigenvector as the normal vector according to the eigenvalue and the weight; The area with curvature greater than a preset value is optimized by weighted averaging of neighborhood normal vectors to make the normal vectors smoother.

9. A three-dimensional model construction system for image registration and matching, characterized in that: include: The data acquisition module is used to use the depth camera to obtain color image data and depth data at each high and low viewing angles of the target. A dual-threaded asynchronous acquisition mechanism is used to achieve spatiotemporal alignment of the image data and depth data through timestamp matching to generate point cloud data at each viewing angle. A preliminary processing module is used to convert the color image data at each viewing angle into grayscale image data; and to convert the point cloud data at each viewing angle from the camera coordinate system to the target coordinate system; An image processing module is used to extract features from the grayscale image data at each viewing angle, obtain multiple key points of the grayscale image data, and generate descriptors based on the gradient features of the area around the key points; Based on the similarity of the descriptors, feature matching is performed on the key points of the grayscale image data of high and low viewing angles to obtain multiple matching point pairs; Based on all matching pairs, the high-viewing angle and low-viewing angle color image data are mixed and superimposed to generate a superimposed image; The point cloud processing module is used to calculate the normal vector and curvature of each point cloud surface based on the point cloud data in the target coordinate system; Using the Poisson reconstruction algorithm and the spherical rotation algorithm, the discrete point cloud data is converted into a continuous three-dimensional surface according to the normal vector and curvature of each point cloud surface, and a three-dimensional polygonal mesh model is constructed.

10. The three-dimensional model construction system according to claim 9, characterized in that: Also includes: The visualization module is used to provide an interactive 3D visualization interface to render and dynamically display image data, point cloud data, and 3D polygon mesh models throughout the entire life cycle; The performance statistics module is used to record the computation time of each processing link; and evaluate the processing efficiency of the system based on the computation time data of each processing link.

Images