Binocular scanner high-speed three-dimensional precise matching and three-dimensional reconstruction system

Abstract

According to the binocular scanner high-speed three-dimensional fine matching and three-dimensional reconstruction system, a binocular 3D scanner optimization scheme for improving FPGA acceleration is provided, an image processing algorithm with a large bottom layer data volume is accelerated in hardware, continuous gray free combination light spots are adopted as structured light images for three-dimensional measurement, FPGA image filtering is improved, and the three-dimensional measurement precision is improved. The method comprises the following steps of: reducing data processing capacity by adopting an image series-connection type subtraction acquisition method, correcting epipolar lines of images to be horizontal by adopting an image binocular correction method, performing scale cost adaptive transformation on the images, accurately detecting edge and angular point features of the images, and reducing dependence on a central pixel by adopting scale cost adaptive transformation based on Gaussian distribution sampling; constructing a triangular mesh by adopting a hollow circular polar angle triangularization method, and describing the surface image of the three-dimensional object; a 3D scanning system platform is established, a three-dimensional curved surface is reconstructed according to a depth map returned by the improved FPGA, the improved FPGA realizes a coprocessor scheme based on a SoCKit development board, and real-time high-precision 3D scanning is realized.

Description

Technical Field

[0001] The present application relates to a three-dimensional matching and reconstruction system of a binocular 3D scanner, and in particular to a high-speed three-dimensional precise matching and stereo reconstruction system of a binocular scanner, belonging to the technical field of three-dimensional image digital processing. Background Art

[0002] With the rapid development of computer and image processing technology, the level of three-dimensional digital technology has gradually improved, and it has been integrated into many different fields in various ways and roles: in the industrial field, the realization of product design and mold development and other links must inevitably use three-dimensional digital measurement technology; in scientific experiments, the three-dimensional color model data of the scene is used to realize virtual reality technology to complete simulation training; in the medical field, common CT machines and MRI nuclear magnetic resonance instruments also use three-dimensional digital technology; other fields include archaeological research, as well as the popular 3D network virtual world and games, in which three-dimensional digital technology has played an important role, and all this proves that the three-dimensional world will have broad uses in the future.

[0003] With the development of the three-dimensional information field, related equipment will also be developed. 3D scanners are one of them. As a front-end device, 3D scanners can realize information input. By analyzing real-world objects or environments, their surface shape data is collected, and the scanned data is used to reconstruct three-dimensional models in the virtual world. The existing technology for constructing three-dimensional models of objects includes: Method 1 is to build a three-dimensional model by using modeling software. The relevant software includes UG, Pro / E and CATIA, etc. The geometric elements such as cubes and balls are scaled, sheared, translated, and rotated to build a relatively complex scene. Since this method can accurately manage the shape and position of the elements, it is often used to construct three-dimensional models such as buildings and furniture. In addition, because it can produce some unique rendering effects and is easy to control lighting and texture, it is also often used in film and television works and advertising special effects. However, the disadvantage is that it requires a full grasp of the scene data and the software operation is complex, so it is necessary to build a three-dimensional model. The modeling cycle is long, which increases the production cost; the second is to use images, videos, etc. to reconstruct the three-dimensional model, and build a realistic scene or object model like a photo based on the image of the real scene. The modeling cost is getting lower and lower, and the labor intensity is also reduced, but the disadvantage of this method is that machine operation cannot replace manual operation, and the modeling accuracy is not high; the third method is to use three-dimensional scanning equipment, including laser, ultrasound or infrared to achieve ranging and obtain three-dimensional information of objects. The measurement results are highly accurate and are often used in applications such as complex mechanical parts and cultural relics that require precision, but there are still many restrictions on its use for larger scenes.

[0004] Developing a robust, high-precision, and high-speed binocular 3D scanner is of great value, but it still faces challenges in many aspects. For example, it is difficult to achieve real-time performance. There are two reasons why real-time performance is difficult to achieve: one is the high algorithm complexity, and the other is the need for a large storage space.

[0005] The global algorithm complexity of binocular 3D matching is much higher than the local algorithm complexity. In the process of implementing 3D matching, in order to achieve real-time performance, in addition to complexity, the required storage space must also be considered. The storage space requirements of the matching algorithm are divided into: the space overhead required to store binocular images, the space overhead required to store DSI or cost matrix, the space overhead required to store disparity maps, and the space overhead required for intermediate results.

[0006] The three-dimensional matching algorithm is to find the point pairs or blocks with the same features on the binocular image pair, and describe these features or block energies with functions, based on which to find the matching parts in the two views and calculate their horizontal difference. The real-time research of three-dimensional matching is generally divided into three types according to the different computing platforms: realizing binocular stereo vision on CPU, CPU, and FPGA. According to the characteristics of each platform, although the CPU can use its high main frequency, multi-core and MMX parallel instruction characteristics to realize binocular stereo vision, compared with FPGA and GPU, two parallel processing systems, the CPU is limited by the computing framework and is not ideal when processing high-complexity and high-density calculations. Therefore, it can only be used to process some applications with small computing volume, which makes the CPU unable to develop in the long run in the realization of real-time binocular vision system. In contrast, the characteristics of FPGA and GPU parallel processing make it possible to achieve satisfactory results when realizing binocular vision system. Therefore, in order to realize high-performance and high-quality real-time binocular stereo vision system, FPGA and GPU are used as the main platform. Compared to GPU, the design process of FPGA platform is a process of IC circuit design using hardware description language (VHDL or Verilog). When FPGA is powered on and configured, it can be regarded as a chip specially made for this application. In this case, even if a lower clock frequency and lower power consumption are used on the FPGA platform than CPU or GPU, its final processing efficiency can still be higher than them.

[0007] In summary, binocular 3D scanners have broad application prospects, but the binocular scanner three-dimensional precise matching and stereo reconstruction system also has technical difficulties that need to be solved, including:

[0008] (1) The existing technology uses modeling software to construct three-dimensional models of objects, which requires a full understanding of scene data and complex software operations. Therefore, the modeling cycle is long, which increases the production cost; the three-dimensional model is reconstructed using images, videos, etc., and machine operations cannot replace manual operations, and the modeling accuracy is not high; the use of three-dimensional scanning equipment for larger scenes still has many limitations. Currently, there is an urgent need for a binocular 3D scanner with strong robustness, high precision and high speed, but there are still many challenges. Due to the high algorithm complexity, a large storage space is required, and it is difficult to achieve real-time performance; the existing technology CPU platform realizes binocular stereo vision, but the CPU is limited by the computing framework and the effect is not ideal when processing high-complexity and high-density calculations. It can only be used to process some applications with small computing volume. The existing technology cannot process data in parallel, lacks hardware acceleration of the underlying image processing algorithm with large data volume, and has the limitation of von Neumann serial processing. The development cost is high and the instrument is large in size.

[0009] (2) The prior art lacks spatial neighborhood coding in the structured light measurement system, and is unable to project freely combined light spots onto the target object and verify the identification effect of the freely combined light spots with correlation factors, and is unable to achieve the goal of obtaining a dense point cloud with a single measurement; it lacks a simulation module for establishing median filtering and Gaussian filtering on DSP Builder, and the required cycle is too long, making it impossible to achieve the real-time performance of the 3D scanner; it lacks the use of Gaussian pyramids for image serial reduction acquisition, and the amount of data to be processed is large. In order to improve the matching accuracy and reduce the processing time, it lacks the method of using image binocular correction, and the matching accuracy is low, and the processing time is long; it is impossible to adjust the image planes of the left and right cameras to the fair plane so that the two images can be strictly aligned in line, and the time required for three-dimensional matching is long. It is impossible to change the search range from the plane to only search on the epipolar line, and the search dimension is large, resulting in poor image matching accuracy and matching speed.

[0010] (3) The prior art lacks a scale-cost adaptive transformation for the corrected image, lacks improvements based on the shortcomings of the scale-cost adaptive transformation, is heavily dependent on the center pixel, and has poor matching effects. It lacks a simulation module for improving the scale-cost adaptive transformation on DSPBuilder for simulation, and is unable to reduce the time required by analyzing the clock cycle. The local structural features of the image cannot be identified; it is unable to implement a method for clipping dissimilarity distances, lacks a method for generating a comparison matrix through code, and then calculating the clipping dissimilarity distance matrix to finally obtain a depth map, lacks a method for parallel processing on hardware, takes a long time to check the approximation of matching points, and cannot accurately judge the approximation of the bit string after the scale-cost adaptive transformation, resulting in a long three-dimensional matching time and poor matching accuracy.

[0011] (4) The prior art lacks a method for calculating the coordinates of a 3D point cloud based on the depth map, and then using the empty circular polar angle triangulation method to sequentially connect the discrete 3D point clouds into a 3D surface, and finally displaying the triangulated reconstructed image. It also lacks a specific idea for implementing this part of the process, which has a complex algorithm but a small amount of data processing, in VTK. It lacks an implementation method for the clipping difference distance, and is unable to evaluate the similarity between two bit strings by calculating the clipping difference distance, and determine whether two pixel points are matching points. It lacks a method for generating a 3D point cloud based on parallax calculation, and lacks a method for constructing a triangular mesh based on the generated discrete point cloud using the empty circular polar angle triangulation method, which results in the inability to build a reliable 3D scanning system platform. It lacks the specific steps of controlling image acquisition by computer, downloading the original image to the FPGA, and reconstructing the 3D surface based on the depth map sent back by the FPGA. It lacks an FPGA coprocessor solution based on the SoCKit development board, and is unable to achieve real-time high-precision 3D scanning. Summary of the invention

[0012] This application proposes an optimization scheme for binocular 3D scanners accelerated by improved FPGA, accelerates the underlying image processing algorithm with large amount of data in hardware, adopts continuous grayscale free combination of light spots as structured light images for three-dimensional measurement, designs and implements image filtering processing based on improved FPGA, adopts image serial subtraction acquisition method to reduce data processing volume, adopts image binocular correction method to correct the polar lines of the image to be horizontal, and performs scale-cost adaptive transformation on the image before three-dimensional matching to accurately detect the edge and corner features of the image, adopts a scale-cost adaptive transformation based on Gaussian distribution sampling to reduce the dependence on the central pixel, and implements it on the improved FPGA; adopts the empty circle polar angle triangulation method to construct a triangular mesh to describe the surface image of the three-dimensional object; finally, builds a 3D scanning system platform based on the above method, adopts a computer to control image acquisition, then downloads the original image to the improved FPGA and reconstructs the three-dimensional surface according to the depth map sent back by the improved FPGA, and the improved FPGA implements a coprocessor solution based on the SoCKit development board to achieve real-time and high-precision 3D scanning.

[0013] In order to achieve the above technical features and effects, the technical solutions adopted in this application are as follows:

[0014] Binocular scanner high-speed 3D precise matching and stereo reconstruction system, first is the binocular scanner 3D matching reconstruction scheme; second is the image preprocessing of binocular 3D scanner based on improved FPGA, including: projected 3D image, image filtering, image cascade subtraction acquisition, image binocular correction; third is the scale cost adaptive transformation of binocular 3D scanner based on improved FPGA; fourth is binocular scanning 3D matching based on improved FPGA, including: approximation calculation evaluation, hardware implementation of clipping different distances, and 3D matching search to generate depth map; fifth is binocular scanner high-speed 3D surface reconstruction, including: reconstruction of 3D image based on binocular depth map, empty circle polar angle triangulation based on VTK; sixth is the construction of binocular 3D scanner system with improved FPGA;

[0015] This application proposes an optimization scheme for binocular 3D scanners accelerated by improved FPGA. By using improved FPGA hardware parallel processing, the image processing algorithm with large underlying data volume is accelerated in hardware. Continuous grayscale free combination of light spots is used as the structured light image for three-dimensional measurement. The image filtering processing based on the improved FPGA is designed and implemented. After the image is filtered, the image is subjected to the image serial subtraction acquisition method to reduce the data processing volume. The image binocular correction method is used to correct the polar lines of the image to be horizontal. When searching for the matching point of the left image corresponding to the right image, it is only necessary to search for the same row of pixels, and finally the binocular correction is implemented on the improved FPGA.

[0016] Before three-dimensional matching, the image is first subjected to a scale-cost adaptive transformation to accurately detect the edge and corner features of the image. A scale-cost adaptive transformation based on Gaussian distribution sampling is used to reduce the dependence on the central pixel and is implemented on an improved FPGA. After the scale-cost adaptive transformation, each pixel of the original image is represented by a bit string. The present application designs an implementation method of the clipping dissimilarity distance in the improved FPGA, and evaluates the similarity between two bit strings by calculating the clipping dissimilarity distance to determine whether the two pixels are matching points. If the clipping dissimilarity distance of the two matching points is less than a critical value, it is determined that the match is successful. The final result is the disparity of the two matching points. A three-dimensional point cloud is generated based on the disparity calculation. Based on the generated discrete point cloud, a triangular mesh is constructed using an empty circular polar angle triangulation method to describe the surface image of the three-dimensional object.

[0017] Finally, a 3D scanning system platform was built based on the above method. The image acquisition was controlled by a computer. The original image was then downloaded to the improved FPGA and the three-dimensional surface was reconstructed according to the depth map sent back by the improved FPGA. The improved FPGA implemented a coprocessor solution based on the SoCKit development board, and the EDA tools of DSP Builder and Qsys were used to achieve real-time and high-precision 3D scanning.

[0018] Preferably, project a three-dimensional image: The goal is to generate a dense point cloud with only one measurement, and a combined gray light spot projection image coding is adopted;

[0019] 1. Generation of combined light spots

[0020] Adopt a combined gray light spot projection image coding, and a dense point cloud can be obtained with a single measurement. The projection generation method is as follows:

[0021] M = Rand(m, n) * A Equation 1

[0022] In the formula, M is the generated combined gray matrix, m = h / Δx, n = w / Δy, A is the image gray level, and the range is 1 - 256. Rand generates a combined number between 0 and 1, excluding 1.

[0023] 2. Relevance of projection area

[0024] During the three-dimensional matching process, a certain feature point in the left image has a unique corresponding point in the right image. With the help of structured light, this corresponding point can be quickly found. An association factor is used to verify the uniqueness of the combined gray light spot projection mark. The association factor is as follows:

[0025]

[0026] In the above formula, 0 ≤ i < Ma + Mb - 1, 0 ≤ j < Na + Nb - 1,, (Ma, Mb) is the size of A, and (Na, Nb) is the size of B.

[0027] Preferably, image filtering: An improved FPGA is used to implement image filtering. A hierarchical window is used to perform block operations on the image. The hierarchical window is moved until the entire image is subjected to block operations, realizing the processing of the entire image. This process is expressed as:

[0028] Q[x, y] = ff(I[x, y],....I[x + Δx, y + Δy]), (Δx, Δy) ∈ W Equation 3

[0029] W is a hierarchical window centered on I[x, y]. Every time the hierarchical window W scans to a position of the input image, an output pixel will be generated at this position, and the value of this output pixel is obtained through Equation 3. f is a filtering function;

[0030] The hierarchical moving hierarchical window filtering has three stages: input row buffer, pixel point operation, and output result. Among them, the input row buffer adopts FIFO hierarchical window filtering. The input image from the memory is continuously input into the FIFO, and pixels are continuously read out from the other end of the FIFO and sent to the arithmetic unit for processing. The input and output speeds are matched, and no additional memory reading and writing is performed in the middle;

[0031] Image tandem subtraction: First, the first tandem subtraction is to generate a downsampled image for the original image, and then the second tandem subtraction is for the newly generated downsampled image. This cycle is repeated to form a set of images. If the obtained images are stacked in sequence, a pyramid shape is formed.

[0032] The pyramid model can be summarized into two parts: the first step is to pass the image through a low-pass filter to smooth the image, and the second step is to sample the image processed in the previous step. The final result is an image that is processed into a size different from the original size.

[0033] Each level of image in the sequence is obtained by low-pass filtering the previous level of image and then performing interlaced row and column serial subtraction acquisition, as follows:

[0034]

[0035] Among them, G k (i, j) is located at the kth layer of the pyramid image. The bottom layer of the pyramid is G0, which is also the original image that needs to be collected in series. At the same time, w(m, n) = h(m)h(n) is a 5×5 low-pass graded window function, h is a Gaussian density distribution function, and the graded window function w(m, n) is expressed as Formula 5:

[0036]

[0037] An image pyramid is constructed through multi-layer serial subtraction processing.

[0038] Preferably, image binocular correction:

[0039] Adjust the image planes of the left and right cameras to the same plane and align the image planes in rows. At this time, the position of the pole is at infinity, and the pole line corresponding to the feature point is the same row of horizontal lines, ensuring that the two images are strictly in line correspondence. The feature point search only needs to be searched in the same row of pixels of the other camera. According to the internal and external parameter matrix of the imaging coordinates, the image binocular correction is made coplanar and row-aligned. The specific steps are:

[0040] Step 1: Divide the rotation matrix R obtained by stereo calibration into two parts r1 and r2, which are called aggregate rotation matrices. Rotate the left and right cameras by half to make the optical axes of the left and right cameras parallel.

[0041] Step 2: Rotate the image around the optical center to make the polar line horizontal and the pole at infinity. Select a vector with the principal point as the origin and the pole direction. This vector is the translation vector e1 between the optical centers. The next vector e2 must be orthogonal to e1 and is obtained by cross product of e1 and the principal optical axis. The third vector is required to be orthogonal to e, e, and is obtained by cross product. Finally, the pole is transformed to infinity. The matrix after row alignment is:

[0042]

[0043] Step 3: Combining the previous two steps, the final image is adjusted to be coplanar, and the correction matrix for parallel alignment is:

[0044] R l =R rect r l

[0045] R r =R rect r r Formula 7

[0046] For each coordinate point of the corrected image, the coordinates of the original image corresponding to the coordinate point are obtained by calculating the correction matrix, camera parameter matrix, and inverse matrix of the distortion moment. The original image generated after inverse mapping has corresponding coordinates of floating point type. The pixel values ​​of these floating point coordinates are obtained by using bilinear interpolation.

[0047] Improving FPGA hardware to implement image binocular correction includes two steps: one is to look up the mapping table for each pixel coordinate; the other is to obtain the pixel value of the new coordinate through bilinear interpolation based on the table lookup result.

[0048] The image binocular correction module consists of: a lookup table, two pixel address generators, a scan line buffer address generator, a pixel value calculator, and the mapping table coordinates of the left and right image binocular correction completed by offline calculation are stored in two LUTX and two LUTY logic unit tables. A coordinate address is obtained by looking up the table, and the integer part of the address determines the coordinate address Pix_address_X of the four adjacent points ABCD pixels of the coordinate, and the pixel values ​​Pixel_X of the four points ABCD are found from the original image. The pixel shift register is completed by a memory with a write port and two read ports. Two pixel values ​​are read out in the first clock cycle, and two pixel values ​​are read out in the next clock cycle. Then these four pixel values ​​are input into the pixel value calculator, and the decimal part of the address obtained from the lookup table is used to calculate the pixel value of the final corrected image.

[0049] Preferably, the scale cost adaptive transformation of the binocular 3D scanner of FPGA is improved: the plane Gaussian distribution N(μ, ∑) is used to weight the transformation grading window, and then the grading window is sampled 2×L times, and every two sampling points i, j form a comparison point pair<i,j> , the set of comparison point pairs is the result of the final scale cost adaptive transformation. The specific implementation process is as follows:

[0050] Step 1: Sampling according to the plane Gaussian distribution, generating 2×(N-1) sampling points, the expression of the plane Gaussian function, where ,

[0051]

[0052] Transform x and y into polar coordinates:

[0053]

[0054] After transforming the original variables x and y into new variables ρ and θ, the probability density function g(ρ,θ) of ρ and θ is obtained according to the probability density function f(x,y) of x and y:

[0055]

[0056] θ follows a uniform distribution on [-π,π], and ρ follows an exponential distribution with λ=-1. The final sampling method is:

[0057] Generate θ: θ = 2πr-π, r∈U[0,1];

[0058] Generate ρ: ρ = -lnr, r∈U[0,1];

[0059] The final result:

[0060] The above sampling obtains continuous values, discretizes the uniform sampling area, and obtains the coordinates of discrete points;

[0061] Step 2: According to the sampling method of step 1, 2×(N-1) sampling points are obtained, and then these sampling points are combined into a scale cost adaptive transformation template. These transformation templates are not unique, and only one sampling result needs to be selected as the subsequent transformation template;

[0062] Step 3: Perform scale cost adaptive transformation based on the obtained comparison template.

[0063] The hardware implementation of scale-cost adaptive transformation consists of two parts: a hierarchical moving grading window module and an element comparison module within the grading window;

[0064] The hierarchical mobile grading window module includes a delay module and a storage delay module. The storage delay module realizes row cache. The input of the module is data_in[7:0], which inputs 8-bit pixel data. After cache and delay processing, r1, r1_d1, and rl_d2 respectively represent the first row and first column elements, the first row and second column elements, and the first row and third column elements of the grading window. r2, r2_d2, r3, r3_d1, and r3_d2 correspond to the elements of the second row and the third row respectively.

[0065] The element comparison module compares the paired elements in the classification window using a comparator according to the generated coordinate template, and then uses the Bus Builder module in DSP Builder to compose the comparison results into 8-bit data output.

[0066] Preferably, the similarity calculation evaluation is: calculating the minimum cost to clip the source string into the target string, the clipping process operations include replacement, deletion, and insertion, and the sum of the three operations is defined as the clipping difference distance of the two strings. The mathematical expression of the clipping difference distance is:

[0067]

[0068] Among them, i and j represent the length of string s1 and s2 respectively, lev [i-1,j] +1 inserts a letter into string s2, lev [i-1,j] +1 deletes a letter from string s1. When the i-th character of string s1 is different from the j-th character of string s2, f(i,j)=1, otherwise it is 0. That is, when calculating the clip difference distance, the source string and the target string are arranged into a table, and the value of each position in the table is calculated one by one according to the calculation formula of the clip difference distance; the values ​​of the first row and the first column do not need to be calculated and are automatically filled. To calculate the value at (i,j), three values ​​(i-1,j), (i,j-1), and (i-1,j-1) are required for comparison.

[0069] Preferably, the hardware implementation of clipping different distances is as follows: in the process of improving the FPGA to perform a three-dimensional matching module, the corrected image and the left and right images after the scale cost adaptive transformation are uniformly placed in the SDRAM memory. In order to realize the synchronous operation of the left and right images, the SDRAM adopts a two-read and two-write port structure to realize the parallel storage of the two images and facilitate reading and writing.

[0070] Converting to improved FPGA hardware enables parallel operation. The calculation of each position depends on the values ​​of the left, top, and top-left positions. Positions on the same diagonal line are calculated simultaneously.

[0071] After obtaining the comparison matrix, the next step is to calculate the clipping difference distance matrix by writing Verilog HDL code. Each calculation simultaneously calculates the values ​​of all positions of the diagonal line. The number of elements of the diagonal line is not fixed and is implemented using combinational logic circuits.

[0072] Finally, the clipping dissimilarity distance is implemented in software and hardware by comparing the number of operations. The clipping dissimilarity distance implemented in hardware does not need to calculate each element one by one, but calculates all the elements in the upper angle at the same time. For two strings of length m and n respectively, the software serialization requires m×n operations, while in hardware, only m+n operations are required. Completing the clipping dissimilarity distance on the improved FPGA not only greatly speeds up the speed of three-dimensional matching, but also has efficient real-time performance.

[0073] Preferably, a three-dimensional matching search generates a depth map: if the clipping difference distance of two feature points does not reach a specified critical value, the two feature points are considered as matching points, and the left and right images are matched for feature points. The three-dimensional matching only needs to be compared and searched in the same row;

[0074] Assume an ideal camera stereo coordinate system, the left and right cameras have the same internal parameters, the same focal length, and the same principal point position. The image coordinates are based on the upper left corner of the image as the origin, the pixels in the same row of the image plane of the left and right images are aligned, the optical axes of the left and right cameras are parallel, and the origin of the camera coordinates is the optical center of the left camera;

[0075] The simplified stereo measurement model uses the triangle method to obtain the coordinates of point P:

[0076]

[0077] Where f is the focal length, (x l -x r ) represents the parallax of two matching points. If the internal and external parameters of the camera are calibrated, the matching points of the left and right images are found (the same point in the physical world is the imaging point of the left and right images), and the x, y, z coordinates of point P in the real scene are calculated. According to formula 10, the difference in coordinates of the matching points (x l -x r ) is parallax.

[0078] Preferably, binocular scanner high-speed 3D surface reconstruction:

[0079] (I) Reconstructing 3D images based on binocular depth maps

[0080] Construct a three-dimensional grid, let P = {p1, p2, ..., p n} is a point set. Triangulation connects each data point to form a number of triangles, resulting in a triangular mesh. The triangular mesh maps the topological connection relationship of each data point and represents the connection of the data points. It is used for a plane point set. The empty circular polar angle triangulation of these points makes the circumscribed circle of each triangle free of any unnecessary points. The steps of the empty circular polar angle triangulation network method include:

[0081] Step 1: Construct a non-optimized initial triangulation;

[0082] Step 2: Iterate the edge exchange optimization of the convex quadrilateral: this quadrilateral is composed of two colateral triangles;

[0083] There are two steps in the reconstruction of the empty circular polar angle triangulation of the surface. The first is to perform analog focal axis transformation. The target of the transformation is the sampling points, which exist on the surface of the original object. The second is to use the inverse transformation method of the focal axis. The result after the transformation is the segmented approximate surface of the original surface. The point clouds of various forms are transmitted and converted into Thiessen depth polygons. Finally, the empty circular polar angle triangulation and weight map are obtained.

[0084] (1) Analog focal axis conversion

[0085] According to the different dimensions, there are two situations, namely, plane and elevation. In the plane, the focal axis is the set of the centers of the circles inside the boundary, while in three dimensions, it is the set of the centers of the largest spheres. In the plane, the circle must pass through at least two points on the boundary, while in the elevation, the sphere must pass through at least three points. However, no matter in the plane or elevation, inside the circle or the sphere, any point on the boundary should not be included. In the process of focal axis transformation, the coordinates of the focal axis and the radius are transformed together, so when the inverse transformation is performed, the original boundary is also represented and is the same as the original.

[0086] (2) Thiessen depth polygon graph

[0087] Assume there is a point set P = {p1, p2, ..., p k}, p i ∈R n , the Thiessen depth polygon partition area is defined as:

[0088]

[0089] Where x is any point, d(x, p i ) is x and p i The distance between them, the Thiessen depth polygon areas do not overlap, there is only one common edge, the two are common faces, and the graph formed by these areas is the Thiessen depth polygon graph;

[0090] (3) Weight Graph

[0091] Based on the approximate description of the surface based on the circumscribed circle, assume a sphere B with a center coordinate of c and a radius of ρ. c，ρ , the weighted distance from spatial point x to c is calculated by formula 12:

[0092] d pow (x,E c，x ) = d 2 (c, x)-ρ 2 Formula 12

[0093] Where d represents the Euclidean distance. Based on the weight graph, the surface is divided into several blocks, and in the point set in each block, there must be one that is closest to the vertex.

[0094] (II) Empty circle polar angle triangulation based on VTK

[0095] VTK is used to perform the empty circle polar angle triangulation method to achieve surface reconstruction of point cloud data. The specific process is as follows:

[0096] Step 1: Use the fopen function to read the 3D point cloud data stored in the txt text;

[0097] Step 2: Get a discrete 3D point cloud, and connect the point cloud into a 3D surface using the empty circle polar angle triangulation method;

[0098] Step 3: Map the triangulated network to the graphics library and display the stereo reconstruction result.

[0099] Preferably, the binocular 3D scanner system based on FPGA is constructed as follows:

[0100] (I) Improved FPGA-based binocular 3D scanner system architecture

[0101] The overall image processing flow is divided into two parts: hardware and software. The hardware part performs image filtering, image binocular correction, non-parametric transformation, and 3D matching. This part is completed by improving FPGA hardware, and the output result is the depth map of the object being measured. The software part calculates the 3D point cloud based on the depth map, and then reconstructs the 3D surface from the 3D point cloud. The algorithm of this part is complex, but the algorithm data volume is small, and it is implemented in computer software.

[0102] Software part: establish nios core to make SDRAM read and write control more convenient, read the image, process it in the module and then store it; for the hardware part, regard it as a custom IP, which includes image filtering, image binocular correction, non-parametric transformation, 3D matching algorithm and SDRAM interface and algorithm interface. The interface enables each module in the IP to exchange information correctly; the interface part is directly described in hardware language and then input into DSP Builder, and DSP Builder is used to complete modeling and improve development speed.

[0103] (II) Improvement of the hardware structure of the binocular 3D scanner based on FPGA

[0104] The system hardware consists of two CMOS sensors on the left and right, a projector, a synchronous trigger circuit, a computer, and a SoCKit development board;

[0105] The left and right cameras use USB interface cameras with CMOS line exposure. According to the size and distance of the measured object, a combination of grayscale light spots with appropriate resolution is selected to project the image. The adjustment criterion is that the light spot on the surface of the measured object is clearly identifiable. The placement height, angle and image brightness of the projector are adjusted to make the collected images of the left and right cameras clear and the grayscale gradient changes obvious.

[0106] The synchronous trigger circuit controls the sampling of the left and right cameras at the same time to ensure that the left and right cameras capture the image of the same object at the same time; the computer is the system control center. During the acquisition phase, it controls the synchronous trigger to generate a pulse signal, collects the images of the left and right cameras through the USB interface, and saves them in the computer; during the stereo reconstruction phase, the computer selects the original image data for stereo reconstruction through the user interface program designed by Qt, and transmits it to the SoCKit development board through Ethernet for image processing; the depth map processed by the SoCKit development board is transmitted back to the computer through the Ethernet interface, and the computer reconstructs the depth map into a three-dimensional surface and displays it to the user through a graded window.

[0107] Compared with the prior art, the innovations and advantages of this application are:

[0108] (1) This application solves the problem that the image processing part consumes too much time and cannot meet the real-time requirements due to factors such as high algorithm complexity. A high-speed and flexible embedded image processing system is designed using FPGA, and an optimization solution for binocular 3D scanner accelerated by improved FPGA is proposed. The image processing algorithm with large underlying data volume is accelerated in hardware by using improved FPGA hardware parallel processing. Continuous grayscale free combination of light spots is used as the structured light image for three-dimensional measurement. Image filtering processing based on improved FPGA is designed and implemented. After image filtering, the image is subjected to image serial subtraction acquisition to reduce the amount of data processing. The image binocular correction method is used to correct the image's epipolar line to a horizontal level. When searching for matching points of the left image corresponding to the right image, only the same row of pixels needs to be searched. Finally, binocular correction is implemented on the improved FPGA, so that a dense point cloud can be obtained in a single measurement. The correlation factor is used to verify that the identification effect of free combination of light spots is good.

[0109] (2) Before three-dimensional matching, the present application first performs a scale-cost adaptive transformation on the image to accurately detect the edge and corner features of the image, and adopts a scale-cost adaptive transformation based on Gaussian distribution sampling to reduce the dependence on the central pixel, and implements it on the improved FPGA; after the scale-cost adaptive transformation, each pixel point of the original image is represented by a bit string, and the present application designs an implementation method of the clipping difference distance in the improved FPGA, and evaluates the similarity between two bit strings by calculating the clipping difference distance to determine whether the two pixels are matching points. If the clipping difference distance of the two matching points is less than the critical value, it is determined that the match is successful, and the final result is the disparity of the two matching points. A three-dimensional point cloud is generated based on the disparity calculation, and based on the generated discrete point cloud, a triangular mesh is constructed using the empty circular polar angle triangulation method to describe the surface image of the three-dimensional object; this method reduces the dependence on the central pixel, and therefore has a better matching effect. The analysis of the clock cycle proves that the time required for implementation on the improved FPGA is reduced.

[0110] (3) This application proposes a binocular 3D scanner based on an improved FPGA, builds a 3D scanning system platform, uses a computer to control image acquisition, then downloads the original image to the improved FPGA and reconstructs the three-dimensional surface based on the depth map sent back by the improved FPGA. The improved FPGA can take advantage of data parallel processing and accelerate the underlying image processing algorithm with large data volume in hardware, getting rid of the limitations of von Neumann serial processing, making the development cost lower and the instrument smaller. The improved FPGA implements a coprocessor solution based on the SoCKit development board, and uses DSP Builder and Qsys EDA tools to achieve real-time high flexibility, reusability, high reliability, and high precision 3D scanning.

[0111] (4) The present application proposes a method of implementing clipping dissimilarity distance on an improved FPGA to make an accurate judgment on the similarity of a bit string. First, a comparison matrix is ​​generated through code, and then the clipping dissimilarity distance matrix is ​​calculated. Finally, a depth map is obtained. At the same time, the method of parallel processing on hardware reduces the processing time for checking the similarity of two matching points. The three-dimensional point cloud coordinates are calculated based on the depth map, and the empty circle polar angle triangulation method connects the discrete three-dimensional point cloud into a three-dimensional surface in sequence. The algorithm is complex, but the amount of data processing is not large. This part is implemented in the design of VTK. By comparing the clock cycles or number of operations required for each part, no matter which part it is, there is a significant degree of efficiency improvement and reduction in processing time compared to the software, which proves the feasibility and efficiency of implementing a 3D scanner on an improved FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 This is the hardware structure diagram of the binocular 3D scanner system with improved FPGA.

[0113] Figure 2 This is a diagram of the hierarchical window structure of the hardware-implemented hierarchical mobile filtering.

[0114] Figure 3 This is a diagram of the hardware implementation of image binocular correction.

[0115] Figure 4 It is a schematic diagram of the hardware structure of the hierarchical mobile graded window module.

[0116] Figure 5 It is a schematic diagram of the hardware structure of the scale cost adaptive transformation.

[0117] Figure 6 This is a schematic diagram of the scale cost adaptive transformation image output result.

[0118] Figure 7 This is a hardware calculation diagram of clipping mutual difference distance.

[0119] Figure 8 It is a simplified model diagram of stereo measurement for generating depth map by three-dimensional matching search.

[0120] Fig. 9 This is a flow chart of the method for triangulating the empty circle polar angle using VTK.

[0121] Fig.10 This is the synchronous trigger circuit diagram that controls the sampling of the left and right cameras at the same time.

[0122] Fig.11 is the original image of the book surface in the experiment.

[0123] Fig.12 This is the surface image of the book reconstructed in the experiment.

[0124] Fig.13 It is the hand scale cost adaptive transformation image in the experiment.

[0125] Fig.14 It is the three-dimensional reconstructed surface image of the human hand surface in the experiment. DETAILED DESCRIPTION

[0126] The following is a detailed description of the specific implementation of the binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system of the present application in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present application and implement it. Those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0127] At present, 3D scanners have special requirements for system processing speed and storage space due to the high complexity of the algorithm. How to make the processing speed faster and more accurate is the main problem faced today. In response to this problem, this application proposes an optimization solution for binocular 3D scanners based on improved FPGA acceleration. By using improved FPGA hardware parallel processing, the image processing algorithm with large underlying data volume is accelerated in hardware, breaking away from the limitations of von Neumann serial processing, reducing the development cost and making the instrument smaller.

[0128] This application uses continuous grayscale free combination spots as structured light images for three-dimensional measurement, designs and implements an image filtering process based on an improved FPGA, filters out the noise generated during the image acquisition process and improves the image quality, and after completing the image filtering, uses an image series reduction acquisition method to reduce the amount of data processing, and uses an image binocular correction method to correct the image's epipolar lines to a horizontal level. When looking for matching points in the left image corresponding to the right image, it is only necessary to search for pixels in the same row, and finally designs and implements image binocular correction on an improved FPGA platform.

[0129] Before 3D matching, the image is first subjected to a scale-cost adaptive transformation to accurately detect the edge and corner features of the image. However, the traditional transformation is overly dependent on the grayscale value of the center pixel, that is, when the grayscale value of the center pixel changes slightly, the transformation result may change dramatically. To avoid this situation, this application proposes a scale-cost adaptive transformation based on Gaussian distribution sampling to reduce the dependence on the center pixel and implement it on an improved FPGA.

[0130] After the scale cost adaptive transformation, each pixel of the original image is represented by a bit string. This application designs an implementation method of the clipping difference distance in an improved FPGA. The clipping difference distance is calculated to evaluate the similarity between two bit strings to determine whether the two pixels are matching points. If the clipping difference distance of the two matching points is less than the critical value, it is determined that the match is successful. The final result is the disparity of the two matching points. A three-dimensional point cloud is generated based on the disparity calculation. Based on the generated discrete point cloud, a triangular mesh is constructed using the empty circular polar angle triangulation method to describe the surface image of the three-dimensional object.

[0131] Finally, based on the above method, a 3D scanning system platform was built, and a computer was used to control image acquisition. Then the original image was downloaded to the improved FPGA and the 3D surface was reconstructed according to the depth map sent back by the improved FPGA. The improved FPGA implemented a coprocessor solution based on the SoCKit development board, and the EDA tools of DSP Builder and Qsys were used to simplify the design workload and improve the development speed. Real-time and high-precision 3D scanning was achieved.

[0132] 1. Binocular Scanner 3D Matching Reconstruction Solution

[0133] This application proposes a binocular 3D scanner based on an improved FPGA. Figure 1 The hardware structure of the system.

[0134] The generated free combination light spot is projected onto the surface of the object to be measured by a projector, and the object to be measured projected with the free combination light spot is obtained through two CMOS sensors. The synchronous trigger controls the left and right cameras to sample at the same time to ensure that the image collected is the same object at the same time. After the image that needs to be processed is obtained by the camera, it is saved in the cloud server, and the image is transmitted to the SoCKit development board for image processing. The obtained depth map is sent back to the cloud server, and finally the depth map is stereo reconstructed, and the final reconstruction result is displayed to the user through the graded window. This application realizes parallel image processing on an improved FPGA, improves the system processing speed, and meets real-time requirements.

[0135] To avoid unnecessary duplication of operations, only partial results of the previous level's operations need to be stored before starting the next level's operations, and RAM is used to cache information for each level's operations. However, the time required for each level to process data is not consistent. To avoid errors, an extra period of time is reserved for each level, and at the same time the read and write addresses of the RAM are controlled.

[0136] To accurately obtain the disparity information of each point, there are four levels: the first level, image filtering; the second level, image binocular correction; the third level, scale cost adaptive transformation; the fourth level, graded window approximation evaluation calculation. The noise in the image is filtered out by image filtering to ensure that the quality of the image is not affected. The image planes of the left and right cameras are corrected to be coplanar by image binocular correction, so that each search for feature points only needs to be searched in the same row of pixels of the other camera, reducing the processing time. After the left and right images are corrected, scale cost adaptive transformation is required before three-dimensional matching is realized. Scale cost adaptive transformation uses the size relationship of the grayscale values ​​of the neighboring pixels for scale cost transformation instead of pixel grayscale value as the approximation measure, so that the anti-interference ability of the collected image is enhanced, and the edge and corner features in the image can be better detected. The image pixel value after scale cost adaptive transformation has been converted into a bit string representing its features. Next, the clipping difference distance is used to evaluate the approximation of the two bit strings to determine whether the two feature points are matching points. After the judgment, the disparity output is obtained.

[0137] 2. Image preprocessing of binocular 3D scanner based on improved FPGA

[0138] (1) Projecting 3D images

[0139] The goal is to generate a dense point cloud with only one measurement, encoded using a combined grayscale spot projection image.

[0140] 1. Free combination spot generation

[0141] Using combined grayscale spot projection image coding, a dense point cloud can be obtained with a single measurement. The projection generation method is as follows:

[0142] M = Rand(m, n) * A Equation 1

[0143] In the equation, M is the generated combined grayscale matrix, m = h / Δx, n = w / Δy, A is the image grayscale level, with a range of 1 - 256, and Rand generates a combination number between 0 - 1, excluding 1.

[0144] 2. Projection area correlation

[0145] During the 3D matching process, a certain feature point in the left image has a unique corresponding point in the right image. With the help of structured light, this corresponding point can be quickly found. An association factor is used to verify the uniqueness of the combined grayscale spot projection mark. The association factor is as follows:

[0146]

[0147] In the above equation, 0 ≤ i < Ma + Mb - 1, 0 ≤ j < Na + Nb - 1, (Ma, Mb) is the size of A, and (Na, Nb) is the size of B.

[0148] (2) Image filtering

[0149] An improved FPGA is used to implement image filtering. A hierarchical window is used to perform block operations on the image. The hierarchical window is moved until the entire image has been subjected to block operations, realizing the processing of the entire image. This process is expressed as:

[0150] Q[x, y] = f(I[x, y],....I[x + Δx, y + Δy]), (Δx, Δy) ∈ W Equation 3

[0151] W is a hierarchical window centered on I[x, y]. Every time the hierarchical window W scans to a position in the input image, an output pixel is generated at this position, and the value of this output pixel is obtained through Equation 3. f is a filtering function.

[0152] The hardware implementation of the hierarchical moving filter hierarchical window structure is as Figure 2 shown. The hierarchical moving hierarchical window filtering has three stages: input row buffer, pixel point operation, and output result. Among them, the input row buffer uses FIFO hierarchical window filtering. The input image from the memory is continuously input into the FIFO, and pixels are continuously read from the other end of the FIFO and sent to the arithmetic unit for processing. The input and output speeds are matched, and no additional memory reads and writes are performed in the middle, improving the processing speed.

[0153] (3) Image serial subtraction acquisition

[0154] First, the first serial subtraction is performed on the original image to generate a downsampled image, and then the second serial subtraction is performed on the downsampled image just generated. This cycle is repeated to form a set of images. If the obtained images are stacked in sequence, a pyramid shape is formed.

[0155] The pyramid model can be summarized into two parts: the first step is to pass the image through a low-pass filter to smooth the image, and the second step is to sample the image processed in the previous step. The final result is an image that is processed to a different size from the original.

[0156] Each level of image in the sequence is obtained by low-pass filtering the previous level of image and then performing interlaced row and column serial subtraction acquisition, as follows:

[0157]

[0158] Among them, G k (i, j) is located at the kth layer of the pyramid image. The bottom layer of the pyramid is G0, which is also the original image that needs to be collected in series. At the same time, w(m, n) = h(m)h(n) is a 5×5 low-pass graded window function, h is a Gaussian density distribution function, and the graded window function w(m, n) is expressed as Formula 5:

[0159]

[0160] An image pyramid is constructed through multi-layer serial subtraction processing.

[0161] (IV) Image Binocular Correction

[0162] Adjust the image planes of the left and right cameras to the same plane and align the image planes in rows. At this time, the position of the pole is at infinity, and the pole line corresponding to the feature point is the same row of horizontal lines, ensuring that the two images are strictly in line correspondence. The feature point search only needs to be searched in the same row of pixels of the other camera. According to the internal and external parameter matrix of the imaging coordinates, the image binocular correction is made coplanar and row-aligned. The specific steps are:

[0163] Step 1: Divide the rotation matrix R obtained by stereo calibration into two parts r1 and r2, which are called aggregate rotation matrices. Rotate the left and right cameras by half to make the optical axes of the left and right cameras parallel.

[0164] Step 2: Rotate the image around the optical center to make the polar line horizontal and the pole at infinity. Select a vector with the principal point as the origin and the pole direction. This vector is the translation vector e1 between the optical centers. The next vector e2 must be orthogonal to e1 and is obtained by cross product of e1 and the principal optical axis. The third vector is required to be orthogonal to e, e, and is obtained by cross product. Finally, the pole is transformed to infinity. The matrix after row alignment is:

[0165]

[0166] Step 3: Combining the previous two steps, the final image is adjusted to be coplanar, and the correction matrix for parallel alignment is:

[0167] R l =R rect r l

[0168] R r =R rcet r r Formula 7

[0169] For each coordinate point of the corrected image, the coordinates of the original image corresponding to the coordinate point are obtained by calculating the correction matrix, camera parameter matrix, and inverse matrix of the distortion moment. The original image generated after inverse mapping has corresponding coordinates that are also floating-point. The pixel values ​​of these floating-point coordinates are obtained using bilinear interpolation.

[0170] Improving FPGA hardware to implement image binocular correction includes two steps: one is to look up the mapping table for each pixel coordinate; the other is to obtain the pixel value of the new coordinate through bilinear interpolation based on the table lookup result.

[0171] like Figure 3 This is the hardware implementation of image binocular correction. The image binocular correction module consists of: a lookup table, two pixel address generators, a scan line buffer address generator, a pixel value calculator, and the coordinates of the mapping table of the left and right image binocular correction completed by offline calculation are stored in two LUTX and two LUTY logical unit tables. The table is looked up to obtain a coordinate address, and the integer part of the address determines the coordinate address Pix_address_X of the four adjacent points ABCD pixels of the coordinate, and the pixel values ​​Pixel_X of the four points ABCD are found from the original image. The pixel shift register is completed by a memory with a write port and two read ports. Two pixel values ​​are read out in the first clock cycle, and two pixel values ​​are read out in the next clock cycle. Then these four pixel values ​​are input into the pixel value calculator, and the decimal part of the address obtained from the lookup table is used to calculate the pixel value of the final corrected image.

[0172] The present application adopts a method for generating a free combination of light spots and selects a 10×10 graded window for self-correlation verification and identification of the generated free combination of light spots. The result of the self-correlation verification proves that the identification effect is good. A specific implementation method of image filtering in an improved FPGA is given, and the image is collected in series. The image binocular correction method is used to make the image planes of the left and right cameras coplanar, and the image binocular correction is implemented in the improved FPGA.

[0173] 3. Improved FPGA-based binocular 3D scanner scale-cost adaptive transformation

[0174] This application proposes an improved binocular 3D scanning scale-cost adaptive transformation, which uses a planar Gaussian distribution N(μ, ∑) to weight the transformation grading window, and then samples the grading window 2×L times, with every two sampling points i, j forming a comparison point pair.<i,j> , the set of comparison point pairs is the result of the final scale cost adaptive transformation. The specific implementation process is as follows:

[0175] Step 1: Sampling according to the plane Gaussian distribution, generating 2×(N-1) sampling points, the expression of the plane Gaussian function, where ,

[0176]

[0177] Transform x and y into polar coordinates:

[0178]

[0179] After transforming the original variables x and y into new variables ρ and θ, the probability density function g(ρ,θ) of ρ and θ is obtained according to the probability density function f(x,y) of x and y:

[0180]

[0181] θ follows a uniform distribution on [-π,π], and ρ follows an exponential distribution with λ=-1. The final sampling method is:

[0182] Generate θ: θ = 2πr-π, r∈U[0,1];

[0183] Generate ρ: ρ = -lnr, r∈U[0,1];

[0184] The final result:

[0185] The above sampling obtains continuous values, discretizes the uniform sampling area, and obtains the coordinates of discrete points.

[0186] Step 2: According to the sampling method of step 1, 2×(N-1) sampling points are obtained, and then these sampling points are combined into a scale cost adaptive transformation template. These transformation templates are not unique, and only one sampling result needs to be selected as the subsequent transformation template;

[0187] Step 3: Perform scale cost adaptive transformation based on the obtained comparison template.

[0188] The improved scale-cost adaptive transform uses sampling to generate comparison point pairs, and the weights are still concentrated on the pixels in the central area. However, compared with the prior art, the reliance on the central pixel is reduced. The results obtained by the prior art are all comparisons between the neighborhood pixels and the central pixel. The improved method samples from the graded window, which not only uses the central pixel, but also the neighborhood pixels for comparison, making more use of the spatial structure information of the graded window.

[0189] 1. Hardware Implementation of Scale-Cost Adaptive Transform

[0190] It consists of two parts: a hierarchical moving grading window module and an element comparison module within the grading window.

[0191] Figure 4 It is the hardware structure of the hierarchical mobile grading window module, including a delay module and a storage delay module. The storage delay module realizes row cache. The input of the module is data_in[7:0], and the input is 8-bit pixel data. After caching and delay processing, r1, r1_d1, and rl_d2 represent the first row and first column elements, the first row and second column elements, and the first row and third column elements of the grading window respectively. r2, r2_d2, r3, r3_d1, and r3_d2 correspond to the elements of the second row and the third row respectively.

[0192] The element comparison module compares the paired elements in the classification window using a comparator according to the generated coordinate template, and then uses the Bus Builder module in DSP Builder to form an 8-bit data output. The hardware structure is as follows: Figure 5 shown.

[0193] The running time of the algorithm is related to the encoding length. The longer the length, the longer the running time. In the experiment, the image resolution is 384×512, SimLim and SimRim are the input ports for the left and right images, data_out and data_outl are the output ports, and the final output result is a 196608×1 string, which is converted into an image output after transformation. The transformation result is shown in Figure 6 shown.

[0194] In software, it takes 1.2985 seconds to perform an improved scale-cost adaptive transformation, while in hardware, under the premise of an amplitude frequency of 10M, it takes only 0.016 seconds. The results clearly show that the time is reduced, which shows the huge advantage of hardware processing.

[0195] 4. Binocular scanning 3D matching based on improved FPGA

[0196] After the image is scale-cost adaptively transformed, each pixel in the original image is represented by the transformed bit string, and the image matching is performed by evaluating the similarity of two bit strings.

[0197] (I) Evaluation of approximation calculation

[0198] Calculate the minimum cost to clip the source string into the target string. The clipping process operations include replacement, deletion, and insertion. The sum of the three operations is defined as the clipping difference distance of the two strings. The mathematical expression of the clipping difference distance is:

[0199]

[0200] Among them, i and j represent the length of string s1 and s2 respectively, lev [i-1,j] +1 inserts a letter into string s2, lev [i-1,j] +1 deletes a letter from string s1. When the i-th character of string s1 is different from the j-th character of string s2, f(i,j)=1, otherwise it is 0. That is, when calculating the clip difference distance, the source string and the target string are arranged into a table, and the value of each position in the table is calculated one by one according to the calculation formula of the clip difference distance; the values ​​of the first row and the first column do not need to be calculated and are automatically filled. To calculate the value at (i,j), three values ​​(i-1,j), (i,j-1), and (i-1,j-1) are required for comparison.

[0201] (II) Hardware Implementation of Clipping Different Distances

[0202] In the process of improving the FPGA for three-dimensional matching module, the corrected image and the left and right images after scale cost adaptive transformation are uniformly placed in the SDRAM memory. In order to realize the synchronous operation of the left and right images, the SDRAM adopts two read and two write port structures to realize the parallel storage of two images and facilitate reading and writing.

[0203] The single process of clip dissimilarity distance only calculates one-digit value in the clip dissimilarity distance table. If two strings of length m and n are compared, m×n operations are required (each operation includes sorting and comparison). When performing three-dimensional matching, if m×n operations are required each time to check the proximity of two matching points, it will consume a lot of time. Clipping can only be processed serially in software, but parallel operation can be achieved by converting to improved FPGA hardware. The calculation of each position depends on the values ​​of the left, top, and top-left positions, so the positions on the same diagonal are calculated simultaneously. The order of operations is as follows: Figure 7 .

[0204] Figure 7Where text is the source string, pattern is the target string, and matrixD is the generated Leveshtein distance matrix. Before generating the Leveshtein distance matrix, some preparation is required. The matrix for approximate comparison of two strings must be calculated first. This process can be obtained by relying on VerilogHDL code. The main idea is that if the characters at the corresponding positions of the strings are equal, then the bit is 0, and if they are not equal, it is 1.

[0205] After obtaining the comparison matrix, the next step is to calculate the clip dissimilarity distance matrix by writing Verilog HDL code. Each calculation simultaneously calculates the values ​​of all positions of the diagonal line. The number of elements of the diagonal line is not fixed and is implemented using combinational logic circuits.

[0206] Finally, by comparing the number of operations, we can determine which is faster in software or hardware. The clip dissimilarity distance implemented in hardware does not need to calculate each element one by one, but calculates all the elements of the upper angle at the same time. For two strings of length m and n, software serialization requires m×n operations, while in hardware, only m+n operations are required. Therefore, it can be concluded that completing the clip dissimilarity distance on the improved FPGA not only greatly speeds up the speed of three-dimensional matching, but also has efficient real-time performance.

[0207] (III) 3D matching search to generate depth map

[0208] After binocular image correction, the left and right images have been aligned, and after scale-cost adaptive transformation, the original image pixel values ​​have been transformed into bit strings representing their features. In the next step, if the clipping difference distance between two feature points does not reach the specified critical value, the two feature points are considered as matching points, and the left and right images are matched for feature point matching. Three-dimensional matching only requires comparison and search in the same row.

[0209] Assume an ideal camera stereo coordinate system, the left and right cameras have the same internal parameters, the same focal length, and the same principal point position. The image coordinates take the upper left corner of the image as the origin, the pixels in the same row of the image plane of the left and right images are aligned (on the same straight line), the optical axes of the left and right cameras are parallel, and the origin of the camera coordinates is the optical center of the left camera. The stereo measurement model can be simplified as Figure 8 .

[0210] The simplified stereo measurement model uses the triangle method to obtain the coordinates of point P:

[0211]

[0212] Where f is the focal length, (x l -x r) represents the parallax of two matching points. If the internal and external parameters of the camera are calibrated, the matching points of the left and right images are found (the same point in the physical world is the imaging point of the left and right images), and the x, y, z coordinates of point P in the real scene are calculated. According to formula 10, the difference in coordinates of the matching points (x l -x r ) is parallax.

[0213] 5. High-speed 3D surface reconstruction with binocular scanner

[0214] (I) Reconstructing 3D images based on binocular depth maps

[0215] Based on the binocular depth map, the three-dimensional coordinates of the point cloud are obtained according to the imaging relationship. The surface image of the three-dimensional object can only be described when the point cloud data is continuous, so it is necessary to construct a three-dimensional grid.

[0216] Let P = {p1, p2, ..., p n} is a point set. Triangulation connects each data point to form a number of triangles, resulting in a triangular mesh. The triangular mesh maps the topological connection relationship of each data point and represents the connection of the data points. It is used for a plane point set. The empty circular polar angle triangulation of these points makes the circumscribed circle of each triangle free of any unnecessary points. The steps of the empty circular polar angle triangulation network method include:

[0217] Step 1: Construct a non-optimized initial triangulation;

[0218] Step 2: Iterate the edge exchange optimization of the convex quadrilateral: this quadrilateral is composed of two colateral triangles;

[0219] There are two steps in surface reconstruction of empty circular polar angle triangulation. The first is to perform analog focal axis transformation. The target of the transformation is the sampling points, which exist on the surface of the original object. The second is to use the focal axis inverse transformation method. The result after the transformation is a piecewise approximate surface of the original surface. Point clouds of various forms are transmitted and converted into Thiessen depth polygons, and finally the empty circular polar angle triangulation and weight map are obtained.

[0220] (1) Analog focal axis conversion

[0221] Depending on the dimension, there are two situations, namely plane and elevation. In the plane, the focal axis is the set of centers of circles inside the boundary, while in three dimensions, it is the set of centers of the largest sphere. In the plane, the circle must pass through at least two points on the boundary, while in the elevation, the sphere must pass through at least three. But whether in the plane or elevation, inside the circle or the sphere, any points on the boundary should not be included. In the process of focal axis transformation, the coordinates of the focal axis and the radius are transformed collaboratively, so when the inverse transformation is performed, the original boundary is also represented and is roughly the same as the original.

[0222] (2) Thiessen depth polygon graph

[0223] Assume there is a point set P = {p1, p2, ..., p k}, p i ∈R n , the Thiessen depth polygon partition area is defined as:

[0224]

[0225] Where x is any point, d(x, p i ) is x and p i The distance between them, there is no overlap in the Thiessen depth polygon areas, there are only common edges, which are common faces, and the graph formed by these areas is the Thiessen depth polygon graph.

[0226] (3) Weight Graph

[0227] Based on the approximate description of the surface based on the circumscribed circle, assume that a sphere B with a center coordinate of c and a radius of ρ c，p , the weighted distance from spatial point x to c is calculated by formula 12:

[0228] d pow (x,E c，x ) = d 2 (c, x)-ρ 2 Formula 12

[0229] Where d represents the Euclidean distance. Based on the weight graph, the surface is divided into several blocks, and in the point set in each block, there must be one that is closest to the vertex.

[0230] (II) Empty circle polar angle triangulation based on VTK

[0231] VTK is used to perform the empty circle polar angle triangulation method to achieve surface reconstruction of point cloud data. The specific process is as follows: Fig. 9 .

[0232] Step 1: Use the fopen function to read the 3D point cloud data stored in the txt text;

[0233] Step 2: Get a discrete 3D point cloud, and connect the point cloud into a 3D surface using the empty circle polar angle triangulation method;

[0234] Step 3: Map the triangulated network to the graphics library and display the stereo reconstruction result.

[0235] 6. Improve the FPGA-based binocular 3D scanner system

[0236] (I) Improved FPGA-based binocular 3D scanner system architecture

[0237] According to the image processing pyramid architecture, the bottom layer algorithm is simple, but the data volume is large and the calculation speed is fast. The upper layer algorithm is complex, but the data volume is not large. The design of this system fully considers the advantages of hardware and software processing, and divides the overall image processing process into hardware and software. The hardware part performs image filtering, image binocular correction, non-parametric transformation, and three-dimensional matching. This part is completed by improving FPGA hardware, and the output result is the depth map of the measured object; the software part calculates the three-dimensional point cloud based on the depth map, and then reconstructs the three-dimensional surface from the three-dimensional point cloud. The algorithm of this part is relatively complex, but the algorithm data volume is small, which is more suitable for computer software implementation.

[0238] Software part: establish nios core to make SDRAM read and write control more convenient, read the image, process it in the module and then store it; for the hardware part, regard it as a custom IP, which includes image filtering, image binocular correction, non-parametric transformation, 3D matching algorithm and SDRAM interface and algorithm interface. The interface enables each module in the IP to exchange information correctly; the interface part is directly described in hardware language and then input into DSP Builder, and DSP Builder is used to complete modeling and improve development speed.

[0239] (II) Improvement of the hardware structure of the binocular 3D scanner based on FPGA

[0240] The system hardware consists of two CMOS sensors, a projector, a synchronous trigger circuit, a computer, and a SoCKit development board. Figure 1 shown.

[0241] The left and right cameras use USB interface cameras with CMOS line exposure. According to the size and distance of the measured object, a combination of grayscale spots with appropriate resolution is selected to project the image. The adjustment criterion is that the light spot on the surface of the measured object is clearly discernible. The placement height, angle, and image brightness of the projector are adjusted to make the captured images of the left and right cameras clear and the grayscale gradient changes obvious.

[0242] The synchronous trigger circuit controls the left and right cameras to sample at the same time, ensuring that the left and right cameras capture the image of the same object at the same time. The trigger circuit diagram is as follows Fig.10 As shown:

[0243] The camera trigger level is 5V, while the serial port output level is only 3.3V, so a logic AND gate chip is used to achieve level conversion. The AND gate chip is powered by 5V voltage, the input pulse voltage is 3.3V, and the output pulse voltage is 5V. The 5V pulse voltage is used to control the camera's acquisition through an optocoupler isolation chip.

[0244] The computer is the system control center. During the acquisition phase, it controls the synchronization trigger to generate pulse signals, collects images from the left and right cameras through the USB interface, and saves them on the computer. During the stereo reconstruction phase, the computer selects the original image data for stereo reconstruction through the user interface program designed by Qt, and transmits it to the SoCKit development board via Ethernet for image processing. The depth map processed by the SoCKit development board is transmitted back to the computer through the Ethernet interface. The computer reconstructs the depth map into a three-dimensional surface and displays it to the user through a graded window.

[0245] VII. Experiment and Results Analysis

[0246] In order to test the performance of the system, a book surface and a human hand surface are tested respectively. The book surface is a typical plane and is the simplest test object. The surface of a human hand is a curved surface with obvious separations between fingers. The curved surface is gentle and suitable for testing.

[0247] Experiment 1, plane detection.

[0248] 1) Image acquisition. First, project a combined grayscale light spot onto the surface of the book, and adjust the distance between the projector and the object to be measured so that the image size of the object to be measured in the image is appropriate. Adjust the code element size of the projected image so that the code elements on the surface of the object are clearly visible. Turn on the camera to collect images, adjust the camera position so that the object to be measured can be clearly seen in the lens, adjust the light transmittance of the lens aperture to ensure that the image has sufficient brightness, and try to ensure that the light transmittance of the left and right cameras is consistent to reduce the brightness difference of the left and right images. The collected image is as follows: Fig.11 shown.

[0249] 2) Image processing. The acquired image is first subjected to image filtering and serial subtraction operations, and then the noise is removed and the size of the original image is reduced. The next step is to perform scale-cost adaptive transformation. The scale-cost adaptive transformation uses a 3×3 graded window to process the image.

[0250] 3) Reconstruct the surface. After the scale cost adaptive transformation, the bit string data is obtained. The clip difference distance is used to evaluate the similarity of the bit string, and the three-dimensional matching points are obtained to obtain the depth map. The depth map is processed by empty circular polar angle triangulation to generate a three-dimensional surface. In order to better display the measured plane, the background image except the book surface is manually segmented before the three-dimensional surface reconstruction. The final stereoscopic reconstructed surface is as follows Fig.12 .

[0251] From left to right are the front of the reconstructed surface, three frames of images rotated 45° around the focal axis, and three frames of images rotated 80° around the focal axis. From the reconstruction results, it can be seen that the generated surface is basically consistent with the real book surface, the surface is smooth, and there is no discrete point cloud around the edge. In general, the reconstructed surface is basically consistent with the real book.

[0252] Experiment 2, surface detection.

[0253] 1) Image acquisition. As in Experiment 1, adjust the distance between the projector and the object to be measured, adjust the size of the projected image code element, adjust the camera position, adjust the light transmittance of the lens aperture, and collect the left and right original images of the human hand.

[0254] 2) Process the image. The acquired image must first undergo a certain amount of preprocessing, namely image filtering and serial subtraction, to remove noise and reduce the size of the original image. Then comes the scale cost adaptive transformation. The specific method of scale cost adaptive transformation is to use a 3×3 graded window to process the image. The processing results are as follows: Fig.13 .

[0255] 3) Reconstruct the surface. After the scale cost adaptive transformation, the bit string data is obtained, and the clip difference distance is used to evaluate the approximation of the bit string. The three-dimensional matching points are obtained, and the depth map can be obtained. The depth map is triangulated by the empty circle polar angle to form a three-dimensional surface. In order to better display the measured plane, the background image except the surface of the human hand is manually segmented before the three-dimensional surface reconstruction.

[0256] Fig.14 It is a reconstructed surface image obtained by rotating the human hand at different angles. From the image, we can see that the structure of the entire hand is reconstructed. The reconstructed image conforms to the surface structure of the human hand and the point cloud is dense. The transition between the surfaces is smooth and there is no obvious stratification.

Claims

1. Binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system, characterized by: The first is the three-dimensional matching reconstruction scheme of binocular scanner; the second is the image preprocessing of binocular 3D scanner based on improved FPGA, including: projecting three-dimensional images, image filtering, image cascade subtraction acquisition, and image binocular correction; the third is the scale cost adaptive transformation of binocular 3D scanner based on improved FPGA; the fourth is the three-dimensional matching of binocular scanning based on improved FPGA, including: approximation calculation evaluation, hardware implementation of clipping different distances, and three-dimensional matching search to generate depth map; the fifth is the high-speed 3D surface reconstruction of binocular scanner, including: reconstruction of three-dimensional images based on binocular depth map, and empty circle polar angle triangulation based on VTK; the sixth is the construction of binocular 3D scanner system based on improved FPGA; This application proposes an optimization scheme for binocular 3D scanners accelerated by improved FPGA. By using improved FPGA hardware parallel processing, the image processing algorithm with large underlying data volume is accelerated in hardware. Continuous grayscale free combination of light spots is used as the structured light image for three-dimensional measurement. The image filtering processing based on the improved FPGA is designed and implemented. After the image is filtered, the image is subjected to the image serial subtraction acquisition method to reduce the data processing volume. The image binocular correction method is used to correct the polar lines of the image to be horizontal. When searching for the matching point of the left image corresponding to the right image, it is only necessary to search for the same row of pixels, and finally the binocular correction is implemented on the improved FPGA. Before three-dimensional matching, the image is first subjected to a scale-cost adaptive transformation to accurately detect the edge and corner features of the image. A scale-cost adaptive transformation based on Gaussian distribution sampling is used to reduce the dependence on the central pixel and is implemented on an improved FPGA. After the scale-cost adaptive transformation, each pixel of the original image is represented by a bit string. The present application designs an implementation method of the clipping dissimilarity distance in the improved FPGA, and evaluates the similarity between two bit strings by calculating the clipping dissimilarity distance to determine whether the two pixels are matching points. If the clipping dissimilarity distance of the two matching points is less than a critical value, it is determined that the match is successful. The final result is the disparity of the two matching points. A three-dimensional point cloud is generated based on the disparity calculation. Based on the generated discrete point cloud, a triangular mesh is constructed using an empty circular polar angle triangulation method to describe the surface image of the three-dimensional object. Finally, a 3D scanning system platform was built based on the above method. The image acquisition was controlled by a computer. The original image was then downloaded to the improved FPGA and the three-dimensional surface was reconstructed according to the depth map sent back by the improved FPGA. The improved FPGA implemented a coprocessor solution based on the SoCKit development board, and the EDA tools of DSP Builder and Qsys were used to achieve real-time and high-precision 3D scanning.

2. According to claim 1, the binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system is characterized in that: Projected 3D image: The goal is to generate a dense point cloud with only one measurement, using combined grayscale spot projection image encoding; 1. Free combination of light spots By using combined grayscale spot projection image encoding, a dense point cloud can be obtained in a single measurement. The projection generation method is: M=Rand(m,n)*A Formula 1 Where M is the generated combined gray matrix, m = h / Δx, n = w / Δy, A is the image gray level, ranging from 1 to 256, and Rand generates a combined number between 0 and 1, excluding 1; 2. Projection area correlation During the 3D matching process, a certain feature point in the left image has a unique corresponding point in the right image. With the help of structured light, this corresponding point can be quickly found. The correlation factor is used to verify the uniqueness of the combined gray light spot projection mark. The correlation factor is as follows: In the above formula, 0 ≤ i < Ma + Mb - 1, 0 ≤ j < Na + Nb - 1, (Ma, Mb) is the size of A, and (Na, Nb) is the size of B.

3. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: Image filtering: Image filtering is implemented using an improved FPGA. A hierarchical window is used to perform block operations on the image. The hierarchical window is moved until the entire image has been subjected to block operations, realizing the processing of the entire image. This process is expressed as: Q[x, y] = f(I[x, y],....I[x + Δx, y + Δy]), (Δx, Δy) ∈ W Equation 3 w is a hierarchical window centered on I[x, y]. Each time the hierarchical window W scans a position in the input image, an output pixel is generated at this position, and the value of this output pixel is obtained through Equation 3. f is a filtering function; The hierarchical moving hierarchical window filtering has three stages: input line buffer, pixel point operation, and output result. Among them, the input line buffer adopts FIFO hierarchical window filtering. The input image from the memory is continuously input into the FIFO, and pixels are continuously read from the other end of the FIFO and sent to the arithmetic unit for processing. The input and output speeds are matched, and no additional memory read and write operations are performed in the middle; Image serial subtraction acquisition: First, the first serial subtraction acquisition is performed on the original image to generate a downsampled image. Then, the second serial subtraction acquisition is performed on the just-generated downsampled image. This process is repeated in a cycle to form a set of image collections. If the obtained images are stacked in sequence, a pyramid shape is formed; The pyramid model is generally divided into two parts: Step one is that the image first passes through a low-pass filter to smooth the image. Step two is to sample the image processed in the previous step. The final result is an image processed to a different size from the original size; Each level of the image in the sequence is obtained by performing interlaced and interleaved serial subtraction acquisition on the previous level of the image after low-pass filtering, as follows: Among them, G k (i, j) is located at the kth layer of the pyramid image. The bottom layer of the pyramid is G0, which is also the original image that needs to be collected in series. At the same time, w(m, n) = h(m)h(n) is a 5×5 low-pass graded window function, h is a Gaussian density distribution function, and the graded window function w(m, n) is expressed as Formula 5: After multi-layer serial subtraction acquisition processing, an image pyramid is formed.

4. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: Image binocular correction: Adjust the left and right camera image planes to be coplanar and align the image plane rows. At this time, the position of the pole is at infinity, and the corresponding epipolar lines of the feature points are the same horizontal line. Ensure that the two images are strictly row-corresponding. The feature point search only needs to be performed on the same row of pixels in the other camera. According to the internal and external parameter matrices of the imaging coordinates, the binocular images are corrected to be coplanar and row-aligned. The specific steps are as follows: Step one: Divide the rotation matrix R obtained from the stereo calibration into two parts r1 and r2, which are called aggregation rotation matrices. Rotate both the left and right cameras by half to make the optical axes of the left and right cameras parallel; Step 2: Rotate the image around the optical center to make the polar line horizontal and the pole at infinity. Select a vector with the principal point as the origin and the pole direction. This vector is the translation vector e1 between the optical centers. The next vector e2 must be orthogonal to e1 and is obtained by cross product of e1 and the principal optical axis. The third vector is required to be orthogonal to e, e, and is obtained by cross product. Finally, the pole is transformed to infinity. The matrix after row alignment is: Step 3: Combining the previous two steps, the final image is adjusted to be coplanar, and the correction matrix for parallel alignment is: R l =R real r l R r =R rect r r Formula 7 For each coordinate point of the corrected image, the coordinates of the original image corresponding to the coordinate point are obtained by calculating the correction matrix, camera parameter matrix, and inverse matrix of the distortion moment. The original image generated after inverse mapping has corresponding coordinates of floating point type. The pixel values ​​of these floating point coordinates are obtained by using bilinear interpolation. Improving FPGA hardware to implement image binocular correction includes two steps: first, each pixel coordinate looks up the mapping table; second, based on the table lookup results, the pixel value of the new coordinate is obtained through bilinear interpolation processing; The image binocular correction module consists of: a lookup table, two pixel address generators, a scan line buffer address generator, a pixel value calculator, and the mapping table coordinates of the left and right image binocular correction completed by offline calculation are stored in two LUTX and two LUTY logic unit tables. A coordinate address is obtained by looking up the table, and the integer part of the address determines the coordinate address Pix_address_X of the four adjacent points ABCD pixels of the coordinate, and the pixel values ​​Pixel_X of the four points ABCD are found from the original image. The pixel shift register is completed by a memory with a write port and two read ports. Two pixel values ​​are read out in the first clock cycle, and two pixel values ​​are read out in the next clock cycle. Then these four pixel values ​​are input into the pixel value calculator, and the decimal part of the address obtained from the lookup table is used to calculate the pixel value of the final corrected image.

5. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: Improved FPGA binocular 3D scanner scale cost adaptive transformation: using planar Gaussian distribution N (μ, ∑), weighting the transformation grading window, and then sampling the grading window 2×L times, every two sampling points i, j form a comparison point pair<i,j> , the set of comparison point pairs is the result of the final scale cost adaptive transformation. The specific implementation process is as follows: Step 1: Sampling according to the plane Gaussian distribution, generating 2×(N-1) sampling points, the expression of the plane Gaussian function, where σ=1, Transform x and y into polar coordinates: After transforming the original variables x and y into new variables ρ and θ, the probability density function g(ρ,θ) of ρ and θ is obtained according to the probability density function f(x,y) of x and y: θ follows a uniform distribution on [-π,π], and ρ follows an exponential distribution with λ=-1. The final sampling method is: Generate θ: θ = 2πr-π, r∈U[0,1]; Generate ρ: ρ = -lnr, r∈U[0,1]; The final result: The above sampling obtains continuous values, discretizes the uniform sampling area, and obtains the coordinates of discrete points; Step 2: According to the sampling method of step 1, 2×(N-1) sampling points are obtained, and then these sampling points are combined into a scale cost adaptive transformation template. These transformation templates are not unique, and only one sampling result needs to be selected as the subsequent transformation template; Step 3: Perform scale cost adaptive transformation based on the obtained comparison template; The hardware implementation of scale-cost adaptive transformation consists of two parts: a hierarchical moving grading window module and an element comparison module within the grading window; The hierarchical mobile grading window module includes a delay module and a storage delay module. The storage delay module realizes row cache. The input of the module is data_in[7:0], which inputs 8-bit pixel data. After cache and delay processing, r1, r1_d1, and rl_d2 respectively represent the first row and first column elements, the first row and second column elements, and the first row and third column elements of the grading window. r2, r2_d2, r3, r3_d1, and r3_d2 correspond to the elements of the second row and the third row respectively. The element comparison module compares the paired elements in the classification window using a comparator according to the generated coordinate template, and then uses the Bus Builder module in DSP Builder to compose the comparison results into 8-bit data output.

6. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: Approximation calculation evaluation: Calculate the minimum cost to edit the source string into the target string. The editing process operations include replacement, deletion, and insertion. The sum of the three operations is defined as the editing difference distance of the two strings. The mathematical expression of the editing difference distance is: Among them, i and j represent the length of string s1 and s2 respectively, lev [i-1,j] +1 inserts a letter into string s2, lev [i-1,j] +1 deletes a letter from string s1. When the i-th character of string s1 is different from the j-th character of string s2, f(i,j)=1, otherwise it is 0. That is, when calculating the clip difference distance, the source string and the target string are arranged into a table, and the value of each position in the table is calculated one by one according to the calculation formula of the clip difference distance; the values ​​of the first row and the first column do not need to be calculated and are automatically filled. To calculate the value at (i,j), three values ​​(i-1,j), (i,j-1), and (i-1,j-1) are required for comparison.

7. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: Hardware implementation of clipping different distances: In the process of improving the FPGA for 3D matching module, the corrected image and the left and right images after scale cost adaptive transformation are uniformly placed in the SDRAM memory. To achieve synchronous operation of the left and right images, the SDRAM adopts a two-read and two-write port structure to achieve parallel storage of two images and facilitate reading and writing; Converting to improved FPGA hardware enables parallel operation. The calculation of each position depends on the values ​​of the left, top, and top-left positions. Positions on the same diagonal line are calculated simultaneously. After obtaining the comparison matrix, the next step is to calculate the clipping difference distance matrix by writing Verilog HDL code. Each calculation simultaneously calculates the values ​​of all positions of the diagonal line. The number of elements of the diagonal line is not fixed and is implemented using combinational logic circuits. Finally, the clipping dissimilarity distance is determined by comparing the number of operations in software and hardware. The clipping dissimilarity distance implemented in hardware does not need to calculate each element one by one, but calculates all the elements in the upper angle at the same time. For two strings of length m and n respectively, software serialization requires m×n operations, while in hardware, only m+n operations are required. Completing the clipping dissimilarity distance on the improved FPGA not only greatly speeds up the speed of three-dimensional matching, but also has efficient real-time performance.

8. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: 3D matching search generates depth map: If the clipping difference distance of two feature points does not reach the specified critical value, then the two feature points are considered as matching points, and the left and right images are matched for feature points. 3D matching only requires comparison and search in the same row; Assume an ideal camera stereo coordinate system, the left and right cameras have the same internal parameters, the same focal length, and the same principal point position; the image coordinates take the upper left corner of the image as the origin, the pixels in the same row of the image plane of the left and right images are aligned, the optical axes of the left and right cameras are parallel, and the origin of the camera coordinates is the optical center of the left camera; The simplified stereo measurement model uses the triangle method to obtain the coordinates of point P: Where f is the focal length, (x l -x r ) represents the parallax of the two matching points. If the internal and external parameters of the camera are calibrated, the matching points of the left and right images are found (the same point in the physical world is the imaging point of the left and right images), and the x, y, and z coordinates of point P in the real scene are calculated, according to formula 10, the difference between the coordinates of the matching points (x l -x r ) is parallax.

9. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: Binocular scanner high-speed 3D surface reconstruction: (I) Reconstructing 3D images based on binocular depth maps Construct a three-dimensional grid, let P = {p1, p2, ..., p n } is a point set. Triangulation connects each data point to form a number of triangles, resulting in a triangular mesh. The triangular mesh maps the topological connection relationship of each data point and represents the connection of the data points. It is used for a plane point set. The empty circular polar angle triangulation of these points makes the circumscribed circle of each triangle free of any unnecessary points. The steps of the empty circular polar angle triangulation network method include: Step 1: Construct a non-optimized initial triangulation; Step 2: Iterate the edge exchange optimization of the convex quadrilateral: this quadrilateral is composed of two colateral triangles; There are two steps in the reconstruction of the empty circular polar angle triangulation of the surface. The first is to perform analog focal axis transformation. The target of the transformation is the sampling points, which exist on the surface of the original object. The second is to use the inverse transformation method of the focal axis. The result after the transformation is the segmented approximate surface of the original surface. The point clouds of various forms are transmitted and converted into Thiessen depth polygons. Finally, the empty circular polar angle triangulation and weight map are obtained. (1) Analog focal axis conversion According to the different dimensions, there are two situations, namely, plane and elevation. In the plane, the focal axis is the set of the centers of the circles inside the boundary, while in three dimensions, it is the set of the centers of the largest spheres. In the plane, the circle must pass through at least two points on the boundary, while in the elevation, the sphere must pass through at least three points. However, no matter in the plane or elevation, inside the circle or the sphere, any point on the boundary should not be included. In the process of focal axis transformation, the coordinates of the focal axis and the radius are transformed together, so when the inverse transformation is performed, the original boundary is also represented and is the same as the original. (2) Thiessen depth polygon graph Assume there is a point set P = {p1, p2, ..., p k }, p l ∈R n , the Thiessen depth polygon partition area is defined as: Where x is any point, d(x, p i ) is x and p i The distance between them, the Thiessen depth polygon areas do not overlap, there is only one common edge, the two are common faces, and the graph formed by these areas is the Thiessen depth polygon graph; (3) Weight Graph Based on the approximate description of the surface based on the circumscribed circle, assume that a sphere B with a center coordinate of c and a radius of ρ c，ρ , the weighted distance from spatial point x to c is calculated by formula 12: d pow (x,B c，x ) = d 2 (c, x)-ρ 2 Formula 12 Where d represents the Euclidean distance. Based on the weight graph, the surface is divided into several blocks, and in the point set in each block, there must be one that is closest to the vertex. (II) Empty circle polar angle triangulation based on VTK VTK is used to perform the empty circle polar angle triangulation method to achieve surface reconstruction of point cloud data. The specific process is as follows: Step 1: Use the fopen function to read the 3D point cloud data stored in the txt text; Step 2: Get a discrete 3D point cloud, and connect the point cloud into a 3D surface using the empty circle polar angle triangulation method; Step 3: Map the triangulated network to the graphics library and display the stereo reconstruction result.

10. The binocular scanner high-speed three-dimensional precise matching and stereo reconstruction system according to claim 1, characterized in that: Improved FPGA binocular 3D scanner system construction: (I) Improved FPGA-based binocular 3D scanner system architecture The overall image processing flow is divided into two parts: hardware and software. The hardware part performs image filtering, image binocular correction, non-parametric transformation, and 3D matching. This part is completed by improving FPGA hardware, and the output result is the depth map of the object being measured. The software part calculates the 3D point cloud based on the depth map, and then reconstructs the 3D surface from the 3D point cloud. The algorithm of this part is complex, but the algorithm data volume is small, and it is implemented in computer software. Software part: Establish the nios core to make SDRAM read and write control more convenient, read the image, process it in the module and then store it; for the hardware part, regard it as a custom IP, which includes image filtering, image binocular correction, non-parametric transformation, 3D matching algorithm and SDRAM interface and algorithm interface. The interface enables each module in the IP to exchange information correctly; the interface part is directly described in hardware language, and then input into DSP Builder, and DSP Builder is used to complete the modeling to improve the development speed; (II) Improvement of the hardware structure of the binocular 3D scanner based on FPGA The system hardware consists of two CMOS sensors on the left and right, a projector, a synchronous trigger circuit, a computer, and a SoCKit development board; The left and right cameras use USB interface cameras with CMOS line exposure. According to the size and distance of the measured object, a combination of grayscale light spots with appropriate resolution is selected to project the image. The adjustment criterion is that the light spot on the surface of the measured object is clearly identifiable. The placement height, angle and image brightness of the projector are adjusted to make the collected images of the left and right cameras clear and the grayscale gradient changes obvious. The synchronous trigger circuit controls the sampling of the left and right cameras at the same time to ensure that the left and right cameras capture the image of the same object at the same time; the computer is the system control center. During the acquisition phase, it controls the synchronous trigger to generate a pulse signal, collects the images of the left and right cameras through the USB interface, and saves them in the computer; during the stereo reconstruction phase, the computer selects the original image data for stereo reconstruction through the user interface program designed by Qt, and transmits it to the SoCKit development board through Ethernet for image processing; the depth map processed by the SoCKit development board is transmitted back to the computer through the Ethernet interface, and the computer reconstructs the depth map into a three-dimensional surface and displays it to the user through a graded window.

Images