Three-dimensional reconstruction method, apparatus, and system

Abstract

This application discloses a 3D reconstruction method, apparatus, and system, including the steps of acquiring a first image and a second image, where the first image and the second image are acquired by collecting predetermined code element images projected onto a surface of an object by different image acquisition devices, the predetermined code element images including a plurality of target code elements randomly distributed according to a predetermined direction, the target code elements being line segment stripes, acquiring first target pixel points of each code element in the first image and determining matching pixel points in the second image for each first target pixel point of each code element in the first image, and determining three-dimensional coordinates of the first target pixel points based on at least the pixel coordinates of the predetermined first target pixel points in the first image and the pixel coordinates of the matching pixel points in the second image to complete the 3D reconstruction.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to a Chinese patent application filed with the China Patent Office on August 10, 2022, bearing application number 202210956350.X and entitled "3D reconstruction method, apparatus and system," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of three-dimensional reconstruction, and more particularly to a three-dimensional reconstruction method, apparatus and system. [Background technology]

[0003] Structured light 3D reconstruction technology projects an optically coded pattern onto the surface of an object to be measured and reconstructs 3D data of the object surface using the collected deformation pattern. It boasts high efficiency and interference resistance, making it widely applicable to various 3D reconstruction scenes. The key issue in structured light 3D reconstruction technology is matching corresponding pixel points. Different matching strategies depend on different coding methods. Depending on the coding method of the projected pattern, structured light technology can be divided into temporal coding and spatial coding. Temporal coding requires projecting multiple frames of patterns onto the measurement scene in time sequence, which typically requires the object to be measured and the projector to remain relatively stationary. This prevents high-frame-rate scanning and limits its application to static scanning scenes. In spatial coding, three-dimensional reconstruction can usually be completed by projecting a single pattern onto the scene to be measured. Currently, related technologies use a circular code element coding method to decode using the relative displacement relationship of neighboring code elements, while another method obtains three-dimensional data by matching between image blocks. The former method has a small code element coding capacity and can only reconstruct sparse code element points, resulting in little single-frame data and low scanning efficiency. The latter method uses image blocks for matching and has low accuracy. To date, no effective solution has been proposed to the above problem. Summary of the Invention [Problem to be solved by the invention]

[0004] The embodiments of the present application provide a 3D reconstruction method, apparatus, and system to at least solve the technical problem of low accuracy in matching code elements based on code element feature points due to blurred images. [Means for solving the problem]

[0005] According to one aspect of an embodiment of the present application, there is provided a three-dimensional reconstruction method, the method including the steps of: acquiring a first image and a second image, the first image and the second image being acquired by collecting predetermined code element images projected onto a surface of a measured object by different image acquisition devices, the predetermined code element images including a plurality of target code elements randomly distributed according to a predetermined direction, the target code elements being line segment stripes; acquiring first target pixel points of each code element in the first image and determining matching pixel points in the second image of the first target pixel points of each code element in the first image; and determining three-dimensional coordinates of the first target pixel points based on at least the pixel coordinates of the predetermined first target pixel points in the first image and the pixel coordinates of the matching pixel points in the second image, thereby completing the three-dimensional reconstruction.

[0006] In some embodiments of the present application, the step of determining a matching pixel point in the second image for a first target pixel point of each code element in the first image includes the steps of: determining pixel coordinates and gradation values of the first target pixel point; determining a first region of a predetermined area in the first image and setting the center point of the first region as the first target pixel point; determining a second region of a predetermined area from the second image, wherein the ordinate coordinate of the center of the second region in the second image is the same as the ordinate coordinate of the center of the first region in the first image; and determining a matching pixel point from the second region that is matched with the first target pixel point.

[0007] In some embodiments of the present application, the step of determining a matching pixel point from the second region to be matched with the first target pixel point includes the steps of: determining a correlation coefficient of each pixel point in the second region one by one, where the correlation coefficient is for characterizing the correlation between the pixel point in the second region and the first target pixel point; and determining the pixel point in the second region with the largest correlation coefficient as the matching pixel point.

[0008] In some embodiments of the present application, the step of determining the correlation coefficient of each pixel point in the second region one by one includes the steps of determining the gradation average value of all pixel points in the first region as a first gradation average value and determining the gradation average value of all pixel points in the second region as a second gradation average value, and determining the correlation coefficient between each pixel point in the second region and the first target pixel point based on the difference between the gradation value of the first target pixel point in the first region and the first gradation average value and the difference between the gradation value of each pixel point in the second region and the second gradation average value.

[0009] In some embodiments of the present application, the step of determining the pixel point in the second region with the largest correlation coefficient as the matching pixel point includes the steps of: determining the pixel point in the second region with the largest correlation coefficient as a candidate matching point; if the candidate matching point overlaps with a second target pixel point in the second image, determining the candidate matching point as the matching pixel point; if the candidate matching point does not overlap with the second target pixel point in the second image, determining a second target pixel point within a predetermined range around the candidate matching point as the matching pixel point.

[0010] In some embodiments of the present application, the predetermined code element image includes a plurality of target code elements randomly distributed according to a predetermined direction, and the step of the target code elements being line stripes includes the steps of: determining a target area in which the target code element is located based on the length of the target code element, the width of the target code element, the pitch between the target code elements, and the pixel coordinate of the center pixel point of the target code element, wherein the pixel coordinate of the center pixel point of the target code element is randomly generated within the area in which the code element image is located; traversing all pixel points within the target area, and generating a target code element in the target area if no target code element is present in the target area, wherein the target code element includes at least one line segment of a predetermined length and two end points corresponding to the line segment of the predetermined length; and generating the target code elements in all target areas within the code element image area.

[0011] In some embodiments of the present application, the method further includes the steps of: determining a first neighborhood code element set of any one code element in the first image and a plurality of second neighborhood code element sets of a plurality of candidate code elements in the second image; determining the number of matches between neighborhood code elements in the plurality of second neighborhood code element sets and neighborhood code elements in the first neighborhood code element set; and determining the second neighborhood code element set with the largest number of matches among the plurality of second neighborhood code element sets as a target second neighborhood code element set; and determining the candidate code elements corresponding to the target second neighborhood code element set as a target code element.

[0012] In some embodiments of the present application, the method further includes determining a plurality of first target pixel points in the first image and a plurality of second target pixel points in the second image, and matching the plurality of first target pixel points one-to-one with the plurality of second target pixel points in the second image.

[0013] According to another aspect of an embodiment of the present application, there is further provided a three-dimensional reconstruction apparatus, the apparatus including: an acquisition module for acquiring a first image and a second image, the first image and the second image being acquired by collecting predetermined code element images projected onto a surface of a measured object by different image acquisition devices, the predetermined code element images including a plurality of target code elements randomly distributed according to a predetermined direction, the target code elements being line segment stripes; a matching module for acquiring first target pixel points of each code element in the first image and determining matching pixel points in the second image of the first target pixel points of each code element in the first image; and a reconstruction module for determining three-dimensional coordinates of the first target pixel points based at least on pixel coordinates of the predetermined first target pixel points in the first image and pixel coordinates of the matching pixel points in the second image to complete the three-dimensional reconstruction.

[0014] According to another aspect of an embodiment of the present application, there is further provided a three-dimensional reconstruction system applicable to a three-dimensional reconstruction method, the system including at least two image acquisition devices, a projection device, and a first processor, wherein the projection device is for projecting predetermined code element images onto a surface of a measured object, the at least two image acquisition modules are for acquiring predetermined code element images from the surface of the measured object to obtain a first image and a second image, the first processor is for acquiring first target pixel points of each code element in the first image and determining matching pixel points in the second image of the first target pixel points of each code element in the first image, and further for determining three-dimensional coordinates of the first target pixel points based at least on pixel coordinates in the first image of the predetermined first target pixel points and pixel coordinates in the second image of the matching pixel points to complete the three-dimensional reconstruction.

[0015] According to another aspect of an embodiment of the present application, there is further provided a non-volatile storage medium containing a stored program, wherein when the program is executed, it controls an apparatus in which the non-volatile storage medium is located to perform the above-mentioned three-dimensional reconstruction method.

[0016] According to another aspect of an embodiment of the present application, there is further provided an electronic device, the electronic device including a memory and a processor, the processor configured to execute a program, wherein when the program is executed, the above-mentioned three-dimensional reconstruction method is performed. [Effects of the Invention]

[0017] In an embodiment of the present application, a first image and a second image are acquired, the first image and the second image being acquired by respectively collecting predetermined code element images projected onto the surface of the object to be measured using different image acquisition devices, the predetermined code element images including a plurality of target code elements randomly distributed according to a predetermined direction, the target code elements being line segment stripes; a first target pixel point of each code element in the first image is acquired, and a matching pixel point in the second image for the first target pixel point of each code element in the first image is determined; and the 3D coordinates of the first target pixel point are determined based at least on the pixel coordinates of the predetermined first target pixel point in the first image and the pixel coordinates of the matching pixel point in the second image to complete the 3D reconstruction. This achieves the objective of replacing the coding method using code element feature points with the coding method for determining the matching pixel point in the second image for the first target pixel point of each code element in the first image, thereby achieving the technical effect of improving the matching accuracy of code elements and solving the technical problem of low accuracy of matching code elements based on code element feature points due to image blur. [Brief explanation of the drawings]

[0018] The drawings described herein are intended to provide a further understanding of the present application, constitute a part of the present application, and the illustrative embodiments and descriptions thereof are intended to provide an understanding of the present application and are not intended to unduly limit the present application. [Figure 1] FIG. 1 is a block diagram of a hardware structure of a computer terminal (or mobile device) that uses a three-dimensional reconstruction method according to an embodiment of the present application. [Figure 2] 1 is a schematic diagram of a three-dimensional reconstruction method according to the present application; [Figure 3] 1 is a schematic diagram of a preferred coding pattern according to an embodiment of the present application; [Figure 4] 1 is a schematic diagram of five preferred code element shapes according to embodiments of the present application. [Figure 5] 1 is a preferred 3D reconstruction system according to an embodiment of the present application; [Figure 6] 1 is a preferred 3D reconstruction apparatus according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0019] In order to help those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described below clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, not all of the embodiments, and all other embodiments obtained by those skilled in the art based on the embodiments of the present application without any creative work shall fall within the protection scope of the present application.

[0020] It should be noted that the terms "first," "second," etc. in the specification and claims of this application and in the drawings are intended to distinguish between similar objects and are not necessarily intended to describe a particular order or chronological order. It should be noted that data used in this manner may be interchanged where appropriate, so that the embodiments of this application described herein may be performed in an order other than that illustrated or described herein. Furthermore, the terms "comprise" and "have" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to the explicitly recited steps or units, but may include other steps or units not explicitly recited or inherent in the process, method, product, or apparatus.

[0021] Embodiments of the present application further provide exemplary model training methods, in which the steps illustrated in the flowcharts of the figures may be performed by a computer system as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than shown here.

[0022] The method embodiments provided by the present application may be implemented in a mobile terminal, a computer terminal, a cloud server, or a similar computing device. FIG. 1 shows a block diagram of the hardware structure of a computer terminal (or a mobile device) for implementing a 3D reconstruction method. As shown in FIG. 1, a computer terminal 10 (or a mobile device 10) may include one or more processors 102 (denoted by 102a, 102b, . . . , 102n in the figure) (the processors 102 may include, but are not limited to, processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 104 for storing data, and a transmission module 106 for communication functions. The computer terminal 10 (or a mobile device 10) may further include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will appreciate that the structure shown in FIG. 1 is merely exemplary and does not limit the structure of the electronic device. For example, computer terminal 10 may include more or fewer components than those shown in FIG. 1, or may include a different configuration than that shown in FIG.

[0023] It should be noted that the one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits." The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination. The data processing circuits may also be a single, independent processing module, or may be incorporated in whole or in part into any one of the other elements within the computer terminal 10 (or mobile device). In accordance with embodiments of the present application, the data processing circuits may act as processors to control (e.g., select variable resistance terminal paths connected to an interface).

[0024] The memory 104 is for storing software programs and modules of application software, such as program instructions / data storage devices corresponding to the 3D reconstruction method in the embodiment of the present application. The processor 102 executes the software programs and modules stored in the memory 104 to perform various functional applications and data processing, i.e., to realize the 3D reconstruction method of the application program. The memory 104 may include high-speed random access memory and may further include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0025] The transmission module 106 is for receiving or transmitting data via a network. A specific example of the network may include a wireless network provided by the carrier of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which is connected to other network devices via a base station and can thereby communicate with the Internet. In one example, the transmission module 106 may be a radio frequency (RF) module, which is for communicating with the Internet wirelessly.

[0026] The display may be, for example, a touch-sensitive liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0027] According to an embodiment of the present application, an embodiment of a three-dimensional reconstruction method is provided, and it is noted that the steps shown in the flowcharts of the drawings can be executed in a computer system as a set of computer-executable instructions, and that although a logical order is shown in the flowcharts, in some cases the steps shown or described may be executed in a different order than here. FIG. 2 is a flowchart of a three-dimensional reconstruction method according to an embodiment of the present application. As shown in FIG. 2, the method includes the following steps S202 to S206.

[0028] In step S202, a first image and a second image are acquired, and the first image and the second image are acquired by respectively collecting predetermined code element images projected onto the surface of the object to be measured by different image acquisition devices, and the predetermined code element images include a plurality of target code elements randomly distributed according to a predetermined direction, and the target code elements are line stripes.

[0029] In step S204, a first target pixel point of each code element in the first image is obtained, and a matching pixel point in the second image of the first target pixel point of each code element in the first image is determined.

[0030] In step S206, the three-dimensional coordinates of the first target pixel point are determined based on at least the pixel coordinates of the predetermined first target pixel point in the first image and the pixel coordinates of the matching pixel point in the second image, thereby completing the three-dimensional reconstruction.

[0031] Through the above steps, the objective of replacing the coding method based on code element feature points with a coding method that determines the matching pixel point in the second image for the first target pixel point of each code element in the first image can be achieved, thereby achieving the technical effect of improving the matching accuracy of code elements and solving the technical problem that the accuracy of matching code elements based on code element feature points is low due to image blur.

[0032] 3D reconstruction refers to establishing a mathematical model of a 3D object suitable for computer display and processing. It is the basis for processing, manipulating, and analyzing its properties in a computer environment, and is also a technology for establishing a virtual reality that represents objective events in a computer. 3D reconstruction technology typically uses structured light temporal encoding and spatial encoding techniques. The difficulty with spatially encoded structured light is that it uses pixel spatial grayscale information to perform stable and reliable coding for each pixel point. One related technology method involves encoding specific pixel points with certain code element information, and then using a coding method for various circular code elements, utilizing epipolar constraints and the relative displacement relationship between neighboring code elements and the code element in the image, thereby reconstructing each code element point. The disadvantages of this method include limited code element encoding capacity, only sparse code element points can be reconstructed, limited single-frame data, low scanning efficiency, and a significant impact from the texture of the object surface. Another method uses the form of random speckle to perform corresponding point matching based on the correlation between pixel blocks. For example, a pseudo-random speckle image is projected onto the surface of the object to be measured using a projection device, and an improved SGM (semi-global matching, or stereo matching) algorithm is used to match the image blocks, thereby obtaining three-dimensional data of the surface of the object to be measured. The drawback of this method is that the image blocks to which matching is applied are large in size, and the accuracy and detail of the reconstructed data are poor, making it difficult to reconstruct complex objects. Related technologies generally use a code element feature point matching method to complete code element matching, but when the image clarity is poor and the feature points are highly blurred, the accuracy of feature point extraction decreases, resulting in a decrease in the accuracy of code element matching.

[0033] The method proposed in this application uses a line segment stripe code element encoding method to obtain a first target pixel point in the line segment stripe, for example, the midpoint of the line segment stripe. Matching is performed using the first target feature point of the code element, replacing the method of matching using code element feature points. This further improves the accuracy of code element matching when the image is unclear. In addition, the line segment stripe code element is used to increase the encoding capacity, thereby improving the data volume of a single frame image and further improving scanning efficiency. In addition, the method proposed in this application uses line segment stripe distribution, and the distribution density of code elements is high, further improving the accuracy of the obtained reconstruction data.

[0034] The method proposed in this application can be applied to scanners such as intraoral scanners, face scanners, industrial scanners, and dedicated scanners, and can realize 3D reconstruction of objects or scenes such as teeth, faces, human bodies, industrial products, industrial equipment, cultural assets, artworks, prosthetic limbs, medical instruments, and architecture.

[0035] In step S202, the first image is an image collected by a first image collecting device, the second image is an image collected by a second image collecting device, multiple target code elements in the first image are oriented in the same direction, and the pitch between the code elements is randomly determined (for example, the pitch between code element 1 and code element 2 is 100 μm, and the pitch between code element 2 and code element 3 is 80 μm). The code elements are randomly distributed vertically along the horizontal direction of the image (the camera epipolar direction), and the code element features include at least two extractable gradation feature point patterns, with the feature points distributed vertically at a predetermined pitch. Figure 3 shows a preferred code element image, where the code elements are randomly distributed vertically.

[0036] In step S204, a first target pixel point of each code element in the first image is obtained, and a matching pixel point in the second image of each first target pixel point of each code element in the first image is determined, where it should be understood that the code element structure in the first image is the same as the code element structure in the second image because the first image and the second image are obtained by different image acquisition devices acquiring projected images projected onto the surface of the object to be measured. As can be seen, take the first target pixel point as the midpoint of a line segment stripe in the first image as an example, and the matching pixel point in the second image is the midpoint of the line segment stripe in the second image.

[0037] In step S206, the first target pixel point in the first image is matched one-to-one with the matching pixel point in the second image, and the three-dimensional coordinate of the first target pixel point is determined based on at least the pixel coordinate of the first target pixel point in the first image and the pixel coordinate of the corresponding matching pixel point in the second image, thereby completing the reconstruction.

[0038] The above steps S202 to S206 will be described in detail below using a specific example.

[0039] In step S204, a matching pixel point in the second image of the first target pixel point can be determined using the pixel point coordinate and the pixel point gray level, specifically, the pixel coordinate and the gray level of the first target pixel point are determined. A first region of a predetermined area is determined in the first image, and the center point of the first region is set as the first target pixel point. A second region of a predetermined area is determined in the second image, where the ordinate of the center of the second region in the second image is the same as the ordinate of the center of the first region in the first image. A matching pixel point that matches the first target pixel point is determined in the second region.

[0040] In one preferred embodiment, the first and second regions may be image regions with an image size of m×n, where m and n are pixel values.

[0041] It will be understood that the above ordinates are pixel coordinates, and that the ordinate in the second image of the center of the second region being the same as the ordinate in the first image of the center of the first region indicates that the distance of the center of the first region from the boundary of the first image in the predetermined direction is the same as the distance of the center of the second region from the same boundary of the second image in the predetermined direction.

[0042] In one preferred embodiment, the correlation coefficient can be determined by comparing the correlation coefficient between the first target pixel point and each pixel point in the second region, for example, determining the correlation coefficient of each pixel point in the second region one by one, the correlation coefficient characterizing the correlation between the pixel point in the second region and the first target pixel point, and determining the pixel point in the second region with the largest correlation coefficient as the matching pixel point.

[0043] The correlation coefficient between each pixel point in the second region and the first target pixel point is determined in the following manner: first, the average gradation value of all pixel points in the first region is determined as the first average gradation value, and the average gradation value of all pixel points in the second region is determined as the second average gradation value; and then, based on the difference between the gradation value of the first target pixel point in the first region and the first average gradation value and the difference between the gradation value of each pixel point in the second region and the second average gradation value, the correlation coefficient between each pixel point in the second region and the first target pixel point is determined.

[0044] Specifically, the first target pixel point (u 1 , v 1 ), the gradation value is I 1 (u 1 , v 1 ) and one of the pixel points is set as the center of the first region of image size m × n, and a second region is determined in the second image at the same pixel row position as the first region, and the image size of the second region is m × n. The correlation coefficient ω of each pixel point in the second region i is calculated one by one, where i=0, 1, W-1, where W is the pixel width of the first image. The calculation formula for the correlation coefficient is as follows: JPEG2025526801000002.jpg15170

[0045] JPEG2025526801000003.jpg29170

[0046] In addition, the correlation coefficient may be a zero-mean normalized correlation coefficient determined by matching using the zero-mean normalized correlation coefficient (ZNCC) method, which describes the similarity between two different image blocks. The higher the correlation coefficient, the more similar the image blocks are and the more likely they are to become matching points.

[0047] For example, if the first target pixel point is the midpoint of a line segment stripe in the first image, the above steps can find a candidate matching point (the pixel point with the largest correlation coefficient) in the second image for each midpoint of the line segment stripe in the first image. Obviously, since the first target pixel point is the midpoint of a line segment stripe, the matching pixel point in the second image that matches with the first target pixel point must be the midpoint of the line segment stripe in the second image. Based on the above principle, all candidate matching points are traversed, and if the candidate matching point overlaps with the second target pixel point in the second image, the candidate matching point is determined as the matching pixel point. If the candidate matching point does not overlap with the second target pixel point in the second image, a second target pixel point within a predetermined range around the candidate matching point is determined as the matching pixel point.

[0048] In another preferred embodiment, after determining the matching pixel point, a first neighborhood code element set of any one code element in the first image and multiple second neighborhood code element sets of multiple candidate code elements in the second image may be determined in a manner using the first target pixel point as a feature point, the number of matches between neighborhood code elements in the multiple second neighborhood code element sets and neighborhood code elements in the first neighborhood code element set may be determined, the second neighborhood code element set with the largest number of matches among the multiple second neighborhood code element sets may be determined as a target second neighborhood code element set, and the candidate code elements corresponding to the target second neighborhood code element set may be determined as code elements matching any one of the code elements.

[0049] Specifically, the first neighborhood code element set is the neighborhood code element set of one code element in the second image. For example, the code element p in the second image is a line segment of a set length and two corresponding end points of the line segment. The neighborhood code element set of the code element p is {p 1 ,p 2 ,p 3 ,p 4}, and the second neighborhood code element set is a neighborhood code element set of a candidate code element of any one code element in the second image. For example, the candidate code element of code element p is code element q, and the neighborhood code element set of code element q is {q 1 ,q 2 ,q 3 ,q 4} and there are three candidate code elements for code element p, e.g., JPEG2025526801000004.jpg10170 When the second neighborhood code element set exists and there are multiple code elements p, the situation is the same as above, and the explanation is omitted here.

[0050] For example, let us consider the code element p. i , where i is a positive integer, first, the code element p 1 is the first neighboring code element of the candidate code element q1 JPEG2025526801000005.jpg44170The candidate code element with the largest number of matches is determined as the target code element. Take code element p as an example. If candidate code element q1 has the largest number of matches with code element p, candidate code element q1 is determined as the target code element that matches code element p.

[0051] In step S202, a predetermined code element image is generated in the following manner: a target area in which the target code element is located is determined based on the length of the target code element, the width of the target code element, the pitch between the target code elements, and the pixel coordinate of the center pixel point of the target code element, where the pixel coordinate of the center pixel point of the target code element is randomly generated within the area in which the code element image is located; all pixel points within the target area are traversed; if a target code element does not exist in the target area, a target code element is generated in the target area, where the target code element includes at least one line segment of a predetermined length and two end points corresponding to the line segment of the predetermined length; and target code elements are generated in all target areas within the code element image area. As shown in Figure 4, there are five target code elements. The code element designated by symbol 1 is composed of one line segment and two circular end points corresponding to the line segment. The code element designated by symbol 2 is composed of one line segment of a first predetermined length and two line segments of a second predetermined length as end points, and the first predetermined length is greater than the second predetermined length. The code element designated by symbol 3 is composed of one line segment of a first predetermined length and three line segments of a second predetermined length as end points. The code element designated by symbol 4 is composed of one line segment and its two end points. The code element designated by symbol 5 is formed by the intersection of one line segment and another line segment.

[0052] After the projection module and the image acquisition module of the reconstruction system are determined, JPEG2025526801000006.jpg24170 The minimum length of a unit pixel of a code element image that can be projected by a projection device is l. min By this formula, the minimum length of the stripes in the first image is determined as L min To ensure randomness of the projected pattern with width W and height H, the maximum length of the stripes in the pattern, L, can be determined as follows: max does not exceed H / 2, so the length of each stripe is L i ∈[L min ,L max ].

[0053] In particular, the length L of the stripes in the projected pattern may be a fixed length value or a random length value. In the case of a random length value, the length of each stripe is determined by a pseudo-random sequence {L i}, and the range of values of the pseudo-random sequence is L i ∈[L min ,L max ].

[0054] If the first target pixel point is the midpoint of a line segment stripe in the first image, the midpoint of the line segment stripe in the second image must be matched with the first target pixel point, and the midpoint of the line segment stripe in the second image is determined as the second target pixel point. The correlation coefficient between each second target pixel point and the first target pixel point is directly determined, and the second target pixel point with the largest correlation coefficient is determined as the matching pixel point for the first target pixel point, thereby reducing the number of pixel point calculations and further improving matching efficiency.

[0055] In some embodiments of the present application, before determining a plurality of candidate code elements in the first image that correspond to the target code elements in the second image, the method further includes a step of adjusting the first image and the second image to be coplanar.

[0056] In the field of stereoscopic vision, an epipolar constraint relationship exists between the first and second images, i.e., feature points with the same epipolar orientation in the projected image will also have the same epipolar orientation in the image acquired by the image acquisition device. Using the epipolar compensation principle, the epipolar orientations of the first and second images can be compensated to the same horizontal direction. After the compensation is complete, the first and second images are subjected to a filter smoothing process, including but not limited to a Gaussian filter.

[0057] The first and second images are generated by projecting the projection target image onto the surface of the object to be measured, and the projection parameters of each code element can be determined based on the preset orientation results.

[0058] Based on the above method, the present application can quickly and accurately reconstruct 3D data of the surface of the workpiece. The fringe center reconstruction method in the present application can achieve accurate 3D data acquisition, and the random fringe method increases the number of coding points (every single pixel point at the fringe center is a coding point), improving data redundancy and further improving scanning efficiency.

[0059] An embodiment of the present application further provides a 3D reconstruction system, as shown in FIG. 5 , the system includes at least two image acquisition devices 503, a projection device 504, and a first processor 502, wherein the projection device 504 is for projecting predetermined code element images onto the surface of the object to be measured; the at least two image acquisition modules 503 are for collecting predetermined code element images from the surface of the object to be measured 501 to obtain a first image and a second image; the first processor 502 is for acquiring first target pixel points of each code element in the first image and determining matching pixel points in the second image of the first target pixel points of each code element in the first image; and further for determining three-dimensional coordinates of the first target pixel points according to the predetermined image parameters of the first image and the second image to complete the three-dimensional reconstruction.

[0060] However, the image acquisition device 503 includes, but is not limited to, a grayscale camera and a color camera, and the projection method of the projection device 504 includes, but is not limited to, DLP (Digital Light Processing), MASK (Mask Projection), DOE (Diffraction Projection), etc., and can project a structured light pattern. In one preferred embodiment, there may be multiple image acquisition devices.

[0061] In one preferred embodiment, the first processor 502 pre-determines and generates a code element image, which is then transmitted to the projection device 504, which projects the code element image onto the surface of the object 501; two image acquisition devices 503 acquire a first image and a second image, which are then transmitted to the first processor 502 for code element matching; and after completing the code element matching, the three-dimensional coordinates of each code element in the first image are determined, thereby completing the three-dimensional reconstruction.

[0062] In some embodiments of the present application, the first processor 502 can be connected to an interactive device to input generation instructions, display code element images on a display interface, modify the displayed code element images, and display multiple types of data in the reconstruction process on the interactive device, thereby facilitating real-time monitoring.

[0063] An embodiment of the present application further provides a model training apparatus, as shown in FIG. 6 , including: an acquisition module 60 configured to acquire a first image and a second image, where the first image and the second image are acquired by collecting predetermined code element images projected onto a surface of an object to be measured by different image acquisition devices, and the predetermined code element images include a plurality of target code elements randomly distributed according to a predetermined direction, and the target code elements are line segment stripes; a matching module 62 configured to acquire first target pixel points of each code element in the first image and determine matching pixel points in the second image for each first target pixel point of each code element in the first image; and a reconstruction module 64 configured to determine three-dimensional coordinates of the first target pixel points based on the predetermined image parameters of the first image and the second image to complete three-dimensional reconstruction.

[0064] The matching module 62 includes a first determination sub-module, which determines the pixel coordinates and grayscale values of a first target pixel point, determines a first region of a predetermined area in the first image, and sets the center point of the first region as the first target pixel point, determines a second region of a predetermined area in the second image, where the ordinate of the center of the second region in the second image is the same as the ordinate of the center of the first region in the first image, and determines a matching pixel point from the second region that matches the first target pixel point.

[0065] the first determination sub-module includes a first determination unit and a second determination unit, the first determination unit determines a correlation coefficient for each pixel point in the second region one by one, the correlation coefficient characterizing the correlation between the pixel point in the second region and the first target pixel point, and is configured to determine the pixel point in the second region with the largest correlation coefficient as the matching pixel point; the second determination unit determines the gradation average value of all pixel points in the first region as the first gradation average value and the gradation average value of all pixel points in the second region as the second gradation average value, and is configured to determine the correlation coefficient between each pixel point in the second region and the first target pixel point based on the difference between the gradation value of the first target pixel point in the first region and the first gradation average value and the difference between the gradation value of each pixel point in the second region and the second gradation average value.

[0066] The first determination unit includes a determination subunit, which is configured to determine a pixel point in the second region having the largest correlation coefficient as a candidate matching point, and if the candidate matching point overlaps with a second target pixel point in the second image, determine the candidate matching point as a matching pixel point, and if the candidate matching point does not overlap with the second target pixel point in the second image, determine a second target pixel point within a predetermined range around the candidate matching point as a matching pixel point.

[0067] The matching module 62 further includes a matching sub-module, which is configured to determine a plurality of first target pixel points in the first image and a plurality of second target pixel points in the second image, and match the plurality of first target pixel points one-to-one with the plurality of second target pixel points in the second image.

[0068] According to another aspect of an embodiment of the present application, there is further provided a non-volatile storage medium, the non-volatile storage medium including a stored program, wherein when the program is executed, it controls an apparatus in which the non-volatile storage medium is located to perform the above-mentioned three-dimensional reconstruction method.

[0069] According to another aspect of the present invention, there is further provided a processor, the processor being configured to execute a program, wherein when the program is executed, the program performs the above three-dimensional reconstruction method.

[0070] The processor is configured to execute a program that performs the following functions: acquire a first image and a second image, the first image and the second image being acquired by respectively collecting predetermined code element images projected onto the surface of the object to be measured by different image acquisition devices, the predetermined code element images including a plurality of target code elements randomly distributed according to a predetermined direction, the target code elements being line segment stripes; acquire a first target pixel point of each code element in the first image; and determine a matching pixel point in the second image for the first target pixel point of each code element in the first image; and determine the three-dimensional coordinates of the first target pixel point based at least on the pixel coordinates of the predetermined first target pixel point in the first image and the pixel coordinates of the matching pixel point in the second image, thereby completing three-dimensional reconstruction.

[0071] The processor executes the three-dimensional reconstruction method, and achieves the purpose of replacing the coding method using code element feature points with a coding method that determines the matching pixel point in the second image for the first target pixel point of each code element in the first image, thereby achieving the technical effect of improving the matching accuracy of code elements and solving the technical problem of low accuracy in matching code elements based on code element feature points due to image blur.

[0072] In the above embodiments of the present application, the description of each embodiment has its own emphasis, and for the parts not described in detail in one embodiment, reference can be made to the relevant descriptions of other embodiments.

[0073] It should be noted that in some embodiments provided in the present application, the disclosed technical contents can be realized in other ways. Here, the device embodiments described above are merely exemplary. For example, the division of units may be division of logical functions, and other division methods may exist when actually realized. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. In other respects, the shown or discussed mutual couplings, direct couplings, or communication connections may be indirect couplings or communication connections via some interfaces, units, or modules, and may be electrical or other types.

[0074] The units described as separate elements may or may not be physically separate, and the elements shown as units may or may not be physical units, and may be located in one place or distributed among multiple units, some or all of which may be selected to achieve the objectives of the solutions of the present embodiment according to actual needs.

[0075] Furthermore, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated units may be realized in the form of hardware or in the form of software functional units.

[0076] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially embodied in the form of a software product, or a part of the technical solution, or all or part of the technical solution, and the computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the steps of each method embodiment of the present application. The aforementioned storage medium includes various media capable of storing program code, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a portable hard disk, a magnetic disk, or an optical disk.

[0077] The above are only preferred embodiments of the present application, and those skilled in the art may make some improvements and modifications without departing from the principles of the present application, and these improvements and modifications should also be considered as within the protection scope of the present application. [Industrial Applicability]

[0078] The technical solution provided by the embodiments of the present application can be applied to the field of three-dimensional reconstruction. In the embodiments of the present application, a first image and a second image are obtained, and the first image and the second image are obtained by respectively collecting predetermined code element images projected onto the surface of the object to be measured by different image acquisition devices, and the predetermined code element images include a plurality of target code elements randomly distributed according to a predetermined direction, and the target code elements are line stripes; a first target pixel point of each code element in the first image is obtained; and a matching pixel point in the second image of the first target pixel point of each code element in the first image is determined; A method is adopted in which the three-dimensional coordinates of the first target pixel point are determined based on the pixel coordinates of the determined first target pixel point in the first image and the pixel coordinates of the matching pixel point in the second image to complete three-dimensional reconstruction, thereby achieving the purpose of replacing the coding method using code element feature points with a coding method that determines the matching pixel point in the second image for the first target pixel point of each code element in the first image, thereby achieving the technical effect of improving the matching accuracy of code elements and solving the technical problem of low accuracy in matching code elements based on code element feature points due to image blur. [Explanation of symbols]

[0079] 501…Object to be measured 502...First processor 503...Image acquisition equipment 504…Projection equipment

Claims

1. acquiring a first image and a second image, wherein the first image and the second image are acquired by collecting predetermined code element images projected onto the surface of the object by different image acquisition devices, the predetermined code element images including a plurality of target code elements randomly distributed according to a predetermined direction, and the target code elements are line stripes; obtaining a first target pixel point of each code element in the first image, and determining a matching pixel point in the second image of the first target pixel point of each code element in the first image; and determining three-dimensional coordinates of the first target pixel point based at least on predetermined pixel coordinates of the first target pixel point in the first image and pixel coordinates of the matching pixel point in the second image, thereby completing three-dimensional reconstruction.

2. determining a matching pixel point in the second image for each first target pixel point of each code element in the first image, determining pixel coordinates of the first target pixel point and a gradation value of the first target pixel point; determining a first region of a predetermined area in the first image, and setting the center point of the first region as the first target pixel point; determining a second region of the predetermined area from the second image, wherein the ordinate of the center of the second region in the second image is the same as the ordinate of the center of the first region in the first image; and determining a matching pixel point from the second region that is matched with the first target pixel point.

3. The step of determining a matching pixel point from the second region that is matched with the first target pixel point includes: determining a correlation coefficient for each pixel point in the second region, the correlation coefficient characterizing a correlation between the pixel point in the second region and the first target pixel point; The method of claim 2 , further comprising determining the pixel point in the second region having the largest correlation coefficient as the matching pixel point.

4. The step of determining the correlation coefficients of each pixel point in the second region one by one includes: determining a gradation average value of all pixel points in the first region as a first gradation average value, and determining a gradation average value of all pixel points in the second region as a second gradation average value; and determining a correlation coefficient between each pixel point in the second region and the first target pixel point based on a difference between the gradation value of the first target pixel point in the first region and the first gradation average value and a difference between the gradation value of each pixel point in the second region and the second gradation average value.

5. The step of determining the pixel point in the second region having the largest correlation coefficient as the matching pixel point includes: determining a pixel point in the second region having the largest correlation coefficient as a candidate matching point; determining the candidate matching point as the matching pixel point if the candidate matching point overlaps with a second target pixel point in the second image; and if the candidate matching point does not overlap with a second target pixel point in the second image, determining the second target pixel point within a predetermined range around the candidate matching point as the matching pixel point.

6. The predetermined code element image includes a plurality of target code elements randomly distributed according to a predetermined direction, and the target code elements are line stripes. determining a target area in which the target code element is located based on the length of the target code element, the width of the target code element, the pitch between the target code elements, and pixel coordinates of the target code element center pixel point, wherein the pixel coordinates of the target code element center pixel point are randomly generated within the area in which the code element image is located; traversing all pixel points within the target area, and if the target code element does not exist in the target area, generating the target code element in the target area, wherein the target code element includes at least one line segment of a predetermined length and two end points corresponding to the line segment of the predetermined length; generating the target code element for all of the target areas within the code element image area.

7. determining a first set of neighboring code elements of any one code element in the first image and a plurality of second sets of neighboring code elements of a plurality of candidate code elements in the second image; determining the number of matches between neighboring code elements in the plurality of second neighboring code element sets and neighboring code elements in the first neighboring code element set, and determining the second neighboring code element set having the largest number of matches among the plurality of second neighboring code element sets as a target second neighboring code element set; The method of claim 1 , further comprising: determining the candidate code elements corresponding to the target second set of neighborhood code elements as the target code elements.

8. determining a plurality of first target pixel points in the first image and a plurality of second target pixel points in the second image; 2. The method of claim 1, further comprising the step of: one-to-one matching the plurality of first target pixel points with a plurality of second target pixel points in the second image.

9. an acquisition module configured to acquire a first image and a second image, the first image and the second image being acquired by collecting predetermined code element images projected onto a surface of the object to be measured by different image acquisition devices, the predetermined code element images including a plurality of target code elements randomly distributed according to a predetermined direction, the target code elements being line segment stripes; a matching module configured to obtain a first target pixel point of each code element in the first image and determine a matching pixel point in the second image of the first target pixel point of each code element in the first image; a reconstruction module configured to determine three-dimensional coordinates of the first target pixel point based at least on predetermined pixel coordinates of the first target pixel point in the first image and pixel coordinates of the matching pixel point in the second image, thereby completing three-dimensional reconstruction.

10. at least two image acquisition devices, a projection device, and a first processor; the projection device is for projecting a predetermined code element image onto a surface of the object to be measured; the at least two image acquisition modules are for acquiring the predetermined code element images from a surface of the workpiece to obtain a first image and a second image; a first processor for obtaining first target pixel points of each code element in the first image and determining matching pixel points in the second image for the first target pixel points of each code element in the first image, and for determining three-dimensional coordinates of the first target pixel points based at least on predetermined pixel coordinates of the first target pixel points in the first image and pixel coordinates of the matching pixel points in the second image to complete three-dimensional reconstruction.

11. A non-volatile storage medium, the non-volatile storage medium containing a stored program, wherein when the program is executed, it controls an apparatus in which the non-volatile storage medium is located to perform a three-dimensional reconstruction method described in any one of claims 1 to 8.

12. An electronic device comprising a memory and a processor, the processor configured to execute a program, wherein when the program is executed, the electronic device performs the three-dimensional reconstruction method according to any one of claims 1 to 8.

Images