Three-dimensional scanning method and device, electronic equipment and program product

Abstract

The invention belongs to the technical field of scanning, and provides a three-dimensional scanning method and device, electronic equipment and a program product.The three-dimensional scanning method comprises the steps that after a high-frequency fringe image is projected to the surface of a measured object through a projector, the surface of the measured object is collected based on a binocular camera assembly, and a speckle fringe image is obtained; wherein a predetermined sparse speckle removal pattern is inserted into the high-frequency fringe image, and the binocular camera assembly comprises a first camera and a second camera which are arranged on the two sides of the projector; decoding the speckle stripe image to obtain a relative phase value of the first camera and the second camera; according to the speckle fringe image and the relative phase value, calculating a sub-pixel-level parallax value of the speckle fringe image; and determining three-dimensional point cloud information of the measured object based on a predetermined camera calibration parameter and the sub-pixel-level parallax value. The technical problem that in the prior art, the number of projected images of a three-dimensional scanning technology is large, and rapid three-dimensional scanning is not facilitated can be solved.

Description

Technical Field

[0001] The present application belongs to the field of scanning technology, and more specifically, to a three-dimensional scanning method, device, electronic equipment and program product. Background Art

[0002] A structured light 3D scanner is a device that can collect three-dimensional shape and size data of real-world objects or environments. A structured light 3D scanner captures the shape and details of objects at all angles through three-dimensional scanning without damaging the surface of the object. It is suitable for scanning fragile or valuable items. Compared with traditional measurement methods, three-dimensional scanning can quickly acquire large amounts of data. Some 3D scanners can achieve very high measurement accuracy and are widely used in industrial manufacturing, cultural relics protection, medical health, construction engineering, virtual reality (VR), augmented reality (AR) and other fields.

[0003] In the prior art, the phase shift algorithm is a commonly used 3D scanning method for obtaining 3D information of an object, also known as phase measurement profilometry. However, the disadvantages of this phase measurement profilometry are that it is not only affected by the global illumination effect, but also has a large number of projected images, which is not conducive to achieving fast 3D scanning. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a three-dimensional scanning method, device, electronic device and program product, aiming to solve the technical problem that the three-dimensional scanning technology in the prior art has a large number of projected images, which is not conducive to achieving fast three-dimensional scanning.

[0005] To achieve the above object, according to a first aspect of the present application, a three-dimensional scanning method is provided, the method comprising:

[0006] After the high-frequency fringe image is projected onto the surface of the object to be measured by a projector, the surface of the object to be measured is captured based on a binocular camera assembly to obtain a speckle fringe image; wherein a predetermined sparse speckle pattern is inserted into the high-frequency fringe image, and the binocular camera assembly includes: a first camera and a second camera arranged on both sides of the projector;

[0007] Decoding the speckle fringe image to obtain a relative phase value between the first camera and the second camera;

[0008] Calculating a sub-pixel disparity value of the speckle fringe image according to the speckle fringe image and the relative phase value;

[0009] Based on predetermined camera calibration parameters and the sub-pixel parallax value, three-dimensional point cloud information of the object under test is determined.

[0010] Optionally, in a possible implementation manner of the first aspect, calculating a sub-pixel disparity value of the speckle fringe image according to the speckle fringe image and the relative phase includes:

[0011] Performing epipolar correction processing on the speckle fringe image by using an epipolar correction algorithm to obtain an epipolar corrected speckle fringe image;

[0012] Using a binocular stereo matching algorithm to perform matching calculation on the epipolar corrected speckle fringe image to obtain an initial pixel-level disparity value;

[0013] A quadratic curve fitting algorithm is adopted to calculate the sub-pixel disparity value of the speckle fringe image based on the initial pixel-level disparity value and the relative phase value.

[0014] Optionally, in a possible implementation manner of the first aspect, determining the three-dimensional point cloud information of the object under test based on a predetermined camera calibration parameter and the sub-pixel disparity value includes:

[0015] Based on the camera calibration parameters, the sub-pixel disparity value is converted into three-dimensional point cloud information of the measured object.

[0016] Optionally, in a possible implementation manner of the first aspect, the method further includes:

[0017] According to the surface characteristics of the measured object and / or the measurement environment factors, the parameters of the initial speckle pattern are adjusted to obtain the predetermined sparse speckle pattern, wherein the predetermined sparse speckle pattern includes at least one of the following: speckle spots, speckle lines, speckle stripes, and speckle grids, and the parameters include at least one of the following: distribution density, size, and grayscale;

[0018] inserting the predetermined sparse speckle pattern into the high-frequency fringe image;

[0019] The high-frequency fringe image is projected onto the surface of the object to be measured by the projector.

[0020] Optionally, in a possible implementation manner of the first aspect, after determining the three-dimensional point cloud information of the object to be measured based on predetermined camera calibration parameters and the sub-pixel parallax value, the method further includes:

[0021] Post-processing the three-dimensional point cloud information, wherein the post-processing includes at least one of the following: filtering processing, smoothing processing, and data compression processing; and

[0022] A visualization image is generated according to the three-dimensional point cloud information, wherein the visualization image is used to display the three-dimensional shape or structure of the object being measured in at least one of a point cloud image, a grid image, and a solid model.

[0023] According to a second aspect of the present application, a three-dimensional scanning device is provided, the device comprising:

[0024] A projector, used for projecting a high-frequency fringe image with a predetermined sparse speckle pattern inserted onto a surface of the object to be measured;

[0025] A binocular camera assembly, comprising a first camera and a second camera disposed on both sides of the projector, for capturing the surface of the object to be measured to obtain a speckle fringe image;

[0026] an image processing module, configured to decode the speckle fringe image to obtain a relative phase value between the first camera and the second camera, and calculate a sub-pixel disparity value of the speckle fringe image according to the speckle fringe image and the relative phase value;

[0027] The three-dimensional information generation module is used to determine the three-dimensional point cloud information of the object under test based on predetermined camera calibration parameters and the sub-pixel parallax value.

[0028] Optionally, in a possible implementation manner of the second aspect, the image processing module further includes:

[0029] an epipolar correction unit, configured to perform epipolar correction processing on the speckle fringe image using an epipolar correction algorithm to obtain an epipolar corrected speckle fringe image;

[0030] A pixel-level disparity calculation unit, used for performing matching calculation on the epipolar-corrected speckle fringe image using a binocular stereo matching algorithm to obtain an initial pixel-level disparity value;

[0031] The sub-pixel disparity optimization unit is used to calculate the sub-pixel disparity value of the speckle fringe image based on the initial pixel-level disparity value and the relative phase value by using a quadratic curve fitting algorithm.

[0032] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0033] According to a third aspect of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements any of the methods described in one embodiment.

[0034] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0035] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0036] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0037] The embodiment of the present application provides a three-dimensional scanning method, device, electronic device and program product. The three-dimensional scanning method, after projecting a high-frequency fringe image onto the surface of the object to be measured by a projector, collects the surface of the object to be measured based on a binocular camera assembly to obtain a speckle fringe image; wherein a predetermined sparse speckle pattern is inserted into the high-frequency fringe image, and the binocular camera assembly includes: a first camera and a second camera arranged on both sides of the projector; decoding the speckle fringe image to obtain a relative phase value between the first camera and the second camera; calculating the sub-pixel disparity value of the speckle fringe image based on the speckle fringe image and the relative phase value; determining the three-dimensional point cloud information of the object to be measured based on the predetermined camera calibration parameters and the sub-pixel disparity value. This can solve the technical problem that the number of projected images of the three-dimensional scanning technology in the prior art is large, which is not conducive to realizing fast three-dimensional scanning. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a flowchart of a three-dimensional scanning method provided in an embodiment of the present application;

[0040] Figure 2 It is a schematic flow chart of an optional three-dimensional scanning method provided in an embodiment of the present application;

[0041] Figure 3 It is a schematic flow chart of an optional three-dimensional scanning method provided in an embodiment of the present application;

[0042] Figure 4 It is a schematic flow chart of an optional three-dimensional scanning method provided in an embodiment of the present application;

[0043] Figure 5 It is a schematic flow chart of an optional three-dimensional scanning method provided in an embodiment of the present application;

[0044] Figure 6 is a structural schematic diagram of a three-dimensional scanning device provided in an embodiment of the present application;

[0045] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0047] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0048] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0050] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0051] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0052] A structured light 3D scanner is a device that can collect three-dimensional shape and size data of real-world objects or environments. It captures the shape and details of objects at all angles through three-dimensional scanning without damaging the surface of the object. It is suitable for scanning fragile or valuable items. Compared with traditional measurement methods, three-dimensional scanning can quickly obtain large amounts of data. Some 3D scanners can achieve very high measurement accuracy and are widely used in industrial manufacturing, cultural relics protection, medical health, construction engineering, virtual reality (VR) / augmented reality (AR) and other fields.

[0053] Among them, the phase shift algorithm is a commonly used 3D scanning method for obtaining 3D information of an object, also known as phase measurement profilometry. Most phase measurement profilometry techniques assume that the scene point is illuminated by an ideal point light source. However, in reality, the assumption of the above phase measurement profilometry is not valid. This is because in addition to the direct illumination light, the scene point is also indirectly illuminated by other light, mainly mutual reflection light and sub-surface scattered light. Mutual reflection light produces mutual reflection effect, and sub-surface scattered light produces sub-surface scattering effect. These effects are strongly dependent on the material properties of the surface or scene. In addition, the projector also has a defocus effect, which, together with the mutual reflection effect and the sub-surface scattering effect, is collectively referred to as the global illumination effect. The global illumination effect affects the direct illumination effect. Therefore, if the influence of the global illumination effect is completely ignored, it will lead to a large error in the calculated wrap-around phase.

[0054] It can be seen that the existing technical solutions to the global illumination effect are mainly to project high-frequency phase-shifted fringe images with three frequencies and four-step phase shifts or three frequencies and three-step phase shifts, which require a total of 12 or 9 images to be projected, and belong to the common phase measurement profilometry. Some solutions also project high-frequency fringe images with three frequencies, where the first frequency projects three images, and the second and third frequencies each project one image, which is used to resolve the absolute phase of the first frequency fringe image.

[0055] In response to the above problems, the embodiment of the present application proposes a binocular structured light three-dimensional scanning method, which only needs to project three images to greatly reduce the influence of the global illumination effect, thereby obtaining three-dimensional point cloud information. The specific technical solution is: a high-frequency cosine fringe image with a specific sparse speckle pattern inserted is projected onto the surface of the object to be measured by a projector, and black-and-white cameras located on the left and right sides of the projection collect the projected fringe image. The fringe analysis method is used to calculate the relative phase of the images collected by the left and right black-and-white cameras, and the fringe images are matched based on the sparse speckle pattern using the epipolar correction algorithm and the binocular stereo matching algorithm to calculate the pixel-level parallax, and then the sub-pixel-level parallax is calculated based on the pixel-level parallax and the relative phase, thereby combining the binocular camera calibration parameters to convert the precise sub-pixel-level parallax into three-dimensional information.

[0056] This application example provides an example of a three-dimensional scanning method, please refer to Figure 1 As shown, Figure 1 A schematic flow chart of a three-dimensional scanning method provided by the present application is shown. As an example but not a limitation, the method can be applied to or run in an electronic device. The method includes:

[0057] S101, after projecting the high-frequency fringe image onto the surface of the object to be measured by a projector, the surface of the object to be measured is captured based on a binocular camera assembly to obtain a speckle fringe image.

[0058] A predetermined sparse speckle pattern is inserted into the high-frequency fringe image, and the binocular camera assembly includes: a first camera and a second camera arranged on both sides of the projector.

[0059] S102, decoding the speckle fringe image to obtain a relative phase value between the first camera and the second camera.

[0060] S103, calculating a sub-pixel disparity value of the speckle fringe image according to the speckle fringe image and the relative phase value.

[0061] S104, determining the three-dimensional point cloud information of the object to be measured based on the predetermined camera calibration parameters and the sub-pixel parallax value.

[0062] In the example of the present application, first, a high-frequency fringe image is generated, and a predetermined sparse speckle pattern is inserted into the high-frequency fringe image. Specifically, an encoding operation can be performed according to a specific mathematical model to generate a high-frequency fringe image. For example, according to the mathematical expression of the high-frequency cosine fringe with three-step phase shift, various parameters of the fringe are determined, including: preset constants, number of phase shift steps, parameters for modulating the brightness of the speckle spots, etc., so as to generate a high-frequency fringe image with a speckle pattern (such as a high-frequency cosine fringe image). Then, the high-frequency fringe image is accurately projected onto the surface of the object to be measured by a projector. It should be understood that in the example of the present application, the projector needs to have sufficient resolution and brightness adjustment capabilities to ensure that the projected image is clear and can cover the effective measurement area of ​​the object to be measured.

[0063] By inserting a predetermined sparse speckle pattern into the high-frequency fringe image, on the one hand, the speckle pattern can increase the texture information of the image, so that more feature points can be used in subsequent matching and calculation, improving the accuracy and stability of the measurement. On the other hand, the speckle pattern can reduce the influence of the global illumination effect to a certain extent, such as reducing the defocus effect, mutual reflection effect and sub-surface scattering effect, so that the collected image can more truly reflect the actual situation of the surface of the measured object.

[0064] Furthermore, in the example of the present application, a binocular camera assembly is formed by setting a first camera and a second camera on both sides of the projector (the specific type may be a black and white camera, i.e., a monochrome camera). For example, the first camera may be a camera set on the left side of the projector, and the second camera may be a camera set on the right side of the projector. Specifically, the two cameras need to be accurately installed and calibrated to ensure that their optical axes are parallel and their relative positions are fixed, so that images with speckle fringes on the surface of the object under test can be accurately captured later. After the projection is completed, the binocular camera simultaneously captures the surface of the object under test. Due to the ups and downs of the object surface and the presence of projection fringes and speckle patterns, the image captured by the camera is a speckle fringe image.

[0065] By using a binocular camera to collect images from different angles at the same time, the depth information of the object surface can be obtained through the principle of triangulation. Images from different perspectives can provide more information about the shape and structure of the object surface, and can capture the three-dimensional features of the measured object more comprehensively and accurately than a monocular camera.

[0066] Furthermore, according to the preset coding rules and algorithms, the acquired speckle fringe image is decoded to analyze or extract information such as the phase of the fringe and the brightness of the speckle in the speckle fringe image. For example, according to the mathematical expression of the speckle fringe image, the relative phase value of the image acquired by the first camera and the second camera is determined by calculating and processing information such as the grayscale value of different pixels in the speckle fringe image. For example, a specific function is operated on the pixels at different positions in the speckle fringe image to restore the phase information given in the projection process, thereby obtaining the relative phase value.

[0067] The relative phase value obtained through decoding processing can reflect the phase difference between the images captured by the two cameras. This phase difference is closely related to factors such as the shape and height change of the surface of the object being measured. Calculating an accurate relative phase value will help improve the accuracy of subsequent calculations of sub-pixel parallax values ​​and the final determination of three-dimensional point cloud information, and thus help improve the precision and accuracy of three-dimensional scanning.

[0068] Afterwards, in the example of this application, an epipolar correction algorithm is used to perform epipolar correction processing on the speckle fringe image. Specifically, the epipolar correction processing is based on the geometric model and imaging principle of the binocular camera, and a geometric transformation model is constructed by acquiring the intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters (such as the relative rotation and translation relationship between cameras) of the binocular camera. Then, a coordinate transformation operation is performed on each pixel point in the speckle fringe image, converting it from the original image coordinate system to the corrected coordinate system, so that the images captured by the two cameras can be aligned in the epipolar direction, and the speckle fringe image after epipolar correction is obtained.

[0069] Optionally, in the example of the present application, the binocular stereo matching algorithm can find corresponding points between the images captured by the two cameras based on the features of the pixel points in the speckle fringe image after epipolar correction (such as grayscale value, texture, edge, etc.). For example, taking the normalized cross-correlation stereo matching algorithm as an example, the left camera image is divided into several small image blocks, and then the image block most similar to each left camera image block is found in the right camera image along the epipolar direction, and their similarity is measured by calculating the normalized cross-correlation coefficient between the two image blocks. When the most similar image block is found, the initial pixel-level disparity value can be obtained by calculating the coordinate difference of the corresponding pixel points in the two corresponding image blocks. It should be understood that the initial pixel-level disparity value reflects the position difference between the two camera-captured images at the pixel level, and is the basis for further calculating the sub-pixel disparity value.

[0070] Furthermore, based on the initial pixel-level disparity value and relative phase value, the quadratic curve fitting algorithm is used to further optimize the disparity value to obtain the sub-pixel disparity value. Specifically, according to the phase value of the pixel points in the same row of the left camera phase image, combined with the pixel-level disparity map, the pixel points that meet specific conditions can be found on the right camera image, and a more accurate disparity value can be determined by quadratic curve fitting, thereby obtaining the sub-pixel disparity value.

[0071] Optionally, the camera calibration parameters include internal parameters of the camera (such as focal length, principal point coordinates, pixel size, etc.) and external parameters (such as relative rotation and translation relationship between cameras). The camera calibration parameters are obtained by pre-calibrating the camera. A specific calibration plate or an object of known shape can be used for shooting and measurement, and then the camera calibration parameters are obtained through calculation and analysis.

[0072] For example, in the example of this application, the triangulation principle can be used to calculate each pixel in the image according to the camera calibration parameters and sub-pixel disparity values, and convert it from image coordinates (pixel coordinates) to coordinates in the camera coordinate system, and then combine the external parameters between the two cameras to calculate through the triangulation formula to obtain the three-dimensional coordinates of the pixel in the world coordinate system (including coordinates in the three directions of x, y, and z). By calculating all the pixels in the image that can obtain valid sub-pixel disparity values, a series of three-dimensional coordinate points can be obtained, which constitute the three-dimensional point cloud information of the object under test. In this way, the conversion from two-dimensional image information (speckle fringe image) to three-dimensional space information (three-dimensional point cloud information) is realized, so that the actual shape and structure of the object under test in three-dimensional space can be accurately restored.

[0073] In the embodiment of the present application, a high-frequency cosine fringe image with a specific sparse speckle pattern inserted is projected onto the surface of the object to be measured by a projector, and the black-and-white cameras on the left and right sides of the projector collect the projected fringe images. The relative phase value of the images collected by the left and right black-and-white cameras is calculated using the fringe analysis method, and the speckle fringe images are matched based on the sparse speckle pattern using the epipolar correction algorithm and the binocular stereo matching algorithm to calculate the pixel-level parallax, and then the sub-pixel-level parallax value is calculated based on the pixel-level parallax and the relative phase, so as to convert the precise sub-pixel-level parallax into three-dimensional information in combination with the binocular camera calibration parameters. Since the high-frequency fringe image projection can effectively solve the influence of the global illumination effect, the embodiment of the present application only needs to project fewer high-frequency fringe images (such as 3 high-frequency fringe images, etc.) to obtain a high-precision three-dimensional point cloud, and the influence of the global illumination effect is weakened, with the advantages of fast scanning speed and high scanning accuracy.

[0074] For a possible implementation, please refer to Figure 2 As shown, Figure 2A schematic flow chart of an optional three-dimensional scanning method provided by the present application is shown, and a sub-pixel disparity value of a speckle fringe image is calculated based on the speckle fringe image and the relative phase, including:

[0075] S201, using an epipolar correction algorithm to perform epipolar correction processing on the speckle fringe image to obtain an epipolar corrected speckle fringe image.

[0076] S202, using a binocular stereo matching algorithm to perform matching calculation on the speckle fringe image after epipolar correction to obtain an initial pixel-level disparity value.

[0077] S203, using a quadratic curve fitting algorithm, based on the initial pixel-level disparity value and the relative phase value, calculate the sub-pixel disparity value of the speckle fringe image.

[0078] Optionally, the epipolar correction algorithm is implemented based on the geometric model and imaging principle of the binocular vision system. When the binocular camera collects speckle fringe images, due to the different positions and angles of the two cameras, the positional relationship of the corresponding points in the collected images is not a simple one-to-one correspondence, but there is a certain geometric transformation relationship. In the example of this application, the purpose of the epipolar correction processing is to eliminate the image geometric distortion caused by the difference in camera position and angle, so that the images collected by the two cameras can be aligned in the epipolar direction for subsequent accurate matching calculations.

[0079] In one example, an epipolar correction algorithm is used to construct a geometric transformation model based on the intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters (such as the relative rotation and translation relationship between cameras) of the binocular camera. Then, by performing a coordinate transformation operation on each pixel in the speckle fringe image, it is converted from the original image coordinate system to the corrected coordinate system. For example, for a pixel point (x, y) in the image, its coordinates are converted to new coordinates (x', y') according to a pre-calculated transformation matrix, so that in the new coordinate system, the corresponding points in the images captured by the two cameras are on the same straight line in the epipolar direction, thereby obtaining a speckle fringe image after epipolar correction.

[0080] After the epipolar correction process, the images captured by the two cameras are aligned in the epipolar direction, so that the corresponding points in the image can be found more accurately when the binocular stereo matching algorithm is subsequently performed. Before the epipolar correction process is performed, the geometric distortion of the image causes incorrect matching of corresponding points, thereby affecting the accuracy of the calculation of the disparity value. Therefore, in the example of this application, this interference factor is eliminated through the epipolar correction process. The corrected image makes the matching calculation process simpler and more efficient, and improves the accuracy and reliability of the matching.

[0081] In the case of epipolar alignment, the search range for corresponding points can be greatly narrowed. There is no need to conduct an extensive search in the entire image area. Instead, a targeted search can be performed along the epipolar direction, thereby reducing the amount of calculation and matching time and improving the efficiency of the entire 3D scanning process.

[0082] Furthermore, the binocular stereo matching algorithm is mainly used to find corresponding points between the images captured by two cameras based on the features of the pixels in the images. For example, the above features may include: grayscale value, texture, edge and other information of the pixels. For example, the binocular stereo matching algorithm includes: region-based matching algorithm (such as normalized cross-correlation stereo matching algorithm), feature-based matching algorithm, etc.

[0083] For the speckle fringe image after epipolar correction, the binocular stereo matching algorithm can be used to divide the left camera image into several small image blocks (for example, divided by a certain window size), and then the image block most similar to each left camera image block is found in the right camera image along the epipolar direction. In the process of finding the most similar image block, the normalized mutual correlation coefficient between the two image blocks is calculated to measure their similarity.

[0084] The initial pixel-level disparity value calculated by the binocular stereo matching algorithm can reflect the position difference between the images captured by the two cameras at the pixel level. The algorithm performs matching calculations based on various features of the pixels in the image, which can make full use of the rich information of the image and make the matching results more accurate and reliable. Different image features (such as grayscale value, texture, etc.) can play a role in different scenes, thus adapting to various complex surface conditions of the measured objects, ensuring that relatively accurate initial pixel-level disparity values ​​can be obtained in different environments.

[0085] Furthermore, based on the initial pixel-level disparity value and the relative phase value, the disparity value is further optimized by a quadratic curve fitting algorithm to obtain a sub-pixel disparity value. Specifically, it can be, but is not limited to, classified according to the phase value of the pixel points in the same row of the left camera phase image. For example, for the same row of the left camera phase image, the phase value of the current pixel point (x, y) is set to φL(x, y). Different situations occur. According to these situations, the corresponding phase value and coordinate are taken for quadratic curve fitting to obtain the coordinate value xL whose phase value is equal to φL(x, y). Then, according to the pixel-level disparity map and the φL(x, y) value, the pixel point (xR, y) that meets the specific conditions is found on the right camera image, and a more accurate disparity value is determined by quadratic curve fitting, thereby obtaining a sub-pixel disparity value, which can further improve the disparity accuracy on the basis of the pixel-level disparity, and provide more refined disparity data for accurately determining the three-dimensional point cloud information.

[0086] As a specific implementation method, the technical solution provided in the embodiment of the present application is to use a structured light 3D scanner, specifically project a high-frequency fringe image with a specific sparse speckle pattern, use the high-frequency fringes in the high-frequency fringe image to reduce the impact of the global illumination effect, and use binocular speckle phase matching to achieve parallax calculation, thereby achieving fast and high-precision 3D scanning. The following will describe in detail the fringe encoding and decoding principle and the precise calculation of parallax:

[0087] 1) In the embodiment of the present application, the mathematical expression of the high-frequency cosine fringes with three-step phase shift is:

[0088]

[0089] In the above formula, k=1,2,3, indicating the kth image; (u,v) indicates the pixel coordinates; C(u,v) modulates the brightness of the scattered speckles; M(u,v)∈{0,1} indicates the distribution of random scattered speckles; A and B respectively indicate the preset constants of the stripes, and A=B; N is the number of phase shift steps, and in the embodiment of the present application, N=3 is set; H indicates the lateral resolution of the projector, that is, the projected stripes are vertical stripes.

[0090] Due to the projected fringe image The maximum amplitude is equal to A+B. In order to obtain a high-quality projection image, that is, random speckle interpolation in the fringe image will not affect the fringe decoding, let:

[0091]

[0092] In the above formula, C max =max{S k (u,v)},C min =min{S k (u,v)},S k (u,v)=A+B* k = 1, 2, 3. Using the above formula (2), a high-frequency fringe image with interpolated speckles can be generated.

[0093] Furthermore, in the example of the present application, a projector is used to project the generated high-frequency fringe image with interpolated scattered spots onto the surface of the object to be measured, and black and white cameras located on the left and right sides of the projector collect images. The obtained speckle fringe image can be expressed by the following formula (3).

[0094]

[0095] In the above formula, x and y represent the horizontal and vertical coordinates of the pixel in the image respectively; is the average grayscale; is grayscale modulation; is the relative phase value, also called the winding relative phase, and its value is in the range [0,2π). It represents the random noise originating from the light source and the camera, which is independent of the warping phase value and the position of the pixel. The calculation formula is as follows:

[0096]

[0097] Using the above formula (4), the relative phase value of each pixel point can be calculated, including the relative phase value of the speckle pixel point.

[0098] The relative phase of the left and right cameras is obtained by decoding the speckle fringe images collected by the left and right cameras. Since there are scattered spots in the projected image, the speckle fringe image collected by the camera is subjected to epipolar correction processing to obtain the epipolar corrected speckle fringe image. Then, the epipolar corrected speckle fringe image is subjected to binocular stereo matching using a binocular stereo matching algorithm (e.g., a normalized cross-correlation stereo matching algorithm). The calculation formula of the normalized cross-correlation stereo matching algorithm is as follows:

[0099]

[0100] In the above formula (5), f and h represent two image blocks of the same size in the speckle fringe images of the left and right cameras after epipolar correction; Respectively represent the mean of the grayscale values ​​in the f and h image blocks; x and y represent the horizontal and vertical coordinates of the pixel points in the image. The larger the δ(f,h) value, the stronger the correlation. Therefore, the pixel-level disparity map can be calculated using the normalized cross-correlation stereo matching method.

[0101] For a possible implementation, please refer to Figure 3 As shown, Figure 3 A schematic flow chart of an optional three-dimensional scanning method provided by the present application is shown, S104, based on predetermined camera calibration parameters and sub-pixel parallax values, determining three-dimensional point cloud information of the object under test, including:

[0102] S1041, based on the camera calibration parameters, convert the sub-pixel disparity value into three-dimensional point cloud information of the measured object.

[0103] In the above implementation, during the three-dimensional scanning process, the internal and external parameters of the camera can be obtained through camera calibration, which are collectively referred to as camera calibration parameters. Specifically, the internal parameters mainly describe the optical characteristics and imaging characteristics of the camera itself, such as focal length, principal point coordinates, pixel size, etc. The focal length determines the distance range in which the camera can clearly image and the imaging magnification; the principal point coordinates are the position of the center of the image plane in the camera coordinate system; the pixel size involves the actual physical size corresponding to each pixel in the image. External parameters are used to reflect the position and posture of the camera in three-dimensional space, that is, the rotation and translation relationship of the camera relative to a certain world coordinate system. For example, in a binocular camera system, the relative rotation and translation relationship between the two cameras is described by external parameters.

[0104] After obtaining the camera calibration parameters and calculating the sub-pixel disparity value, based on the principle of triangulation, the sub-pixel disparity value can be converted into three-dimensional point cloud information of the object under test. Specifically, in the example of this application, the binocular camera observes the object under test from different perspectives. Due to the different positions of the two cameras, the position of the same point on the object will be different in the images captured by the two cameras. This position difference is the disparity. Through the known camera calibration parameters (including the internal parameters of the two cameras and the external parameters between them), a mathematical relationship between the disparity and the depth information of the object in three-dimensional space (i.e., the distance to the camera plane) can be established.

[0105] For example, assuming that the angle between the optical axes of the two cameras (determined by the external parameters of the cameras) and the focal length of the cameras (internal parameters) are known, after obtaining the sub-pixel disparity value of a certain point in the two camera images, the depth coordinates of the pixel in the three-dimensional space can be calculated according to the triangulation formula. For example, for each pixel in the image, the corresponding three-dimensional point cloud information can be calculated according to the following steps:

[0106] First, the correspondence between the pixel points in the left and right camera images is determined based on the sub-pixel disparity value. Then, the image coordinates (pixel coordinates) are converted into coordinates in the camera coordinate system using the internal parameters of the camera. Next, the three-dimensional coordinates of the pixel point in the world coordinate system (including coordinates in the x, y, and z directions) are calculated using the triangulation formula, combining the external parameters between the two cameras and the coordinates in the camera coordinate system obtained previously. Repeat the above calculation process, and calculate all the pixel points in the image that can obtain valid sub-pixel disparity values ​​to obtain a series of three-dimensional coordinate points, which constitute the three-dimensional point cloud information of the object being measured.

[0107] By converting sub-pixel parallax values ​​into three-dimensional point cloud information of the object under test based on camera calibration parameters, the conversion from two-dimensional image information (image with speckle fringes collected by the camera) to three-dimensional space information is realized. Multiple factors such as the camera's optical properties, imaging properties, and the geometric relationship between cameras are taken into consideration. Therefore, the obtained three-dimensional point cloud information can well reflect the surface undulations, contours and other features of the object, and can more accurately restore the actual shape and structure of the object under test in three-dimensional space.

[0108] In addition, the calculation of sub-pixel disparity values ​​is to improve the accuracy of disparity measurement, and on this basis, the conversion of 3D point cloud information combined with camera calibration parameters can further reduce measurement errors. Moreover, the generated 3D point cloud information can be easily used for subsequent data processing operations, such as filtering, smoothing, generating visual images, etc. (as mentioned above). At the same time, this standard 3D point cloud data format is also easy to use in different software platforms and application scenarios, such as further modeling and rendering in 3D modeling software, or size measurement, shape comparison and other operations in engineering analysis software, thereby broadening the application scope of 3D scanning data.

[0109] As an optional example, in the example of this application, the three-dimensional point cloud calculated using the pixel-level disparity map has the problem of poor accuracy. In order to further obtain a high-precision three-dimensional point cloud, the sub-pixel disparity is further optimized and solved in combination with the relative phase based on the pixel-level disparity. Since the speckle fringe image is periodic and the relative phase value is in the range [0,2π), the phase value within a period is an increasing relationship. Therefore, based on this principle and the calculated pixel-level disparity map, the phase value of the pixel point (x, y) in the left camera image can be preliminarily determined. The corresponding pixel position in the right camera image. For the same row of the left camera phase image, assume that the phase value of the current pixel (x, y) is This includes the following situations:

[0110] when When The phase value obtained by fitting a quadratic curve with the coordinates x-1, x, and x+1 is equal to The coordinate value x L .

[0111] when When The phase value obtained by fitting a quadratic curve with the coordinates x-1, x, and x+1 is equal to The coordinate value x L .

[0112] when When The phase value obtained by fitting a quadratic curve with the coordinates x-1, x, and x+1 is equal to The coordinate value x L .

[0113] According to the pixel-level disparity map and Value, we can find the pixel point (x R ,y):

[0114] Condition 1: If Then take the phase value The phase value obtained by fitting the quadratic curve with the coordinate values ​​x+i-1, x+i, x+i+1, x+i+2 is equal to The coordinate value x R .

[0115] Condition 2: Then take the phase value The phase value obtained by fitting the quadratic curve with the coordinate values ​​x+i-1, x+i, x+i+1, x+i+2 is equal to The coordinate value x R .

[0116] Condition 3: and Then take the phase value The phase value obtained by fitting the quadratic curve with the coordinate values ​​x+i-1, x+i, x+i+1, x+i+2 is equal to The coordinate value x R .

[0117] Condition 4: Then take the phase value The phase value obtained by fitting the quadratic curve with the coordinate values ​​x+i-1, x+i, x+i+1, x+i+2 is equal to The coordinate value x R .

[0118] Condition 5: Then take the phase value The phase value obtained by fitting the quadratic curve with the coordinate values ​​x+i-1, x+i, x+i+1, x+i+2 is equal to The coordinate value x R .

[0119] Assumptions The initial pixel-level disparity value is d, satisfying the condition d-1≤|x L -x R |≤d+1 coordinate value x L With x R Used to calculate the sub-pixel disparity value d subpixel =|x L -x R |. Using the system calibration parameters of the binocular camera, the sub-pixel parallax can be converted into a 3D point cloud.

[0120] The principle of micro-phase high-frequency fringe encoding in the embodiment of the present application is to generate a three-step high-frequency fringe image by a phase shift method, randomly generate a speckle image, reasonably select the grayscale of the speckle spots according to the high-frequency fringe image, and interpolate the speckle image into the high-frequency fringe image without affecting the relative phase solution of the speckle fringe image collected by the camera. The two-step high-precision disparity calculation method proposed in the embodiment of the present application first uses a binocular stereo matching method to match the speckle fringe image after epipolar correction to calculate the pixel-level disparity, and based on the pixel-level disparity and the relative phase of the left and right cameras, the quadratic curve fitting method is used to further optimize the disparity value to obtain the sub-pixel level disparity. The binocular micro-phase structured light three-dimensional scanning method proposed in the embodiment of the present application can effectively reduce the influence of global illumination effects such as defocus effect, mutual reflection effect and sub-surface scattering effect.

[0121] Compared with the prior art, the embodiment of the present application utilizes binocular speckle combined with a stereoscopic vision method, and can solve the problem of accurate parallax calculation without relative phase unwrapping, thereby reducing the number of projected images to 3, facilitating rapid dynamic scanning, and having the characteristics of rapidity, accuracy, and strong robustness. If the relative phase calculation accuracy of the two-step phase shift method can be improved, the embodiment of the present application can be extended to a two-step phase shift method, that is, the number of projected images can be reduced to 2 at least, achieving faster dynamic scanning.

[0122] For a possible implementation, please refer to Figure 4 As shown, Figure 4 A schematic flow chart of an optional three-dimensional scanning method provided by the present application is shown, the method further comprising:

[0123] S301, adjusting parameters of an initial speckle pattern according to surface characteristics of a measured object and / or measurement environment factors to obtain a predetermined sparse speckle pattern.

[0124] The predetermined sparse speckle pattern includes at least one of the following: speckle spots, speckle lines, speckle stripes, and speckle grids, and the parameters include at least one of the following: distribution density, size, and grayscale.

[0125] S302, inserting a predetermined sparse speckle pattern into the high-frequency fringe image.

[0126] S303, projecting the high-frequency fringe image onto the surface of the object to be measured by a projector.

[0127] In this application example, before performing a three-dimensional scan, the surface characteristics of the object to be measured and the measurement environment factors are carefully analyzed and evaluated. In the specific implementation scenario, the surface characteristics of the object to be measured may include surface roughness, reflectivity, color and other aspects. For example, objects with rough surfaces can scatter light differently from smooth surfaces; objects with high reflectivity will produce strong reflections, affecting image acquisition effects; objects of different colors also have differences in light absorption and reflection. Measurement environment factors cover the intensity of ambient light, color temperature, whether there are interfering light sources and other content. For example, in a strong light environment, the projected stripes and speckle patterns will be obscured, and in an environment with colored light interference, the color perception of the image will be changed.

[0128] In one example, if the surface of the object being measured is relatively rough, in order to better capture its surface features, the distribution density of the speckle pattern can be appropriately increased so that the speckle can cover the surface of the object more evenly and increase the number of feature points that can be used for analysis. On the contrary, for objects with relatively smooth surfaces, too high a distribution density may cause the speckle to be too dense, increasing the complexity of image analysis. In this case, the distribution density can be appropriately reduced. For example, when scanning a rough rock surface, the distribution density of the speckle pattern is set higher; when scanning a smooth metal plate, the distribution density is appropriately reduced.

[0129] In another example, when the surface of the object being measured is rich in details and small in size, such as some tiny mechanical parts, in order to clearly distinguish these details, the size of the speckle will be reduced so that the speckle can fit the tiny features of the object surface more accurately. For larger objects with relatively simple surface features, the size of the speckle can be appropriately increased to improve the recognition of the speckle pattern in the image.

[0130] In another example, for objects with high reflectivity, such as reflective metal products, in order to prevent the speckle pattern from being too bright and difficult to distinguish in the image, the gray value of the speckle can be reduced to provide a better contrast with the reflection of the object surface. In a measurement environment with low light, in order to ensure that the speckle pattern can be clearly seen in the image, the gray value of the speckle can be appropriately increased.

[0131] By flexibly adjusting the parameters of the speckle pattern according to the surface characteristics of the object being measured and the measurement environment factors, the generated predetermined sparse speckle pattern can be better adapted to different scanning scenarios, thereby improving the adaptability of the three-dimensional scanning method to various objects and environments, so that in the subsequent image acquisition, analysis and processing processes, the information on the surface of the object can be more accurately obtained, thereby improving the accuracy of three-dimensional scanning.

[0132] After generating a predetermined sparse speckle pattern that meets the requirements, it is inserted into the high-frequency fringe image. Specifically, the pixel values ​​of the speckle pattern can be combined with the pixel values ​​of the high-frequency fringe image according to certain rules based on the corresponding relationship between the pixel coordinates of the high-frequency fringe image and the predetermined sparse speckle pattern. For example, a weighted average method can be adopted to perform weighted summation of the pixel values ​​of the speckle pattern and the corresponding pixel values ​​of the high-frequency fringe image according to a set weight coefficient to obtain the pixel values ​​of the high-frequency fringe image after the speckle pattern is inserted. Alternatively, a direct replacement method can be adopted to replace the original pixel values ​​in a specific area of ​​the high-frequency fringe image with the pixel values ​​of the speckle pattern, thereby realizing the insertion of the speckle pattern.

[0133] During the insertion process, it is necessary to ensure the compatibility of the speckle pattern with the high-frequency fringe image, that is, the inserted image can still be processed according to the established encoding and decoding rules. Therefore, it is necessary to fully consider the encoding method of the high-frequency fringe image and the needs of subsequent image decoding, analysis and other operations. For example, the inserted speckle pattern cannot destroy the original phase information in the high-frequency fringe image, so that key data such as relative phase values ​​can be accurately obtained in subsequent processing.

[0134] Inserting a predetermined sparse speckle pattern into a high-frequency fringe image can further enrich the feature information of the image. The high-frequency fringe image itself provides fringe-based phase information for three-dimensional measurement, while the inserted speckle pattern adds speckle-related features, such as the distribution, size, and grayscale of the speckle. These rich features can provide more references in the subsequent image analysis process, helping to improve the accuracy and reliability of three-dimensional scanning.

[0135] The high-frequency fringe image is projected onto the surface of the object to be measured by a projector, which may include adjusting the focal length, brightness, contrast and other parameters of the projector. At the same time, the projector may be calibrated so that its position and angle relationship with the binocular camera assembly meets the requirements of three-dimensional scanning. For example, it is ensured that the optical axis of the projector and the optical axis of the binocular camera assembly are within a certain angle range so that the image can be accurately captured by the camera after projection.

[0136] During the projection process, the projection distance or angle of the projector can be adjusted according to the size of the object to be measured to ensure that the effective measurement area of ​​the object can be covered. At the same time, according to the shape and undulation of the object surface, the brightness and other parameters of the projector can be adjusted in time to ensure that the projected image can be clearly seen on the surface of the object, and the image in some areas will not be blurred or missing due to the unevenness of the object surface.

[0137] In the example of this application, the projector projects a high-frequency fringe image onto the surface of the object to be measured, forming an image with speckle fringes on the surface of the object, providing the original image data for the subsequent acquisition, decoding, matching calculation and other operations of the binocular camera component, thereby realizing the three-dimensional scanning measurement of the object to be measured. In addition, by timely adjusting the parameters of the projector to adapt to the shape and undulation of the object surface, it can be ensured that the projected image presents a good effect on the surface of the object, so that the subsequent acquired image can truly reflect the actual situation of the object surface, thereby improving the accuracy and reliability of three-dimensional scanning.

[0138] For a possible implementation, please refer to Figure 5 As shown, Figure 5 A schematic flow chart of an optional three-dimensional scanning method provided by the present application is shown. After determining the three-dimensional point cloud information of the object to be measured based on predetermined camera calibration parameters and sub-pixel parallax values, the method further includes:

[0139] S401, post-processing the three-dimensional point cloud information, wherein the post-processing includes at least one of the following: filtering processing, smoothing processing, and data compression processing.

[0140] Optionally, in the example of this application, filtering processing is intended to remove noise points in the three-dimensional point cloud information. For example, specific filtering methods that can be used include: statistical filtering, radius filtering, bilateral filtering, etc. Through filtering processing, the quality of three-dimensional point cloud information can be effectively improved, making subsequent point cloud-based analysis, modeling and other operations more accurate. After removing noise points, the point cloud data is purer and can more realistically reflect the surface shape and structure of the measured object, reducing misjudgments or inaccurate modeling results caused by noise interference.

[0141] Optionally, in the example of this application, the smoothing process is mainly to make the surface of the object represented by the three-dimensional point cloud information smoother and reduce the roughness and discontinuity of the surface. For example, smoothing methods include: average smoothing, Gaussian smoothing, etc. The smoothed point cloud can more intuitively display the approximate shape of the measured object, especially for some objects with slight irregularities on the surface. After smoothing, its three-dimensional shape is visually clearer and more continuous, which is convenient for subsequent visualization and further analysis and processing, such as shape matching, size measurement and other operations.

[0142] Optionally, in the example of this application, data compression processing is to reduce the storage space of point cloud data without affecting the main features and use effects of point cloud information. In practical applications, especially for large-scale three-dimensional point cloud data, data compression can improve the efficiency of data processing and reduce storage costs. At the same time, when point cloud data is needed in the future, point cloud information that meets application requirements can be restored through decompression operations.

[0143] S402, generating a visualization image based on the three-dimensional point cloud information, wherein the visualization image is used to display the three-dimensional shape or structure of the object being measured in at least one of a point cloud image, a grid image, and a solid model.

[0144] Optionally, the point cloud map is an image directly drawn by drawing the points in the three-dimensional point cloud information according to their coordinate positions in the three-dimensional space. Each point cloud can be represented by a small dot, and the color of the point cloud can be set according to different attributes, such as the height of the point, the distance from the camera, the category of the point (if classified), etc. By displaying these points in three-dimensional space, you can intuitively see the approximate shape of the object being measured and the distribution of the points. Specifically, the point cloud map can quickly and intuitively present the three-dimensional shape of the object being measured. It is especially suitable for preliminary observation and analysis of the shape of the object, and can show some detailed features of the surface of the object, such as protrusions, depressions, holes, etc.

[0145] Optionally, generating a mesh map requires first performing triangulation on the 3D point cloud information, and converting the point cloud into a mesh structure composed of triangular facets. For example, by connecting adjacent points into triangular facets to form a continuous mesh surface, and then rendering the mesh facets according to the attributes of the points (such as color, normal direction, etc.), a realistic mesh map can be obtained. Compared with point cloud maps, mesh maps are smoother and more continuous, and can better show the overall shape and surface features of objects.

[0146] Optionally, based on the mesh map, the mesh map is first converted into a format that can be used for 3D modeling (such as STL format, etc.), and then processed using professional 3D modeling software. In the 3D modeling software, various operations can be performed on the model as needed, such as adding details, adjusting the shape, setting the material, etc., and finally a complete solid model is generated to use the solid model to display the 3D shape and structure of the object being measured.

[0147] By post-processing the 3D point cloud information and generating different forms of visual images, the 3D shape or structure of the object being measured can be better displayed, meeting the needs of different application scenarios and improving the practicality and usability of 3D scanning data.

[0148] Through the three-dimensional scanning method provided by the example of this application, the structured light three-dimensional scanner does not directly use laser to measure the distance, but projects a pre-inserted specific sparse speckle pattern onto the surface of the object to be measured. For example, the above-mentioned specific sparse speckle pattern can be points, lines, stripes or grids. Due to the ups and downs of the surface of the object, the projected structured light will be deformed. The scanner captures these deformed patterns through the built-in camera and calculates the depth information of the surface of the object using the principle of triangulation. The embodiment of the present application projects a high-frequency cosine fringe image with a specific sparse speckle pattern inserted, uses high-frequency stripes to reduce the influence of the global illumination effect, and uses binocular speckle image matching and phase matching to achieve sub-pixel parallax calculation, thereby achieving fast and high-precision three-dimensional scanning.

[0149] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0150] Corresponding to the three-dimensional scanning method of the above embodiment, Figure 6 is a schematic diagram of the structure of a three-dimensional scanning device provided in an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be Figure 7 Electronic equipment shown.

[0151] Reference Figure 6 , the three-dimensional scanning device comprises:

[0152] A projector 601 is used to project a high-frequency fringe image inserted with a predetermined sparse speckle pattern onto the surface of the object to be measured;

[0153] A binocular camera assembly 602, comprising a first camera and a second camera disposed on both sides of the projector, for capturing the surface of the object to be measured to obtain a speckle fringe image;

[0154] The image processing module 603 is used to decode the speckle fringe image to obtain the relative phase value between the first camera and the second camera, and calculate the sub-pixel disparity value of the speckle fringe image according to the speckle fringe image and the relative phase value;

[0155] The three-dimensional information generating module 604 is used to determine the three-dimensional point cloud information of the measured object based on the predetermined camera calibration parameters and the sub-pixel parallax value.

[0156] Further, based on any of the above embodiments, as an example of the present application, the image processing module further includes:

[0157] An epipolar correction unit, used for performing epipolar correction processing on the speckle fringe image by using an epipolar correction algorithm to obtain an epipolar corrected speckle fringe image;

[0158] A pixel-level disparity calculation unit is used to perform matching calculation on the speckle fringe image after epipolar line correction using a binocular stereo matching algorithm to obtain an initial pixel-level disparity value;

[0159] The sub-pixel disparity optimization unit is used to calculate the sub-pixel disparity value of the speckle fringe image based on the initial pixel-level disparity value and the relative phase value by using a quadratic curve fitting algorithm.

[0160] Further, based on any of the above embodiments, as an example of the present application, determining the three-dimensional point cloud information of the object under test based on predetermined camera calibration parameters and sub-pixel parallax values ​​includes:

[0161] Based on the camera calibration parameters, the sub-pixel disparity value is converted into the three-dimensional point cloud information of the measured object.

[0162] Further, based on any of the above embodiments, as an example of the present application, the device is further specifically used for:

[0163] According to the surface characteristics of the measured object and / or the measurement environment factors, the parameters of the initial speckle pattern are adjusted to obtain a predetermined sparse speckle pattern, wherein the predetermined sparse speckle pattern includes at least one of the following: speckle spots, speckle lines, speckle stripes, and speckle grids, and the parameters include at least one of the following: distribution density, size, and grayscale;

[0164] inserting a predetermined sparse speckle pattern into the high-frequency fringe image;

[0165] The high-frequency fringe image is projected onto the surface of the object to be measured by a projector.

[0166] Further, based on any of the above embodiments, as an example of the present application, the device is further specifically used for:

[0167] Post-processing the three-dimensional point cloud information, wherein the post-processing includes at least one of the following: filtering processing, smoothing processing, and data compression processing; and

[0168] A visualization image is generated based on the three-dimensional point cloud information, wherein the visualization image is used to display the three-dimensional shape or structure of the object being measured in at least one of a point cloud image, a grid image, and a solid model.

[0169] It should be noted that the three-dimensional scanning device provided in the above embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0170] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0171] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0172] An embodiment of the present application further provides an electronic device, the electronic device comprising one or more processors and a memory;

[0173] The memory is coupled to one or more processors, and the memory is used to store computer program codes, the computer program codes include computer instructions, and one or more processors call the computer instructions to enable the electronic device to execute the three-dimensional scanning method shown above.

[0174] Figure 7 The schematic diagram of the structure of an electronic device provided in the embodiment of the present application is that the electronic device 700 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, or a communication device such as a server, a storage device, a base station, or a smart car, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0175] The memory 701 can be used to store computer software programs 702 and modules, and the processor 703 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 701. The memory 701 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 701 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0176] Among them, the processor 703 may include one or more processors such as a central processing unit, an application processor (AP), a baseband processor, etc. The processor may be the nerve center and command center of the wireless router. The processor 703 may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. The memory 701 may be used to store computer executable program codes, and the executable program codes include instructions. The processor 703 executes various functional applications and data processing of the network device by running the instructions stored in the memory. The memory 701 may include a program storage area and a data storage area, such as storing data of a sound signal to be played. For example, the memory may be a double rate synchronous dynamic random access memory DDR or a flash memory Flash.

[0177] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium is run on an electronic device, the electronic device executes the three-dimensional scanning method shown above.

[0178] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0179] The embodiment of the present application also provides a computer program product including computer instructions. When the computer program product is run on an electronic device, the electronic device can execute the three-dimensional scanning method shown above.

[0180] The computer storage medium and computer program product provided in the above-mentioned embodiments of the present application are used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the method provided above, and will not be repeated here.

[0181] In the above embodiments, it can also be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0182] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0183] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments applied for herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0184] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-dimensional scanning method, characterized in that: include: After the high-frequency fringe image is projected onto the surface of the object to be measured by a projector, the surface of the object to be measured is captured based on a binocular camera assembly to obtain a speckle fringe image; wherein a predetermined sparse speckle pattern is inserted into the high-frequency fringe image, and the binocular camera assembly includes: a first camera and a second camera arranged on both sides of the projector; Decoding the speckle fringe image to obtain a relative phase value between the first camera and the second camera; Calculating a sub-pixel disparity value of the speckle fringe image according to the speckle fringe image and the relative phase value; Based on predetermined camera calibration parameters and the sub-pixel parallax value, three-dimensional point cloud information of the object under test is determined.

2. The method according to claim 1, characterized in that The calculating, according to the speckle fringe image and the relative phase, a sub-pixel disparity value of the speckle fringe image comprises: Performing epipolar correction processing on the speckle fringe image by using an epipolar correction algorithm to obtain an epipolar corrected speckle fringe image; Using a binocular stereo matching algorithm to perform matching calculation on the epipolar corrected speckle fringe image to obtain an initial pixel-level disparity value; A quadratic curve fitting algorithm is adopted to calculate the sub-pixel disparity value of the speckle fringe image based on the initial pixel-level disparity value and the relative phase value.

3. The method according to claim 1, characterized in that: The determining of the three-dimensional point cloud information of the object under test based on the predetermined camera calibration parameters and the sub-pixel parallax value includes: Based on the camera calibration parameters, the sub-pixel disparity value is converted into three-dimensional point cloud information of the measured object.

4. The method according to claim 1, characterized in that: The method further comprises: According to the surface characteristics of the measured object and / or the measurement environment factors, the parameters of the initial speckle pattern are adjusted to obtain the predetermined sparse speckle pattern, wherein the predetermined sparse speckle pattern includes at least one of the following: speckle spots, speckle lines, speckle stripes, and speckle grids, and the parameters include at least one of the following: distribution density, size, and grayscale; inserting the predetermined sparse speckle pattern into the high-frequency fringe image; The high-frequency fringe image is projected onto the surface of the object to be measured by the projector.

5. The method according to claim 1, characterized in that: After determining the three-dimensional point cloud information of the measured object based on the predetermined camera calibration parameters and the sub-pixel parallax value, the method further includes: Post-processing the three-dimensional point cloud information, wherein the post-processing includes at least one of the following: filtering processing, smoothing processing, and data compression processing; and A visualization image is generated according to the three-dimensional point cloud information, wherein the visualization image is used to display the three-dimensional shape or structure of the object being measured in at least one of a point cloud image, a grid image, and a solid model.

6. A three-dimensional scanning device, characterized in that: include: A projector, used for projecting a high-frequency fringe image with a predetermined sparse speckle pattern inserted onto a surface of the object to be measured; A binocular camera assembly, comprising a first camera and a second camera disposed on both sides of the projector, for capturing the surface of the object to be measured to obtain a speckle fringe image; an image processing module, configured to decode the speckle fringe image to obtain a relative phase value between the first camera and the second camera, and calculate a sub-pixel disparity value of the speckle fringe image according to the speckle fringe image and the relative phase value; The three-dimensional information generation module is used to determine the three-dimensional point cloud information of the object under test based on predetermined camera calibration parameters and the sub-pixel parallax value.

7. The three-dimensional scanning device according to claim 6, characterized in that: The image processing module further comprises: an epipolar correction unit, configured to perform epipolar correction processing on the speckle fringe image using an epipolar correction algorithm to obtain an epipolar corrected speckle fringe image; A pixel-level disparity calculation unit, used for performing matching calculation on the epipolar-corrected speckle fringe image using a binocular stereo matching algorithm to obtain an initial pixel-level disparity value; The sub-pixel disparity optimization unit is used to calculate the sub-pixel disparity value of the speckle fringe image based on the initial pixel-level disparity value and the relative phase value by using a quadratic curve fitting algorithm.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The invention comprises a computer program which, when executed, causes the method according to any one of claims 1 to 5 to be performed.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images