Three-dimensional reconstruction method, device and equipment

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method and device, equipment and a storage medium. A mark point can be arranged on the surface of a target object, and structured light is projected to the surface of the target object in at least partial time period of scanning the target object so as to acquire an image frame set of the target object. And reconstructing based on each image frame containing the mark point pattern to obtain a three-dimensional point cloud of each frame of mark point, and reconstructing based on each image frame containing the structured light pattern to obtain a three-dimensional point cloud of each frame of target object. Since the coordinate transformation relationship between the three-dimensional point clouds of each frame of mark point can be accurately determined, and the coordinate transformation relationship between the three-dimensional point clouds of the single frame of mark point and the three-dimensional point clouds of the single frame of target object can also be determined, the splicing of the three-dimensional point clouds of the multiple frames of target objects can be assisted by means of the mark points; and thus, a more accurate splicing result can be obtained.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of three-dimensional scanning technology, and in particular, to a three-dimensional reconstruction method, device, equipment and storage medium. Background Art

[0002] In the process of three-dimensionally reconstructing a target object using the data collected by a three-dimensional scanning device, it is usually necessary to use the three-dimensional scanning device to scan the target object multiple times from different perspectives to obtain multiple frames of local point cloud data, and then use the multiple frames of local point cloud data to splice to obtain the global point cloud data of the target object.

[0003] In the related art, the common splicing method is geometric splicing, that is, using the geometric features of the target object itself for splicing. This splicing method requires the target object to have rich and non-repetitive geometric features. When scanning some geometrically regular target objects, such as cylindrical bases, planting rods, etc., which have repetitive geometric features or weak feature information, due to the limited number of feature points extracted, a high-precision splicing result cannot be calculated, resulting in a poor accuracy of the finally reconstructed three-dimensional model. Summary of the Invention

[0004] The embodiments of the present application provide a three-dimensional reconstruction method, device, equipment and storage medium.

[0005] According to the first aspect of the embodiments of the present application, a three-dimensional reconstruction method is provided. The method includes:

[0006] Obtain a set of image frames collected during the process of a three-dimensional scanning device scanning a target object, wherein marker points are provided on the surface of the target object, and structured light is projected onto the surface of the target object during at least part of the time period of scanning the target object;

[0007] Perform three-dimensional reconstruction on the marker points based on the image frames including marker point patterns in the set of image frames to obtain three-dimensional point clouds of multiple frames of marker points, and determine a first coordinate transformation relationship between the three-dimensional point clouds of the multiple frames of marker points;

[0008] Perform three-dimensional reconstruction on the target object based on the image frames including structured light patterns in the set of image frames to obtain three-dimensional point clouds of multiple frames of the target object;

[0009] For each three-dimensional point cloud of the target object, determine a second coordinate transformation relationship between the three-dimensional point cloud of the target object and the three-dimensional point cloud of one of the frames of marker points, and splice the three-dimensional point clouds of the multiple frames of the target object based on the first coordinate transformation relationship and the second coordinate transformation relationship.

[0010] According to the second aspect of the embodiments of the present application, a three-dimensional reconstruction device is provided. The three-dimensional reconstruction device includes:

[0011] An acquisition module, configured to acquire a set of image frames collected during the process of a three-dimensional scanning device scanning a target object, wherein marker points are arranged on the surface of the target object, and structured light is projected onto the surface of the target object during at least a part of the time period of scanning the target object;

[0012] A three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the marker points based on the image frames including marker point patterns in the set of image frames to obtain three-dimensional point clouds of multiple frames of marker points, and determine a first coordinate transformation relationship between the three-dimensional point clouds of the multiple frames of marker points; and perform three-dimensional reconstruction on the target object based on the image frames including structured light patterns in the set of image frames to obtain three-dimensional point clouds of multiple frames of the target object;

[0013] A stitching module, configured to, for each three-dimensional point cloud of the target object, determine a second coordinate transformation relationship between the three-dimensional point cloud of the target object and the three-dimensional point cloud of one of the marker points, and stitch the three-dimensional point clouds of the multiple frames of the target object based on the first coordinate transformation relationship and the second coordinate transformation relationship.

[0014] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including a processor, a memory, and computer instructions stored in the memory and executable by the processor. When the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0015] According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are executed, the method mentioned in the first aspect above is implemented.

[0016] In the embodiments of the present application, marker points can be arranged on the surface of the target object, and structured light is projected onto the surface of the target object during at least a part of the time period of scanning the target object, so as to acquire a set of image frames of the target object. Then, the three-dimensional point clouds of each frame of marker points can be reconstructed based on each image frame including a marker point pattern, and the three-dimensional point clouds of each frame of the target object can be reconstructed based on each image frame including a structured light pattern. Since the coordinate transformation relationship between the three-dimensional point clouds of each frame of marker points can be accurately determined, and the coordinate transformation relationship between the three-dimensional point cloud of a single frame of marker points and the three-dimensional point cloud of a single frame of the target object can also be determined, the stitching of the three-dimensional point clouds of the multiple frames of the target object can be assisted by the marker points, so as to obtain a more accurate stitching result.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the embodiments of the present application. Description of the Drawings

[0018] The accompanying drawings here are incorporated into the specification and form a part of this specification. These drawings show embodiments that conform to the embodiments of the present application, and are used together with the specification to illustrate the technical solutions of the embodiments of the present application.

[0019] Figure 1 It is a schematic diagram of an application scenario of an embodiment of the present application.

[0020] Figure 2 It is a flowchart of a three-dimensional reconstruction method of an embodiment of the present application.

[0021] Figure 3 It is a schematic diagram of a fiducial point of an embodiment of the present application.

[0022] Figure 4 It is a schematic diagram of the fiducial point being blocked by a structured light pattern in an embodiment of the present application.

[0023] Figure 5 It is a schematic diagram of simultaneously collecting a structured light pattern and a fiducial point pattern in an embodiment of the present application.

[0024] Figure 6 It is a schematic diagram of collecting a structured light pattern and a fiducial point pattern in sequence in an embodiment of the present application.

[0025] Figure 7 It is another schematic diagram of collecting a structured light pattern and a fiducial point pattern in sequence in an embodiment of the present application.

[0026] Figure 8 It is a schematic diagram of a three-dimensional reconstruction device of an embodiment of the present application.

[0027] Figure 9 It is a schematic logical structure diagram of an electronic device of an embodiment of the present application. Detailed implementation manners

[0028] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0029] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms "a", "the", and "said" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. Additionally, the term "at least one" as used herein represents any one of a plurality or any combination of at least two of a plurality.

[0030] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to a determination".

[0031] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application and make the above-mentioned objectives, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] The three-dimensional reconstruction technology based on structured light projects an optical coding pattern (i.e., structured light) onto the target object to be reconstructed, and determines the depth information of the surface of the target object based on the deformed pattern of the structured light collected by the camera, so as to restore the three-dimensional data of the surface of the target object. The three-dimensional reconstruction technology based on structured light has characteristics such as high efficiency and anti-interference, and is widely used in various three-dimensional reconstruction scenarios. During the scanning process, limited by the size of the projection area of the structured light or the shooting field of view of the camera, usually only local three-dimensional point clouds of the target object can be obtained in a single measurement. To obtain the global three-dimensional point cloud of the target object, the local point clouds collected from multiple perspectives need to be stitched together.

[0033] Currently, the commonly used stitching method is geometric stitching, that is, using the geometric features of the target object itself for stitching. This stitching method requires the target object to have rich and non-repetitive geometric features. When scanning some geometrically regular target objects, such as cylindrical pedestals, planting rods, scanning rods, etc., which have repetitive geometric features or weak feature information, due to the limited number of feature points extracted, a high-precision stitching result cannot be calculated, resulting in a poor accuracy of the finally reconstructed three-dimensional model.

[0034] In order to improve the stitching accuracy, in the embodiments of the present application, it is considered that for some target objects with weak geometric rules and feature information, fiducial points can be set on the surface of the target object, and structured light can be projected onto the surface of the target object during at least part of the time period of scanning the target object, so as to collect a set of image frames of the target object. Then, the three-dimensional point cloud of each frame of fiducial points is reconstructed based on each image frame containing the fiducial point pattern, and the three-dimensional point cloud of each frame of the target object is reconstructed based on each image frame containing the structured light pattern. Since the coordinate transformation relationship between the three-dimensional point clouds of each frame of fiducial points can be accurately determined, and the coordinate transformation relationship between the three-dimensional point cloud of a single frame of fiducial points and the three-dimensional point cloud of a single frame of the target object can also be determined, thus, the stitching of the three-dimensional point clouds of multiple frames of the target object can be assisted by the fiducial points to obtain a more accurate stitching result.

[0035] The three-dimensional reconstruction method provided by the embodiments of the present application can be executed by a three-dimensional scanning device, or by other devices connected to the three-dimensional scanning device wirelessly or wiredly. The other devices can be mobile phones, tablets, laptop computers, cloud servers or server clusters, etc.

[0036] For example, in some scenarios, if the computing power of the three-dimensional scanning device itself is good, it can collect images of the target object and perform three-dimensional reconstruction on the target object using the collected images to obtain a three-dimensional model of the target object.

[0037] In some scenarios, if the computing power of the three-dimensional scanning device itself is poor, three-dimensional reconstruction can be achieved by relying on other devices. For example, scanning software can be installed on the other device, and the scanning software can perform real-time three-dimensional reconstruction based on the images collected by the three-dimensional scanning device and display the reconstruction result to the user.

[0038] For example, as Figure 1 shown, it is a schematic diagram of an application scenario of the embodiments of the present application. Considering that the computing power of the three-dimensional scanning device (taking an oral scanner as an example in the figure) is weak, the images collected by the three-dimensional scanning device are usually sent to a terminal device with better processing performance (such as a laptop computer) communicatively connected thereto. The terminal device can install scanning software, and the scanning software can perform real-time three-dimensional reconstruction based on the collected images and display the reconstruction result to the user.

[0039] The three-dimensional scanning device in the embodiments of the present application can be an oral scanner, a facial scanner, an industrial scanner, a professional scanner, and can be used for three-dimensional scanning and three-dimensional reconstruction of items such as teeth, human faces, human bodies, industrial products, industrial equipment, cultural relics, artworks, prosthetics, medical appliances, and buildings.

[0040] The target object in the embodiments of the present application can be various objects to be reconstructed provided with fiducial points. For example, it can be teeth, implant posts, scanning posts, abutments, industrial products, etc., and the embodiments of the present application do not limit this.

[0041] As Figure 2 shown, the 3D reconstruction method may include the following steps:

[0042] S202. Obtain a set of image frames collected during the scanning of the target object by a 3D scanning device, wherein fiducial points are provided on the surface of the target object, and structured light is projected onto the surface of the target object during at least part of the time period of scanning the target object;

[0043] In step S202, a set of image frames collected during the scanning of the target object by a 3D scanning device can be obtained. Among them, the image frames in the set of image frames can be collected by one or more cameras. Fiducial points are provided on the surface of the target object. The fiducial points can be formed on the surface of the target object by various methods such as projection, spraying, or pasting. The shape, color, quantity, size, etc. of the fiducial points can be set based on actual needs, and the embodiments of the present application do not limit this. As Figure 3 shown, in some scenarios, the fiducial points may not carry coding information ( Figure 3 the leftmost point), and in some scenarios, in order to better distinguish each fiducial point on the surface of the target object, the fiducial points can also be fiducial points carrying coding information ( Figure 3 the two points on the right).

[0044] Among them, in order to obtain the depth information of the surface of the target object, structured light is projected onto the surface of the target object during at least part of the time period of scanning the target object. Among them, the structured light can be structured light of various patterns such as dot matrices, lines, and light spots, and the embodiments of the present application do not limit this.

[0045] Among them, structured light can be projected onto the target object throughout the scanning process, or structured light can be projected onto the target object only during part of the time period of the scanning process. For example, in order to ensure that each image frame collected includes a structured light pattern, structured light can be projected onto the target object throughout the scanning process. Of course, in some scenarios, in order to make some image frames not include a structured light pattern, structured light can also not be projected onto the target object at some moments.

[0046] S204. Perform 3D reconstruction on the fiducial points based on the image frames including fiducial point patterns in the set of image frames to obtain a 3D point cloud of multiple frames of fiducial points, and determine a first coordinate transformation relationship between the 3D point clouds of the multiple frames of fiducial points;

[0047] In step S204, after obtaining the set of image frames, an image frame including the fiducial point image can be obtained from the image frames, and 3D reconstruction of the fiducial points can be performed based on the image frame including the fiducial point pattern to obtain the 3D point clouds of multiple frames of fiducial points. Among them, one image frame including the fiducial point pattern corresponds to one 3D point cloud of the fiducial points.

[0048] For example, in some scenarios, the 3D scanning device includes at least two cameras. The fiducial points on the surface of the target object can be image-captured by the at least two cameras to obtain multiple sets of image frame sequences including the fiducial points. For each fiducial point, the depth information of the pixel point corresponding to the fiducial point can be obtained based on the parallax of the pixel points corresponding to the fiducial point in the images captured by the at least two cameras. Furthermore, the 3D coordinates of the fiducial point can be determined. In the above manner, the 3D point cloud corresponding to the fiducial point in each image frame can be determined, and then the coordinate transformation relationship between different image frames (i.e., the 3D point clouds of different frames of fiducial points) can be determined through the 3D coordinates of the 3D point clouds of the same fiducial point in different image frames, hereinafter referred to as the first coordinate transformation relationship.

[0049] In some scenarios, the 3D scanning device may also include only a single camera. Then, the images of the fiducial points can be captured by the single camera at different perspectives. For each fiducial point, the depth information of the pixel point corresponding to the fiducial point can be obtained based on the parallax of the pixel points corresponding to the fiducial point in the images captured by the camera at different perspectives, so that the 3D point cloud of the single frame of fiducial points corresponding to each image frame can be determined, and the first coordinate transformation relationship between the 3D point clouds of the multiple frames of fiducial points can be determined.

[0050] Among them, in some scenarios, the 3D coordinates of each fiducial point may be the 3D coordinates of the center of each fiducial point pattern. For example, taking the fiducial point carrying the coding information as shown in Figure 3 as an example, the fiducial point may be composed of different patterns. For such a fiducial point, the center of the pattern of the fiducial point can be extracted, and 3D reconstruction can be performed on the center point, and the coordinates of the center point obtained by the reconstruction are used as the 3D coordinates of the fiducial point.

[0051] S206. Perform 3D reconstruction on the target object based on the image frame including the structured light pattern in the set of image frames to obtain the 3D point clouds of multiple frames of the target object;

[0052] In step S206, after obtaining the set of image frames, an image frame including a structured light pattern can be obtained from the set of image frames, and then based on these image frames, three-dimensional reconstruction of the target object is performed to obtain a multi-frame three-dimensional point cloud of the target object. Among them, for each image frame, the depth information of the surface of the target object in the image frame can be determined based on the structured light pattern in the image frame, so that a single-frame three-dimensional point cloud of the target object can be obtained. Among them, the image frame including the structured light pattern can also be collected by one or more cameras.

[0053] S208. For the three-dimensional point cloud of each frame of the target object, determine the second coordinate transformation relationship between the three-dimensional point cloud of this frame of the target object and the three-dimensional point cloud of one frame of the fiducial points, and splice the three-dimensional point clouds of the multi-frame target objects based on the first coordinate transformation relationship and the second coordinate transformation relationship.

[0054] In step S208, since the coordinate transformation relationship between the three-dimensional point clouds of different frames of fiducial points can be determined, therefore, the coordinate transformation relationship between the three-dimensional point clouds of different frames of the target object can be determined with the help of the coordinate transformation relationship between the three-dimensional point clouds of different frames of fiducial points. For example, for the three-dimensional point cloud of each frame of the target object, the coordinate transformation relationship between the three-dimensional point cloud of this frame of the target object and the three-dimensional point cloud of one frame of the fiducial points can be determined. For the sake of easy distinction, it is hereinafter referred to as the second coordinate transformation relationship. Among them, there is a certain relationship between the three-dimensional point cloud of one frame of the fiducial points and the three-dimensional point cloud of this frame of the target object. For example, both are reconstructed based on the same image frame, both are collected by different cameras at the same time, or the time interval between their collections is short, etc. Then, the coordinate transformation relationship between the three-dimensional point clouds of the multi-frame target objects can be determined based on the first coordinate transformation relationship and the second coordinate transformation relationship, and further, the coordinate systems between the three-dimensional point clouds of the multi-frame target objects can be unified to splice the three-dimensional point clouds of the multi-frame target objects.

[0055] By setting fiducial points on the surface of the target object, since the coordinate transformation relationship between the three-dimensional point clouds of different frames of fiducial points can be accurately determined, the accurate splicing between the three-dimensional point clouds of different frames of the target object can be assisted by using the fiducial points, and an accurate splicing result can be obtained.

[0056] In some embodiments, such as Figure 4As shown, when the three-dimensional scanning device scans a target object, it can simultaneously collect the structured light pattern and the fiducial point pattern projected on the surface of the target object, that is, the image frame set includes a plurality of composite image frames, and each composite image frame includes both the structured light pattern and the fiducial point pattern. Since the structured light pattern and the fiducial point pattern are located in the same image frame, therefore, the coordinate transformation relationship between any two frames of the three-dimensional point cloud of the target object obtained by reconstruction can be determined by the fiducial points in the two image frames corresponding to the two frames of the three-dimensional point cloud of the target object. For example, assume that the image frame set includes composite image frame A and composite image frame B. A single-frame three-dimensional point cloud P1 of the target object can be reconstructed based on the structured light pattern in composite image frame A, a single-frame three-dimensional point cloud P2 of the target object can be reconstructed based on the structured light pattern in composite image frame B, a single-frame three-dimensional point cloud O1 of the fiducial points can be reconstructed based on the fiducial point pattern in composite image frame A, and a single-frame three-dimensional point cloud O2 of the fiducial points can be reconstructed based on the fiducial point pattern in composite image frame B. Since the coordinate transformation relationship R1 between O1 and O2 can be determined, and further, since the coordinate transformation relationship R2 between P1 and P2 can also be determined, that is, R2 = R1.

[0057] As Figure 5 shown, for some scenarios where the beams or light spots of the structured light are relatively dense, or the structured light is a black-and-white striped pattern, if the structured light pattern and the fiducial point pattern are included in the same frame of the image, the fiducial points on the surface of the target object are often blocked by the structured light pattern, resulting in the subsequent inability to extract these fiducial points from the image, or the extracted fiducial points being inaccurate, leading to an inaccurate stitching result obtained based on the fiducial points, and further, the accuracy of the reconstructed three-dimensional model is not high.

[0058] To reduce the occlusion of the fiducial points by the structured light pattern, in some embodiments, the density of the structured light pattern in the composite image frame is determined based on the size of the fiducial point. That is, for the scenario of simultaneously collecting the structured light pattern and the fiducial point pattern, the density of the structured light projected onto the surface of the target object can be determined based on the size of the fiducial points set on the surface of the target object, so as to minimize the occlusion of the structured light image to the fiducial points. For example, taking the structured light as line structured light as an example, the distance between adjacent lines can be made greater than the size of the fiducial point, so that the fiducial point can be located between the lines, thereby reducing the occlusion of the fiducial point.

[0059] In some embodiments, to avoid occlusion of the fiducial points by the structured light pattern, different cameras may be used to capture the fiducial point pattern and the structured light pattern on the surface of the target object respectively, or the same camera may be used to capture the fiducial point pattern and the structured light pattern on the surface of the target object at different times, so that the fiducial point pattern and the structured light pattern are located in different image frames. That is, the set of image frames includes multiple structured light image frames and multiple fiducial point image frames. Each structured light image frame only includes the structured light pattern, and each fiducial point image frame only includes the fiducial point pattern. Among them, the multiple structured light image frames can be captured by one or more cameras, the multiple fiducial point image frames can be captured by one or more cameras, and the multiple structured light image frames and the multiple fiducial point image frames can be captured by the same camera or by different cameras.

[0060] Therefore, in some embodiments, when projecting the structured light pattern onto the surface of the target object, the acquisition methods of the two patterns may also be determined based on the density of the structured light pattern projected by the 3D scanning device. For example, if the density of the structured light pattern projected by the 3D scanning device is less than the preset density, the structured light pattern and the fiducial point pattern may be captured simultaneously by the same camera, that is, the set of image frames includes multiple composite image frames. If the density of the structured light pattern projected by the 3D scanning device is greater than or equal to the preset density, the structured light pattern and the fiducial point pattern may be captured at different times by the same camera, or the structured light pattern and the fiducial point pattern may be captured separately by different cameras, that is, the set of image frames includes multiple structured light image frames and multiple fiducial point image frames.

[0061] In a scenario where the structured light pattern projected by the 3D scanning device is sparse, that is, the density is lower than the preset density, the acquisition method of capturing the structured light pattern and the fiducial point pattern simultaneously by the same camera may be adopted. Since the structured light pattern and the fiducial point pattern are located in the same image frame, the coordinate systems of the 3D point cloud of the fiducial points and the 3D point cloud of the target object reconstructed based on the same image frame are the same. That is, the coordinate transformation relationship of the 3D point clouds of the target object in different frames can be determined quickly, improving the point cloud stitching efficiency. In a scenario where the structured light pattern projected by the 3D scanning device is dense, that is, the density is greater than or equal to the preset density, capturing the structured light pattern and the fiducial point pattern at different times by the same camera or capturing the structured light pattern and the fiducial point pattern separately by different cameras can make the structured light pattern and the fiducial point pattern located in different images, avoiding occlusion of the fiducial points by the structured light pattern, and thus the fiducial points can be extracted from the images more accurately, improving the stitching accuracy.

[0062] In one embodiment, the three-dimensional scanning device includes a structured light projector for projecting structured light onto a target object. When the three-dimensional scanning device scans the target object, it first acquires one or more initial images. Based on the width or diameter of the fiducial points obtained from the initial images, and according to the width or diameter of the fiducial points and the acquisition method preset by the user, it calculates the density of the required structured light pattern and adjusts the parameters of the structured light projector to modify the pattern projected subsequently.

[0063] In some embodiments, the three-dimensional scanning device includes a structured light projector for projecting structured light onto a target object, and a fill light for filling light for the target object. The composite image frame is acquired when the above-mentioned structured light projector is in the on state and the fill light is in the on state. For example, for a scene where the brightness of the scanning environment is low, the camera can clearly acquire the image of the fiducial points only when the fill light is turned on. For example, taking the scanning rod in the oral cavity as an example, usually the brightness in the oral cavity is low, and the fill light needs to be turned on to clearly capture the fiducial points. Therefore, in order to acquire the structured light pattern and the fiducial point pattern simultaneously, the image of the target object can be acquired when the structured light projector is in the on state and the fill light is in the on state, to obtain the above-mentioned composite image frame. Among them, the fill light can be an LED light.

[0064] In some embodiments, the set of image frames includes at least one set of first image frame sequences. Each set of first image frame sequences is acquired by one camera in the three-dimensional scanning device. Each set of first image frame sequences includes multiple structured light image frames and multiple fiducial point image frames, and the structured light image frames and the fiducial point image frames are acquired by the camera at different times. For example, taking the three-dimensional scanning device including a monocular camera as an example, the monocular camera can acquire images of the fiducial point pattern and the structured light pattern in sequence to obtain a set of first image frame sequences. Taking the three-dimensional scanning device including a multiocular camera (such as binocular or trinocular or four or more eyes) as an example, each camera can be controlled to acquire images of the fiducial point pattern and the structured light pattern in sequence to obtain multiple sets of first image frame sequences. Among them, for the scenario where the three-dimensional scanning device includes a multiocular camera (such as binocular or trinocular or four or more eyes), at the same time, multiple cameras can be used to acquire images of the fiducial points from different perspectives simultaneously, and then the depth information of the fiducial points can be determined more accurately through methods such as triangulation to perform three-dimensional reconstruction of the fiducial points.

[0065] In some embodiments, the 3D scanning device includes a structured light projector for projecting structured light onto a target object, and a fill light for illuminating the target object; the structured light image frame is acquired when the structured light projector is in the on state and the fill light is in the off state, and the fiducial point image frame is acquired when the structured light projector is in the off state and the fill light is in the on state. For example, for a scenario where the brightness of the scanning environment is low, the camera can clearly acquire the images of the fiducial points only when the fill light is turned on. Therefore, the on / off states of the structured light projector and the fill light can be controlled according to the timing, so that the same camera can acquire the structured light pattern and the fiducial point pattern according to the timing.

[0066] In some embodiments, the 3D scanning device includes a structured light projector, which includes a structured light mode for projecting structured light and a uniform light mode for projecting uniform light. The structured light image frame is acquired when the structured light projector is in the structured light mode, and the fiducial point image frame is acquired when the structured light projector is in the uniform light mode. For some structured light projectors, they include multiple projection modes, that is, they can project structured light with encoded information or unstructured uniform light, and the uniform light can be white light, or colored light such as blue light or red light. Therefore, the uniform light projected by it can be used to illuminate the target object, so that there is no need to additionally set a fill light in the 3D scanning device, simplifying the structure of the 3D scanning device.

[0067] For a scenario where the structured light image frame and the fiducial point image frame are acquired according to the timing, since the structured light image frame and the fiducial point image frame are acquired by the camera at different poses, considering that the camera usually moves at a constant speed, therefore, the motion of the camera can be estimated based on at least two adjacent image frames of the same type, and the pose transformation of the camera when acquiring two adjacent image frames can be determined, so that the coordinate transformation relationship between the structured light image frame A and the adjacent fiducial point image frame B can be determined, that is, the coordinate transformation relationship between the 3D point cloud of the single-frame target object corresponding to the structured light image frame A and the 3D point cloud of the single-frame target object corresponding to the fiducial point image frame B.

[0068] Among them, there are various ways to estimate the motion of the camera. In some scenarios, the ICP (Iterative Closest Point) algorithm can be used to determine the camera motion relationship between adjacent frames. Of course, other algorithms can also be used, and the embodiments of the present application do not make any limitations.

[0069] Thus, in some embodiments, if the structured light image frames and the fiducial point image frames are acquired in sequence, for the three-dimensional point cloud of the target object in each frame, the coordinate transformation relationship between the three-dimensional point cloud of the target object in this frame and the three-dimensional point cloud of one of the fiducial point frames can be determined, where the fiducial point image frame corresponding to the three-dimensional point cloud of this one of the fiducial point frames is adjacent to the structured light image frame corresponding to the three-dimensional point cloud of this frame of the target object. When determining the coordinate transformation relationship between the three-dimensional point cloud of this frame of the target object and the three-dimensional point cloud of this one of the fiducial point frames, motion estimation can be performed on at least two adjacent image frames of the same type in the first image frame sequence to determine the pose transformation relationship when the camera acquires two adjacent image frames, where the acquisition time interval between the at least two image frames of the same type and the structured light image frame corresponding to the three-dimensional point cloud of this frame of the target object is less than a preset time interval, and then the above-mentioned second coordinate transformation relationship can be determined based on this pose transformation relationship. Considering that during the scanning process, the motion of the camera can be regarded as uniform motion in a short period of time, in order to accurately estimate the coordinate transformation relationship between the three-dimensional point cloud of this frame of the target object and the three-dimensional point cloud of this one of the fiducial point frames, the acquisition times of the at least two image frames of the same type should be as close as possible to the acquisition time of the three-dimensional point cloud of this frame of the target object, and since motion estimation requires the same content to be included in two image frames, therefore, the camera motion can be estimated based on at least two image frames of the same type, and the at least two image frames of the same type can be both structured light image frames or both fiducial point image frames.

[0070] In some embodiments, as Figure 6 shown, the structured light image frames and the fiducial point image frames in each group of the first image frame sequence are arranged alternately. That is, the same camera can alternately acquire the structured light pattern and the fiducial point pattern.

[0071] For the above acquisition method, when determining the three-dimensional point clouds of the target object in different frames, the following method can be adopted:

[0072] Assume that the first image frame sequence includes: structured light image frame 1, fiducial point image 1, structured light image frame 2, fiducial point image 2, structured light image frame 3, fiducial point image 3...

[0073] The single-frame point clouds constructed based on each image frame are: three-dimensional point cloud 1 of the target object, three-dimensional point cloud 1 of the fiducial point, three-dimensional point cloud 2 of the target object, three-dimensional point cloud 2 of the fiducial point, three-dimensional point cloud 3 of the target object, three-dimensional point cloud 3 of the fiducial point...

[0074] For the three-dimensional point cloud 1 of the target object, the coordinate transformation relationship with the three-dimensional point cloud 1 of the landmark points can be determined. Specifically, based on the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 2 of the target object, the motion of the camera can be estimated, and the pose transformation of the camera when collecting the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 2 of the target object can be determined. Assuming R, considering that the camera can be regarded as moving at a constant speed in a short time, therefore, the pose transformation between the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 1 of the landmark points is R / 2, and this pose transformation is the coordinate transformation relationship between the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 1 of the landmark points.

[0075] In some embodiments, as Figure 7 shown, each group of the first image frame sequences includes multiple repetitively arranged image groups, each image group includes a structured light image frame and a landmark point structure frame, and each image group includes at least two continuously arranged image frames of the same type.

[0076] Considering that usually the camera can only be regarded as moving at a constant speed in a short time, therefore, in order to more accurately estimate the motion of the camera, when collecting the structured light pattern frames or landmark point patterns in sequence, the camera can be controlled to continuously collect at least two image frames of the same type, and then collect another type of image frame, and then repeat the above collection order to obtain multiple image groups. For example, at least two structured light image frames can be continuously collected and then the landmark point image frames can be collected, or at least two landmark point image frames can be continuously collected and then the structured light image frames can be collected. Thus, the motion of the camera can be estimated based on the at least two structured light image frames (or at least two landmark point image frames), and the motion information of the camera can be determined, and this motion information can be used as the motion information of the camera when collecting this image group, so as to facilitate determining the pose transformation of the camera when collecting the adjacent structured light image frame and landmark point image frame in this image group.

[0077] Considering that the image frames in the same image group share the same motion information, in order to ensure that the motion information is as accurate as possible, the continuously arranged image frames of the same type in the image group can include only two. For example, in some embodiments, each image group includes two continuously arranged structured light image frames and one landmark point image frame. In some embodiments, each image group includes two continuously arranged landmark point image frames and one structured light image frame. Since the collection time interval is short, thus, when applying the camera motion information determined by the two continuous structured light image frames (or two continuous structured light image frames) to the structured light image frame and the adjacent landmark point image frame in the image group, it will be more accurate.

[0078] For example, for the above collection method, when determining the three-dimensional point cloud of the target object in different frames, the following method can be adopted:

[0079] Suppose the first frame image sequence includes: structured light image frame 1, structured light image frame 2, fiducial point image 1, structured light image frame 3, structured light image frame 4, fiducial point image frame 2, structured light image frame 5, structured light image frame 6, fiducial point image frame 3...

[0080] The single-frame point clouds constructed based on each image frame are: 3D point cloud 1 of the target object, 3D point cloud 2 of the target object, 3D point cloud 1 of the fiducial point, 3D point cloud 3 of the target object, 3D point cloud 4 of the target object, 3D point cloud 2 of the fiducial point, 3D point cloud 5 of the target object, 3D point cloud 6 of the target object, 3D point cloud 3 of the fiducial point...

[0081] Motion estimation can be performed based on the 3D point cloud 1 of the target object and the 3D point cloud 2 of the target object to determine the pose transformation of the camera, and then this pose transformation is used as the pose transformation when the camera captures the 3D point cloud 2 of the target object and the 3D point cloud 1 of the fiducial point. Considering that the acquisition time intervals of the above three frames of point clouds are very short, that is, the motion speed of the camera when capturing the 3D point cloud 1 of the target object and the 3D point cloud 2 of the target object can be regarded as the same as the motion speed of the camera when capturing the 3D point cloud 2 of the target object and the 3D point cloud 1 of the fiducial point. Therefore, the pose transformation of the camera when capturing the 3D point cloud 2 of the target object and the 3D point cloud 1 of the fiducial point determined in this way is relatively accurate.

[0082] In some embodiments, in order to make the structured light pattern and the fiducial point pattern distributed in different image frames, different cameras can also be used to capture images of the structured light pattern and the fiducial point pattern respectively. For example, the image frame set includes a second image frame sequence and a third image frame sequence. The second image frame sequence includes the above-mentioned multiple structured light image frames, and the third image frame sequence includes the above-mentioned multiple fiducial point image frames. The second image frame sequence and the third image frame sequence are respectively captured by two cameras in the 3D scanning device. Considering that different cameras have different response characteristics to light beams of different wavelengths, the structured light projected by the structured light projector and the light beam projected by the fill light can be set as light beams of different wavelengths, such as light beam 1 and light beam 2. Then, two cameras, camera 1 and camera 2, are set in the 3D scanning device. Among them, the sensor in camera 1 can only sense light beam 1, and the sensor in camera 2 can only sense light beam 2. Therefore, when the structured light projector and the fill light are both turned on, the above two cameras can be used to capture images of the target object simultaneously. Camera 1 captures the structured light image frame, and camera 2 captures the fiducial point image frame.

[0083] In some embodiments, if the structured light image frame and the fiducial point image frame are respectively acquired by two cameras, for the three-dimensional point cloud of the target object in each frame, the coordinate transformation relationship between the three-dimensional point cloud of the target object in this frame and the three-dimensional point cloud of one of the fiducial point frames can be determined, where the acquisition time of the three-dimensional point cloud of one of the fiducial point frames is the same as that of the three-dimensional point cloud of the target object. When determining the second coordinate transformation relationship between the three-dimensional point cloud of the target object in this frame and the three-dimensional point cloud of one of the fiducial point frames, the second coordinate transformation relationship between the three-dimensional point cloud of the target object in this frame and the three-dimensional point cloud of one of the fiducial point frames can be determined based on the extrinsic parameters of the two pre-calibrated cameras. That is, since the structured light image frame and the fiducial point image frame are acquired simultaneously by two cameras, the coordinate transformation relationship between the two can be determined by the extrinsic parameters of the two cameras.

[0084] In some embodiments, the three-dimensional scanning device is an oral scanner, and the target object is the oral cavity. Among them, a scanning rod is installed in the oral cavity, fiducial points are arranged on the scanning rod, or fiducial points are pasted on the teeth and gums in the oral cavity, or a target is installed in the oral cavity, and fiducial points are arranged on the target.

[0085] In some embodiments, the three-dimensional scanning device is a facial scanner, and the target object is the face. Among them, fiducial points are pasted on the face, or a target is installed on the face, and fiducial points are arranged on the target.

[0086] In some embodiments, the three-dimensional scanning device is a handheld scanner, and the target object is an industrial product or a cultural relic or a building. Among them, fiducial points are pasted or projected or sprayed on the scanned objects such as industrial products or cultural relics or buildings.

[0087] In some embodiments, after stitching the three-dimensional point clouds of multiple frames of the target object based on the first coordinate transformation relationship and the second coordinate transformation relationship, the three-dimensional model of the target object obtained by stitching can be displayed in real time on the interaction interface. The three-dimensional model can be with a texture map pasted or without a texture map pasted.

[0088] In some embodiments, the three-dimensional scanning device is an oral scanner, and the target object is an edentulous oral cavity. Among them, a scanning rod is installed in the oral cavity, and fiducial points are arranged on the scanning rod. When it can be seen on the interaction interface that as the scanning process progresses, the scanning rod and the edentulous oral cavity are gradually displayed on the interaction interface, the fiducial point texture image that has been pasted can be selected to be displayed or not to be displayed on the scanning rod according to user needs, meeting the user's development requirements.

[0089] It is not difficult to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions in the case of no conflict. Due to space limitations, they are not listed one by one in the embodiments of this application.

[0090] Correspondingly, an embodiment of the present application further provides a 3D reconstruction device, as Figure 8 shown, the 3D reconstruction device 80 includes:

[0091] An acquisition module 81, configured to acquire a set of image frames collected by a 3D scanning device during the process of scanning a target object, wherein marker points are provided on the surface of the target object, and structured light is projected on the surface of the target object during at least a part of the time period of scanning the target object;

[0092] A 3D reconstruction module 82, configured to perform 3D reconstruction on the marker points based on the image frames including marker point patterns in the set of image frames to obtain 3D point clouds of multiple frames of marker points, and determine a first coordinate transformation relationship between the 3D point clouds of the multiple frames of marker points; and perform 3D reconstruction on the target object based on the image frames including structured light patterns in the set of image frames to obtain 3D point clouds of multiple frames of the target object;

[0093] A stitching module 83, configured to determine a second coordinate transformation relationship between the 3D point cloud of each frame of the target object and the 3D point cloud of one of the frames of marker points, and stitch the 3D point clouds of the multiple frames of the target object based on the first coordinate transformation relationship and the second coordinate transformation relationship.

[0094] Wherein, the specific steps for the above device to execute the task processing method can refer to the description in the above method embodiments, and will not be elaborated herein.

[0095] Furthermore, an embodiment of the present application further provides an electronic device, as Figure 9 shown, the device includes a processor 91, a memory 92, and computer instructions stored in the memory 92 and executable by the processor 91. When the processor 91 executes the computer instructions, the method described in any one of the above embodiments is implemented.

[0096] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the foregoing embodiments is implemented.

[0097] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0098] From the description of the above embodiments, those skilled in the art can clearly understand that the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments of the present application.

[0099] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or a combination of any several of these devices.

[0100] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated. When implementing the solutions of the embodiments of the present application, the functions of the modules can be implemented in one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the solutions of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0101] The above are only specific implementation manners of the embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the embodiments of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the embodiments of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire a set of image frames collected by a three-dimensional scanning device during a process of scanning a target object, wherein a marker point is set on the surface of the target object, and structured light is projected on the surface of the target object during at least a portion of a time period of scanning the target object; Performing three-dimensional reconstruction of the marker points based on the image frames including the marker point pattern in the image frame set to obtain three-dimensional point clouds of multiple-frame marker points, and determining a first coordinate transformation relationship between the three-dimensional point clouds of the multiple-frame marker points; Performing three-dimensional reconstruction on the target object based on the image frames including the structured light pattern in the image frame set to obtain a three-dimensional point cloud of the target object in multiple frames; For each frame of the three-dimensional point cloud of the target object, determine the second coordinate transformation relationship between the three-dimensional point cloud of the target object in the frame and the three-dimensional point cloud of one of the frame marker points, and splice the three-dimensional point clouds of the target objects in multiple frames based on the first coordinate transformation relationship and the second coordinate transformation relationship.

2. The method according to claim 1, characterized in that The image frame set includes a plurality of structured light image frames and a plurality of marker point image frames, each structured light image frame includes only a structured light pattern, and each marker point image frame includes only a marker point pattern; or The image frame set includes a plurality of composite image frames, and each composite image frame includes both a structured light pattern and a marker point pattern.

3. The method according to claim 2, characterized in that The density of the structured light pattern in the composite image frame is determined based on the size of the marker point.

4. The method according to claim 2, characterized in that: If the density of the structured light pattern projected by the three-dimensional scanning device is less than a preset density, the image frame set includes the multiple composite image frames; If the density of the structured light pattern projected by the three-dimensional scanning device is greater than or equal to a preset density, the image frame set includes the multiple structured light image frames and the multiple marker point image frames.

5. The method according to claim 2, characterized in that: The three-dimensional scanning device includes a structured light projector for projecting structured light onto the target object, and a fill light for providing fill light for the target object; the composite image frame is acquired when the structured light projector is in an on state and the fill light is in an on state.

6. The method according to claim 2, characterized in that The image frame set includes at least one group of first image frame sequences, each group of first image frame sequences is acquired by a camera in the three-dimensional scanning device, each group of first image frame sequences includes multiple structured light image frames and multiple marker point image frames, and the structured light image frames and the marker point image frames are acquired by the camera in a time-sharing manner.

7. The method according to claim 6, characterized in that The three-dimensional scanning device includes a structured light projector for projecting structured light onto the target object, and a fill light for filling light for the target object; the structured light image frame is acquired when the structured light projector is in an on state and the fill light is in an off state, and the mark point image frame is acquired when the structured light projector is in an off state and the fill light is in an on state; or The three-dimensional scanning device includes a structured light projector, and the structured light projector includes a structured light mode for projecting structured light and a uniform light mode for projecting uniform light; the structured light image frame is acquired when the structured light projector is in the structured light mode, and the marker point image frame is acquired when the structured light projector is in the uniform light mode.

8. The method according to claim 6, characterized in that The structured light image frames and the marker point image frames in the first image frame sequence are arranged alternately.

9. The method according to claim 6, characterized in that The first image frame sequence includes a plurality of repeatedly arranged image groups, each image group includes the structured light image frame and the marker point structure frame, and each image group includes at least two continuously arranged image frames of the same type.

10. The method according to claim 9, characterized in that Each image group includes two consecutively arranged structured light image frames and one marker point image frame; or Each image group includes two consecutively arranged marker point image frames and one structured light image frame.

11. The method according to claim 6, characterized in that The marker point image frame corresponding to the three-dimensional point cloud of one frame of marker points is adjacent to the structured light image frame corresponding to the three-dimensional point cloud of the target object in the frame; For each frame of the target object's three-dimensional point cloud, determining the second coordinate transformation relationship between the three-dimensional point cloud of the target object in the frame and the three-dimensional point cloud of one of the frame's marker points includes: Performing motion estimation on at least two adjacent image frames of the same type in the first image frame sequence to determine a posture transformation relationship when the camera captures two adjacent image frames, wherein the acquisition time interval between the at least two image frames of the same type and the structured light image frame corresponding to the three-dimensional point cloud of the target object in the frame is less than a preset time interval; The second coordinate transformation relationship is determined based on the posture transformation relationship.

12. The method according to claim 2, characterized in that: The image frame set includes a second image frame sequence and a third image frame sequence, the second image frame sequence includes the multiple structured light image frames, the third image frame sequence includes the multiple marker point image frames, and the second image frame sequence and the third image frame sequence are respectively acquired by two cameras in the three-dimensional scanning device.

13. The method according to claim 12, characterized in that The acquisition time of the three-dimensional point cloud of the marker point in one frame is the same as that of the three-dimensional point cloud of the target object in the frame; for each three-dimensional point cloud of the target object in the frame, determining the second coordinate transformation relationship between the three-dimensional point cloud of the target object in the frame and the three-dimensional point cloud of the marker point in one frame, including: Based on the pre-calibrated external parameters of the two cameras, a second coordinate transformation relationship between the three-dimensional point cloud of the target object in the frame and the three-dimensional point cloud of the marker point in one of the frames is determined.

14. The method according to claim 1, characterized in that The three-dimensional scanning device is an oral scanner, the target object is an oral cavity, and the oral cavity is an oral cavity where a scanning rod is installed or where marking points are attached to teeth and gums or an oral cavity where a target is installed; and / or After the three-dimensional point clouds of the target object in the multiple frames are spliced ​​together based on the first coordinate transformation relationship and the second coordinate transformation relationship, the three-dimensional model of the target object currently spliced ​​together is displayed in real time on the interactive interface.

15. A three-dimensional reconstruction device, characterized in that: The three-dimensional reconstruction device comprises: an acquisition module, configured to acquire a set of image frames collected by a three-dimensional scanning device during a process of scanning a target object, wherein a mark point is set on the surface of the target object, and structured light is projected on the surface of the target object during at least a portion of a time period during which the target object is scanned; a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction of the marker points based on the image frames including the marker point pattern in the image frame set, obtain three-dimensional point clouds of multiple frames of marker points, and determine a first coordinate transformation relationship between the three-dimensional point clouds of the multiple frames of marker points; and perform three-dimensional reconstruction of the target object based on the image frames including the structured light pattern in the image frame set, obtain three-dimensional point clouds of the multiple frames of target object; The splicing module is used to determine the second coordinate transformation relationship between the three-dimensional point cloud of the target object in each frame and the three-dimensional point cloud of one frame of the marker point, and splice the three-dimensional point clouds of the target objects in multiple frames based on the first coordinate transformation relationship and the second coordinate transformation relationship.

16. An electronic device, characterized in that: The electronic device includes a processor, a memory, and computer instructions stored in the memory and executable by the processor. When the processor executes the computer instructions, the method according to any one of claims 1 to 14 is implemented.

Images