Three-dimensional reconstruction method, device and equipment

Abstract

Embodiments of the present application provide a three-dimensional reconstruction method, device, equipment and storage medium. A landmark point can be set on a surface of a target object, and a structured light is projected to the surface of the target object in at least a part of a time period during which the target object is scanned, so as to collect a set of image frames of the target object. Then, a three-dimensional point cloud of landmark points of each frame is reconstructed based on each image frame containing a landmark point pattern, and a three-dimensional point cloud of the target object of each frame is reconstructed based on each image frame containing a structured light pattern. Since a coordinate conversion relationship between the three-dimensional point clouds of the landmark points of each frame can be accurately determined, and a coordinate conversion relationship between the three-dimensional point cloud of a single frame of landmark points and the three-dimensional point cloud of a single frame of the target object can also be determined, the landmark points can be used to assist the splicing of the three-dimensional point clouds of the multiple frames of the target object, so as to obtain a more accurate splicing result.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of three-dimensional scanning, and particularly relate to a three-dimensional reconstruction method, device, equipment and storage medium. BACKGROUND

[0002] In the process of three-dimensional reconstruction of a target object by using data collected by a three-dimensional scanning device, the three-dimensional scanning device is usually used to scan the target object from different angles multiple times to obtain multiple frames of local point cloud data, and then the multiple frames of local point cloud data are spliced to obtain global point cloud data of the target object.

[0003] In related technologies, a common splicing method is geometric splicing, that is, splicing is performed by using geometric features of the target object. This splicing method requires the target object to have rich and non-repeated geometric features. When scanning some target objects with regular geometry, such as a cylindrical base or a planting rod, the target objects have repeated geometric features or weak feature information. Because the number of feature points extracted is limited, a high-precision splicing result cannot be calculated, thereby resulting in a poor precision of the three-dimensional model finally reconstructed. SUMMARY

[0004] Embodiments of the present application provide a three-dimensional reconstruction method, device, equipment and storage medium.

[0005] According to a first aspect of embodiments of the present application, a three-dimensional reconstruction method is provided, and the method comprises:

[0006] obtaining a set of image frames collected by a three-dimensional scanning device in the process of scanning a target object, wherein a mark point is arranged on a surface of the target object, and structured light is projected on the surface of the target object in at least part of a time period during which the target object is scanned;

[0007] performing three-dimensional reconstruction on the mark point based on an image frame including a mark point pattern in the set of image frames, obtaining multiple frames of three-dimensional point clouds of the mark point, and determining a first coordinate conversion relationship between the multiple frames of three-dimensional point clouds of the mark point;

[0008] performing three-dimensional reconstruction on the target object based on an image frame including a structured light pattern in the set of image frames, obtaining multiple frames of three-dimensional point clouds of the target object;

[0009] for each frame of three-dimensional point clouds of the target object, determining a second coordinate conversion relationship between the frame of three-dimensional point clouds of the target object and one frame of three-dimensional point clouds of the mark point, and splicing the multiple frames of three-dimensional point clouds of the target object based on the first coordinate conversion relationship and the second coordinate conversion relationship.

[0010] According to a second aspect of embodiments of the present application, a three-dimensional reconstruction device is provided, and the three-dimensional reconstruction device comprises:

[0011] an acquisition module, configured to acquire a set of image frames collected by the three-dimensional scanning device in a process of scanning a target object, wherein a landmark point is arranged on a surface of the target object, and structured light is projected on the surface of the target object in at least a part of a time period during which the target object is scanned;

[0012] a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the landmark point based on image frames including landmark point patterns in the set of image frames to obtain a plurality of frames of three-dimensional point clouds of the landmark point, and determine a first coordinate conversion relationship between the plurality of frames of three-dimensional point clouds of the landmark point; and perform three-dimensional reconstruction on the target object based on image frames including structured light patterns in the set of image frames to obtain a plurality of frames of three-dimensional point clouds of the target object;

[0013] a splicing module, configured to, for each frame of three-dimensional point clouds of the target object, determine a second coordinate conversion relationship between the frame of three-dimensional point clouds of the target object and one frame of three-dimensional point clouds of the landmark point, and splice the plurality of frames of three-dimensional point clouds of the target object based on the first coordinate conversion relationship and the second coordinate conversion relationship.

[0014] According to a third aspect of the embodiments of the present application, an electronic device is provided, which includes a processor, a memory, and computer instructions stored in the memory and executable by the processor. When the processor executes the computer instructions, the method mentioned in the first aspect can be implemented.

[0015] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions. When the computer instructions are executed, the method mentioned in the first aspect can be implemented.

[0016] In the embodiments of the present application, a landmark point can be arranged on a surface of a target object, and structured light can be projected on the surface of the target object in at least a part of a time period during which the target object is scanned, so as to acquire a set of image frames of the target object. Then, three-dimensional point clouds of each frame of landmark points can be reconstructed based on image frames containing landmark point patterns, and three-dimensional point clouds of each frame of target objects can be reconstructed based on image frames containing structured light patterns. Since the coordinate conversion relationship between the three-dimensional point clouds of each frame of landmark points can be accurately determined, and the coordinate conversion relationship between the three-dimensional point clouds of each frame of landmark points and each frame of target objects can also be determined, the splicing of the plurality of frames of three-dimensional point clouds of the target objects can be assisted by the landmark points, so as to obtain a more accurate splicing result.

[0017] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, rather than limiting the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description serve to explain the principles of the present application.

[0019] Figure 1 is a schematic diagram of an application scenario of the present application.

[0020] Figure 2 is a flowchart of a three-dimensional reconstruction method of the present application.

[0021] Figure 3 is a schematic diagram of a landmark point of the present application.

[0022] Figure 4 is a schematic diagram of a landmark point of the present application being blocked by a structured light pattern.

[0023] Figure 5 is a schematic diagram of the present application of simultaneously collecting a structured light pattern and a landmark pattern.

[0024] Figure 6 is a schematic diagram of the present application of collecting a structured light pattern and a landmark pattern in time sequence.

[0025] Figure 7 is a schematic diagram of another embodiment of the present application of collecting a structured light pattern and a landmark pattern in time sequence.

[0026] Figure 8 is a schematic diagram of a three-dimensional reconstruction device of the present application.

[0027] Figure 9 is a schematic diagram of a logic structure of an electronic device of the present application. DETAILED DESCRIPTION

[0028] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings, in which like reference numerals represent like elements, and identical or similar components are denoted by like reference numerals throughout the various figures and / or embodiments of the present application. The following exemplary embodiments described in the present application do not represent all the implementations consistent with the present application. Instead, they only represent a number of devices and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0029] The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the description of the application and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It also will be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. In addition, the term "at least one of' as used herein means any one of or any combination of any two or more of the items in the list.

[0030] It should be understood that although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a particular order or hierarchy. These terms are used only to distinguish one type of information from another type of information. For example, a first information can be termed a second information, and similarly, a second information can be termed a first information without departing from the scope of the present application. The word "if' as used herein means "when" or "upon" or "in response to the determination" depending on the context.

[0031] In order to enable persons skilled in the art to better understand the technical solutions in the application embodiments, and to make the above-mentioned purposes, characteristics and advantages of the application embodiments more apparent and understandable, the technical solutions in the application embodiments will be further described in detail below with reference to the drawings.

[0032] The three-dimensional reconstruction technology based on structured light projects an optical coding pattern (i.e. structured light) to a target object to be reconstructed, and determines the depth information of the surface of the target object based on the deformed pattern of the structured light captured by a camera, to recover the three-dimensional data of the surface of the target object. The three-dimensional reconstruction technology based on structured light has the characteristics of high efficiency and anti-interference, and is widely used in various three-dimensional reconstruction scenes. In the scanning process, due to the size of the projection plane of the structured light or the size of the shooting field of view of the camera, only a local three-dimensional point cloud of the target object can be obtained in a single measurement. In order to obtain a global three-dimensional point cloud of the target object, the local point clouds obtained from multiple perspectives need to be spliced.

[0033] At present, the commonly used splicing method is geometric splicing, that is, splicing is performed by using the geometric characteristics of the target object. This splicing method requires the target object to have rich and non-repeated geometric characteristics. When scanning some geometrically regular target objects, such as cylindrical base, planting rod, scanning rod and other target objects with repeated geometric characteristics or weak characteristic information, due to the limited number of feature points extracted, a high-precision splicing result cannot be calculated, thereby resulting in a poor precision of the three-dimensional model finally reconstructed.

[0034] To improve stitching accuracy, this application proposes that for target objects with weak geometric regularity and feature information, marker points can be set on the surface of the target object, and structured light can be projected onto the surface of the target object for at least a portion of the scanning time to acquire a set of image frames of the target object. Then, based on each image frame containing the marker point pattern, a 3D point cloud of each frame of marker points is reconstructed, and based on each image frame containing the structured light pattern, a 3D point cloud of each frame of the target object is reconstructed. Since the coordinate transformation relationship between the 3D point clouds of each frame of marker points can be accurately determined, and the coordinate transformation relationship between the 3D point cloud of a single frame of marker points and the 3D point cloud of a single frame of the target object can also be determined, the marker points can be used to assist in stitching the 3D point clouds of the target object across multiple frames, resulting in a more accurate stitching result.

[0035] The three-dimensional reconstruction method provided in this application embodiment can be executed by a three-dimensional scanning device or by other devices connected to the three-dimensional scanning device wirelessly or wiredly. These other devices can be mobile phones, tablets, laptops, cloud servers, or server clusters, etc.

[0036] For example, in some scenarios, if the 3D scanning device has good computing power, it can acquire images of the target object and use the acquired images to perform 3D reconstruction of the target object to obtain a 3D model of the target object.

[0037] In some scenarios, if the computing power of the 3D scanning device itself is insufficient, other devices can be used to achieve 3D reconstruction. For example, scanning software can be installed on the other device, which can then perform real-time 3D reconstruction based on the images acquired by the 3D scanning device and display the reconstruction results to the user.

[0038] For example, such as Figure 1 The diagram shown is an application scenario illustration of an embodiment of this application. Considering that the computing power of a 3D scanning device (taking an oral scanner as an example in the figure) is relatively weak, the images collected by the 3D scanning device are usually sent to a terminal device with better processing performance (such as a laptop computer) that is connected to it. The terminal device can install scanning software, which can perform real-time 3D reconstruction based on the collected images and display the reconstruction results to the user.

[0039] The 3D scanning device in this application embodiment can be an oral scanner, facial scanner, industrial scanner, or professional scanner, which can be used to perform 3D scanning and 3D reconstruction of items such as teeth, faces, bodies, industrial products, industrial equipment, cultural relics, artworks, prostheses, medical devices, and buildings.

[0040] The target object in the embodiments of the present application can be various to-be-reconstructed objects provided with a marker point, such as a tooth, an implant rod, a scanning rod, an abutment, an industrial product, and the like, and the embodiments of the present application do not make any limitation.

[0041] As shown in Figure 2 The three-dimensional reconstruction method can include the following steps:

[0042] S202, acquiring a set of image frames collected by a three-dimensional scanning device in the process of scanning a target object, wherein the surface of the target object is provided with a marker point, and the surface of the target object is projected with structured light in at least part of the time period of scanning the target object;

[0043] In step S202, a set of image frames collected by a three-dimensional scanning device in the process of scanning a target object can be acquired, wherein the image frames in the set of image frames can be collected by one or more cameras. The surface of the target object is provided with a marker point, which can be a marker point formed on the surface of the target object by projection, spraying or pasting, etc. The shape, color, number, size, etc. of the marker point can be set based on actual requirements, and the embodiments of the present application do not make any limitation. As shown in Figure 3 In some scenarios, the marker point can not carry coded information (the first point on the left), and in some scenarios, in order to better distinguish each marker point on the surface of the target object, the marker point can also be a marker point carrying coded information (the two points on the right). Figure 3 Figure 3

[0044] In order to obtain the depth information of the surface of the target object, structured light is projected on the surface of the target object in at least part of the time period during the scanning of the target object. The structured light can be dot matrix, line, light spot, etc. and the embodiments of the present application do not make any limitation.

[0045] The structured light can be projected on the target object throughout the scanning process, or can be projected on the target object only in part of the time period of the scanning process. For example, in order to realize that each image frame collected includes a structured light pattern, the structured light can be projected on the target object throughout the scanning process. Of course, in some scenarios, in order to let some image frames not include a structured light pattern, the structured light can not be projected on the target object at some moments.

[0046] S204, three-dimensional reconstruction of the marker point based on the image frames including the marker point pattern in the set of image frames, obtaining a plurality of three-dimensional point clouds of the marker point, and determining a first coordinate conversion relationship between the plurality of three-dimensional point clouds of the marker point;

[0047] ​​In step S204, after obtaining the image frame set, an image frame including a landmark point image can be obtained from the image frames, and a three-dimensional reconstruction is performed on the landmark points based on the image frame including the landmark point pattern to obtain a three-dimensional point cloud of the landmark points in multiple frames, wherein one image frame including a landmark point pattern corresponds to one three-dimensional point cloud of the landmark points.

[0048] For example, in some scenarios, the three-dimensional scanning device includes at least two cameras, and the landmark points on the surface of the target object can be imaged by the at least two cameras to obtain a plurality of image frame sequences including landmark points. For each landmark point, the depth information of the pixel point corresponding to the landmark point can be obtained based on the parallax of the pixel point corresponding to the landmark point in the images captured by the at least two cameras, and then the three-dimensional coordinates of the landmark point can be determined. In this way, the three-dimensional point cloud corresponding to the landmark points in each image frame can be determined, and then the coordinate transformation relationship between different image frames (i.e., the three-dimensional point clouds of the landmark points in different frames) can be determined based on the three-dimensional coordinates of the three-dimensional point clouds of the same landmark point in different image frames, which is referred to as the first coordinate transformation relationship.

[0049] In some scenarios, the three-dimensional scanning device can also include only a single camera, and then the images of the landmark points can be captured by the single camera at different angles. For each landmark point, the depth information of the pixel point corresponding to the landmark point can be obtained based on the parallax of the pixel point corresponding to the landmark point in the images captured by the camera at different angles, so that the three-dimensional point cloud of the landmark points in each image frame can be determined, and the first coordinate transformation relationship between the three-dimensional point clouds of the landmark points in multiple frames can be determined.

[0050] In some scenarios, the three-dimensional coordinates of the landmark points can be the three-dimensional coordinates of the centers of the landmark point patterns. For example, as shown in the landmark point carrying the encoding information, the landmark point can be composed of different patterns. For such landmark points, the center of the pattern of the landmark point can be extracted, and the three-dimensional reconstruction is performed on the center point, and the coordinates of the reconstructed center point are taken as the three-dimensional coordinates of the landmark point. Figure 3

[0051] S206, performing three-dimensional reconstruction on the target object based on the image frames including the structured light pattern in the image frame set to obtain a three-dimensional point cloud of the target object in multiple frames;

[0052] ​In step S206, after obtaining the image frame set, an image frame including a structured light pattern can be obtained from the image frame set, and then three-dimensional reconstruction is performed on the target object based on the image frame to obtain a plurality of three-dimensional point clouds of the target object. For each image frame, the depth information of the surface of the target object in the image frame can be determined based on the structured light pattern in the image frame, so that a single three-dimensional point cloud of the target object can be obtained. The image frame including the structured light pattern can also be obtained by one or more cameras.

[0053] In step S208, for each three-dimensional point cloud of the target object, a second coordinate transformation relationship between the three-dimensional point cloud of the target object and the three-dimensional point cloud of one of the mark points is determined, and the plurality of three-dimensional point clouds of the target object are spliced based on the first coordinate transformation relationship and the second coordinate transformation relationship.

[0054] In step S208, since the coordinate transformation relationship between the three-dimensional point clouds of different frames of mark points can be determined, the coordinate transformation relationship between the three-dimensional point clouds of different frames of target objects can be determined by means of the coordinate transformation relationship between the three-dimensional point clouds of different frames of mark points. For example, for each three-dimensional point cloud of the target object, a coordinate transformation relationship between the three-dimensional point cloud of the target object and the three-dimensional point cloud of one of the mark points can be determined, which is referred to as a second coordinate transformation relationship below for ease of distinction. The three-dimensional point cloud of the mark point and the three-dimensional point cloud of the target object have a certain relationship, such as being reconstructed based on the same image frame, being captured by different cameras at the same time, or having a short time interval between the capture times, and the like. Then, the coordinate transformation relationship between the plurality of three-dimensional point clouds of the target object can be determined based on the first coordinate transformation relationship and the second coordinate transformation relationship, and the coordinate systems between the plurality of three-dimensional point clouds of the target object are unified to splice the plurality of three-dimensional point clouds of the target object.

[0055] By setting the mark points on the surface of the target object, the coordinate transformation relationship between the three-dimensional point clouds of different frames of mark points can be accurately determined, and the mark points can be used to assist accurate splicing between the three-dimensional point clouds of different frames of target objects to obtain an accurate splicing result.

[0056] In some embodiments, as Figure 4As shown, when scanning a target object, this 3D scanning device can simultaneously acquire structured light patterns and marker point patterns projected onto the surface of the target object. That is, the image frame set includes multiple composite image frames, each containing both the structured light pattern and the marker point pattern. Since the structured light pattern and the marker point pattern are located in the same image frame, the coordinate transformation relationship between any two reconstructed 3D point clouds of the target object can be determined by the marker points in the corresponding two image frames of the target object's 3D point clouds. For example, suppose the image frame set includes composite image frame A and composite image frame B. Based on the structured light pattern in composite image frame A, a single-frame 3D point cloud P1 of the target object can be reconstructed. Based on the structured light pattern in composite image frame B, a single-frame 3D point cloud P2 of the target object can be reconstructed. Based on the marker point pattern in composite image frame A, a single-frame 3D point cloud O1 of the marker point can be reconstructed. Based on the marker point pattern in composite image frame B, a single-frame 3D point cloud O2 of the marker point can be reconstructed. Since the coordinate transformation relationship R1 between O1 and O2 can be determined, the coordinate transformation relationship R2 between P1 and P2 can also be determined, i.e., R2 = R1.

[0057] like Figure 5 As shown, in some scenarios where the beams or spots of structured light are relatively dense, or in scenarios where the structured light consists of alternating black and white stripes, if the same frame of the image includes both structured light patterns and marker point patterns, the marker points on the surface of the target object are often obscured by the structured light patterns. This makes it impossible to extract these marker points from the image later, or the extracted marker points are inaccurate, resulting in inaccurate stitching results based on the marker points, and consequently, low accuracy of the reconstructed 3D model.

[0058] To reduce the occlusion of marker points by the structured light pattern, in some embodiments, the density of the structured light pattern in the composite image frame is determined based on the size of the marker point. That is, for scenarios where both structured light patterns and marker point patterns are acquired simultaneously, the density of the structured light projected onto the target object's surface can be determined based on the size of the marker point set on the target object's surface, in order to minimize the occlusion of the marker point by the structured light image. For example, taking linear structured light as an example, the distance between adjacent lines can be made greater than the size of the marker point, allowing the marker point to be located between the lines, thereby reducing the occlusion of the marker point.

[0059] In some embodiments, in order to avoid the occlusion of the structured light pattern to the landmark points, the landmark pattern and the structured light pattern on the target object surface can also be captured by different cameras respectively, or the landmark pattern and the structured light pattern on the target object surface can be captured by the same camera in different time, so that the landmark pattern and the structured light pattern are located in different image frames. That is, the image frame set includes a plurality of structured light image frames and a plurality of landmark image frames, each structured light image frame only includes a structured light pattern, and each landmark image frame only includes a landmark pattern. Wherein, the plurality of structured light image frames can be captured by one or more cameras, the plurality of landmark image frames can be captured by one or more cameras, the plurality of structured light image frames and the plurality of landmark image frames can be captured by the same camera, or can be captured by different cameras.

[0060] Therefore, in some embodiments, when the structured light pattern is projected to the target object surface, the acquisition mode of the two patterns can also be determined based on the density of the structured light pattern projected by the three-dimensional scanning device. For example, if the density of the structured light pattern projected by the three-dimensional scanning device is less than the preset density, the structured light pattern and the landmark pattern can be captured by the same camera at the same time, that is, the image frame set includes a plurality of composite image frames. If the density of the structured light pattern projected by the three-dimensional scanning device is greater than or equal to the preset density, the structured light pattern and the landmark pattern can be captured by the same camera in different time, or the structured light pattern and the landmark pattern can be captured by different cameras respectively, that is, the image frame set includes a plurality of structured light image frames and a plurality of landmark image frames.

[0061] In the scene where the structured light pattern projected by the three-dimensional scanning device is relatively sparse, that is, the density is lower than the preset density, the acquisition mode of capturing the structured light pattern and the landmark pattern by the same camera at the same time can be adopted. Since the structured light pattern and the landmark pattern are located in the same image frame, the coordinate system of the three-dimensional point cloud of the landmark and the three-dimensional point cloud of the target object reconstructed based on the same image frame is the same, that is, the coordinate transformation relationship of the three-dimensional point cloud of the target object in different frames can be quickly determined, and the point cloud splicing efficiency is improved. In the scene where the structured light pattern projected by the three-dimensional scanning device is relatively dense, that is, the density is greater than or equal to the preset density, the structured light pattern and the landmark pattern can be captured by the same camera in different time, or the structured light pattern and the landmark pattern can be captured by different cameras respectively, so that the structured light pattern and the landmark pattern are located in different images, avoiding the occlusion of the structured light pattern to the landmark, so that the landmark can be more accurately extracted from the image, and the splicing accuracy is improved.

[0062] In one embodiment, the 3D scanning device includes a structured light projector for projecting structured light onto a target object. When scanning the target object, the 3D scanning device first acquires one or more initial images, obtains the width or diameter of the marker points based on the scan in the initial images, calculates the required density of the structured light pattern based on the width or diameter of the marker points and the acquisition method preset by the user, and adjusts the parameters of the structured light projector to modify the subsequently projected pattern.

[0063] In some embodiments, the 3D scanning device includes a structured light projector for projecting structured light onto a target object, and a supplementary light for providing illumination to the target object. The composite image frame is acquired when both the structured light projector and the supplementary light are on. For example, in low-light environments, the camera can only clearly capture images of marker points when the supplementary light is on. For instance, when scanning a scanning rod inside a mouth, the brightness inside the mouth is typically low, requiring the supplementary light to clearly capture the marker points. Therefore, to simultaneously acquire both the structured light pattern and the marker point pattern, the target object can be imaged when both the structured light projector and the supplementary light are on, resulting in the composite image frame. The supplementary light can be an LED light.

[0064] In some embodiments, the image frame set includes at least one set of first image frame sequences. Each set of first image frame sequences is acquired by a camera in a 3D scanning device. Each set of first image frame sequences includes multiple structured light image frames and multiple marker point image frames, which are acquired by the camera in a time-division multiplexing manner. For example, if the 3D scanning device includes a monocular camera, the monocular camera can acquire images of the marker point pattern and the structured light pattern in a time sequence to obtain a set of first image frame sequences. If the 3D scanning device includes multiple cameras (such as binocular, triocular, or quad-camera systems), each camera can be controlled to acquire images of the marker point pattern and the structured light pattern in a time sequence to obtain multiple sets of first image frame sequences. In scenarios where the 3D scanning device includes multiple cameras (such as binocular, triocular, or quad-camera systems), multiple cameras can simultaneously acquire images of the marker points from different perspectives. This allows for more accurate determination of the depth information of the marker points through methods such as triangulation, enabling 3D reconstruction of the marker points.

[0065] In some embodiments, the three-dimensional scanning device comprises a structured light projector for projecting structured light to the target object, and a fill light for filling light for the target object; the structured light image frame is captured when the structured light projector is in an open state and the fill light is in a closed state, and the marker point image frame is captured when the structured light projector is in a closed state and the fill light is in an open state. For example, for a scene with low scanning environment brightness, the camera can clearly capture the image of the marker point only when the fill light is turned on. Therefore, the on-off state of the structured light projector and the fill light can be controlled in time sequence, so that the same camera can capture the structured light pattern and the marker point pattern in time sequence.

[0066] In some embodiments, the three-dimensional scanning device comprises a structured light projector, which comprises a structured light mode for projecting structured light, and a uniform light mode for projecting uniform light. The structured light image frame is captured when the structured light projector is in the structured light mode, and the marker point image frame is captured when the structured light projector is in the uniform light mode. For some structured light projectors, they comprise multiple projection modes, i.e. they can project structured light with coded information, or uniform light without structure. The uniform light can be white light, or colored light such as blue light or red light. Therefore, the uniform light projected by the structured light projector can be used to fill light for the target object, so that an additional fill light does not need to be provided in the three-dimensional scanning device, simplifying the structure of the three-dimensional scanning device.

[0067] For the scene where the structured light image frame and the marker point image frame are captured in time sequence, since the structured light image frame and the marker point image frame are captured by the camera in different poses, and considering that the camera usually moves at a constant speed, the camera motion estimation can be performed based on at least two adjacent image frames of the same type, to determine the pose transformation of the camera when capturing adjacent two image frames, so as to determine the coordinate transformation relationship between the structured light image frame A and the adjacent marker point image frame B, i.e. the coordinate transformation relationship between the three-dimensional point cloud of the single-frame target object corresponding to the structured light image frame A and the three-dimensional point cloud of the single-frame target object corresponding to the marker point image frame B.

[0068] The camera motion estimation can be performed in multiple ways. In some scenarios, the ICP (Iterative Closest Point) algorithm can be used to determine the camera motion relationship between adjacent frames. Of course, other algorithms can also be used, and the embodiments of the present application do not make any limitation.

[0069] Therefore, in some embodiments, if the structured light image frames and the marker point image frames are acquired in a time sequence, for each frame of the three-dimensional point cloud of the target object, a coordinate transformation relationship between the frame of the three-dimensional point cloud of the target object and a frame of the three-dimensional point cloud of the marker point can be determined, wherein the marker point image frame corresponding to the frame of the three-dimensional point cloud of the marker point is adjacent to the structured light image frame corresponding to the frame of the three-dimensional point cloud of the target object. In determining the coordinate transformation relationship between the frame of the three-dimensional point cloud of the target object and the frame of the three-dimensional point cloud of the marker point, motion estimation can be performed on at least two adjacent image frames of the same type in the first image frame sequence to determine a pose transformation relationship of the camera when acquiring the two adjacent image frames, wherein the time interval between the acquisition of the at least two image frames of the same type and the structured light image frame corresponding to the frame of the three-dimensional point cloud of the target object is less than a preset time interval, and then the second coordinate transformation relationship can be determined based on the pose transformation relationship. Considering that the motion of the camera in a short time during the scanning process can be regarded as uniform motion, in order to accurately estimate the coordinate transformation relationship between the frame of the three-dimensional point cloud of the target object and the frame of the three-dimensional point cloud of the marker point, the acquisition time of the at least two image frames of the same type should be as close as possible to the acquisition time of the frame of the three-dimensional point cloud of the target object, and since motion estimation requires that the same content is included in the two image frames, the motion of the camera can be estimated based on the at least two image frames of the same type, which can be both structured light image frames or both marker point image frames.

[0070] In some embodiments, as shown in FIG. 1, the structured light image frames and the marker point image frames in each group of the first image frame sequence are arranged alternately. That is, the same camera can alternately acquire the structured light pattern and the marker point pattern. Figure 6

[0071] For the above acquisition mode, when determining the three-dimensional point cloud of different frames of the target object, the following method can be used:

[0072] Suppose the first image frame sequence includes: structured light image frame 1, marker point image 1, structured light image frame 2, marker point image 2, structured light image frame 3, marker point image 3, and so on.

[0073] The single-frame point clouds constructed based on the respective image frames are: three-dimensional point cloud 1 of the target object, three-dimensional point cloud 1 of the marker point, three-dimensional point cloud 2 of the target object, three-dimensional point cloud 2 of the marker point, three-dimensional point cloud 3 of the target object, three-dimensional point cloud 3 of the marker point, and so on.

[0074] ​For the three-dimensional point cloud 1 of the target object, a coordinate conversion relationship between the three-dimensional point cloud 1 and the three-dimensional point cloud 1 of the marker point can be determined. Specifically, motion estimation can be performed on the camera based on the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 2 of the target object, and a pose transformation of the camera when the three-dimensional point cloud 1 of the target object is collected and the three-dimensional point cloud 2 of the target object is collected is determined. Assuming R, the camera can be considered as uniform motion in a short time, and therefore, the pose transformation between the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 1 of the marker point is R / 2. The pose transformation is the coordinate conversion relationship between the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 1 of the marker point.

[0075] In some embodiments, as shown in FIG. 6, each of the first image frame sequences includes a plurality of repeatedly arranged image groups, each of the image groups includes a structured light image frame and a marker point structure frame, and each of the image groups includes at least two consecutively arranged image frames of the same type. Figure 7

[0076] Considering that the camera can be considered as uniform motion in a short time, in order to more accurately perform motion estimation on the camera, when the structured light pattern frame or the marker point pattern is collected in time sequence, the camera can be controlled to continuously collect at least two image frames of the same type, and then collect an image frame of another type, and then repeat the above collection sequence to obtain a plurality of image groups. For example, at least two structured light image frames can be continuously collected, and then a marker point image frame can be collected, or at least two marker point image frames can be continuously collected, and then a structured light image frame can be collected. Thus, motion estimation can be performed on the camera based on the at least two structured light image frames (or at least two marker point image frames), and motion information of the camera can be determined. The motion information can be used as the motion information of the camera when the image group is collected, so as to facilitate determination of the pose transformation of the camera when adjacent structured light image frames and marker point image frames in the image group are collected.

[0077] Considering that the image frames in the same image group share the same motion information, in order to ensure that the motion information is as accurate as possible, the consecutively arranged image frames of the same type in the image group can include only two. For example, in some embodiments, each image group includes two consecutively arranged structured light image frames and one marker point image frame. In some embodiments, each image group includes two consecutively arranged marker point image frames and one structured light image frame. Since the collection time interval is short, the camera motion information determined by using the two consecutive structured light image frames (or two consecutive structured light image frames) is more accurate when it is applied to the structured light image frames and the adjacent marker point image frames in the image group.

[0078] For example, for the above collection method, when determining the three-dimensional point cloud of different frames of the target object, the following method can be used:

[0079] ​Suppose the first image frame sequence comprises: structured light image frame 1, structured light image frame 2, landmark point image 1, structured light image frame 3, structured light image frame 4, landmark point image frame 2, structured light image frame 5, structured light image frame 6, landmark point image frame 3……

[0080] The single-frame point clouds constructed based on the respective image frames are: three-dimensional point cloud 1 of the target object, three-dimensional point cloud 2 of the target object, three-dimensional point cloud 1 of the landmark point, three-dimensional point cloud 3 of the target object, three-dimensional point cloud 4 of the target object, three-dimensional point cloud 2 of the landmark point, three-dimensional point cloud 5 of the target object, three-dimensional point cloud 6 of the target object, three-dimensional point cloud 3 of the landmark point……

[0081] Motion estimation can be performed based on the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 2 of the target object, and the pose transformation of the camera is determined, which is then taken as the pose transformation of the camera when the three-dimensional point cloud 2 of the target object and the three-dimensional point cloud 1 of the landmark point are captured. It is considered that the time intervals of the above three point clouds are very short, i.e. the motion speed of the camera when the three-dimensional point cloud 1 of the target object and the three-dimensional point cloud 2 of the target object are captured is consistent with the motion speed of the camera when the three-dimensional point cloud 2 of the target object and the three-dimensional point cloud 1 of the landmark point are captured, and thus the pose transformation of the camera when the three-dimensional point cloud 2 of the target object and the three-dimensional point cloud 1 of the landmark point are captured determined in this way is relatively accurate.

[0082] In some embodiments, in order to distribute the structured light pattern and the landmark point pattern in different image frames, different cameras can also be used to capture the structured light pattern and the landmark point pattern respectively. For example, the image frame set comprises a second image frame sequence and a third image frame sequence, the second image frame sequence comprises the above-mentioned multiple structured light image frames, the third image frame sequence comprises the above-mentioned multiple landmark point image frames, and the second image frame sequence and the third image frame sequence are captured by two cameras in the three-dimensional scanning device respectively. It is considered that the response characteristics of different cameras to different waveband light beams are different, and thus the structured light projected by the structured light projector and the light beam projected by the supplementary light lamp can be set as light beams of different wavebands, for example, light beam 1 and light beam 2, and then two cameras, camera 1 and camera 2, are arranged in the three-dimensional scanning device, wherein the sensor in camera 1 can only sense light beam 1, and the sensor in camera 2 can only sense light beam 2. Therefore, when the structured light projector and the supplementary light lamp are both in the open state, the above-mentioned two cameras can be used to capture images of the target object at the same time, and camera 1 captures structured light image frames and camera 2 captures landmark point image frames.

[0083] In some embodiments, if the structured light image frame and the landmark point image frame are captured by two cameras respectively, for each frame of the three-dimensional point cloud of the target object, a coordinate conversion relationship between the three-dimensional point cloud of the frame of the target object and the three-dimensional point cloud of one of the frames of the landmark points can be determined, wherein the three-dimensional point cloud of the one of the frames of the landmark points has the same capture time as the three-dimensional point cloud of the target object. In determining the second coordinate conversion relationship between the three-dimensional point cloud of the frame of the target object and the three-dimensional point cloud of the one of the frames of the landmark points, the second coordinate conversion relationship between the three-dimensional point cloud of the frame of the target object and the three-dimensional point cloud of the one of the frames of the landmark points can be determined based on the extrinsic parameters of the two cameras determined in advance. That is, since the structured light image frame and the landmark point image frame are captured by two cameras at the same time, the coordinate conversion relationship between the two can be determined by the extrinsic parameters of the two cameras.

[0084] In some embodiments, the three-dimensional scanning device is an oral cavity scanner, and the target object is an oral cavity, wherein the oral cavity is provided with a scanning rod provided with the landmark points, or the teeth and gums in the oral cavity are pasted with the landmark points, or the oral cavity is provided with a target provided with the landmark points.

[0085] In some embodiments, the three-dimensional scanning device is a face scanner, and the target object is a face, wherein the face is pasted with the landmark points, or the face is provided with a target provided with the landmark points.

[0086] In some embodiments, the three-dimensional scanning device is a handheld scanner, and the target object is an industrial product or an artifact or a building, wherein the industrial product or the artifact or the building is pasted or projected or sprayed with the landmark points.

[0087] In some embodiments, after the three-dimensional point clouds of the multiple frames of the target object are spliced based on the first coordinate conversion relationship and the second coordinate conversion relationship, a three-dimensional model of the target object currently spliced can be displayed in real time on the interactive interface. The three-dimensional model can be pasted with a texture image or not.

[0088] In some embodiments, the three-dimensional scanning device is an oral cavity scanner, and the target object is an edentulous jaw oral cavity, wherein the oral cavity is provided with a scanning rod provided with the landmark points. When the scanning process is advancing, the scanning rod and the edentulous jaw oral cavity can be displayed step by step on the interactive interface, and the landmark point texture image pasted can be displayed or not displayed on the scanning rod according to the user's demand, so as to meet the user's development demand.

[0089] It is not difficult to understand that the schemes described in the above embodiments can be freely combined to obtain new schemes in the absence of conflicts. Due to the length of the application, the embodiments of the present application are not listed one by one.

[0090] Correspondingly, the embodiment of the present application further provides a three-dimensional reconstruction device, as shown in Figure 8 The three-dimensional reconstruction device 80 comprises:

[0091] An acquisition module 81 is configured to acquire a set of image frames collected by a three-dimensional scanning device in a process of scanning a target object, wherein the target object is provided with a landmark point, and the target object surface is projected with a structured light in at least a part of a time period of scanning the target object;

[0092] A three-dimensional reconstruction module 82 is configured to perform three-dimensional reconstruction on the landmark point based on an image frame including a landmark point pattern in the set of image frames, to obtain a three-dimensional point cloud of a plurality of frames of landmark points, and to determine a first coordinate conversion relationship between the three-dimensional point clouds of the plurality of frames of landmark points; and perform three-dimensional reconstruction on the target object based on an image frame including a structured light pattern in the set of image frames, to obtain a three-dimensional point cloud of a plurality of frames of target objects;

[0093] A splicing module 83 is configured to, for each frame of three-dimensional point cloud of the target object, determine a second coordinate conversion relationship between the three-dimensional point cloud of the frame of target object and the three-dimensional point cloud of one frame of landmark point, and splice the three-dimensional point clouds of the plurality of frames of target objects based on the first coordinate conversion relationship and the second coordinate conversion relationship.

[0094] The device performs the specific steps of the task processing method, which can refer to the description in the above method embodiments, and will not be repeated here.

[0095] Further, the embodiment of the present application further provides an electronic device, as shown in Figure 9 The device comprises a processor 91, a memory 92, computer instructions stored in the memory 92 and executable by the processor 91, and the processor 91 executes the computer instructions to implement the method of any one of the above embodiments.

[0096] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method of any one of the above embodiments.

[0097] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carriers.

[0098] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of the present application can be implemented by means of software plus necessary universal hardware platforms. Based on such an understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0099] The systems, devices, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0100] The various embodiments described in the specification are intended to be exemplary only and the scope of the application is not intended to be limited by the embodiments described in the specification. The same parts in each of the embodiments are identical in reference, and each of the embodiments focuses on the differences from other embodiments. In particular, the device embodiments are described more simply because they are basically similar to the method embodiments, and the relevant parts are described in the method embodiments. The device embodiments described above are only schematic, and the modules described as separate components can or can not be physically separated, and the functions of each module can be implemented in the same or multiple software and / or hardware. Some or all of the modules can be selected to achieve the purpose of the embodiments of the present application according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0101] The above is only a specific implementation of the embodiments of the present application. It should be noted that for those skilled in the art, without departing from the principles of the embodiments of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the embodiments of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: A set of image frames acquired by a 3D scanning device during the scanning of a target object is obtained. The target object surface is marked with marker points, and structured light is projected onto the target object surface for at least a portion of the scanning time. The image frame set includes multiple composite image frames, each containing both a structured light pattern and a marker point pattern; or, the image frame set includes multiple structured light image frames and multiple marker point image frames, each structured light image frame containing only a structured light pattern, and each marker point image frame containing only a marker point pattern. Based on the image frames containing marker point patterns in the image frame set, the marker points are reconstructed in 3D to obtain a 3D point cloud of multiple marker points, and a first coordinate transformation relationship is determined between the 3D point clouds of the multiple marker points. Based on the image frames including structured light patterns in the image frame set, the target object is reconstructed in three dimensions to obtain a three-dimensional point cloud of the target object in multiple frames. For each frame of the target object's 3D point cloud, a second coordinate transformation relationship is determined between the 3D point cloud of the target object in that frame and the 3D point cloud of a marker point in one of the frames. Based on the first coordinate transformation relationship and the second coordinate transformation relationship, the 3D point clouds of the target object in multiple frames are stitched together. The 3D point cloud of the marker point in one of the frames and the 3D point cloud of the target object in that frame satisfy one of the following conditions: they are reconstructed from the same frame image; the image of the reconstructed 3D point cloud of the marker point in one of the frames and the image of the reconstructed 3D point cloud of the target object in that frame are acquired by different cameras at the same time; the acquisition time interval between the image of the reconstructed 3D point cloud of the marker point in one of the frames and the image of the reconstructed 3D point cloud of the target object in that frame is less than a preset time.

2. The method according to claim 1, characterized in that, The density of the structured light pattern in the composite image frame is determined based on the size of the marker points.

3. The method according to claim 1, characterized in that, If the density of the structured light pattern projected by the 3D scanning device is less than the preset density, then the image frame set includes the plurality of composite image frames; If the density of the structured light pattern projected by the 3D scanning device is greater than or equal to a preset density, then the image frame set includes the plurality of structured light image frames and the plurality of marker point image frames.

4. The method according to claim 1, characterized in that, The 3D scanning device includes a structured light projector for projecting structured light onto the target object, and a fill light for supplementing the target object with light; the composite image frame is acquired when the structured light projector is in the on state and the fill light is in the on state.

5. The method according to claim 1, characterized in that, The image frame set includes at least one set of first image frame sequences. Each set of first image frame sequences is acquired by one camera in the three-dimensional scanning device. Each set of first image frame sequences includes multiple structured light image frames and multiple marker point image frames. The structured light image frames and the marker point image frames are acquired by the camera in a time-division manner.

6. The method according to claim 5, characterized in that, The 3D scanning device includes a structured light projector for projecting structured light onto the target object, and a supplementary light for providing supplementary lighting to the target object; the structured light image frame is acquired when the structured light projector is on and the supplementary light is off, and the marker point image frame is acquired when the structured light projector is off and the supplementary light is on; or The 3D scanning device includes a structured light projector, which includes a structured light mode for projecting structured light and a uniform light mode for projecting uniform light. The structured light image frame is acquired when the structured light projector is in the structured light mode, and the marker point image frame is acquired when the structured light projector is in the uniform light mode.

7. The method according to claim 5, characterized in that, The structured light image frames and the marker point image frames in the first image frame sequence are arranged alternately.

8. The method according to claim 5, characterized in that, The first image frame sequence includes multiple repeated image groups, each image group including the structured light image frame and the marker point image frame, and each image group includes at least two consecutively arranged image frames of the same type.

9. The method according to claim 8, characterized in that, Each image group consists of two consecutively arranged structured light image frames and one marker image frame; or Each image group consists of two consecutively arranged marker image frames and one structured light image frame.

10. The method according to claim 5, characterized in that, The image frame of the marker point corresponding to the three-dimensional point cloud of one of the marker points is adjacent to the structured light image frame of the three-dimensional point cloud of the target object in that frame. For each frame of the target object's 3D point cloud, determine the second coordinate transformation relationship between the 3D point cloud of the target object in that frame and the 3D point cloud of a marker point in one of the frames, including: Motion estimation is performed on at least two adjacent image frames of the same type in the first image frame sequence to determine the pose transformation relationship when the camera acquires two adjacent image frames. The acquisition time interval between the at least two image frames of the same type and the structured light image frame corresponding to the three-dimensional point cloud of the target object in the frame is less than a preset time interval. The second coordinate transformation relationship is determined based on the pose transformation relationship.

11. The method according to claim 1, characterized in that, The image frame set includes a second image frame sequence and a third image frame sequence. The second image frame sequence includes the plurality of structured light image frames, and the third image frame sequence includes the plurality of marker point image frames. The second image frame sequence and the third image frame sequence are acquired by two cameras in the three-dimensional scanning device, respectively.

12. The method according to claim 11, characterized in that, The acquisition time of the 3D point cloud of one frame of marker points is the same as that of the 3D point cloud of the target object in that frame; for each frame of the 3D point cloud of the target object, the second coordinate transformation relationship between the 3D point cloud of the target object in that frame and the 3D point cloud of one frame of marker points is determined, including: Based on the pre-calibrated extrinsic parameters of the two cameras, determine the second coordinate transformation relationship between the 3D point cloud of the target object in this frame and the 3D point cloud of the marker point in one of the frames.

13. The method according to claim 1, characterized in that, The three-dimensional scanning device is an oral scanner, the target object is the oral cavity, and the oral cavity is an oral cavity with a scanning rod or tooth and gum adhesive markers installed, or an oral cavity with a target installed. and / or After stitching together the 3D point clouds of the target object from multiple frames based on the first coordinate transformation relationship and the second coordinate transformation relationship, the 3D model of the target object obtained by stitching together is displayed in real time on the interactive interface.

14. A three-dimensional reconstruction device, characterized in that, The three-dimensional reconstruction device includes: An acquisition module is used to acquire a set of image frames collected by a 3D scanning device during the scanning of a target object. The target object surface is provided with marker points, and structured light is projected onto the target object surface during at least a portion of the scanning time. The image frame set includes multiple composite image frames, each composite image frame including both a structured light pattern and a marker point pattern; or, the image frame set includes multiple structured light image frames and multiple marker point image frames, each structured light image frame including only a structured light pattern, and each marker point image frame including only a marker point pattern. The 3D reconstruction module is used to perform 3D reconstruction of the marker points based on image frames containing marker point patterns in the image frame set, to obtain a 3D point cloud of multiple frames of marker points, and to determine a first coordinate transformation relationship between the 3D point clouds of the multiple frames of marker points; and to perform 3D reconstruction of the target object based on image frames containing structured light patterns in the image frame set, to obtain a 3D point cloud of the target object. The stitching module is used to determine the second coordinate transformation relationship between the 3D point cloud of the target object in each frame and the 3D point cloud of a marker point in one frame, and to stitch the 3D point clouds of the target object in multiple frames based on the first coordinate transformation relationship and the second coordinate transformation relationship. The 3D point cloud of the marker point in one frame and the 3D point cloud of the target object in that frame satisfy one of the following conditions: they are reconstructed from the same frame image; the image of the reconstructed 3D point cloud of the marker point in one frame and the image of the reconstructed 3D point cloud of the target object in that frame are acquired by different cameras at the same time; the acquisition time interval between the image of the reconstructed 3D point cloud of the marker point in one frame and the image of the reconstructed 3D point cloud of the target object in that frame is less than a preset time.

15. An electronic device, characterized in that, The electronic device includes a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein the processor executes the computer instructions to implement the method as described in any one of claims 1-13.

Images