Three-dimensional reconstruction method, device, equipment and storage medium

Abstract

Embodiments of the present disclosure provide a three-dimensional reconstruction method, device, equipment and storage medium. The motion speed of a three-dimensional scanning device can be determined based on the pose change when a camera in the three-dimensional scanning device captures two or more images and the time interval of capturing the two or more images. For any target image in the two or more images or a target image with a smaller time interval of capturing the two or more images, the second pose when the target image ends exposure can be determined based on the exposure time of the target image, the motion speed of the three-dimensional scanning device and the first pose when the three-dimensional scanning device starts capturing the target image. Then, the first position information and the second position information of the three-dimensional point corresponding to each pixel point in the target image can be solved based on the first pose and the second pose respectively, and some target three-dimensional points with smaller errors are selected based on the first position information and the second position information, which are used to construct a three-dimensional model, so as to improve the accuracy of the three-dimensional model.

Description

Technical Field

[0001] This disclosure relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, apparatus, device and storage medium. Background Technology

[0002] Currently, 3D scanning equipment is widely used in various fields to achieve 3D reconstruction of target scenes. Taking handheld 3D scanning equipment as an example, users can move the device around the target object and generate a 3D model of that object in real time. Some 3D scanning equipment includes a camera, which collects image data of the target object for 3D reconstruction. However, during image acquisition, the camera requires a certain exposure time. During this time, the 3D scanning equipment and the target object may move relative to each other. For example, in a scenario where the user is holding the 3D scanning equipment, hand tremors can cause the equipment to move relative to the target object, resulting in inaccurate data and consequently, an inaccurate reconstructed 3D model. Summary of the Invention

[0003] This disclosure provides a three-dimensional reconstruction method, apparatus, device, and storage medium.

[0004] According to a first aspect of the present disclosure, a three-dimensional reconstruction method is provided, the method comprising:

[0005] The method involves acquiring at least two frames of images captured by a camera in a 3D scanning device, determining the pose change of the camera when capturing the at least two frames, determining the movement speed of the 3D scanning device based on the pose change and the acquisition time interval between the at least two frames, and for any target image, determining the second pose of the camera when the exposure of the target image ends based on the first pose of the camera when the exposure of the target image begins, the movement speed, and the exposure duration of the target image; wherein the target image is any frame of the at least two frames, or the maximum acquisition time interval between the target image and each frame of the at least two frames is less than a first preset duration.

[0006] For a pixel in the target image, first position information of a three-dimensional point corresponding to the pixel is determined based on the first pose, and second position information of a three-dimensional point corresponding to the pixel is determined based on the second pose.

[0007] Based on the difference between the first location information and the second location information, target 3D points are selected from the 3D points corresponding to the pixels of the target image, so as to use the target 3D points for 3D reconstruction.

[0008] According to a second aspect of the present disclosure, a three-dimensional scanning apparatus is provided, the apparatus comprising:

[0009] The acquisition module is used to acquire at least two frames of images captured by a camera in a 3D scanning device and determine the pose changes of the camera when acquiring the at least two frames of images.

[0010] A motion speed determination module is used to determine the motion speed of the three-dimensional scanning device based on the pose change and the acquisition time interval between at least two frames of images.

[0011] The pose determination module is used to determine, for any target image, the second pose of the camera when the exposure of the target image ends, based on the first pose of the camera when the exposure of the target image begins, the movement speed, and the exposure duration of the target image; wherein, the target image is any frame of the at least two frames, or the maximum acquisition time interval between the target image and each frame of the at least two frames is less than a first preset duration.

[0012] The position information determination module is used to determine, based on the first pose, a first position information of a three-dimensional point corresponding to a pixel in the target image, and to determine, based on the second pose, a second position information of the three-dimensional point corresponding to the pixel.

[0013] The filtering module is used to filter out target 3D points from the 3D points corresponding to the pixels of the target image based on the difference between the first location information and the second location information, so as to use the target 3D points for 3D reconstruction.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device including a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein when the processor executes the computer instructions, it can implement the method mentioned in the first aspect above.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing computer instructions that, when executed, implement the method mentioned in the first aspect above.

[0016] In this embodiment of the disclosure, during the process of 3D reconstruction of a target object using a 3D scanning device, the motion speed of the 3D scanning device can be determined based on the pose changes when the camera in the 3D scanning device acquires two or more frames of images, and the acquisition time interval between these two or more frames of images. For any frame of the target image in these two or more frames, or a target image with a small acquisition time interval with these two or more frames, the pose changes of the 3D scanning device during the exposure process of that frame of target image can be determined based on the exposure duration of that frame of target image and the motion speed of the 3D scanning device. Since the first pose at the start of acquisition of that frame of target image by the 3D scanning device can be determined, the second pose at the end of the exposure of the target image can be determined based on the pose changes during the exposure process. Then, the first position information and the second position information of the 3D points corresponding to each pixel in the target image can be solved based on the first pose and the second pose, respectively. Target 3D points with smaller errors are selected based on the difference between the first position information and the second position information, and a 3D model is constructed using the target 3D points, thereby improving the accuracy of the constructed 3D model.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0019] Figure 1 This is a schematic diagram illustrating an embodiment of the present disclosure of scanning an object using a handheld 3D scanning device.

[0020] Figure 2 This is a schematic diagram of the structure of a structured light 3D scanning device according to an embodiment of the present disclosure.

[0021] Figure 3 This is a schematic diagram of the structure of a three-dimensional scanning device according to an embodiment of the present disclosure.

[0022] Figure 4 This is a schematic diagram of a three-dimensional reconstruction method according to an embodiment of the present disclosure.

[0023] Figure 5 This is a schematic diagram of a three-dimensional reconstruction method according to an embodiment of the present disclosure.

[0024] Figure 6 This is a schematic diagram of the logical structure of a three-dimensional scanning device according to an embodiment of the present disclosure.

[0025] Figure 7 This is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0027] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0029] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0030] 3D scanning equipment is widely used in fields such as dentistry, industrial product manufacturing, and mold making to achieve 3D reconstruction of target scenes. For example... Figure 1 The diagram illustrates a method of scanning a target object using a handheld 3D scanning device. Users can move the handheld 3D scanning device around the target object and send the collected data to other devices (such as tablets or computers). These other devices can then generate and display a 3D model of the target object in real time based on the collected data.

[0031] Non-contact 3D scanning equipment can generally be divided into active and passive types. Active 3D scanning equipment is mainly divided into 3D scanning equipment based on TOF (Time of Flight) technology and structured light 3D scanning equipment.

[0032] The principle of Time-of-Flight (TOF) based 3D scanning equipment is as follows: electromagnetic waves (such as lasers or millimeter waves) are emitted towards the target object, and the electromagnetic waves reflected back from the target object are received by a detector. Based on the interval between the emission and reception times of the electromagnetic waves, as well as the propagation speed of the electromagnetic waves, the distance between the target object and the 3D scanning equipment can be obtained, thus yielding the 3D data of the target object. Structured light 3D scanning equipment works based on the principle of optical triangulation. It consists of an optical projector (such as a laser) and a camera. Its principle is as follows: the optical projector projects a pattern encoded according to certain rules and patterns onto the target object. The encoded pattern is modulated by the surface shape of the target object, causing deformation. The deformed structured light is captured by the camera. The 3D shape of the object can be determined by the positional relationship between the camera and the projection light source and the degree of deformation of the structured light. Structured light can be categorized into point, line, grating, and surface structured light, depending on the type of beam projected by the projector. For example, ... Figure 2 The diagram shows a schematic of a structured light 3D scanning device. The device includes a laser and a camera. The laser projects a laser line, which forms a laser plane in three-dimensional space. This plane intersects with the curved surface of the object to be measured, forming a curve. After the curve is captured by the camera, the three-dimensional coordinates of each point on the curve can be obtained through the principle of triangulation.

[0033] Passive 3D scanning equipment uses a camera to capture images of the same target object from different angles, and then reconstructs the object in 3D based on these images. Unlike active 3D scanning equipment, passive 3D scanning equipment does not require actively projecting light onto the target object. Figure 3 The diagram shows a passive 3D scanning device. The 3D scanning device includes two cameras, which can acquire images of a target object. For a 3D point on the target object, the distance between the 3D point and the camera can be determined based on the parallax of the pixel coordinates of the 3D point in the images acquired by the two cameras, thereby obtaining the 3D data of the target object.

[0034] For 3D scanning devices that include cameras, these devices rely on image data acquired by the camera for 3D reconstruction. During image acquisition, each frame requires a certain exposure time, during which the 3D scanning device and the target object may move relative to each other, leading to inaccurate data. For example, in scenarios where a user holds the 3D scanning device, hand tremors can cause significant movement between the device and the target object, resulting in blurred images. Consequently, the positions of 3D points determined from these images will have large errors, leading to poor accuracy in the final 3D model. In fields such as dentistry, where high accuracy is required for the reconstructed 3D model is necessary, excessive relative movement between the scanning device and the target object will render the constructed model inadequate for these scenarios.

[0035] To address the aforementioned issues, one readily apparent approach is to minimize the exposure time during image acquisition by the camera, thereby controlling the relative motion between the 3D scanning device and the target object within a certain range. However, when employing this method, for 3D reconstruction of target objects with reflective or low-reflectivity surface materials, the short exposure time can easily result in a low signal-to-noise ratio and poor image quality.

[0036] While it is possible to improve the signal-to-noise ratio of structured light 3D scanning equipment by increasing the energy of the structured light laser during the measurement process, this can lead to excessive power consumption and may also be user-unfriendly due to the high laser energy.

[0037] Another approach that is easy to think of is to install a speed sensor in the 3D scanning device. By measuring the real-time speed of the device through the speed sensor, the amount of movement of the 3D scanning device during the exposure time can be obtained based on the integral of the speed over the exposure time. Furthermore, the user can be alerted when the speed is high. However, this solution would greatly increase the hardware cost and is quite cumbersome.

[0038] Based on this, embodiments of this disclosure provide a three-dimensional reconstruction method. During the three-dimensional reconstruction of a target object using a three-dimensional scanning device, the motion speed of the three-dimensional scanning device can be determined based on the pose changes when the camera in the three-dimensional scanning device acquires two or more frames of images, and the acquisition time interval between these two or more frames. For any target image in these two or more frames, or a target image with a small acquisition time interval from the two or more frames, the pose changes of the three-dimensional scanning device during the exposure process of that target image can be determined based on the exposure duration of that target image and the motion speed of the three-dimensional scanning device. Since the first pose (i.e., the pose when the target image begins exposure) of the three-dimensional scanning device at the start of acquiring that target image can be determined, the second pose at the end of the exposure of the target image can be determined based on the pose changes during the exposure process. Then, the first position information and the second position information of the three-dimensional points corresponding to each pixel in the target image can be solved based on the first pose and the second pose, respectively. Based on the first position information and the second position information, some three-dimensional points with large motion are identified and discarded, thereby removing some data with large errors. A three-dimensional model is constructed using data from three-dimensional points with higher accuracy, resulting in a more accurate three-dimensional model.

[0039] The three-dimensional reconstruction method provided in this embodiment can be executed by a three-dimensional scanning device. Of course, it can also be executed by other devices that are connected to the three-dimensional scanning device. For example, the three-dimensional scanning device can be connected to other devices (such as tablets, laptops, etc.) and then send the collected image data to other devices so that other devices can perform three-dimensional reconstruction in real time and display the reconstructed three-dimensional model.

[0040] The 3D scanning device in this disclosure can be any type of photogrammetric 3D scanning device, which includes at least one camera. For example, in some embodiments, the 3D scanning device can be a structured light 3D scanning device. In some embodiments, the 3D scanning device can also be a non-structured light binocular 3D scanning device, that is, a passive 3D scanning device that includes only two cameras. In some embodiments, the 3D scanning device can be a device obtained by improving upon the principles of the above two types of 3D scanning devices, and this disclosure does not impose any limitations.

[0041] The 3D scanning device in this embodiment can be a handheld 3D scanning device or other types of 3D scanning devices. As long as there is a large relative motion between the 3D scanning device and the target object during the acquisition of a frame of image, resulting in errors in the acquired data, the method of this embodiment can be used to reduce the errors.

[0042] The target object in this embodiment can be any target object that needs to be reconstructed in three dimensions, such as the oral cavity, teeth, various industrial products, etc. This embodiment does not impose any limitations.

[0043] like Figure 4 The diagram shown is a flowchart of a three-dimensional reconstruction method provided in an embodiment of this disclosure. Figure 5 This is a schematic diagram of the 3D reconstruction method, which may specifically include the following steps:

[0044] S402. Acquire at least two frames of images captured by the camera in the 3D scanning device, and determine the pose change of the camera when it acquires the at least two frames of images;

[0045] In step S402, during the process of using a 3D scanning device to acquire images of the target object for 3D reconstruction, at least two frames of images captured by the camera in the 3D scanning device can be obtained. Then, the pose changes of the camera when acquiring these at least two frames of images can be determined. For example, the pose changes of two adjacent frames in these at least two frames can be determined, or the pose changes of the first and last frames in these at least two frames can be determined.

[0046] like Figure 5 As shown, the example uses two frames of images; the method is similar for more than two frames. For instance, the at least two frames are image A and image B, where image A is acquired starting at time t1 and image B is acquired starting at time t2. The pose (position and orientation) of the 3D scanning device when acquiring images A and B can then be determined, for example, as R1T1 and R2T2 respectively. Here, R represents the rotation matrix of the camera relative to the origin of the world coordinate system (which can be customized), and T represents the translation matrix of the camera relative to the origin of the world coordinate system. The pose change of the camera when acquiring two frames can then be determined: ΔRT = R2T2 - R1T1, that is, the rotation angle and movement distance of the camera during the time interval (t2-t1) between the acquisition of these two frames.

[0047] The camera is fixedly installed in the 3D scanning device, and the change in the camera's pose is the change in the pose of the 3D scanning device.

[0048] In some scenarios, there is an overlapping region in at least two image frames. When determining the pose change between these at least two image frames captured by the camera, the pose change can be determined based on the overlapping region. In some scenarios, if the camera is mounted on a device such as a gimbal, the pose change when the camera captures these at least two image frames can also be determined based on the rotation angle of the gimbal. Of course, the embodiments disclosed herein are not limited to the above methods; any method that can determine the pose change when the camera captures different image frames is applicable.

[0049] S404. Based on the pose change when the camera acquires the at least two frames of images and the acquisition time interval of the at least two frames of images, determine the movement speed of the three-dimensional scanning device;

[0050] In step S404, after determining the pose change when the camera acquires at least two frames of images, the current motion speed of the 3D scanning device can be determined based on the pose change and the time interval between the acquisition of these at least two frames. The motion speed can be the average speed of the 3D scanning device during the acquisition of these at least two frames. For example, the overall pose change when the camera acquires the first and last frames of these at least two frames, as well as the acquisition time interval between the first and last frames, can be determined, and then the average speed of the 3D scanning device during the acquisition of these at least two frames can be calculated. Alternatively, the average speed between two adjacent frames can be calculated separately, and then the average value can be taken. The motion speed can include one or more of the 3D scanning device's rotational speed and translational speed.

[0051] For example, such as Figure 5 As shown, taking a scene with only two frames of images as an example, after determining the pose changes when the camera acquires images A and B, the acquisition time interval between the two frames can be determined based on the time t1 when image A starts acquiring and the time t2 when image B starts acquiring. Then, the movement speed of the 3D scanning device during this time can be determined based on the pose changes when acquiring the two frames of images and the acquisition time interval.

[0052] S406. For any target image, based on the first pose of the camera when the target image begins exposure, the movement speed, and the exposure duration of the target image, determine the second pose of the camera when the target image ends exposure; wherein, the target image is any frame among the at least two frames, or the maximum acquisition time interval between the target image and each frame among the at least two frames is less than a first preset duration.

[0053] In step S406, since the determined motion speed is the average speed of the 3D scanning device when acquiring these at least two frames of images, and considering that the speed of the 3D scanning device does not change much in a short period of time, the motion speed of the 3D scanning device during the exposure process of these at least two frames of images, or other images with a short acquisition time interval between these at least two frames of images, is also close to this motion speed. Therefore, for any frame of the target image in these at least two frames of images, or a target image whose maximum acquisition time interval between each frame of these at least two frames of images is less than the first preset duration, the pose change of the 3D scanning device during the exposure process of the target image can be determined using this motion speed. Thus, the first pose of the camera when the target image begins exposure and the second pose of the camera when the exposure ends can be obtained.

[0054] Taking a scene with only two frames as an example, the determined motion speed is the average speed during the time from the start of image A to the start of image B. The time from the start of image B (i.e., the start of image B's exposure) to the end of image B's exposure is short, and the camera's motion speed can be considered constant during this period. Therefore, for any frame of the target image in either image A or image B, the camera's pose at the start of exposure (hereinafter referred to as the first pose) has already been determined. For example, for image A, its first pose is R1T1, and for image B, its first pose is R2T2. Simultaneously, based on the camera's motion speed and the exposure duration of the target image, the pose change of the camera during the exposure process of the target image can be determined, thus obtaining the camera's pose at the end of the exposure of the target image, hereinafter referred to as the second pose. For example, for image B, the pose change during the exposure process is ΔR'T', then the second pose is: R3T3=R2T2+ΔR'T'.

[0055] S408. For the pixels of the target image, determine the first position information of the three-dimensional point corresponding to the pixel based on the first pose, and determine the second position information of the three-dimensional point corresponding to the pixel based on the second pose.

[0056] In step S408, after determining the first pose at the start of exposure and the second pose at the end of exposure of the target image, the position information of the three-dimensional points corresponding to each pixel in the target image can be determined based on the first pose, hereinafter referred to as the first position information. Simultaneously, the position information of the three-dimensional points corresponding to each pixel in the target image can be determined based on the second pose, hereinafter referred to as the second position information. For example, the depth information of the pixels can be solved based on the camera pose, thereby obtaining the position information of the three-dimensional points corresponding to each pixel. The specific method for solving the pixel depth information can be determined based on the type of 3D scanning device. For example, for a structured light 3D scanning device, the depth information of the pixels in the target image can be determined based on the triangulation principle. For a binocular 3D scanning device, the depth information of the pixels in the target image can be determined based on the parallax of the pixels in the images acquired by the two cameras.

[0057] S410. Based on the difference between the first position information and the second position information, target three-dimensional points are selected from the three-dimensional points corresponding to the pixels of the target image, so as to perform three-dimensional reconstruction using the target three-dimensional points.

[0058] In step S410, after determining the position information of the 3D points corresponding to each pixel in the target image in the first and second poses, the influence of camera motion on the final solved 3D point positions can be determined based on the difference between the two position information. For example, if the difference between the two position information is small, it means that the camera motion has little impact on the determination of the 3D point's position, that is, the determined 3D point's position information is relatively accurate with a small error. Conversely, it means that the camera motion has a significant impact on the determination of the 3D point's position, and the current determination of the 3D point's coordinates has a large error and is not accurate enough. 3D points with large errors can be discarded, thereby selecting target 3D points with higher accuracy.

[0059] For each frame of images acquired by a 3D scanning device, the above method can be used to filter out target 3D points with smaller errors from the corresponding 3D points of the pixels in the image. Then, the target 3D points determined from each frame of images are registered and fused to obtain the 3D model of the target object.

[0060] Of course, when determining the pose change of the camera during the exposure of the target image, the average speed during the acquisition of at least two frames is used as the motion speed of the exposure process. If the acquisition time interval between these two frames is too long, then using the average motion speed of this process as the motion speed of the target image exposure process will inevitably be inaccurate. To make the determined motion speed during the exposure of the target image more accurate, in some embodiments, the acquisition time interval between any two frames of the at least two images can be controlled to be less than a second preset duration, or the number of frames between any two frames of the at least two images can be controlled to be less than a preset number of frames. For example, taking a scenario with only two frames as an example, these two frames can be two frames continuously acquired by the camera. Since the frame rate of the 3D scanning device is high and the time interval is short, the motion speed during the exposure process determined by the above method is more accurate for two continuously acquired frames.

[0061] In some embodiments, when determining the pose change of the camera when capturing two frames of images, feature points can be extracted from these two frames respectively. These feature points can be points in the images with significant pixel value changes, such as contour points or boundary points. After extracting the feature points from the two frames, the feature points in the two frames can be matched to obtain matching feature point pairs. That is, the two pixels in a feature point pair correspond to the same three-dimensional point in three-dimensional space. After obtaining the feature point pairs, the poses corresponding to the two frames captured by the camera can be calculated based on these feature point pairs, thereby determining the pose change of the camera when capturing the two frames.

[0062] In some embodiments, to facilitate the determination of the camera's pose, a calibration object can be placed within the camera's field of view to assist in the calibration of the camera's intrinsic and extrinsic parameters. For example, the calibration object may include multiple corner points, the distances between these corner points being known. After the camera acquires the two frames of images, these corner points can be identified from the images. Then, based on the pixel coordinates of these corner points and the distances between them, the poses corresponding to the two frames of images can be calculated. For example, the PnP (Perspective-n-Point) algorithm can be used to calculate the camera's pose, thereby determining the pose changes of the camera when acquiring the two frames of images.

[0063] In some embodiments, after determining the motion speed of the 3D scanning device based on the pose changes during the acquisition of two or more image frames by the camera and the acquisition time interval of these two or more image frames, a preliminary judgment can be made based on the motion speed. For example, if the motion speed is relatively low, the motion of the 3D scanning device will not have a significant impact on the final measured 3D data, i.e., the impact of the motion speed can be ignored. In this case, there is no need to perform the subsequent step of removing 3D points with large errors, and 3D reconstruction can be performed directly using the image frames, thereby improving processing efficiency and saving processing resources. Of course, if the motion speed is high, the motion of the 3D scanning device will have a certain impact on the final measured 3D data. For example, it will cause some errors in the position information of certain pixels in the image frame. In this case, in order to ensure the accuracy of the measured 3D data, the subsequent step of removing 3D points with large errors can be performed. Therefore, two motion speed thresholds can be preset, namely a first motion speed threshold and a second motion speed threshold. When the motion speed is less than the first motion speed threshold, 3D reconstruction can be performed directly using the image frames, without performing the subsequent step of removing 3D points with large errors. If the motion speed is between the first motion speed threshold and the second motion speed threshold, then the subsequent steps of determining the second pose based on the motion speed and filtering the target 3D points are executed.

[0064] Of course, in some embodiments, if the motion speed exceeds a second motion speed threshold, it indicates that the motion speed is too high. Excessive motion speed may lead to inaccurate measurement of the entire frame of image data, resulting in inaccurate determination of the position information of the 3D points corresponding to that frame, severely affecting the accuracy of the reconstructed 3D model. In this case, at least two frames of image data can be directly deleted, and the step of selecting the target 3D points from these at least two frames can be skipped. By discarding image frames with significantly large errors, the accuracy of the reconstructed 3D model can be improved, and computational resources can be saved.

[0065] In some embodiments, for each pixel in the target image corresponding to a 3D point, after determining the first position information of the 3D point in a first pose and the second position information in a second pose, the moving distance of the 3D point can be determined based on the first and second position information. Then, 3D points whose moving distance is less than a preset distance can be selected as target 3D points. For example, the first position information corresponds to a 3D coordinate, and the second position information corresponds to a 3D coordinate. The distance between the two 3D coordinates can be calculated as the moving distance of the 3D point. This distance can be Euclidean distance, Mahalanobis distance, etc. If the moving distance is small, it means that the relative motion between the 3D scanning device and the target object has little impact on the determination of the 3D point's position, that is, the error of the 3D point's position data is small, and therefore it can be used as a target 3D point for subsequent 3D model construction. Conversely, if the moving distance is large, it is a 3D point with a large error and therefore needs to be discarded. In this way, more accurate 3D data can be obtained, and the constructed 3D model is also more accurate.

[0066] In some implementations, the 3D scanning device can be a handheld 3D scanning device. If the scanning speed is detected to be too fast, for example, if the scanning speed exceeds a third preset speed threshold, the user can be prompted to reduce the scanning speed to avoid inaccurate 3D data due to excessive speed. The third preset speed threshold can be the same as the first preset speed threshold, the same as the second preset speed threshold, or different from both the first and second preset speed thresholds.

[0067] In this process, prompts can be provided to the user via voice or visual cues. For example, an LED light can be installed on the handheld 3D scanning device to flash when excessive movement speed is detected. Alternatively, a voice prompt device can be installed on the handheld 3D scanning device to provide a voice prompt to the user when excessive movement speed is detected. It is easy to understand that the solutions described in the above embodiments can be combined where there is no conflict, and not all of them are listed in this disclosure.

[0068] Accordingly, this disclosure also provides a three-dimensional reconstruction apparatus, such as... Figure 6 As shown, the device 60 includes:

[0069] The acquisition module 61 is used to acquire at least two frames of images captured by the camera in the 3D scanning device and determine the pose change of the camera when it acquires the at least two frames of images.

[0070] The motion speed determination module 62 is used to determine the motion speed of the three-dimensional scanning device based on the pose change and the acquisition time interval between the at least two frames of images.

[0071] The pose determination module 63 is used to determine, for any target image, the second pose of the camera when the exposure of the target image ends, based on the first pose of the camera when the exposure of the target image begins, the movement speed, and the exposure duration of the target image; wherein the target image is any frame of the at least two frames, or the maximum acquisition time interval between the target image and each frame of the at least two frames is less than a first preset duration.

[0072] The position information determination module 64 is used to determine, based on the first pose, a first position information of a three-dimensional point corresponding to a pixel in the target image, and to determine, based on the second pose, a second position information of a three-dimensional point corresponding to a pixel.

[0073] The filtering module 65 is used to filter out target 3D points from the 3D points corresponding to the pixels of the target image based on the difference between the first position information and the second position information, so as to use the target 3D points for 3D reconstruction.

[0074] The specific steps for implementing the three-dimensional reconstruction method using the above-mentioned device can be found in the description of the above method embodiments, and will not be repeated here.

[0075] Furthermore, embodiments of this disclosure also provide a device, such as... Figure 7 As shown, the device includes a processor 71, a memory 72, and computer instructions stored in the memory 72 that can be executed by the processor 71. When the processor 71 executes the computer instructions, it implements the method described in any of the above embodiments.

[0076] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0077] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0078] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0079] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0080] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0081] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Acquire at least two frames of images captured by a camera in a 3D scanning device, and determine the pose changes of the camera when acquiring the at least two frames of images; The motion speed of the 3D scanning device is determined based on the pose change and the acquisition time interval between at least two frames of images. For any target image, based on the first pose of the camera when the target image begins exposure, the movement speed, and the exposure duration of the target image, the second pose of the camera when the target image ends exposure is determined; wherein, the target image is any frame among the at least two frames, or the maximum acquisition time interval between the target image and each frame among the at least two frames is less than a first preset duration; For a pixel in the target image, first position information of a three-dimensional point corresponding to the pixel is determined based on the first pose, and second position information of a three-dimensional point corresponding to the pixel is determined based on the second pose. Based on the difference between the first location information and the second location information, target 3D points are selected from the 3D points corresponding to the pixels of the target image, so as to use the target 3D points for 3D reconstruction.

2. The method according to claim 1, characterized in that, The time interval between any two frames in the at least two frames is less than the second preset duration, or The number of frames between any two frames in the at least two frames is less than a preset number of frames.

3. The method according to claim 1, characterized in that, Determining the pose changes of the camera when acquiring at least two frames of images includes: If there is an overlapping region in the at least two frames of images, feature points are extracted from each of the at least two frames of images. These feature points are then matched to obtain matched feature point pairs. Based on these feature point pairs, the poses of the camera when capturing the at least two frames of images are determined, and the pose changes are determined based on these poses. Each of the at least two frames of images includes a pre-set calibration object, which includes multiple corner points. The distance between each corner point is known. The corner points are extracted from the at least two frames of images. Based on the distance of the corner points and the pixel coordinates of the corner points, the poses corresponding to the camera when capturing the at least two frames of images are determined respectively. The pose changes are determined based on the corresponding poses.

4. The method according to claim 1, characterized in that, Before determining the second pose of the camera when the exposure of the target image ends, based on the first pose of the camera when the exposure of the target image begins, the movement speed, and the exposure duration of the target image, the method further includes: Determine whether the movement speed is greater than a first preset speed threshold and less than a second preset speed threshold; If so, then the operation of determining the second pose of the camera when the exposure of the target image ends is performed based on the first pose of the camera when the exposure of the target image begins, the movement speed, and the exposure duration of the target image.

5. The method according to claim 1 or 4, characterized in that, After determining the movement speed of the three-dimensional scanning device, the method further includes... Determine whether the movement speed is greater than a second preset speed threshold; If so, delete the at least two frames of images.

6. The method according to claim 1, characterized in that, Based on the difference between the first location information and the second location information, target 3D points are selected from the 3D points corresponding to the pixels of the target image, including: For the three-dimensional points corresponding to the pixels of the target image, the moving distance of the three-dimensional points is determined based on the first position information and the second position information of the three-dimensional points; The three-dimensional point whose moving distance is less than a preset distance is selected as the target three-dimensional point.

7. The method according to claim 1, characterized in that, The 3D scanning device is a handheld 3D scanning device, and the method further includes: If the movement speed of the 3D scanning device is determined to be greater than a third preset speed threshold, the user is prompted to reduce the scanning speed.

8. A three-dimensional scanning device, characterized in that, The device includes: The acquisition module is used to acquire at least two frames of images captured by a camera in a 3D scanning device and determine the pose changes of the camera when acquiring the at least two frames of images. A motion speed determination module is used to determine the motion speed of the three-dimensional scanning device based on the pose change and the acquisition time interval between at least two frames of images. The pose determination module is used to determine, for any target image, the second pose of the camera when the exposure of the target image ends, based on the first pose of the camera when the exposure of the target image begins, the movement speed, and the exposure duration of the target image; wherein, the target image is any frame of the at least two frames, or the maximum acquisition time interval between the target image and each frame of the at least two frames is less than a first preset duration. The position information determination module is used to determine, based on the first pose, a first position information of a three-dimensional point corresponding to a pixel in the target image, and to determine, based on the second pose, a second position information of the three-dimensional point corresponding to the pixel. The filtering module is used to filter out target 3D points from the 3D points corresponding to the pixels of the target image based on the difference between the first location information and the second location information, so as to use the target 3D points for 3D reconstruction.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein the processor executes the computer instructions to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images