Three-dimensional scanning method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional scanning method and device, equipment and a storage medium. In the process of scanning the target object by using the three-dimensional scanning equipment, the three-dimensional model obtained by performing real-time three-dimensional reconstruction on the target object based on the data acquired by the three-dimensional scanning equipment can be acquired, and meanwhile, the real-scene image of the target object can be acquired through the camera on the AR equipment; and projecting the three-dimensional model into the live-action image based on the matching relationship between the three-dimensional model and the target object in the live-action image so as to determine and mark the scanned area in the live-action image to obtain the augmented reality image, and then displaying the augmented reality image through a display screen of the AR device. Therefore, the user can see the real-time conditions of the target object and the three-dimensional scanning equipment and can also see the current scanning condition of the target object by checking the augmented reality image, the sight line does not need to be switched back and forth in the scanning process, and the operation is very convenient.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scanning technology and more particularly to a three-dimensional scanning method, device, equipment and storage medium. Background Art

[0002] With the development of 3D scanning technology, 3D scanning technology has been gradually applied to various fields such as medicine, industry, archaeology, etc., and 3D scanning equipment suitable for various fields has gradually emerged. At present, in the process of using 3D scanning equipment to scan the target object to reconstruct the 3D model of the target object, the 3D scanning equipment usually needs to be connected to terminal devices with display screens such as laptops and tablets, and the collected data is sent to these terminal devices in real time. These terminal devices perform real-time 3D reconstruction of the target object based on the collected data to obtain the 3D model of the target object, and display the currently reconstructed 3D model through the display screen.

[0003] When the user uses the 3D scanning device to scan the target object, it is necessary to adjust the position, movement speed, etc. of the 3D scanning device in real time based on the reconstruction of the current 3D model. Therefore, the user's line of sight needs to constantly switch back and forth between the 3D scanning device and the display screen of the terminal device, which brings a lot of inconvenience to the user and affects the user experience. Summary of the invention

[0004] The present invention provides a three-dimensional scanning method, device, equipment and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a three-dimensional scanning method is provided, the method comprising:

[0006] In the process of scanning the target object by using the three-dimensional scanning device, a three-dimensional model of the target object is obtained in real time, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device;

[0007] Acquire the real-life image of the target object captured by the AR device in real time;

[0008] Determining a matching relationship between the three-dimensional model and a target object in the real scene image, and projecting the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image;

[0009] The scanned area in the real scene image is marked to obtain an augmented reality image, so as to display the augmented reality image in real time through the AR device.

[0010] According to a second aspect of an embodiment of the present disclosure, a three-dimensional scanning device is provided, the device comprising:

[0011] A three-dimensional model acquisition module, used to acquire a three-dimensional model of a target object in real time during the process of scanning the target object using a three-dimensional scanning device, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device;

[0012] A real-scene image acquisition module is used to acquire the real-scene image of the target object collected by the AR device in real time;

[0013] A projection module, used to determine a matching relationship between the three-dimensional model and a target object in the real scene image, and project the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image;

[0014] The identification module identifies the scanned area in the real scene image to obtain an augmented reality image, so as to display the augmented reality image in real time through the AR device.

[0015] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and computer instructions stored in the memory and executable by the processor, wherein when the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the method mentioned in the first aspect is implemented.

[0017] In the disclosed embodiment, when a user uses a three-dimensional scanning device to scan a target object, a three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device can be obtained. At the same time, a real-life image of the target object can be collected by a camera on the AR device, and a matching relationship between the three-dimensional model and the target object in the real-life image can be determined. The three-dimensional model is projected onto the real-life image based on the matching relationship to determine a scanned area in the real-life image, and the scanned area in the real-life image is identified to obtain an augmented reality image. The augmented reality image can then be displayed on a display screen of the AR device, so that the user can see the real-time status of the target object and the three-dimensional scanning device by viewing the augmented reality image, and can also see the current scanning status of the target object. Therefore, the user can determine how to adjust the position of the three-dimensional scanning device and determine the scanning method by only viewing the content displayed on the display screen of the AR device, without having to switch the line of sight back and forth during the scanning process. The operation is very convenient, which improves the user experience during the scanning process.

[0018] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0020] Figure 1 It is a schematic diagram of using a three-dimensional scanning device to perform real-time scanning on a target object according to an embodiment of the present disclosure.

[0021] Figure 2 It is a schematic diagram of an application scenario of an embodiment of the present disclosure.

[0022] Figure 3 It is a flow chart of a three-dimensional scanning method according to an embodiment of the present disclosure.

[0023] Figure 4 It is a schematic diagram of identifying a scanning trajectory line in an augmented reality image according to an embodiment of the present disclosure.

[0024] Figure 5 It is a schematic diagram of identifying scanning quality in an augmented reality image according to an embodiment of the present disclosure.

[0025] Figure 6 It is a schematic diagram of identifying a specific type of area in an augmented reality image according to an embodiment of the present disclosure.

[0026] Figure 7 It is a schematic diagram of an embodiment of the present disclosure, which identifies the tooth currently under discussion in an augmented reality image displayed by AR glasses worn by a doctor and a patient.

[0027] Figure 8 It is a schematic diagram of the logical structure of a three-dimensional scanning device according to an embodiment of the present disclosure.

[0028] Fig. 9 It is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0031] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0033] At present, when using 3D scanning equipment to scan a target object to reconstruct a 3D model of the target object, the 3D scanning equipment usually needs to be connected to terminal devices with display screens such as laptops and tablets, and the collected data is sent to these terminal devices in real time. These terminal devices perform real-time 3D reconstruction of the target object based on the collected data to obtain a 3D model of the target object, and display the currently reconstructed 3D model on the display screen. Taking the oral scanning scene as an example, Figure 1 As shown, usually the oral scanning device can be connected to the doctor's computer for communication. When the doctor uses the oral scanning device to scan the patient's oral cavity, the oral scanning device can send the collected data to the computer in real time. The computer performs real-time three-dimensional reconstruction of the patient's oral cavity based on the collected data to obtain a reconstructed three-dimensional model of the oral cavity, and displays the real-time reconstructed three-dimensional model on the computer display.

[0034] When a user uses a 3D scanning device to scan a target object, he needs to determine the current scanning status of the target object based on the reconstruction status of the current 3D model, that is, which areas have been scanned, which areas have not been scanned, and the scanning quality of the scanned areas, so as to adjust the position and scanning speed of the 3D scanning device in real time. Therefore, the user's line of sight needs to switch back and forth between the target object and the display screen of the terminal device, which brings a lot of inconvenience to the user and affects the user experience. For example, Figure 1 As shown, the doctor needs to check the real-time reconstructed three-dimensional model displayed on the display screen, adjust the scanning method based on the reconstruction of the three-dimensional model, and then look at the patient's mouth to adjust the position and scanning speed of the oral scanning device. The line of sight needs to keep switching back and forth between the patient's mouth and the display screen, which is very cumbersome.

[0035] In view of this, the present disclosure provides a three-dimensional scanning method, which can combine AR (Augmented Reality) technology with three-dimensional scanning technology. In the process of a user using a three-dimensional scanning device to scan a target object, a three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device can be obtained. At the same time, the real-life image of the target object can be collected in real time by a camera on the AR device, and the matching relationship between the three-dimensional model and the target object in the real-life image can be determined. The three-dimensional model is projected into the real-life image based on the matching relationship to determine the scanned area in the real-life image, and the scanned area is marked to obtain an augmented reality image, which can then be displayed on the display screen of the AR device. Thus, by viewing the augmented reality image, the user can see the real-time status of the target object and the three-dimensional scanning device, as well as the current scanning status of the target object. Therefore, the user can determine how to adjust the position of the three-dimensional scanning device and determine the scanning method by only viewing the content displayed on the display screen of the AR device, without switching the line of sight back and forth during the scanning process. The operation is very convenient, which improves the user experience during the scanning process.

[0036] In some scenarios, the method of the embodiments of the present disclosure can be executed by an AR device. For example, the AR device can be connected to a three-dimensional scanning device for communication, and obtain data collected by the three-dimensional scanning device from the three-dimensional scanning device in real time, and then use the collected data to reconstruct a three-dimensional model of the target object. At the same time, the AR device can collect real-life images of the target object through a camera in real time, and project the three-dimensional model into the currently collected real-life image to obtain and display an augmented reality image.

[0037] In some scenarios, the method of the embodiment of the present disclosure may also be executed by other devices connected to the AR device, which may be devices with good processing performance such as laptops, tablet computers, and cloud servers. These devices can obtain data collected by the three-dimensional scanning device, and use these data to perform three-dimensional reconstruction of the target object to obtain a three-dimensional model. At the same time, they can obtain the real scene image collected by the AR device, project the three-dimensional model into the real scene image, obtain an augmented reality image, and then send it to the AR device, which will display the augmented reality image.

[0038] In some scenarios, the method of the embodiment of the present disclosure may also be that some steps are performed by AR devices, and some steps are performed by other terminal devices. For example, the step of obtaining a three-dimensional model by three-dimensional reconstruction is performed by other terminal devices, and the step of projecting the three-dimensional model into a real-scene image to generate an augmented reality image and displaying it is performed by the AR device. Among them, the three-dimensional scanning device can be a handheld or fixed scanner such as an oral scanner, a facial scanner, an ear scanner, an industrial scanner, a professional scanner, etc., which can realize the three-dimensional reconstruction of objects or scenes such as teeth, human faces, ears, human bodies, industrial products, industrial equipment, cultural relics, artworks, prostheses, medical devices, buildings, etc.

[0039] Among them, the specific settings of the execution entity can be flexibly set based on actual needs, and the embodiments of the present disclosure are not limited.

[0040] For example, Figure 2 As shown, it is a schematic diagram of an application scenario of an embodiment of the present disclosure. The user can hold a handheld oral scanning device to perform a three-dimensional scan of the patient's oral cavity. At the same time, the user can wear AR glasses. The AR glasses can obtain the data collected by the oral scanning device, and use the obtained data to reconstruct a three-dimensional model in real time, and project the three-dimensional model into the real-scene image of the oral cavity collected by the AR glasses, determine which area of ​​the real-scene image of the oral cavity has been scanned, and then mark the scanned area to obtain an augmented reality image, and display it on the display screen of the AR glasses. By viewing the content displayed by the AR glasses, the user can understand the current scanning status of the oral cavity and adjust the scanning method based on the scanning status.

[0041] The target object in the embodiment of the present disclosure may be any object that needs to be 3D reconstructed, such as an oral cavity, a building, a cultural relic, etc. Correspondingly, the 3D scanning device in the embodiment of the present disclosure may also be any type of 3D scanning device, such as an oral scanning device, an industrial scanning device, etc.

[0042] The AR device in the embodiments of the present disclosure may be various types of AR devices, for example, AR glasses, AR displays, etc. The AR device includes a camera and a display screen, wherein the camera is used to collect a real-life image of a target object, and the display screen is used to display the resulting augmented reality image. In addition, the AR device may also include various sensors for determining the current position of the user's line of sight.

[0043] like Figure 3 As shown, the method may include the following steps:

[0044] S302, in the process of scanning the target object by using the three-dimensional scanning device, obtaining a three-dimensional model of the target object in real time, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device;

[0045] In step S302, when the user uses the 3D scanning device to scan the target object, the 3D model of the target object can be obtained in real time. The 3D model can be obtained by real-time 3D reconstruction of the target object based on the data collected by the 3D scanning device. For example, all the data collected at the current moment can be 3D reconstructed at fixed intervals (for example, 0.5s) to obtain the 3D model at the current moment. As time goes by, the collected data gradually increases, and the real-time reconstructed 3D model also tends to be more complete.

[0046] S304, acquiring a real-scene image of the target object collected by the AR device in real time;

[0047] In step S304, a real-scene image of the target object captured by the AR device can be obtained in real time. The AR device may include a camera, and the real-scene image including the target object can be captured by the camera. At the same time, the relative position relationship between the three-dimensional scanning device and the target object can also be known based on the real-scene image. Of course, since the real-scene image also changes in real time, a frame of the real-scene image can also be captured at fixed intervals.

[0048] S306, determining a matching relationship between the three-dimensional model and the target object in the real scene image, and projecting the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image;

[0049] In step S306, after obtaining the three-dimensional model and the real scene image at the current moment, the matching relationship between the three-dimensional model and the target object in the real scene image can be determined, wherein the matching relationship refers to the correspondence between the three-dimensional points in the three-dimensional model and the pixel points in the real scene image. Since the target objects in the three-dimensional model and the real scene image both correspond to the real target objects in the real world, the coordinate systems of the two can be unified and converted to the same coordinate system. The conversion relationship is the matching relationship between the two. Then, based on the matching relationship, the currently reconstructed three-dimensional model can be projected into the real scene image to determine the projection area of ​​the three-dimensional model in the real scene image, and the projection area is the scanned area in the current real scene image.

[0050] S308: Mark the scanned area in the real scene image to obtain an augmented reality image, so as to display the augmented reality image in real time through the AR device.

[0051] Usually, when a user performs a three-dimensional scan on a target object, the user is more concerned about which areas of the current target object have been scanned and which areas have not been scanned, so that the user can perform a supplementary scan on the unscanned areas and collect data of the unscanned areas. In order to facilitate the user to intuitively know which areas are scanned areas, in step S308, the scanned areas in the real scene image can be marked to obtain an augmented reality image. Among them, there are many ways to mark the processing, as long as the user can distinguish which areas have been scanned. For example, the scanned area and the unscanned area can be rendered in different colors, or the scanned area can be framed out, or the projection points when the three-dimensional model is projected to the area can be displayed in the scanned area, wherein the density of the projection points can indicate the density of the three-dimensional model corresponding to the area. After obtaining the augmented reality image after the marking process, the augmented reality image can be displayed in real time through the AR device, so that the user can understand the current real-time scanning situation of the target object by viewing the augmented reality image. Considering that for different types of target objects, there are often corresponding better scanning trajectories during the scanning process, scanning the target object based on the better scanning trajectory can not only perform a comprehensive scan of the target object, but also improve the scanning efficiency. Therefore, in some embodiments, in order to facilitate user scanning, before the augmented reality image is displayed in real time through the AR device, the scanning trajectory data matching the target object can be obtained first, and then the scanning trajectory line is drawn in the augmented reality image based on the acquired scanning trajectory data, wherein the scanning trajectory line is used to guide the user to use the three-dimensional scanning device to perform a three-dimensional scan of the target object. For example, for each type of target object, the best scanning trajectory line for that type of target object can be pre-designed, and the data related to the scanning trajectory line, that is, the scanning trajectory data, can be stored, and the scanning trajectory data is used to indicate the position information and scanning direction information of the scanning trajectory line. By drawing the scanning trajectory line in the augmented reality image, the augmented reality image finally displayed to the user also includes the scanning trajectory line, so that the user can move the three-dimensional scanning device along the scanning trajectory line in the augmented reality image to scan the target object. In this way, the scanning efficiency can be greatly improved, the user operation is convenient, and the target object that the user has not scanned before can also be accurately operated to obtain a better scanning result.

[0052] like Figure 4 As shown, taking the target object as the oral cavity as an example, a scanning trajectory line can be drawn in the augmented reality image to guide the user to scan the oral cavity.

[0053] In some embodiments, when the scanned area in the real scene image is marked to obtain an augmented reality image, the three-dimensional model can be projected into the real scene image to obtain a projection image, and then the projection image is rendered into different colors and superimposed on the real scene image to mark the scanned area. The augmented reality image may include multiple layers, and the scanning trajectory line, the real scene image, and the projection image of the three-dimensional model may be located in different layers. When the three-dimensional model is superimposed on the real scene image, in order to avoid the three-dimensional model from blocking the scanning trajectory line, the layer where the scanning trajectory line is located can be placed on the top layer.

[0054] In addition, when users are scanning the target object, the scanning quality is also an issue that users are more concerned about, such as whether there are problems with missed scans in various areas of the target object, whether the scanning density of various areas meets the standard, whether there are problems such as sparse 3D points after reconstruction in some areas, and insufficient precision. In order to allow users to understand the scanning quality of each area that has been scanned in real time, in some embodiments, the scanned area can also be divided into multiple sub-areas, which can be divided equally or unequally, and then the scanning quality of each sub-area can be determined, and the scanning quality of each sub-area can be identified in the real-life image, so that users can intuitively understand the scanning status of the current scanned area. For example, after reconstructing a three-dimensional model based on data collected by a three-dimensional scanning device, the matching relationship between the three-dimensional model and the target object in the real-life image can be determined, and then the number and density of three-dimensional points corresponding to each sub-area of ​​the target object in the real-life image can be determined based on the matching relationship, wherein the denser the three-dimensional points, the greater the scanning density of the area, and the better the scanning quality. In some scenarios, the scanning quality of each sub-area can be identified by the depth of the color, for example, the darker the color, the denser the 3D points reconstructed after scanning the sub-area, and the better the scanning quality. Figure 5 As shown, the color depth can be used to indicate the scanning quality. Darker teeth indicate that the reconstructed 3D points are denser, and lighter teeth indicate that the reconstructed 3D points are sparser and need to be re-scanned. In some scenarios, some points can also be covered in each sub-area. The denser the covered points, the denser the 3D points reconstructed after scanning the sub-area, and the better the scanning quality. Of course, the specific identification method can be flexibly selected based on actual needs, as long as the user can intuitively understand the scanning quality of each sub-area of ​​the scanned area based on the final augmented reality image, so that the user can re-scan those sub-areas with poor scanning quality, thereby collecting more and more complete data.

[0055] Generally speaking, during the scanning process, some specific types of areas of the target object may be areas of concern to the user, so these areas can be marked out for the user's convenience. For example, in some scenarios, the target object is the oral cavity, and the specific type of area may be the diseased area in the oral cavity, such as caries. Doctors usually pay more attention to these parts of the oral cavity, so these areas can be marked out. In some scenarios, the target object may be a pipe, and the specific type of area may be an area with cracks in the pipe, or an area with rust and corrosion in the pipe. Of course, for different scenarios, specific areas can be flexibly set based on actual needs.

[0056] In order to allow the user to intuitively view a specific type of area during the scanning process, in some embodiments, after obtaining the real scene image captured by the AR device, the real scene image can also be recognized. For example, a pre-trained image recognition model can be used to recognize the real scene image to determine whether there is a specific type of area in the real scene image. If so, the specific type of area is marked, for example, Figure 6 As shown, the area can be framed or rendered in a specific color, etc. The specific identification method can be flexibly set based on actual needs.

[0057] Considering that while the user is scanning the target object in real time, there may be other users who also want to know the real-time scanning status of the target object. For example, taking oral scanning as an example, when the doctor is scanning the patient's teeth with an oral scanning device, the patient may also want to know the condition of his or her oral cavity at the same time. Therefore, in some embodiments, the AR device may be a first AR device, and after generating the augmented reality image, the first AR device may send the augmented reality image to the second AR device so that the second AR device also displays the augmented reality image. For example, the first AR device may be AR glasses worn by a doctor, and the doctor can perform a real-time scan of the patient's oral cavity based on the augmented reality image displayed by the AR glasses. At the same time, the patient can also wear AR glasses, that is, the second AR device, and the patient's AR glasses can also obtain the augmented reality image and display it to the patient. Thus, during the scanning process, both the doctor and the patient can see the condition of the oral cavity and the current scanning condition.

[0058] In some embodiments, considering that during the scanning of the target object, the user wearing the first AR device and the user wearing the second AR device may need to communicate and discuss a specific part of the target object. In order to facilitate one of the users to know which area of ​​the target object the other user is currently introducing, a user instruction input by the user through the first AR device or the second AR device can be received. For example, the user can use the control component of the AR device, or the user can input the user instruction through voice or gesture. After receiving the user instruction, the intention of the user instruction can be analyzed, the target area indicated by the user instruction can be determined, and the target area can be identified from the augmented reality image, and the target area can be marked, for example, the target area can be framed to display the augmented reality image after marking in the first AR device and the second AR device.

[0059] In some embodiments, the target object may be an oral cavity, and the target area may be a tooth. In order to accurately identify the tooth that the doctor or patient wants to discuss, when identifying the target area indicated by the user instruction from the augmented reality image and identifying the target area, the augmented reality image may be first semantically segmented to obtain the various teeth in the patient's mouth, and then the tooth indicated by the user instruction may be identified from the segmented teeth, and the tooth may be identified.

[0060] For example, Figure 7As shown, during the scanning process, the doctor may need to introduce the patient's oral condition, such as telling the patient that a certain tooth is caries and needs treatment, or that two teeth do not bite well and need correction, etc. In order to let the patient know which tooth or area in the mouth the doctor is currently introducing, the doctor can enter a user instruction through the AR glasses he wears before introducing it to the patient. The user instruction can indicate the target area that the doctor wants to introduce. For example, the doctor can input instructions through voice or gestures and point to the target area he wants to introduce. For example, the doctor can voice input "the Nth tooth from left to right in the upper row". At this time, the AR glasses can recognize the voice instruction, identify the tooth indicated by the voice instruction from the augmented reality image, and mark the tooth, and then display the augmented reality image after marking in the AR glasses worn by the doctor and the AR glasses worn by the patient. As a result, both the doctor and the patient can see the marked teeth through the augmented reality image displayed by the AR glasses they wear, and know which tooth the doctor is currently introducing. The same principle applies to patients. When a patient wants to consult a doctor about a tooth, he or she can also input user instructions for the tooth through the AR glasses they wear. The AR glasses can recognize the tooth from the augmented reality image and mark it. The marked augmented reality image is then displayed in the doctor's and patient's AR glasses, so that the doctor also knows which tooth the patient is currently consulting about.

[0061] When the three-dimensional model is projected onto the real-life image to determine the current scanned area, accurately determining the matching relationship between the three-dimensional model and the target object in the real-life image is a prerequisite for obtaining an augmented reality image with better effects. In some embodiments, if the target object is an oral cavity, in order to obtain a more accurate matching relationship, the matching relationship can be constrained or corrected with the help of the correspondence between the teeth in the three-dimensional model and the teeth in the real-life image to obtain a more accurate matching relationship. For example, the three-dimensional model can be semantically segmented to obtain multiple tooth models, and the oral cavity area in the real-life image can be semantically segmented to obtain multiple tooth areas. Among them, when the three-dimensional model is semantically segmented, the gum area and each tooth in the three-dimensional model of the oral cavity can be identified to obtain a gum model and multiple tooth models. Similarly, when the real-life image is semantically segmented, the gum area and each tooth in the image can also be identified to obtain a gum area and multiple tooth areas.

[0062] Since the multiple tooth models correspond to the teeth in the real mouth, and the tooth regions also correspond to the teeth in the real mouth, the correspondence between the multiple tooth models and the multiple tooth regions can be determined based on the tooth numbers, and the two can be matched one by one. Then, the correspondence can be used as a constraint to determine the matching relationship between the three-dimensional model and the target object in the real scene image. Since the corresponding relationship between teeth is added as a constraint, the matching relationship finally determined is also more accurate.

[0063] For example, after obtaining a tooth model and a tooth region corresponding to a tooth in the real world, an easily extractable feature line (e.g., upper edge line) or an easily identifiable feature point in the tooth, such as a center point or the highest point of a protrusion, can be determined from the tooth model and the tooth region, respectively. The corresponding relationship between these feature lines or feature points in the tooth model and the tooth region is established as a constraint to determine the matching relationship between the three-dimensional model and the target object in the real scene image.

[0064] In some embodiments, in order to determine the matching feature line or feature point from the tooth model and the tooth region as accurately as possible, the feature line may be the boundary line between the tooth and the gum. Since each tooth and the gum region can be identified during the semantic segmentation process, and the boundary line between the tooth and the gum is relatively clear and easy to extract, the tooth model and the tooth region corresponding to the same tooth can be determined from the three-dimensional model and the current frame real scene image, respectively, and then the boundary line between the tooth and the gum can be extracted from the two, respectively, and the corresponding relationship of the above boundary line is used as a constraint to determine the matching relationship.

[0065] For example, when using the correspondence between teeth as a constraint to determine the matching relationship, the tooth model and tooth area corresponding to the same tooth can be determined based on the correspondence between multiple tooth models and tooth areas, and then the boundary between the tooth and the gums can be extracted from the tooth model, hereinafter referred to as the first boundary line. At the same time, the boundary between the tooth and the gums can be extracted from the tooth area, hereinafter referred to as the second boundary line, wherein the first boundary line and the second boundary line correspond to the boundary between the tooth and the gums in the same real tooth, and then the first boundary line can be projected into the real scene image to obtain a projection line, and a reprojection error between the projection line and the second boundary line is constructed, and then the matching relationship is solved with the minimization of the reprojection error as a constraint condition.

[0066] Since the boundary between teeth and gums is relatively clear and easy to extract, the tooth model and tooth area corresponding to the same tooth can be determined from the three-dimensional model and the real-life image respectively, and then the boundary between teeth and gums can be extracted from the two respectively. The above boundary line is used to determine the reprojection error, and then the reprojection error is used as a constraint to solve the matching relationship, so that the final matching relationship is more accurate.

[0067] For example, suppose a 3D scanning device is at t i The three-dimensional model of the tooth scanned at each moment is in the coordinate system of the three-dimensional scanning device: AR devices are i The real scene image collected at each moment is The real-life image can be used to identify and segment the tooth position number, and the tooth region set corresponding to each tooth can be obtained. Where k represents the tooth position number.

[0068] Similarly, the tooth 3D model can also be segmented and identified to obtain a 3D mesh subset belonging to each tooth. in represents the mesh subset belonging to tooth position k in the tooth model (the three-dimensional model of each tooth). The boundary line of the tooth is the boundary between the tooth and the gum, recorded as It is also a part with special semantics.

[0069] Through the AR device camera internal and external parameter matrix K, T i , the 3D point P in the 3D tooth model can be projected onto the real-scene image plane captured by the camera to obtain the image coordinates of the projection point: u = KT i P. The internal parameter matrix K is only related to the optical system of AR glasses and is constant during the shooting process. The external parameter matrix T i is the term to be solved, which is used to describe the relative position relationship between the 3D scanning device and the camera, and accurately solve the external parameter matrix T i It's the key, KT i This is the aforementioned matching relationship.

[0070] Solving the external parameter matrix requires finding the correspondence between the teeth in the 3D model and the teeth in the 2D image, which can be done with the help of the tooth boundary line. Find the correspondence between the tooth boundary line in the 3D model and the 2D image, and determine the reprojection error between the two. Then, the external parameter matrix can be solved with minimizing the reprojection error as a constraint, as shown in formula (1):

[0071]

[0072] in, is the pixel coordinate of the jth tooth-gingival boundary line in the image at time ti, is the pixel coordinate of the jth tooth-gingival boundary line in the image at time ti-1. is the three-dimensional coordinate of the jth tooth-gingival boundary line in the three-dimensional model at time ti (i.e., the tooth-tooth boundary line of a certain tooth model), is the 3D coordinate of the jth tooth and gum boundary line in the 3D model at time ti-1, K is the camera intrinsic parameter matrix, Ti is the external parameter matrix of the camera at time ti, T i -1 is the external parameter matrix of the camera at time ti-1, ω 1 and ω 2 is the preset coefficient.

[0073] When determining the external parameter matrix, by introducing the reprojection error of the tooth-gingival junction line as a constraint condition, the final matching relationship can be made more accurate.

[0074] In some embodiments, the three-dimensional scanning device may include two scanning modes, namely, a high-precision scanning mode and a low-precision scanning mode. In the high-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is small, that is, the measurement error is small, while in the low-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is large, that is, the measurement error is large. Considering that certain areas of the target object may be areas that the user needs to focus on, these areas require more refined measurements and have higher requirements for errors. For example, when scanning a pipeline, for crack areas in the pipeline, since the size of the cracks must be accurately determined in order to take corresponding treatment measures based on the size of the cracks, these areas need to be scanned more finely. Therefore, after the AR device acquires the real-life image, it can determine whether the area currently scanned by the 3D scanning device is a specific type of area based on the real-life image. For example, based on the matching relationship between the reconstructed 3D model and the real-life image, it can determine which area of ​​the real-life image the current 3D model corresponds to. This area is the area currently being scanned. The trained model can then be used to identify the area to determine whether it is a specific type of area. If so, the 3D scanning device can be notified to switch the scanning mode to a high-precision scanning mode to enable a more refined scan of the area.

[0075] In some embodiments, the method can be performed by an AR device. Considering that the process of determining the matching relationship between the three-dimensional model and the target object in the real scene image requires a large amount of calculation, some AR devices have weak processing performance, which may affect the real-time display of the augmented reality image.

[0076] The AR device can detect whether there is a target device in the surrounding environment that can provide computing services. The target device can be a device with higher processing performance. After detecting the target device, the three-dimensional model and the real scene image can be sent to the target device, and the matching relationship between the two can be determined by the target device. Based on the matching relationship, the three-dimensional model is superimposed on the real scene image to obtain an augmented reality image, and then the augmented reality image is received from the target device and displayed. By using a device with better processing performance to determine the matching relationship between the three-dimensional model and the real scene image, the processing efficiency can be improved to ensure that the augmented reality image is displayed in real time in the AR device.

[0077] In some scenarios, for certain target objects, it may be necessary to scan them multiple times to determine their changing trends over time. For example, for the patient's mouth, it can be scanned at regular intervals to determine the changing trends of its teeth, or for certain pipes, it can be scanned at regular intervals to determine the changes in its corrosion position over time, or the changes in cracks over time, etc. Therefore, in some embodiments, after the three-dimensional model of the target object is reconstructed in real time based on the current scanned data, a historical three-dimensional model reconstructed based on the historical scan data of the target object can also be obtained, and then the historical three-dimensional model can be compared with the currently reconstructed three-dimensional model to determine whether there are some risk areas, such as areas where pipe cracks increase sharply, and then these risk areas can be marked in the augmented reality image so that they can be displayed to the user through the AR device, and at the same time, the user can also be warned.

[0078] In the process of superimposing the three-dimensional model of the target object onto the real-life image video stream of the target object, the final generated augmented reality image usually has a jitter problem due to matching errors. For example, taking the target object as teeth as an example, for the current frame of the real-life image, there may be errors in the determined matching relationship, resulting in the teeth in the three-dimensional model and the teeth in the current frame of the real-life image not being completely aligned, so that the three-dimensional model is projected onto the current frame of the real-life image, and the teeth in the projection area and the real-life image cannot be aligned, for example, the teeth in the three-dimensional model are offset to the left by a certain distance compared to the teeth in the real-life image. For the next frame of the real-life image, there may be errors in the determined matching relationship, resulting in the teeth in the three-dimensional model being offset to the right by a certain distance compared to the teeth in the real-life image. Therefore, when the user sees the final generated two frames of augmented reality images, the position of the three-dimensional model may suddenly jump from the left position of the image to the right position, that is, there is a jump problem. From the overall display effect, the video stream of the augmented reality image seen by the user has been jittering, which seriously affects the display effect of the augmented reality image video stream.

[0079] Based on this, in some embodiments, in the process of superimposing the three-dimensional model onto the real-time video stream captured by the AR device, when determining the matching relationship between the three-dimensional model and the current frame real-scene image in the video stream, the matching relationship between the three-dimensional model and the one or more frames of real-scene images before the current frame real-scene image can be determined together. That is, the matching relationship between the three-dimensional model and the previous one or more frames of real-scene images is used to constrain and optimize the matching relationship between the three-dimensional model and the current frame real-scene image in the video stream, which is equivalent to smoothing the matching relationship between the three-dimensional model and different frames of real-scene images, thereby eliminating the jitter and jump problems in the video stream of the augmented reality image finally generated, and improving the display effect of the video stream of the augmented reality image.

[0080] For example, after obtaining the currently reconstructed three-dimensional model and the current frame real-scene image from the video stream captured by the AR device, the initial matching relationship between the three-dimensional model and the current frame real-scene image can be determined, and the initial matching relationship can be optimized based on the matching relationship between the three-dimensional model and the reference real-scene image to obtain a target matching relationship; wherein the reference real-scene image is one or more frames of images before the current frame real-scene image in the video stream; and then the three-dimensional model can be projected into the current frame real-scene image based on the target matching relationship, the scanned area can be determined, and the identification process can be performed to obtain an augmented reality image.

[0081] In some embodiments, when the initial matching relationship is optimized based on the matching relationship between the three-dimensional model and the reference real-scene image to obtain the target matching relationship, a constraint item can be first determined based on the deviation between the matching relationship between the three-dimensional model and the reference real-scene image and the initial matching relationship, and then the constraint item is used to optimize the initial matching relationship to obtain the target matching relationship.

[0082] In some embodiments, when the initial matching relationship is optimized based on the matching relationship between the three-dimensional model and the reference image to obtain the target matching relationship, a first matching error can be constructed based on the matching relationship between the three-dimensional model and the reference image, and a second matching error can be constructed based on the target matching relationship, and then the target matching relationship is solved with the minimization of the first matching error and the second matching error as constraints. That is, the target matching relationship finally calculated will minimize the matching error between the three-dimensional model and the reference real scene image, and also minimize the matching error between the three-dimensional model and the real scene image of the current frame.

[0083] It is not difficult to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space reasons, they are not listed one by one in the embodiments of this disclosure.

[0084] Accordingly, the present disclosure also provides a three-dimensional scanning device, such as Figure 8As shown, the device includes: a three-dimensional model acquisition module 81, which is used to acquire a three-dimensional model of the target object in real time during the process of scanning the target object using a three-dimensional scanning device, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device;

[0085] A real scene image acquisition module 82, used to acquire a real scene image of a target object collected by an AR device in real time;

[0086] A projection module 83, configured to determine a matching relationship between the three-dimensional model and a target object in the real scene image, and project the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image;

[0087] The identification module 84 identifies the scanned area in the real scene image to obtain an augmented reality image, so as to display the augmented reality image in real time through the AR device.

[0088] Among them, the specific steps of the above-mentioned device to perform the three-dimensional scanning method can refer to the description in the above-mentioned method embodiment, and will not be repeated here.

[0089] Furthermore, the present disclosure also provides a device, such as Fig. 9 As shown, the device includes a processor 91, a memory 92, and computer instructions stored in the memory 92 for execution by the processor 91. When the processor 91 executes the computer instructions, the method described in any one of the above embodiments is implemented.

[0090] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0091] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0092] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure or some parts of the embodiments.

[0093] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0094] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present disclosure. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0095] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present disclosure.

Claims

1. A three-dimensional scanning method, characterized in that: The method comprises: In the process of scanning the target object by using the three-dimensional scanning device, a three-dimensional model of the target object is obtained in real time, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device; Acquire the real-life image of the target object captured by the AR device in real time; Determining a matching relationship between the three-dimensional model and a target object in the real scene image, and projecting the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image; The scanned area in the real scene image is marked to obtain an augmented reality image, so as to display the augmented reality image in real time through the AR device.

2. The method according to claim 1, characterized in that Before displaying the augmented reality image in real time through the AR device, the method further includes: Acquire scanning trajectory data matching the target object; A scanning trajectory line is drawn in the augmented reality image based on the scanning trajectory data, wherein the scanning trajectory line is used to guide the user to use the three-dimensional scanning device to perform a three-dimensional scan on the target object.

3. The method according to claim 1, characterized in that The method further comprises: dividing the scanned area into a plurality of sub-areas; The scanning quality of each sub-region in the scanned region is determined, and the scanning quality of each sub-region is identified in the augmented reality image.

4. The method according to claim 1, characterized in that: Before displaying the augmented reality image in real time through the AR device, the method further includes: A specific type of region is identified from the augmented reality image, and identification processing is performed on the specific type of region.

5. The method according to claim 4, characterized in that The target object is the oral cavity, and the specific type of area includes an oral disease area; The target object is a pipeline, and the specific type of area includes an area where cracks or corrosion occur.

6. The method according to claim 1, characterized in that The AR device is a first AR device, and the method further includes: The augmented reality image is sent to a second AR device so that the second AR device displays the augmented reality image.

7. The method according to claim 6, characterized in that The method further comprises: receiving a user instruction input by a user through the first AR device or the second AR device; A target area indicated by the user instruction is identified from the augmented reality image, and a marking process is performed on the target area, so as to display the augmented reality image after the marking process in the first AR device and the second AR device.

8. The method according to claim 7, characterized in that The target object is an oral cavity, the target area is a tooth, and the target area indicated by the user instruction is identified from the augmented reality image, and the target area is marked, including: Performing semantic segmentation processing on the augmented reality image to obtain individual teeth in the oral cavity; The tooth indicated by the user instruction is identified from the teeth obtained by segmentation, and the tooth is marked.

9. The method according to claim 1, characterized in that: The target object is an oral cavity, and determining a matching relationship between the three-dimensional model and the target object in the real scene image includes: Performing semantic segmentation processing on the three-dimensional model to obtain multiple tooth models; Performing semantic segmentation processing on the target object in the real scene image to obtain multiple tooth regions; Determining a correspondence between the plurality of tooth models and the plurality of tooth regions based on the tooth numbers; A matching relationship between the three-dimensional model and the target object in the real scene image is determined based on the corresponding relationship.

10. The method according to claim 9, characterized in that Determining a matching relationship between the three-dimensional model and a target object in the real scene image based on the corresponding relationship includes: Determining a tooth model and a tooth region corresponding to a same tooth from among the plurality of tooth models and the plurality of tooth regions based on the corresponding relationship; Extracting a first boundary line between teeth and gums from the tooth model, and extracting a second boundary line between teeth and gums from the tooth region; Projecting the first dividing line into the real scene image to obtain a projection line, and constructing a reprojection error between the projection line and the second dividing line; The matching relationship between the three-dimensional model and the target object in the real scene image is determined at least with the minimization of the reprojection error as a constraint condition.

11. The method according to claim 1, characterized in that: The scanning mode of the three-dimensional scanning device includes a high-precision scanning mode and a low-precision scanning mode, and the method further includes: In the case where it is determined based on the real scene image that the area currently scanned by the three-dimensional scanning device is an area of ​​a specific type, the three-dimensional scanning device is notified to switch the scanning mode to a high-precision scanning mode.

12. The method according to claim 1, characterized in that The method is performed by the AR device, and the method further includes: Detect whether there is a target device in the surrounding environment that can provide computing services; After detecting the target device, the three-dimensional model and the real scene image are sent to the target device so that the target device determines the matching relationship, and based on the matching relationship, the three-dimensional model is projected into the real scene image to determine the scanned area in the real scene image, and the scanned area in the real scene image is marked to obtain an augmented reality image; The augmented reality image is received from the target device and displayed.

13. The method according to claim 1, characterized in that Before displaying the augmented reality image in real time through the AR device, the method further includes: Acquire a historical three-dimensional model of the target object, wherein the historical three-dimensional model is reconstructed based on historical scanning data of the target object; Comparing the historical three-dimensional model with the three-dimensional model reconstructed in real time to determine the risk area in the target object; The risk area is identified in the real scene image.

14. A three-dimensional scanning device, characterized in that: The device comprises: A three-dimensional model acquisition module, used to acquire a three-dimensional model of a target object in real time during the process of scanning the target object using a three-dimensional scanning device, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device; A real-scene image acquisition module is used to acquire the real-scene image of the target object collected by the AR device in real time; A projection module, used to determine a matching relationship between the three-dimensional model and a target object in the real scene image, and project the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image; The identification module identifies the scanned area in the real scene image to obtain an augmented reality image, so as to display the augmented reality image in real time through the AR device.

15. An electronic device, characterized in that: The electronic device includes a processor, a memory, and computer instructions stored in the memory and executable by the processor. When the processor executes the computer instructions, the method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images