Three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of three-dimensional reconstruction, and provides a three-dimensional reconstruction method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first image frame set of an oral cavity in a first state, and the oral cavity in the first state represents that at least one actual scanning rod is installed in the oral cavity; identifying a scanning rod area of each image frame in the first image frame set; and based on the feature information in the scanning rod region, splicing the plurality of image frames in the first image frame set, and reconstructing to obtain target three-dimensional data corresponding to an actual scanning rod in the oral cavity. According to the method, splicing is carried out based on the rigid structure, the influence of anatomical structures (such as gingiva, lip and tongue) in the image frame on the splicing precision can be reduced, and the overall precision of three-dimensional reconstruction can be improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, device, electronic device and storage medium. Background Art

[0002] During the oral scanning process, multiple image frames need to be spliced ​​and aligned through a common area (or transition area) to reconstruct a complete three-dimensional model. In related technologies, in scenarios such as edentulous jaw implants and large-span implants, soft tissue areas such as gums are usually selected as transition areas, and data from different areas or different image frames are spliced. However, due to the lack of rigid support of soft tissues such as gums, they are easily deformed by external forces (such as opening movements, instrument contact) or muscle traction (such as grinning), and the patient's physiological activities (such as breathing, saliva secretion, etc.) are also prone to cause slight displacement of soft tissues, resulting in changes in the morphology of the soft tissue area. In this case, if the soft tissue area is selected as the transition area, the coordinates of the transition area in the scan data will be offset, affecting the splicing accuracy of adjacent areas or adjacent frame data, thereby reducing the accuracy of three-dimensional reconstruction. The final restoration denture is based on the deformed gum data for computer-aided design (Computer Aided Design, CAD), which will cause the restoration to compress the gums or create gaps when it is in place in the mouth. If the deformation exceeds a certain limit, the patient will feel uncomfortable in mild cases, and the implant may fail in severe cases. Summary of the invention

[0003] In view of this, the present application provides a three-dimensional reconstruction method, device, electronic device and storage medium to solve the problem of large stitching errors, which in turn affects the accuracy of three-dimensional reconstruction.

[0004] A first aspect of an embodiment of the present application provides a three-dimensional reconstruction method, which is applied to an electronic device, and the three-dimensional reconstruction method includes: acquiring a first image frame set of an oral cavity in a first state, wherein the oral cavity in the first state indicates that at least one actual scanning rod is installed in the oral cavity; identifying a scanning rod area of ​​each image frame in the first image frame set; based on feature information in the scanning rod area, splicing multiple image frames in the first image frame set, and reconstructing target three-dimensional data corresponding to the actual scanning rod in the oral cavity.

[0005] In some embodiments, before the reconstruction obtains the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, the method also includes: identifying the soft tissue area of ​​each image frame in the first image frame set; the soft tissue area does not participate in the three-dimensional reconstruction of the target three-dimensional data or participates in the three-dimensional reconstruction of the target three-dimensional data according to a preset weight.

[0006] In some embodiments, the method further includes: based on identifying characteristic information of the soft tissue area in each image frame in the first image frame set and / or characteristic information of the scanning rod area, performing one or more of a removal operation, a marking operation, a hiding operation or a segmentation operation on the soft tissue area in each image frame or the scanning rod area.

[0007] In some embodiments, the first image frame set includes texture image frames and depth image frames, and identifying the scanning rod area of ​​each image frame in the first image frame set includes: using a preset recognition model to perform feature extraction and feature fusion on the texture image frame and the depth image frame to obtain a mask map, and the mask map includes the scanning rod area.

[0008] In some embodiments, based on the feature information in the scanning rod area, multiple image frames in the first image frame set are stitched together, and target three-dimensional data corresponding to the actual scanning rod in the oral cavity is reconstructed, including: based on the feature information in the scanning rod area, multiple image frames in the first image frame set are stitched together in real time; based on the real-time stitched image frames, real-time three-dimensional reconstruction is performed to obtain target three-dimensional data corresponding to the actual scanning rod.

[0009] In some embodiments, based on the feature information in the scanning rod area, multiple image frames in the first image frame set are stitched together, and target three-dimensional data corresponding to the actual scanning rod in the oral cavity is reconstructed, including: based on the feature information of the scanning rod area, multiple image frames in the first image frame set are stitched together in real time; based on the stitched image frames, real-time three-dimensional reconstruction is performed to obtain actual scanning data corresponding to the actual scanning rod in real time; based on the feature information of the scanning rod area, standard data of the actual scanning rod is matched from a standard scanning rod database, and the matching is a regular matching during the real-time scanning process. During the real-time scanning process, each time a certain number of actual scanning data is obtained or each time interval, the current actual scanning data is feature-stitched with the standard data, and as the scanning data increases, the position and posture of the standard data is updated; and the scanning data of the actual scanning rod is replaced with the matched standard data of the actual scanning rod to obtain the target three-dimensional data.

[0010] In some embodiments, the method further includes: acquiring a second image frame set of the oral cavity in a second state, wherein the oral cavity in the second state indicates that no actual scanning rod is installed in the oral cavity; based on the second image frame set, acquiring regional three-dimensional data of the soft tissue area in each image frame in the second image frame set; acquiring a third image frame set, wherein the third image frame set includes partial image frames of the first image frame set or updated image frames acquired when the oral cavity is in the first state; based on the feature information of the soft tissue area in each image frame in the third image frame set and the feature information of the scanning rod area, merging the regional three-dimensional data corresponding to the soft tissue area with the target three-dimensional data to obtain an oral model.

[0011] In some embodiments, the method further includes: based on feature information in the scanning rod area, displaying the target three-dimensional data in real time on the interactive interface of the electronic device when acquiring a first set of image frames of the oral cavity in the first state; or after acquiring a second set of image frames of the oral cavity in the second state, displaying the oral model on the interactive interface in response to a user's switching instruction.

[0012] In some embodiments, the stitching of multiple image frames in the first image frame set based on the feature information in the scanning rod area includes: judging whether there is overlapping feature data between adjacent image frames in the multiple image frames based on the feature information in the scanning rod area; if there is overlapping feature data between adjacent image frames in the multiple image frames, stitching the adjacent image frames in real time based on the overlapping feature data; if there is no overlapping feature data between adjacent image frames in the multiple image frames, issuing a prompt message to prompt the user that the stitching fails, and / or issuing a prompt message to prompt the user to return to the initial position for scanning.

[0013] In some embodiments, the method further includes: generating a design model based on the target three-dimensional data and sending the design model to a 3D printing device; receiving a user request for modifying the design model in real time; and outputting a revised three-dimensional design model based on the user request and the target three-dimensional model.

[0014] A second aspect of an embodiment of the present application provides a three-dimensional reconstruction device, which is applied to an electronic device, and the three-dimensional reconstruction device includes: an acquisition module, which is used to acquire a first image frame set of an oral cavity in a first state, wherein the oral cavity in the first state indicates that at least one actual scanning rod is installed in the oral cavity; an identification module, which is used to identify the scanning rod area of ​​each image frame in the first image frame set; and a reconstruction module, which is used to splice multiple image frames in the first image frame set based on feature information in the scanning rod area, and reconstruct target three-dimensional data corresponding to the actual scanning rod in the oral cavity.

[0015] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned three-dimensional reconstruction method when executing the computer-readable instructions.

[0016] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the above-mentioned three-dimensional reconstruction method is implemented.

[0017] The embodiment of the present application provides a three-dimensional reconstruction method. When an actual scanning rod is installed in the oral cavity, a first image frame set of the oral cavity is obtained, and the scanning rod area in each image frame in the first image frame set is identified. Based on the feature information of the scanning rod area, multiple image frames in the first image frame set are spliced ​​to reconstruct the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, thereby realizing image frame reconstruction based on a rigid structure. Since the rigid structure of the scanning rod is not easily deformed during the scanning process and has good stability, splicing multiple image frames in the first image frame set based on the feature information of the scanning rod area helps to improve the splicing accuracy, thereby improving the overall accuracy of the three-dimensional reconstruction, and subsequent restoration design based on the target three-dimensional data can improve the success rate of implantation, improve the matching degree between the restoration product and the patient, and improve the comfort of the patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a device diagram of the three-dimensional reconstruction method provided in an embodiment of the present application.

[0020] Figure 2 It is a flow chart for implementing the three-dimensional reconstruction method provided in an embodiment of the present application.

[0021] Figure 3 This is an example diagram of the interactive interface provided in the first embodiment of the present application.

[0022] Figure 4 This is an example diagram of the interactive interface provided in the second embodiment of the present application.

[0023] Figure 5 This is an example diagram of the scanning rod area in the image frame provided by the first embodiment of the present application.

[0024] Figure 6 This is an example diagram of the scanning rod area in the image frame provided by the second embodiment of the present application.

[0025] Figure 7 This is an example diagram of the scanning rod area in the image frame provided in the third embodiment of the present application.

[0026] Figure 8 This is an example diagram of target three-dimensional data provided by an embodiment of the present application.

[0027] Fig. 9 This is an example diagram of the oral model provided in the embodiment of the present application.

[0028] Fig.10 It is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field in this application. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can represent A or B. "And / or" in this application is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c seven situations.

[0032] See also Figure 1 , is a diagram of a device for a three-dimensional reconstruction method provided in one embodiment of the present application. Figure 1 As shown, the three-dimensional reconstruction method provided in the embodiment of the present application can be applied to an electronic device 100, and the electronic device 100 can include one or more devices such as a mobile phone, a tablet computer, a smart wearable device, a laptop computer, a scanning device (such as an oral digital impression device), etc. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0033] like Figure 1 As shown, the electronic device 100 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the I / O interface 104 through the bus 105.

[0034] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), bluetooth (blue tooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0035] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0036] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 103. The non-volatile memory may include a disk storage device and a flash memory.

[0037] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the three-dimensional reconstruction method executed on the electronic device 100 can be implemented.

[0038] In other embodiments, the electronic device 100 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 100 .

[0039] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0040] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned three-dimensional reconstruction method.

[0041] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information or visualize information. In addition, the I / O interface 104 can also be used to connect various sensor devices, such as radar sensors, image sensors, etc., so as to obtain the required radar point cloud data, image data and other data.

[0042] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 100 .

[0043] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0044] Figure 1 The scenes shown are only illustrative examples, and the three-dimensional reconstruction method provided by the present application can also be applied in other scenes. For example, in some scenes, an electronic device 100 and a scanning device may be included, and the electronic device 100 and the scanning device are separately arranged, and the electronic device is a computer, a tablet computer, a mobile phone, etc.; in some scenes, the electronic device 100 can be arranged in the scanning device. In some scenes, other types of devices may also be included. In summary, the embodiments of the present application do not limit the specific application scenarios of the three-dimensional reconstruction method.

[0045] See also Figure 2 , which is a flowchart of the implementation of the 3D reconstruction method provided in the embodiment of the present application. The method is applied to electronic devices, and the embodiment of the present application is a computer program product of the method running on (for example Figure 1 The electronic device 100 in FIG. 1 is used as an example for description. The method includes the following steps.

[0046] S11: Acquire a first image frame set of the oral cavity in a first state.

[0047] In some embodiments, the oral cavity in the first state indicates that at least one actual scanning rod is installed in the oral cavity. The actual scanning rod can be installed on an implant or a base in the oral cavity. The oral cavity can be the oral cavity of a patient with edentulous jaws, such as the oral cavity of a fully edentulous jaw patient or the oral cavity of a half-edentulous jaw patient. Therefore, the user can use a scanning device such as an oral scanner, an extraoral scanner, etc. to scan the oral cavity in the first state to obtain a first set of image frames.

[0048] In some embodiments, the actual scanning rod surface is provided with specific geometric patterns and / or marks, etc. These geometric patterns or marks can be accurately captured by the scanning device during scanning, and are used to provide reference points to help the scanning device identify and locate the scanning rod at multiple angles or positions. For example, the geometric pattern can be a high-contrast pattern, a regular geometric shape (such as a circle, a square), an irregular geometric shape (such as a groove, a protrusion), and the mark can be a non-coded mark or a coded mark, such as a non-coded mark point or a coded mark point. The specific geometric patterns and / or marks can be evenly distributed or unevenly distributed on the actual scanning rod.

[0049] In some embodiments, the first image frame set includes multiple image frames. An image frame represents a single two-dimensional image captured by a scanning device at a specified time point during the process of scanning the oral cavity. When scanning the oral cavity in the first state, each image frame in the first image frame set includes an intraoral scanning rod and surrounding tissues (e.g., soft tissues such as teeth, gums, and mucosa).

[0050] S12: Identify the scanning rod area of ​​each image frame in the first image frame set.

[0051] In some embodiments, the scanbar region represents a region in the image frame that includes local information containing an actual scanbar.

[0052] In some embodiments of the present application, the first image frame set includes texture image frames and depth image frames, and identifying the scanning rod area of ​​each image frame in the first image frame set includes: using a preset recognition model to extract and fuse features of the texture image frames and the depth image frames to obtain a mask map, and the mask map includes the scanning rod area.

[0053] In some embodiments, the texture image frame represents a two-dimensional color image of the interior of the oral cavity, which is used to reflect the surface visual features of the teeth, gums, scanning rods and other tissues, including details such as color, texture, spots, cracks, restorations (such as fillings or crowns), surface patterns, uncoded landmarks or coded landmarks, etc. The depth image frame represents the two-dimensional spatial information of the oral structure. Each pixel value in the depth image frame can represent the distance (depth value) from the corresponding position to the scanning device, including the geometric shape, grooves and protrusions on the scanning rod, etc., which are usually represented in grayscale or pseudo-color (for example, bright near and dark far).

[0054] In some embodiments, the electronic device may use optical three-dimensional scanning technology (such as structured light technology, laser triangulation technology, binocular stereo vision technology, etc.) to obtain depth image frames. For example, a texture image frame may be obtained by a scanning device projecting white light on the scanned object through a color camera, and a depth image frame may be obtained by a scanning device projecting structured light on the scanned object through a black and white camera. The specific method of obtaining the depth image frame is not limited in the embodiments of the present application.

[0055] In some embodiments, the preset recognition model may be a deep learning model, such as a convolutional network model, a recurrent network model, a generative adversarial network model, a semantic segmentation model, etc.

[0056] In some embodiments, the preset recognition model includes a multi-layer convolutional network and a multi-layer deconvolutional layer. The electronic device uses the preset recognition model to extract features and fuse information on the current frame image and the depth image, and outputs a mask map, including: using a multi-layer convolutional network to extract features and fuse features on the current frame image and the depth image to obtain a feature map; using a multi-layer deconvolutional layer to perform upsampling and recovery processing on the feature map to obtain a mask map.

[0057] In some embodiments, the electronic device uses multiple layers of deconvolution layers to perform upsampling and recovery processing on the feature map to obtain a mask map, including performing a transposed convolution operation on the feature map, upsampling, splicing the shallow features of the corresponding layer of the encoder with the upsampled deep features by channel, and merging details and semantic information. Repeat the above steps, and when the resolution of the feature map is restored to the input size, use the activation function to predict the category probability of each pixel in the feature map to obtain a mask map.

[0058] S13: Based on the feature information in the scanning rod area, multiple image frames in the first image frame set are spliced, and target three-dimensional data corresponding to the actual scanning rod in the oral cavity is reconstructed.

[0059] In some embodiments, the feature information in the scanning rod area includes but is not limited to geometric features (such as point cloud data, edges and contours, preset key points, etc.), and information of preset marks (such as geometric patterns, marks) on the actual scanning rod.

[0060] In some embodiments, the target three-dimensional data may include three-dimensional point cloud data and a three-dimensional model corresponding to the actual scanning rod. The target three-dimensional data includes the shape information of the scanning rod and the positioning information of the scanning rod. Therefore, the present application, based on the characteristic information of the scanning rod area, stitching multiple image frames in the first image frame set helps to improve the stitching accuracy, thereby improving the overall accuracy of the three-dimensional reconstruction, obtaining more accurate scanning rod positioning information, and making subsequent restoration designs (such as crown, bridge, denture, etc.) based on the target three-dimensional data, improving the implant success rate, improving the matching degree between the restoration product and the patient, and improving the patient's comfort.

[0061] In some embodiments of the present application, before reconstructing the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, the method also includes: identifying the soft tissue area of ​​each image frame in the first image frame set; the soft tissue area does not participate in the three-dimensional reconstruction or participates in the three-dimensional reconstruction of the target three-dimensional data according to a preset weight.

[0062] Among them, the soft tissue includes one or more of the soft tissue objects such as tongue, gums, lips, etc. in the oral region. The soft tissue region refers to the region where the soft tissue is located in the image frame. The preset weight can be a weight input by the user, or a preset weight can be calculated based on the user's edentulous jaw condition and age using a preset model (e.g., an AI model); and / or the preset weight can be determined by analyzing the quality of the image frames scanned by the scanning device (e.g., the image frames in the first image frame set or the second image frame set) using a preset model; and / or the preset weight can be determined based on the splicing error or splicing success rate of the target three-dimensional data obtained by real-time splicing using a preset model.

[0063] In some embodiments, the preset model may be a deep learning model, such as a convolutional network model, etc. The embodiments of the present application do not limit the preset model.

[0064] In some embodiments, the electronic device may pre-train a deep learning model based on feature data of soft tissue objects such as the tongue, gums, and lips, such as the shape, edge features, and color features of the gums, to identify the soft tissue area in each image frame in the first image frame set. In other embodiments, the electronic device may also identify the soft tissue area in each image frame in the first image frame set by other means. The embodiments of the present application are not limited thereto.

[0065] It should be noted that one embodiment may be to only identify the scanning rod area, remove all areas of the image frames in the first image frame set except the scanning rod, and reconstruct only the scanning rod area; one embodiment may be to semantically identify each part of the image frames in the first image frame set, distinguish whether it is soft tissue or scanning rod, and determine that the soft tissue area does not participate in the reconstruction. One embodiment may be to semantically identify each part of the image frames in the first image frame set, distinguish the soft tissue and scanning rod in the image frames, thereby determining the soft tissue area and participating in the reconstruction according to the preset weight to avoid splicing loss. One embodiment may be to further distinguish the soft tissue area, and the soft tissue can be further distinguished into tongue, gums, and lips, and the tongue and lips are removed as miscellaneous data, and the gum data can be retained for subsequent operations.

[0066] Specifically, semantic recognition is performed on each part of the image frame in the first image frame set to distinguish whether it is a tongue, gum, lip or scanning rod. For example, if it is a tongue and lip, it is deleted, if it is a scanning rod, it is given a weight of 80%, and if it is a gum, it is given a weight of 20%. When splicing multiple image frames, it is determined whether there is overlapping feature data between adjacent image frames in the multiple image frames; if there is overlapping feature data between adjacent image frames in the multiple image frames, the adjacent image frames are spliced ​​in real time based on the overlapping feature data. Among them, the electronic device can determine whether there is overlapping feature data between adjacent image frames in the multiple image frames based on the weight of the scanning rod (for example, 80%) and the weight of the gum (for example, 20%). In another example, the electronic device can set different weights for splicing according to different classifications of the semantic recognition results. Specifically, the electronic device can perform semantic recognition on each part of the image frame in the first image frame set to distinguish whether it is a tongue, gum, lip or scanning rod, and give weights to the scanned data obtained based on the semantic recognition results. For example, the initial value of the scan data in each image frame is set to 1. If the semantic recognition result is the tongue, the weight is reduced based on the initial value of the scan data to obtain the corresponding weight when the semantic recognition result is the tongue, for example, it can be reduced by 80%, so the corresponding weight of the tongue is 20%; if it is the lips, the weight is reduced based on the initial value of the scan data to obtain the corresponding weight when the semantic recognition result is the lips, for example, it is reduced by 80%, so the corresponding weight of the lips is 20%; if the semantic recognition result is the scanning rod, the weight is increased based on the initial value of the scan data to obtain the corresponding weight when the semantic recognition result is the scanning rod, for example, the weight is increased by 100%; if the semantic recognition result is the gums, the weight is increased based on the initial value of the scan data to obtain the corresponding weight when the semantic recognition result is the gums, for example, the weight is increased by 20%. The electronic device splices and reconstructs the scan data whose weights meet the preset threshold range according to all the scan data after multiple verifications and calculations to obtain the target three-dimensional data.

[0067] In some embodiments of the present application, the electronic device may perform one or more of a removal operation, a marking operation, a hiding operation, or a segmentation operation on the soft tissue area or the scanning rod area in each image frame based on identifying characteristic information of the soft tissue area and / or characteristic information of the scanning rod area in each image frame in the first image frame set.

[0068] It should be noted that the removal operation can be deletion, the marking operation can be marking with different colors, the segmentation operation can be segmentation by determining the boundary, and the hiding operation can be hiding the pre-selected area. The marking operation, hiding operation or segmentation operation does not affect the integrity of the scan data. If necessary, the entire scan data can be restored and displayed through settings.

[0069] In some embodiments, the feature information of the soft tissue area may include the soft tissue's geometric shape (e.g., the wavy shape of the gum margin, the tongue coating contour, etc.), color and texture features, three-dimensional structural features (e.g., topological structure, spatial deformation features, etc.), function and physiological state features, etc. In the case of installing a scanning rod in the oral cavity, the feature information of the soft tissue area may also include the interaction features between the soft tissue and the scanning rod, such as the contact area between the soft tissue and the scanning rod, pressure distribution, etc. The specific content of the feature information of the soft tissue area is not limited in the embodiments of the present application.

[0070] In the prior art, by combining scanning equipment (such as oral digital impression device / intraoral scanner, extraoral scanner) with scanning rods for three-dimensional reconstruction, the anatomical morphology of the patient's oral cavity (such as dentition, gums, occlusal relationship) can be restored for digital design of restorations or implants. However, when the scanning rod is installed in the patient's mouth, during the scanning process, the complex anatomical structure around the scanning rod (such as gums, lips and tongue and other soft tissues) is prone to data overlap or interference with the scanning rod area, resulting in mismatching of common areas when splicing image frames, thereby causing errors to accumulate frame by frame, and ultimately manifested as distortion of the three-dimensional model or even splicing fracture, which in turn affects the accuracy and clinical adaptability of subsequent restoration design.

[0071] For example Figure 3 As shown in the figure, the upper left part of the figure shows the window of the scanning device in the oral cavity, and the lower right part of the figure shows the reconstructed 3D model. Figure 3 As shown, the reconstructed 3D model contains large soft tissue areas (e.g. Figure 3 The soft tissue area is prone to deformation, which may cause data overlap or interference with the scanning rod area.

[0072] In order to solve the above problem, the electronic device can perform one or more of a removal operation, a marking operation, a hiding operation or a segmentation operation on the soft tissue area in each image frame in the first image frame set based on the feature information of the soft tissue area and / or the feature information of the scanning rod area, so that the electronic device can stitch multiple image frames based only on the feature information of the scanning rod area, thereby improving the stitching accuracy and further improving the accuracy of 3D reconstruction. For example Figure 4 As shown, the soft tissue area in each image frame is removed, and the image frames after the removal operation are spliced ​​and three-dimensionally reconstructed to obtain the target three-dimensional data (three-dimensional model) (for example Figure 4 Soft tissue areas are not shown in the lower right image.

[0073] In some embodiments of the present application, based on the feature information in the scanning rod area, multiple image frames in the first image frame set are spliced, and the target three-dimensional data corresponding to the actual scanning rod in the oral cavity is reconstructed, including: based on the feature information in the scanning rod area, multiple image frames in the first image frame set are spliced ​​in real time; based on the real-time spliced ​​image frames, real-time three-dimensional reconstruction is performed to obtain the target three-dimensional data corresponding to the actual scanning rod. Among them, when the first image frame set of the oral cavity in the first state is obtained, the target three-dimensional data is displayed in real time on the interactive interface of the electronic device. Therefore, in this embodiment, the recognition of the scanning rod area or soft tissue area of ​​each image frame in the first image frame set is also a real-time operation, so that real-time scanning and real-time stitching and reconstruction can be realized. During the scanning process, as the user moves in the oral cavity, the target three-dimensional data is gradually obtained. Figure 4 The target three-dimensional data shown in the lower right corner is obtained as follows Figure 8 The target three-dimensional data shown is also displayed in real time on the interactive interface, making it easy for users to understand the scanning progress.

[0074] In some embodiments of the present application, based on the feature information in the scanning rod area, multiple image frames in the first image frame set are spliced, and target three-dimensional data corresponding to the actual scanning rod in the oral cavity is reconstructed, including: based on the feature information in the scanning rod area, multiple image frames in the first image frame set are spliced ​​in real time; based on the spliced ​​image frames, real-time three-dimensional reconstruction is performed to obtain actual scanning data corresponding to the actual scanning rod in real time; based on the feature information of the scanning rod area, standard data of the actual scanning rod is matched from a standard scanning rod database; and the scanning data of the actual scanning rod is replaced with the matched standard data of the actual scanning rod to obtain the target three-dimensional data.

[0075] The standard scanning rod database stores standard data of various types of scanning rods, which represent a complete, non-missing, and error-free reference parameter set of the scanning rod in an ideal state, such as a designed scanning rod CAD model.

[0076] Therefore, in this embodiment, the target three-dimensional data represents the structure of the standard scanning rod and the position and posture of the actual scanning rod.

[0077] In some embodiments, the electronic device acquires the actual scanning data corresponding to the actual scanning rod in real time during the process of real-time stitching of multiple image frames in the first image frame set based on the characteristic information of the scanning rod area. However, the actual scanning data may be missing due to reasons such as occlusion or scanning device failure during the scanning process, which may affect the accuracy of subsequent image frame stitching and further affect the accuracy of the target three-dimensional data. In order to solve the above problem, the electronic device matches the standard data of the actual scanning rod from the standard scanning rod database based on the characteristic information of the scanning rod area, and replaces the scanning data of the actual scanning rod with the matched standard data of the actual scanning rod to obtain the target three-dimensional data. By replacing the actual scanning data of the actual scanning rod with the standard data of the actual scanning rod, it is ensured that whether the scanning data of the actual scanning rod is complete or partial, it can be replaced with the complete standard data of the actual scanning rod, thereby helping to improve the accuracy of the reconstructed target three-dimensional data.

[0078] In some embodiments, in the process of matching the standard data of the actual scanning rod from the standard scanning rod database, the matching means regular matching during the real-time scanning process. During the real-time scanning process, each time a certain amount of actual scanning data is obtained or each time interval, the current actual scanning data is feature-spliced ​​with the standard data. As the scanning data increases, the position and posture of the standard data of the actual scanning rod that has been feature-spliced ​​are updated. By updating the position and posture of the standard data of the actual scanning rod that has been feature-spliced ​​as the scanning data increases, the splicing accuracy and splicing success rate of the standard data of the actual scanning rod and the scanning data of the actual scanning rod can be improved, and the number of scanning rods is limited, whether it is one actual scanning rod or multiple actual scanning rods, feature splicing matching can be performed. Among them, the scanning data of the target scanning rod in the current scanning data is feature-spliced ​​and matched with the standard data, and the standard data will undergo coordinate transformation during the feature splicing process.

[0079] In some embodiments, matching the standard data of the actual scanning rod from the standard scanning rod database based on the feature information of the scanning rod area includes: determining the identification information of the actual scanning rod based on the feature information of the scanning rod area; and matching the standard data of the actual scanning rod from the standard scanning rod database based on the identification information.

[0080] Among them, the standard scanning rod database stores standard data of various types of scanning rods. The standard data of each type of scanning rod corresponds to identification information such as model. The electronic device can determine the model of the actual scanning rod in the scanning rod area based on the characteristic information of the scanning rod area, and based on the model of the actual scanning rod, retrieve the corresponding standard model of each actual scanning rod from the standard scanning rod database, and then form the target three-dimensional data based on the corresponding standard model of each actual scanning rod and the posture of each scanning rod in the scanning data.

[0081] In some embodiments, the electronic device may load a standard scanbar database into a cloud server, and download standard data of an actual scanbar from the cloud server.

[0082] In some embodiments of the present application, multiple image frames in a first image frame set are spliced ​​based on feature information in a scanning rod area, including: judging whether there is overlapping feature data between adjacent image frames in the multiple image frames based on the feature information in the scanning rod area; if there is overlapping feature data between adjacent image frames in the multiple image frames, splicing the adjacent image frames in real time based on the overlapping feature data; if there is no overlapping feature data between adjacent image frames in the multiple image frames, issuing a prompt message to prompt a user that the splicing has failed.

[0083] After all the multiple image frames in the first image frame set are stitched together, the target three-dimensional data may be globally optimized.

[0084] In some embodiments, the overlapping feature data represents feature data covering the same area (common area) between different image frames. The overlapping feature data may include geometric overlapping features, texture overlapping features, semantic overlapping features, and the like.

[0085] Taking the use of an oral digital scanner to scan the oral cavity in a certain state to obtain the first set of image frames as an example, when the oral scanner moves in the patient's mouth, multiple images will be continuously taken or point cloud data will be captured. The geometric features on the scanning rod exist as fixed reference points in these images. Each scan (single frame) will capture some features of the scanning rod, and these features form a common area between different image frames, that is, multiple image frames contain data of the same partial features. For example Figures 5 to 7 As shown, Figure 5 The image frame shown is Figure 6 There are common areas between the image frames shown, such as Figure 7 Shown in red area.

[0086] If each image frame contains feature information of the scanning rod area, such as some geometric features on the scanning rod, the electronic device can determine the common area between different image frames through the feature information of the scanning rod, and align and splice the different image frames according to the features of the common area. For example, if two adjacent image frames capture the same mark on the scanning rod, the electronic device can use the position information of the mark to determine the common area between the two adjacent image frames, calculate the relative position and angle between the two adjacent image frames based on the feature data of the common area, and splice the two image frames into a larger three-dimensional model based on the relative position and angle between the two adjacent image frames. With the real-time splicing of the image frames, a three-dimensional model of the oral cavity is reconstructed, for example Figure 8 shown.

[0087] However, in the case of a user scanning error or scanning device failure, some image frames may not contain the scanning rod area, for example, a plurality of consecutive image frames, image frame A (with scanning rod area), image frame B (without scanning rod area), image frame C (without scanning rod area), image frame D (with scanning rod area). In this case, based on the feature information of the scanning rod area, it will be determined that there is no overlapping feature data between image frame A and image frame B, image frame B and image frame C, and image frame C and image frame D. At this time, the electronic device can issue a prompt message to inform the user that the stitching failed.

[0088] In some embodiments, the prompting method may include voice, prompt box, scanning box color change, etc. The embodiment of the present application does not limit the prompting method.

[0089] In some embodiments, while or after prompting the user that stitching has failed, the electronic device may also issue a prompt message to prompt the user to return to the initial position to scan, and indicate the initial position on the interactive interface, where the initial position may include the scanning position corresponding to the last image frame that was successfully stitched.

[0090] In other embodiments of the present application, if stitching fails and there is no overlapping feature data between adjacent image frames in multiple image frames, no prompt message may be issued, and the current image frame that failed to be stitched is stitched with other image frames remaining in the first image frame or the target three-dimensional data obtained by real-time stitching to find a common area until stitching is successful.

[0091] In some embodiments, assuming that adjacent image frames include a first image frame and a second image frame, the electronic device can determine the overlapping feature data between adjacent image frames in the following manner: If both adjacent image frames contain preset marks, the electronic device can obtain the first geometric data of the preset mark and the adjacent mark in the first image frame, and obtain the second geometric data of the preset mark and the adjacent mark in the second image frame. Among them, the adjacent mark includes other marks within a preset area with the preset mark as the center. If the first geometric data and the second geometric data meet the preset conditions, the preset radius is determined according to the first geometric data and the second geometric data, and the feature data of the area enclosed by the preset mark and the preset radius in the first image frame and the second image frame is used as the overlapping feature data.

[0092] In this embodiment, the preset mark includes but is not limited to a coding mark and a geometric pattern mark. The preset area range can be customized according to the characteristic information of the actual scanning rod. The preset radius can be customized according to the first geometric data and the second geometric data.

[0093] In this embodiment, the first geometric data includes data such as the distance and relative angle between the preset mark and the adjacent mark in the first image frame. The second geometric data includes data such as the distance and relative angle between the preset mark and the adjacent mark in the second image frame. The preset condition can be set by user. For example, the preset condition can be set to that the error between the first geometric data and the second geometric data is less than a preset error threshold. The preset error threshold can be set by user.

[0094] As an example, the first image frame includes markers A1 and A2, and the second image frame includes markers A2 and A3. The marker A2 in the first image frame and the marker A2 in the second image frame are the same marker, and the electronic device can determine that the preset marker A2 exists in both the first image frame and the second image frame. The marker A1 included in the first image frame is an adjacent marker of the preset marker A2. Assume that the preset error threshold includes a distance error threshold and an angle error threshold, the distance error threshold is set to 0.2 mm, and the angle error threshold is set to 1. ° The electronic device can calculate the distance d1 and angle θ1 between marker A1 and marker A2, for example, d1=5mm, θ1=30 ° The second image frame includes a mark A3 which is an adjacent mark of the preset mark A2. The electronic device can calculate the distance d2 and the angle θ2 between the mark A3 and the mark A2, for example, d1=5.1 mm, θ1=29.5 ° The electronic device calculates the error between the first geometric data and the second geometric data: |d1-d2|=0.1mm, |θ1-θ2|=0.5 °. According to the fact that the value of |d1-d2| is less than the distance error threshold and the value of |θ1-θ2| is less than the angle error threshold, the electronic device can determine that the first geometric data and the second geometric data meet the preset conditions. At this time, the electronic device can determine that the marks A1 and A2 in the first image frame and A3 in the second image frame are the same group of marks. The electronic device can set a preset radius according to the distance d1 between the preset mark and the adjacent mark in the first image frame and the distance d2 between the preset mark and the adjacent mark in the second image frame. For example, the preset radius can be set to 5mm. The electronic device uses the feature data of the area enclosed by the preset mark and the preset radius in the first image frame and the second image frame as overlapping feature data.

[0095] In some embodiments, the electronic device may use an image processing algorithm (eg, an HSV color segmentation algorithm, a morphological filtering algorithm) to locate the mark on the scanning rod. This embodiment of the present application is not limited.

[0096] In other embodiments, the electronic device may also determine the overlapping feature data between adjacent image frames in other ways. For example, the overlapping feature data between adjacent image frames may be determined based on the geometric features of the feature marks of the scanning rod area in the adjacent image frames. This embodiment of the application is not limited to this.

[0097] In some embodiments, after determining the overlapping feature data between adjacent image frames, the electronic device may splice the adjacent image frames based on the overlapping feature data. Specifically, the electronic device may calculate the relative posture parameters between adjacent image frames, such as the optimal rotation matrix, translation vector and other parameters, based on the overlapping feature data. The electronic device maps the adjacent image frames to the same coordinate system based on the relative posture parameters, and fuses the point cloud data corresponding to the adjacent image frames to splice the adjacent image frames.

[0098] In actual applications, some doctors need to display a global oral model in order to quickly and accurately capture anatomical information in the oral cavity, such as details of the gums and surrounding tissues, to facilitate the design optimization of implants and restorations.

[0099] To display a global oral model, in some embodiments of the present application, the method also includes the following steps: S14: Acquire a second image frame set of the oral cavity in a second state; S15: Based on the second image frame set, acquire regional three-dimensional data of the soft tissue area in each image frame in the second image frame set; S16: Acquire a third image frame set, the third image frame set including partial image frames of the first image frame set or updated image frames acquired when the oral cavity is in the first state; S17: Based on the feature information of the soft tissue area in each image frame in the third image frame set and the feature information of the scanning rod area, merge the regional three-dimensional data corresponding to the soft tissue area with the target three-dimensional data to obtain an oral model.

[0100] The oral cavity in the second state indicates that no actual scanning rod is installed in the oral cavity, and the regional three-dimensional data includes three-dimensional point cloud data corresponding to the soft tissue area. In this embodiment, the three-dimensional data of the area obtained by scanning the oral cavity without the actual scanning rod installed will have a higher scanning accuracy. Finally, based on the third image frame set, the soft tissue model not interfered by the scanning rod and the scanning rod model not interfered by the gums can be combined into a global oral cavity model, and the reconstruction accuracy is higher.

[0101] Steps S14 and S15 can be performed before step S11, and S16 and S17 can be performed after step S13. Specifically, a scanning device can be used to scan the soft tissue (such as gums) in the oral cavity in the second state, and the scanning device sends the second image frame obtained by the scan to the electronic device. The electronic device obtains a second image frame set of the oral cavity in the second state, and based on the second image frame set, obtains regional three-dimensional data of the soft tissue area in each image frame in the second image frame set. After scanning the soft tissue in the oral cavity in the second state, the scanning device is used to scan the oral cavity in the first state, and the electronic device obtains a first image frame set. Based on the first image frame set, the electronic device obtains target three-dimensional data corresponding to the actual scanning rod in each image frame in the first image frame set. After obtaining the target three-dimensional data, the scanning device is used to scan the oral cavity in the first state to obtain a third image frame set. Based on the feature information of the soft tissue area in each image frame in the third image frame set and the feature information of the scanning rod area, the electronic device merges the regional three-dimensional data corresponding to the soft tissue area with the target three-dimensional data to obtain a global oral model such as Fig. 9 shown.

[0102] It should be noted that the third image frame set in steps S16 and S17 includes partial image frames of the first image frame set or updated image frames obtained when the mouth is in the first state, that is, they can be re-scanned and obtained using a scanning device, or they can be historical image frames obtained in step S11.

[0103] After acquiring the second set of image frames of the oral cavity in the second state, the global oral cavity model can be displayed on the interactive interface in response to the user's switching instruction. The interactive interface can have a switching instruction for displaying gingival soft tissue. After steps S14 and S15, the user can select and execute the above steps S16 and S17 to generate and display the global oral cavity model for the user to view.

[0104] In other embodiments of the present application, in response to a user's switching instruction, the second image frame set in steps S14 and S15 may not be obtained, and the soft tissue area identified in the first image frame set is spliced ​​with the target three-dimensional model to obtain a global oral model. At this time, since the target three-dimensional model obtained at the beginning has no interference from gum data, the final global oral model will also have better accuracy.

[0105] In some embodiments of the present application, the method further includes: generating a design model based on the target three-dimensional data and sending the design model to a 3D printing device; receiving a user request for modifying the design model in real time; and outputting a revised three-dimensional design model based on the user request and the target three-dimensional model. In the embodiment of the present application, a revised three-dimensional design model can be output based on the user request and the target three-dimensional model, so that the user can set dentures of different quantities, sizes and shapes.

[0106] The embodiment of the present application provides a three-dimensional reconstruction method. When an actual scanning rod is installed in the oral cavity, a first image frame set of the oral cavity is obtained, and the scanning rod area in each image frame in the first image frame set is identified. Based on the feature information of the scanning rod area, multiple image frames in the first image frame set are spliced ​​to reconstruct the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, thereby realizing image frame reconstruction based on a rigid structure. Since the rigid structure of the scanning rod is not easily deformed during the scanning process and has good stability, splicing multiple image frames in the first image frame set based on the feature information of the scanning rod area helps to improve the splicing accuracy, thereby improving the overall accuracy of the three-dimensional reconstruction, and making subsequent restoration designs based on the target three-dimensional data, thereby improving the success rate of implantation, and improving the matching degree between the restoration product and the patient and the comfort of the patient.

[0107] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0108] See also Fig.10 , Fig.10 The structure diagram of the 3D reconstruction device provided in the embodiment of the present application is shown, which can realize the details of the 3D reconstruction method in the above embodiment and achieve the same effect. Fig.10As shown, the three-dimensional reconstruction device 10 can be applied to an electronic device with a data processing function, and the three-dimensional reconstruction device 10 includes: an acquisition module 11, used to acquire a first image frame set of an oral cavity in a first state, wherein the oral cavity in the first state indicates that at least one actual scanning rod is installed in the oral cavity; an identification module 12, used to identify the scanning rod area of ​​each image frame in the first image frame set; and a reconstruction module 13, used to splice multiple image frames in the first image frame set based on feature information in the scanning rod area, and reconstruct target three-dimensional data corresponding to the actual scanning rod in the oral cavity.

[0109] For the specific definition of the 3D reconstruction device 10, please refer to the definition of the 3D reconstruction method above, which will not be repeated here. Each module in the above 3D reconstruction device 10 can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0110] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the three-dimensional reconstruction method in the above-mentioned embodiments of the present application. The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A three-dimensional reconstruction method, applied to electronic equipment, characterized in that: The three-dimensional reconstruction method comprises: Acquire a first image frame set of an oral cavity in a first state, wherein the oral cavity in the first state indicates that at least one actual scanning rod is installed in the oral cavity; identifying a scanning rod region of each image frame in the first set of image frames; Based on the feature information in the scanning rod area, multiple image frames in the first image frame set are spliced, and target three-dimensional data corresponding to the actual scanning rod in the oral cavity is reconstructed.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: Before reconstructing and obtaining the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, the method further includes: identifying a soft tissue region in each image frame in the first set of image frames; The soft tissue area does not participate in the three-dimensional reconstruction of the target three-dimensional data or participates in the three-dimensional reconstruction of the target three-dimensional data according to a preset weight.

3. The three-dimensional reconstruction method according to claim 1 or 2, characterized in that: The method further comprises: Based on identifying feature information of the soft tissue area in each image frame in the first image frame set and / or feature information of the scanning rod area, one or more of a removal operation, a marking operation, a hiding operation or a segmentation operation is performed on the soft tissue area in each image frame or the scanning rod area.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The first image frame set includes a texture image frame and a depth image frame, and the identifying a scanning rod area of ​​each image frame in the first image frame set includes: The texture image frame and the depth image frame are subjected to feature extraction and feature fusion by using a preset recognition model to obtain a mask image, wherein the mask image includes the scanning rod area.

5. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of splicing a plurality of image frames in the first image frame set based on the feature information in the scanning rod area and reconstructing target three-dimensional data corresponding to the actual scanning rod in the oral cavity includes: Based on the feature information in the scanning rod area, multiple image frames in the first image frame set are stitched in real time; Real-time three-dimensional reconstruction is performed based on the real-time spliced ​​image frames to obtain target three-dimensional data corresponding to the actual scanning rod.

6. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of splicing a plurality of image frames in the first image frame set based on the feature information in the scanning rod area and reconstructing target three-dimensional data corresponding to the actual scanning rod in the oral cavity includes: Based on the characteristic information of the scanning rod area, multiple image frames in the first image frame set are stitched in real time; Performing real-time three-dimensional reconstruction based on the spliced ​​image frames to obtain actual scanning data corresponding to the actual scanning rod in real time; Based on the feature information of the scanning rod area, the standard data of the actual scanning rod is matched from the standard scanning rod database. The matching is a regular matching in the real-time scanning process. During the real-time scanning process, each time a certain amount of actual scanning data is obtained or each time interval, the current actual scanning data is feature spliced ​​with the standard data. As the scanning data increases, the position and posture of the standard data is updated; The scan data of the actual scan rod is replaced by the matched standard data of the actual scan rod to obtain the target three-dimensional data.

7. The three-dimensional reconstruction method according to claim 1, characterized in that: The method further comprises: Acquire a second set of image frames of the oral cavity in a second state, wherein the oral cavity in the second state indicates that no actual scanning rod is installed in the oral cavity; Based on the second image frame set, acquiring regional three-dimensional data of a soft tissue region in each image frame in the second image frame set; Acquire a third image frame set, where the third image frame set includes partial image frames of the first image frame set or updated image frames acquired when the oral cavity is in the first state; Based on the feature information of the soft tissue area in each image frame in the third image frame set and the feature information of the scanning rod area, the regional three-dimensional data corresponding to the soft tissue area is merged with the target three-dimensional data to obtain an oral model.

8. The three-dimensional reconstruction method according to claim 7, characterized in that: The method further comprises: Based on the feature information in the scanning rod area, when acquiring a first set of image frames of the oral cavity in the first state, displaying the target three-dimensional data in real time on the interactive interface of the electronic device; or After acquiring a second set of image frames of the oral cavity in the second state, the oral cavity model is displayed on the interactive interface in response to a switching instruction of the user.

9. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of stitching a plurality of image frames in the first image frame set based on the feature information in the scanning rod region comprises: Based on the feature information in the scanning rod area, determining whether there is overlapping feature data between adjacent image frames in the plurality of image frames; If there is overlapping feature data between adjacent image frames in the plurality of image frames, then based on the overlapping feature data, the adjacent image frames are spliced ​​in real time; If there is no overlapping feature data between adjacent image frames in the multiple image frames, a prompt message is issued to prompt the user that the stitching fails, and / or a prompt message is issued to prompt the user to return to the initial position for scanning.

10. The three-dimensional reconstruction method according to claim 1, characterized in that: The method further comprises: Generate a design model based on the target three-dimensional data and send the design model to the 3D printing device; receiving user requests for modifying design models in real time; Output a revised 3D design model based on user request and target 3D model.

11. A three-dimensional reconstruction device, applied to electronic equipment, characterized in that: The three-dimensional reconstruction device comprises: An acquisition module, configured to acquire a first image frame set of an oral cavity in a first state, wherein the oral cavity in the first state indicates that at least one actual scanning rod is installed in the oral cavity; an identification module, used to identify a scanning rod area of ​​each image frame in the first image frame set; A reconstruction module is used to splice multiple image frames in the first image frame set based on feature information in the scanning rod area, and reconstruct the target three-dimensional data corresponding to the actual scanning rod in the oral cavity.

12. An electronic device, characterized in that: The method comprises a memory, a processor and computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, implement the three-dimensional reconstruction method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 10 is implemented.

Images