Three-dimensional reconstruction method, apparatus, electronic device, and storage medium

Abstract

The influence of anatomical structures (such as soft tissues like gums, lips, and tongue) within the image frame on stitching accuracy can be reduced, thereby improving the overall accuracy of three-dimensional reconstruction. [Solution] The present application relates to the field of three-dimensional reconstruction technology and provides a three-dimensional reconstruction method, apparatus, electronic device, and storage medium. The method includes the steps of: acquiring a first set of image frames of an oral cavity in a first state, wherein the oral cavity in a first state refers to a state in which at least one actual scan rod is installed in the oral cavity; identifying the scan rod region in each image frame of the first set of image frames; and stitching together a plurality of image frames of the first set of image frames based on characteristic information of the scan rod region to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity. The method performs stitching based on a rigid structure.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and particularly to a three-dimensional reconstruction method, apparatus, electronic device, and storage medium.

Background Art

[0002] In the oral scanning process, a plurality of image frames need to be stitched and aligned through a common area (or transition area) to reconstruct a complete three-dimensional model. In related technologies, in scenes such as edentulous implants and large-span implants, usually a soft tissue area such as the gingiva is selected as the transition area to stitch data of different areas or different image frames. However, since soft tissues such as the gingiva lack rigid support, they are easily deformed by external forces (such as opening movements and instrument contact) or muscle traction (such as mouth-opening movements), and the patient's physiological activities (such as breathing and saliva secretion) are also likely to cause minute displacements of the soft tissues, thereby changing the morphology of the soft tissue area. In this case, when the soft tissue area is selected as the transition area, the coordinates of the transition area in the scan data shift, affecting the stitching accuracy of data in adjacent areas or adjacent frames, and further reducing the accuracy of three-dimensional reconstruction. Finally, when performing computer-aided design (CAD) of a prosthesis (denture) based on the deformed gingiva data, there is a possibility that the prosthesis may compress the gingiva or cause voids when worn in the oral cavity. If the deformation exceeds a certain limit, it may cause discomfort to the patient in mild cases and implant failure in severe cases.

Summary of the Invention

[0003] In view of the above, this application provides a three-dimensional reconstruction method, apparatus, electronic device, and storage medium to solve the problem of large stitching errors and further affecting the accuracy of three-dimensional reconstruction.

[0004] A first embodiment of the present invention provides a three-dimensional reconstruction method applicable to electronic equipment, the three-dimensional reconstruction method comprising the steps of: obtaining a first set of image frames of an oral cavity in a first state, wherein the oral cavity in a first state refers to a state in which at least one actual scan rod is fitted inside the oral cavity; identifying a scan rod region in each image frame of the first set of image frames; and stitching together a plurality of image frames of the first set of image frames based on characteristic information of the scan rod region to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity.

[0005] In some embodiments, prior to the step of reconstructing target three-dimensional data corresponding to the actual scan rod in the oral cavity, the method further includes the step of identifying soft tissue regions in each image frame of the first image frame set, wherein the soft tissue regions either do not participate in the three-dimensional reconstruction of the target three-dimensional data or participate in the three-dimensional reconstruction of the target three-dimensional data according to a predetermined weight.

[0006] In some embodiments, the method further includes the step of performing one or more of the following operations on the soft tissue region or the scan rod region in each image frame, based on identifying feature information of the soft tissue region and / or feature information of the scan rod region in each image frame of the first image frame set: removal, marking, concealment, or splitting.

[0007] In some embodiments, the first image frame set includes a texture image frame and a depth image frame, and the step of identifying the scan rod region in each image frame of the first image frame set includes the step of performing feature extraction and feature fusion on the texture image frame and the depth image frame using a pre-configured identification model to obtain a mask image, the mask image including the scan rod region.

[0008] In some embodiments, the step of stitching together multiple image frames of the first image frame set based on the characteristic information of the scan rod region to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity includes the steps of stitching together multiple image frames of the first image frame set in real time based on the characteristic information of the scan rod region, and performing three-dimensional reconstruction in real time based on the stitched image frames in real time to obtain target three-dimensional data corresponding to the actual scan rod.

[0009] In some embodiments, the step of stitching together a plurality of image frames of the first image frame set based on the feature information of the scan rod region to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity includes the steps of stitching together a plurality of image frames of the first image frame set in real time based on the feature information of the scan rod region, performing three-dimensional reconstruction in real time based on the stitched image frames to acquire actual scan data corresponding to the actual scan rod in real time, matching standard data of the actual scan rod from a standard scan rod database based on the feature information of the scan rod region, wherein the matching is periodic matching in the real-time scanning process, and in the real-time scanning process, each time a certain amount of actual scan data is acquired or at regular intervals, the current actual scan data and the standard data are feature-stitched together, and the orientation of the standard data is updated as the scan data increases, and the scan data of the actual scan rod is replaced with the standard data of the actual scan rod obtained by matching to acquire the target three-dimensional data.

[0010] In some embodiments, the method further includes the steps of: acquiring a second set of image frames of the oral cavity in a second state, where the oral cavity in a second state refers to a state in which no actual scan rod is inserted into the oral cavity; acquiring three-dimensional region data of the soft tissue region in each image frame of the second set of image frames based on the second set of image frames; acquiring a third set of image frames, where the third set of image frames includes some image frames from the first set of image frames, or updated image frames acquired when the oral cavity is in the first state; and merging the three-dimensional region data corresponding to the soft tissue region and the target three-dimensional data based on the feature information of the soft tissue region and the feature information of the scan rod region in each image frame of the third set of image frames to acquire an oral cavity model. Here, data integration (data merging) includes two forms: one is a data fusion method, and the other is a method of unifying coordinate systems.

[0011] In some embodiments, the method further includes the steps of: displaying the target three-dimensional data in real time on the interaction interface of the electronic device when acquiring a first set of image frames of the oral cavity in a first state based on the feature information of the scan rod region; or, after acquiring a second set of image frames of the oral cavity in a second state, displaying the oral cavity model on the interaction interface in response to a user's switching command.

[0012] In some embodiments, the step of stitching a plurality of image frames of the first image frame set based on the feature information of the scan rod region includes the steps of: determining whether or not there is overlapping feature data between adjacent image frames in the plurality of image frames based on the feature information of the scan rod region; if there is overlapping feature data between adjacent image frames in the plurality of image frames, stitching the adjacent image frames in real time based on the overlapping feature data; and if there is no overlapping feature data between adjacent image frames in the plurality of image frames, issuing a notification to the user to indicate stitching failure and / or issuing a notification to the user to return to the initial position and scan again.

[0013] In some embodiments, the method further includes the steps of generating a design model based on target three-dimensional data and transmitting the design model to a 3D printing device; receiving a user request to modify the design model in real time; and outputting a modified three-dimensional design model based on the user request and the target three-dimensional model.

[0014] A second embodiment of the present invention provides a three-dimensional reconstruction apparatus applicable to an electronic device, the three-dimensional reconstruction apparatus comprising an acquisition module, an identification module, and a reconstruction module, wherein the acquisition module is a module for acquiring a first set of image frames of a first state oral cavity, the first state oral cavity being a state in which at least one actual scan rod is installed in the oral cavity, the identification module is a module for identifying the scan rod region in each image frame of the first set of image frames, and the reconstruction module is a module for stitching together a plurality of image frames of the first set of image frames based on the characteristic information of the scan rod region and reconstructing target three-dimensional data corresponding to the actual scan rod in the oral cavity.

[0015] A third embodiment of the present invention provides an electronic device including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the above-described three-dimensional reconstruction method is realized when the processor executes the computer-readable instructions.

[0016] A fourth embodiment of the present invention provides a computer-readable storage medium in which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the above-described three-dimensional reconstruction method is realized.

[0017] The embodiment of the present invention provides a three-dimensional reconstruction method in which, when an actual scan rod is placed in the oral cavity, a first set of image frames of the oral cavity is acquired, and the scan rod region is identified in each image frame of the first set of image frames. Based on the characteristic information of the scan rod region, multiple image frames of the first set of image frames are stitched together, and target three-dimensional data corresponding to the actual scan rod in the oral cavity is reconstructed, thereby realizing image frame stitching based on a rigid structure. Since the rigid structure of the scan rod is less prone to deformation during the scanning process and has good stability, stitching multiple image frames of the first set of image frames based on the characteristic information of the scan rod region contributes to improving stitching accuracy and further improves the overall accuracy of the three-dimensional reconstruction. In addition, by performing subsequent prosthetic design based on the target three-dimensional data, the success rate of implants, the fit between the prosthetic product and the patient, and the patient's comfort can be improved. [Brief explanation of the drawing]

[0018] To more clearly explain the three-dimensional reconstruction method, the following are some of the drawings necessary for describing the embodiments of this application. Clearly, the following drawings represent only a portion of the embodiments of this application, and those skilled in the art can obtain other drawings based on these without expending any creative effort. [Figure 1] This is an equipment diagram of the three-dimensional reconstruction method provided in the embodiment of the present application. [Figure 2] This is a flowchart showing the implementation process of the three-dimensional reconstruction method provided in the embodiments of the present application. [Figure 3] This is an exemplary diagram of the interaction interface provided in the first embodiment of the present application. [Figure 4] This is an exemplary diagram of the interaction interface provided in the second embodiment of the present application. [Figure 5] This is an exemplary diagram of the scan rod area in the image frame provided in the first embodiment of the present application. [Figure 6] This is an exemplary diagram of the scan rod area in the image frame provided in the second embodiment of the present application. [Figure 7] This is an exemplary diagram of the scan rod area in the image frame provided in the third embodiment of the present application. [Figure 8] This is an exemplary diagram of the target three-dimensional data provided in the embodiments of the present application. [Figure 9] This is an exemplary diagram of the oral model provided in the embodiments of the present application. [Figure 10] This is a structural schematic diagram of the three-dimensional reconstruction device provided in the embodiments of the present application.

Modes for Carrying Out the Invention

[0019] Hereinafter, referring to the drawings of the embodiments of the present application, the technical solutions of the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application.

[0020] Hereinafter, the terms “First” or “Second” are used solely for descriptive purposes and should not be understood as indicating or implying relative importance, or as implying the number of technical features to be described. Therefore, features limited by “First” or “Second” may explicitly or implicitly include one or more such features. In the description of the embodiments of this application, terms such as “exemplary” or “for example” are used for example, case, or explanation. Any embodiment or design solution described as “exemplary” or “for example” in the embodiments of this application should not be construed as preferable or superior to other embodiments or design solutions. More precisely, terms such as “exemplary” or “for example” are intended to present the relevant concepts in a specific manner.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art relating to the present application. The terms used in this specification are for illustrative purposes only and are not intended to limit the application. In this application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B. In this application, "and / or" indicates only a related relationship describing the subject matter in question, and indicates that three relationships may exist. For example, A and / or B may indicate three cases: A only exists, A and B exist simultaneously, and B only exists. "At least one" means one or more. "Multiple" means two or more. For example, at least one of a, b or c may indicate seven cases: a, b, c, a and b, a and c, b and c, a, b and c.

[0022] Referring to FIG. 1, FIG. 1 is a device diagram of a three-dimensional reconstruction method provided in an embodiment of the present application. As shown in FIG. 1, the three-dimensional reconstruction method provided in the embodiment of the present application can be applied to an electronic device 100, and the electronic device 100 can include one or more of devices such as a mobile phone, a tablet computer, a smart wearable device, a notebook computer, and a scanning device (such as an oral digital impression device). The embodiment of the present application does not limit the specific type of the electronic device in any way.

[0023] As shown in FIG. 1, the electronic device 100 can include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is connected to the communication module 101, the memory 102, and the I / O interface 104 via the bus 105, respectively.

[0024] The communication module 101 can include a wired communication module and / or a wireless communication module. The wired communication module can provide one or more solutions for wired communication such as a universal serial bus (USB) and a controller area network bus (CAN). The wireless communication module can provide one or more solutions for wireless communication such as wireless fidelity (Wi-Fi), Bluetooth (BT), a mobile communication network, frequency modulation (FM), near field communication technology (NFC), and infrared technology (IR).

[0025] Memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). Random access memories can be directly read and written by the processor 103 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc. Random access memories may include static random access memories (SRAM), dynamic random access memories (DRAM), synchronous dynamic random access memories (SDRAM), and double datarate synchronous dynamic random access memories (DDR SDRAM), etc.

[0026] Non-volatile memory can also store executable program data, user and application data, etc., and can be pre-loaded into random access memory and directly read and written by processor 103. Non-volatile memory can include disk storage devices and flash memory.

[0027] Memory 102 is used to store one or more computer programs. One or more computer programs are configured to be executed by processor 103. The one or more computer programs include multiple instructions, and when the multiple instructions are executed by processor 103, a three-dimensional reconstruction method can be realized to be executed on electronic device 100.

[0028] In other embodiments, the electronic device 100 further includes an external memory interface, which is used to connect to external memory and extend the storage capacity of the electronic device 100.

[0029] The processor 103 may include one or more processing units, for example, an application processor (AP), a modulation / demodulation processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Here, different processing units may be independent devices or may be integrated into one or more processors.

[0030] The processor 103 provides computation and control capabilities; for example, the processor 103 is used to execute a computer program stored in memory 102, thereby realizing the three-dimensional reconstruction method described above.

[0031] The I / O interface 104 is used to provide user input or output channels. For example, it can be used to connect to various input / output devices such as a mouse, keyboard, touch device, and display, enabling users to input or visualize information. The I / O interface 104 can also be used to connect to various sensor devices such as radar sensors and image sensors, allowing for the acquisition of necessary data such as radar point cloud data and image data.

[0032] Bus 105 is used to provide a channel for mutual communication between at least the communication module 101, memory 102, processor 103, and I / O interface 104 in the electronic device 100.

[0033] To ensure that it is understood that the structures shown in the embodiments of this application do not constitute a specific limitation of the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than those shown, some components may be combined, some components may be separated, and may have different component arrangements. The illustrated components can be realized by hardware, software, or a combination of software and hardware.

[0034] The scene shown in Figure 1 is merely an illustrative example, and the three-dimensional reconstruction method provided in this application can be applied to other scenes. For example, in some scenes, an electronic device 100 and a scanning device may be included, the electronic device 100 and the scanning device may be installed separately and communicate by wired or wireless means, the electronic device may be a computer, tablet computer, and / or mobile phone, and the scanning device may be an oral scanner (intraoral scanner or extraoral scanner) or an imaging device attached to an implant handpiece. In some scenes, the electronic device 100 may be installed inside the scanning device. In some scenes, other types of devices may also be included. As described above, the embodiments of this application do not limit the specific application scenes of the three-dimensional reconstruction method.

[0035] As shown in Figure 2, Figure 2 is a flowchart illustrating the implementation of a three-dimensional reconstruction method provided in an embodiment of the present application. The method is applied to electronic devices, and the embodiment of the present application describes an example in which a computer program product of the method is executed on an electronic device (for example, on electronic device 100 in Figure 1). The method includes the following steps:

[0036] S11: Obtain the first set of image frames of the oral cavity in the first state.

[0037] In some embodiments, the first state of the oral cavity refers to a state in which at least one actual scan rod is mounted in the oral cavity. The actual scan rod can be mounted on an implant or base in the oral cavity. The oral cavity may be that of a patient with missing teeth, for example, the oral cavity of a patient with complete edentulism or a patient with partial edentulism. Therefore, the user can scan the oral cavity in the first state using a scanning device such as an intraoral scanner and an extraoral scanner, or an imaging device mounted on an implant handpiece, and obtain a first set of image frames.

[0038] In some embodiments, specific geometric patterns and / or marks are provided on the surface of the actual scan rod, and these geometric patterns or marks can be accurately captured by the scanning device during scanning, providing reference points to help the scanning device identify and position the scan rod from multiple angles or positions. For example, the geometric patterns may be high-contrast patterns, regular geometric shapes (e.g., circles, squares), or irregular geometric shapes (e.g., grooves and protrusions), and the marks may be unsigned marks or signed marks, for example, unsigned markers or signed markers. Here, the specific geometric patterns and / or marks may be uniformly or non-uniformly distributed on the actual scan rod.

[0039] In some embodiments, the first image frame set includes multiple image frames. An image frame refers to a single two-dimensional image captured by the scanning device at a specified point in time during the oral cavity scanning process. When scanning the oral cavity in a first state, each image frame in the first image frame set includes the scanning rod in the oral cavity and surrounding tissues (e.g., soft tissues such as teeth, gums, and mucous membranes).

[0040] S12: Identify the scan rod region in each image frame of the first image frame set.

[0041] In some embodiments, the scan rod region refers to a region in the image frame that contains local information of the actual scan rod.

[0042] In some embodiments of the present invention, the first image frame set includes texture image frames and depth image frames, and the step of identifying scan rod regions in each image frame of the first image frame set includes the step of performing feature extraction and feature fusion on the texture image frame and depth image frame using a pre-configured identification model to obtain a mask image, the mask image including scan rod regions.

[0043] In some embodiments, a texture image frame refers to a two-dimensional color image of the inside of the oral cavity, used to reflect the surface visual features of tissues such as teeth, gums, or scan rods, and includes details such as color, texture, spots, cracks, prostheses (e.g., fillings or crowns), surface patterns, and unsigned or signed markers. A depth image frame refers to two-dimensional spatial information of the oral cavity structure, where each pixel value in the depth image frame can represent the distance (depth value) from the corresponding position to the scanning device, and includes details such as geometric shapes and grooves on the scan rod, and is usually represented in grayscale or pseudocolor (e.g., brighter near and darker far).

[0044] In some embodiments, electronic devices can acquire depth image frames using optical three-dimensional scanning techniques (e.g., structural optics, laser triangulation, or binocular stereo vision techniques). For example, texture image frames can be acquired by a color camera when a scanning device projects white light onto the object being scanned, while depth image frames can be acquired by a monochrome camera when a scanning device projects structural light onto the object being scanned. The embodiments of this application do not limit the specific method for acquiring depth image frames.

[0045] In some embodiments, the pre-configured discrimination model may be a deep learning model, such as a convolutional network model, a circular network model, a generative counter-network model, or a semantic segmentation model.

[0046] In some embodiments, a pre-configured identification model includes a multi-layer convolutional network and a multi-layer deconvolutional network. The electronic device uses the pre-configured identification model to perform feature extraction and information fusion on the current frame image and depth image, and outputs a mask image. This step includes using a multi-layer convolutional network to perform feature extraction and feature fusion on the current frame image and depth image to obtain a feature map, and using a multi-layer deconvolutional network to perform upsampling and restoration on the feature map to obtain a mask image.

[0047] In some embodiments, the electronic device uses multiple inverse convolutional layers to perform upsampling and decompression on the feature map and obtain a mask image. This step involves performing a transpose convolution operation on the feature map, upsampling, stitching together the shallow features of the corresponding layers of the encoder with the deep features after upsampling on a channel-by-channel basis, and integrating detail and semantic information. The above steps are repeated until the resolution of the feature map is restored to the input dimensions, then an activation function is used to predict the category probability of each pixel in the feature map and obtain a mask image.

[0048] S13: Based on the feature information of the scan rod region, multiple image frames from the first image frame set are stitched together to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity.

[0049] In some embodiments, the feature information of the scan rod region includes, but is not limited to, geometric features (e.g., point cloud data, edges and contours, and pre-defined keypoints) and information on pre-defined marks on the actual scan rod (e.g., geometric patterns and marks).

[0050] In some embodiments, the target three-dimensional data may include three-dimensional point cloud data and a three-dimensional model corresponding to the actual scan rod. The target three-dimensional data includes shape information and positioning information of the scan rod. Therefore, by stitching together multiple image frames of the first image frame set based on the characteristic information of the scan rod region, this invention contributes to improving stitching accuracy, further improving the overall accuracy of three-dimensional reconstruction, and enabling the acquisition of more accurate scan rod positioning information. In addition, by performing subsequent prosthetic design (e.g., design of crowns, bridges, and dentures) based on the target three-dimensional data, the success rate of implants, the fit between the prosthetic product and the patient, and patient comfort can be improved.

[0051] In some embodiments of the present invention, prior to reconstructing target three-dimensional data corresponding to an actual scan rod in the oral cavity, the method further includes the step of identifying soft tissue regions in each image frame of a first set of image frames, wherein the soft tissue regions either do not participate in the three-dimensional reconstruction of the target three-dimensional data or participate in the three-dimensional reconstruction of the target three-dimensional data according to a predetermined weight.

[0052] Here, soft tissue includes one or more soft tissue objects within the oral region, such as the tongue, gums, and lips. The soft tissue region refers to the region in the image frame where soft tissue is located. The pre-set weights may be weights entered by the user, or they may be pre-set weights calculated using a pre-set model (e.g., an AI model) based on the user's edentulous status and age. And / or, the pre-set weights are determined by performing a quality analysis of the image frames scanned by the scanning device (e.g., image frames in the first image frame set or the second image frame set) using a pre-set model. And / or, the pre-set weights are determined based on the stitching error or stitching success rate of the target three-dimensional data obtained by real-time stitching using a pre-set model.

[0053] In some embodiments, the pre-configured model may be a deep learning model, such as a convolutional network model. The embodiments of this application are not limited to pre-configured models.

[0054] In some embodiments, the electronic device can be used to identify soft tissue regions in each image frame of a first image frame set by training a deep learning model based on feature data of soft tissue objects such as the tongue, gums, or lips, for example, the shape, edge features, or color features of the gums. In other embodiments, the electronic device can also identify soft tissue regions in each image frame of a first image frame set by other means. The embodiments of this application are not limited to these.

[0055] It should be explained that in one embodiment, only the scan rod region can be identified, all regions other than the scan rod can be removed from the image frames in the first image frame set, and reconstruction can be performed only on the scan rod region. In one embodiment, semantic identification can be performed on each part of the image frames in the first image frame set to distinguish whether it is soft tissue or a scan rod, and it can be determined that the soft tissue region does not participate in the reconstruction. In one embodiment, semantic identification can be performed on each part of the image frames in the first image frame set to distinguish between soft tissue and a scan rod in the image frame, thereby determining the soft tissue region and having it participate in the reconstruction according to a pre-set weight, thereby avoiding stitching loss. In one embodiment, the soft tissue region can be further distinguished, and the soft tissue can be further distinguished into tongue, gingiva, or lips, the tongue and lips can be removed as miscellaneous data, and the gingival data can be saved for subsequent operations.

[0056] Specifically, semantic identification is performed on each part of the image frames in the first set of image frames to distinguish whether it is the tongue, gums, lips, or scan rod. For example, if it is the tongue or lips, it is deleted; if it is the scan rod, it is assigned an 80% weight; and if it is the gums, it is assigned a 20% weight. When stitching multiple image frames, it is determined whether there is overlapping feature data between adjacent image frames within the multiple image frames. If there is overlapping feature data between adjacent image frames within the multiple image frames, the adjacent image frames are stitched in real time based on the overlapping feature data. Here, the electronic device can determine whether there is overlapping feature data between adjacent image frames within the multiple image frames based on the weight of the scan rod (e.g., 80%) and the weight of the gums (e.g., 20%). As another example, the electronic device can set different weights based on different classifications of the semantic identification results and use them for stitching. Specifically, the electronic device performs semantic identification on each part of the image frame in the first image frame set, distinguishing whether it is the tongue, gums, lips, or scan rod, and assigns weights to the scan data obtained by the scan based on the semantic identification result. For example, if the initial value of the scan data in each image frame is set to 1, and the semantic identification result is the tongue, the weight is decreased based on the initial value of the scan data, and the corresponding weight for the tongue is obtained, which can be decreased by, for example, 80%, in which case the corresponding weight for the tongue is 20%. If it is the lips, the weight is decreased based on the initial value of the scan data, and the corresponding weight for the lips is obtained, which can be decreased by, for example, 80%, in which case the corresponding weight for the lips is 20%. If the semantic identification result is the scan rod, the weight is increased based on the initial value of the scan data, and the corresponding weight for the scan rod is obtained, which can be increased by, for example, 100%.If the semantic identification result is gingiva, the weight is increased based on the initial value of the scan data to obtain the corresponding weight for when the semantic identification result is gingiva, for example, by increasing the weight by 20%. Based on all the scan data after multiple calculations and verifications, the electronic device stitches and reconstructs the scan data where the weights meet a pre-set threshold range to obtain the target three-dimensional data.

[0057] In some embodiments of the present application, the electronic device can perform one or more of the following operations on the soft tissue region or scan rod region in each image frame, based on identifying feature information of the soft tissue region and / or feature information of the scan rod region in each image frame of the first image frame set: removal, marking, concealment, or division.

[0058] It should be explained that removal operations may be deletions, markings may be markings with different colors, splittings may be divisions with defined boundaries, and concealments may be concealing pre-selected areas. Marking, concealment, or splitting operations do not affect the integrity of the scan data, and if necessary, they can be restored and all scan data can be displayed through the settings.

[0059] In some embodiments, the soft tissue region feature information may include the geometric morphology of the soft tissue (e.g., gingival edge waveform, tongue coating contour, etc.), color and texture features, three-dimensional structural features (e.g., topological structure, spatial deformation features, etc.), and functional and physiological state features. When a scan rod is inserted into the oral cavity, the soft tissue region feature information may further include interaction features between the soft tissue and the scan rod, such as the contact area and pressure distribution between the soft tissue and the scan rod. The embodiments of this application do not limit the specific content of the soft tissue region feature information.

[0060] In conventional technology, a scanning device (e.g., an oral digital impression device / intraoral scanner or extraoral scanner) and a scanning rod are combined to perform three-dimensional reconstruction, which allows for the restoration of the anatomical morphology of the patient's oral cavity (e.g., dentition, gingiva, and occlusal relationships) and is used for the digital design of prostheses or implants. However, when the scanning rod is placed inside the patient's oral cavity, the complex anatomical structures surrounding the scanning rod (e.g., soft tissues such as gingiva, lips, and tongue) easily cause data overlap or interference with the scanning rod area during the scanning process, leading to mismatches in common areas during image frame stitching. As a result, errors accumulate frame by frame, ultimately causing distortion or stitching breaks in the three-dimensional model, further affecting the accuracy and clinical fit of subsequent prosthetic designs.

[0061] In some embodiments, texture image frames and depth image frames can be acquired synchronously. In other embodiments, both can be acquired by continuously capturing the same location or through a sequence of preceding and succeeding frames. The texture image frames and depth image frames can be marked as associated images.

[0062] For example, as shown in Figure 3, the viewport of the intraoral scanning device is displayed in the upper left of the illustration, and the reconstructed three-dimensional model is displayed in the lower right of the illustration. As shown in Figure 3, the reconstructed three-dimensional model includes a large area of ​​soft tissue (for example, the area enclosed by the frame in Figure 3), and soft tissue areas are easily deformable, easily causing data overlap or interference with the scan rod area.

[0063] To solve the above problem, the electronic device can perform one or more of the following operations on the soft tissue regions in each image frame of the first image frame set, based on the feature information of the soft tissue regions and / or the feature information of the scan rod region in each image frame. This allows the electronic device to stitch multiple image frames based only on the feature information of the scan rod region, improving the accuracy of stitching and further improving the accuracy of three-dimensional reconstruction. For example, as shown in Figure 4, a removal operation is performed on the soft tissue regions in each image frame, and the soft tissue regions are not displayed in the target three-dimensional data (three-dimensional model) obtained by stitching and three-dimensionally reconstructing the image frames after the removal operation (for example, shown in the lower right of Figure 4).

[0064] In some embodiments of the present invention, the step of stitching together multiple image frames of a first image frame set based on characteristic information of the scan rod region to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity includes the steps of stitching together multiple image frames of a first image frame set in real time based on characteristic information of the scan rod region, and performing three-dimensional reconstruction in real time based on the stitched image frames in real time to obtain target three-dimensional data corresponding to the actual scan rod. Here, when acquiring the first image frame set of the oral cavity in a first state, the target three-dimensional data is displayed in real time on the interaction interface of the electronic device. Therefore, in this embodiment, the identification of the scan rod region or soft tissue region of each image frame in the first image frame set is also a real-time operation, thereby enabling real-time scanning and real-time stitching reconstruction. During the scanning process, as the user moves within the oral cavity, the target three-dimensional data shown in Figure 8 can be gradually obtained from the target three-dimensional data shown in the lower right of Figure 4, and simultaneously displayed in real time on the interaction interface, making it easier for the user to grasp the progress of the scan.

[0065] In some embodiments of the present invention, the step of stitching together multiple image frames of a first image frame set based on characteristic information of the scan rod region to reconstruct target three-dimensional data corresponding to an actual scan rod in the oral cavity includes: stitching together multiple image frames of a first image frame set in real time based on characteristic information of the scan rod region; performing three-dimensional reconstruction in real time based on the stitched image frames to acquire actual scan data corresponding to an actual scan rod in real time; matching standard data of an actual scan rod from a standard scan rod database based on characteristic information of the scan rod region; and replacing the scan data of an actual scan rod with the standard data of an actual scan rod obtained by matching to acquire target three-dimensional data.

[0066] Here, the standard scanrod database stores standard data for multiple types of scanrods. Standard data refers to a complete, incomplete, and error-free set of reference parameters for a scanrod in an ideal state, such as a CAD model of a designed scanrod. Therefore, in this embodiment, the target three-dimensional data refers to the structure of the standard scanrod and the orientation of the actual scanrod.

[0067] In some embodiments, the electronic device acquires actual scan data corresponding to the actual scan rod in real time during the process of stitching multiple image frames of a first image frame set in real time, based on the characteristic information of the scan rod region. However, actual scan data may be lost due to occlusion during the scanning process or failure of the scanning device, which can affect the accuracy of subsequent image frame stitching and further affect the accuracy of the target three-dimensional data. To solve the above problem, the electronic device matches the standard data of the actual scan rod from a standard scan rod database based on the characteristic information of the scan rod region, replaces the scan data of the actual scan rod with the standard data of the actual scan rod obtained through matching, and acquires the target three-dimensional data. By replacing the actual scan data of the actual scan rod with the standard data of the actual scan rod, it is possible to replace the scan data of the actual scan rod with complete standard data of the actual scan rod, regardless of whether the scan data of the actual scan rod is complete or partial, which is advantageous in improving the accuracy of the reconstructed target three-dimensional data.

[0068] In some embodiments, in the process of matching standard data of actual scan rods from a standard scan rod database, matching refers to periodic matching during the real-time scanning process. During the real-time scanning process, feature stitching is performed between the current actual scan data and the standard data each time a certain amount of actual scan data is acquired, or at regular intervals. As the scan data increases, the orientation of the feature-stitched standard data of the actual scan rods is updated. By updating the orientation of the feature-stitched standard data of the actual scan rods as the scan data increases, the stitching accuracy and success rate between the standard data of the actual scan rods and the scan data of the actual scan rods can be improved. Furthermore, feature stitching matching can be performed for one or multiple actual scan rods, regardless of the number of scan rods. Here, feature stitching matching is performed between the scan data of the target scan rod in the current scan data and the standard data, and the standard data undergoes a coordinate transformation during the feature stitching process.

[0069] In some embodiments, the step of matching standard data of an actual scan rod from a standard scan rod database based on characteristic information of the scan rod region includes the step of determining identification information of an actual scan rod based on characteristic information of the scan rod region, and the step of matching standard data of an actual scan rod from a standard scan rod database based on the identification information.

[0070] Here, the standard scan rod database stores standard data for multiple types of scan rods. Each type of scan rod's standard data is associated with identification information, such as a model number. Based on the characteristic information of the scan rod area, the electronic device can determine the model number of the actual scan rod in the scan rod area. Based on the actual scan rod model number, it retrieves the standard model corresponding to each actual scan rod from the standard scan rod database, and constructs the target three-dimensional data based on the standard model corresponding to each actual scan rod and the orientation of each scan rod in the scan data.

[0071] In some embodiments, the electronic device can load a standard scan rod database onto a cloud server and download standard data for the actual scan rod from the cloud server.

[0072] In some embodiments of the present invention, the step of stitching a plurality of image frames of a first image frame set based on feature information of a scan rod region includes the step of determining whether or not there is overlapping feature data between adjacent image frames in the plurality of image frames based on the feature information of the scan rod region; if there is overlapping feature data between adjacent image frames in the plurality of image frames, the step of stitching the adjacent image frames in real time based on the overlapping feature data; and if there is no overlapping feature data between adjacent image frames in the plurality of image frames, the step of issuing notification information to the user to indicate that stitching has failed.

[0073] Here, after the stitching of multiple image frames from the first image frame set is complete, global optimization can be performed on the target three-dimensional data.

[0074] In some embodiments, overlapping feature data refers to feature data that covers the same region (common region) across different image frames. Overlapping feature data can include geometric overlapping features, texture overlapping features, and semantic overlapping features.

[0075] To give an example of using an oral digital scanner to perform an intraoral scan of the oral cavity in a first state and obtain a first set of image frames, the oral scanner continuously captures multiple images or point cloud data as it moves through the patient's oral cavity. In these images, geometric features on the scan rod exist as fixed reference points. Each scan (single frame) captures partial features of the scan rod, and these features form a common region between different image frames, meaning that the same part of the feature data is included in multiple image frames. For example, as shown in Figures 5 to 7, a common region exists between the image frame shown in Figure 5 and the image frame shown in Figure 6, and the common region is as shown by the dashed line in Figure 7.

[0076] If each image frame contains feature information of the scan rod region, such as partial geometric features on the scan rod, the electronic device can determine common regions between different image frames based on the scan rod feature information, and then align and stitch the different image frames based on the features of the common region. For example, if two adjacent image frames both capture the same mark on the scan rod, the electronic device uses the position information of this mark to determine the common region between the two adjacent image frames, calculates the relative position and angle between the two adjacent image frames based on the feature data of the common region, and stitches the two image frames into a larger three-dimensional model based on the relative position and angle between the two adjacent image frames. With real-time stitching of image frames, a three-dimensional model of the oral cavity can be reconstructed, as shown in Figure 8, for example.

[0077] However, in the event of a user scan error or a malfunction of the scanning device, some image frames may not contain the scan rod area. For example, a series of consecutive image frames might be image frame A (with the scan rod area), image frame B (without the scan rod area), image frame C (without the scan rod area), and image frame D (with the scan rod area). In this case, based on the feature information of the scan rod area, it is determined that there is no overlapping feature data between image frame A and image frame B, between image frame B and image frame C, and between image frame C and image frame D. At this point, the electronic device can issue a notification to the user indicating a stitching failure.

[0078] In some embodiments, the presentation method may include methods such as sound, a presentation box, and a color change of a scan box. The embodiments of the present application are not limited to the presentation method.

[0079] In some embodiments, simultaneously with or after presenting the user with a stitching failure, the electronic device may issue presentation information and instruct the user to return to the initial position and scan, and the initial position may be indicated on the interaction interface, where the initial position may include the scan position corresponding to the last image frame that was previously successfully stitched.

[0080] In some other embodiments of the present application, if stitching fails, i.e., if there is no overlapping feature data between adjacent image frames in a plurality of image frames, it is not necessary to issue presentation information. Instead, a common region is searched between the currently unstitched image frame and the remaining other image frames in the first image frame, and stitches the currently unstitched image frame and the remaining other image frames in the first image frame based on the common region, continuing until stitching is successful. Alternatively, a common region is searched between the currently unstitched image frame and the target three-dimensional data obtained by real-time stitching, and stitches the currently unstitched image frame and the target three-dimensional data obtained by real-time stitching based on the common region, continuing until stitching is successful.

[0081] In some embodiments, assuming that adjacent image frames include a first image frame and a second image frame, the electronic device can determine overlapping feature data between adjacent image frames in the following manner. If both adjacent image frames contain a pre-defined mark, the electronic device can obtain first geometric data of the pre-defined mark and adjacent marks in the first image frame, and second geometric data of the pre-defined mark and adjacent marks in the second image frame. Here, adjacent marks include other marks within a pre-defined area centered on the pre-defined mark. If the first and second geometric data satisfy the pre-defined conditions, the electronic device determines a pre-defined radius based on the first and second geometric data, and the feature data of the area enclosed by the pre-defined mark and the pre-defined radius in the first and second image frames is considered overlapping feature data.

[0082] In this embodiment, the pre-set marks include, but are not limited to, coded marks and geometric pattern marks. The pre-set area range can be customized based on the actual scan rod's feature information. The pre-set radius can be customized based on the first geometric data and the second geometric data.

[0083] In this embodiment, the first geometric data includes data such as the distance and relative angle between a pre-set mark and an adjacent mark in the first image frame. The second geometric data includes data such as the distance and relative angle between a pre-set mark and an adjacent mark in the second image frame. The pre-set conditions can be customized; for example, the pre-set condition can be set to be that the error between the first geometric data and the second geometric data is less than a pre-set error threshold. The pre-set error threshold can be customized.

[0084] As an example, suppose the first image frame contains marks A1 and A2, and the second image frame contains marks A2 and A3. Mark A2 in the first image frame and mark A2 in the second image frame are the same mark, and the electronic device can determine that a pre-set mark A2 exists in both the first and second image frames. Mark A1 is contained in the first image frame and is an adjacent mark to the pre-set mark A2. Assume that the pre-set error thresholds include a distance error threshold and an angle error threshold, setting the distance error threshold to 0.2 mm and the angle error threshold to 1°. The electronic device can calculate the distance d1 and angle θ1 between marks A1 and A2, for example, d1 = 5 mm and θ1 = 30°. Mark A3 is contained in the second image frame and is an adjacent mark to the pre-set mark A2. The electronic device can calculate the distance d2 and angle θ2 between marks A3 and A2, for example, d1 = 5.1 mm and θ1 = 29.5°. The electronic device calculates the error between the first geometric data and the second geometric data as |d1-d2|=0.1mm and |θ1-θ2|=0.5°. Based on the fact that the value of |d1-d2| is smaller than the distance error threshold and the value of |θ1-θ2| is smaller than the angle error threshold, the electronic device can determine that the first and second geometric data satisfy the pre-set conditions. At this point, the electronic device can determine that marks A1 and A2 in the first image frame and mark A3 in the second image frame belong to the same group of marks. Based on the distance d1 between the pre-set marks and adjacent marks in the first image frame and the distance d2 between the pre-set marks and adjacent marks in the second image frame, the electronic device can set a pre-set radius, for example, by setting the pre-set radius to 5mm. The electronic device considers the feature data of the region enclosed by the pre-set marks and the pre-set radius in the first and second image frames as duplicate feature data.

[0085] In some embodiments, the electronic device may use an image processing algorithm (e.g., an HSV color segmentation algorithm or a morphological filtering algorithm) to position the marks on the scan rod. The embodiments of this application are not limited to these.

[0086] In other embodiments, the electronic device may also determine overlapping feature data between adjacent image frames by other means. For example, overlapping feature data between adjacent image frames may be determined based on the geometric features of feature marks in the scan rod region of adjacent image frames. The embodiments of the present application are not limited thereto.

[0087] In some embodiments, after determining the overlapping feature data between adjacent image frames, the electronic device can stitch the adjacent image frames together based on the overlapping feature data. Specifically, the electronic device can calculate relative pose parameters between adjacent image frames, such as an optimal rotation matrix and a translation vector, based on the overlapping feature data. Based on the relative pose parameters, the electronic device maps the adjacent image frames to the same coordinate system, merges the point cloud data corresponding to the adjacent image frames, and stitches the adjacent image frames together.

[0088] In practical applications, some physicians need to display a holistic oral model, which allows them to quickly and accurately capture anatomical information within the oral cavity, such as the details of the gums and surrounding tissues, and further facilitates the optimization of implant and prosthetic design.

[0089] To display an overall oral cavity model, in some embodiments of the present application, the method further includes the following steps: Step S14: Obtain a second set of image frames of the oral cavity in a second state. Step S15: Based on the second set of image frames, obtain three-dimensional region data of the soft tissue areas in each image frame of the second set of image frames. Step S16: Obtain a third set of image frames, the third set of image frames includes some image frames from the first set of image frames, or updated image frames obtained when the oral cavity is in a first state. Step S17: Based on the feature information of the soft tissue areas and the feature information of the scan rod areas in each image frame of the third set of image frames, merge the three-dimensional region data corresponding to the soft tissue areas with the target three-dimensional data to obtain an oral cavity model.

[0090] Here, the second state of the oral cavity refers to a state in which no actual scan rod is inserted into the oral cavity, and the regional three-dimensional data includes three-dimensional point cloud data corresponding to the soft tissue region. In this embodiment, the regional three-dimensional data obtained by scanning without an actual scan rod being inserted into the oral cavity has higher scanning accuracy, and finally, based on the third set of image frames, a soft tissue model that is not interfered with by the scan rod and a scan rod model that is not interfered with by the gingiva can be combined to construct an overall oral cavity model, resulting in higher reconstruction accuracy.

[0091] In the embodiments of this application, there are, but are not limited to, various implementation methods for integrating the region 3D data corresponding to the soft tissue region with the target 3D data. One implementation method is to connect and merge the region 3D data and the target 3D data. Another implementation method is to unify the coordinate systems of the region 3D data and the target 3D data. To facilitate user viewing, there are, but are not limited to, various representation formats for the oral cavity model. One representation format is to display the region 3D data and target 3D data after connection and fusion in the interaction interface. Another representation format is to display the region 3D data and target 3D data superimposed under the unified coordinate system of the interaction interface, while maintaining the separation of these two data sets. Another representation format is two independent data files after the coordinate systems have been unified. Here, data integration (data merging) includes two forms: one is the data fusion method, and the other is the method of unifying the coordinate systems.

[0092] In some embodiments, steps S14 and S15 can be performed before step S11, and steps S16 and S17 can be performed after step S13, for example, the execution order of the steps is S14, S15, S11, S12, S13, S16 and S17. Specifically, the scanning device can be used to scan the soft tissue (e.g., gingiva) in the oral cavity in a second state, and the scanning device transmits the second image frames acquired by the scan to the electronic device. The electronic device acquires a second set of image frames of the oral cavity in the second state, and based on the second set of image frames, acquires three-dimensional data of the soft tissue region in each image frame of the second set of image frames. After the scanning of the soft tissue in the oral cavity in the second state is completed, the scanning device is used to scan the oral cavity in a first state, and the electronic device acquires a first set of image frames, and based on the first set of image frames, acquires target three-dimensional data corresponding to the actual scan rod in each image frame of the first set of image frames. After acquiring the target three-dimensional data, the scanning device is used to scan the oral cavity in the first state again, and a third set of image frames is acquired. Based on the feature information of the soft tissue region and the scan rod region in each image frame of the third image frame set, the electronic device merges the region 3D data corresponding to the soft tissue region with the target 3D data to obtain an overall oral model, as shown in Figure 9.

[0093] It should be explained that the third set of image frames in steps S16 and S17 includes some image frames from the first set of image frames, or updated image frames acquired when the oral cavity is in the first state, that is, they can be acquired by scanning again using a scanning device, or they may be historical image frames acquired in step S11.

[0094] Here, after acquiring the second set of image frames of the oral cavity in the second state, the overall oral cavity model can be displayed on the interaction interface in response to the user's switching command. A switching command to display gingival soft tissue can be provided on the interaction interface, and if selected by the user after steps S14 and S15, steps S16 and S17 described above can be executed to generate and display the overall oral cavity model, which is convenient for the user to view.

[0095] In some other embodiments of the present application, steps S14 and S15 may be performed before step S11, S16 may be performed in sync with any of steps S11, S12, or S13, or after any of steps S11, S12, or S13, and S17 may be performed after step S13. For example, any order of execution of steps being S14, S15, S11, S16, S12, S13, and S17, or any other reasonable order of execution of steps being S14, S15, S11, S12, S16, S13, and S17, is within the scope of protection of the present application.

[0096] In some other embodiments of the present application, steps S14 and S15 may be performed after step S13, S16 may be performed in sync with any of steps S11, S12, or S13, or after any of steps S11, S12, or S13, and S17 may be performed after step S13 or step S15. For example, any reasonable order of execution of steps, such as S11, S12, S13, S14, S15, S16 and S17, or S11, S16, S12, S13, S14, S15 and S17, or S11, S12, S16, S13, S14, S15 and S17, is within the scope of protection of the present application.

[0097] In some other embodiments of the present invention, it is not necessary to acquire the second set of image frames in steps S14 and S15 in response to a user switching command. Instead, the soft tissue regions identified in the first set of image frames and the target three-dimensional model can be stitched together to obtain an overall oral cavity model. In this case, since the target three-dimensional model acquired initially is not affected by gingival data, the accuracy of the final overall oral cavity model is also good.

[0098] In some embodiments of the present invention, the three-dimensional reconstruction method further includes the steps of: generating a design model based on target three-dimensional data and transmitting the design model to a 3D printing device; receiving a user request in real time to modify the design model; and outputting a modified three-dimensional design model based on the user request and the target three-dimensional model. In embodiments of the present invention, a modified three-dimensional design model can be output based on the user request and the target three-dimensional model, thereby allowing the user to easily set dentures of different quantities, dimensions, and shapes.

[0099] The embodiment of the present invention provides a three-dimensional reconstruction method in which, when an actual scan rod is placed in the oral cavity, a first set of image frames of the oral cavity is acquired, and the scan rod region is identified in each image frame of the first set of image frames. Based on the characteristic information of the scan rod region, multiple image frames of the first set of image frames are stitched together, and target three-dimensional data corresponding to the actual scan rod in the oral cavity is reconstructed, thereby realizing image frame stitching based on a rigid structure. Since the rigid structure of the scan rod is less prone to deformation during the scanning process and has good stability, stitching multiple image frames of the first set of image frames based on the characteristic information of the scan rod region contributes to improving stitching accuracy and further improves the overall accuracy of the three-dimensional reconstruction. In addition, by performing subsequent prosthetic design based on the target three-dimensional data, the success rate of implants, the fit between the prosthetic product and the patient, and the patient's comfort can be improved.

[0100] It should be understood that the order of the steps in the above-described embodiments does not indicate the order of execution, and the execution order of each process should be determined by its function and inherent logic, and does not imply any limitation on the implementation process of the embodiments of this application.

[0101] Referring to Figure 10, which is a structural diagram of a three-dimensional reconstruction apparatus provided in an embodiment of the present application, it can realize the details of the three-dimensional reconstruction method in the above-described embodiment and achieve the same effects. As shown in Figure 10, the three-dimensional reconstruction apparatus 10 can be applied to an electronic device having a data processing function, and the three-dimensional reconstruction apparatus 10 includes an acquisition module 11, an identification module 12, and a reconstruction module 13. The acquisition module 11 is for acquiring a first set of image frames of a first state oral cavity, where the first state oral cavity refers to a state in which at least one actual scan rod is installed in the oral cavity. The identification module 12 is for identifying the scan rod region in each image frame of the first set of image frames. The reconstruction module 13 stitches together multiple image frames of the first set of image frames based on the characteristic information of the scan rod region to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity.

[0102] In some embodiments, the identification module 12 is further configured to identify soft tissue regions in each image frame included in the first image frame set before reconstructing target three-dimensional data corresponding to the actual scan rod in the oral cavity. The reconstruction module 13 is further configured to either exclude soft tissue regions from the three-dimensional reconstruction of the target three-dimensional data, or to include them in the three-dimensional reconstruction of the target three-dimensional data based on pre-set weights.

[0103] In some embodiments, the reconstruction module 13 is further configured to perform one or more of the following operations on the soft tissue regions or scan rod regions in each image frame, based on the results of identifying feature information of soft tissue regions and / or scan rod regions in each image frame included in the first image frame set: removal, marking, hiding, or splitting. In some embodiments, the first image frame set includes texture image frames and depth image frames, and the identification module 12 is further configured to perform feature extraction and feature fusion on the texture image frames and depth image frames using a pre-configured identification model to obtain a mask image, the mask image including scan rod regions.

[0104] In some embodiments, the reconstruction module 13 is further configured to stitch together multiple image frames included in the first image frame set in real time based on feature information in the scan rod region, perform real-time three-dimensional reconstruction based on the stitched image frames, and acquire target three-dimensional data corresponding to the actual scan rod.

[0105] In some embodiments, the reconstruction module 13 is configured to stitch together multiple image frames included in the first image frame set in real time based on feature information of the scan rod region; perform three-dimensional reconstruction in real time based on the stitched image frames to acquire actual scan data corresponding to the actual scan rod in real time; match the standard data of the actual scan rod from the standard scan rod database based on feature information of the scan rod region, where this matching is a periodic matching during the real-time scanning process, and feature stitching of the current actual scan data and standard data each time a certain amount of actual scan data is acquired during the real-time scanning process, or at regular time intervals, to update the pose of the standard data as the scan data increases; and replace the scan data of the actual scan rod with the matched standard data of the actual scan rod to acquire the target three-dimensional data.

[0106] In some embodiments, the acquisition module 11 is further configured to acquire a second set of image frames of the oral cavity in a second state (where the oral cavity in a second state refers to a state in which no actual scan rod is inserted into the oral cavity); and based on the second set of image frames, it acquires three-dimensional regional data of the soft tissue region in each image frame included in the second set of image frames.

[0107] In some embodiments, the acquisition module 11 is further configured to acquire a third set of image frames (the third set of image frames includes some image frames from the first set of image frames, or updated image frames acquired when the oral cavity is in a first state); the reconstruction module 13 is further configured to acquire an oral cavity model by merging regional 3D data corresponding to soft tissue regions and target 3D data based on feature information of soft tissue regions and feature information of scan rod regions in each image frame included in the third set of image frames.

[0108] In some embodiments, the three-dimensional reconstruction apparatus 10 further comprises a display module, which is configured to display target three-dimensional data in real time on the interaction interface of an electronic device when acquiring a first set of image frames of the oral cavity in a first state based on feature information in the scan rod region, or to display an oral cavity model on the interaction interface in response to a user switching command after acquiring a second set of image frames of the oral cavity in a second state.

[0109] In some embodiments, the reconstruction module 13 is further configured to determine, based on feature information in the scan rod region, whether or not there is overlapping feature data between adjacent image frames in multiple image frames; if there is overlapping feature data between adjacent image frames in multiple image frames, to stitch the adjacent image frames in real time based on the overlapping feature data; and if there is no overlapping feature data between adjacent image frames in multiple image frames, to issue prompting information to notify the user of stitching failure and / or to issue prompting the user to return to the initial position and scan again.

[0110] In some embodiments, the three-dimensional reconstruction apparatus 10 further comprises a design module, which is configured to generate a design model based on target three-dimensional data and transmit it to a 3D printer, receive user requests in real time for modifications to the design model, and output a modified three-dimensional design model based on the user requests and the target three-dimensional model.

[0111] Specific limitations regarding the three-dimensional reconstruction apparatus 10 can be found in the limitations regarding the three-dimensional reconstruction method described above, and will be omitted again here. Each module in the three-dimensional reconstruction apparatus 10 described above can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules may be built into the processor of the electronic device in hardware form, or attached to the processor of the electronic device as an independent component, or stored in the memory of the electronic device in software form, and can be called by the processor to perform the operations corresponding to each of the above modules.

[0112] Embodiments of the present application further provide a computer-readable storage medium in which a computer program is stored, the computer program includes program instructions, and the method realized when the program instructions are executed can refer to the three-dimensional reconstruction method in each of the embodiments of the present application described above.

[0113] Here, the computer-readable storage medium may be the internal memory of the electronic device in the above-described embodiment, for example, the hard disk or memory of the electronic device. Alternatively, the computer-readable storage medium may be an external storage device of the electronic device, for example, a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard provided in the electronic device.

[0114] Furthermore, computer-readable storage media can primarily include program storage and data storage areas, where program storage can store the operating system, application programs necessary for at least one function, etc., and data storage can store data created through the use of electronic devices, etc.

[0115] Finally, it should be noted that the above embodiments are merely illustrative and not limiting to the technical solutions of the present application. While the present application has been described in detail with reference to better embodiments, those skilled in the art should understand that modifications or substitutions with equivalents of the technical solutions of the present application are possible and should fall within the scope of protection of the present application, provided that they do not deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A three-dimensional reconstruction method applicable to electronic devices, The aforementioned three-dimensional reconstruction method is A step of obtaining a first set of image frames of the oral cavity in a first state, wherein the oral cavity in the first state refers to a state in which at least one actual scan rod is inserted into the oral cavity. The steps include identifying the scan rod region in each image frame of the first image frame set, The steps include stitching together multiple image frames of the first image frame set based on the characteristic information of the scan rod region to reconstruct target three-dimensional data corresponding to the actual scan rod in the oral cavity, A three-dimensional reconstruction method characterized by including the following.

2. Before reconstructing the target three-dimensional data corresponding to the actual scan rod in the oral cavity, the three-dimensional reconstruction method further includes the step of identifying soft tissue regions in each image frame of the first image frame set, The soft tissue region either does not participate in the three-dimensional reconstruction of the target three-dimensional data, or participates in the three-dimensional reconstruction of the target three-dimensional data according to a pre-set weight. The three-dimensional reconstruction method according to feature 1.

3. The three-dimensional reconstruction method according to claim 1 or 2, further comprising the step of performing one or more of the following operations on the soft tissue region or the scan rod region in each image frame, based on identifying feature information of the soft tissue region and / or feature information of the scan rod region in each image frame of the first image frame set: removal operation, marking operation, concealment operation, or division operation.

4. The aforementioned first set of image frames includes texture image frames and depth image frames. The step of identifying the scan rod region in each image frame of the first image frame set includes the step of using a pre-configured identification model to perform feature extraction and feature fusion on the texture image frame and the depth image frame to obtain a mask image, the mask image including the scan rod region. The three-dimensional reconstruction method according to feature 3.

5. The step of stitching together multiple image frames of the first image frame set based on the characteristic information of the scan rod region and reconstructing target three-dimensional data corresponding to the actual scan rod in the oral cavity is as follows: The steps include stitching together multiple image frames of the first image frame set in real time based on the characteristic information of the scan rod region, The steps include: performing real-time three-dimensional reconstruction based on image frames stitched in real time to acquire target three-dimensional data corresponding to the actual scan rod; The three-dimensional reconstruction method according to claim 1, characterized by including the following:

6. The step of stitching together multiple image frames of the first image frame set based on the characteristic information of the scan rod region and reconstructing target three-dimensional data corresponding to the actual scan rod in the oral cavity includes the step of stitching together multiple image frames of the first image frame set in real time based on the characteristic information of the scan rod region, The steps include performing a three-dimensional reconstruction in real time based on the stitched image frames and acquiring the actual scan data corresponding to the actual scan rod in real time, A step of matching the standard data of the actual scan rod from a standard scan rod database based on the characteristic information of the scan rod region, wherein the matching is a periodic matching during the real-time scanning process, and during the real-time scanning process, each time a certain amount of actual scan data is acquired, or at regular intervals, the current actual scan data and the standard data are feature-stitched together, and the orientation of the standard data is updated as the scan data increases. The steps include: replacing the scan data of the actual scan rod with the standard data of the actual scan rod obtained by matching, and acquiring the target three-dimensional data; The three-dimensional reconstruction method according to claim 1, characterized by including the following:

7. The aforementioned three-dimensional reconstruction method further, A step of obtaining a second set of image frames of the oral cavity in a second state, wherein the oral cavity in the second state refers to a state in which no actual scan rod is inserted into the oral cavity. The steps include: obtaining three-dimensional data of the soft tissue region in each image frame of the second image frame set based on the second image frame set; A step of obtaining a third set of image frames, wherein the third set of image frames includes a portion of the image frames from the first set of image frames, or updated image frames obtained when the oral cavity is in the first state, Based on the feature information of the soft tissue region and the feature information of the scan rod region in each image frame of the third image frame set, the three-dimensional data of the region corresponding to the soft tissue region and the target three-dimensional data are merged to obtain an oral cavity model. The three-dimensional reconstruction method according to claim 1, characterized by including the following:

8. The aforementioned three-dimensional reconstruction method further, When acquiring a first set of image frames of the oral cavity in the first state based on the characteristic information of the scan rod region, the steps include displaying the target three-dimensional data in real time on the interaction interface of the electronic device, or After obtaining a second set of image frames of the oral cavity in the second state, the step of displaying the oral cavity model on the interaction interface in response to a user switching command, The three-dimensional reconstruction method according to claim 7, characterized by including the following:

9. The step of stitching together a plurality of image frames of the first image frame set based on the characteristic information of the scan rod region is as follows: The steps include determining whether or not there is overlapping feature data between adjacent image frames in the plurality of image frames based on the feature information of the scan rod region, If overlapping feature data exists between adjacent image frames in the plurality of image frames, the step of stitching the adjacent image frames in real time based on the overlapping feature data, If there is no overlapping feature data between adjacent image frames in the plurality of image frames, the step of issuing a notification to the user to indicate stitching failure and / or issuing a notification to the user to return to the initial position and scan again, The three-dimensional reconstruction method according to claim 1, characterized by including the following:

10. The aforementioned three-dimensional reconstruction method further, The steps include generating a design model based on target three-dimensional data and transmitting the design model to a 3D printing device, Steps include receiving user requests in real time to modify the design model, The steps include: outputting a modified 3D design model based on the user's request and target 3D model; The three-dimensional reconstruction method according to claim 1, characterized by including the following:

11. A three-dimensional reconstruction device applied to electronic equipment, The three-dimensional reconstruction apparatus includes an acquisition module, an identification module, and a reconstruction module, The acquisition module is a module for acquiring a first set of image frames of the oral cavity in a first state, wherein the oral cavity in a first state refers to a state in which at least one actual scan rod is inserted into the oral cavity. The identification module is a module for identifying the scan rod region in each image frame of the first image frame set, The reconstruction module is a module for reconstructing target three-dimensional data corresponding to the actual scan rod in the oral cavity by stitching together multiple image frames of the first image frame set based on the characteristic information of the scan rod region. A three-dimensional reconstruction apparatus characterized by the following features.

12. It is an electronic device, Including memory, processor, and computer-readable instructions, An electronic device characterized in that, when the computer-readable instruction is executed by the processor, the three-dimensional reconstruction method described in claim 1 or 2 is realized.

13. A computer-readable storage medium, Computer-readable instructions are stored in the aforementioned computer-readable storage medium. A computer-readable storage medium characterized in that, when the computer-readable instruction is executed by a processor, the three-dimensional reconstruction method described in claim 1 or 2 is realized.

Images