3D Reconstruction Method, Device, Electronic Device, and Storage Medium

Abstract

The present application relates to the field of three-dimensional reconstruction technology, and provides a three-dimensional reconstruction method, apparatus, electronic device, and storage medium. The method includes: obtaining a first set of image frames of the oral cavity in a first state, where the oral cavity in the first state means that at least one actual scanning rod is installed in the oral cavity; identifying the scanning rod regions of each image frame in the first set of image frames; and based on the feature information in the scanning rod regions, splicing a plurality of image frames in the first set of image frames and reconstructing the target three-dimensional data corresponding to the actual scanning rod in the oral cavity. This method performs splicing based on a rigid structure, which can reduce the influence of anatomical structures (such as soft tissues like gums, lips, and tongue) in the image frames on the splicing accuracy, thereby improving the overall accuracy of three-dimensional reconstruction.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular, to a three-dimensional reconstruction method, apparatus, electronic device, and storage medium. Background Art

[0002] During oral scanning, multiple image frames need to be stitched and aligned through a common area (or transition area) to reconstruct a complete three-dimensional model. In related technologies, in scenarios such as edentulous implant and large-span implant, soft tissue areas such as gums are usually selected as the transition area to stitch data from different areas or different image frames. However, due to the lack of rigid support in soft tissues such as gums, they are prone to deformation due to external forces (such as opening movements, instrument contact) or muscle traction (such as grinning), and the patient's physiological activities (such as breathing, saliva secretion, etc.) are also likely to cause slight displacement of soft tissues, resulting in changes in the morphology of soft tissue areas. In this case, if a soft tissue area is selected as the transition area, it will cause coordinate offsets in the transition area of the scanned data, affecting the stitching accuracy of adjacent areas or adjacent frame data, thereby reducing the accuracy of three-dimensional reconstruction. Finally, when a prosthetic denture is designed by computer-aided design (CAD) based on the deformed gum data, the prosthesis may compress the gums or create gaps when placed in the mouth. If the deformation exceeds a certain limit, it will cause discomfort to the patient in the light case, and may even lead to implant failure in the severe case. Summary of the Invention

[0003] In view of this, this application provides a three-dimensional reconstruction method, apparatus, electronic device, and storage medium to solve the problem of large stitching errors, which in turn affects the accuracy of three-dimensional reconstruction.

[0004] In the first aspect of the embodiments of this application, a three-dimensional reconstruction method is provided, which is applied to an electronic device. The three-dimensional reconstruction method includes: obtaining a first set of image frames of the oral cavity in a first state, where the oral cavity in the first state means that at least one actual scanning rod is installed in the oral cavity; identifying the scanning rod areas of each image frame in the first set of image frames; and based on the feature information in the scanning rod areas, stitching multiple image frames in the first set of image frames and reconstructing to obtain target three-dimensional data corresponding to the actual scanning rods in the oral cavity.

[0005] In some embodiments, before reconstructing to obtain the target three-dimensional data corresponding to the actual scanning rods in the oral cavity, the method further includes: identifying the soft tissue areas of each image frame in the first set of image frames; the soft tissue areas do not participate in the three-dimensional reconstruction of the target three-dimensional data or participate in the three-dimensional reconstruction of the target three-dimensional data according to a preset weight.

[0006] In some embodiments, the method further includes: performing one or more of a removal operation, a marking operation, a hiding operation, or a segmentation operation on the soft tissue region or the scanning rod region in each image frame based on the feature information of the soft tissue region and / or the feature information of the scanning rod region in each image frame of the first image frame set.

[0007] In some embodiments, the first image frame set includes a texture image frame and a depth image frame. Identifying the scanning rod region in each image frame of the first image frame set includes: performing feature extraction and feature fusion on the texture image frame and the depth image frame using a preset recognition model to obtain a mask image, where the mask image includes the scanning rod region.

[0008] In some embodiments, based on the feature information in the scanning rod region, stitching multiple image frames in the first image frame set and reconstructing to obtain target three-dimensional data corresponding to the actual scanning rod in the oral cavity includes: performing real-time stitching on multiple image frames in the first image frame set based on the feature information in the scanning rod region; performing real-time three-dimensional reconstruction on the stitched image frames to obtain the target three-dimensional data corresponding to the actual scanning rod.

[0009] In some embodiments, based on the feature information in the scanning rod region, stitching multiple image frames in the first image frame set and reconstructing to obtain target three-dimensional data corresponding to the actual scanning rod in the oral cavity includes: performing real-time stitching on multiple image frames in the first image frame set based on the feature information of the scanning rod region; performing real-time three-dimensional reconstruction on the stitched image frames to obtain real-time scanning data corresponding to the actual scanning rod in real time; based on the feature information of the scanning rod region, matching the standard data of the actual scanning rod from a standard scanning rod database, where the matching is a periodic matching during the real-time scanning process, and during the real-time scanning process, every time a certain number of actual scanning data is obtained or every time a certain period of time elapses, the current actual scanning data is feature-stitched with the standard data, and as the scanning data increases, updating the pose of the standard data; using the matched standard data of the actual scanning rod to replace the scanning data of the actual scanning rod to obtain the target three-dimensional data.

[0010] In some embodiments, the method further includes: obtaining a second set of image frames of the oral cavity in a second state, where the oral cavity in the second state indicates that no actual scanning rod is installed in the oral cavity; based on the second set of image frames, obtaining regional three-dimensional data of the soft tissue region in each image frame of the second set of image frames; obtaining a third set of image frames, where the third set of image frames includes some image frames of the first set of image frames or updated image frames obtained when the oral cavity is in the first state; based on the feature information of the soft tissue region and the feature information of the scanning rod region in each image frame of the third set of image frames, merging the regional three-dimensional data corresponding to the soft tissue region with the target three-dimensional data to obtain an oral cavity model.

[0011] In some embodiments, the method further includes: based on the feature information in the scanning rod region, displaying the target three-dimensional data in real time on the interaction interface of the electronic device when obtaining the first set of image frames of the oral cavity in the first state; or after obtaining the second set of image frames of the oral cavity in the second state, displaying the oral cavity model on the interaction interface in response to a user switching instruction.

[0012] In some embodiments, the splicing of multiple image frames in the first set of image frames based on the feature information in the scanning rod region includes: based on the feature information in the scanning rod region, determining whether there is overlapping feature data between adjacent image frames in the multiple image frames; if there is overlapping feature data between adjacent image frames in the multiple image frames, then based on the overlapping feature data, performing real-time splicing of the adjacent image frames; if there is no overlapping feature data between adjacent image frames in the multiple image frames, then sending a prompt message to prompt the user that the splicing fails, and / or sending a prompt message to prompt the user to return to the initial position for scanning.

[0013] In some embodiments, the method further includes: generating a design model according to the target three-dimensional data and sending the design model to a 3D printing device; receiving in real time a user request for modifying the design model; and outputting a revised three-dimensional design model according to the user request and the target three-dimensional model.

[0014] A second aspect of the embodiments of the present application provides a three-dimensional reconstruction device, which is applied to an electronic device. The three-dimensional reconstruction device includes: an acquisition module, configured to acquire a first set of image frames of the oral cavity in a first state, where the oral cavity in the first state indicates that at least one actual scanning rod is installed in the oral cavity; an identification module, configured to identify the scanning rod regions of the respective image frames in the first set of image frames; and a reconstruction module, configured to splice multiple image frames in the first set of image frames based on the feature information in the scanning rod region and reconstruct to obtain target three-dimensional data corresponding to the actual scanning rod in the oral cavity.

[0015] In a third aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the above-mentioned three-dimensional reconstruction method is implemented.

[0016] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the above-mentioned three-dimensional reconstruction method is implemented.

[0017] The embodiments of the present application provide a three-dimensional reconstruction method. In the case of installing an actual scanning rod in the oral cavity, a first set of image frames of the oral cavity is obtained, and the scanning rod regions in each image frame of the first set of image frames are identified. Based on the feature information of the scanning rod regions, multiple image frames in the first set of image frames are spliced to reconstruct the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, realizing image frame stitching based on a rigid structure. Since the rigid structure of the scanning rod is not easily deformed during the scanning process and has good stability, splicing multiple image frames in the first set of image frames based on the feature information of the scanning rod regions helps to improve the splicing accuracy, and further can improve the overall accuracy of three-dimensional reconstruction. Moreover, based on the target three-dimensional data for subsequent repair design can improve the success rate of implantation, improve the matching degree between the repair product and the patient, and the patient comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a device diagram of the three-dimensional reconstruction method provided by the embodiments of the present application.

[0020] Figure 2 It is a flowchart of the implementation of the three-dimensional reconstruction method provided by the embodiments of the present application.

[0021] Figure 3 It is an example diagram of an interaction interface provided by the first embodiment of the present application.

[0022] Figure 4 It is an example diagram of an interaction interface provided by the second embodiment of the present application.

[0023] Figure 5 It is an example diagram of the scanning rod region in an image frame provided by the first embodiment of the present application.

[0024] Figure 6 It is an exemplary diagram of the scanning rod area in the image frame provided by the second embodiment of the present application.

[0025] Figure 7 It is an exemplary diagram of the scanning rod area in the image frame provided by the third embodiment of the present application.

[0026] Figure 8 It is an exemplary diagram of the target three-dimensional data provided by the embodiment of the present application.

[0027] Figure 9 It is an exemplary diagram of the oral cavity model provided by the embodiment of the present application.

[0028] Figure 10 It is a schematic structural diagram of the three-dimensional reconstruction device provided by the embodiment of the present application. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0030] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" is intended to present related concepts in a specific manner.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that unless otherwise stated in this application, " / " means "or". For example, A / B may mean A or B. The "and / or" in this application is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone, these three situations. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b, or c may mean: a, b, c, a and b, a and c, b and c, a, b, and c, these seven situations.

[0032] Please refer to Figure 1 , which is a device diagram of a three-dimensional reconstruction method provided by an embodiment of this application. As Figure 1 shown, the three-dimensional reconstruction method provided by the embodiment of this application can be applied to the electronic device 100. The electronic device 100 may include one or more of devices such as mobile phones, tablet computers, smart wearable devices, laptop computers, scanning devices (such as oral digital impression devices), etc. The embodiment of this application does not impose any restrictions on the specific type of the electronic device.

[0033] As Figure 1 shown, the electronic device 100 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is respectively coupled to the communication module 101, the memory 102, and the I / O interface 104 through the bus 105.

[0034] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more of wired communication solutions such as a universal serial bus (USB), a controller area network bus (CAN). The wireless communication module may provide one or more of wireless communication solutions such as wireless fidelity (Wi-Fi), Bluetooth (BT), a mobile communication network, frequency modulation (FM), near field communication (NFC), and infrared technology (IR).

[0035] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store the operating system or executable programs of other running programs (such as machine instructions), and can also be used to store user and application data, etc. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0036] The non-volatile memory can also store executable programs and store user and application data, etc., and can be pre-loaded into the random access memory for direct reading and writing by the processor 103. The non-volatile memory may include disk storage devices, flash memory.

[0037] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 103, a three-dimensional reconstruction method executable on the electronic device 100 can be implemented.

[0038] In other embodiments, the electronic device 100 further includes an external memory interface for connecting to an external memory to implement the expansion of the storage capacity of the electronic device 100.

[0039] The processor 103 may include one or more processing units. For example, the processor 103 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0040] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute the computer program stored in the memory 102 to implement the above-mentioned 3D reconstruction method.

[0041] The I / O interface 104 is used to provide channels for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information or visualize the information. In addition, the I / O interface 104 can also be used to connect various sensor devices, such as a radar sensor, an image sensor, etc., so as to obtain data such as required radar point cloud data and image data.

[0042] The bus 105 is at least used to provide a communication channel for mutual communication between the communication module 101, the memory 102, the processor 103, and the I / O interface 104 in the electronic device 100.

[0043] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0044] Figure 1 The scene shown is only a schematic example. The 3D reconstruction method provided by the present application can also be applied to other scenes. For example, in some scenes, it may include the electronic device 100 and a scanning device. The electronic device 100 and the scanning device are separately arranged, and the electronic device is a computer, a tablet computer, a mobile phone, etc.; in some scenes, the electronic device 100 can be arranged inside the scanning device. In some scenes, other types of devices may also be included. In summary, the embodiments of the present application do not limit the specific application scenarios of the 3D reconstruction method.

[0045] Please refer to Figure 2 as shown, which is a flowchart of the implementation of the three-dimensional reconstruction method provided by the embodiments of the present application. This method is applied to an electronic device. In the embodiments of the present application, a computer program product of this method is run on (for example Figure 1 the electronic device 100 in) as an example for illustration. This method includes the following steps.

[0046] S11: Obtain a first set of image frames of the oral cavity in a first state.

[0047] In some embodiments, the oral cavity in the first state means that at least one actual scanning rod is installed in the oral cavity. The actual scanning rod can be installed on an implant or abutment in the oral cavity. The oral cavity can be the oral cavity of a patient with tooth loss, such as the oral cavity of a completely edentulous patient or a partially edentulous patient. Therefore, the user can use scanning devices such as an intraoral scanner or an extraoral scanner to scan the oral cavity in the first state to obtain the first set of image frames.

[0048] In some embodiments, specific geometric patterns and / or marks, etc. are provided on the surface of the actual scanning rod. These geometric patterns or marks can be accurately captured by the scanning device during scanning and are used to provide reference points to help the scanning device identify and locate the scanning rod at multiple angles or positions. For example, the geometric pattern can be a high-contrast pattern, a regular geometric shape (such as a circle, a square), an irregular geometric shape (such as a groove, a protrusion), and the mark can be an unencoded mark or an encoded mark, such as an unencoded fiducial point or an encoded fiducial point. Among them, the specific geometric patterns and / or marks on the actual scanning rod can be evenly distributed or unevenly distributed.

[0049] In some embodiments, the first set of image frames contains multiple image frames. An image frame represents a single two-dimensional image captured by the scanning device at a specified time point during the scanning of the oral cavity. In the case of scanning the oral cavity in the first state, each image frame in the first set of image frames includes the scanning rod and surrounding tissues (such as soft tissues such as teeth, gums, and mucosa) in the oral cavity.

[0050] S12: Identify the scanning rod regions of each image frame in the first set of image frames.

[0051] In some embodiments, the scanning rod region means a region that contains local information of the actual scanning rod in the image frame.

[0052] In some embodiments of the present application, the first set of image frames includes texture image frames and depth image frames. Identifying the scanning rod regions of each image frame in the first set of image frames includes: performing feature extraction and feature fusion on the texture image frames and depth image frames using a preset recognition model to obtain a mask image, and the mask image includes the scanning rod regions.

[0053] In some embodiments, the texture image frame represents a two-dimensional color image of the oral cavity interior, which is used to reflect the surface visual characteristics of tissues such as teeth, gums, and scanning rods, including details such as color, texture, spots, cracks, restorations (such as fillings or crowns), surface patterns, unencoded fiducial points, or encoded fiducial points. The depth image frame represents the two-dimensional spatial information of the oral cavity structure. Each pixel value in the depth image frame can represent the distance (depth value) from the corresponding position to the scanning device, including details such as the geometry, grooves, and protrusions on the scanning rod, usually represented in grayscale or pseudo-color (for example, bright for near and dark for far).

[0054] In some embodiments, the electronic device can use optical three-dimensional scanning technologies (such as structured light technology, laser triangulation technology, binocular stereo vision technology, etc.) to obtain the depth image frame. For example: the texture image frame can be obtained by the scanning device projecting white light onto the object to be scanned through a color camera, and the depth image frame can be obtained by the scanning device projecting structured light onto the object to be scanned through a black-and-white camera. The embodiments of the present application do not limit the specific manner of obtaining the depth image frame.

[0055] In some embodiments, the preset recognition model can be a deep learning model, such as a convolutional network model, a recurrent network model, a generative adversarial network model, a semantic segmentation model, etc.

[0056] In some embodiments, the preset recognition model includes multiple convolutional layers and multiple deconvolutional layers. The electronic device uses the preset recognition model to perform feature extraction and information fusion on the current frame image and the depth image, and outputs a mask image, including: using multiple convolutional layers to perform feature extraction and feature fusion on the current frame image and the depth image to obtain a feature map; using multiple deconvolutional layers to perform upsampling restoration processing on the feature map to obtain a mask image.

[0057] In some embodiments, the electronic device uses multiple deconvolutional layers to perform upsampling restoration processing on the feature map to obtain a mask image, including performing a transposed convolution operation on the feature map for upsampling, splicing the shallow features of the corresponding layer of the encoder and the upsampled deep features by channels, and combining the detailed and semantic information. Repeating the above steps, when the resolution of the feature map is restored to the input size, using an activation function to predict the class probability of each pixel in the feature map to obtain a mask image.

[0058] S13: Based on the feature information in the scanning rod area, splice multiple image frames in the first image frame set, and reconstruct to obtain the target three-dimensional data corresponding to the actual scanning rod in the oral cavity.

[0059] In some embodiments, the feature information in the scanning rod area includes but is not limited to geometric features (such as point cloud data, edges and contours, preset key points, etc.), information on preset marks (such as geometric patterns, marks) on the actual scanning rod.

[0060] In some embodiments, the target three-dimensional data may include three-dimensional point cloud data and three-dimensional models corresponding to the actual scanning rod. The target three-dimensional data includes the shape information of the scanning rod and the positioning information of the scanning rod. Therefore, based on the feature information of the scanning rod area, stitching multiple image frames in the first image frame set helps to improve the stitching accuracy, and further can improve the overall accuracy of three-dimensional reconstruction, obtain more accurate positioning information of the scanning rod, and perform subsequent repair designs (such as crown, bridge, denture, etc.) based on the target three-dimensional data, improve the success rate of implantation, and improve the matching degree between the repair product and the patient as well as the patient comfort.

[0061] In some embodiments of the present application, before reconstructing the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, the method further includes: identifying the soft tissue area of each image frame in the first image frame set; the soft tissue area does not participate in three-dimensional reconstruction or participates in the three-dimensional reconstruction of the target three-dimensional data according to a preset weight.

[0062] Among them, the soft tissue includes one or more of soft tissue objects such as the tongue, gums, and lips in the oral cavity area. The soft tissue area represents the area where the soft tissue in the image frame is located. The preset weight can be a weight input by the user, or a preset model (such as an AI model) can be used to calculate the preset weight based on the edentulous situation and age situation of the user; and / or use the preset model to analyze the quality of the image frames scanned by the scanning device (such as the image frames in the first image frame set or the second image frame set) to determine the preset weight; and / or use the preset model to determine the preset weight according to the stitching error or stitching success rate of the target three-dimensional data obtained by real-time stitching.

[0063] In some embodiments, the preset model may be a deep learning model, such as a convolutional network model, etc. The embodiments of the present application do not limit the preset model.

[0064] In some embodiments, the electronic device may pre-train a deep learning model based on the feature data of soft tissue objects such as the tongue, gums, and lips, such as the shape, edge features, and color features of the gums, for identifying the soft tissue area of each image frame in the first image frame set. In other embodiments, the electronic device may also identify the soft tissue area of each image frame in the first image frame set by other means. The embodiments of the present application do not limit this.

[0065] It should be noted that in one embodiment, only the scanning rod area may be recognized, and the areas other than the scanning rod in the image frames in the first image frame set are removed, and only the scanning rod area is reconstructed; in one embodiment, semantic recognition may be performed on each part of the image frames in the first image frame set to distinguish between soft tissue and the scanning rod, and it is determined that the soft tissue area does not participate in the reconstruction. In one embodiment, semantic recognition may be performed on each part of the image frames in the first image frame set to distinguish the soft tissue from the scanning rod in the image frames, so as to determine the soft tissue area and participate in the reconstruction according to a preset weight to avoid splicing loss. In one embodiment, the soft tissue area may be further distinguished. The soft tissue may be further distinguished into the tongue, gums, and lips. The tongue and lips are removed as miscellaneous data, and the gum data may be retained for subsequent operations.

[0066] Specifically, semantic recognition is performed on each part of the image frames in the first image frame set to distinguish whether it is the tongue, gums, lips or the scanning rod. For example, if it is the tongue and lips, they are deleted; if it is the scanning rod, a weight of 80% is assigned; if it is the gums, a weight of 20% is assigned. When splicing multiple image frames, it is determined whether there is overlapping feature data between adjacent image frames among the multiple image frames; if there is overlapping feature data between adjacent image frames among the multiple image frames, the adjacent image frames are spliced in real time based on the overlapping feature data. Among them, the electronic device may judge whether there is overlapping feature data between adjacent image frames among the multiple image frames according to the weight of the scanning rod (such as 80%) and the weight of the gums (such as 20%). Another example is that the electronic device may set different weights according to different classifications of the semantic recognition results for splicing. Specifically, the electronic device may perform semantic recognition on each part of the image frames in the first image frame set to distinguish whether it is the tongue, gums, lips or the scanning rod, and assign weights to the scanned data based on the semantic recognition results. For example, the initial value of the scanned data in each image frame is set to 1. If the semantic recognition result is the tongue, the weight is reduced based on the initial value of the scanned data to obtain the corresponding weight when the semantic recognition result is the tongue. For example, it may be reduced by 80%, so the corresponding weight of the tongue is 20%; if it is the lips, the weight is reduced based on the initial value of the scanned data to obtain the corresponding weight when the semantic recognition result is the lips. For example, it is reduced by 80%, so the corresponding weight of the lips is 20%; if the semantic recognition result is the scanning rod, the weight is increased based on the initial value of the scanned data to obtain the corresponding weight when the semantic recognition result is the scanning rod. For example, the weight is increased by 100%; if the semantic recognition result is the gums, the weight is increased based on the initial value of the scanned data to obtain the corresponding weight when the semantic recognition result is the gums. For example, the weight is increased by 20%. The electronic device splices and reconstructs the scanned data whose weights meet the preset threshold range according to all the scanned data after multiple verification calculations to obtain the target three-dimensional data.

[0067] In some embodiments of the present application, the electronic device may perform one or more of a removal operation, a marking operation, a hiding operation, or a segmentation operation on the soft tissue region or the scanning rod region in each image frame based on identifying the feature information of the soft tissue region and / or the feature information of the scanning rod region in each image frame of the first image frame set.

[0068] It should be noted that the removal operation may be deletion, the marking may be marking with different colors, the segmentation may be determining the boundary for segmentation, and the hiding may be hiding the pre-selected area. The marking operation, the hiding operation, or the segmentation operation does not affect the integrity of the scan data. If necessary, all scan data can be restored and displayed by setting.

[0069] In some embodiments, the feature information of the soft tissue region may include the geometric shape of the soft tissue (such as the wavy shape of the gingival margin, the contour of the tongue coating, etc.), color and texture features, three-dimensional structural features (such as topological structure, spatial deformation features, etc.), functional and physiological state features, etc. In the case where a scanning rod is installed in the oral cavity, the feature information of the soft tissue region may further include the interaction features between the soft tissue and the scanning rod, such as the contact area between the soft tissue and the scanning rod, the pressure distribution, etc. The specific content of the feature information of the soft tissue region in the embodiments of the present application is not limited.

[0070] In the prior art, three-dimensional reconstruction can be performed by combining a scanning device (such as an oral digital impression device / intraoral scanner, extraoral scanner) with a scanning rod to restore the anatomical morphology of the patient's oral cavity (such as dentition, gingiva, occlusion relationship) for digital design of restorations or implants, etc. However, when the scanning rod is installed in the patient's oral cavity, during the scanning process, the complex anatomical structures (such as soft tissues such as gingiva, lips, and tongue) around the scanning rod are prone to data overlap or interference with the scanning rod region, resulting in misalignment of the common area during image frame stitching, thereby causing error accumulation frame by frame, and finally manifested as distortion or even stitching breakage of the three-dimensional model, which in turn affects the accuracy and clinical adaptability of subsequent restoration design.

[0071] For example Figure 3 As shown, the window of the scanning device in the oral cavity is shown in the upper left of the figure, and the reconstructed three-dimensional model is shown in the lower right of the figure. As Figure 3 shown, the reconstructed three-dimensional model contains a large area of soft tissue region (such as the Figure 3 area boxed in the figure), and the soft tissue region is prone to deformation and is likely to have data overlap or interference with the scanning rod region.

[0072] To solve the above problems, the electronic device may perform one or more of the removal operation, marking operation, hiding operation, or segmentation operation on the soft tissue region in each image frame based on the feature information of the soft tissue region and / or the feature information of the scanning rod region in each image frame of the first image frame set, so that the electronic device can perform stitching of multiple image frames only based on the feature information of the scanning rod region, improve the stitching accuracy, and further improve the accuracy of three-dimensional reconstruction. For example Figure 4 As shown, after performing the removal operation on the soft tissue regions in each image frame and stitching and three-dimensional reconstructing the image frames after the removal operation, the target three-dimensional data (three-dimensional model) (such as Figure 4 shown in the lower right figure) does not display the soft tissue region.

[0073] In some embodiments of the present application, based on the feature information in the scanning rod region, multiple image frames in the first image frame set are stitched and reconstructed to obtain the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, including: performing real-time stitching on multiple image frames in the first image frame set based on the feature information in the scanning rod region; performing real-time three-dimensional reconstruction on the image frames after real-time stitching to obtain the target three-dimensional data corresponding to the actual scanning rod. Among them, when obtaining the first image frame set of the oral cavity in the first state, the target three-dimensional data is displayed in real time on the interaction interface of the electronic device. Therefore, the recognition of the scanning rod region or the soft tissue region of each image frame in the first image frame set in this embodiment is also a real-time operation, so that real-time scanning, real-time stitching, and reconstruction can be realized. During the scanning process, as the user moves in the oral cavity, it gradually changes from Figure 4 the target three-dimensional data shown in the lower right figure in the middle to obtain the target three-dimensional data as shown in Figure 8 shown, and at the same time, it is displayed in real time on the interaction interface, which is convenient for the user to understand the scanning progress.

[0074] In some embodiments of the present application, based on the feature information in the scanning rod region, multiple image frames in the first image frame set are stitched and reconstructed to obtain the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, including: performing real-time stitching on multiple image frames in the first image frame set based on the feature information of the scanning rod region; performing real-time three-dimensional reconstruction on the stitched image frames to obtain the actual scanning data corresponding to the actual scanning rod in real time; based on the feature information of the scanning rod region, matching the standard data of the actual scanning rod from the standard scanning rod database; using the matched standard data of the actual scanning rod to replace the scanning data of the actual scanning rod to obtain the target three-dimensional data.

[0075] Among them, the standard scanning rod database stores the standard data of various types of scanning rods. The standard data represents a set of benchmark parameters of the scanning rod in an ideal state, complete, without missing, and without errors. For example, the designed CAD model of the scanning rod.

[0076] Therefore, in this embodiment, the target 3D data represents the structure of the standard scanning rod and the pose of the actual scanning rod.

[0077] In some embodiments, during the process of real-time stitching of multiple image frames in the first image frame set by the electronic device based on the feature information of the scanning rod area, the actual scanning data corresponding to the actual scanning rod will be obtained in real time. However, due to reasons such as occlusion during the scanning process or scanning device failure, the actual scanning data may be missing, which may affect the accuracy of subsequent image frame stitching and thus the accuracy of the target 3D data. To solve the above problems, the electronic device matches the standard data of the actual scanning rod from the standard scanning rod database based on the feature information of the scanning rod area, and uses the matched standard data of the actual scanning rod to replace the scanning data of the actual scanning rod to obtain the target 3D data. By replacing the actual scanning data of the actual scanning rod with the standard data of the actual scanning rod, it is ensured that whether the scanning data of the actual scanning rod is complete or partial, it can be replaced with the complete standard data of the actual scanning rod, which helps to improve the accuracy of the reconstructed target 3D data.

[0078] In some embodiments, during the process of matching the standard data of the actual scanning rod from the standard scanning rod database, the matching represents periodic matching during the real-time scanning process. During the real-time scanning process, every time a certain number of actual scanning data are obtained or every certain period of time, the current actual scanning data is stitched with the standard data in terms of features. As the scanning data increases, the pose of the standard data of the actual scanning rod that has been stitched with features is updated. By updating the pose of the standard data of the actual scanning rod that has been stitched with features as the scanning data increases, the stitching accuracy and success rate of the standard data of the actual scanning rod and the scanning data of the actual scanning rod can be improved, and regardless of the number of scanning rods, whether it is one actual scanning rod or multiple actual scanning rods, feature stitching and matching can be performed. Among them, the scanning data of the target scanning rod in the current scanning data is stitched and matched with the standard data, and coordinate transformation will occur to the standard data during the feature stitching process.

[0079] In some embodiments, matching the standard data of the actual scanning rod from the standard scanning rod database based on the feature information of the scanning rod area includes: determining the identification information of the actual scanning rod based on the feature information of the scanning rod area; and matching the standard data of the actual scanning rod from the standard scanning rod database based on the identification information.

[0080] Among them, the standard scanning rod database stores standard data of various types of scanning rods. The standard data of each type of scanning rod corresponds to identification information such as a model number. The electronic device can determine the model number of the actual scanning rod in the scanning rod area according to the characteristic information of the scanning rod area, and based on the model number of the actual scanning rod, retrieve the corresponding standard model of each actual scanning rod from the standard scanning rod database, and then form target three-dimensional data based on the corresponding standard model of each actual scanning rod and the pose of each scanning rod in the scanning data.

[0081] In some embodiments, the electronic device can load the standard scanning rod database into the cloud server and download the standard data of the actual scanning rod from the cloud server.

[0082] In some embodiments of the present application, based on the characteristic information in the scanning rod area, stitching multiple image frames in the first image frame set includes: based on the characteristic information in the scanning rod area, determining whether there is overlapping feature data between adjacent image frames in the multiple image frames; if there is overlapping feature data between adjacent image frames in the multiple image frames, then based on the overlapping feature data, perform real-time stitching of the adjacent image frames; if there is no overlapping feature data between adjacent image frames in the multiple image frames, then send a prompt message to prompt the user that the stitching fails.

[0083] Among them, after all the multiple image frames in the first image frame set are stitched, global optimization can be performed on the target three-dimensional data.

[0084] In some embodiments, the overlapping feature data represents feature data that covers the same area (common area) between different image frames. The overlapping feature data can include geometric overlapping features, texture overlapping features, semantic overlapping features, etc.

[0085] Taking the example of using an oral digital scanner to perform oral scanning on an oral cavity in a state to obtain a first image frame set, when the oral scanner moves in the patient's oral cavity, multiple images or point cloud data will be continuously captured. The geometric features on the scanning rod exist as fixed reference points in these images. Each scan (single frame) will capture some features of the scanning rod, and these features form a common area between different image frames, that is, the data of the same part of the features is included in multiple image frames. For example Figures 5 to 7 shown, Figure 5 the image frame shown and Figure 6 the image frame shown have a common area, and the common area is as shown in Figure 7 the red area in.

[0086] If each image frame contains the feature information of the scanning rod area, such as some geometric features on the scanning rod, the electronic device can determine the common area between different image frames through the feature information of the scanning rod, and align and splice different image frames according to the features of the common area. For example, if the same mark on the scanning rod is captured in two adjacent image frames, the electronic device can use the position information of this mark to determine the common area between the two adjacent image frames, calculate the relative position and angle between the two adjacent image frames based on the feature data of the common area, and splice the two image frames into a larger three-dimensional model based on the relative position and angle between the two adjacent image frames. With the real-time splicing of image frames, a three-dimensional model of the oral cavity is reconstructed, such as Figure 8 as shown

[0087] However, in the case of user scanning errors or scanning device failures, etc., it may cause some image frames not to contain the scanning rod area. For example, for a series of consecutive image frames, image frame A (with a scanning rod area), image frame B (without a scanning rod area), image frame C (without a scanning rod area), image frame D (with a scanning rod area). In this case, based on the feature information of the scanning rod area, it will be determined that there is no overlapping feature data between image frame A and image frame B, image frame B and image frame C, and image frame C and image frame D. At this time, the electronic device can send a prompt message to prompt the user that the splicing fails.

[0088] In some embodiments, the prompt method may include methods such as voice, a prompt box, and the scanning box changing color. The embodiments of the present application do not limit the prompt method.

[0089] In some embodiments, when or after prompting the user that the splicing fails, the electronic device can also send a prompt message to prompt the user to return to the initial position for scanning and indicate the initial position on the interaction interface, where the initial position may include the scanning position corresponding to the last successfully spliced image frame.

[0090] In other embodiments of the present application, if there is no overlapping feature data between adjacent image frames among multiple image frames when the splicing fails, a prompt message may not be sent, and the currently failed spliced image frame may be spliced with the remaining other image frames in the first image frame or the target three-dimensional data obtained by real-time splicing to find a common area until the splicing is successful.

[0091] In some embodiments, assuming that adjacent image frames include a first image frame and a second image frame, the electronic device can determine the overlapping feature data between the adjacent image frames in the following manner: If the preset markers are included in both adjacent image frames, the electronic device can obtain the first geometric data of the preset marker and the adjacent markers in the first image frame, and obtain the second geometric data of the preset marker and the adjacent markers in the second image frame. Among them, the adjacent markers include other markers within a preset area range centered on the preset marker. If the first geometric data and the second geometric data meet the preset conditions, a preset radius is determined according to the first geometric data and the second geometric data, and the feature data of the area enclosed by the preset marker and the preset radius in the first image frame and the second image frame is used as the overlapping feature data.

[0092] In this embodiment, the preset markers include, but are not limited to, coding markers and geometric pattern markers. The preset area range can be custom-set according to the characteristic information of the actual scanning rod. The preset radius can be custom-set according to the first geometric data and the second geometric data.

[0093] In this embodiment, the first geometric data includes data such as the distance and relative angle between the preset marker and the adjacent markers in the first image frame. The second geometric data includes data such as the distance and relative angle between the preset marker and the adjacent markers in the second image frame. The preset conditions can be custom-set. For example, the preset conditions can be set such that the error between the first geometric data and the second geometric data is less than a preset error threshold. The preset error threshold can be custom-set.

[0094] As an example, the first image frame includes markers A1 and A2, and the second image frame includes markers A2 and A3. The marker A2 in the first image frame and the marker A2 in the second image frame are the same marker, and the electronic device can determine that the preset marker A2 exists in both the first image frame and the second image frame. The marker A1 included in the first image frame is an adjacent marker of the preset marker A2. Assuming that the preset error threshold includes a distance error threshold and an angle error threshold, the distance error threshold is set to 0.2 mm, and the angle error threshold is set to 1 ° . The electronic device can calculate the distance d1 and the angle θ1 between the marker A1 and the marker A2. For example, d1 = 5 mm and θ1 = 30 ° . The marker A3 included in the second image frame is an adjacent marker of the preset marker A2. The electronic device can calculate the distance d2 and the angle θ2 between the marker A3 and the marker A2. For example, d1 = 5.1 mm and θ1 = 29.5 ° . The electronic device calculates the error between the first geometric data and the second geometric data: |d1 - d2| = 0.1 mm, |θ1 - θ2| = 0.5 °According to that the value of |d1 - d2| is less than the distance error threshold and the value of |θ1 - θ2| is less than the angle error threshold, the electronic device can determine that the first geometric data and the second geometric data meet the preset conditions. At this time, the electronic device can determine that the markers A1 and A2 in the first image frame and A3 in the second image frame are the same group of markers. The electronic device can set a preset radius according to the distance d1 between the preset marker and the adjacent marker in the first image frame and the distance d2 between the preset marker and the adjacent marker in the second image frame. For example, the preset radius can be set to 5 mm. The electronic device uses the feature data of the area enclosed by the preset marker and the preset radius in the first image frame and the second image frame as the overlapping feature data.

[0095] In some embodiments, the electronic device can use image processing algorithms (such as HSV color segmentation algorithm, morphological filtering algorithm) to locate the markers on the scanning rod. The embodiments of the present application are not limited thereto.

[0096] In other embodiments, the electronic device can also determine the overlapping feature data between adjacent image frames in other ways. For example, the overlapping feature data between adjacent image frames can be determined based on the geometric features of the feature markers in the scanning rod area of the adjacent image frames. The embodiments of the present application are not limited thereto.

[0097] In some embodiments, after determining the overlapping feature data between adjacent image frames, the electronic device can splice the adjacent image frames based on the overlapping feature data. Specifically, the electronic device can calculate the relative pose parameters between adjacent image frames based on the overlapping feature data, such as parameters such as the optimal rotation matrix and translation vector. The electronic device maps the adjacent image frames to the same coordinate system according to the relative pose parameters, and fuses the point cloud data corresponding to the adjacent image frames to splice the adjacent image frames.

[0098] In practical applications, individual doctors need to display a global oral model to quickly and accurately capture anatomical information in the oral cavity, such as details of the gums and surrounding tissues, so as to facilitate the design optimization of implants and prosthetics.

[0099] To display a global oral model, in some embodiments of the present application, the method further includes the following steps: S14: Obtain a second set of image frames of the oral cavity in the second state; S15: Based on the second set of image frames, obtain the regional three-dimensional data of the soft tissue area in each image frame of the second set of image frames; S16: Obtain a third set of image frames, where the third set of image frames includes some image frames of the first set of image frames or updated image frames obtained when the oral cavity is in the first state; S17: Based on the feature information of the soft tissue area and the feature information of the scanning rod area in each image frame of the third set of image frames, merge the regional three-dimensional data corresponding to the soft tissue area with the target three-dimensional data to obtain an oral model.

[0100] Among them, the oral cavity in the second state indicates that no actual scanning rod is installed in the oral cavity, and the regional three-dimensional data includes the three-dimensional point cloud data corresponding to the soft tissue region. In this embodiment, when scanning the oral cavity without an actual scanning rod installed to obtain the regional three-dimensional data, the scanning accuracy will be higher. Finally, based on the third image frame set, a global oral cavity model can be composed of a soft tissue model not interfered by the scanning rod and a scanning rod model not interfered by the gingiva, with higher reconstruction accuracy.

[0101] Steps S14 and S15 can be performed before step S11, and S16 and S17 can be performed after step S13. Specifically, a scanning device can be used to scan the soft tissues (such as the gingiva) in the oral cavity in the second state, and the scanning device sends the second image frames obtained by scanning to the electronic device. The electronic device obtains the second image frame set of the oral cavity in the second state, and based on the second image frame set, obtains the regional three-dimensional data of the soft tissue region in each image frame in the second image frame set. After scanning the soft tissues in the oral cavity in the second state, use the scanning device to scan the oral cavity in the first state. The electronic device obtains the first image frame set, and based on the first image frame set, obtains the target three-dimensional data corresponding to the actual scanning rod in each image frame in the first image frame set. After obtaining the target three-dimensional data, use the scanning device to scan the oral cavity in the first state again to obtain the third image frame set. The electronic device merges the regional three-dimensional data corresponding to the soft tissue region and the target three-dimensional data based on the feature information of the soft tissue region and the feature information of the scanning rod region in each image frame in the third image frame set to obtain a global oral cavity model as Figure 9 shown.

[0102] It should be noted that the third image frame set in steps S16 and S17 includes some image frames of the first image frame set or updated image frames obtained when the oral cavity is in the first state, that is, it can be obtained by re-scanning with the scanning device, or it can be the historical image frames obtained in step S11.

[0103] Among them, after obtaining the second image frame set of the oral cavity in the second state, a global oral cavity model can be displayed on the interaction interface in response to the user's switching instruction. There can be a switching instruction for displaying the gingival soft tissue on the interaction interface. After steps S14 and S15, after the user selects, the above steps S16 and S17 can be executed to generate and display the global oral cavity model for the user to view.

[0104] In some other embodiments of the present application, in response to a user's switching instruction, the second set of image frames in steps S14 and S15 may not be acquired, and the soft tissue regions identified in the first set of image frames are stitched with the target three-dimensional model to obtain a global oral model. At this time, since the initially obtained target three-dimensional model has no gum data interference, the accuracy of the final global oral model will also be relatively good.

[0105] In some embodiments of the present application, the method further includes: generating a design model according to the target three-dimensional data and sending the design model to a 3D printing device; receiving in real time a user request for modifying the design model; and outputting a revised three-dimensional design model according to the user request and the target three-dimensional model. In the embodiments of the present application, a revised three-dimensional design model can be output according to the user request and the target three-dimensional model, which is convenient for the user to set dentures with different quantities, sizes and shapes.

[0106] The embodiments of the present application provide a three-dimensional reconstruction method. In the case of installing an actual scanning rod in the oral cavity, a first set of image frames of the oral cavity is acquired, and the scanning rod regions in each image frame of the first set of image frames are identified. By using the feature information of the scanning rod regions, multiple image frames in the first set of image frames are stitched to reconstruct the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, realizing image frame stitching based on a rigid structure. Since the rigid structure of the scanning rod is not easily deformed during the scanning process and has good stability, stitching multiple image frames in the first set of image frames based on the feature information of the scanning rod regions helps to improve the stitching accuracy, and further can improve the overall accuracy of three-dimensional reconstruction. Moreover, subsequent repair designs are based on the target three-dimensional data, which can improve the success rate of implantation, the matching degree between the repair product and the patient, and the patient's comfort.

[0107] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0108] Please refer to Figure 10 , Figure 10 FIG. shows the structural diagram of the three-dimensional reconstruction device provided by the embodiments of the present application, which can implement the details of the three-dimensional reconstruction method in the above embodiments and achieve the same effect. As Figure 10As shown in the figure, the three-dimensional reconstruction device 10 can be applied to an electronic device with data processing functions. The three-dimensional reconstruction device 10 includes: an acquisition module 11, configured to acquire a first set of image frames of the oral cavity in a first state, where the oral cavity in the first state means that at least one actual scanning rod is installed in the oral cavity; an identification module 12, configured to identify the scanning rod regions of the respective image frames in the first set of image frames; and a reconstruction module 13, configured to splice a plurality of image frames in the first set of image frames based on the feature information in the scanning rod regions, and reconstruct the target three-dimensional data corresponding to the actual scanning rod in the oral cavity.

[0109] For the specific limitations of the three-dimensional reconstruction device 10, reference can be made to the limitations of the three-dimensional reconstruction method in the foregoing text, which will not be elaborated here. Each module in the above three-dimensional reconstruction device 10 can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or stored in the memory of the electronic device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0110] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and the method implemented when the program instructions are executed can refer to the three-dimensional reconstruction method in each of the above embodiments of the present application.

[0111] Among them, the computer-readable storage medium can be the internal memory of the electronic device in the above embodiment, such as the hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc.

[0112] Further, the computer-readable storage medium mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the electronic device, etc.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A three-dimensional reconstruction method, applied to an electronic device, characterized in that, The three-dimensional reconstruction method includes: Obtaining a first set of image frames of the oral cavity in a first state, where the oral cavity in the first state means that at least one actual scanning rod is installed in the oral cavity; Identifying the scanning rod regions in each image frame of the first set of image frames; Based on the feature information in the scanning rod regions, stitching multiple image frames in the first set of image frames and reconstructing to obtain target three-dimensional data corresponding to the actual scanning rod in the oral cavity; The three-dimensional reconstruction method further includes: Obtaining a second set of image frames of the oral cavity in a second state, where the oral cavity in the second state means that no actual scanning rod is installed in the oral cavity; Based on the second set of image frames, obtaining regional three-dimensional data of the soft tissue region in each image frame of the second set of image frames; Obtaining a third set of image frames, where the third set of image frames includes some image frames of the first set of image frames or updated image frames obtained when the oral cavity is in the first state; Based on the feature information of the soft tissue region and the feature information of the scanning rod region in each image frame of the third set of image frames, merging the regional three-dimensional data corresponding to the soft tissue region with the target three-dimensional data to obtain an oral cavity model.

2. The 3D reconstruction method according to claim 1, wherein Before reconstructing to obtain the target three-dimensional data corresponding to the actual scanning rod in the oral cavity, the method further includes: Identifying the soft tissue regions in each image frame of the first set of image frames; The soft tissue regions do not participate in the three-dimensional reconstruction of the target three-dimensional data or participate in the three-dimensional reconstruction of the target three-dimensional data according to a preset weight.

3. The three-dimensional reconstruction method according to claim 1 or 2, characterized in that, The method further includes: Based on identifying the feature information of the soft tissue region and / or the feature information of the scanning rod region in each image frame of the first set of image frames, performing one or more of a removal operation, a marking operation, a hiding operation, or a segmentation operation on the soft tissue region or the scanning rod region in each image frame.

4. The three-dimensional reconstruction method according to claim 3, characterized in that The first set of image frames includes texture image frames and depth image frames. Identifying the scanning rod regions in each image frame of the first set of image frames includes: Using a preset recognition model to perform feature extraction and feature fusion on the texture image frames and the depth image frames to obtain a mask image, where the mask image includes the scanning rod regions.

5. The three-dimensional reconstruction method according to claim 1, characterized in that, Based on the feature information in the scanning rod regions, stitching multiple image frames in the first set of image frames and reconstructing to obtain target three-dimensional data corresponding to the actual scanning rod in the oral cavity, includes: Based on the feature information in the scanning rod regions, performing real-time stitching on multiple image frames in the first set of image frames; Based on the image frames after real-time stitching, performing real-time three-dimensional reconstruction to obtain the target three-dimensional data corresponding to the actual scanning rod.

6. The three-dimensional reconstruction method according to claim 1, wherein Based on the feature information in the scanning rod regions, stitching multiple image frames in the first set of image frames and reconstructing to obtain target three-dimensional data corresponding to the actual scanning rod in the oral cavity, includes: Based on the feature information of the scanning rod regions, performing real-time stitching on multiple image frames in the first set of image frames; Perform real-time three-dimensional reconstruction based on the spliced image frames to obtain the actual scan data corresponding to the actual scan rod in real time; Based on the feature information of the scan rod area, match the standard data of the actual scan rod from the standard scan rod database. The matching is a periodic matching during the real-time scanning process. During the real-time scanning process, every time a certain number of actual scan data are obtained or every certain period of time, the current actual scan data is feature-spliced with the standard data. As the scan data increases, update the pose of the standard data; Replace the scan data of the actual scan rod with the matched standard data of the actual scan rod to obtain the target three-dimensional data.

7. The three-dimensional reconstruction method according to claim 1, wherein The method further includes: Based on the feature information in the scan rod area, display the target three-dimensional data on the interaction interface of the electronic device in real time when obtaining the first image frame set of the oral cavity in the first state; or After obtaining the second image frame set of the oral cavity in the second state, display the oral cavity model on the interaction interface in response to a user's switching instruction.

8. The three-dimensional reconstruction method according to claim 1, wherein, The splicing of multiple image frames in the first image frame set based on the feature information in the scan rod area includes: Based on the feature information in the scan rod area, determine whether there is overlapping feature data between adjacent image frames in the multiple image frames; If there is overlapping feature data between adjacent image frames in the multiple image frames, perform real-time splicing of the adjacent image frames based on the overlapping feature data; If there is no overlapping feature data between adjacent image frames in the multiple image frames, send a prompt message to prompt the user that the splicing fails, and / or send a prompt message to prompt the user to return to the initial position for scanning.

9. The three-dimensional reconstruction method according to claim 1, wherein The method further includes: Generate a design model according to the target three-dimensional data and send the design model to a 3D printing device; Receive a user request for modifying the design model in real time; Output a revised three-dimensional design model according to the user request and the target three-dimensional model.

10. A three-dimensional reconstruction device, applied to an electronic device, characterized in that, The three-dimensional reconstruction device includes: An acquisition module for acquiring a first image frame set of the oral cavity in the first state, where the oral cavity in the first state means that at least one actual scan rod is installed in the oral cavity; An identification module for identifying the scan rod area of each image frame in the first image frame set; A reconstruction module for splicing multiple image frames in the first image frame set based on the feature information in the scan rod area and reconstructing to obtain the target three-dimensional data corresponding to the actual scan rod in the oral cavity; The three-dimensional reconstruction device further includes: The acquisition module for acquiring a second image frame set of the oral cavity in the second state, where the oral cavity in the second state means that no actual scan rod is installed in the oral cavity; The acquisition module is further configured to obtain the regional three-dimensional data of the soft tissue area in each image frame in the second image frame set based on the second image frame set; The acquisition module is further configured to acquire a third image frame set, where the third image frame set includes some image frames of the first image frame set or updated image frames obtained when the oral cavity is in the first state; The reconstruction module is further configured to merge the regional three-dimensional data corresponding to the soft tissue region with the target three-dimensional data based on the feature information of the soft tissue region and the feature information of the scanning rod region in each image frame of the third image frame set, so as to obtain an oral cavity model.

11. An electronic device, characterized in that, It includes a memory, a processor, and computer-readable instructions. When the computer-readable instructions are executed by the processor, the three-dimensional reconstruction method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 9 is implemented.

Images