Real-time reconstruction method and device for oral cavity live-action three-dimensional digital model

Abstract

The invention provides an oral cavity live-action three-dimensional digital model real-time reconstruction method and device, and the method comprises the steps: collecting oral cavity scanning data, including texture images, point cloud data and data synchronously obtained through an IMU sensor; carrying out real-time preprocessing on the acquired data by adopting an image cascade network based on attention optimization, and generating a soft tissue mask to eliminate soft tissue interference; in combination with IMU sensor data, constructing an ICP odometer based on luminosity error enhancement, and optimizing point cloud registration by fusing geometric errors and luminosity errors; on-line graph optimization is carried out based on closed-loop matching information detected in real time in the background, accumulative errors are corrected, and the global model precision is improved; and screening texture information by adopting a probability graph optimization method to realize uniform model texture mapping. When high-resolution images are collected, a high-definition oral cavity image real-time recognition function is achieved through the deep neural network, masks are generated in real time, other soft tissues of the oral cavity on the images are eliminated, and high-frame-rate real-time three-dimensional reconstruction is supported.

Description

Technical Field

[0001] The present invention belongs to the field of oral three-dimensional reconstruction, and in particular relates to a technical solution for real-time reconstruction of a three-dimensional digital model of an oral cavity. Background Art

[0002] Oral 3D modeling technology is the core foundation of digital dental diagnosis and treatment. Its accuracy directly determines the reliability of clinical applications such as implant navigation and invisible orthodontics. Traditional oral 3D imaging technologies (such as cone-beam CT and structured light scanning) can achieve accuracies of 50-100μm in static environments. However, they face significant challenges in dynamic oral scenarios: 1) physiological vibrations and spontaneous movements of oral soft tissues (such as tongue displacement and cheek contraction) produce motion artifacts; 2) specular reflections from the salivary membrane interfere with the optical acquisition system; and 3) low-frequency vibrations of 0.01-0.1Hz caused by patient respiration can cause sub-pixel registration errors. Existing research using motion compensation algorithms (such as ICP iterative closest point) can reduce errors to 200μm, but are limited by acquisition frame rates below 30Hz, making it difficult to capture transient soft tissue deformations (such as swallowing, which can occur at up to 500ms). Summary of the Invention

[0003] The present invention provides a technical solution for real-time reconstruction of a three-dimensional digital model of an oral cavity, which can solve the problem of the influence of tiny movements in the oral cavity on high-precision three-dimensional modeling.

[0004] The present invention provides a method for real-time reconstruction of a three-dimensional digital model of an oral cavity, comprising the following steps: Collect oral scan data, including texture images, point cloud data, and data obtained synchronously through IMU sensors; Using an attention-optimized image cascade network to pre-process the collected data in real time, generating a soft tissue mask to eliminate soft tissue interference; Combined with IMU sensor data, an ICP odometry based on photometric error enhancement is constructed, and point cloud registration is optimized by fusing geometric error and photometric error. Perform online graph optimization based on closed-loop matching information detected in real time in the background to correct accumulated errors and improve global model accuracy; The probabilistic graph optimization method is used to filter texture information, and uniform model texture mapping is achieved by combining local and global color adjustment.

[0005] Moreover, the image cascade network based on attention optimization includes low, medium and high resolution branches, wherein: The low-resolution branch extracts refined low-resolution features based on the channel-space cascade attention module combined with dilated convolution; The medium-resolution branch extracts features and fuses them with the results obtained by the low-resolution branch through a cascade feature fusion module to obtain refined medium-resolution features; The high-resolution branch adopts the UNet structure and is fused with the results obtained by the medium-resolution branch through a cascade feature fusion module.

[0006] Moreover, the channel-space cascade attention module combined with the dilated convolution includes the following operations: The input features are sequentially upgraded through dilated convolution, channel attention module, spatial attention module and dilated convolution; The channel attention module captures global context features through global average pooling and global maximum pooling operations, giving priority to feature channels containing rich oral textures; The spatial attention module refines the spatial mask through dilated convolution kernels.

[0007] Moreover, the method of optimizing point cloud registration by fusing geometric error and photometric error includes introducing photometric error into the optimization target of point cloud registration and minimizing a composite loss function including geometric error and photometric error.

[0008] Furthermore, the online graph optimization is based on a factor graph approach, including: Integrate front-end sensor data and closed-loop matching information into node and edge constraints; The pose drift and accumulated error are corrected in real time by minimizing the global error function.

[0009] Furthermore, the uniform model texture mapping comprises: Matching texture fragments with geometric features based on voxel grids; When filtering high-quality texture information through probability map optimization, texture information with high texture quality and good consistency between regions is filtered out by maximizing the corresponding objective function.

[0010] Moreover, the uniform model texture mapping further comprises: Perform local color adjustment on the filtered texture to eliminate color difference; Perform global color grading for color balance and consistency.

[0011] On the other hand, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for real-time reconstruction of a three-dimensional digital model of an oral cavity as described above is implemented.

[0012] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for real-time reconstruction of a three-dimensional digital model of an oral cavity.

[0013] On the other hand, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above-mentioned method for real-time reconstruction of a three-dimensional digital model of an oral cavity.

[0014] According to the disclosed real-time reconstruction scheme for a three-dimensional digital model of an oral cavity, an initial data set is first acquired by collecting oral scan data. Deep learning techniques are then used to preprocess the acquired data in real time, automatically separating teeth, surrounding gums, and other oral soft tissues from the high-definition texture data. An iterative closest point (ICP) odometer based on photometric error enhancement is then constructed using an inertial measurement unit (IMU) sensor for precise positioning and tracking. Furthermore, online graph optimization is performed based on closed-loop matching information from real-time background detection to achieve accurate reconstruction of the oral three-dimensional model. Finally, a probabilistic graph optimization method is combined to achieve efficient mapping of uniform model textures. This scheme enables real-time recognition of high-definition oral images through a specifically improved deep neural network during high-resolution image acquisition, generates masks in real time, and eliminates other oral soft tissues from the image, enabling the system to perform real-time three-dimensional reconstruction at a high frame rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings: Figure 1 A schematic flow chart of a method for real-time reconstruction of a three-dimensional digital model of an oral cavity provided by an embodiment of the present invention.

[0016] Figure 2 A schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the structure of a real-time reconstruction device for a three-dimensional digital model of an oral cavity provided by an embodiment of the present invention.

[0018] Figure 4 Schematic diagram of the attention-enhanced image cascade network structure provided by an embodiment of the present invention.

[0019] Figure 5 A schematic diagram of a CFF module provided in an embodiment of the present invention.

[0020] Figure 6 Schematic diagram of a dilated convolution module combined with an attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0022] The present invention provides a method for real-time reconstruction of a three-dimensional digital model of the oral cavity, which can efficiently and accurately construct a three-dimensional digital model from oral imaging data in real time. This method is applicable to various electronic devices and can be implemented in different hardware and system architectures, particularly on devices such as smart terminal devices, computer devices, and cloud computing platforms, providing powerful real-time computing and data processing capabilities. The electronic devices include but are not limited to smartphones, tablets, laptops, personal computers, industrial computers, and cloud computing-based server clusters. These devices acquire image data from the interior of the oral cavity by integrating computer programs and image acquisition equipment, such as an oral three-dimensional scanner, and generate a three-dimensional digital model based on this data in real time.

[0023] Furthermore, the real-time reconstruction method of the present invention is not limited to a specific hardware configuration or system architecture but rather has broad adaptability. The type and processing power of electronic devices do not limit the application of the present method, and the method can be flexibly adjusted based on the performance of different devices. This innovative method can rapidly acquire a three-dimensional oral model and provide dentists with real-time, high-precision imaging data, thereby assisting in the diagnosis of oral diseases, the development of treatment plans, and surgical procedures.

[0024] Example 1 The present invention provides a method for real-time reconstruction of a three-dimensional digital model of a real oral scene, comprising: collecting oral scan data; preprocessing the data in real time based on deep learning technology; constructing an ICP odometer based on photometric error enhancement in combination with an IMU sensor; performing online graph optimization based on closed-loop matching information detected in real time in the background; and uniform model texture mapping based on probability graph optimization.

[0025] Figure 1 The flowchart of an embodiment of a method for real-time reconstruction of a three-dimensional digital model of an oral cavity disclosed in the present invention is shown. When implemented, the method can be applied to electronic devices.

[0026] 110: Collect oral scan data.

[0027] In practice, users use an oral 3D scanner to scan the oral area, acquiring high-resolution texture images and ordered point cloud data. Specifically, the 3D scanner uses high-precision optical sensors to scan the interior of the oral cavity in all directions, generating detailed surface morphology data while also recording the color and texture information of the oral surface. The collected point cloud data consists of spatially distributed 3D coordinate points, reflecting the precise geometry and structure of the oral surface.

[0028] At the same time, IMU (Inertial Measurement Unit) sensors are also used for data collection. In specific implementation, it is recommended that the IMU sensor module be embedded in the handle of the oral 3D scanner and fixed coaxially with the optical sensor to ensure that the IMU's motion coordinate system is physically aligned with the optical scanning coordinate system. The IMU sensor captures the position and motion information of the oral scanner in real time, obtains dynamic data such as acceleration and angular velocity, and uploads it together with the collected point cloud data. It can be transmitted to a remote server or computer device for subsequent processing. This IMU data provides the necessary reference information for subsequent motion compensation and data alignment, helping to improve data accuracy and stability during subsequent reconstruction.

[0029] During the acquisition process, texture images and point cloud data will be associated through a preset synchronization mechanism and stored synchronously in the device's processing system.

[0030] 120: Real-time data preprocessing based on deep learning technology.

[0031] During oral scanning, teeth and periodontal and gingival tissue can be considered fixed structures, while oral soft tissue is mobile. Therefore, during high-resolution image acquisition, real-time identification and generation of soft tissue masks, eliminating soft tissue from the image, is crucial for achieving high-frame-rate, real-time 3D reconstruction. To improve the real-time and accuracy of artificial intelligence (AI) intelligent recognition, the real-time modeling of soft tissue must be effectively addressed.

[0032] In order to achieve accurate real-time segmentation, the embodiment of the present invention proposes an ICNet structure based on attention optimization, that is, an attention-enhanced image cascade network for real-time preprocessing of data, whose structure is shown in the figure Figure 4 As shown in the figure, the recommended image and feature sizes are shown in brackets. The core framework of this method is the ICNet network (Image Cascade Network), which consists of three network branches: low-, medium-, and high-resolution. This paper optimizes and improves this framework. The top branch is the low-resolution branch (1 / 4 the original image size), the middle branch is the medium-resolution branch (1 / 2 the original image size), and the bottom branch is the high-resolution branch (original image size). The branches are connected by CFF. The low- and medium-resolution branches are terminated with 7×7 convolution and pooling layers, while the high-resolution branch maintains lightweight feature extraction.

[0033] Specifically, the preferred embodiments of the embodiments are as follows: The low-resolution branch draws on the improved ResNet50 structure. First, based on a 7×7 convolution with a stride of 2, it obtains features of 1 / 8 the size of the original image; then it passes through conv1_x (corresponding to Layer 1 in ResNet50) in sequence to obtain features of 1 / 16 the size of the original image; then it passes through conv2_x (corresponding to Layer 2 in ResNet50) to obtain features of 1 / 32 the size of the original image; finally, it passes through conv3_x and conv4_x (corresponding to Layer 3 and Layer 4 in ResNet50), inputs the CBAM+ optimization module proposed in this invention, and finally passes through 7×7 convolution and pooling layers to obtain refined low-resolution features.

[0034] The structure of the mid-resolution branch is identical to that of the low-resolution branch, except that the conv3_x and conv4_x layers and the CBAM+ optimization module are removed to improve computational efficiency. First, a 7×7 convolution with a stride of 2 is performed to obtain features at 1 / 4 the original image size. This is followed by conv1_x (corresponding to Layer 1 in ResNet50) to obtain features at 1 / 8 the original image size. This is followed by conv2_x (corresponding to Layer 2 in ResNet50) to obtain features at 1 / 16 the original image size. Finally, a 7×7 convolution and pooling layer are performed, and the CFF is used to fuse the results with those from the low-resolution branch to obtain the refined mid-resolution features.

[0035] The high-resolution branch adopts a lightweight design and a classic UNet architecture. It undergoes three 3×3 convolutions, a ReLU activation function, and a batch normalization layer to obtain features at 1 / 2, 1 / 4, and 1 / 8 the original image size. After feature fusion with the results from the medium-resolution branch through CFF, the image is restored to its original size through three upsampling layers, followed by a 1×1 convolution to transform the feature dimension.

[0036] The present invention adopts corresponding scale networks for different resolutions to achieve a balance between feature extraction and network performance, and does not miss detailed features in high-resolution images while ensuring real-time performance.

[0037] However, since oral soft tissue and the gingival margin are difficult to distinguish, ICNet can retain most semantic information when performing feature extraction in the low-resolution branch, but the loss of details and edge information will affect segmentation accuracy. Therefore, an optimized design combining the attention mechanism and dilated convolution is added to the low-resolution branch, namely the CBAM+ optimization module proposed in this paper. CBAM is a channel-space cascaded attention module. This paper proposes a CBAM optimized with dilated convolution, referred to as the CBAM+ optimization module.

[0038] like Figure 6 As shown, the CBAM+ optimization module employs a cascaded processing pipeline: input features are first dimensionalized via a 1×1 dilated convolution (with dilation=2) and a 3×3 dilated convolution (with dilation=2). This is then fed into a channel-attention module, processed by a spatial attention module, and then reduced by a 3×3 dilated convolution (with dilation=2). Finally, residual fusion is performed with the original features. The channel-attention module comprises a dual-way pooling layer and a shared MLP (with two fully connected layers and a 16:1 channel compression ratio in the middle layer). The spatial attention module adds a 3×3 dilated convolution kernel to the traditional architecture. In the low-resolution branch, dynamic feature calibration is achieved via a channel-spatial cascaded attention module (CBAM+). Channel-attention suppresses interference signals such as instrument reflections, while spatial attention with dilated convolution kernels precisely locates soft tissue edges. These two processes form a closed-loop optimization loop using a "cascade-residual" architecture. Combined with multi-stage dilated convolutions, the network achieves both fine-grained edge detection and large-scale anatomical structure consistency.

[0039] Specific preferred solutions such as Figure 6 As shown, the channel attention module in CBAM simultaneously performs global average pooling (GAP) and global maximum pooling (GMP) on the input features. The dual-pooling results are fed into a shared two-layer MLP (the first layer reduces the number of channels to C / 16, and the second layer restores it to C). Channel weight vectors are generated and then multiplied channel-by-channel with the original features. The channel attention module captures global contextual features through global average pooling and global maximum pooling, prioritizing feature channels rich in oral texture. The spatial attention module performs mean-maximum merging of the channel attention output features along the channel axis to generate a two-channel feature map. This is then processed through a 3×3 dilated convolution (dilation=2) followed by a sigmoid activation function to generate a spatial mask, which is then spatially weighted with the input features. The spatial attention module further refines the feature representation at each pixel by aggregating cross-channel features. In this way, CBAM enables precise adjustment and optimization of features at both the channel and spatial levels, effectively capturing and refining detailed features critical for soft tissue segmentation. Next, a 3×3 dilated convolutional layer is added after the spatial attention module to further refine the features highlighted by the channel attention. This step not only optimizes the details of local features but also enhances the model's adaptability to different oral environments. Finally, a residual learning mechanism is employed to address the vanishing gradient problem and accelerate the initial learning process.

[0040] To suppress the impact of useless features on the model, the CBAM attention model is applied to the final convolutional block of the low-resolution branch. By adding this module to the quarter-resolution branch, segmentation accuracy is improved. The convolutional blocks at the front of the low- and medium-resolution branches are primarily used to extract image features, while the attention layer further enhances feature extraction accuracy. The high-resolution branch provides better guidance for the low- and medium-resolution branches, thereby optimizing segmentation results.

[0041] Finally, the feature maps generated by the three branches are fused through the CFF (Cascade Feature Fusion) module, as shown in Figure 5 As shown in Figure 1, the CFF module takes three inputs: feature maps F1, F2, and labels. When processing F1, it is first upsampled twice to the same size as F2. A 3×3 dilated convolution with a dilation rate of 1 is then used to refine the F1 features. F2 is then convolved with a uniform channel number (1×1 convolution) to match the number of channels in F1. It is then normalized through a batch normalization layer. The resulting F2 features are summed (SUM) with F1 and activated with a ReLU to produce F2'. To enhance the learning capability of the F1 features, the F1 features are upsampled twice and convolved through the classification head to obtain the final result. The auxiliary loss is then calculated with the true labels. Gradient backpropagation is used to optimize the underlying feature learning, thereby improving the optimization effect of the loss function. In the first CFF module, F1 and F2 are derived from the low-resolution and medium-resolution branches, respectively, and output 1 / 2 resolution features. The second-level CFF fuses these features with the high-resolution branch features as the new F1 and F2, respectively, ultimately outputting a segmentation result at the original image size. Through this fusion operation, the accuracy of the segmentation results is further improved.

[0042] To further enhance the performance of the oral image segmentation model, a large number of oral scans covering various oral structures and soft tissue conditions were collected to ensure data diversity and representativeness. These scans encompass a wide range of samples, including age, gender, and oral health status, ensuring the model's adaptability to various oral environments. All collected oral images were manually annotated using professional annotation software to ensure accuracy and high quality. Dataset partitioning is crucial, so the annotated dataset was divided into training, validation, and test sets using a commonly used ratio of 8:1:1. The training set, which accounts for 80% of the total dataset, is primarily used for model training and parameter optimization; the validation set, which accounts for 10%, is used for model tuning and preventing overfitting during training; and the test set, which also accounts for 10%, is used to ultimately evaluate the model's generalization and real-world performance. This data partitioning ensures fairness in training, validation, and testing, and effectively avoids overlap between the training and test sets, thereby improving model reliability and accuracy.

[0043] During data processing and preprocessing, several optimization steps were performed to address the characteristics of oral images, including image normalization, cropping, and enhancement. This improved the model's robustness across varying image quality, lighting conditions, and viewing angles. Furthermore, to further reduce overfitting, data augmentation techniques such as random rotation, flipping, and scaling were employed to increase the diversity of the dataset and better adapt the model to various practical application scenarios.

[0044] After the dataset is prepared, it is input into the constructed segmentation model for training. During the training process, the corresponding loss functions of 1 / 16, 1 / 8, and 1 / 4 size labels are calculated respectively. Figure 4 The dotted line indicates that the training process is complete only. The model uses the attention-based ICNet architecture, combined with multi-resolution network branches for training, to produce accurate segmentation results.

[0045] 130: Build an ICP odometry based on photometric error enhancement by combining IMU sensors.

[0046] Based on pre-processed point cloud and texture data, combined with IMU data collected by the IMU sensor, more precise state estimation and positioning tasks can be performed to further improve the accuracy and robustness of oral 3D reconstruction. Specifically, by fusing these sensor data, the drift and accumulated errors caused by relying solely on visual information can be significantly reduced. Especially in situations of rapid movement or limited field of view, IMU data can provide powerful compensation and correction, ensuring the stability and accuracy of the reconstruction process.

[0047] On this basis, we construct an ICP odometry based on photometric error enhancement. In traditional ICP algorithms, point cloud registration typically relies on geometric information while ignoring texture and photometric information. To address this issue, in addition to geometric matching, this method introduces photometric error as an auxiliary optimization objective, improving registration accuracy by minimizing the photometric difference between image textures. The specific implementation steps include: First, the camera's pose information is acquired synchronously through IMU data and visual data. The IMU provides preliminary camera motion information, including acceleration and angular velocity, to help estimate the camera's pose, while the visual data provides texture and photometric information from the images captured by the camera, which is used for subsequent calculation of photometric error.

[0048] The point clouds are then preliminarily registered using a traditional ICP algorithm. This step relies solely on geometric information, such as the Euclidean distance between points, and ignores photometric information in the preliminary registration. Next, based on the preliminary registration of the point clouds, the photometric error is calculated for each pair of corresponding points. Photometric error is measured based on the difference between the grayscale and color values in the texture image and reflects the quality of the texture matching of the point cloud. Each point in the point cloud is mapped to a pixel position in the image using projection or depth estimation techniques, and the photometric difference between corresponding points in the point clouds is calculated.

[0049] During the optimization process, photometric error helps accurately align point clouds by minimizing the difference between image textures. Specifically, photometric error is introduced into the optimization objective function of point cloud registration. The optimization objective includes not only geometric error (the distance difference between point clouds) but also photometric error. The effects of geometric error and photometric error are balanced by weighting coefficients, so that both geometric and photometric matching can be optimized during registration. The optimization objective function can be expressed as:

[0050] in, is the geometric error, is the photometric error, and It is a balance coefficient used to adjust the weights of geometric error and photometric error in optimization.

[0051] Ultimately, by minimizing a composite loss function that incorporates both geometric and photometric errors, the system achieves more accurate point cloud registration results. This not only achieves precise alignment of point clouds in geometric space, but also accounts for the effects of varying illumination or uneven textures, enhancing the stability and accuracy of the registration process. In particular, the introduction of photometric errors effectively improves the stability and robustness of the matching process in situations with varying illumination or sparse textures.

[0052] In traditional ICP algorithms, point cloud registration usually relies on geometric information while ignoring texture and photometric information. To address this problem, in addition to geometric matching, this method introduces photometric error as an auxiliary optimization objective, improving the accuracy of registration by minimizing the photometric difference in image texture. The specific implementation steps include: first, synchronously acquiring the camera's pose information through IMU data and visual data; then, using the traditional ICP algorithm to perform a preliminary registration of the point cloud, and then calculating the photometric error between each pair of point clouds and adding photometric constraints during the optimization process, thereby simultaneously optimizing the geometric and photometric registration effects. In this way, the system can not only accurately align point clouds in geometric space, but also take into account the additional constraints brought by texture information, enhancing the stability and accuracy of the registration process, especially in the case of changing lighting or uneven texture, and maintaining efficient matching results.

[0053] 140: Perform online graph optimization based on closed-loop matching information detected in real time in the background.

[0054] Online graph optimization, based on front-end data and closed-loop matching information from real-time back-end detection, aims to improve the system's positioning accuracy and stability in real time. On the front end, sensor data (such as cameras and IMUs) provides current pose estimates and environmental information. Over time, this data may deviate due to sensor noise and accumulated errors. Back-end closed-loop matching information identifies loop points to detect whether the system has returned to the known position and provides necessary corrections. Combined with this closed-loop matching information, the system can optimize previous estimates.

[0055] In its implementation, closed-loop matching information is obtained through a loop detection algorithm, and these loop points are used as constraints for graph optimization. Front-end data and closed-loop matching information are integrated through factor graph optimization, with the goal of minimizing the system's global error. During graph optimization, each node represents a state (such as the camera pose), while edges represent constraints between nodes (such as the relative relationship between position and time). Ultimately, by maximizing the optimization objective function and correcting accumulated errors, the system ensures long-term, high-precision positioning in complex and dynamic environments. This optimization process occurs in real time, without the need for offline processing, enabling the system to maintain high accuracy over long periods of operation. This significantly improves the system's positioning and mapping accuracy, particularly in complex and dynamic environments.

[0056] 150: Uniform model texture mapping based on probabilistic graph optimization.

[0057] In view of the dense texture and geometric features of the texture data, we first extract and match each texture fragment with the corresponding geometric object to obtain the corresponding matching information. This matching information is used to construct a voxel grid model. As an effective 3D data representation, the voxel grid can accurately integrate texture and geometric features. The color information in the voxel grid and the fineness of the texture detail on the triangle facets can be used as a preliminary assessment of the texture mapping quality, which can be used as a basis for further optimization of the texture mapping process.

[0058] Next, the texture information is filtered using a probabilistic graph optimization approach. In this process, graph optimization methods construct nodes and edges to represent the various features of the 3D model and the constraints between them.

[0059] Specifically, each node represents a texture or geometric feature, and the edge represents the matching relationship between features. The optimization objective function is:

[0060] in, is the total optimization error, and Represent the geometric matching error and ideal value of the i-th and j-th feature points respectively, and represents the quality metric of texture information, is the weight coefficient, It is the balancing factor for texture optimization. By maximizing this objective function, we can filter out texture information with high texture quality and good inter-regional consistency, and eliminate inconsistent or low-quality textures, thereby improving the overall texture effect and mapping quality.

[0061] The filtered texture information is further optimized through local and global color grading. Local color grading primarily adjusts areas of texture with color differences, making the colors of detailed details more natural and blending with the surrounding areas. Global color grading adjusts color balance and consistency across the entire model, eliminating color variations caused by lighting changes or scanning errors, ensuring a uniform and natural texture across the entire surface.

[0062] In practice, those skilled in the art may employ software technology to automate the above process. Accordingly, providing a solution for real-time reconstruction of a three-dimensional digital model of an oral cavity, including a computer or server, and executing the above process on the computer or server to perform real-time reconstruction of a three-dimensional digital model of an oral cavity, would also fall within the scope of protection of the present invention.

[0063] In another embodiment, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned method for real-time reconstruction of a three-dimensional digital model of a real oral scene.

[0064] In another embodiment, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned method for real-time reconstruction of a three-dimensional digital model of an oral cavity.

[0065] Example 2 The present invention also provides an electronic device, including a processor, a memory, a computer program stored in the memory and executed by the processor, a communication interface, and a communication bus. The processor, memory, and communication interface communicate through the communication bus. When the processor executes the computer program, the real-time reconstruction method of the three-dimensional digital model of the oral cavity described in the first aspect is implemented.

[0066] Figure 2The following is a schematic diagram of the structure of an electronic device embodiment of the present invention. The specific embodiments of the present invention do not impose any specific limitations on the implementation of the electronic device. The device includes the following components: a processor 220, a communication interface 250, an input device 230, an output device 240, and a memory 200, which are interconnected and communicate with each other via a communication bus 260. The communication interface 250 is responsible for network communication with other devices (such as clients or other servers).

[0067] The processor 220 is used to execute the programs 210, which execute the relevant steps of the method for real-time reconstruction of the oral cavity real-scene three-dimensional digital model.

[0068] Specifically, program 210 includes program code, which includes computer operating instructions. These instructions enable processor 220 to perform specific computing tasks. Processor 220 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or an integrated circuit configured to perform the functions required by embodiments of the present invention. The processors in an electronic device may be multiple processors of the same type (e.g., multiple CPUs) or a combination of different types (e.g., a combination of a CPU and an ASIC).

[0069] The memory 200 is used to store a program 210, which may include a high-speed RAM memory and may also include a non-volatile memory (such as a disk memory). The function of the program 210 is to enable the processor 220 to perform the specific operations in the above-mentioned method for real-time reconstruction of a three-dimensional digital model of a real oral cavity.

[0070] It should be noted that the algorithms or display functions described herein are not directly associated with a specific computer, virtual system, or other device; any general-purpose system can be used with the described approach. Based on the foregoing, the structure required to construct such a system is readily apparent. Furthermore, the embodiments of the present invention are not limited to a specific programming language. A variety of programming languages can be used to implement the functions described herein, and the description of a specific programming language is provided solely to provide examples of preferred embodiments.

[0071] Example 3 The present invention also provides a real-time reconstruction device for a three-dimensional digital model of a real oral scene, which includes: a data acquisition module for collecting oral scan data; a preprocessing module for performing real-time preprocessing of the data based on deep learning technology; a front-end module for constructing an ICP odometer based on photometric error enhancement in combination with an IMU sensor; a back-end module for performing online graph optimization based on closed-loop matching information detected in real time in the background; and a post-processing module for performing global uniform model texture mapping based on probability graph optimization.

[0072] Figure 3FIG. 3 is a schematic diagram of a device for real-time reconstruction of a three-dimensional digital model of an oral cavity according to the present invention. As shown in the figure, the device 300 includes: The data acquisition module 310 is responsible for collecting oral scan data, using high-precision sensors to capture the three-dimensional structure and texture information of the oral cavity in real time, ensuring the comprehensiveness and accuracy of the data and providing basic data for subsequent processing.

[0073] The preprocessing module 320 is based on an attention-optimized ICNet structure. The preprocessing module performs real-time recognition of the collected oral scan data and generates a soft tissue mask to eliminate the soft tissue part on the image. This is an important step in achieving high frame rate real-time 3D reconstruction and provides support for significantly improving reconstruction efficiency and quality.

[0074] Front-end module 330, combined with an IMU sensor, builds an ICP odometry system based on photometric error enhancement. By fusing IMU data with visual information, it improves positioning accuracy and stability, overcoming the positioning drift problem caused by dynamic environments and sensor errors in traditional methods.

[0075] Backend module 340 performs online graph optimization based on real-time loop matching information. When a loop match is detected, the system matches the known position with the current observation and uses graph optimization techniques to make global adjustments, effectively eliminating error accumulation and improving the accuracy of the overall model.

[0076] Post-processing module 350 optimizes texture mapping using a technique based on probability graph optimization. By filtering high-quality texture information and performing local and global color processing, it ultimately achieves globally uniform model texture mapping, enhancing the visual quality and realism of the 3D model.

[0077] In summary, the above solution of the present invention can achieve the following beneficial effects: Through an attention-optimized ICNet structure, real-time recognition and generation of soft tissue masks are achieved, eliminating the soft tissue part on the image and realizing high-frame-rate real-time 3D reconstruction.

[0078] By combining a multi-branch network structure with low, medium and high resolution, especially adding the attention mechanism (CBAM) and void convolution in the low-resolution branch, it can effectively capture the details of soft tissue and gingival margins, overcoming the problem of detail loss in traditional methods when the resolution is insufficient, and ensuring high accuracy of real-time soft tissue segmentation.

[0079] By combining channel-wise and spatial-level attention, the algorithm leverages global context-aware features to enhance important features and suppress less important ones, improving the network's ability to learn details. This is particularly impressive when processing complex oral soft tissue boundaries. Furthermore, the introduction of dilated convolutions further refines features, optimizing local feature details and enhancing the model's adaptability to diverse oral environments and scanning conditions. This ensures that the model maintains high segmentation accuracy in diverse scenarios.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of these modules may be selected to achieve the objectives of the present embodiments based on practical needs. Those skilled in the art will be able to understand and implement these embodiments without inventive effort. Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a required general-purpose hardware platform, or, of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or portions thereof. The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for real-time reconstruction of a three-dimensional digital model of an oral cavity, characterized in that: The following processes are included: Collect oral scan data, including texture images, point cloud data, and data obtained synchronously through IMU sensors; Using an attention-optimized image cascade network to pre-process the collected data in real time, generating a soft tissue mask to eliminate soft tissue interference; Combined with IMU sensor data, an ICP odometry based on photometric error enhancement is constructed, and point cloud registration is optimized by fusing geometric error and photometric error. Perform online graph optimization based on closed-loop matching information detected in real time in the background to correct accumulated errors and improve global model accuracy; The probabilistic graph optimization method is used to filter texture information, and uniform model texture mapping is achieved by combining local and global color adjustment.

2. The method according to claim 1, characterized in that The image cascade network based on attention optimization includes low, medium and high resolution branches, where: The low-resolution branch extracts refined low-resolution features based on the channel-space cascade attention module combined with dilated convolution; The medium-resolution branch extracts features and fuses them with the results obtained by the low-resolution branch through a cascade feature fusion module to obtain refined medium-resolution features; The high-resolution branch adopts the UNet structure and is fused with the results obtained by the medium-resolution branch through a cascade feature fusion module.

3. The method according to claim 2, characterized in that The channel-space cascade attention module combined with dilated convolution includes the following operations: The input features are sequentially upgraded through dilated convolution, channel attention module, spatial attention module and dilated convolution; The channel attention module captures global context features through global average pooling and global maximum pooling operations, giving priority to feature channels containing rich oral textures; The spatial attention module refines the spatial mask through dilated convolution kernels.

4. The method according to claim 1, wherein The point cloud registration is optimized by fusing geometric error and photometric error, including introducing photometric error into the optimization target of point cloud registration and minimizing a composite loss function including geometric error and photometric error.

5. The method according to claim 1, wherein The online graph optimization is based on a factor graph approach and includes: Integrate front-end sensor data and closed-loop matching information into node and edge constraints; The pose drift and accumulated error are corrected in real time by minimizing the global error function.

6. The method according to claim 1, characterized in that The uniform model texture mapping includes: Matching texture fragments with geometric features based on voxel grids; When filtering high-quality texture information through probability map optimization, texture information with high texture quality and good consistency between regions is filtered out by maximizing the corresponding objective function.

7. The method according to claim 6, characterized in that The uniform model texture mapping further comprises: Perform local color adjustment on the filtered texture to eliminate color difference; Perform global color grading for color balance and consistency.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for real-time reconstruction of a three-dimensional digital model of an oral cavity according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for real-time reconstruction of a three-dimensional digital model of an oral cavity according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the method for real-time reconstruction of a three-dimensional digital model of an oral cavity according to any one of claims 1 to 7 is implemented.

Images