Control method and device, system, and storage medium for an oral procedure device

Abstract

The present disclosure relates to the technical field of control, and particularly relates to a control method and device, system and storage medium of an oral operation device, comprising: acquiring a plurality of types of first data, the plurality of types of first data comprising multiple types of oral image data, oral scanning data and oral recording data; performing multi-modal fusion on the first data according to a mapping relationship between different types of first data to generate an oral model, wherein the oral model comprises a plurality of structure layers, and each structure layer in the plurality of structure layers corresponds to a type of first data; generating a planning path of the oral operation device according to the oral model; and controlling the oral operation device to move to a target position according to the planning path.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, and in particular to a control method, device, system, and storage medium for an oral manipulation device. Background Technology

[0002] As dental procedures become more precise and minimally invasive, the requirements for accuracy and stability in procedures such as dental implants, tooth preparation, and orthodontic assistance are increasing. Traditional dental procedures mainly rely on manual operation using hand instruments, and their accuracy is limited by factors such as hand tremors and limited field of vision, making it difficult to guarantee consistency and repeatability in complex anatomical environments.

[0003] As a result, various dental manipulation devices are gradually being applied, such as using robotic arms or end effectors to replace or automatically complete key intraoral operations such as drilling, positioning, and implantation. Summary of the Invention

[0004] In related technologies, the automation level of oral manipulation devices is low, and automatic control of oral manipulation devices cannot be achieved.

[0005] In view of this, the present disclosure provides a control method, device, system, and storage medium for an oral manipulation device. By performing multimodal fusion on multiple types of data, an accurate oral model is generated, and subsequent path planning is performed based on the oral model, so as to achieve high-precision automatic control of the oral manipulation device.

[0006] According to one aspect of this disclosure, a control method for an oral manipulation device is provided, comprising: acquiring multiple types of first data, the multiple types of first data including oral imaging data, oral scan data, and oral record data; performing multimodal fusion on the first data according to the mapping relationship between different types of first data to generate an oral model, wherein the oral model includes multiple structural layers, each of the multiple structural layers corresponding to a type of first data; generating a planned path for the oral manipulation device according to the oral model; and controlling the oral manipulation device to move to a target position according to the planned path.

[0007] In some embodiments, generating an oral cavity model by multimodal fusion of the first data according to the mapping relationship between different types of first data includes: determining an initial mapping relationship between different types of first data using a first feature, wherein the first feature exists in each type of first data; registering the multiple types of first data according to the initial mapping relationship to determine a target mapping relationship between the different types of first data; and multimodal fusion of the first data according to the target mapping relationship to generate the oral cavity model.

[0008] In some embodiments, registering the multiple types of first data according to the initial mapping relationship to determine the target mapping relationship includes: registering the multiple types of first data using one of a plurality of registration methods to obtain a first mapping relationship, wherein the plurality of registration methods include at least one of surface registration, feature registration, and deep learning registration; determining the target mapping relationship based on the first mapping relationship when the deviation between the first mapping relationship and the initial mapping relationship is less than or equal to a preset threshold; updating the initial mapping relationship again using other registration methods among the plurality of registration methods until a second mapping relationship is obtained when the deviation between the second mapping relationship and the initial mapping relationship is less than or equal to the preset threshold.

[0009] In some embodiments, the plurality of structural layers include an anatomical structural layer, a surface structural layer, a soft tissue structural layer, and an occlusal contact layer. The oral imaging data includes computed tomography (CT) image data, the oral scanning data includes intraoral scanning data and facial scanning data, and the oral recording data includes occlusal recording data. The process of multimodal fusion of the first data based on the mapping relationship between different types of first data to generate an oral model includes: generating the anatomical structural layer based on the CT image data; generating the surface structural layer based on the intraoral scanning data; generating the soft tissue structural layer based on the facial scanning data; generating the occlusal contact layer based on the occlusal recording data; and fusing the anatomical structural layer, the surface structural layer, the soft tissue structural layer, and the occlusal contact layer according to the mapping relationship to generate the oral model.

[0010] In some embodiments, generating an oral cavity model by multimodal fusion of the first data according to the mapping relationship between different types of first data includes: preprocessing the first data, wherein the preprocessing includes at least one of artifact suppression, structural segmentation, three-dimensional reconstruction, noise removal, hole detection, surface smoothing, and coordinate normalization.

[0011] In some embodiments, generating a planned path for the oral manipulation device based on the oral model includes: determining a second feature in the oral model based on the grayscale features, morphological features, and continuity of the oral model, wherein the second feature includes at least one of anatomical structure, edentulous area parameters, and occlusal features; and generating the planned path based on the second feature.

[0012] In some embodiments, generating the planned path according to the second feature includes: determining motion constraints of the oral manipulation device and parameters of the target position according to the second feature, wherein the motion constraints include the motion space of the oral manipulation device; and generating the planned path according to the motion constraints and the parameters of the target position.

[0013] In some embodiments, generating the planned path based on the second feature includes: determining the operating mode of the oral manipulation device based on the second feature; and generating the planned path based on the operating mode.

[0014] In some embodiments, the control method further includes: receiving second data, the second data including feedback data of the oral manipulation device during movement; and updating the planned path based on the second data.

[0015] In some embodiments, the second data includes at least one of the pose, motion state, force, torque, drive current, drilling direction, and drilling depth of the oral manipulation device. Updating the planned path based on the second data includes: determining the actual path of the oral manipulation device based on the second data and the oral model; and dynamically adjusting the planned path based on the deviation between the actual path and the planned path.

[0016] According to a second aspect of this disclosure, a control device for an oral manipulation apparatus is provided, comprising: a data acquisition module configured to acquire multiple types of first data, the first data including oral imaging data, oral scan data, and oral record data; a model generation module configured to perform multimodal fusion of the first data according to the mapping relationship between different types of first data to generate an oral model, wherein the oral model includes multiple structural layers, each of the multiple structural layers corresponding to a type of first data; a path generation module configured to generate a planned path for the oral manipulation apparatus according to the oral model, the planned path being used to move the oral manipulation apparatus to a target position; and a control module configured to control the oral manipulation apparatus according to the planned path.

[0017] According to a third aspect of this disclosure, a control device for an oral manipulation apparatus is provided, comprising: at least one memory; and at least one processor coupled to said at least one memory, said at least one processor being configured to execute a control method as described in any embodiment of this disclosure based on instructions stored in said at least one memory.

[0018] According to a fourth aspect of this disclosure, a system is provided, comprising: a control device as described in any embodiment of this disclosure; and an oral manipulation device configured to operate according to a planned path generated by the control device.

[0019] In some embodiments: the control device is further configured to receive second data, update the planned path based on the second data, and control the oral manipulation device based on the updated planned path, the second data including feedback data of the oral manipulation device during movement; the oral manipulation device is further configured to send the second data to the control device during movement.

[0020] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the control method as described in any embodiment of this disclosure.

[0021] According to a sixth aspect of this disclosure, a computer program product is provided that, when run on a computer, causes the computer to implement the control method as described in any embodiment of this disclosure. Attached Figure Description

[0022] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0023] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0024] Figure 1 A flowchart illustrating a control method according to some embodiments of the present disclosure is shown;

[0025] Figure 2 A flowchart illustrating multimodal fusion according to some embodiments of the present disclosure is shown;

[0026] Figure 3 A flowchart illustrating the generation of a planned path according to some embodiments of the present disclosure is shown;

[0027] Figure 4 A flowchart illustrating a control method according to other embodiments of the present disclosure is shown;

[0028] Figure 5 A schematic diagram illustrating the process of controlling an oral manipulation device to perform implantation operations in some embodiments of this disclosure is shown;

[0029] Figure 6 A block diagram of a control device according to some embodiments of the present disclosure is shown;

[0030] Figure 7 Block diagrams of control devices according to other embodiments of the present disclosure are shown;

[0031] Figure 8 This is a block diagram illustrating a system for implementing some embodiments of the present disclosure;

[0032] Figure 9 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0033] It should be understood that the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. Furthermore, the same or similar reference numerals denote the same or similar components. Detailed Implementation

[0034] Various embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The descriptions of the embodiments are merely illustrative and are in no way intended to limit the scope of the disclosure or its application or use. The present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to make the disclosure thorough and complete, and to fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps set forth in these embodiments should be interpreted as merely illustrative and not as limiting.

[0035] The terms “first,” “second,” and similar words used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different parts. Words such as “including” mean that the element preceding the word covers the element listed after the word, and do not exclude the possibility of covering other elements as well.

[0036] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0037] All terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in a general dictionary, such as a dictionary, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] In related technologies, the automation level of oral manipulation devices is low, and automatic control of oral manipulation devices cannot be achieved.

[0040] In view of this, this disclosure proposes a control method, device, system, and storage medium for an oral manipulation device. By performing multimodal fusion on multiple types of data, an accurate oral model is generated, and subsequent path planning is performed based on the oral model to achieve high-precision automatic control of the oral manipulation device.

[0041] First, combined Figure 1 The control method of the oral manipulation device in this disclosure is described. Figure 1 A flowchart illustrating a control method according to some embodiments of the present disclosure is shown.

[0042] like Figure 1 As shown, the control method for an oral manipulation device may include: Step S1, acquiring multiple types of first data, including oral imaging data, oral scan data, and oral record data; Step S2, performing multimodal fusion on the first data according to the mapping relationship between different types of first data to generate an oral model, wherein the oral model includes multiple structural layers, and each of the multiple structural layers corresponds to a type of first data; Step S3, generating a planned path for the oral manipulation device based on the oral model; Step S4, controlling the oral manipulation device to move to a target position according to the planned path.

[0043] In step S1, multimodal data acquisition can be performed first. For example, data from different sources can be read through paths such as device interfaces, system interfaces, or data file interfaces.

[0044] In the first data type, oral imaging data includes, for example, radiographic images acquired through medical imaging equipment, used to characterize the deep hard tissue structures within the oral cavity. For instance, oral imaging data may include cone-beam computed tomography (CBCT) images, used to characterize the spatial location and morphological information of anatomical structures such as the dentition, jawbone, temporomandibular joint, and periodontal tissues.

[0045] Oral scan data, such as optical surface scan data acquired through intraoral scanning equipment, is used to characterize the geometry of soft and hard tissues in the oral cavity or the external surface of the face. For example, oral scan data can include three-dimensional scan data of the entire dentition and oral soft tissues, used to characterize precise geometric information of surface structures such as tooth crown shape, gingival contour, and mucosal morphology.

[0046] Oral recording data, such as contact relationship data of the upper and lower jaw teeth collected by an occlusal recording device, is used to characterize the location, timing, and force distribution characteristics of occlusal contact. For example, oral recording data can include digital occlusal recording data, used to characterize occlusal function information such as the distribution of occlusal contact points, occlusal contact sequence, occlusal contact area, and occlusal force of the upper and lower jaws.

[0047] By using the various types of data mentioned above, the information in the oral cavity model obtained through data fusion can be more comprehensive and accurate, thereby improving the precision of the control process.

[0048] In step S2, after data acquisition, the acquired multimodal data can be fused.

[0049] In some embodiments, performing multimodal fusion on the first data to generate an oral cavity model based on the mapping relationship between different types of first data may include: preprocessing the first data, wherein the preprocessing includes at least one of artifact suppression, structural segmentation, three-dimensional reconstruction, noise removal, hole detection, surface smoothing, and coordinate normalization.

[0050] In the above embodiments, data preprocessing can be performed before data fusion. For example, data noise reduction, calibration, and alignment algorithms can be used to automatically remove interference information from the data and accurately align multi-source data to ensure data consistency and accuracy.

[0051] Furthermore, different preprocessing methods can be used for different types of data. For example, for oral imaging data, such as CBCT data, data preprocessing can be performed through artifact suppression, anatomical structure segmentation, and 3D reconstruction. For oral scan data and oral record data, data preprocessing can be performed through noise removal, hole inspection, surface smoothing, and coordinate standardization.

[0052] Artifact suppression, for example, uses algorithms to suppress or eliminate interference such as metal artifacts or motion artifacts in image data, thereby improving image clarity and structural discernibility and avoiding interference from artifacts in subsequent analysis.

[0053] Anatomical segmentation, for example, is based on grayscale, texture, or shape features to separate and extract specific anatomical structures from the background and surrounding tissues, providing a basis for subsequent analysis and processing.

[0054] Three-dimensional reconstruction, for example, involves reconstructing a series of two-dimensional tomographic images into a three-dimensional model using algorithms such as volume rendering, so that it can be integrated with other types of data later.

[0055] Noise removal, for example, involves using filtering algorithms to remove random noise and outliers from image or point cloud data, thereby improving the signal-to-noise ratio and ensuring the accuracy of subsequent segmentation, registration, and other processing.

[0056] Hole detection, for example, involves identifying and locating holes in 3D models or point cloud data, i.e., missing data areas, so that subsequent data repair can be performed based on boundary topological features.

[0057] Surface smoothing, for example, is the smoothing process used to smooth the surface of a model, eliminating scanning staircase effects and surface jaggedness, making the model surface closer to the natural smooth shape of real tissue.

[0058] Coordinate standardization, for example, involves operations such as translation, rotation, and scaling to unify data from various sources into the same standard coordinate system or reference frame, eliminating differences in acquisition angles and image sizes, and establishing a unified benchmark for subsequent data processing.

[0059] The data preprocessing methods have been introduced in detail above. Below, we will combine... Figure 2 We will continue to introduce how to perform multimodal data fusion. Figure 2 A flowchart illustrating multimodal fusion according to some embodiments of the present disclosure is shown.

[0060] like Figure 2 As shown, step S2, which involves multimodal fusion of the first data based on the mapping relationship between different types of first data to generate an oral cavity model, may include: step S21, determining an initial mapping relationship between different types of first data using a first feature, wherein the first feature exists in each type of first data; step S22, registering the multiple types of first data according to the initial mapping relationship to determine a target mapping relationship between the different types of first data; and step S23, multimodal fusion of the first data based on the target mapping relationship to generate the oral cavity model.

[0061] In step S21, initial registration between data can be performed using cross-modal corresponding features.

[0062] Cross-modal features are those that exist in various types of data and can be used as reference points in the data fusion process. For example, if there are enough natural teeth in the collected data of multiple types, natural teeth can be used as the first feature, i.e., a cross-modal feature, for initial registration.

[0063] After determining the first feature, initial registration can be performed using this feature. For example, based on the vector between the image center and the first feature, the initial mapping relationship between different types of first data can be determined through the spatial variation relationship between the vectors.

[0064] It should be understood that the above-described natural teeth are merely examples, and other features can also be selected as the first feature to assist in initial registration. Furthermore, the first feature can be a single specified feature or multiple specified features. Using a small number of features for initial registration can improve the efficiency of subsequent registration.

[0065] In step S22, based on the above initial mapping relationship, further fine registration can be performed between the data.

[0066] The fine registration in the above steps can be carried out by a variety of registration methods, such as surface registration, feature registration, and deep learning registration.

[0067] Surface iterative registration, for example, starts from the initial pose and iteratively finds the nearest point pairs between the data model or data point cloud, thereby continuously optimizing the rigid body transformation between the data and gradually minimizing the distance error between corresponding points.

[0068] The initial mapping relationship determined in step S21 is used in surface iterative registration, for example, to determine the initial pose of the iterative process, so that the iteration starts at a more accurate initial position and speeds up the efficiency of iterative solution.

[0069] Feature registration, for example, involves further extracting discriminative feature points or feature point clouds from point clouds or images, establishing correspondences through feature matching, and then solving the mapping relationship between the data.

[0070] The above initial mapping relationship is used in feature registration, for example, to determine the matching features or feature point cloud ranges between data, so that matching features can be determined within the range and the correspondence between features can be established quickly.

[0071] Deep learning registration, for example, is based on learning and training machine learning models to perform end-to-end registration transformations, and then applying the trained neural network to configure data, thereby achieving data-driven automatic registration.

[0072] The aforementioned initial mapping relationship is used in deep learning registration, for example, as a reference parameter provided to the machine learning model, thereby improving the accuracy and efficiency of model registration.

[0073] Based on the aforementioned multiple registration methods, registering the multiple types of first data according to the initial mapping relationship to determine the target mapping relationship may include: registering the multiple types of first data using one of the multiple registration methods to obtain a first mapping relationship; determining the target mapping relationship based on the first mapping relationship when the deviation between the first mapping relationship and the initial mapping relationship is less than or equal to a preset threshold; updating the initial mapping relationship again using other registration methods among the multiple registration methods until a second mapping relationship is obtained, and determining the target mapping relationship based on the second mapping relationship, wherein the deviation between the second mapping relationship and the initial mapping relationship is less than or equal to the preset threshold.

[0074] In other words, registration can be performed first using one of several registration methods to obtain the spatial variation matrix between different modal data, and the registration quality can be evaluated after registration. If the registration quality is insufficient, registration can be performed again using other methods.

[0075] As an example, if the initial mapping between image data and scanned data is a 50-degree rotation, in the case of fine registration through surface iterative registration, the 50-degree rotation can be used as the initial value for iteration. For instance, the rotation angle can be gradually updated based on the deviation between a specific pair of points in the image data after rotating it 50 degrees and the corresponding pair of points in the scanned data.

[0076] If the preset threshold is 10 degrees, and the first mapping relationship obtained after iterative update is, for example, rotated by 55 degrees, the deviation between the first mapping relationship and the initial mapping relationship is a rotation of 5 degrees, which is less than the preset threshold.

[0077] Under the above circumstances, it can be considered that the mapping relationship obtained by this fine registration is close to the initial prediction, the registration process has high reliability, and the registration quality meets the requirements. Therefore, the above first mapping relationship can be directly determined as the final target mapping relationship.

[0078] If the first mapping relationship obtained after iterative update is, for example, rotated by 70 degrees, the deviation between the first mapping relationship and the initial mapping relationship is a rotation of 20 degrees, which is greater than the preset threshold.

[0079] In the above situation, it can be assumed that the mapping relationship obtained from this registration differs significantly from the initial prediction, indicating that there may be some problems in the registration process, and the registration quality cannot meet the requirements. The aforementioned first mapping relationship cannot be directly determined as the target mapping relationship.

[0080] If the registration quality cannot meet the requirements, other methods can be used to re-register, such as feature registration or deep learning registration.

[0081] After feature registration, if the obtained mapping relationship is, for example, rotated by 56 degrees, and the deviation is 6 degrees which is less than the threshold, then the mapping relationship obtained by feature registration can be used as the second mapping relationship, and the target mapping relationship can be further determined.

[0082] It should be understood that the above method for determining the registration quality is merely exemplary. The registration quality of the mapping relationship obtained by registration can also be evaluated based on specific indicators such as the distance between anatomical landmarks, the average deviation of the overlapping surface, the root mean square error, and the maximum local deviation.

[0083] In addition, if the registration error exceeds the preset threshold and none of the registration methods can meet the requirements, you can reselect the registration area, adjust the registration parameters, or output a prompt message for confirmation.

[0084] The previous section detailed how to register data. Below, we will continue by explaining how to fuse data to generate a dental model.

[0085] In some embodiments, the plurality of structural layers of the oral cavity model include an anatomical structural layer, a surface structural layer, a soft tissue structural layer, and an occlusal contact layer. The oral cavity imaging data includes computed tomography (CT) imaging data, the oral cavity scan data includes intraoral scan data and facial scan data, and the oral cavity recording data includes occlusal recording data.

[0086] In the above embodiments, the process of multimodal fusion of the first data to generate an oral cavity model based on the mapping relationship between different types of first data may include: generating the anatomical structure layer based on the computed tomography image data; generating the surface structure layer based on the intraoral scan data; generating the soft tissue structure layer based on the facial scan data; generating the occlusal contact layer based on the occlusal record data; and fusing the anatomical structure layer, the surface structure layer, the soft tissue structure layer, and the occlusal contact layer according to the mapping relationship to generate the oral cavity model.

[0087] In other words, during the process of multimodal data fusion, corresponding data layers can be constructed based on the anatomical and functional information reflected by different data.

[0088] Computed tomography (CBCT) data can be used to construct anatomical layers of the jawbone, alveolar bone, tooth roots, nerve canals, maxillary sinuses, and mental foramen within the oral cavity. Intraoral scan data can be used to construct surface structural layers of teeth, gingiva, and edentulous areas within the oral cavity. Facial scan data can be used to construct reference layers of facial soft tissue structures and facial aesthetics. Occlusal recording data can be used to determine the spatial transformation relationship between the maxilla and mandible and the occlusal contact layer.

[0089] After registration is completed, different types of data can be integrated into a unified oral model in a hierarchical manner. The fused oral model can simultaneously present reference information from various aspects, such as critical anatomical structures like the jawbone, high-precision dentition and soft tissue surfaces, maxillary and mandibular occlusal relationships, and facial aesthetic reference information, providing a data foundation for subsequent path planning and control.

[0090] In addition, the oral model described above can retain the data source, transformation matrix, and credibility information of each data layer to facilitate subsequent data tracing.

[0091] The preceding text elaborated on how to perform multimodal data fusion to generate oral cavity models. The following section will introduce... Figure 3 Continuing to introduce Figure 1 How to perform path planning based on the generated model in step S3. Figure 3 A flowchart illustrating the generation of a planned path according to some embodiments of this disclosure is shown.

[0092] like Figure 3 As shown, step S3, generating the planned path of the oral manipulation device based on the oral model, includes: step S31, determining the second feature in the oral model based on the grayscale features, morphological features and continuity of the oral model, wherein the second feature includes at least one of anatomical structure, edentulous area parameters and occlusal features; step S32, generating the planned path based on the second feature.

[0093] In step S31, feature extraction from the oral cavity model can be performed first.

[0094] For example, by using artificial intelligence semantic segmentation and instance segmentation methods, combined with grayscale features, geometric shapes, spatial locations and anatomical continuity in the model, structures such as the jawbone, alveolar bone, teeth, tooth roots, inferior alveolar nerve canal, mental foramen and maxillary sinus in the oral cavity can be automatically identified.

[0095] For example, features such as the crown, gingiva, edentulous area, and alveolar ridge surface can be identified through the surface structure layer in the model, that is, the structure layer corresponding to the intraoral scan data, and spatial correspondence can be made with the identification results of other structure layers in the model.

[0096] After completing the initial anatomical structure identification, the identification results can be corrected by rules such as the connectivity between structures, tubular morphology, and positional relationship between adjacent structures, thereby achieving accurate structural identification in the oral cavity model.

[0097] In addition, the location and number of missing teeth in the oral model can be used to automatically identify parameters of the edentulous area in the oral cavity, such as the alveolar bone width, height, and bone density of the edentulous area, so as to assess the feasibility of performing various operations such as dental implantation using oral manipulation devices.

[0098] Furthermore, based on the oral cavity model, it can automatically analyze the occlusal characteristics of the oral cavity, such as occlusal type (e.g., neutral occlusion, mesial occlusion, distal occlusion) and occlusal force distribution, thereby identifying occlusal interference points and providing a reference for subsequent path planning.

[0099] In step S32, a planned path for controlling the oral manipulation device can be generated based on the features extracted from the oral cavity model.

[0100] In some embodiments, step S32, generating the planned path according to the second feature, may include: determining the motion constraints of the oral manipulation device and the parameters of the target position according to the second feature, wherein the motion constraints include the motion space of the oral manipulation device; and generating the planned path according to the motion constraints and the parameters of the target position.

[0101] In the above embodiments, the dangerous areas in the operation process can be determined by extracting the features, thereby realizing the path planning of the oral manipulation device.

[0102] The following section will use the implant procedure performed by an oral manipulation device as an example to illustrate the above-mentioned path planning process.

[0103] For the above operations, the target location of the oral manipulation device can be, for example, the edentulous area within the oral cavity. Therefore, based on the features extracted from the model, parameters of the edentulous area within the oral cavity, such as the location of the edentulous area, alveolar bone conditions, and occlusal relationships at the edentulous area, can be determined as parameters for the target location.

[0104] The parameters of the target location mentioned above can be used to determine parameters such as planting position, planting angle, and planting depth during the planting operation, so as to facilitate subsequent path planning.

[0105] Motion constraints on oral manipulation devices can be, for example, safe operating boundaries or movement spaces during the movement of a robotic arm. These safe operating boundaries or movement spaces can be determined based on dangerous areas within the oral cavity, such as bone tissue, nerves, and the maxillary sinus. By avoiding these dangerous areas, safe intraoral manipulation can be ensured.

[0106] Based on the aforementioned movement constraints, the operating path of the oral manipulation device can be determined according to the implant location, implant angle, and implant depth, so that the device can safely complete the aforementioned operations.

[0107] Furthermore, for implantation procedures, the implants and prostheses used in the above procedures can be determined based on the second features extracted from the model.

[0108] For example, based on the above parameters of the target location, such as the location of the edentulous area, alveolar bone conditions, and occlusal relationship in the edentulous area, the diameter and length of the implant can be automatically matched, thereby recommending the model and specifications of the implant.

[0109] After the implant is determined, the corresponding prosthesis can be determined based on the extracted features.

[0110] For example, the type of prosthesis can be determined based on the parameters of the implant, combined with the characteristics of the occlusal relationship and facial soft tissue structure, such as single crown restoration, multi-crown restoration, bridge restoration, attachment restoration, etc.

[0111] Furthermore, it can automatically design the shape, size, occlusal surface morphology, and materials of the restoration based on the type of restoration, ensuring that the restoration is coordinated with other adjacent teeth and has a good occlusal relationship.

[0112] By extracting feature parameters from the model to determine the implant and its corresponding prosthesis, the coordination and adaptation between various steps of the procedure can be improved. Specifically, by linking implant planning data and prosthesis design data in real time, the fixation method and connection accuracy of the prosthesis can be automatically adjusted to ensure a precise fit between the prosthesis and the implant, avoiding problems such as poor fixation and occlusal interference.

[0113] Based on the oral cavity model mentioned above, a three-dimensional simulation model of the restored oral cavity can also be generated to demonstrate the effect of the implant procedure.

[0114] The above text uses the example of performing implant procedures with a dental manipulator to illustrate how to determine the planning path of the dental manipulator.

[0115] In addition to specific operations, dental manipulation devices can also have multiple types of operating modes. In light of the above, generating the planned path based on the second feature may include: determining the operating mode of the dental manipulation device based on the second feature; and generating the planned path based on the operating mode.

[0116] For example, when an oral manipulation device has two operation modes, implantation and bone augmentation, after determining the implant for the implantation operation, the expected stability of the implant can be assessed based on parameters such as alveolar bone width, height, cortical bone thickness, and relative bone quality, combined with relevant implant parameters.

[0117] If the expected stability meets the requirements, the operation mode of the oral manipulation device can be determined as implantation, and the corresponding planning path can be generated according to the method described above.

[0118] In cases where the expected stability is low, such as insufficient alveolar bone volume, which means that the implantation procedure cannot be performed directly, the operation mode of the oral manipulation device can be determined to be bone augmentation operation to generate a corresponding planning path for subsequent implantation and other operations.

[0119] After generating the planned path, the operational accuracy of the planned path can be verified by comparing it with the anatomical structures in the oral cavity model. This includes verifying the precision of the target location's coordinates, angles, and depth. If the operational accuracy is low, meaning the deviation between the planned path and the actual structures in the oral cavity model exceeds a threshold, the planned path can be automatically optimized and adjusted based on this deviation.

[0120] After determining the final planned path, in step S4, the oral manipulation device can be controlled to execute according to the planned path, thereby realizing the automatic control of the oral manipulation device.

[0121] The above text combines Figure 1 One implementation method of the control approach is introduced. By performing multimodal fusion on multiple types of data, an accurate oral cavity model is generated, and subsequent path planning is performed based on the oral cavity model to achieve high-precision automatic control of the oral cavity manipulation device.

[0122] Below, we will combine Figure 4 This section introduces another way to implement the control method. Figure 4 A flowchart illustrating a control method according to other embodiments of the present disclosure is shown.

[0123] like Figure 4 As shown, in Figure 1 Based on this, the control method may further include: step S5, receiving second data, the second data including feedback data of the oral manipulation device during the movement process; step S6, updating the planned path according to the second data.

[0124] In other words, the above embodiments can also update the planned path in real time based on feedback data from the oral manipulation device during operation.

[0125] The second data fed back by the oral manipulation device may include, for example, at least one of the following: the position, motion state, force, torque, drive current, drilling direction, and drilling depth of the oral manipulation device.

[0126] The aforementioned data may be collected in real time or at preset time intervals. Furthermore, the second data may also include, for example, positional change information of the manipulated object obtained by visual positioning, optical positioning, or scanning devices.

[0127] In some embodiments, updating the planned path based on the second data includes: determining the actual path of the oral manipulation device based on the second data and the oral model; and dynamically adjusting the planned path based on the deviation between the actual path and the planned path.

[0128] In other words, the second data can be used for registration verification of the planned path, monitoring of execution deviations, and local dynamic adjustment of the planning scheme.

[0129] For example, after receiving the second data, the aforementioned feedback data can be converted into the coordinate system of the oral cavity model, and the position of the oral cavity manipulation device at each moment can be determined based on the feedback data, thereby determining the actual path of the oral cavity manipulation device.

[0130] After determining the actual path of the device, the path can be adjusted according to the deviation between the actual path and the planned path, so that the oral manipulation device can move from the current actual position to the target position, realizing dynamic adjustment during the operation.

[0131] In addition to dynamically adjusting the path, feedback data can also be used to verify the anatomical structures in the oral cavity model. For example, based on the feedback force and torque of the oral cavity manipulator, it can be determined whether the anatomical structures around the manipulator are bone or soft tissue structures, thereby validating the oral cavity model.

[0132] Furthermore, during the implantation procedure, the stability of the implant can be further determined. For example, during the implantation procedure, real-time data such as drilling resistance, force or torque, implantation torque, resonance frequency analysis results, and path deviation of the dental manipulation device can be used to dynamically verify the initial stability determined above.

[0133] As an example, based on the resistance of the oral manipulation device, the stress, strain, and interfacial micromotion risk of the peri-implant bone tissue can be independently reviewed through finite element analysis or a validated mechanical surrogate model.

[0134] If the actual bone density in the oral cavity is determined to be less than the bone density extracted from the oral model, the initial stability can be considered relatively high. At this point, the stability assessment of the implant can be dynamically reduced based on feedback data. If the stability decreases below a threshold, the result can be fed back to the path planning unit to readjust the implant specifications, placement, angle, depth, or to change to another operating path.

[0135] The above describes the control method proposed in this disclosure. By performing multimodal fusion on multiple types of data, an accurate oral cavity model is generated, and subsequent path planning is performed based on the oral cavity model to achieve high-precision automatic control of the oral cavity manipulation device.

[0136] Below, we will combine Figure 5 This paper presents a specific example of controlling an oral manipulation device according to the control method proposed in this disclosure.

[0137] Figure 5 This illustration shows a flowchart of controlling an oral manipulation device to perform implantation procedures in some embodiments of this disclosure. For example... Figure 5 As shown, the control method may include processes 501 to 511.

[0138] In process 501, multimodal data can first be received and preprocessed. Multimodal data may include, for example, CBCT image data, intraoral scan data, occlusal records, facial scan data, and periodontal data.

[0139] Through data preprocessing, interference information in image data can be automatically removed, and multi-source data can be accurately aligned to ensure data consistency and accuracy. Furthermore, it supports automatic data format conversion to adapt to various digital devices and planning algorithms.

[0140] In process 502, multimodal data fusion can be performed. Through data registration, the data received in process 501 can be fused in multiple modes to generate a dental model in a unified coordinate system.

[0141] As an example, the initial registration method can be determined based on whether there is sufficient natural tooth overlap between the data.

[0142] When there is sufficient overlap of natural teeth, initial registration can be completed based on features of the natural tooth surface, such as cusps, pits, fissures, and incisal edges.

[0143] In the absence of sufficient natural tooth overlap, initial registration can be completed based on other auxiliary references, such as braces, occlusion records, clamps, or pre-defined markers.

[0144] After the initial registration, further fine-grained registration of the data can be completed, such as unifying the spatial coordinate system of each data point and generating spatial transformation matrices for each modality.

[0145] Different types of data can correspond to different structural layers. After fusing the data from various modalities, a complete oral cavity model can be generated.

[0146] In process 503, after the model is generated, the reliability of the model can be determined based on the registration quality to see if it meets the requirements.

[0147] If the reliability of the oral model does not meet the requirements, the process can proceed to step 504, for example, by reselecting the registration method, re-registering the registration area in step 502, or outputting a prompt message for manual confirmation.

[0148] If the oral cavity model meets the reliability requirements, the process can proceed to step 505 to perform structural recognition based on the oral cavity model.

[0149] In process 505, dangerous structures, edentulous areas, bone structures, soft tissue structures, etc., can be identified from the oral model, and the occlusal relationship within the oral cavity can be determined.

[0150] In process 506, based on the above structure, collaborative planning for implantation and restoration can be carried out. This planning may include, for example, two parts: operational pathway planning and collaborative design of the restoration.

[0151] Operation path planning can be achieved, for example, by determining the safety space during device operation based on the model, as described above, thereby determining the planned path so that the device can safely reach the target location.

[0152] In addition, for the planting operation, the above planning process can also determine parameters such as the diameter and length of the implant in order to recommend a suitable implant model.

[0153] Collaborative prosthesis design can be achieved by generating an initial prosthesis based on the parameters of the implant, and then optimizing the material, shape, and size of the initial prosthesis based on factors such as adjacency and soft tissue constraints, occlusion and functional constraints, aesthetic and morphological constraints, and connection and manufacturing constraints, ultimately determining a prosthesis that matches the implant.

[0154] Once the implant and prosthesis are determined, parameters such as stability, safety distance, prosthetic fit, occlusal function, and facial aesthetics can be comprehensively considered to synergistically optimize the implant and prosthesis parameters. For example, these parameters can be iteratively adjusted until a suitable match is achieved between the implant and prosthesis parameters.

[0155] In process 507, after the above solution is determined, the solution can be verified and a comprehensive evaluation can be conducted to determine whether the solution meets the standards.

[0156] For example, the plan can be evaluated based on multiple dimensions such as safety, stability, adaptability, interlocking relationship, mechanical relationship, or manufacturability to determine whether the plan meets the standards.

[0157] If the proposed solution does not meet the requirements, you can proceed to step 508 to adjust the parameters or weights in the planning process and replan the solution.

[0158] If the plan meets the requirements, you can proceed to step 509 and control the oral cavity operating device to execute the above plan.

[0159] In process 510, during execution, feedback data from the oral manipulation device can be received to perform closed-loop detection and adjustment of the operation.

[0160] For example, feedback data from the oral manipulation device can be used to detect data registration during the model generation process and path deviations during the execution process, thereby updating the planned path or pausing execution if the deviation exceeds a threshold.

[0161] In process 511, after execution, data archiving can be performed to store the initial multimodal data, the generated oral cavity model, the determined planning path, and the actual execution path, so as to serve as a reference for subsequent path planning.

[0162] Through traceable closed-loop control of implant restoration, the precision and success rate of implant procedures can be improved. Furthermore, fully automated planning throughout the process can enhance the operational efficiency of dental manipulation devices.

[0163] The above describes the control method for the oral manipulation device provided in the embodiments of this disclosure. See below for further details. Figure 6 and Figure 7 The control device of the oral manipulation apparatus according to embodiments of the present disclosure is described, which is used to execute any of the above-described scheduling methods. Figure 6 A block diagram of a control device according to some embodiments of the present disclosure is shown.

[0164] like Figure 6As shown, the control device 6 of the oral manipulation device includes: a data acquisition module 61, configured to acquire multiple types of first data, including oral imaging data, oral scan data, and oral record data; a model generation module 62, configured to perform multimodal fusion of the first data according to the mapping relationship between different types of first data to generate an oral model, wherein the oral model includes multiple structural layers, and each of the multiple structural layers corresponds to a type of first data; a path generation module 63, configured to generate a planned path for the oral manipulation device according to the oral model, the planned path being used to move the oral manipulation device to a target position; and a control module 64, configured to control the oral manipulation device according to the planned path.

[0165] The data acquisition module 61 of the control device 6 can be used, for example, to perform... Figure 1 Step S1. The model generation module 62 of the control device 6 can, for example, be used to perform... Figure 1 Step S2. The path generation module 63 of the control device 6 can, for example, be used to execute... Figure 1 Step S3. The control module 64 of the control device 6 can, for example, be used to perform... Figure 1 Step S4.

[0166] It should be understood that the control device may also include other modules for performing other steps in the control method of the embodiments of this disclosure, such as a stability determination module, a second data receiving module, etc.

[0167] Figure 7 A block diagram of a control device according to other embodiments of the present disclosure is shown.

[0168] like Figure 7 As shown, the control device 7 includes: at least one memory 71; and at least one processor 72 coupled to the at least one memory 71, the at least one processor 72 being configured to execute the scheduling method as described in any of the foregoing embodiments based on instructions stored in the at least one memory 71.

[0169] Memory 71 is used to store one or more computer-readable instructions. Memory 71 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory, dynamic random access memory, static random access memory, read-only memory, and flash memory. Memory 71 may, for example, store operating systems, application programs, boot loaders, databases, and other programs, as well as various application programs and various data.

[0170] Processor 72 is configured to execute computer-readable instructions to implement the scheduling method described in any of the foregoing embodiments. Specific implementations of each step of the method can be found in the above embodiments, for example... Figure 1 The steps involved are repeated here, so the details will not be repeated.

[0171] The processor 72 can be various processing devices, such as a central processing unit (CPU), a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The CPU can be based on x86 or ARM architectures, etc.

[0172] The processor 72 and the memory 71 can communicate with each other directly or indirectly. For example, the processor 72 and the memory 71 can communicate via a network. The network can include a wireless network, a wired network, and / or any combination of wireless and wired networks. The processor 72 and the memory 71 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0173] It should be noted that Figure 7 The components of the control device 7 shown are merely exemplary and not limiting; the control device 7 may have other components depending on the actual application requirements. The processor 72 can operate other components in the control device 7 to perform desired functions. The control device 7 can be implemented by software, firmware, and / or hardware and can be integrated into a device with the relevant application installed.

[0174] Based on the control device described above, this disclosure also proposes a system. Figure 8 This is a block diagram illustrating a system for implementing some embodiments of the present disclosure.

[0175] like Figure 8 As shown, the system may include: a control device 81 as described in any embodiment of this disclosure; and an oral manipulation device 82 configured to operate according to a planned path generated by the control device 81.

[0176] The above system can automatically plan and accurately execute various types of oral procedures, improving operational efficiency and accuracy, and realizing automatic control of oral operating devices.

[0177] Figure 9 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0178] like Figure 9 As shown, computer system 9 can be represented as a general-purpose computing device. Computer system 9 includes memory 91, processor 92, and bus 90 connecting different system components.

[0179] The memory 91 can be various forms of computer-readable storage media, such as system memory, non-volatile storage media, etc. System memory may store, for example, an operating system, application programs, a bootloader, and other programs. System memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. Non-volatile storage media may store, for example, instructions for executing corresponding embodiments of the scheduling method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0180] Processor 92 can be implemented using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module can be implemented by executing instructions in the central processing unit (CPU) memory to perform the corresponding steps, or by implementing dedicated circuits to perform the corresponding steps.

[0181] Bus 90 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0182] The computer system 9 may also include an input / output interface 93, a network interface 94, and a storage interface 95. These interfaces 93, 94, and 95, as well as the memory 91 and processor 92, can be connected via a bus 90. The input / output interface 93 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 94 provides a connection interface for various networked devices. The storage interface 95 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0183] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to implement the scheduling method described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, the computer instructions containing program code for performing the methods shown in the flowcharts.

[0184] Various embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0185] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for controlling an oral manipulation device, comprising: Acquire multiple types of first data, including various types of oral imaging data, oral scan data, and oral record data; Based on the mapping relationship between different types of first data, the first data is fused in a multimodal manner to generate an oral cavity model, wherein the oral cavity model includes multiple structural layers, and each of the multiple structural layers corresponds to a type of first data; Based on the oral cavity model, the planned path of the oral cavity manipulation device is generated; According to the planned path, the oral manipulation device is controlled to move to the target position.

2. The control method according to claim 1, wherein, Based on the mapping relationship between different types of first data, the first data is fused in a multimodal manner to generate an oral cavity model, including: The first feature determines the initial mapping relationship between different types of first data, and the first feature exists in each type of first data; Based on the initial mapping relationship, the first data of the multiple types are registered to determine the target mapping relationship between the first data of different types; Based on the target mapping relationship, the first data is fused in a multimodal manner to generate the oral cavity model.

3. The control method according to claim 2, wherein, Based on the initial mapping relationship, the registration of the multiple types of first data is performed to determine the target mapping relationship, including: The first data of the multiple types are registered using one of the multiple registration methods to obtain a first mapping relationship. The multiple registration methods include at least one of surface registration, feature registration, and deep learning registration. If the deviation between the first mapping relationship and the initial mapping relationship is less than or equal to a preset threshold, the target mapping relationship is determined based on the first mapping relationship. If the deviation between the first mapping relationship and the initial mapping relationship is greater than the preset threshold, the initial mapping relationship is updated again using other registration methods among the multiple registration methods until a second mapping relationship is obtained. Based on the second mapping relationship, the target mapping relationship is determined, wherein the deviation between the second mapping relationship and the initial mapping relationship is less than or equal to the preset threshold.

4. The control method according to claim 1, wherein, The multiple structural layers include an anatomical structural layer, a surface structural layer, a soft tissue structural layer, and an occlusal contact layer. The oral imaging data includes computed tomography (CT) imaging data. The oral scanning data includes intraoral scanning data and facial scanning data. The oral recording data includes occlusal recording data. Based on the mapping relationship between different types of first data, the first data is fused in a multimodal manner to generate an oral model, including: The anatomical structure layer is generated based on the computed tomography (CT) image data; The surface structure layer is generated based on the intraoral scan data; The soft tissue structure layer is generated based on the facial scan data; The occlusal contact layer is generated based on the occlusal record data; Based on the mapping relationship, the anatomical structure layer, the surface structure layer, the soft tissue structure layer, and the occlusal contact layer are fused to generate the oral cavity model.

5. The control method according to claim 1, wherein, Based on the mapping relationship between different types of first data, the first data is fused in a multimodal manner to generate an oral cavity model, including: The first data is preprocessed, and the preprocessing includes at least one of artifact suppression, structure segmentation, 3D reconstruction, noise removal, hole detection, surface smoothing, and coordinate normalization.

6. The control method according to claim 1, wherein, Based on the oral cavity model, the planned path for the oral manipulation device includes: Based on the grayscale features, morphological features, and continuity of the oral cavity model, a second feature is determined in the oral cavity model. The second feature includes at least one of the following: anatomical structure, edentulous area parameters, and occlusal features. The planned path is generated based on the second feature.

7. The control method according to claim 6, wherein, Based on the second feature, generating the planned path includes: Based on the second feature, the motion restriction conditions of the oral manipulation device and the parameters of the target position are determined, wherein the motion restriction conditions include the motion space of the oral manipulation device; The planned path is generated based on the motion constraints and the parameters of the target location.

8. The control method according to claim 6, wherein, Based on the second feature, generating the planned path includes: Based on the second feature, the operation mode of the oral manipulation device is determined; The planned path is generated according to the described operation method.

9. The control method according to claim 1, further comprising: Receive second data, which includes feedback data of the oral manipulation device during movement; The planned path is updated based on the second data.

10. The control method according to claim 9, wherein, The second data includes at least one of the following: the position, motion state, force, torque, drive current, drilling direction, and drilling depth of the oral manipulation device. Updating the planned path based on the second data includes: Based on the second data and the oral cavity model, the actual path of the oral cavity manipulation device is determined; The planned path is dynamically adjusted based on the deviation between the actual path and the planned path.

11. A control device for an oral manipulation apparatus, comprising: The data acquisition module is configured to acquire multiple types of first data, including various types of oral imaging data, oral scan data, and oral record data. The model generation module is configured to perform multimodal fusion on the first data according to the mapping relationship between different types of first data to generate an oral cavity model, wherein the oral cavity model includes multiple structural layers, and each of the multiple structural layers corresponds to a type of first data; The path generation module is configured to generate a planned path for the oral manipulation device based on the oral cavity model, the planned path being used to move the oral manipulation device to a target position; The control module is configured to control the oral manipulation device according to the planned path.

12. A control device for an oral manipulation apparatus, comprising: At least one memory; as well as At least one processor coupled to the at least one memory, the at least one processor being configured to execute the control method as described in any one of claims 1 to 10 based on instructions stored in the at least one memory.

13. A system comprising: The control device as described in claim 11 or 12; The oral manipulation device is configured to operate according to a planned path generated by the control device.

14. The system according to claim 13, wherein: The control device is further configured to receive second data, update the planned path based on the second data, and control the oral manipulation device based on the updated planned path, wherein the second data includes feedback data of the oral manipulation device during movement. The oral manipulation device is also configured to send second data to the control device during movement.

15. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the control method as described in any one of claims 1 to 10.

16. A computer program product, when run on a computer, causes the computer to implement the control method as described in any one of claims 1 to 10.

Images