Oral cavity motion tracking method and device, equipment and storage medium

By using probabilistic graphical processing of 3D oral models and 2D images, the problem of auxiliary devices affecting data reliability was solved, enabling tooth and jaw movement tracking without device assistance and improving the accuracy and naturalness of data acquisition.

CN121129485APending Publication Date: 2025-12-16SHINING 3D TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511405282.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-28
Publication Date
2025-12-16

AI Technical Summary

Technical Problem

In existing technologies, when using electronic face bows to track jaw movements, the use of auxiliary devices causes patient discomfort and affects the physiological accuracy of the data.

Method used

By acquiring a 3D oral cavity model and multiple sets of 2D images, and utilizing probabilistic graphs and image residual optimization techniques, the tooth registration pose can be adjusted to generate oral motion tracking results without the need for fixed auxiliary devices in the user's oral cavity.

Benefits of technology

It improves the reliability and accuracy of jaw movement tracking data, allowing users to move naturally and simplifying the data collection process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121129485A_ABST
    Figure CN121129485A_ABST
Patent Text Reader

Abstract

The invention provides an oral cavity motion tracking method and device, equipment and a storage medium. The method comprises the following steps: acquiring a three-dimensional oral cavity model and at least one group of two-dimensional images; generating a first probability graph corresponding to the two-dimensional image; based on the visual angle of the two-dimensional image, generating a second probability graph corresponding to the three-dimensional oral cavity model under the visual angle; aiming at each group of two-dimensional images, according to a first probability graph corresponding to each two-dimensional image in the group of two-dimensional images, a second probability graph corresponding to each two-dimensional image in the group of two-dimensional images, of the three-dimensional oral cavity model, and a current pose of the three-dimensional oral cavity model relative to each two-dimensional image in the group of two-dimensional images, calculating the three-dimensional oral cavity model according to the first probability graph corresponding to each two-dimensional image in the group of two-dimensional images; determining tooth registration poses of the group of two-dimensional images in the shooting time period; and outputting an oral cavity motion tracking result according to the tooth registration poses corresponding to the shooting time periods. In this way, tooth jaw motion tracking can be achieved without depending on auxiliary equipment, and the reliability of collected data is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of oral motion trajectory acquisition technology, and more specifically, to an oral motion tracking method, device, equipment, and storage medium. Background Technology

[0002] In today's rapidly developing field of dentistry and oral medicine, precise measurement and analysis of jaw movements has become a crucial aspect of many clinical applications.

[0003] In existing technologies, dental and jaw movement tracking is typically achieved through electronic face bows. The technical principle of electronic face bows is to use high-precision optical or electromagnetic sensors to capture the movement of an auxiliary device fixed to the patient's teeth, thereby enabling dental and jaw movement tracking.

[0004] The presence of assistive devices can easily cause discomfort to patients and affect natural jaw movement, thereby reducing the physiological accuracy of the data. Summary of the Invention

[0005] The purpose of this application is to provide an oral motion tracking method, device, electronic device, and computer-readable storage medium to solve the problem in related technologies that require the use of auxiliary equipment to achieve dental motion tracking, thereby affecting the reliability of the collected data.

[0006] This application provides an oral motor tracking method, including: Acquire a three-dimensional oral cavity model and at least one set of two-dimensional images, with the difference in shooting time corresponding to the same set of two-dimensional images within a preset range; A first probability map corresponding to the two-dimensional image is generated; the value of each pixel in the first probability map is used to characterize the probability that the pixel belongs to a specified tooth region; Based on the perspective of the two-dimensional image, a second probability map is generated corresponding to the three-dimensional oral cavity model under the perspective; the value of each pixel in the second probability map is used to characterize the probability that the pixel belongs to a specified tooth region; For each set of two-dimensional images, the tooth registration pose of the shooting time period of the set of two-dimensional images is determined based on the first probability map corresponding to each two-dimensional image in the set of two-dimensional images, the second probability map corresponding to the three-dimensional oral model under the view of each two-dimensional image in the set of two-dimensional images, and the current pose of the three-dimensional oral model relative to each two-dimensional image in the set of two-dimensional images. Based on the tooth registration pose corresponding to each of the aforementioned shooting time periods, the oral cavity motion tracking results are output.

[0007] In the above implementation process, a three-dimensional oral cavity model of the user's mouth and multiple sets of two-dimensional images of the mouth are acquired. Then, the position and pose of the three-dimensional oral cavity model are adjusted based on each set of two-dimensional images. This allows the user's oral cavity posture within the shooting time period of each set of two-dimensional images to be reproduced through the three-dimensional oral cavity model, i.e., tooth registration pose, thereby achieving jaw movement tracking. Furthermore, this method does not require the fixation of auxiliary devices in the user's mouth, allowing the user to perform jaw movements in a completely natural state, thus improving the reliability of the acquired data.

[0008] Further, the step of outputting oral motion tracking results based on the tooth registration poses corresponding to each of the shooting time periods includes: driving the three-dimensional oral model in real time based on each of the tooth registration poses in chronological order of the shooting time periods, so as to display the oral motion tracking results on the interactive interface of the display device; or, The two-dimensional images are displayed in the first preset area of ​​the interactive interface in chronological order of shooting time; the three-dimensional oral cavity model is driven in real time based on the registration pose of each tooth in chronological order of shooting time period, so as to display the oral cavity motion tracking results in the second preset area of ​​the interactive interface.

[0009] In the above implementation, the 3D oral cavity model is driven to move in real time based on the tooth registration pose corresponding to each shooting time period. Oral cavity motion tracking results are obtained and displayed, allowing users to access these results through an interactive interface, facilitating data analysis. Furthermore, by displaying the 2D image sequence and the oral cavity motion tracking results generated based on it on the interactive interface, synchronous comparison between the 2D images and the 3D dynamic model can be achieved. This facilitates users in verifying registration accuracy, identifying motion anomalies, and conducting data analysis.

[0010] Further, generating a second probability map of the three-dimensional oral cavity model corresponding to the viewpoint of the two-dimensional image includes: Based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the stated viewpoint, the 3D oral cavity model is rendered to obtain a second probability map corresponding to the 3D oral cavity model from the stated viewpoint.

[0011] In the above implementation, by rendering the 3D oral cavity model relative to the current pose of the 3D oral cavity model and camera parameters based on the perspective of the 2D image, a 2D probability image, i.e., a second probability map, can be drawn from the perspective of the 2D image. This allows for the construction of a probability map error, and by optimizing the error, the pose of the 3D oral cavity model can be adjusted, facilitating the accurate determination of the tooth registration pose in subsequent steps.

[0012] Furthermore, before rendering the three-dimensional oral cavity model based on the current pose of the three-dimensional oral cavity model relative to the two-dimensional image and camera parameters from the stated viewpoint to obtain the second probability map corresponding to the three-dimensional oral cavity model from the stated viewpoint, the oral cavity motion tracking method further includes: Identify the crown region and non-crown region on the three-dimensional oral cavity model, mark the crown region as the rendering region, and / or mark the non-crown region as the non-rendering region; Correspondingly, based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the stated viewpoint, the 3D oral cavity model is rendered, including: Based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the stated perspective, the rendering area on the 3D oral cavity model is rendered.

[0013] In the above implementation, by identifying the crown and non-crown regions on the 3D oral cavity model, the rendering and non-rendering regions can be obtained. Thus, only the crown region on the 3D oral cavity model can be rendered, and the probability map obtained from the rendering can represent the probability that each pixel value belongs to the tooth region. Furthermore, ignoring the non-rendering regions on the 3D oral cavity model reduces the amount of rendering data and improves rendering efficiency.

[0014] Furthermore, generating the first probability map corresponding to the two-dimensional image includes: generating the first probability map corresponding to the two-dimensional image based on a region detection algorithm.

[0015] In the above implementation, a first probability map corresponding to the two-dimensional image is generated using a region detection algorithm. This not only generates the first probability map corresponding to the two-dimensional image but also divides the two-dimensional image into regions, identifying specific tooth regions within the image, so that the first probability map only includes pixels from the specified tooth regions. This reduces computational load and speeds up the determination of tooth registration poses in subsequent processes.

[0016] In the above implementation, after generating the first probability map corresponding to the two-dimensional image based on the region detection algorithm, the oral cavity motion tracking method further includes: Based on the region detection results of the two-dimensional image, the occluded region corresponding to the specified tooth region in the two-dimensional image is identified, and the pixels corresponding to the occluded region are marked as unregistered pixels.

[0017] In the above implementation, by determining the occluded region corresponding to the specified tooth region, pixels in the occluded region can be ignored during the subsequent tooth registration pose determination process based on pixels in the specified tooth region. Since the occluded region may contain erroneous features, excluding the occluded region from the specified tooth region for tooth registration pose determination helps improve the accuracy of tooth registration pose determination.

[0018] Further, for each set of two-dimensional images, based on the first probability map corresponding to each two-dimensional image in the set, the second probability map corresponding to the three-dimensional oral model at the viewpoint of each two-dimensional image in the set, and the current pose of the three-dimensional oral model relative to each two-dimensional image in the set, the tooth registration pose for the time period in which the set of two-dimensional images is captured is determined, including: For each two-dimensional image in the set of two-dimensional images, the image residual corresponding to the two-dimensional image is calculated using the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model under the view of the two-dimensional image. The comprehensive residual is obtained based on the image residuals corresponding to each of the two-dimensional images; Based on the comprehensive residual, the current pose of the three-dimensional oral cavity model is optimized, and the tooth registration pose of the two-dimensional images during the shooting time period is output.

[0019] In the above implementation, by utilizing the image residual between the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model from the viewpoint of the two-dimensional image, and obtaining a comprehensive residual based on the image residuals corresponding to each two-dimensional image, the difference between the three-dimensional oral cavity model and the user's oral cavity posture during the shooting time period of the set of two-dimensional images can be quantified. Simultaneously, the current pose of the three-dimensional oral cavity model is optimized based on the comprehensive residual. This allows for pose adjustment of the three-dimensional oral cavity model based on the quantified difference between the three-dimensional oral cavity model and the user's oral cavity posture during the shooting time period of the set of two-dimensional images, making the pose of the three-dimensional oral cavity model converge with the user's oral cavity posture during the shooting time period of the set of two-dimensional images, thereby achieving jaw movement tracking.

[0020] Further, the designated tooth region includes a whole-jaw tooth region and / or a single-tooth tooth region. For each two-dimensional image in the set of two-dimensional images, using the first probability map corresponding to that two-dimensional image and the second probability map corresponding to the three-dimensional oral model at the viewpoint of that two-dimensional image, the image residual corresponding to that two-dimensional image is calculated, including: For each specified tooth region in the two-dimensional image, the region probability error is calculated based on the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model under the view of the two-dimensional image. The image residual corresponding to the two-dimensional image is calculated based on the sum of the probability errors of all regions.

[0021] In the above implementation, for each specified tooth region in the 2D image, based on the first probability map corresponding to the 2D image and the second probability map corresponding to the 3D oral model from the viewpoint of the 2D image, the regional probability error is calculated. This allows for the division of the 2D image into regions and the parallel calculation of the regional probability error for each region, thereby accelerating the acquisition of image residuals. Furthermore, when the specified tooth region includes single-tooth regions, calculating the regional probability error corresponding to the single-tooth region allows for a more accurate determination of whether adjacent regions belong to the tooth region. Moreover, when the specified tooth region includes both whole-occlusal tooth regions and single-tooth regions, accumulating the regional probability errors of the whole-occlusal tooth regions and the single-tooth regions allows for the integration of global and local errors, resulting in stronger constraints. Therefore, in the subsequent pose adjustment process based on image residuals, the accuracy of the relative positions between teeth can be improved, and the precise pose of individual teeth can be optimized, achieving a balance between overall stability and local accuracy, and enhancing the robustness and accuracy of pose registration.

[0022] Further, the optimization of the current pose of the three-dimensional oral cavity model based on the comprehensive residual, and the output of the tooth registration pose corresponding to the shooting time period, includes: If the overall residual does not exceed the preset residual target, the current pose will be output as the tooth registration pose; If the overall residual exceeds the preset residual target, the current pose is adjusted, and based on the adjusted current pose, the second probability map is generated again and the overall residual is calculated. If the second probability map generation step is executed a preset number of times, the current pose is output as the tooth registration pose.

[0023] Since the overall residual is a quantitative indicator of the matching accuracy between a 3D oral cavity model and a set of 2D images, and the preset residual target is a pre-set tolerance value, meaning that if the overall residual does not exceed the preset residual target, the matching accuracy between the 3D oral cavity model and the set of 2D images can be considered to have reached an acceptable level, in the above implementation, by outputting the current pose as the tooth registration pose when the overall residual does not exceed the preset residual target, tooth and jaw movement tracking can be achieved.

[0024] Furthermore, if the second probability map generation step is executed a preset number of times, the current pose is output as the tooth registration pose. This limits the maximum number of times the second probability map generation step can be executed, avoiding continuous regeneration of the second probability map and recalculation of the comprehensive residual if the overall residual exceeds a preset residual target, thus reducing the waste of computational resources.

[0025] Furthermore, after acquiring the three-dimensional oral cavity model and at least one set of two-dimensional images, the oral cavity motion tracking method further includes: For this set of two-dimensional images, if the set of two-dimensional images includes a depth map, generate the corresponding three-dimensional tooth point cloud based on the depth map; Based on the three-dimensional tooth point cloud and the three-dimensional oral cavity model, the three-dimensional point alignment residual corresponding to the two-dimensional image is calculated; the three-dimensional point alignment residual is used to calculate the comprehensive residual.

[0026] In the above implementation, by obtaining the three-dimensional point alignment residual corresponding to the two-dimensional image and calculating the comprehensive residual based on the three-dimensional point alignment residual, the accuracy of tooth registration pose can be improved.

[0027] Furthermore, after acquiring the three-dimensional oral cavity model and at least one set of two-dimensional images, the oral cavity motion tracking method further includes: For each two-dimensional image in the set of two-dimensional images, based on the two-dimensional image and the three-dimensional oral cavity model, the reprojection residual corresponding to the two-dimensional image is calculated. The reprojection residuals corresponding to each two-dimensional image are accumulated and summed to obtain the three-dimensional reprojection residual. The three-dimensional point reprojection residual is used to calculate the comprehensive residual.

[0028] In the above implementation, by obtaining the three-dimensional point reprojection residual corresponding to the two-dimensional image and calculating the comprehensive residual based on the three-dimensional point reprojection residual, the accuracy of tooth registration pose can be improved.

[0029] Secondly, embodiments of this application provide an oral cavity motion tracking device, comprising: The acquisition module is used to acquire a three-dimensional oral cavity model and at least one set of two-dimensional images, wherein the difference in the shooting time corresponding to the two-dimensional images in the same set is within a preset range. The first generation module is used to generate a first probability map corresponding to the two-dimensional image; the value of each pixel in the first probability map is used to characterize the probability that the pixel belongs to a specified tooth region. The second generation module is used to generate a second probability map of the three-dimensional oral cavity model corresponding to the viewpoint of the two-dimensional image; the value of each pixel in the second probability map is used to characterize the probability that the pixel belongs to a specified tooth region. The determination module is used to determine the tooth registration pose of each set of two-dimensional images during the shooting time period based on the first probability map corresponding to each two-dimensional image in the set of two-dimensional images, the second probability map corresponding to the three-dimensional oral model under the viewpoint of each two-dimensional image in the set of two-dimensional images, and the current pose of the three-dimensional oral model relative to each two-dimensional image in the set of two-dimensional images. The output module is used to output oral motion tracking results based on the tooth registration pose corresponding to each of the shooting time periods.

[0030] Thirdly, embodiments of this application also provide an electronic device, including a processor, a memory, and a communication bus; the communication bus is used to realize the connection and communication between the processor and the memory; the processor is used to execute one or more programs stored in the memory to implement any of the oral motion tracking methods described above.

[0031] Fourthly, this application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement any of the oral motion tracking methods described above. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating an oral motion tracking method provided in an embodiment of this application; Figure 2(a) is a schematic diagram of a three-dimensional oral cavity model provided in an embodiment of this application; Figure 2(b) is a schematic diagram of a three-dimensional oral cavity model provided in an embodiment of this application; Figure 2(c) is a schematic diagram of a three-dimensional oral cavity model provided in an embodiment of this application; Figure 3(a) is a schematic diagram of a two-dimensional image provided in an embodiment of this application; Figure 3(b) is a schematic diagram of a first probability map provided in an embodiment of this application; Figure 3(c) is a schematic diagram of another first probability diagram provided in an embodiment of this application; Figure 4(a) is a schematic diagram of a second probability map provided in an embodiment of this application; Figure 4(b) is a schematic diagram of another second probability diagram provided in an embodiment of this application; Figure 5(a) is a superimposed probability distribution diagram provided in an embodiment of this application; Figure 5(b) is another superimposed probability distribution diagram provided by an embodiment of this application; Figure 6 A flowchart illustrating a method for determining tooth registration pose provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an oral motion tracking device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Example 1: To address the issue that related technologies require auxiliary equipment for tooth and jaw movement tracking, which can affect the reliability of the collected data, this application provides an oral motion tracking method. (See also...) Figure 1 As shown, Figure 1 This is a flowchart illustrating the oral motion tracking method provided in the embodiments of this application, including: Step S101: Obtain a three-dimensional oral cavity model and at least one set of two-dimensional images, wherein the difference in shooting time corresponding to the same set of two-dimensional images is within a preset range.

[0036] Optionally, the three-dimensional oral cavity model is a user's three-dimensional oral cavity model, which may include a three-dimensional oral cavity model of the maxilla and / or a three-dimensional oral cavity model of the mandible.

[0037] For example, when the three-dimensional oral cavity model includes a three-dimensional oral cavity model of the maxilla and a three-dimensional oral cavity model of the mandible, the three-dimensional oral cavity model can be as shown in Figure 2(a). When the three-dimensional oral cavity model includes a three-dimensional oral cavity model of the maxilla, the three-dimensional oral cavity model can be as shown in Figure 2(b). When the three-dimensional oral cavity model includes a three-dimensional oral cavity model of the mandible, the three-dimensional oral cavity model can be as shown in Figure 2(c).

[0038] In some embodiments, a user's oral cavity can be scanned to obtain a three-dimensional oral cavity model. For example, an intraoral scanning device can be used to scan the user's oral cavity to obtain a three-dimensional oral cavity model.

[0039] Two-dimensional images include images of the user's mouth. These can be obtained by photographing the user's mouth, recording video of the user's mouth, and then cropping an image from the recorded video. Alternatively, the user's face can be photographed to obtain a two-dimensional image including the user's mouth, or the user's face can be recorded to obtain a two-dimensional image including the user's mouth.

[0040] In one optional implementation of this application, multiple cameras can be used simultaneously to capture images of the user's mouth from different shooting angles, obtaining a set of two-dimensional images. Since camera delays may vary, even if multiple cameras are triggered simultaneously, there may be a slight time difference in the actual shooting time. Therefore, the time difference between the shooting times of the same set of two-dimensional images is limited to a preset range.

[0041] In one optional implementation of this application, when the user's mouth is in a specific state, at least one camera can be used to capture images of the user's mouth from different perspectives to obtain a set of two-dimensional images.

[0042] A set of two-dimensional images can include data of different modalities, namely texture maps and depth maps. Texture maps can represent information such as color, texture, and boundaries inside the oral cavity. The pixel data corresponding to the texture map stores color distribution information, which can be black and white information (such as representing shades of gray) or color information. Depth maps represent the distance information (i.e., depth value) of each pixel point from the camera.

[0043] The three-dimensional oral cavity model and the two-dimensional image in step S101 are obtained based on the oral cavity of the same user. For example, if the three-dimensional oral cavity model is obtained by scanning the oral cavity of user a, the two-dimensional image is obtained by photographing the oral cavity of user a.

[0044] Step S102: Generate a first probability map corresponding to the two-dimensional image. The value of each pixel in the first probability map is used to characterize the probability that the pixel belongs to a specified tooth region.

[0045] For example, the value range of a pixel can be [0, 1]. When a pixel value is equal to 1, it indicates that the probability of the pixel belonging to the specified tooth region is 100%. When a pixel value is equal to 0, it indicates that the probability of the pixel belonging to the specified tooth region is 0. When a pixel value is equal to 0.3, it indicates that the probability of the pixel belonging to the specified tooth region is 30%.

[0046] In one optional implementation of this application, the two-dimensional image can be divided into regions to obtain multiple specified tooth regions, and a first probability map corresponding to each specified tooth region can be obtained. Correspondingly, the value of each pixel in the first probability map is used to characterize the probability that the pixel belongs to the specified tooth region corresponding to the first probability map. The specified tooth regions may include whole-jaw tooth regions and / or single-tooth tooth regions. The first probability map corresponding to each specified tooth region is the first probability map corresponding to the two-dimensional image.

[0047] For example, as shown in Figure 3(a), when the specified tooth region is the whole jaw tooth region, the two-dimensional image can be divided into regions to obtain the maxillary tooth region and the mandibular tooth region. The first probability map corresponding to the maxillary tooth region can be shown in Figure 3(b), and the first probability map corresponding to the mandibular tooth region can be shown in Figure 3(c). In the first probability maps shown in Figures 3(b) and 3(c), the whiter the color, the greater the probability of belonging to the corresponding specified tooth region, and the darker the color, the smaller the probability of belonging to the corresponding specified tooth region.

[0048] In another optional implementation of this application, for each two-dimensional image in each group of two-dimensional images, a first probability map corresponding to the two-dimensional image can be generated based on a region detection algorithm.

[0049] Step S103: Based on the perspective of the two-dimensional image, generate a second probability map corresponding to the three-dimensional oral cavity model under the perspective; the value of each pixel in the second probability map is used to characterize the probability that the pixel belongs to a specified tooth region.

[0050] Similarly, for each of the two-dimensional images in a set of two-dimensional images, a second probability map of the three-dimensional oral cavity model is generated based on the viewpoint of that two-dimensional image.

[0051] Optionally, based on the viewpoint of the two-dimensional image, a second probability map corresponding to the three-dimensional oral cavity model is generated, including: identifying the crown region on the three-dimensional oral cavity model and marking the crown region as a rendering region. Based on the current pose of the three-dimensional oral cavity model relative to the two-dimensional image and camera parameters under the viewpoint, the rendering region is rendered to obtain the second probability map corresponding to the three-dimensional oral cavity model under the viewpoint. Correspondingly, in identifying the crown region on the three-dimensional oral cavity model, non-crown regions on the three-dimensional oral cavity model can also be identified, and these non-crown regions can be marked as non-rendering regions.

[0052] Optionally, based on the viewpoint of the two-dimensional image, a second probability map corresponding to the three-dimensional oral cavity model under the viewpoint is generated, including: Based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the perspective of the 2D image, the 3D oral cavity model is rendered to obtain the second probability map corresponding to the 3D oral cavity model from the perspective.

[0053] The current pose represents the instantaneous spatial position and orientation of the 3D oral cavity model in the camera coordinate system. It includes the rotation matrix R and the translation vector t. The rotation matrix R represents the rotation of the 3D oral cavity model around the x, y, and z axes in the camera coordinate system, and the translation vector t represents the position coordinates of the preset reference origin of the 3D oral cavity model in the camera coordinate system. The camera coordinate system represents the virtual coordinate system of the virtual space in which the 3D oral cavity model resides.

[0054] Camera intrinsics are a set of parameters that describe the inherent properties of a camera and are used to project points in three-dimensional space onto the camera's two-dimensional image plane.

[0055] In this embodiment of the application, the camera intrinsic parameters of the three-dimensional oral cavity model relative to the two-dimensional image are the same as the camera intrinsic parameters when the two-dimensional image was captured.

[0056] In some embodiments, a differentiable contour renderer can be used to render the 3D oral cavity model based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the viewpoint of the 2D image, thereby obtaining a second probability map of the 3D oral cavity model from the viewpoint.

[0057] In some embodiments, tooth contour information can be further processed after the probability map is generated, and then the tooth contour can be extracted.

[0058] For example, based on the viewpoint of the two-dimensional image shown in Figure 3(a), the current pose of the three-dimensional oral cavity model Figure 2(b) relative to the two-dimensional image Figure 3(a) and the camera parameters, the three-dimensional oral cavity model rendering Figure 2(b) is rendered to obtain the second probability map shown in Figure 4(a).

[0059] Based on the perspective of the two-dimensional image shown in Figure 3(a), the current pose of the three-dimensional oral cavity model Figure 2(c) relative to the two-dimensional image Figure 3(a) and the camera parameters, the rendering of the three-dimensional oral cavity model Figure 2(c) can be rendered to obtain the second probability map shown in Figure 4(b).

[0060] By superimposing the second probability map shown in Figure 4(a) and the first probability map shown in Figure 3(b), we can obtain the superimposed probability distribution map shown in Figure 5(a). By superimposing the second probability map shown in Figure 4(b) and the first probability map shown in Figure 3(c), we can obtain the superimposed probability distribution map shown in Figure 5(b).

[0061] Based on the superimposed probability distribution diagrams shown in Figures 5(a) and 5(b), it can be seen that there are some differences between the second probability diagram shown in Figure 4(a) and the first probability diagram shown in Figure 3(b). Furthermore, there are significant differences between the second probability diagram shown in Figure 4(b) and the first probability diagram shown in Figure 3(c).

[0062] This indicates that the current poses of the 3D oral cavity model in Figure 2(b) relative to the 2D image in Figure 3(a) and the 3D oral cavity model in Figure 2(c) relative to the 2D image in Figure 3(a) are both incorrect and need to be adjusted.

[0063] Step S104: For each set of two-dimensional images, determine the tooth registration pose during the shooting time period of the set of two-dimensional images based on the first probability map corresponding to each two-dimensional image in the set of two-dimensional images, the second probability map corresponding to the three-dimensional oral model under the viewpoint of each two-dimensional image in the set of two-dimensional images, and the current pose of the three-dimensional oral model relative to each two-dimensional image in the set of two-dimensional images.

[0064] Step S105: Output oral motion tracking results based on the tooth registration pose corresponding to each shooting time.

[0065] In one optional implementation of this application, the three-dimensional oral cavity model can be driven in real time based on the registration pose of each tooth according to the order of each shooting time period from front to back, so as to display the oral cavity motion tracking results on the interactive interface of the display device.

[0066] In another optional implementation of this application, each two-dimensional image can be displayed in the first preset area of ​​the interactive interface in chronological order of shooting time; and the three-dimensional oral model can be driven in real time based on the registration pose of each tooth in chronological order of each shooting time period to display the oral motion tracking results in the second preset area of ​​the interactive interface.

[0067] The oral motion tracking method provided in this application acquires a three-dimensional oral model of the user's mouth and multiple sets of two-dimensional images of the mouth. Then, based on each set of two-dimensional images, the position and pose of the three-dimensional oral model are adjusted. This allows the user's oral posture within the time period of each set of two-dimensional images to be reproduced using the three-dimensional oral model, i.e., the tooth registration pose, thereby achieving jaw motion tracking. Furthermore, this method does not require the installation of auxiliary devices in the user's mouth, allowing the user to perform jaw movements in a completely natural state, thus improving the reliability of the acquired data.

[0068] Combination Figure 6 The flowchart shown illustrates the method for determining the tooth registration pose. Step S104 may include the following steps: Step S601: For each two-dimensional image in the set of two-dimensional images, use the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model under the view of the two-dimensional image to calculate the image residual corresponding to the two-dimensional image.

[0069] In one optional implementation of this application, for each two-dimensional image in the set of two-dimensional images, the image residual corresponding to the two-dimensional image is calculated using a first probability map corresponding to the two-dimensional image and a second probability map corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image. This includes: for each two-dimensional image in the set of two-dimensional images, using the values ​​of each pixel in the first probability map to construct a first probability map matrix corresponding to the two-dimensional image; using the values ​​of each pixel in the second probability map corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image to construct a second probability map matrix corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image; and calculating the image residual corresponding to the two-dimensional image based on the first probability map matrix and the second probability map matrix corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image.

[0070] In another optional implementation of this application embodiment, after step S102, the following steps may also be performed: based on the region detection results of the two-dimensional image, identify the occluded region corresponding to the specified tooth region in the two-dimensional image, and mark the pixel points corresponding to the occluded region as unregistered pixel points.

[0071] Correspondingly, for each two-dimensional image in this set of two-dimensional images, using the first probability map corresponding to that two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model at the viewpoint of that two-dimensional image, the image residual corresponding to that two-dimensional image is calculated, including: In the first probability map corresponding to the two-dimensional image, the values ​​of pixels other than those that are not registered are used to construct the first probability map matrix corresponding to the two-dimensional image. For the second probability map corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image, the values ​​of pixels other than those that are not registered are used to construct the second probability map matrix corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image. Based on the first probability map matrix corresponding to the two-dimensional image and the second probability map matrix corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image, the image residual corresponding to the two-dimensional image is calculated.

[0072] Optionally, the image residual can characterize one of the following: the cross-entropy, joint probability, or squared error between the second probability map matrix corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image and the first probability map matrix corresponding to the two-dimensional image.

[0073] In one optional implementation of this application, the designated tooth region includes an orthognathic tooth region and / or a single-tooth tooth region. For each two-dimensional image in the set of two-dimensional images, the image residual corresponding to the two-dimensional image is calculated using a first probability map corresponding to the two-dimensional image and a second probability map corresponding to the three-dimensional oral model at the viewpoint of the two-dimensional image. This includes: for each designated tooth region in the two-dimensional image, calculating the region probability error based on the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral model at the viewpoint of the two-dimensional image; and calculating the image residual corresponding to the two-dimensional image based on the sum of all region probability errors.

[0074] In other words, for each specified tooth region in the two-dimensional image, a first region map matrix for that specified tooth region can be determined in the first probability map corresponding to the two-dimensional image. A second region map matrix for that specified tooth region is determined in the second probability map corresponding to the three-dimensional oral cavity model from the viewpoint of the two-dimensional image. Based on the first and second region map matrices corresponding to the specified tooth regions, the region probability error of that specified tooth region in the two-dimensional image is calculated. The region probability error can characterize one of the following: cross-entropy, joint probability, or squared error between the first and second region map matrices corresponding to the specified tooth region.

[0075] Step S602: Obtain the comprehensive residual based on the image residuals corresponding to each two-dimensional image.

[0076] In one optional implementation of this application, obtaining the comprehensive residual based on the image residuals corresponding to each two-dimensional image includes: summing the image residuals corresponding to each two-dimensional image to obtain the comprehensive residual.

[0077] For example, when the image residual represents the squared error between the second probability map matrix corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image and the first probability map matrix corresponding to the two-dimensional image, it can be expressed by the formula... The combined residual between any set of two-dimensional images and a three-dimensional oral cavity model is calculated.

[0078] Among them, P d The first probability map matrix representing the two-dimensional image. P represents the first probability map matrix corresponding to the i-th two-dimensional image. r The second probability map matrix representing the three-dimensional oral cavity model. The second probability map matrix represents the 3D oral cavity model from the viewpoint of the i-th 2D image. T represents the current pose of the 3D oral cavity model. T = {R, t}, where R represents the rotation matrix and t represents the translation vector.

[0079] In another optional implementation of this application, obtaining the comprehensive residual based on the image residuals corresponding to each two-dimensional image includes: for the set of two-dimensional images, if the set of two-dimensional images includes a depth map, generating a three-dimensional tooth point cloud based on the depth map; calculating the three-dimensional point alignment residual based on the three-dimensional tooth point cloud and the three-dimensional oral cavity model; and summing the three-dimensional point alignment residual and the image residuals corresponding to each two-dimensional image to obtain the comprehensive residual.

[0080] Optionally, generating a 3D tooth point cloud based on a depth map includes: transforming each pixel in the depth map into a camera coordinate system using camera intrinsics to obtain a 3D tooth point cloud.

[0081] Optionally, based on the 3D tooth point cloud and the 3D oral cavity model, the 3D point alignment residual is calculated, including: obtaining at least one reference 3D point in the 3D oral cavity model; for each reference 3D point, transforming the reference 3D point to the camera coordinate system based on the viewpoint of the depth map corresponding to the 3D tooth point cloud; determining the target 3D point corresponding to the reference 3D point in the 3D tooth point cloud; calculating the difference vector between the coordinate-transformed reference 3D point and the target 3D point corresponding to the reference 3D point; projecting the difference vector based on the normal direction of the target 3D point corresponding to the reference 3D point to obtain the alignment error of the reference 3D point in the 3D tooth point cloud; and squaring and summing the alignment errors of all reference 3D points in the 3D tooth point cloud to obtain the 3D point alignment residual.

[0082] For example, it can be calculated Obtain the 3D point alignment residual. Among them, Characterizing the k-th reference 3D point in the 3D oral cavity model, Characterizing reference three-dimensional points The target 3D point in the 3D tooth point cloud Characterizing reference three-dimensional points Normal vector in a 3D tooth point cloud. Representation vector sum vector The vector dot product between them. By calculating It can realize a perspective based on the depth map corresponding to the 3D tooth point cloud, and reference 3D points. Transform to the camera coordinate system. Characterizing the reference 3D point after coordinate transformation With the target three-dimensional point The difference vector between them.

[0083] In another optional implementation of this application embodiment, obtaining the comprehensive residual based on the image residuals corresponding to each two-dimensional image includes: for each two-dimensional image in the group of two-dimensional images, calculating the reprojection residual corresponding to the two-dimensional image based on the two-dimensional image and the three-dimensional oral cavity model; summing the reprojection residuals corresponding to each two-dimensional image to obtain the three-dimensional reprojection residual. The comprehensive residual is obtained by summing the image residuals corresponding to each two-dimensional image and the three-dimensional point reprojection residuals.

[0084] Optionally, based on the two-dimensional image and the three-dimensional oral cavity model, the reprojection residual corresponding to the two-dimensional image is calculated, including: Based on the perspective of the two-dimensional image, a 2D projection is performed on each reference three-dimensional point in the three-dimensional oral cavity model to obtain the reference probability point corresponding to each reference three-dimensional point.

[0085] In this two-dimensional image, obtain the 2D feature points corresponding to each reference three-dimensional point.

[0086] For each reference 3D point, obtain the distance error between the reference probability point corresponding to the reference 3D point and the 2D feature point corresponding to the reference 3D point. Summing all distance errors yields the reprojection residual corresponding to the two-dimensional image.

[0087] 2D feature point representation includes only pixel coordinates and does not include depth information for pixels.

[0088] For example, it can be calculated Obtain the three-dimensional point reprojection residual.

[0089] in, This refers to the pose after transforming the reference pose (i.e., variable T) to the viewpoint of the i-th 2D image. For multi-view cameras, a reference viewpoint is typically chosen to define the reference pose, while the poses between the various viewpoints are determined during device calibration; let's call this the calibration pose. Therefore, based on the reference pose and the calibration pose, the poses of the tooth model relative to each viewpoint can be obtained. Of course, it's not hard to understand that the image chosen as the reference viewpoint corresponds to... . This represents a 2D projection onto a 3D point x. In the embodiments of this application, The representation is based on the viewpoint of the i-th two-dimensional image, representing the reference three-dimensional point in the three-dimensional oral cavity model. Perform 2D projection. Characterizing reference three-dimensional points The coordinates of the corresponding two-dimensional feature point in the i-th two-dimensional image.

[0090] In another optional implementation of this application, obtaining the comprehensive residual based on the image residual corresponding to each two-dimensional image includes: for the set of two-dimensional images, if the set of two-dimensional images includes a depth map, generating a corresponding three-dimensional tooth point cloud based on the depth map; and calculating the three-dimensional point alignment residual based on the three-dimensional tooth point cloud and the three-dimensional oral cavity model.

[0091] For each two-dimensional image in the set of two-dimensional images, the reprojection residual corresponding to the two-dimensional image is calculated based on the two-dimensional image and the three-dimensional oral cavity model. The reprojection residuals corresponding to each two-dimensional image are summed to obtain the three-dimensional reprojection residual.

[0092] Obtain the error weighting factors corresponding to the 3D point alignment residual and the 3D point ghosting residual. Both the error weighting factors for the 3D point alignment residual and the 3D point ghosting residual are constants, and their sum equals 1.

[0093] Using the error weighting factors corresponding to the 3D point alignment residual and the 3D point ghosting residual, the 3D point alignment residual and the 3D point ghosting residual are weighted and summed. The weighted sum is then summed with the image residuals corresponding to each 2D image to obtain the comprehensive residual.

[0094] For example, it can be calculated Obtain the comprehensive residual. Among them, The error weighting factor characterizing the 3D point alignment residual. Error weighting factors characterize the residuals of 3D point reprojection. and It can also be set to 0, which means that there is no or no error term is used.

[0095] Step S603: Optimize the current pose of the 3D oral cavity model based on the comprehensive residual, and output the tooth registration pose corresponding to the shooting time period.

[0096] Optionally, the current pose of the 3D oral cavity model is optimized based on the comprehensive residual, and the tooth registration pose corresponding to the shooting time period is output, including: Determine whether the overall residual exceeds the preset residual target; if the overall residual exceeds the preset residual target, adjust the current pose of the 3D oral model; based on the adjusted current pose, generate the second probability map again and calculate the overall residual until the overall residual does not exceed the preset residual target, or until the number of times the second probability map generation step is executed reaches the preset number of executions, and output the current pose of the 3D oral model as the tooth registration pose.

[0097] Optionally, adjusting the current pose of the 3D oral cavity model includes: performing gradient descent optimization on the second probability map matrix set corresponding to the 3D oral cavity model in the set of 2D images to obtain the adjusted current pose. The second probability map matrix set corresponding to the 3D oral cavity model in the set of 2D images includes: the second probability map matrix corresponding to the 3D oral cavity model at the viewpoint of each 2D image in the set of 2D images.

[0098] In some embodiments, given a set of two-dimensional images, the current pose of the three-dimensional oral cavity model relative to the set of two-dimensional images can be obtained by manually annotating or automatically recognizing feature points of the three-dimensional oral cavity model and the corresponding feature points of the two-dimensional images, and then by using the commonly used PnP (Persistent-n-Points) algorithm.

[0099] In a sequence of multiple time-ordered two-dimensional images, for the first set of two-dimensional images, the current pose of the three-dimensional oral cavity model relative to the set of two-dimensional images can be obtained by manually annotating or automatically recognizing the feature points of the three-dimensional oral cavity model and the feature points of the corresponding two-dimensional images, and then by using the commonly used PnP (Persipective-n-Points) algorithm.

[0100] For each set of two-dimensional images other than the first set, the tooth registration pose of the previous set of two-dimensional images can be used as the current pose of that set of two-dimensional images.

[0101] Example 2: Based on the same inventive concept, this application also provides an oral motion tracking device 700 in its embodiments. Please refer to... Figure 7 As shown, Figure 7 It shows the use of Figure 1The oral motion tracking device of the method shown should be understood to have specific functions described above; to avoid repetition, detailed descriptions are omitted here. Device 700 includes at least one software function module that can be stored in memory or embedded in the operating system of device 700 in the form of software or firmware. Specifically: See Figure 7 As shown, the device 700 includes: an acquisition module 701, a first generation module 702, a second generation module 703, a determination module 704, and an output module 705. Wherein: The acquisition module 701 acquires a 3D oral cavity model and at least one set of 2D images, wherein the difference in shooting time corresponding to the same set of 2D images is within a preset range. The first generation module 702 generates a first probability map corresponding to the 2D images; the value of each pixel in the first probability map represents the probability that the pixel belongs to a specified tooth region. The second generation module 703 generates a second probability map corresponding to the 3D oral cavity model at the viewpoint of the 2D images; the value of each pixel in the second probability map represents the probability that the pixel belongs to a specified tooth region. The determination module 704, for each set of 2D images, determines the tooth registration pose for the shooting time period of the set of 2D images based on the first probability map corresponding to each 2D image in the set, the second probability map corresponding to the 3D oral cavity model at the viewpoint of each 2D image in the set, and the current pose of the 3D oral cavity model relative to each 2D image in the set. The output module 705 outputs the oral cavity motion tracking results based on the tooth registration pose corresponding to each shooting time period.

[0102] In one feasible embodiment of this application, the output module 705 is specifically used to drive the three-dimensional oral cavity model in real time based on the registration pose of each tooth, according to the sequential order of each shooting time period from front to back, so as to display the oral cavity motion tracking results on the interactive interface of the display device; or, The two-dimensional images are displayed in the first preset area of ​​the interactive interface in chronological order of shooting time. The three-dimensional oral cavity model is driven in real time based on the registration pose of each tooth in chronological order of shooting time period, so as to display the oral cavity motion tracking results in the second preset area of ​​the interactive interface.

[0103] In one feasible embodiment of this application, the second generation module 703 is specifically used to render the three-dimensional oral cavity model according to the current pose of the three-dimensional oral cavity model relative to the two-dimensional image and the camera parameters under the viewpoint, so as to obtain the second probability map corresponding to the three-dimensional oral cavity model under the viewpoint.

[0104] In one feasible embodiment of this application, the second generation module 703 can also be used to identify the crown region and non-crown region on the three-dimensional oral cavity model, mark the crown region as the rendering region, and / or mark the non-crown region as the non-rendering region; and render the rendering region on the three-dimensional oral cavity model according to the current pose of the three-dimensional oral cavity model relative to the two-dimensional image and the camera parameters.

[0105] In one feasible embodiment of this application, the first generation module 703 is specifically used to generate a first probability map corresponding to a two-dimensional image based on a region detection algorithm.

[0106] In one feasible embodiment of this application, the first generation module 703 can also be used to identify the occluded region corresponding to the specified tooth region in the two-dimensional image based on the region detection result of the two-dimensional image, and mark the pixel point corresponding to the occluded region as a non-registered pixel point.

[0107] In one feasible embodiment of this application, the determining module 704 is specifically used to calculate the image residual corresponding to each two-dimensional image in the set of two-dimensional images using the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral model under the view of the two-dimensional image; obtain the comprehensive residual based on the image residual corresponding to each two-dimensional image; optimize the current pose of the three-dimensional oral model based on the comprehensive residual, and output the tooth registration pose of the shooting time period of the set of two-dimensional images.

[0108] In one feasible embodiment of this application, the determining module 704 is specifically used to calculate the regional probability error for each specified tooth region in the two-dimensional image, based on the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model at the viewpoint of the two-dimensional image; and to calculate the image residual corresponding to the two-dimensional image based on the sum of all regional probability errors.

[0109] In one feasible embodiment of this application, the determining module 704 is specifically used to output the current pose as the tooth registration pose if the comprehensive residual does not exceed the preset residual target; if the comprehensive residual exceeds the preset residual target, adjust the current pose, and based on the adjusted current pose, generate the second probability map again and calculate the comprehensive residual; if the number of times the second probability map generation step is executed reaches the preset number of executions, output the current pose as the tooth registration pose.

[0110] In one feasible embodiment of this application, the determining module 704 can also be used to generate a corresponding three-dimensional tooth point cloud based on the depth map when the set of two-dimensional images includes a depth map; calculate the three-dimensional point alignment residual based on the three-dimensional tooth point cloud and the three-dimensional oral cavity model; and use the three-dimensional point alignment residual to calculate the comprehensive residual.

[0111] In one feasible embodiment of this application, the determining module 704 can also be used to calculate the reprojection residual corresponding to each two-dimensional image in the group of two-dimensional images based on the two-dimensional image and the three-dimensional oral cavity model, and to sum up the reprojection residuals corresponding to each two-dimensional image to obtain the three-dimensional reprojection residual; the three-dimensional point reprojection residual is used to calculate the comprehensive residual.

[0112] It should be understood that, for the sake of brevity, some of the content described in Embodiment 1 will not be repeated in this embodiment.

[0113] Example 3: Based on the same inventive concept, this embodiment provides an electronic device, see [link to relevant documentation]. Figure 8 As shown, it includes a processor 801 and a memory 802. Wherein: The processor 801 is used to execute one or more programs stored in the memory 802 to implement the oral motion tracking method described above.

[0114] It is understandable that the processor 801 can be a processor core or processor chip, or other circuitry capable of program configuration and execution. The memory 802 can be RAM (Random Access Memory), ROM (Read-Only Memory), flash memory, etc., but this is not a limitation.

[0115] It's understandable. Figure 8 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 Different configurations are shown. For example, it may also have an internal communication bus for communication between the processor 801 and the memory 802; or it may have an external communication interface, such as a USB (Universal Serial Bus) interface, a CAN (Controller Area Network) bus interface, etc.; or it may have an information display component such as a display screen, but this is not a limitation.

[0116] Based on the same inventive concept, this embodiment also provides a computer-readable storage medium, such as a floppy disk, optical disk, hard disk, flash memory, USB flash drive, SD (Secure Digital Memory Card), MMC (Multimedia Card), etc., in which one or more programs implementing the above steps are stored. These one or more programs can be executed by one or more processors to implement the above-described oral motion tracking method. Further details will not be elaborated here.

[0117] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0118] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0120] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0121] In this article, "multiple" refers to two or more.

[0122] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for tracking oral motor functions, characterized in that, include: Acquire a three-dimensional oral cavity model and at least one set of two-dimensional images, with the difference in shooting time corresponding to the same set of two-dimensional images within a preset range; Generate the first probability map corresponding to the two-dimensional image; The value of each pixel in the first probability map is used to represent the probability that the pixel belongs to a specified tooth region; Based on the perspective of the two-dimensional image, a second probability map is generated corresponding to the three-dimensional oral cavity model under the perspective; the value of each pixel in the second probability map is used to characterize the probability that the pixel belongs to a specified tooth region; For each set of two-dimensional images, the tooth registration pose of the shooting time period of the set of two-dimensional images is determined based on the first probability map corresponding to each two-dimensional image in the set of two-dimensional images, the second probability map corresponding to the three-dimensional oral model under the view of each two-dimensional image in the set of two-dimensional images, and the current pose of the three-dimensional oral model relative to each two-dimensional image in the set of two-dimensional images. Based on the tooth registration pose corresponding to each of the aforementioned shooting time periods, the oral cavity motion tracking results are output.

2. The oral motor tracking method according to claim 1, characterized in that, The step of outputting oral motion tracking results based on the tooth registration pose corresponding to each of the aforementioned shooting time periods includes: According to the sequence of the shooting time periods from front to back, the three-dimensional oral cavity model is driven in real time based on the registration pose of each tooth, so as to display the oral cavity motion tracking results on the interactive interface of the display device; or... The two-dimensional images are displayed in the first preset area of ​​the interactive interface in chronological order of shooting time; the three-dimensional oral cavity model is driven in real time based on the registration pose of each tooth in chronological order of shooting time period, so as to display the oral cavity motion tracking results in the second preset area of ​​the interactive interface.

3. The oral motor tracking method according to claim 1, characterized in that, The step of generating a second probability map of the three-dimensional oral cavity model corresponding to the viewpoint of the two-dimensional image includes: Based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the stated viewpoint, the 3D oral cavity model is rendered to obtain a second probability map corresponding to the 3D oral cavity model from the stated viewpoint.

4. The oral motor tracking method according to claim 3, characterized in that, Before rendering the three-dimensional oral cavity model based on the current pose of the three-dimensional oral cavity model relative to the two-dimensional image and camera parameters from the stated viewpoint to obtain the second probability map corresponding to the three-dimensional oral cavity model from the stated viewpoint, the oral cavity motion tracking method further includes: Identify the crown region and non-crown region on the three-dimensional oral cavity model, mark the crown region as the rendering region, and / or mark the non-crown region as the non-rendering region; Correspondingly, based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the stated viewpoint, the 3D oral cavity model is rendered, including: Based on the current pose of the 3D oral cavity model relative to the 2D image and the camera parameters from the stated perspective, the rendering area on the 3D oral cavity model is rendered.

5. The oral motor tracking method according to claim 1, characterized in that, The generation of the first probability map corresponding to the two-dimensional image includes: Based on the region detection algorithm, the first probability map corresponding to the two-dimensional image is generated.

6. The oral motor tracking method according to claim 5, characterized in that, After generating the first probability map corresponding to the two-dimensional image based on the region detection algorithm, the oral cavity motion tracking method further includes: Based on the region detection results of the two-dimensional image, the occluded region corresponding to the specified tooth region in the two-dimensional image is identified, and the pixels corresponding to the occluded region are marked as unregistered pixels.

7. The oral motor tracking method according to any one of claims 1-6, characterized in that, For each set of two-dimensional images, based on the first probability map corresponding to each two-dimensional image in the set, the second probability map corresponding to the three-dimensional oral model at the viewpoint of each two-dimensional image in the set, and the current pose of the three-dimensional oral model relative to each two-dimensional image in the set, the tooth registration pose for the time period in which the set of two-dimensional images is captured is determined, including: For each two-dimensional image in the set of two-dimensional images, the image residual corresponding to the two-dimensional image is calculated using the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model under the view of the two-dimensional image. The comprehensive residual is obtained based on the image residuals corresponding to each of the two-dimensional images; Based on the comprehensive residual, the current pose of the three-dimensional oral cavity model is optimized, and the tooth registration pose of the two-dimensional images during the shooting time period is output.

8. The oral motor tracking method according to claim 7, characterized in that, The designated tooth region includes whole-jaw tooth regions and / or single-tooth tooth regions. For each two-dimensional image in the set of two-dimensional images, using the first probability map corresponding to that two-dimensional image and the second probability map corresponding to the three-dimensional oral model at the viewpoint of that two-dimensional image, the image residual corresponding to that two-dimensional image is calculated, including: For each specified tooth region in the two-dimensional image, the region probability error is calculated based on the first probability map corresponding to the two-dimensional image and the second probability map corresponding to the three-dimensional oral cavity model under the view of the two-dimensional image. The image residual corresponding to the two-dimensional image is calculated based on the sum of the probability errors of all regions.

9. The oral motor tracking method according to claim 7, characterized in that, The optimization of the current pose of the 3D oral cavity model based on the comprehensive residual, and the output of the tooth registration pose corresponding to the shooting time period, includes: If the overall residual does not exceed the preset residual target, the current pose will be output as the tooth registration pose; If the overall residual exceeds the preset residual target, the current pose is adjusted, and based on the adjusted current pose, the second probability map is generated again and the overall residual is calculated. If the second probability map generation step is executed a preset number of times, the current pose is output as the tooth registration pose.

10. The oral motor tracking method according to claim 7, characterized in that, After acquiring the three-dimensional oral cavity model and at least one set of two-dimensional images, the oral cavity motion tracking method further includes: For this set of two-dimensional images, if the set of two-dimensional images includes a depth map, generate the corresponding three-dimensional tooth point cloud based on the depth map; Based on the three-dimensional tooth point cloud and the three-dimensional oral cavity model, the three-dimensional point alignment residual is calculated; the three-dimensional point alignment residual is used to calculate the comprehensive residual.

11. The oral motor tracking method according to claim 7, characterized in that, After acquiring the three-dimensional oral cavity model and at least one set of two-dimensional images, the oral cavity motion tracking method further includes: For each two-dimensional image in the set of two-dimensional images, based on the two-dimensional image and the three-dimensional oral cavity model, the reprojection residual corresponding to the two-dimensional image is calculated. The reprojection residuals corresponding to each two-dimensional image are accumulated and summed to obtain the three-dimensional reprojection residual. The three-dimensional point reprojection residual is used to calculate the comprehensive residual.

12. An oral motor tracking device, characterized in that, The oral cavity motion tracking device includes: The acquisition module is used to acquire a three-dimensional oral cavity model and at least one set of two-dimensional images, wherein the difference in the shooting time corresponding to the two-dimensional images in the same set is within a preset range. The first generation module is used to generate a first probability map corresponding to the two-dimensional image; the value of each pixel in the first probability map is used to characterize the probability that the pixel belongs to a specified tooth region. The second generation module is used to generate a second probability map of the three-dimensional oral cavity model corresponding to the viewpoint of the two-dimensional image; the value of each pixel in the second probability map is used to characterize the probability that the pixel belongs to a specified tooth region. The determination module is used to determine the tooth registration pose of each set of two-dimensional images during the shooting time period based on the first probability map corresponding to each two-dimensional image in the set of two-dimensional images, the second probability map corresponding to the three-dimensional oral model under the viewpoint of each two-dimensional image in the set of two-dimensional images, and the current pose of the three-dimensional oral model relative to each two-dimensional image in the set of two-dimensional images. The output module is used to output oral motion tracking results based on the tooth registration pose corresponding to each of the shooting time periods.

13. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to execute the oral motion tracking method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that can be executed by a processor to perform the oral motion tracking method according to any one of claims 1-11.