Orthodontic treatment monitoring method, apparatus, device, and storage medium

Abstract

Embodiments of the present disclosure relate to an orthodontic treatment monitoring method, device, equipment and storage medium, wherein the orthodontic treatment monitoring method comprises: acquiring a plurality of current intraoral images of a user at a current time; determining a first tooth point cloud of each target tooth in the user's oral cavity based on the plurality of current intraoral images; and comparing the first tooth point cloud of each target tooth with a second tooth point cloud of the target tooth to obtain a deviation value corresponding to the target tooth, wherein the second tooth point cloud is a tooth point cloud of the tooth before the current time. According to the embodiments of the present disclosure, the automatic comparison of the first tooth point cloud and the second tooth point cloud can be realized, thereby improving the orthodontic treatment monitoring efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for monitoring orthodontic treatment. Background Technology

[0002] Orthodontic treatment involves periodically wearing braces to guide the teeth to grow in the direction designed by the braces, thereby gradually changing the position and posture of the teeth.

[0003] Currently, orthodontic treatment monitoring during the treatment process mainly relies on patients visiting the clinic regularly for checkups. Doctors visually assess the orthodontic progress at each stage of treatment. If the doctor finds that the orthodontic results are not developing in the direction planned, adjustments to the braces are necessary to ensure the smooth progress of treatment. However, this method of orthodontic treatment monitoring is inefficient and cannot meet the needs of users. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, the present disclosure provides an orthodontic treatment monitoring method, device, equipment and storage medium.

[0005] A first aspect of this disclosure provides a method for monitoring orthodontic treatment, the method comprising:

[0006] Acquire multiple frames of the user's intraoral image at the current moment;

[0007] Based on multiple frames of current intraoral images, determine the first tooth point cloud of each target tooth in the user's oral cavity;

[0008] For each target tooth, the first tooth point cloud and the second tooth point cloud of the target tooth are compared to obtain the deviation value corresponding to the target tooth. The second tooth point cloud is the tooth point cloud of the tooth before the current time.

[0009] A second aspect of this disclosure provides an orthodontic treatment monitoring device, the device comprising:

[0010] The first acquisition module is used to acquire multiple frames of current intraoral images of the user at the current moment;

[0011] The first determining module is used to determine the first tooth point cloud of each target tooth in the user's oral cavity based on multiple frames of current intraoral images;

[0012] The comparison module is used to compare the first tooth point cloud and the second tooth point cloud of each target tooth to obtain the deviation value corresponding to the target tooth. The second tooth point cloud is the tooth point cloud of the tooth before the current time.

[0013] A third aspect of this disclosure provides an electronic device comprising: a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method described in the first aspect.

[0014] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the method of the first aspect described above.

[0015] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0016] This embodiment of the disclosure can acquire multiple frames of current intraoral images of a user at the current moment, and based on these multiple frames, determine the first tooth point cloud for each target tooth in the user's oral cavity. Then, for each target tooth, the first tooth point cloud and the second tooth point cloud are compared to obtain the corresponding deviation value. The second tooth point cloud represents the tooth's point cloud at a time prior to the current moment. Using this technical solution, automatic comparison of the first and second tooth point clouds can be achieved. Compared to traditional visual comparison by a doctor, this method allows for convenient, fast, and accurate acquisition of the deviation value between the first and second tooth point clouds, thereby improving the efficiency of orthodontic treatment monitoring. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an orthodontic treatment monitoring method provided in an embodiment of this disclosure;

[0020] Figure 2 This is a flowchart of another orthodontic treatment monitoring method provided in this embodiment;

[0021] Figure 3 This is a logical diagram illustrating instance segmentation provided in an embodiment of this disclosure;

[0022] Figure 4This is a schematic diagram of the structure of an orthodontic treatment monitoring device provided in an embodiment of this disclosure;

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0024] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0025] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0026] Figure 1 This is a flowchart illustrating an orthodontic treatment monitoring method provided in this disclosure, which can be executed by an electronic device. This electronic device can be exemplarily understood as a device with page display capabilities, such as a mobile phone, tablet computer, laptop computer, desktop computer, or smart TV. Figure 1 As shown, the method provided in this embodiment includes the following steps:

[0027] S110. Obtain multiple frames of the current intraoral image of the user at the current moment.

[0028] In this embodiment of the disclosure, in order to monitor the orthodontic treatment process of the user, the electronic device needs to acquire multiple frames of current intraoral images of the user at the current moment.

[0029] Specifically, the current intraoral image is the image obtained by using an image acquisition device to capture images of the user's intraoral cavity at the current moment.

[0030] The specific type of image acquisition device can be set by those skilled in the art according to the actual situation, and is not limited here. For example, the image acquisition device may include an RGBD camera, in which case the current intraoral image includes an RGB image and a depth image, but it is not limited to this.

[0031] During image acquisition, images of the user's oral cavity can be acquired from different angles and / or different distances to ensure that at least each target tooth in the oral cavity is imaged, such as all teeth in the oral cavity, where the target teeth are those that require orthodontic treatment monitoring.

[0032] In some embodiments, the electronic device may directly receive multiple frames of current intraoral images sent by the image acquisition device, or it may read multiple frames of current intraoral images pre-stored from a local or storage device. However, it is not limited to these embodiments.

[0033] S120. Based on multiple frames of current intraoral images, determine the first tooth point cloud for each target tooth in the user's oral cavity.

[0034] In this embodiment of the disclosure, the electronic device can determine at least the first tooth point cloud for each target tooth based on multiple frames of current intraoral images, so as to subsequently compare the first tooth point cloud and the second tooth point cloud of the target tooth.

[0035] Specifically, the first tooth point cloud is a point cloud of teeth obtained based on the current intraoral image.

[0036] In some embodiments, S120 may include: for each target tooth, segmenting a tooth image (or tooth region) of the target tooth from multiple frames of current intraoral images to obtain at least one tooth image, and determining a first tooth point cloud of the target tooth based on the at least one tooth image.

[0037] Specifically, a tooth image is the image region containing the teeth.

[0038] Specifically, there are various ways to segment a tooth image, and no particular method is specified here. For example, the segmentation process can be performed manually; it can also be performed using a two-dimensional object segmentation algorithm, etc., but it is not limited to these methods.

[0039] Specifically, there are various implementation methods for determining the first tooth point cloud, and no limitation is made here. For example, three-dimensional reconstruction can be performed based on the at least one tooth image to obtain the first tooth point cloud of the target tooth; alternatively, for each tooth image, three-dimensional reconstruction can be performed based on the tooth image to obtain the first tooth sub-point cloud of the target tooth, thereby obtaining at least one first tooth sub-point cloud corresponding to the target tooth, and the at least one first tooth sub-point cloud corresponding to the at least one tooth image can be fused and then sampled to obtain the first tooth point cloud of the target tooth, etc., but not limited to these methods.

[0040] In other embodiments, S120 may include: performing three-dimensional reconstruction based on multiple frames of current intraoral images to obtain the current dental point cloud of the user's intraoral jaw; and segmenting the first tooth point cloud of the target tooth from the current dental point cloud.

[0041] Specifically, the current dental point cloud is the overall point cloud of all upper and / or lower teeth in the user's oral cavity.

[0042] Specifically, there are various ways to segment the point cloud of the first tooth, which are not limited here. For example, the segmentation process can be performed manually or using a 3D point cloud segmentation algorithm, but it is not limited to these methods.

[0043] S130. For each target tooth, compare the first tooth point cloud and the second tooth point cloud of the target tooth to obtain the deviation value corresponding to the target tooth, wherein the second tooth point cloud is the tooth point cloud of the tooth before the current time.

[0044] In this embodiment of the disclosure, the electronic device can pre-store the second tooth point cloud of the target tooth. Thus, for each target tooth that needs orthodontic treatment monitoring, the first tooth point cloud and the second tooth point cloud of the target tooth can be compared to obtain the deviation value between the two.

[0045] Specifically, "before the current moment" can refer to any moment before the current moment, such as before treatment or after the last treatment, but is not limited to this.

[0046] Specifically, the deviation value is the difference between the point cloud of the first tooth and the point cloud of the second tooth of the same tooth. This difference can include differences in position, orientation, etc. Accordingly, the deviation value can include parameters such as position deviation and orientation deviation.

[0047] Specifically, there are several ways to determine the deviation value. The following are typical examples, but are not limited to these.

[0048] In some embodiments, S130 may include: for each target tooth, registering the first tooth point cloud and the second tooth point cloud of the target tooth to obtain the transformation matrix corresponding to the target tooth, and determining the deviation value based on the transformation matrix corresponding to the target tooth, thereby obtaining the deviation value corresponding to each target tooth.

[0049] Those skilled in the art can use any possible registration algorithm to register the point cloud of the first tooth and the point cloud of the second tooth; no limitation is made here.

[0050] Specifically, the transformation matrix includes a translation matrix and a rotation matrix. The attitude deviation can be determined from the rotation matrix, and the position parameters can be determined from the translation matrix.

[0051] It is understandable that, since the embodiments of this disclosure can automatically compare the first tooth point cloud and the second tooth point cloud and obtain quantitative comparison results (i.e., deviation values), compared with the traditional comparison by the doctor's naked eye, it can reduce the doctor's burden, improve the comparison speed, reduce confusion and misjudgment, and detect some minor deviations that are not easily detected by the naked eye, thereby improving the accuracy of the comparison results. Therefore, the embodiments of this disclosure can conveniently, quickly, and accurately obtain the deviation value between the first tooth point cloud and the second tooth point cloud, thereby improving the efficiency of orthodontic treatment monitoring. Furthermore, the embodiments of this disclosure achieve full digitalization. If the digital data such as the first tooth point cloud, the second tooth point cloud, and the deviation value of the user's target tooth are subsequently transmitted to doctors located in different geographical locations via the Internet or other means for review, the user can receive advice and guidance from expert doctors in different geographical locations during the process of dental diagnosis or follow-up visits. This allows expert doctors to provide professional services to a wider range of patients without being constrained by geographical space. It also greatly facilitates the doctor's tracking and monitoring of the patient's treatment process and is more conducive to the timely advancement, adjustment, and improvement of the treatment plan.

[0052] This embodiment of the disclosure can acquire multiple frames of current intraoral images of a user at the current moment, and based on these multiple frames, determine the first tooth point cloud for each target tooth in the user's oral cavity. Then, for each target tooth, the first tooth point cloud and the second tooth point cloud are compared to obtain the corresponding deviation value. The second tooth point cloud represents the tooth's point cloud at a time prior to the current moment. Using this technical solution, automatic comparison of the first and second tooth point clouds can be achieved. Compared to traditional visual comparison by a doctor, this method allows for convenient, fast, and accurate acquisition of the deviation value between the first and second tooth point clouds, thereby improving the efficiency of orthodontic treatment monitoring.

[0053] Figure 2 This is a flowchart of another orthodontic treatment monitoring method provided in this disclosure. This disclosure optimizes the above embodiments and can be combined with various optional solutions from one or more of the above embodiments.

[0054] like Figure 2 As shown, this orthodontic treatment monitoring method may include the following steps.

[0055] S210. Obtain multiple frames of the current intraoral image of the user at the current moment.

[0056] Specifically, S210 is similar to S110, and will not be described in detail here.

[0057] S220. For each frame of the current intraoral image, determine the first tooth sub-point cloud and tooth position of each tooth in the current intraoral image, wherein the tooth position is used to indicate the arrangement position of the teeth on the jaw.

[0058] In this embodiment of the disclosure, each frame of the current intraoral image includes at least one tooth. Therefore, for each frame of the current intraoral image, the first tooth sub-point cloud and tooth position of each tooth in the current intraoral image can be determined based on the current intraoral image. In this way, the first tooth sub-point cloud and tooth position of each tooth in any frame of the current intraoral image can be obtained.

[0059] Specifically, the first tooth sub-point cloud is a point cloud obtained by three-dimensional reconstruction based on the tooth image (or tooth region) in the current intraoral image.

[0060] Specifically, all teeth on the jaw (upper jaw and lower jaw) are usually arranged in sequence. The tooth position is used to indicate the position of the tooth on its respective jaw, such as the nth tooth from left to right, or the mth tooth from right to left, etc.

[0061] In some embodiments, the current intraoral image includes an RGB image and a depth image; wherein, S220 may include:

[0062] S221. Perform instance segmentation on the RGB image to obtain the instance segmentation result;

[0063] Specifically, any possible instance segmentation method can be used to segment RGB images; no limitation is made here.

[0064] S222. Based on the instance segmentation results, perform region segmentation on the RGB image and depth image to obtain the RGB image and depth image of each tooth in the current oral cavity image.

[0065] Specifically, the RGB image of a tooth is the image region containing the tooth in the RGB image, and the depth image of a tooth is the image region containing the tooth in the depth image.

[0066] Specifically, since there is a one-to-one correspondence between the pixels of the RGB image and the depth image (referred to as the first association relationship), the RGB image of each tooth can be segmented from the RGB image based on the instance segmentation result. Furthermore, based on the RGB image of each tooth and the first association relationship, the RGB image of the tooth corresponding to the RGB image of the tooth can be segmented from the depth image.

[0067] S223. For each tooth, perform three-dimensional reconstruction based on the tooth's RGB image and tooth depth image to obtain the first tooth sub-point cloud.

[0068] Specifically, the RGB image provides the color information of the point, and it also provides the x and y coordinates of the point in the pixel coordinate system. The depth image directly provides the Z coordinate in the camera coordinate system, which is the distance between the camera and the point. Therefore, based on the camera's intrinsic parameters, the coordinates of any point in the camera coordinate system can be calculated, thereby achieving three-dimensional reconstruction and obtaining the first tooth sub-point cloud of the tooth.

[0069] Of course, those skilled in the art can also input the current intraoral image into a trained first neural network model to obtain the first tooth sub-point cloud of each tooth in the current intraoral image output by the first neural network model.

[0070] S224. For each tooth, determine the tooth position based on the first tooth sub-point cloud of the tooth, the tooth RGB image, and the second tooth point cloud of multiple teeth in the user's oral cavity.

[0071] Specifically, the first tooth sub-point cloud, the tooth RGB image, and the second tooth point clouds of multiple teeth in the user's oral cavity can be input into a trained second neural network model to obtain the tooth position corresponding to the first tooth sub-point cloud output by the second neural network model. However, it is not limited to this.

[0072] It is understandable that obtaining the first tooth sub-point cloud of a tooth through instance segmentation and 3D reconstruction makes the acquisition method of the first tooth sub-point cloud logically simple and easy to implement, which helps to reduce the difficulty of obtaining the first tooth sub-point cloud.

[0073] S230. For each target tooth, acquire the first tooth sub-point cloud of the tooth with the same tooth position as the target tooth in multiple frames of the current intraoral image, and obtain at least one first tooth sub-point cloud corresponding to the target tooth.

[0074] In this embodiment of the disclosure, each frame of the current intraoral image includes at least one tooth, and different current intraoral images may include overlapping teeth. Therefore, the same tooth may appear in different current intraoral images. Thus, after S220, one or more first tooth sub-point clouds of the same tooth can be obtained. Since the tooth position can determine which tooth on the jaw is the target tooth, for each target tooth, the tooth in the current intraoral image that is at the same position as the target tooth is the target tooth. By selecting the first tooth sub-point cloud of the target tooth from the multiple first tooth sub-point clouds corresponding to the current intraoral image containing the target tooth, at least one first tooth sub-point cloud corresponding to the target tooth can be obtained.

[0075] S240. For each target tooth, determine the first tooth point cloud of the target tooth based on at least one first tooth sub-point cloud corresponding to the target tooth.

[0076] In this embodiment of the disclosure, the electronic device can determine a more complete first tooth point cloud of the target tooth based on at least one first tooth sub-point cloud corresponding to the target tooth.

[0077] Specifically, if the current intraoral image contains a complete target tooth, then the complete point cloud of the target tooth can be obtained by performing three-dimensional reconstruction based on the RGB image and RGB depth image of the target tooth segmented from the current intraoral image. That is, the first tooth sub-point cloud obtained by three-dimensional reconstruction is the first tooth point cloud.

[0078] Typically, for a given tooth, due to factors such as the image acquisition angle, the tooth in the current intraoral image is usually incomplete, meaning it only includes a portion of the tooth. In this case, the first tooth sub-point cloud obtained through 3D reconstruction only corresponds to a part of the tooth. Since the image acquisition angles of different current intraoral images are usually different, multiple first tooth sub-point clouds of the tooth obtained based on multiple previous intraoral images usually include different parts of the tooth. Thus, a more complete first tooth point cloud of the tooth can be obtained based on multiple first tooth sub-point clouds of the tooth.

[0079] In some embodiments, determining the first tooth point cloud of the target tooth based on at least one first tooth sub-point cloud corresponding to the target tooth includes: performing a fusion process on at least one first tooth sub-point cloud corresponding to the target tooth and then performing a sampling process to obtain the first tooth point cloud of the target tooth.

[0080] Specifically, any possible fusion processing method can be used to fuse at least one first tooth sub-point cloud corresponding to the target tooth, without any limitation here.

[0081] Understandably, through fusion and sampling processing, a more complete and concise first tooth point cloud of the target tooth can be obtained. On the one hand, because the first tooth point cloud is more complete, subsequent comparisons between the first and second tooth point clouds allow for a more detailed comparison of the target tooth's condition at different stages, making the obtained deviation values ​​more consistent with the actual growth changes of the target tooth, thus improving the accuracy of orthodontic monitoring. On the other hand, because the first tooth point cloud is more concise, i.e., has a smaller data volume, the computational load when comparing the first and second tooth point clouds can be reduced, improving the comparison effect and thus enhancing the overall efficiency of orthodontic monitoring.

[0082] S250. For each target tooth, compare the first tooth point cloud and the second tooth point cloud of the target tooth to obtain the deviation value corresponding to the target tooth, wherein the second tooth point cloud is the tooth point cloud of the tooth before the current time.

[0083] Specifically, S250 is similar to S130, and will not be described in detail here.

[0084] This disclosure enables the determination of the first tooth sub-point cloud and tooth position of each tooth in the current intraoral image, and the acquisition of at least one first tooth sub-point cloud corresponding to the target tooth based on the tooth position. Thus, the first tooth point cloud of the target tooth is determined based on the at least one first tooth sub-point cloud corresponding to the target tooth. This makes the acquisition of the first tooth point cloud simple and easy to implement. Furthermore, since the first tooth point cloud is more complete, it is possible to make a more detailed comparison of the tooth condition of the target tooth at different stages, thereby making the deviation value obtained from the comparison more consistent with the actual growth changes of the target tooth, that is, improving the accuracy of orthodontic monitoring.

[0085] In another embodiment of this disclosure, the RGB image is segmented to obtain the instance segmentation result, including:

[0086] S2211. Input the RGB image into a feature extraction network based on Win Transformer to extract features and obtain a feature map;

[0087] Specifically, the Swin transformer is a deep learning model that can be used as a general backbone network for computer vision tasks, and can be used for a series of downstream vision tasks such as image classification, image segmentation, and object detection.

[0088] S2212. Determine the instance segmentation result based on the feature map.

[0089] For example, Figure 3 This is a logical diagram illustrating instance segmentation according to an embodiment of this disclosure. See also... Figure 3First, a feature extraction network 311 based on the Swin Transformer is used to extract features from the input RGB image 321 to obtain a feature map 322. Then, a Region Proposal Network (RPN) 312 is used to extract the region of interest from the obtained feature map 322 and perform a localization operation to obtain preliminary positive samples and their localization information, which are denoted as candidate boxes 323. Next, the obtained feature map 322 and candidate boxes 323 are input into the Region of Interest Align (Rol Align) layer 313. Localization information, category information, and a fixed-size feature map are obtained through regression. The detection box generation network 315 generates detection boxes 324 based on the localization information, and the classification network 316 generates classification results 325 based on the category information. The fixed-size feature map is then used to perform semantic segmentation using an image segmentation network 314 based on Fully Convolutional Networks (FCN) to obtain the final instance segmentation result 326.

[0090] Understandably, Mask R-CNN is derived from Faster R-CNN. Specifically, a Mask branch is added to the output module of Faster R-CNN to segment objects within each bounding box. The feature extraction network used for feature map extraction is crucial because Mask R-CNN searches for regions of interest (ROIs) on feature maps at different levels of the feature extraction network for subsequent precise object localization. Traditional CNN-based feature extraction networks have localized receptive fields, typically requiring network deepening to expand them. Attention-based methods, however, can capture global regional relationships. When performing instance segmentation on images of teeth within the oral cavity (i.e., RGB images in the current oral cavity image), considering the cluttered appearance of non-tooth regions within the oral cavity and the indistinct boundary between some teeth and gums, the Swing Transformer can be selected as the feature extraction network for Mask R-CNN. On one hand, it employs a sliding window mechanism similar to CNNs, calculating only the attention between patches within each window, thus solving the problem of high computational cost of the first-generation Vision Transformer. On the other hand, it achieves information interaction between different windows through pixel cyclic movement, ensuring smooth information flow between different regions and thereby improving the final tooth instance segmentation effect.

[0091] In another embodiment of this disclosure, the tooth position is determined based on a first tooth sub-point cloud, a tooth RGB image, and second tooth point clouds of multiple teeth in the user's oral cavity, including:

[0092] S2241. For each second tooth point cloud, after coarsely registering the first tooth sub-point cloud and the second tooth point cloud, determine the region intersection-union ratio of the overlapping area of ​​the projection region and the tooth RGB image when the second tooth point cloud is projected onto the plane of the tooth RGB image corresponding to the first tooth sub-point cloud.

[0093] Specifically, coarse registration refers to registration when the relative poses of two point clouds are completely unknown. Coarse registration can achieve a rough alignment between the first tooth sub-point cloud and the second tooth point cloud.

[0094] Specifically, when the target tooth is a tooth in the upper jaw, the multiple teeth in the user's mouth are all the teeth in the upper jaw; when the target tooth is a tooth in the lower jaw, the multiple teeth in the user's mouth are all the teeth in the lower jaw. Of course, the multiple teeth in the user's mouth can also be all the teeth in the user's mouth, and there is no limitation on this.

[0095] Any possible coarse registration method can be used to coarsely register the point cloud of the first tooth with the point cloud of the second tooth; no limitation is made here. For example, feature-based coarse registration methods, coarse registration methods based on the random sample consensus (RANSAC) framework, etc., can be used, but are not limited to these.

[0096] Specifically, after coarse registration of the first tooth sub-point cloud and the second tooth point cloud, the coordinate systems of the first tooth sub-point cloud and the second tooth point cloud are both the camera coordinate system corresponding to the first tooth sub-point cloud. Therefore, based on the camera's intrinsic parameters, the coordinates of any point on the second tooth point cloud can be calculated and projected onto the pixel coordinate system to obtain the projection area. In this way, the overlap area between the projection area and the tooth RGB image corresponding to the first tooth sub-point cloud can be calculated. The union area after taking the union of the projection area and the tooth RGB image corresponding to the first tooth point cloud can also be calculated. Then, the ratio of the overlap area to the union area can be calculated to obtain the region intersection-union ratio of the overlap area.

[0097] S2242. The tooth position corresponding to the second tooth point cloud with the largest intersection-union ratio in the region is taken as the tooth position of the target tooth.

[0098] Specifically, for each target tooth, the first tooth sub-point cloud of the target tooth is coarsely registered with the second tooth point clouds of multiple teeth in the user's oral cavity, the overlapping area is projected, and the region crossover ratio of the overlapping area is calculated. The tooth position of the tooth with the largest region crossover ratio is assigned to the target tooth.

[0099] Understandably, by traversing through multiple teeth in the user's mouth and matching each tooth with the target tooth (i.e., coarse registration and calculating the intersection-union ratio of the regions), and assigning the position of the tooth that best matches the target tooth to the target tooth, the method of determining the tooth position can be simplified and easy to implement. It can also avoid missing a match with a tooth in the mouth, thereby finding the tooth that best matches the target tooth and improving the accuracy of tooth position determination.

[0100] In another embodiment of this disclosure, the first tooth point cloud and the second tooth point cloud of the target tooth are compared to obtain the deviation value corresponding to the target tooth, including: performing coarse registration on the first tooth point cloud and the second tooth point cloud to obtain a first transformation matrix; continuing to perform fine registration on the first tooth point cloud and the second tooth point cloud to obtain a second transformation matrix; and obtaining the deviation value corresponding to the target tooth based on the first transformation matrix and the second transformation matrix.

[0101] Specifically, coarse registration can provide good initial values ​​for the transformation matrix (i.e., the first transformation matrix) for fine registration.

[0102] Those skilled in the art can use any possible coarse registration method to continue coarse registration of the point clouds of the first and second teeth, and there is no limitation here. For example, the Lepard method can be used, but it is not limited to this.

[0103] Specifically, fine registration involves calculating a more accurate transformation matrix given the initial value of the transformation matrix (i.e., the first transformation matrix) (which is already roughly correct).

[0104] Those skilled in the art can use any possible fine registration algorithm to continue fine registration of the first and second tooth point clouds after coarse registration, and there are no limitations here. For example, iterative closest point (IPC) algorithm, non-rigid iterative closest point (NICP) algorithm, etc., can be used, but are not limited to these.

[0105] Specifically, the first transformation matrix includes a first rotation matrix and a first translation matrix, and the second transformation matrix includes a second rotation matrix and a second translation matrix. The product of the first rotation matrix and the second rotation matrix, and the product of the first translation matrix and the second translation matrix, can be used as the rotation matrix and translation matrix of the first tooth point cloud and the second tooth point cloud of the target tooth, respectively. The pose deviation of the target tooth can be determined based on the rotation matrix, and the position deviation of the target tooth can be determined based on the translation matrix.

[0106] Understandably, coarse registration of the first and second tooth point clouds can achieve rapid initial alignment. Then, fine registration can be performed based on the initial alignment to achieve a more refined alignment effect. This can improve the alignment speed and accuracy of the first and second tooth point clouds.

[0107] Of course, after obtaining the rotation matrix and translation matrix between the first and second tooth point clouds of the target tooth, the second tooth point cloud can be rotated and translated according to the rotation matrix and translation matrix, and the error between the rotated and translated second tooth point cloud and the first tooth point cloud can be calculated. This verifies the accuracy of the rotation matrix and translation matrix, so that when the error exceeds the threshold, the first and second tooth point clouds of the target tooth can be compared again, or relevant personnel can be notified to check whether there is a problem with the orthodontic treatment monitoring.

[0108] Figure 4 This is a schematic diagram of the structure of an orthodontic treatment monitoring device provided in an embodiment of this disclosure. This orthodontic treatment monitoring device can be understood as the aforementioned electronic device or a functional module within the aforementioned electronic device. For example... Figure 4 As shown, the orthodontic treatment monitoring device 400 includes:

[0109] The first acquisition module 410 is used to acquire multiple frames of current intraoral images of the user at the current moment;

[0110] The first determining module 420 is used to determine the first tooth point cloud of each target tooth in the user's oral cavity based on multiple frames of current intraoral images;

[0111] The comparison module 430 is used to compare the first tooth point cloud and the second tooth point cloud of each target tooth to obtain the deviation value corresponding to the target tooth. The second tooth point cloud is the tooth point cloud of the tooth before the current time.

[0112] In another embodiment of this disclosure, the first determining module 420 may include:

[0113] The first determining submodule is used to determine the first tooth sub-point cloud and tooth position of each tooth in the current intraoral image for each frame of the current intraoral image, wherein the tooth position is used to indicate the arrangement position of the teeth on the jaw.

[0114] The first acquisition submodule is used to acquire, for each target tooth, the first tooth sub-point cloud of the tooth with the same tooth position as the target tooth in multiple frames of the current intraoral image, and obtain at least one first tooth sub-point cloud corresponding to the target tooth.

[0115] The second determination submodule is used to determine the first tooth point cloud of each target tooth based on at least one first tooth sub-point cloud corresponding to the target tooth.

[0116] In yet another embodiment of this disclosure, the current intraoral image includes an RGB image and a depth image;

[0117] The first determining submodule may include:

[0118] The instance segmentation unit is used to perform instance segmentation on the RGB image to obtain the instance segmentation result;

[0119] The region segmentation unit is used to segment the RGB image and depth image into regions based on the instance segmentation results, so as to obtain the RGB image and depth image of each tooth in the current oral cavity image;

[0120] The 3D reconstruction unit is used to perform 3D reconstruction for each tooth based on the tooth's RGB image and tooth depth image to obtain the first tooth sub-point cloud;

[0121] The first determining unit is used to determine the tooth position for each tooth based on the first tooth sub-point cloud of the tooth, the tooth RGB image, and the second tooth point cloud of multiple teeth in the user's oral cavity.

[0122] In another embodiment of this disclosure, the instance segmentation unit may include:

[0123] The feature extraction subunit is used to input the RGB image into the Swin Transformer-based feature extraction network to extract features and obtain a feature map.

[0124] The first determining subunit is used to determine the instance segmentation result based on the feature map.

[0125] In another embodiment of this disclosure, the first determining unit may include:

[0126] The coarse registration and projection sub-unit is used to determine the region crossover ratio of the overlapping area between the projection area and the tooth RGB image when the second tooth point cloud is projected onto the plane of the tooth RGB image corresponding to the first tooth sub-point cloud for each second tooth point cloud.

[0127] The second determining sub-unit is used to take the tooth position of the tooth corresponding to the second tooth point cloud with the largest intersection-union ratio of the region as the tooth position of the target tooth.

[0128] In another embodiment of this disclosure, the second determining submodule may include:

[0129] The first determining unit is used to perform fusion processing on at least one first tooth sub-point cloud corresponding to the target tooth and then perform sampling processing to obtain the first tooth point cloud of the target tooth.

[0130] In another embodiment of this disclosure, the comparison module 430 may include:

[0131] The coarse registration submodule is used to perform coarse registration on the first tooth point cloud and the second tooth point cloud to obtain the first transformation matrix.

[0132] The fine registration submodule is used to further fine register the first tooth point cloud and the second tooth point cloud to obtain the second transformation matrix.

[0133] The third determining submodule is used to obtain the deviation value corresponding to the target tooth based on the first transformation matrix and the second transformation matrix.

[0134] The apparatus provided in this embodiment can execute the methods of any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0135] This disclosure also provides an electronic device, which includes: a memory storing a computer program; and a processor for executing the computer program, wherein when the computer program is executed by the processor, it can implement the methods of any of the above embodiments.

[0136] Example, Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device 500 in the embodiments of this disclosure. The electronic device 500 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0137] like Figure 5As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0138] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0139] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0140] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0141] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0142] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0143] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0144] Acquire multiple frames of the user's intraoral image at the current moment;

[0145] Based on multiple frames of current intraoral images, determine the first tooth point cloud of each target tooth in the user's oral cavity;

[0146] For each target tooth, the first tooth point cloud and the second tooth point cloud of the target tooth are compared to obtain the deviation value corresponding to the target tooth. The second tooth point cloud is the tooth point cloud of the tooth before the current time.

[0147] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0150] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0151] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0152] This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0153] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for monitoring orthodontic treatment, characterized in that, include: Acquire multiple frames of current intraoral images of the user at the current moment, wherein the current intraoral images include RGB images; Based on the multiple frames of current intraoral images, a first tooth point cloud is determined for each target tooth in the user's oral cavity. The first tooth point cloud is the first tooth sub-point cloud of the tooth whose position is the same as the target tooth in the first tooth sub-point clouds of each tooth in the multiple frames of current intraoral images. The tooth positions of each tooth in the multiple frames of current intraoral images are obtained as follows: For each second tooth point cloud, after coarse registration of the first tooth sub-point cloud and the second tooth point cloud, the intersection-union ratio (IUGR) of the projection area of ​​the second tooth point cloud onto the plane of the corresponding tooth RGB image of the first tooth sub-point cloud and the region of the tooth RGB image is determined. The tooth position corresponding to the second tooth point cloud with the largest IUGR is taken as the tooth position of the tooth. The tooth RGB image is the image region containing the tooth in the RGB image. For each target tooth, the first tooth point cloud and the second tooth point cloud of the target tooth are compared to obtain the deviation value corresponding to the target tooth, wherein the second tooth point cloud is the tooth point cloud of the tooth before the current time.

2. The method according to claim 1, characterized in that, The step of determining the first tooth point cloud for each target tooth in the user's oral cavity based on the multiple frames of current intraoral images includes: For each frame of the current intraoral image, determine the first tooth sub-point cloud and tooth position of each tooth in the current intraoral image, wherein the tooth position is used to indicate the arrangement position of the teeth on the jaw. For each target tooth, a first tooth sub-point cloud of the tooth with the same tooth position as the target tooth in the current intraoral images of the multiple frames is obtained, and at least one first tooth sub-point cloud corresponding to the target tooth is obtained. For each target tooth, a first tooth point cloud is determined based on at least one first tooth sub-point cloud corresponding to the target tooth.

3. The method according to claim 2, characterized in that, The current intraoral image includes an RGB image and a depth image; The step of determining the first tooth sub-point cloud and tooth position of each tooth in the current intraoral image includes: The RGB image is segmented into instances to obtain the instance segmentation results; Based on the instance segmentation results, the RGB image and the depth image are segmented into regions to obtain the RGB image and depth image of each tooth in the current intraoral image; For each tooth, a three-dimensional reconstruction is performed based on the tooth's RGB image and tooth depth image to obtain the first tooth sub-point cloud; For each tooth, the tooth position is determined based on the first tooth sub-point cloud of the tooth, the tooth RGB image, and the second tooth point cloud of multiple teeth in the user's oral cavity.

4. The method according to claim 3, characterized in that, The step of performing instance segmentation on the RGB image to obtain instance segmentation results includes: The RGB image is input into a Swin Transformer-based feature extraction network to extract features and obtain a feature map. Based on the feature map, the instance segmentation result is determined.

5. The method according to claim 3, characterized in that, The method of determining the tooth position based on the first tooth sub-point cloud, the tooth RGB image, and the second tooth point cloud of multiple teeth in the user's oral cavity includes: For each second tooth point cloud, after coarsely registering the first tooth sub-point cloud with the second tooth point cloud, the region intersection-union ratio of the overlapping region of the projection area and the tooth RGB image when the second tooth point cloud is projected onto the plane of the tooth RGB image corresponding to the first tooth sub-point cloud is determined. The tooth position of the tooth corresponding to the second tooth point cloud with the largest intersection-union ratio is taken as the tooth position of the tooth.

6. The method according to claim 2, characterized in that, Determining the first tooth point cloud of the target tooth based on at least one first tooth sub-point cloud corresponding to the target tooth includes: The first tooth point cloud of the target tooth is obtained by fusing at least one first tooth sub-point cloud and then sampling.

7. The method according to claim 1, characterized in that, The first tooth point cloud and the second tooth point cloud of the target tooth are compared to obtain the deviation value corresponding to the target tooth, including: Coarse registration is performed on the first tooth point cloud and the second tooth point cloud to obtain the first transformation matrix; Continue to perform fine registration on the first tooth point cloud and the second tooth point cloud to obtain the second transformation matrix; Based on the first transformation matrix and the second transformation matrix, the deviation value corresponding to the target tooth is obtained.

8. An orthodontic treatment monitoring device, characterized in that, include: The first acquisition module is used to acquire multiple frames of current intraoral images of the user at the current moment, wherein the current intraoral images include RGB images; The first determining module is used to determine the first tooth point cloud of each target tooth in the user's oral cavity based on the multiple frames of current intraoral images. The first tooth point cloud is the first tooth sub-point cloud of the tooth whose position is the same as the target tooth in the first tooth sub-point cloud of each tooth in the multiple frames of current intraoral images. The tooth positions of each tooth in the multiple frames of current intraoral images are obtained as follows: for each second tooth point cloud, after coarse registration of the first tooth sub-point cloud and the second tooth point cloud, the intersection-union ratio (IUGR) of the projection area of ​​the second tooth point cloud onto the plane of the corresponding RGB image of the tooth is determined, and the tooth position corresponding to the second tooth point cloud with the largest IUGR is taken as the tooth position of the tooth. The RGB image of the tooth is the image region of the RGB image containing the tooth. The comparison module is used to compare the first tooth point cloud and the second tooth point cloud of each target tooth to obtain the deviation value corresponding to the target tooth, wherein the second tooth point cloud is the tooth point cloud of the tooth before the current time.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores a computer program that, when executed by the processor, performs the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images