Orthodontic treatment monitoring method and apparatus, device, and storage medium

Abstract

The present invention relates to an orthodontic treatment monitoring method and apparatus, a device, and a storage medium. The orthodontic treatment monitoring method comprises: acquiring multiple intraoral images and a first dental model of a user; determining initial transformation parameters corresponding to each pair of tooth models in the first dental model and a second dental model corresponding to the multiple intraoral images; determining a loss function corresponding to initial camera parameters and the initial transformation parameters; if the loss function corresponding to the initial camera parameters and the initial transformation parameters satisfies a preset condition, determining the initial transformation parameters to be target transformation parameters. According to an embodiment of the present disclosure, a user can monitor orthodontic treatment more conveniently.

Description

Orthodontic treatment monitoring method, device, equipment and storage medium

[0001] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on June 12, 2023, with application number 202310693808.1 and invention name “Orthodontic treatment monitoring method, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to an orthodontic treatment monitoring method, apparatus, device, and storage medium. Background Art

[0003] Orthodontics is the process of straightening teeth, correcting misalignment and deformity, and achieving beautiful teeth.

[0004] At present, orthodontic treatment monitoring mainly relies on patients to visit the clinic for regular checkups, and the doctor evaluates the orthodontic effect of the current treatment stage through visual observation, or the doctor uses a mouth scanner to scan the patient's teeth, and then quantitatively compares the scanned teeth with the expected teeth to evaluate the orthodontic effect of the current treatment stage. However, some patients are unable to visit the clinic for regular checkups due to busy work, time and space constraints, etc., resulting in the inability to adjust the dental braces in a timely manner, which in turn leads to longer orthodontic treatment cycles, poor treatment results, and poor patient treatment experience. Therefore, there is an urgent need for a more convenient orthodontic treatment monitoring method.

[0005] Summary of the Invention

[0006] (1) Technical issues to be resolved

[0007] The technical problem to be solved by the present disclosure is to solve the problem that some patients are unable to visit the clinic regularly for checkups due to busy work, time and space constraints, etc., resulting in the inability to adjust their orthodontic appliances in a timely manner, which in turn leads to longer orthodontic treatment cycles, poor treatment results, and a poor orthodontic experience for patients. Therefore, there is an urgent need for a more convenient orthodontic treatment monitoring method.

[0008] (2) Technical solution

[0009] In order to solve the above technical problems, the embodiments of the present disclosure provide an orthodontic treatment monitoring method, device, equipment and storage medium.

[0010] In a first aspect, the present disclosure provides an orthodontic treatment monitoring method, the method comprising:

[0011] Acquire multiple intraoral images of the user and a first dental model, wherein the first dental model includes at least one first tooth model;

[0012] Determining initial transformation parameters corresponding to each pair of tooth models in the first dental jaw model and the second dental jaw model corresponding to the multiple intra-oral images, wherein the second dental jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0013] Determining a loss function corresponding to initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of the tooth models after a similarity transformation is performed on each pair of the tooth models according to the corresponding initial transformation parameters;

[0014] If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet a preset condition, the initial transformation parameters are determined as target transformation parameters.

[0015] In a second aspect, the present disclosure provides an orthodontic treatment monitoring device, the device comprising:

[0016] A first acquisition module is configured to acquire a plurality of intraoral images of a user and a first dental model, wherein the first dental model includes at least one first tooth model;

[0017] a first determining module, configured to determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0018] a second determination module, configured to determine a loss function corresponding to initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the plurality of intraoral images, and the loss function is configured to characterize a total deviation between the first tooth model and the second tooth model in each pair of the tooth models after a similarity transformation is performed on each pair of the tooth models according to the corresponding initial transformation parameters;

[0019] The third determining module is configured to determine the initial transformation parameters as target transformation parameters if the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet a preset condition.

[0020] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the device comprising:

[0021] one or more processors;

[0022] a storage device for storing one or more programs,

[0023] When one or more programs are executed by one or more processors, the one or more processors implement the orthodontic treatment monitoring method provided by the first aspect.

[0024] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the orthodontic treatment monitoring method provided in the first aspect.

[0025] (3) Beneficial effects

[0026] The above-mentioned technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art: an orthodontic treatment monitoring method, apparatus, device and storage medium of the embodiments of the present disclosure can obtain multiple intraoral images of the user, and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model; determine the initial transformation parameters corresponding to each pair of tooth models in the first dental jaw model and the second dental jaw model corresponding to the multiple intraoral images, wherein the second dental jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth; determine the loss function corresponding to the initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are the camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of tooth models after each pair of tooth models is similarly transformed according to the corresponding initial transformation parameters; if the loss function corresponding to the initial camera parameters and the initial transformation parameters meets the preset conditions, the initial transformation parameters are determined as the target transformation parameters. It can be seen that by adopting the above technical solution and using an oral scanner to scan the user's teeth, only multiple intraoral images are needed to obtain the target transformation parameters corresponding to each pair of tooth models in the second jaw model and the first jaw model (that is, the jaw model corresponding to the expected orthodontic effect) corresponding to the multiple intraoral images. The doctor can know the orthodontic treatment effect based on the target transformation parameters, so that the user can monitor the orthodontic treatment more conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0028] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] FIG1 is a schematic diagram showing a flow chart of an orthodontic treatment monitoring method provided by an embodiment of the present disclosure;

[0030] FIG2 is a schematic diagram showing a flow chart of another orthodontic treatment monitoring method provided by an embodiment of the present disclosure;

[0031] FIG3 shows a schematic structural diagram of an orthodontic treatment monitoring device provided by an embodiment of the present disclosure;

[0032] FIG4 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure are clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0034] In order to solve the above problems, the embodiments of the present disclosure provide an orthodontic treatment monitoring method, device, equipment and storage medium.

[0035] The orthodontic treatment monitoring method provided by the embodiments of the present disclosure is described below with reference to Figures 1 to 3. In the embodiments of the present disclosure, the orthodontic treatment monitoring method can be performed by an electronic device. The electronic device can include a device with communication capabilities, such as a tablet computer, desktop computer, or laptop computer, or a device simulated by a virtual machine or simulator.

[0036] FIG1 shows a schematic flow chart of an orthodontic treatment monitoring method provided by an embodiment of the present disclosure.

[0037] As shown in FIG1 , the orthodontic treatment monitoring method may include the following steps.

[0038] S110 , obtaining a plurality of intraoral images of the user and a first dental model, wherein the first dental model includes at least one first tooth model.

[0039] Specifically, the intraoral image is an image obtained by capturing the interior of the oral cavity using an imaging device. The image capturing device may include a mobile phone, a computer, a camera, etc., but is not limited thereto.

[0040] Exemplarily, obtaining multiple intraoral images of the user may include: receiving multiple intraoral images of the user sent by other electronic devices, or reading multiple intraoral images of the user from a storage device, etc., but is not limited thereto.

[0041] Specifically, the first dental model is a dental model corresponding to a desired dental orthodontic effect, and the first dental model includes a first tooth model corresponding to at least one tooth.

[0042] The first dental model may include a dental model corresponding to the first stage orthodontic effect, a dental model corresponding to the second stage orthodontic effect, or a dental model corresponding to the final orthodontic effect, but is not limited thereto.

[0043] The specific representation form of the first dental jaw model and the first tooth model may include a three-dimensional mesh model.

[0044] Exemplarily, obtaining the first dental model may include: performing mesh segmentation on the models corresponding to the teeth and gums to segment the tooth models corresponding to each tooth, and using DeepSDF technology to complete the segmented tooth models into a watertight mesh to obtain the first dental model, but is not limited thereto.

[0045] S120. Determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the multiple intraoral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth.

[0046] Specifically, the second dental model is a dental model corresponding to (or behind) multiple intraoral images, and the second dental model includes at least one second tooth model. Those skilled in the art should understand that the first dental model is a dental model corresponding to the desired dental orthodontic effect, and the second dental model is a dental model corresponding to multiple intraoral images. Since the first dental model and the second dental model both correspond to the user's teeth, the first dental model can be transformed into the second dental model by optimizing the posture of each first tooth model in the first dental model, and then the difference between each first tooth model and the first tooth model corresponding thereto can be obtained. Based on this, the embodiment of the present disclosure needs to determine the target transformation parameters for transforming the first dental model into the second dental model.

[0047] Specifically, for each pair of tooth models, the geometry of the first tooth model and the second tooth model included therein are consistent, and the difference between the two is a similarity transformation. In other words, the first tooth model and the second tooth model can be aligned by performing a similarity transformation on the first tooth model. The transformation parameters based on the similarity transformation include a scale, a translation matrix, and a translation matrix. Since the scale between each pair of tooth models is the scale between the world coordinate systems where the first dental model and the second dental model are located, the scale of each pair of tooth models is the same. In this way, each tooth model pair in the first dental model and the second dental model includes a common scale, and their corresponding translation matrices, and rotation matrices.

[0048] Specifically, the initial transformation parameters are initial transformation parameters corresponding to the tooth model pair.

[0049] In some embodiments, S120 may include: randomly generating initial transformation parameters corresponding to each pair of tooth models.

[0050] In other embodiments, S120 may include: performing three-dimensional reconstruction based on multiple intraoral images to obtain a third dental model; performing coarse registration processing on the first dental model and the third dental model to obtain initial transformation parameters corresponding to each pair of dental models.

[0051] Specifically, the third dental jaw model is a dental jaw model corresponding to the user's dental jaw obtained by three-dimensional reconstruction based on multiple intraoral images.

[0052] The specific representation form of the third dental model may include a three-dimensional point cloud model, a three-dimensional mesh model, etc., which is not limited here.

[0053] Specifically, the initial camera parameters are rough camera parameters corresponding to the image acquisition of multiple intraoral images. In the disclosed embodiment, during the process of optimizing the posture of each first tooth model in the first jaw model so that the first jaw model becomes the second jaw model, the actual camera parameters corresponding to the image acquisition of the multiple intraoral images (i.e., target camera parameters) are gradually determined. For example, the initial camera parameters are optimized to gradually approach the target camera parameters.

[0054] Among them, camera parameters may include camera intrinsic parameters and camera extrinsic parameters. Camera intrinsic parameters may include camera focal length, and the projection coordinates of the camera center on the pixel plane, etc. Camera extrinsic parameters may include camera pose. The camera pose may include, for example, camera rotation matrix and camera translation matrix. The camera rotation matrix and camera translation matrix together describe how to transform a point from the world coordinate system to the camera coordinate system.

[0055] For example, the Colmap algorithm is used to estimate the camera parameters (i.e., initial camera parameters) and sparse point cloud (i.e., third dental model) when acquiring multiple intraoral images. If the image acquisition device is a pinhole camera, the camera parameters include three camera intrinsic parameters, i.e., the camera focal length f, and the projection coordinates of the camera center on the pixel plane (c u ,c v ), each intraoral image corresponds to 6 camera extrinsics (corresponding to 6 degrees of freedom).

[0056] Specifically, the first and third dental models are roughly registered to obtain transformation parameters for overall alignment of the first and third dental models, and the transformation parameters are used as initial transformation parameters corresponding to each dental model pair.

[0057] For example, the first and third dental jaw models may be coarsely registered using a feature-based coarse registration method, a coarse registration method based on a random sample consensus (RANSAC) framework, etc., but the present invention is not limited thereto.

[0058] It is understandable that since the second and third jaw models both correspond to multiple intraoral images, when the first and third jaw models are roughly aligned as a whole, the first and second jaw models will also be roughly aligned as a whole, that is, the first and second tooth models in the tooth model pair are at least roughly aligned. Therefore, the initial transformation parameters obtained through the rough registration process are closer to the target transformation parameters, which is conducive to obtaining the target transformation parameters corresponding to the tooth model pair as soon as possible and improving the efficiency of orthodontic treatment monitoring. In addition, by performing rough registration processing on the first and third jaw models as a whole to obtain the initial transformation parameters, the initial transformation parameters corresponding to each pair of tooth models can be quickly obtained.

[0059] In some further embodiments, S120 may include: performing three-dimensional reconstruction based on multiple intraoral images to obtain a third dental model, wherein the third dental model includes at least one third tooth model; performing coarse alignment processing on each pair of tooth models in the first dental model and the third dental model to obtain initial transformation parameters corresponding to each pair of tooth models in the first dental model and the second dental model.

[0060] S130. Determine the loss function corresponding to the initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are the camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of tooth models after each pair of tooth models is similarly transformed according to the corresponding initial transformation parameters.

[0061] Specifically, the deviation between the first tooth model and the second tooth model in the tooth model pair may include deviations in color values, mask values, etc., but is not limited thereto.

[0062] S140: If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions, the initial transformation parameters are determined as target transformation parameters.

[0063] Specifically, the initial camera parameters may also be determined as the target camera parameters.

[0064] Specifically, the preset condition may include: the loss function is less than a preset threshold, wherein the specific value of the preset threshold can be set by those skilled in the art according to actual conditions, but is not limited thereto.

[0065] Specifically, if the loss function corresponding to the initial camera parameters and the initial transformation parameters meets the preset conditions, it indicates that after the first tooth model in each pair of tooth models is similarly transformed according to its corresponding initial transformation parameters, the alignment accuracy with the second tooth model is high (or the requirements are met), and the difference between the initial camera parameters and the actual camera parameters (i.e., the actual camera parameters corresponding to the image acquisition of multiple intraoral images) is small (or the requirements are met). At this time, the initial transformation parameters can be determined as the target transformation parameters.

[0066] Those skilled in the art should understand that, since the first tooth model in the first jaw model is aligned with the second jaw model after similarity transformation according to its corresponding target transformation parameters, in other words, the jaw model obtained after similarity transformation of the first tooth model in the first jaw model according to its corresponding target transformation parameters can be regarded as the second jaw model. Therefore, the tooth position difference between the first tooth model and the second tooth model can be determined according to the target transformation parameters to realize orthodontic treatment monitoring.

[0067] It is understandable that the technical solution provided by the embodiment of the present disclosure eliminates the need for users to go to the clinic in person to have the doctor observe with the naked eye or use an oral scanner to scan their teeth. Instead, they only need to take multi-view images of their own teeth (i.e., multiple intraoral images) and then upload them. In this way, users do not need to go to the clinic frequently for follow-up visits, which saves users' time and frees them from time and space constraints. In addition, since doctors do not need to make judgments with the naked eye, nor do they need to operate an oral scanner to frequently scan users' teeth, doctors' time can also be saved. Moreover, compared to doctors' visual judgment, the technical solution provided by the embodiment of the present disclosure can quantitatively obtain the target transformation parameters corresponding to each pair of tooth models, avoiding misjudgment problems caused by doctors' lack of experience and other reasons, and improving the accuracy of orthodontic treatment monitoring.

[0068] In the embodiment of the present disclosure, there is no need to use an oral scanner to scan the user's teeth. Only multiple intraoral images are needed to obtain the target transformation parameters corresponding to each pair of tooth models in the second jaw model and the first jaw model (that is, the jaw model corresponding to the expected orthodontic effect) corresponding to the multiple intraoral images. The doctor can know the orthodontic treatment effect based on the target transformation parameters, so that the user can monitor the orthodontic treatment more conveniently.

[0069] In another embodiment of the present disclosure, the method further comprises:

[0070] S150: If the loss function corresponding to the initial camera parameters and the initial transformation parameters does not meet the preset conditions, the initial camera parameters and the initial transformation parameters are optimized until the corresponding loss function meets the preset conditions.

[0071] S160: Determine the optimized initial transformation parameters as target transformation parameters.

[0072] Specifically, the optimized initial camera parameters may also be determined as target camera parameters.

[0073] Specifically, if the loss function corresponding to the initial camera parameters and the initial transformation parameters does not meet the preset conditions, it indicates that after the first tooth model in at least one pair of tooth models is similarly transformed according to its corresponding initial transformation parameters, the alignment accuracy with the second tooth model is low (or does not meet the requirements), and the gap between the initial camera parameters and the actual camera parameters is large (or does not meet the requirements). In this case, the initial camera parameters and the initial transformation parameters can be optimized, and then the loss function corresponding to the optimized initial camera parameters and the initial transformation parameters is determined. If the loss function corresponding to the optimized initial camera parameters and the initial transformation parameters meets the preset conditions, the optimized initial transformation parameters are determined as the target transformation parameters. If the loss function corresponding to the optimized initial camera parameters and the initial transformation parameters does not meet the preset conditions, the initial camera parameters and the initial transformation parameters are continuously optimized until the corresponding loss function meets the preset conditions, and finally the optimized initial transformation parameters are determined as the target transformation parameters. Of course, the final optimized initial camera parameters can also be determined as the target camera parameters.

[0074] It can be understood that by continuously optimizing the initial camera parameters and initial transformation parameters, the alignment accuracy of the first and second tooth models can be increasingly improved after the tooth model pairs undergo similarity transformations according to the corresponding optimized initial transformation parameters, thereby obtaining the target transformation parameters for each pair of tooth models. This makes the acquisition of the target transformation parameters simple and convenient.

[0075] Figure 2 shows a flow chart of another orthodontic treatment monitoring method provided by an embodiment of the present disclosure. The present embodiment is optimized based on the above embodiment, and the present embodiment can be combined with various optional solutions in one or more of the above embodiments.

[0076] As shown in FIG2 , the orthodontic treatment monitoring method may include the following steps.

[0077] S210: Acquire multiple intraoral images of the user and a first dental model, wherein the first dental model includes at least one first tooth model.

[0078] Specifically, S210 is similar to S110 and will not be described in detail here.

[0079] S220. Determine initial transformation parameters corresponding to each pair of tooth models in the first dental model and the second dental model corresponding to the multiple intraoral images, wherein the second dental model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth.

[0080] Specifically, S220 is similar to S120 and will not be described in detail here.

[0081] S230. For each pixel point in the multiple intraoral images, based on the initial camera parameters and the initial transformation parameters, detect whether the ray corresponding to the pixel point in the world coordinate system and the first dental model after similarity transformation according to the initial transformation parameters intersect, obtain the detection result, and determine the first color value and the first mask value of the pixel point based on the detection result.

[0082] Specifically, the first color value is the color value of the pixel estimated based on the initial camera parameters and the initial transformation parameters. The color value may include a grayscale value or an RGB value, but is not limited thereto.

[0083] Specifically, the first mask value is the mask value of the pixel estimated based on the initial camera parameters and the initial transformation parameters. The mask value is used to indicate whether the pixel is a background pixel or a pixel corresponding to a tooth. For example, (0,0,0) is used to indicate a background pixel, and (1,1,1) is used to indicate a pixel corresponding to a tooth.

[0084] Specifically, the first dental jaw model after similarity transformation according to the initial transformation parameters refers to the first dental jaw model after similarity transformation of each first tooth model according to the corresponding initial transformation parameters.

[0085] Specifically, for each pixel point, the ray corresponding to the pixel point in the world coordinate system is determined based on the initial camera parameters corresponding to the intra-oral image where the pixel point is located. Then, it is detected whether the ray corresponding to the pixel point and the first dental model after similarity transformation according to the initial transformation parameters intersect to obtain the detection result. Then, the first color value of the pixel point is determined based on the detection result.

[0086] In some embodiments, the first color value of the pixel point is determined based on the detection result, including: if the detection result is intersection, querying the initial color field network based on the intersection coordinates and the direction of the ray to obtain the first color value of the pixel point; if the detection result is non-intersection, determining the first color value of the pixel point to be 0.

[0087] Specifically, the network parameters in the initial color field network are obtained by random settings.

[0088] For example, the initial color field network is denoted as f θ (x,ν), which is a function of the three-dimensional coordinate x and direction ν, outputs the RGB value rendered by NDS, and θ is the network parameter. If the intersection coordinate is X0 and the direction of the ray is V0, bring X0 and V0 into f θ (x,ν), you can get the first color value.

[0089] In some embodiments, determining the first mask value of the pixel point based on the detection result includes: if the detection result is intersection, determining the first mask value of the pixel point to be a first value; if the detection result is non-intersection, determining the first mask value of the pixel point to be a second value.

[0090] The first value is used to indicate that the pixel is a tooth pixel, and the second value is used to indicate that the pixel is a background pixel. For example, the first value may be (1, 1, 1), and the second value may be (0, 0, 0).

[0091] S240 , determining color loss according to the first color value and the second color value of each pixel point in the plurality of intra-oral images.

[0092] Specifically, the second color value is the actual color value of the pixel in the intraoral image to which it belongs.

[0093] In some embodiments, S240 includes: determining, for each pixel in the plurality of intraoral images, a color difference between a first color value and a second color value of the pixel; and determining color loss based on the color difference of each pixel in the plurality of intraoral images.

[0094] Exemplarily, determining the color loss based on the color difference of each pixel in the plurality of intraoral images may include: summing the squares (or absolute values, etc.) of the color difference of each pixel to obtain the color loss, but is not limited thereto.

[0095] S250 , determining a mask loss according to a first mask value and a second mask value of each pixel point in the multiple intraoral images, wherein the second mask value is determined by multiple mask images corresponding to the multiple intraoral images.

[0096] Specifically, for each intraoral image, semantic segmentation processing is performed on the intraoral image to distinguish the tooth part and the background part, and the mask value of the pixel point of the tooth part is set to the first value, and the mask value of the pixel point of the background part is set to the second value, so as to obtain the corresponding mask image.

[0097] Specifically, the second mask value is the real mask value of the pixel in the corresponding mask image.

[0098] In some embodiments, S250 includes: determining, for each pixel in the plurality of intraoral images, a mask difference between a first mask value and a second mask value of the pixel; and determining a mask loss based on the mask difference of each pixel in the plurality of intraoral images.

[0099] Exemplarily, determining the mask loss based on the mask difference of each pixel in the plurality of intraoral images may include: summing the squares (or absolute values, etc.) of the mask differences of each pixel to obtain the mask loss, but is not limited thereto.

[0100] S260: Determine a loss function corresponding to the initial camera parameters and the initial transformation parameters based on the color loss and the mask loss.

[0101] In some embodiments, S260 includes: performing weighted summation processing on the color loss and the mask loss to obtain a loss function corresponding to the initial camera parameters and the initial transformation parameters.

[0102] It is understandable that the weighted summation process can flexibly adjust the proportions of color loss and mask loss, respectively, so that the loss function focuses on the loss type that is more concerned, and thus the target transformation parameters more accurately reflect the differences in the aspects that doctors are more concerned about in the tooth model.

[0103] Of course, you can also directly add the color loss and mask loss to get the loss function.

[0104] To illustrate the process of determining the loss function provided by the embodiment of the present disclosure in detail, a detailed description will be given below based on a specific example.

[0105] N semantically segmented intraoral images are denoted as Among them, I i is the i-th intraoral image after semantic segmentation, the pixel coordinates of a pixel in the intraoral image after semantic segmentation are marked (u, v), and the corresponding RGB value (i.e., the second color value) is recorded as C (u,v) For each intraoral image after semantic segmentation, the mask value of the pixel corresponding to the tooth part is recorded as 1, and the mask value of the pixel corresponding to the background part is recorded as 0 to generate a mask image. The mask image generated after semantic segmentation of N intraoral images is recorded as Among them, m i is the mask image generated after semantic segmentation of the i-th intraoral image. Since each intraoral image is taken by the same image acquisition device, the camera intrinsic parameters corresponding to the N intraoral images are the same, denoted as (f,c u ,c v ). Each intraoral image has an independent camera pose, and the camera extrinsic parameter of the i-th intraoral image is denoted as p i ∈R 6 (6 degrees of freedom), the overall camera parameters are recorded as Φ=(f,c u ,c v ,p1,p2,...,p N )∈R 6N+3 .

[0106] The first dental model includes K first tooth models, and the K first tooth models have a common scaling ratio s and independent rigid body transformation parameters (i.e., translation matrix and rotation matrix) o i ∈R 6 (6 degrees of freedom), the overall note is Ψ=(s,o1,o2,...,o K )∈R 6K+1 .

[0107] The color field network of the first dental model is denoted as f θ (x,ν), which is a function of the three-dimensional spatial coordinate x and direction ν, outputs the RGB value rendered by NDS, and θ is the network parameter.

[0108] The process of calculating the loss function corresponding to the initial camera parameters and initial transformation parameters is as follows: for each pixel in the N intraoral images, the ray of the pixel in the world coordinate system can be calculated through the initial camera parameters corresponding to the intraoral image to which it belongs, and then the intersection of the ray and the first dental model after similarity transformation according to the initial transformation parameters can be calculated. If there is no intersection, the color rendered by NDS (i.e., the first color value) is 0; if there is intersection, the intersection coordinates and ray direction are substituted into f θ (x, ν), we get the color rendered by NDS (i.e. the first color value). Since the intersection coordinates are related to the camera parameters and transformation parameters, and the color rendered by NDS is also related to the network parameters of the color field network, the color rendered by NDS (i.e. the first color value) is recorded as If there is no intersection, the first mask value of the pixel is (0,0,0); if there is intersection, the first mask value of the pixel is (1,1,1). Since whether there is intersection is related to the camera parameters and transformation parameters, the first mask value is recorded as

[0109] Compare the first color value and the second color value of each pixel in N intra-oral images, and the color loss is:

[0110] Among them, L color (θ, Φ, Ψ) is the color loss, is the first color value of the pixel with pixel coordinates (u, v), C (u,v) The second color value of the pixel with pixel coordinates (u, v).

[0111] Compare the first mask value and the second mask value of each pixel in N intra-mouth images, and get the mask loss as:

[0112] Among them, L mask(Φ,Ψ) is the mask loss, is the first mask value of the pixel with pixel coordinates (u, v), m (u,v) The second mask value of the pixel with pixel coordinates (u, v).

[0113] The weighted sum of color loss and mask loss is used to obtain the loss function:

[0114] L(θ, Φ, Ψ) = αL color +λL mask

[0115] Among them, L(θ,Φ,Ψ) is the loss function, α is the color loss L color The weight value of λ is the mask loss L mask The weight value of .

[0116] S270: If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions, the initial transformation parameters are determined as target transformation parameters.

[0117] Specifically, if the loss functions corresponding to the initial camera parameters, initial transformation parameters, and initial color field network do not meet the preset conditions, the initial transformation parameters are determined as the target transformation parameters. In this case, the initial camera parameters are the target camera parameters, and the initial color field network is the color field network of the second dental model.

[0118] Specifically, if the loss functions corresponding to the initial camera parameters, initial transformation parameters, and initial color field network do not meet the preset conditions, the initial camera parameters, initial transformation parameters, and initial color field network are optimized until the corresponding loss functions meet the preset conditions. At this point, the optimized initial camera parameters become the target camera parameters, and the optimized initial color field network becomes the color field network of the second dental model.

[0119] Those skilled in the art should understand that "determining the loss function corresponding to the optimized initial camera parameters, the optimized initial transformation parameters, and the optimized color field network" is similar to S230-S260, except that the optimized initial camera parameters, the optimized initial transformation parameters, and the optimized color field network are involved in the loss function calculation, which will not be repeated here.

[0120] The disclosed embodiment utilizes NDS (Neural Deferred Shading) to jointly optimize camera parameters and transformation parameters, and can simply and quickly extract target transformation parameters from multiple intraoral images and the first dental model taken by the user, thereby obtaining tooth posture differences that accurately meet orthodontic requirements.

[0121] FIG3 shows a schematic structural diagram of an orthodontic treatment monitoring device provided by an embodiment of the present disclosure.

[0122] As shown in FIG3 , the orthodontic treatment monitoring device 300 may include:

[0123] A first acquisition module 310 is configured to acquire a plurality of intraoral images of a user and a first dental model, wherein the first dental model includes at least one first tooth model;

[0124] A first determining module 320 is configured to determine initial transformation parameters corresponding to each pair of tooth models in the first dental jaw model and the second dental jaw model corresponding to the plurality of intraoral images, wherein the second dental jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0125] a second determination module 330 for determining a loss function corresponding to initial camera parameters and initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the plurality of intraoral images, and the loss function is used to characterize a total deviation between the first tooth model and the second tooth model in each pair of tooth models after similarity transformation is performed on each pair of tooth models according to the corresponding initial transformation parameters;

[0126] The third determining module 340 is configured to determine the initial transformation parameters as target transformation parameters if the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet a preset condition.

[0127] In another embodiment of the present disclosure, the device further includes:

[0128] An optimization module, configured to optimize the initial camera parameters and the initial transformation parameters until the corresponding loss functions satisfy the preset conditions if the loss functions corresponding to the initial camera parameters and the initial transformation parameters do not satisfy the preset conditions;

[0129] The fourth determining module is configured to determine the optimized initial transformation parameters as target transformation parameters.

[0130] In yet another embodiment of the present disclosure, the first determining module 320 includes:

[0131] The reconstruction submodule is used to perform three-dimensional reconstruction based on multiple intraoral images to obtain the third dental model;

[0132] The registration submodule is used to perform a coarse registration process on the first dental model and the third dental model to obtain the initial transformation parameters corresponding to each pair of dental models.

[0133] In yet another embodiment of the present disclosure, the second determining module 330 includes:

[0134] A first determination submodule is used to detect, for each pixel point in the plurality of intraoral images, whether a ray corresponding to the pixel point in the world coordinate system intersects with a first dental model after similarity transformation according to the initial transformation parameters, based on initial camera parameters and initial transformation parameters, obtain a detection result, and determine a first color value and a first mask value of the pixel point according to the detection result;

[0135] a second determining submodule, configured to determine color loss based on the first color value and the second color value of each pixel in the plurality of intraoral images;

[0136] a third determination submodule, configured to determine a mask loss based on a first mask value and a second mask value of each pixel in the plurality of intraoral images, wherein the second mask value is determined by a plurality of mask images corresponding to the plurality of intraoral images;

[0137] The fourth determination submodule is used to determine the loss function corresponding to the initial camera parameters and the initial transformation parameters based on the color loss and the mask loss.

[0138] In yet another embodiment of the present disclosure, the first sub-determining module includes:

[0139] a query unit, configured to query the initial color field network to obtain a first color value of the pixel point according to the coordinates of the intersection point and the direction of the ray if the detection result is an intersection;

[0140] A first determining unit, configured to determine a first color value of the pixel point to be 0 if the detection result is non-intersection;

[0141] The second determining submodule includes:

[0142] a second determining unit, configured to determine, for each pixel in the plurality of intraoral images, a color difference between a first color value and a second color value of the pixel;

[0143] The third determining unit is configured to determine the color loss according to the color difference of each pixel in the plurality of intraoral images.

[0144] In yet another embodiment of the present disclosure, the first sub-determining module includes:

[0145] a fourth determining unit, configured to determine a first mask value of the pixel point as a first value if the detection result is an intersection;

[0146] a fifth determining unit, configured to determine the first mask value of the pixel point to be a second value if the detection result is non-intersection;

[0147] The third determining submodule includes:

[0148] a sixth determining unit, configured to determine, for each pixel point in the plurality of intraoral images, a mask difference between the first mask value and the second mask value of the pixel point;

[0149] The seventh determining unit is configured to determine the mask loss according to the mask difference of each pixel point in the plurality of intraoral images.

[0150] In yet another embodiment of the present disclosure, the fourth determining submodule includes:

[0151] The eighth determination unit is used to perform weighted summation processing on the color loss and the mask loss to obtain a loss function corresponding to the initial camera parameters and the initial transformation parameters.

[0152] It should be noted that the orthodontic treatment monitoring device 300 shown in FIG3 can execute each step in the method embodiment shown in FIG1 to FIG2 and realize each process and effect in the method embodiment shown in FIG1 to FIG2, which will not be elaborated here.

[0153] FIG4 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0154] As shown in FIG4 , the electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0155] Specifically, the processor 401 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0156] Memory 402 may include a large-capacity memory for information or instructions. By way of example, and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway device. In certain embodiments, memory 402 is non-volatile solid-state memory. In certain embodiments, memory 402 includes read-only memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0157] The processor 401 reads and executes the computer program instructions stored in the memory 402 to perform the steps of the orthodontic treatment monitoring method provided in the embodiment of the present disclosure.

[0158] In one example, the electronic device may further include a transceiver 403 and a bus 404. As shown in FIG4 , the processor 401, the memory 402, and the transceiver 403 are connected via the bus 404 and communicate with each other.

[0159] Bus 404 includes hardware, software or both. For example, and not limitation, bus may include Accelerated Graphics Port (AGP) or other graphics bus, Extended Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industrial Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local Bus (VLB) bus or other suitable bus or two or more of these combinations. Where appropriate, bus 404 may include one or more buses. Although the present application embodiment describes and illustrates a specific bus, the present application considers any suitable bus or interconnection.

[0160] The following is an embodiment of the computer-readable storage medium provided in the embodiment of the present disclosure. The computer-readable storage medium and the orthodontic treatment monitoring method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the computer-readable storage medium, please refer to the embodiment of the above-mentioned orthodontic treatment monitoring method.

[0161] This embodiment provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform an orthodontic treatment monitoring method. The method includes:

[0162] Acquire multiple intraoral images of the user and a first dental model, wherein the first dental model includes at least one first tooth model;

[0163] Determining initial transformation parameters corresponding to each pair of tooth models in the first dental jaw model and the second dental jaw model corresponding to the plurality of intraoral images, wherein the second dental jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0164] determining a loss function corresponding to initial camera parameters and initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the plurality of intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of tooth models after a similarity transformation is performed on each pair of tooth models according to the corresponding initial transformation parameters;

[0165] If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet the preset conditions, the initial transformation parameters are determined as the target transformation parameters.

[0166] Of course, the storage medium containing computer-executable instructions provided in the embodiment of the present disclosure is not limited to the above method operations, and the computer-executable instructions can also execute related operations in the orthodontic treatment monitoring method provided in any embodiment of the present disclosure.

[0167] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present disclosure can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented through hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer cloud platform (which can be a personal computer, server, or network cloud platform, etc.) to execute the orthodontic treatment monitoring method provided by each embodiment of the present disclosure.

[0168] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0169] The foregoing description is intended only to provide specific embodiments of the present disclosure, which will enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments shown herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein. Industrial Applicability

[0170] The disclosed embodiment provides an orthodontic treatment monitoring method, which can use an oral scanner to scan the user's teeth. Only multiple intraoral images are needed to obtain the target transformation parameters corresponding to each pair of tooth models in the second dental model and the first dental model (that is, the dental model corresponding to the expected orthodontic effect) corresponding to the multiple intraoral images. The doctor can know the orthodontic treatment effect based on the target transformation parameters. In this way, the user can monitor orthodontic treatment more conveniently, which has strong industrial practicality.

Claims

1. A method for monitoring orthodontic treatment, characterized in that: include: Acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model; Determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth; Determine a loss function corresponding to initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of the tooth model pairs after each pair of the tooth model pairs is similarly transformed according to the corresponding initial transformation parameters; If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions, the initial transformation parameters are determined as target transformation parameters.

2. The method according to claim 1, characterized in that Also includes: If the loss function corresponding to the initial camera parameters and the initial transformation parameters does not satisfy the preset condition, optimizing the initial camera parameters and the initial transformation parameters until the corresponding loss function satisfies the preset condition; The optimized initial transformation parameters are determined as the target transformation parameters.

3. The method according to claim 1 or 2, characterized in that: The determining of initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intra-oral images comprises: Performing three-dimensional reconstruction according to the multiple intraoral images to obtain a third dental jaw model; The first jaw model and the third jaw model are roughly registered to obtain the initial transformation parameters corresponding to each pair of the tooth models.

4. The method according to claim 1 or 2, characterized in that: The determining of the loss function corresponding to the initial camera parameters and the initial transformation parameters includes: For each pixel point in the multiple intraoral images, according to the initial camera parameters and the initial transformation parameters, detect whether the ray corresponding to the pixel point in the world coordinate system and the first dental model after similarity transformation according to the initial transformation parameters intersect, obtain a detection result, and determine the first color value and the first mask value of the pixel point according to the detection result; determining a color loss based on the first color value and the second color value of each pixel in the plurality of intra-oral images; Determine the mask loss according to the first mask value and the second mask value of each pixel point in the multiple intraoral images, wherein the second mask value is obtained by Determined by multiple mask images corresponding to the image; A loss function corresponding to the initial camera parameters and the initial transformation parameters is determined according to the color loss and the mask loss.

5. The method according to claim 4, characterized in that Determining the first color value of the pixel according to the detection result includes: If the detection result is an intersection, querying the initial color field network according to the intersection coordinates and the direction of the ray to obtain the first color value of the pixel point; If the detection result is non-intersection, determining the first color value of the pixel point to be 0; Wherein, determining the color loss according to the first color value and the second color value of each pixel point in the plurality of intraoral images comprises: For each pixel in the plurality of intra-oral images, determining a color difference between the first color value and the second color value of the pixel; The color loss is determined according to the color difference of each pixel in the multiple intra-oral images.

6. The method according to claim 4, characterized in that The determining a first mask value of the pixel point according to the detection result includes: If the detection result is an intersection, determining the first mask value of the pixel point to be a first value; If the detection result is non-intersection, determining the first mask value of the pixel point to be a second value; Wherein, determining the mask loss according to the first mask value and the second mask value of each pixel point in the plurality of intraoral images includes: For each of the pixel points in the plurality of intra-oral images, determining a mask difference between the first mask value and the second mask value of the pixel point; The mask loss is determined according to the mask difference of each pixel in the multiple intra-oral images.

7. The method according to claim 4, characterized in that The determining, according to the color loss and the mask loss, a loss function corresponding to the initial camera parameter and the initial transformation parameter comprises: The color loss and the mask loss are weighted and added to obtain a loss function corresponding to the initial camera parameters and the initial transformation parameters.

8. An orthodontic treatment monitoring device, characterized in that: include: A first acquisition module, used to acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model; A first determination module is used to determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the multiple intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth; A second determination module is used to determine the loss function corresponding to the initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are the camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of the tooth model pairs after each pair of the tooth model pairs is similarly transformed according to the corresponding initial transformation parameters; The third determination module is used to determine the initial transformation parameters as target transformation parameters if the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions.

9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; Wherein, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the orthodontic treatment monitoring method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor implements the orthodontic treatment monitoring method described in any one of claims 1 to 7.

Images