Orthodontic treatment monitoring method, device and equipment and storage medium

Abstract

The invention relates to an orthodontic treatment monitoring method, device and equipment and a storage medium. The orthodontic treatment monitoring method comprises the following steps: acquiring a plurality of intraoral images of a user and a first dental model; determining initial transformation parameters corresponding to each tooth model pair in the first tooth jaw model and the second tooth jaw model corresponding to the plurality of intraoral images; determining loss functions corresponding to the initial camera parameters and the initial transformation parameters; and if the loss function corresponding to the initial camera parameter and the initial transformation parameter satisfies a preset condition, determining the initial transformation parameter as a target transformation parameter. According to the embodiment of the invention, a user can carry out orthodontic treatment monitoring more conveniently.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an orthodontic treatment monitoring method, device, equipment and storage medium. Background Art

[0002] Orthodontics is the process of straightening teeth, correcting malocclusions and deformities, in order to achieve beautiful teeth.

[0003] At present, orthodontic treatment monitoring mainly relies on patients to go to the clinic for regular check-ups, and the doctor evaluates the orthodontic effect of the current treatment stage through naked eye observation, or the doctor uses a mouth scanner to scan the patient's teeth, and then compares the scanned teeth with the expected teeth to evaluate the orthodontic effect of the current treatment stage. However, some patients are unable to go to the clinic for regular check-ups due to busy work, time and space constraints, etc., resulting in the inability to adjust the dental braces in time, which in turn leads to longer dental treatment cycles, poor treatment effects, and poor patient treatment experience. Therefore, there is an urgent need for a more convenient orthodontic treatment monitoring method. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides an orthodontic treatment monitoring method, device, equipment and storage medium.

[0005] In a first aspect, the present disclosure provides an orthodontic treatment monitoring method, the method comprising:

[0006] Acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model;

[0007] Determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0008] Determine a loss function corresponding to initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of the tooth model pairs after each pair of the tooth model pairs is similarly transformed according to the corresponding initial transformation parameters;

[0009] If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions, the initial transformation parameters are determined as target transformation parameters.

[0010] In a second aspect, the present disclosure provides an orthodontic treatment monitoring device, the device comprising:

[0011] A first acquisition module, used to acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model;

[0012] A first determination module is used to determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the multiple intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0013] A second determination module is used to determine the loss function corresponding to the initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are the camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of the tooth model pairs after each pair of the tooth model pairs is similarly transformed according to the corresponding initial transformation parameters;

[0014] The third determination module is used to determine the initial transformation parameters as target transformation parameters if the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When one or more programs are executed by one or more processors, the one or more processors implement the orthodontic treatment monitoring method provided in the first aspect.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the orthodontic treatment monitoring method provided in the first aspect.

[0020] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0021] An orthodontic treatment monitoring method, apparatus, device and storage medium of an embodiment of the present disclosure can obtain multiple intraoral images of a user, and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model; determine initial transformation parameters corresponding to each pair of tooth models in the first dental jaw model and the second dental jaw model corresponding to the multiple intraoral images, wherein the second dental jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth; determine the loss function corresponding to the initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are the camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of tooth models after each pair of tooth models is similarly transformed according to the corresponding initial transformation parameters; if the loss function corresponding to the initial camera parameters and the initial transformation parameters meets the preset conditions, the initial transformation parameters are determined as the target transformation parameters. It can be seen that by adopting the above technical solution and using an oral scanner to scan the user's teeth, only multiple intraoral images are needed to obtain the target transformation parameters corresponding to each pair of tooth models in the second jaw model and the first jaw model (that is, the jaw model corresponding to the expected orthodontic effect) corresponding to the multiple intraoral images, and the doctor can know the orthodontic treatment effect based on the target transformation parameters. In this way, users can monitor orthodontic treatment more conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 A schematic diagram of a flow chart of an orthodontic treatment monitoring method provided by an embodiment of the present disclosure is shown;

[0025] Figure 2 A schematic diagram showing a flow chart of another orthodontic treatment monitoring method provided by an embodiment of the present disclosure;

[0026] Figure 3 A schematic structural diagram of an orthodontic treatment monitoring device provided by an embodiment of the present disclosure is shown;

[0027] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] In order to solve the above problems, the embodiments of the present disclosure provide an orthodontic treatment monitoring method, device, equipment and storage medium.

[0031] Combine the following Figures 1 to 3 The orthodontic treatment monitoring method provided by the embodiment of the present disclosure is described. In the embodiment of the present disclosure, the orthodontic treatment monitoring method can be executed by an electronic device. The electronic device can include a device with communication function such as a tablet computer, a desktop computer, a laptop computer, etc., and can also include a device simulated by a virtual machine or a simulator.

[0032] Figure 1 A schematic flow chart of an orthodontic treatment monitoring method provided by an embodiment of the present disclosure is shown.

[0033] like Figure 1 As shown, the orthodontic treatment monitoring method may include the following steps.

[0034] S110, acquiring a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model.

[0035] Specifically, the intraoral image is an image obtained by capturing the inside of the oral cavity using an imaging device. The image capturing device may include a mobile phone, a computer, a camera, etc., but is not limited thereto.

[0036] Exemplarily, acquiring multiple intraoral images of the user may include: receiving multiple intraoral images of the user sent by other electronic devices, or reading multiple intraoral images of the user from a storage device, etc., but is not limited thereto.

[0037] Specifically, the first dental model is a dental model corresponding to a desired dental orthodontic effect, and the first dental model includes a first tooth model corresponding to at least one tooth.

[0038] The first dental model may include a dental model corresponding to a first-stage orthodontic effect, a dental model corresponding to a second-stage orthodontic effect, or a dental model corresponding to a final orthodontic effect, but is not limited thereto.

[0039] The specific representation form of the first dental jaw model and the first tooth model may include a three-dimensional mesh model.

[0040] Exemplarily, obtaining the first dental model may include: performing mesh segmentation on the models corresponding to the teeth and gums, segmenting the tooth models corresponding to each tooth, and using DeepSDF technology to complete the segmented tooth models into a watertight mesh, thereby obtaining the first dental model, but is not limited thereto.

[0041] S120. Determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the multiple intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth.

[0042] Specifically, the second dental model is a dental model corresponding to (or behind) multiple intraoral images, and the second dental model includes at least one second tooth model. Those skilled in the art should understand that the first dental model is a dental model corresponding to the desired dental orthodontic effect, and the second dental model is a dental model corresponding to multiple intraoral images. Since the first dental model and the second dental model both correspond to the user's teeth, the first dental model can be transformed into a second dental model by optimizing the posture of each first tooth model in the first dental model, and then the difference between each first tooth model and the first tooth model corresponding thereto is known. Based on this, the embodiment of the present disclosure needs to determine the target transformation parameters for transforming the first dental model into the second dental model.

[0043] Specifically, for each pair of tooth models, the geometry of the first tooth model and the second tooth model included therein are consistent, and the difference between the two is a similarity transformation. In other words, the first tooth model and the second tooth model can be aligned by performing a similarity transformation on the first tooth model. Among them, the transformation parameters based on the similarity transformation include a scaling ratio (scale), a translation matrix (rotation), and a rotation matrix (translation). Since the scaling ratio (scale) between each pair of tooth models is the scaling ratio between the world coordinate systems where the first dental model and the second dental model are located, the scaling ratios of each pair of tooth models are the same. In this way, each pair of tooth models in the first dental model and the second dental model includes a common scaling ratio (scale), and their corresponding translation matrices (rotation), and rotation matrices (translation).

[0044] Specifically, the initial transformation parameters are initial transformation parameters corresponding to the tooth model pair.

[0045] In some embodiments, S120 may include: randomly generating initial transformation parameters corresponding to each pair of tooth models.

[0046] In other embodiments, S120 may include: performing three-dimensional reconstruction based on multiple intraoral images to obtain a third dental model; and performing coarse registration processing on the first dental model and the third dental model to obtain initial transformation parameters corresponding to each pair of tooth models.

[0047] Specifically, the third dental jaw model is a dental jaw model corresponding to the dental jaw of the user obtained by three-dimensional reconstruction based on a plurality of intraoral images.

[0048] The specific representation form of the third dental model may include a three-dimensional point cloud model, a three-dimensional mesh model, etc., which is not limited here.

[0049] Specifically, the initial camera parameters are rough camera parameters corresponding to the acquisition of multiple intraoral images. In the disclosed embodiment, in the process of optimizing the posture of each first tooth model in the first jaw model so that the first jaw model becomes the second jaw model, the real camera parameters (i.e., target camera parameters) corresponding to the acquisition of multiple intraoral images will also be slowly determined, for example, by optimizing the initial camera parameters to gradually approach the target camera parameters.

[0050] Among them, the camera parameters may include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters may include the camera focal length, the projection coordinates of the camera center on the pixel plane, etc. The camera extrinsic parameters may include the camera pose. The camera pose may include, for example, a camera rotation matrix and a camera translation matrix. The camera rotation matrix and the camera translation matrix together describe how to transform a point from the world coordinate system to the camera coordinate system.

[0051] For example, the Colmap algorithm is used to estimate the camera parameters (i.e., initial camera parameters) and sparse point cloud (i.e., the third dental model) when acquiring multiple intraoral images. If the image acquisition device is a pinhole camera, the camera parameters include three camera intrinsic parameters, i.e., the camera focal length f, and the projection coordinates of the camera center on the pixel plane (c u ,c v ), each intraoral image corresponds to 6 camera extrinsics (corresponding to 6 degrees of freedom).

[0052] Specifically, the first jaw model and the third jaw model are roughly registered to obtain transformation parameters for overall alignment of the first jaw model and the third jaw model, and the transformation parameters are used as initial transformation parameters corresponding to each tooth model pair.

[0053] Exemplarily, the first jaw model and the third jaw model may be coarsely registered using a feature-based coarse registration method, a coarse registration method based on a random sample consensus (RANSAC) framework, etc., but the present invention is not limited thereto.

[0054] It can be understood that since the second jaw model and the third jaw model both correspond to multiple intraoral images, when the first jaw model and the third jaw model are roughly aligned as a whole, the first jaw model and the second jaw model are also roughly aligned as a whole, that is, the first tooth model and the second tooth model in the tooth model pair are at least roughly aligned. Therefore, the initial transformation parameters obtained by the rough registration process are closer to the target transformation parameters, which is conducive to obtaining the target transformation parameters corresponding to the tooth model pair as soon as possible and improving the efficiency of orthodontic treatment monitoring. In addition, by performing a rough registration process on the first jaw model and the third jaw model as a whole to obtain the initial transformation parameters, the initial transformation parameters corresponding to each pair of tooth models can be quickly obtained.

[0055] In some other embodiments, S120 may include: performing three-dimensional reconstruction based on multiple intraoral images to obtain a third dental jaw model, wherein the third dental jaw model includes at least one third tooth model; performing coarse alignment processing on each pair of tooth models in the first dental jaw model and the third dental jaw model to obtain initial transformation parameters corresponding to each pair of tooth models in the first dental jaw model and the second dental jaw model.

[0056] S130. Determine a loss function corresponding to initial camera parameters and initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to a plurality of intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of tooth models after each pair of tooth models is similarly transformed according to the corresponding initial transformation parameters.

[0057] Specifically, the deviation between the first tooth model and the second tooth model in the tooth model pair may include deviations in color values, mask values, etc., but is not limited thereto.

[0058] S140: If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions, the initial transformation parameters are determined as target transformation parameters.

[0059] Specifically, the initial camera parameters may also be determined as the target camera parameters.

[0060] Specifically, the preset condition may include: the loss function is less than a preset threshold, wherein the specific value of the preset threshold may be set by those skilled in the art according to actual conditions, but is not limited thereto.

[0061] Specifically, if the loss function corresponding to the initial camera parameters and the initial transformation parameters meets the preset conditions, it indicates that after the first tooth model in each pair of tooth models is similarly transformed according to its corresponding initial transformation parameters, the alignment accuracy with the second tooth model is high (or has met the requirements), and the difference between the initial camera parameters and the actual camera parameters (i.e., the actual camera parameters corresponding to when multiple intraoral images are acquired) is small (or has met the requirements). At this time, the initial transformation parameters can be determined as the target transformation parameters.

[0062] Those skilled in the art should understand that, since the first tooth model in the first jaw model is aligned with the second jaw model after a similarity transformation according to its corresponding target transformation parameters, in other words, the jaw model obtained after a similarity transformation of the first tooth model in the first jaw model according to its corresponding target transformation parameters can be regarded as the second jaw model. Therefore, the tooth position difference between the first tooth model and the second tooth model can be determined according to the target transformation parameters to achieve orthodontic treatment monitoring.

[0063] It is understandable that the technical solution provided by the embodiment of the present disclosure eliminates the need for users to go to the clinic in person to have the doctor observe with the naked eye or use an oral scanner to scan their teeth. Instead, they only need to take multi-view images of their own teeth (i.e., multiple intraoral images) and then upload them. In this way, users do not need to frequently go to the clinic for follow-up visits, which saves users’ time and frees them from time and space constraints. In addition, since doctors do not need to judge with the naked eye or operate an oral scanner to frequently scan users’ teeth, doctors’ time can also be saved. Moreover, compared to doctors’ visual judgment, the technical solution provided by the embodiment of the present disclosure can quantitatively obtain the target transformation parameters corresponding to each pair of tooth models, avoiding the problem of misjudgment caused by doctors’ lack of experience and other reasons, and improving the accuracy of orthodontic treatment monitoring.

[0064] According to the disclosed embodiment, there is no need to use an oral scanner to scan the user's teeth. Only a plurality of intraoral images are needed to obtain the target transformation parameters corresponding to each pair of tooth models in the second jaw model and the first jaw model (i.e., the jaw model corresponding to the expected orthodontic effect) corresponding to the plurality of intraoral images. The doctor can obtain the orthodontic treatment effect based on the target transformation parameters, thereby allowing the user to monitor the orthodontic treatment more conveniently.

[0065] In another embodiment of the present disclosure, the method further comprises:

[0066] S150: If the loss function corresponding to the initial camera parameters and the initial transformation parameters does not meet the preset conditions, the initial camera parameters and the initial transformation parameters are optimized until the corresponding loss function meets the preset conditions.

[0067] S160: Determine the optimized initial transformation parameters as target transformation parameters.

[0068] Specifically, the optimized initial camera parameters may also be determined as target camera parameters.

[0069] Specifically, if the loss function corresponding to the initial camera parameters and the initial transformation parameters does not meet the preset conditions, it indicates that after the first tooth model in at least one pair of tooth models is similarly transformed according to its corresponding initial transformation parameters, the alignment accuracy with the second tooth model is low (or does not meet the requirements), and the gap between the initial camera parameters and the actual camera parameters is large (or does not meet the requirements). At this time, the initial camera parameters and the initial transformation parameters can be optimized, and then the loss function corresponding to the optimized initial camera parameters and the initial transformation parameters is determined. If the loss function corresponding to the optimized initial camera parameters and the initial transformation parameters meets the preset conditions, the optimized initial transformation parameters are determined as the target transformation parameters. If the loss function corresponding to the optimized initial camera parameters and the initial transformation parameters does not meet the preset conditions, the initial camera parameters and the initial transformation parameters are continuously optimized until the corresponding loss function meets the preset conditions, and finally the optimized initial transformation parameters are determined as the target transformation parameters. Of course, the final optimized initial camera parameters can also be determined as the target camera parameters.

[0070] It can be understood that by continuously optimizing the initial camera parameters and the initial transformation parameters, the alignment accuracy of the first tooth model and the second tooth model can be made higher and higher after the tooth model pair is similarly transformed according to the corresponding optimized initial transformation parameters, thereby obtaining the target transformation parameters corresponding to each pair of tooth models. In this way, the method for obtaining the target transformation parameters can be simple and convenient.

[0071] Figure 2 The flowchart of another orthodontic treatment monitoring method provided by the embodiment of the present disclosure is shown. The embodiment of the present disclosure is optimized on the basis of the above embodiment, and the embodiment of the present disclosure can be combined with various optional solutions in one or more of the above embodiments.

[0072] like Figure 2 As shown, the orthodontic treatment monitoring method may include the following steps.

[0073] S210, acquiring a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model.

[0074] Specifically, S210 is similar to S110 and will not be described in detail here.

[0075] S220. Determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the multiple intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth.

[0076] Specifically, S220 is similar to S120 and will not be described in detail here.

[0077] S230. For each pixel point in the multiple intraoral images, based on the initial camera parameters and the initial transformation parameters, detect whether the ray corresponding to the pixel point in the world coordinate system and the first dental model after similarity transformation according to the initial transformation parameters intersect, obtain the detection result, and determine the first color value and the first mask value of the pixel point based on the detection result.

[0078] Specifically, the first color value is the color value of the pixel estimated according to the initial camera parameters and the initial transformation parameters, wherein the color value may include a grayscale value or an RGB value, etc., but is not limited thereto.

[0079] Specifically, the first mask value is a mask value of a pixel estimated according to the initial camera parameters and the initial transformation parameters, wherein the mask value is used to characterize whether the pixel is a background pixel or a pixel corresponding to a tooth, for example, (0,0,0) is used to characterize a background pixel, and (1,1,1) is used to characterize a pixel corresponding to a tooth.

[0080] Specifically, the first dental jaw model after similarity transformation according to the initial transformation parameters refers to the first dental jaw model after similarity transformation of each first tooth model according to the corresponding initial transformation parameters.

[0081] Specifically, for each pixel point, the ray corresponding to the pixel point in the world coordinate system is determined according to the initial camera parameters corresponding to the intra-oral image where the pixel point is located, and then, it is detected whether the ray corresponding to the pixel point and the first dental model after similarity transformation according to the initial transformation parameters intersect to obtain the detection result, and then, the first color value of the pixel point is determined according to the detection result.

[0082] In some embodiments, the first color value of the pixel point is determined based on the detection result, including: if the detection result is intersection, querying the initial color field network according to the intersection coordinates and the direction of the ray to obtain the first color value of the pixel point; if the detection result is non-intersection, determining the first color value of the pixel point to be 0.

[0083] Specifically, the network parameters in the initial color field network are obtained by random settings.

[0084] For example, the initial color field network is denoted as f θ(x, ν), which is a function of the spatial three-dimensional coordinate x and the direction ν, outputs the RGB value rendered by NDS, and θ is the network parameter. If the intersection coordinate is X0 and the direction of the ray is V0, X0 and V0 are fed into f θ (x,ν), you can get the first color value.

[0085] In some embodiments, determining the first mask value of the pixel point according to the detection result includes: if the detection result is intersection, determining the first mask value of the pixel point to be a first value; if the detection result is non-intersection, determining the first mask value of the pixel point to be a second value.

[0086] The first value is used to indicate that the pixel is a pixel corresponding to a tooth, and the second value is used to indicate that the pixel is a background pixel. For example, the first value may be (1, 1, 1), and the second value may be (0, 0, 0).

[0087] S240, determining color loss according to the first color value and the second color value of each pixel in the plurality of intra-oral images.

[0088] Specifically, the second color value is the real color value of the pixel in the intraoral image to which it belongs.

[0089] In some embodiments, S240 includes: determining, for each pixel in the multiple intraoral images, a color difference between a first color value and a second color value of the pixel; and determining color loss based on the color difference of each pixel in the multiple intraoral images.

[0090] Exemplarily, determining the color loss based on the color difference of each pixel in the plurality of intraoral images may include: adding the squares (or absolute values, etc.) of the color difference of each pixel to obtain the color loss, but is not limited thereto.

[0091] S250. Determine the mask loss according to the first mask value and the second mask value of each pixel in the multiple intraoral images, wherein the second mask value is determined by the multiple mask images corresponding to the multiple intraoral images.

[0092] Specifically, for each intra-oral image, semantic segmentation processing is performed on the intra-oral image to distinguish the tooth part and the background part, and the mask value of the pixel point of the tooth part is set to the first value, and the mask value of the pixel point of the background part is set to the second value, so as to obtain the corresponding mask image.

[0093] Specifically, the second mask value is a real mask value of the pixel in the mask image corresponding to it.

[0094] In some embodiments, S250 includes: determining, for each pixel in the plurality of intraoral images, a mask difference between a first mask value and a second mask value of the pixel; and determining a mask loss based on the mask difference of each pixel in the plurality of intraoral images.

[0095] Exemplarily, determining the mask loss according to the mask difference of each pixel in the plurality of intraoral images may include: adding the squares (or absolute values, etc.) of the mask difference of each pixel to obtain the mask loss, but is not limited thereto.

[0096] S260: Determine a loss function corresponding to the initial camera parameters and the initial transformation parameters according to the color loss and the mask loss.

[0097] In some embodiments, S260 includes: performing weighted sum processing on color loss and mask loss to obtain a loss function corresponding to the initial camera parameters and the initial transformation parameters.

[0098] It can be understood that the weighted addition process can flexibly adjust the proportions of color loss and mask loss, respectively, so that the loss function focuses on the loss type that is more concerned, and then the target transformation parameters more accurately reflect the differences in the aspects that doctors are more concerned about in the tooth model.

[0099] Of course, you can also directly add the color loss and mask loss to get the loss function.

[0100] To illustrate the process of determining the loss function provided by the embodiment of the present disclosure in detail, a detailed description will be given below based on a specific example.

[0101] N semantically segmented intraoral images are denoted as Among them, I i is the i-th intraoral image after semantic segmentation, the pixel coordinates of a pixel in the intraoral image after semantic segmentation are marked (u, v), and the corresponding RGB value (i.e., the second color value) is recorded as C (u,v) For each intraoral image after semantic segmentation, the mask value of the pixel corresponding to the tooth part is recorded as 1, and the mask value of the pixel corresponding to the background part is recorded as 0 to generate a mask image. The mask image generated after semantic segmentation of N intraoral images is recorded as Among them, m i is the mask image generated after semantic segmentation of the i-th intraoral image. Since each intraoral image is taken by the same image acquisition device, the camera intrinsic parameters corresponding to the N intraoral images are the same, denoted as (f, c u ,c v ). Each intraoral image has an independent camera pose, and the camera extrinsic parameter of the i-th intraoral image is denoted by pi ∈R 6 (6 degrees of freedom), the overall camera parameters are recorded as Φ = (f, c u ,c v ,p1,p2,...p, N )∈R 6N+3 .

[0102] The first dental model includes K first tooth models, and the K first tooth models have a common scaling ratio s and independent rigid body transformation parameters (i.e., translation matrix and rotation matrix) o i ∈R 6 (6 degrees of freedom), the overall note is Ψ=(s,o1,o2,...,o K )∈R 6K+1 .

[0103] The color field network of the first dental model is denoted as f θ (x,ν), which is a function of the spatial three-dimensional coordinate x and the direction ν, outputs the RGB value rendered by NDS, and θ is the network parameter.

[0104] The process of calculating the loss function corresponding to the initial camera parameters and initial transformation parameters is as follows: for each pixel in the N intraoral images, the ray of the pixel in the world coordinate system can be calculated through the initial camera parameters corresponding to the intraoral image to which it belongs, so as to calculate the intersection of the ray and the first dental model after similarity transformation according to the initial transformation parameters. If there is no intersection, the color rendered by NDS (i.e., the first color value) is 0; if there is an intersection, the intersection coordinates and the ray direction are substituted into f θ (x, ν), and get the color rendered by NDS (i.e., the first color value). Since the intersection coordinates are related to the camera parameters and transformation parameters, and the color rendered by NDS is also related to the network parameters of the color field network, the color rendered by NDS (i.e., the first color value) is recorded as If they do not intersect, the first mask value of the pixel is (0,0,0); if they intersect, the first mask value of the pixel is (1,1,1). Since whether they intersect is related to both camera parameters and transformation parameters, the first mask value is recorded as

[0105] Compare the first color value and the second color value of each pixel in N intra-mouth images, and the color loss is:

[0106]

[0107] Among them, L color (θ, Φ, Ψ) is the color loss, is the first color value of the pixel with pixel coordinates (u, v), C (u,v)The second color value of the pixel with pixel coordinates (u, v).

[0108] Compare the first mask value and the second mask value of each pixel in N mouth images, and get the mask loss as:

[0109]

[0110] Among them, L mask (Φ,Ψ) is the mask loss, is the first mask value of the pixel with pixel coordinates (u, v), m (u,v) It is the second mask value of the pixel with pixel coordinates (u, v).

[0111] The color loss and mask loss are weighted and summed to obtain the loss function:

[0112] L(θ,Φ,Ψ)=αL color +λL mask

[0113] Among them, L(θ,Φ,Ψ) is the loss function, α is the color loss L color The weight value of λ is the mask loss L mask The weight value of .

[0114] S270: If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions, the initial transformation parameters are determined as target transformation parameters.

[0115] Specifically, if the loss functions corresponding to the initial camera parameters, the initial transformation parameters, and the initial color field network do not meet the preset conditions, the initial transformation parameters are determined as the target transformation parameters. At this time, the initial camera parameters are the target camera parameters, and the initial color field network is the color field network of the second dental model.

[0116] Specifically, if the loss functions corresponding to the initial camera parameters, initial transformation parameters, and initial color field network do not meet the preset conditions, the initial camera parameters, initial transformation parameters, and initial color field network are optimized until the corresponding loss functions meet the preset conditions. At this time, the optimized initial camera parameters are the target camera parameters, and the optimized initial color field network is the color field network of the second dental model.

[0117] Those skilled in the art should understand that "determining the loss function corresponding to the optimized initial camera parameters, the optimized initial transformation parameters, and the optimized color field network" is similar to S230-S260, except that the optimized initial camera parameters, the optimized initial transformation parameters, and the optimized color field network are involved in the loss function calculation, which will not be repeated here.

[0118] The disclosed embodiment utilizes NDS (Neural Deferred Shading) to jointly optimize camera parameters and transformation parameters, and can simply and quickly extract target transformation parameters from multiple intraoral images taken by the user and the first dental model, thereby obtaining tooth posture differences that accurately meet orthodontic requirements.

[0119] Figure 3 A schematic structural diagram of an orthodontic treatment monitoring device provided in an embodiment of the present disclosure is shown.

[0120] like Figure 3 As shown, the orthodontic treatment monitoring device 300 may include:

[0121] A first acquisition module 310 is used to acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model;

[0122] A first determination module 320 is used to determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intraoral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0123] A second determination module 330 is used to determine a loss function corresponding to initial camera parameters and initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to a plurality of intraoral images, and the loss function is used to characterize a total deviation between a first tooth model and a second tooth model in each pair of tooth models after each pair of tooth models is similarly transformed according to the corresponding initial transformation parameters;

[0124] The third determination module 340 is configured to determine the initial transformation parameters as target transformation parameters if the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet a preset condition.

[0125] In another embodiment of the present disclosure, the device further includes:

[0126] An optimization module, for optimizing the initial camera parameters and the initial transformation parameters until the corresponding loss functions meet the preset conditions if the loss functions corresponding to the initial camera parameters and the initial transformation parameters do not meet the preset conditions;

[0127] The fourth determining module is used to determine the optimized initial transformation parameters as target transformation parameters.

[0128] In yet another embodiment of the present disclosure, the first determining module 320 includes:

[0129] A reconstruction submodule is used to perform three-dimensional reconstruction based on multiple intraoral images to obtain a third dental jaw model;

[0130] The registration submodule is used to perform a rough registration process on the first jaw model and the third jaw model to obtain initial transformation parameters corresponding to each pair of tooth models.

[0131] In yet another embodiment of the present disclosure, the second determining module 330 includes:

[0132] A first determination submodule is used to detect, for each pixel point in the multiple intraoral images, whether the ray corresponding to the pixel point in the world coordinate system intersects with the first dental model after similarity transformation according to the initial transformation parameters according to the initial camera parameters and the initial transformation parameters, obtain a detection result, and determine a first color value and a first mask value of the pixel point according to the detection result;

[0133] a second determination submodule, for determining color loss according to a first color value and a second color value of each pixel in the plurality of intraoral images;

[0134] a third determination submodule, configured to determine the mask loss according to the first mask value and the second mask value of each pixel in the plurality of intraoral images, wherein the second mask value is determined by the plurality of mask images corresponding to the plurality of intraoral images;

[0135] The fourth determination submodule is used to determine the loss function corresponding to the initial camera parameters and the initial transformation parameters according to the color loss and the mask loss.

[0136] In yet another embodiment of the present disclosure, the first sub-determination module includes:

[0137] A query unit, configured to query the initial color field network to obtain a first color value of the pixel point according to the coordinates of the intersection point and the direction of the ray if the detection result is an intersection;

[0138] A first determining unit, configured to determine that the first color value of the pixel point is 0 if the detection result is that the pixels do not intersect;

[0139] Wherein, the second determination submodule includes:

[0140] a second determination unit, configured to determine, for each pixel in the plurality of intra-oral images, a color difference between a first color value and a second color value of the pixel;

[0141] The third determination unit is used to determine the color loss according to the color difference of each pixel point in the plurality of intra-oral images.

[0142] In yet another embodiment of the present disclosure, the first sub-determination module includes:

[0143] a fourth determining unit, configured to determine a first mask value of the pixel point as a first value if the detection result is an intersection;

[0144] a fifth determining unit, configured to determine the first mask value of the pixel point to be a second value if the detection result is disjoint;

[0145] Wherein, the third determination submodule includes:

[0146] a sixth determining unit, configured to determine, for each pixel in the plurality of intra-oral images, a mask difference between a first mask value and a second mask value of the pixel;

[0147] The seventh determination unit is used to determine the mask loss according to the mask difference of each pixel point in the plurality of intra-oral images.

[0148] In yet another embodiment of the present disclosure, the fourth determining submodule includes:

[0149] The eighth determination unit is used to perform weighted sum processing on the color loss and the mask loss to obtain a loss function corresponding to the initial camera parameters and the initial transformation parameters.

[0150] It should be noted that Figure 3 The orthodontic treatment monitoring device 300 shown can perform Figure 1 to Figure 2 The various steps in the method embodiment shown in the figure are implemented Figure 1 to Figure 2 The various processes and effects in the method embodiment shown are not described in detail here.

[0151] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0152] like Figure 4 As shown, the electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0153] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0154] The memory 402 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 402 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 402 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 402 is a non-volatile solid-state memory. In a particular embodiment, the memory 402 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0155] The processor 401 reads and executes the computer program instructions stored in the memory 402 to perform the steps of the orthodontic treatment monitoring method provided in the embodiment of the present disclosure.

[0156] In one example, the electronic device may further include a transceiver 403 and a bus 404. Figure 4 As shown, the processor 401, the memory 402 and the transceiver 403 are connected via a bus 404 and communicate with each other.

[0157] The bus 404 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 404 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0158] The following is an embodiment of a computer-readable storage medium provided in an embodiment of the present disclosure. The computer-readable storage medium and the orthodontic treatment monitoring method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the computer-readable storage medium, reference can be made to the above-mentioned embodiment of the orthodontic treatment monitoring method.

[0159] This embodiment provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to perform an orthodontic treatment monitoring method, the method comprising:

[0160] Acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model;

[0161] Determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth;

[0162] Determine a loss function corresponding to initial camera parameters and initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the plurality of intraoral images, and the loss function is used to characterize a total deviation between a first tooth model and a second tooth model in each pair of tooth models after each pair of tooth models is similarly transformed according to the corresponding initial transformation parameters;

[0163] If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet the preset conditions, the initial transformation parameters are determined as the target transformation parameters.

[0164] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present disclosure is not limited to the above method operations, and the computer executable instructions can also execute related operations in the orthodontic treatment monitoring method provided in any embodiment of the present disclosure.

[0165] Through the above description of the implementation methods, technicians in the relevant field can clearly understand that the present disclosure can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented through hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure can essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer cloud platform (which can be a personal computer, server, or network cloud platform, etc.) to execute the orthodontic treatment monitoring method provided by each embodiment of the present disclosure.

[0166] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. A method for monitoring orthodontic treatment, characterized in that: include: Acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model; Determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth; Determine a loss function corresponding to initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of the tooth model pairs after each pair of the tooth model pairs is similarly transformed according to the corresponding initial transformation parameters; If the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions, the initial transformation parameters are determined as target transformation parameters.

2. The method according to claim 1, characterized in that Also includes: If the loss function corresponding to the initial camera parameters and the initial transformation parameters does not satisfy the preset condition, optimizing the initial camera parameters and the initial transformation parameters until the corresponding loss function satisfies the preset condition; The optimized initial transformation parameters are determined as the target transformation parameters.

3. The method according to claim 1 or 2, characterized in that: The determining of initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the plurality of intra-oral images comprises: Performing three-dimensional reconstruction according to the multiple intraoral images to obtain a third dental jaw model; The first jaw model and the third jaw model are roughly registered to obtain the initial transformation parameters corresponding to each pair of the tooth models.

4. The method according to claim 1 or 2, characterized in that: The determining of the loss function corresponding to the initial camera parameters and the initial transformation parameters includes: For each pixel point in the multiple intraoral images, according to the initial camera parameters and the initial transformation parameters, detect whether the ray corresponding to the pixel point in the world coordinate system and the first dental model after similarity transformation according to the initial transformation parameters intersect, obtain a detection result, and determine the first color value and the first mask value of the pixel point according to the detection result; determining a color loss based on the first color value and the second color value of each pixel in the plurality of intra-oral images; Determining a mask loss according to the first mask value and the second mask value of each pixel in the plurality of intra-oral images, wherein the second mask value is determined by a plurality of mask images corresponding to the plurality of intra-oral images; A loss function corresponding to the initial camera parameters and the initial transformation parameters is determined according to the color loss and the mask loss.

5. The method according to claim 4, characterized in that Determining the first color value of the pixel point according to the detection result includes: If the detection result is an intersection, querying the initial color field network according to the intersection coordinates and the direction of the ray to obtain the first color value of the pixel point; If the detection result is non-intersection, determining the first color value of the pixel point to be 0; Wherein, determining the color loss according to the first color value and the second color value of each pixel point in the plurality of intraoral images comprises: For each pixel in the plurality of intra-oral images, determining a color difference between the first color value and the second color value of the pixel; The color loss is determined according to the color difference of each pixel in the multiple intra-oral images.

6. The method according to claim 4, characterized in that The determining a first mask value of the pixel point according to the detection result includes: If the detection result is an intersection, determining the first mask value of the pixel point to be a first value; If the detection result is non-intersection, determining the first mask value of the pixel point to be a second value; Wherein, determining the mask loss according to the first mask value and the second mask value of each pixel point in the plurality of intraoral images includes: For each of the pixel points in the plurality of intra-oral images, determining a mask difference between the first mask value and the second mask value of the pixel point; The mask loss is determined according to the mask difference of each pixel in the multiple intra-oral images.

7. The method according to claim 4, characterized in that The determining, according to the color loss and the mask loss, a loss function corresponding to the initial camera parameter and the initial transformation parameter comprises: The color loss and the mask loss are weighted and added to obtain a loss function corresponding to the initial camera parameters and the initial transformation parameters.

8. An orthodontic treatment monitoring device, characterized in that: include: A first acquisition module, used to acquire a plurality of intraoral images of the user and a first dental jaw model, wherein the first dental jaw model includes at least one first tooth model; A first determination module is used to determine initial transformation parameters corresponding to each pair of tooth models in the first jaw model and the second jaw model corresponding to the multiple intra-oral images, wherein the second jaw model includes at least one second tooth model, and the tooth model pair includes a first tooth model and a second tooth model corresponding to the same tooth; A second determination module is used to determine the loss function corresponding to the initial camera parameters and the initial transformation parameters, wherein the initial camera parameters are the camera parameters corresponding to the multiple intraoral images, and the loss function is used to characterize the total deviation between the first tooth model and the second tooth model in each pair of the tooth model pairs after each pair of the tooth model pairs is similarly transformed according to the corresponding initial transformation parameters; The third determination module is used to determine the initial transformation parameters as target transformation parameters if the loss functions corresponding to the initial camera parameters and the initial transformation parameters meet preset conditions.

9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; Wherein, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the orthodontic treatment monitoring method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor implements the orthodontic treatment monitoring method described in any one of claims 1 to 7.

Images