Photo-based assessment of dental treatments and procedures

Abstract

The present invention relates to a method and system for monitoring the progress of a dental patient during a treatment process. A three-dimensional model of the desired position of the patient's teeth can be time-planned based on a three-dimensional model of the patient's teeth prepared before the start of treatment. A digital camera is used to take one or more two-dimensional photographs of the patient's teeth, which are input into a monitoring system. The monitoring system determines virtual camera parameters for each two-dimensional input image relative to the time-planned three-dimensional model, generates a two-dimensional image from the three-dimensional model using the determined virtual camera parameters, and then compares each input photograph with the corresponding generated two-dimensional image in order to determine how closely the three-dimensional arrangement of the patient's teeth corresponds to the time-planned three-dimensional arrangement.

Description

[0001] This application is a divisional application of the invention patent application with Chinese application number 201680048710.9 (corresponding to PCT international application number PCT / IB2016 / 001170), application date August 18, 2016, and invention name “Photo-based assessment of dental treatment and surgery”. Technical Field

[0002] The present invention relates to methods and systems for monitoring patient progress during dental treatment and surgery, and in particular to a photo-based monitoring method, and to a system for performing the method, which uses two-dimensional photographs taken during an examination of the patient to determine how well the patient's teeth correspond to a three-dimensional representation of the desired position of the patient's teeth according to a treatment plan. Background Art

[0003] Prior to the development of oral imaging and modeling systems, dentists used mechanical indentation methods to generate three-dimensional models of teeth and underlying tissues in order to facilitate the manufacture of various types of dentures, including crowns and bridges. Mechanical indentation techniques generally involve biting into a viscous thixotropic material by the patient, which retains an accurate impression of the patient's teeth and underlying tissues when the material is removed from the patient's teeth. This material can be used as a mold for casting a positive three-dimensional model of the patient's teeth and underlying gingival tissue, or as a mold for casting a denture device. Although mechanical indentation techniques have been used by dentists for decades, they are associated with various defects, including a considerable possibility that the indentation may be adversely altered or destroyed during the removal of the hardened viscous thixotropic material from the patient's teeth and during the transportation of the indentation to the laboratory where the positive three-dimensional model is cast and the denture is manufactured. In addition, for many patients, this procedure is time-consuming and uncomfortable.

[0004] Recently, semi-automatic oral imaging and modeling systems have been developed that electronically generate digital 3D models of the teeth and underlying tissues based on images of the patient's mouth taken by an optomechanical endoscope or scanning pen. The endoscope or scanning pen is introduced into the patient's mouth by a technician in order to acquire a sufficient number of 2D photographs from which a 3D digital model of the patient's teeth and underlying tissues is generated by a computer. This oral imaging and modeling system has been shown to be faster, more accurate, more robust, and more cost-effective than mechanical indentation techniques.

[0005] Thus, in many cases, dental professionals can prepare accurate three-dimensional models of a patient's teeth and use the three-dimensional models to analyze the patient's dental condition and then develop treatment plans for various types of defects and conditions. Furthermore, the three-dimensional models can be electronically replicated to prepare planned three-dimensional configurations of the patient's teeth for various points during the treatment planning process. In order to monitor the progress of dental patients during the treatment process, dental equipment suppliers, dentists, and ultimately dental patients are seeking cost-effective and time-effective methods and systems for using three-dimensional information. Summary of the invention

[0006] The present invention relates to a method and system for monitoring the progress of a dental patient during a treatment process. At any specific time point during the treatment process, a three-dimensional model of the patient's teeth in the desired position at that time point can be planned in time based on a three-dimensional model of the patient's teeth prepared before the start of treatment. During the treatment process, a digital camera is used to take one or more two-dimensional photos of the patient's teeth, which are input into a monitoring system. These input two-dimensional photos represent the actual position of the patient's teeth. The monitoring system determines virtual camera parameters for each two-dimensional input image relative to the planned three-dimensional model, and uses the determined virtual camera parameters to generate a two-dimensional image based on the three-dimensional model. The generated two-dimensional image represents the desired or desired position of the patient's teeth. The monitoring system then compares each input photo with the corresponding generated two-dimensional image in order to determine how closely the three-dimensional arrangement of the patient's teeth corresponds to the three-dimensional arrangement planned in time. When the level of correspondence is below a threshold level, an indication that the treatment is not proceeding as planned is returned to the dentist so that the dentist can take corrective action. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Illustration of a 3D model of a dental patient's teeth.

[0008] Figure 2A-2B The relationship between the virtual camera position and the 3D model of the patient's teeth is illustrated.

[0009] Figures 3A-3D An example is given of a way to map points in the world coordinate system to corresponding points on the virtual camera image plane.

[0010] Figure 4 Illustrated are a series of three-dimensional models of a dental patient's teeth that may be electronically prepared to reflect a desired arrangement of the dental patient's teeth during a proposed treatment planning process.

[0011] Figures 5A-5D The present invention is partly directed to a method of monitoring therapy, to which the present invention relates, which is illustrated schematically.

[0012] Figure 6A control flow chart example is provided to illustrate the operation of the therapy monitoring system and therapy monitoring method according to the present invention.

[0013] Figure 7A-7B The control flow diagram is used to illustrate Figure 6 Step 604.

[0014] Figure 8 Provides Figure 6 The control flow chart of step 610 is a method for determining virtual camera parameters.

[0015] Fig. 9 Provides Figure 8 8 is a control flow chart of the initial virtual camera parameter adjustment method called in step 805.

[0016] Figures 10A-10C Provides information about Fig. 9 Additional details of steps 904, 908 and 912 in FIG.

[0017] Fig.11 Show Figure 8 The second virtual camera parameter adjustment method called in step 806.

[0018] Fig.12 Provides Fig.11 Control flow diagram of the optimization step in .

[0019] Figures 13A-13C An example illustrates gradient-based loss calculation.

[0020] Fig.14 Provides Figure 8 A control flow chart of the first virtual camera parameter fine-tuning method called in step 807 .

[0021] Fig.15 An example of a tooth group is illustrated, Fig.14 The inner loop of the illustrated method iterates over these groups of teeth.

[0022] Fig.16 Using the same illustration conventions used in FIG. 13 , the Laplacian vector calculation for an image element and the generation of a Laplacian vector field are illustrated.

[0023] Figures 17A-17B The calculation of contrast values ​​or correlation coefficients is illustrated.

[0024] Fig.18 A general architectural diagram is provided for various types of computers, including computers used to implement a dental treatment monitoring system.

[0025] Fig.19An example is given of a distributed computer system based on the Internet.

[0026] Fig. 20 An illustration of cloud computing.

[0027] Fig.21 The generalized hardware and software components of a general-purpose computer system are illustrated, for example, a general-purpose computer system having an architecture similar to Figure 1 A general purpose computer system is shown.

[0028] Figures 22A-22B Give examples to illustrate two types of virtual machines and virtual machine operating environments. DETAILED DESCRIPTION

[0029] Figure 1 It is a three-dimensional model of the teeth of a dental patient. The three-dimensional model 102 includes a three-dimensional model of teeth associated with the upper jaw 104 of the dental patient and a three-dimensional model of teeth associated with the lower jaw 106 of the dental patient. The three-dimensional model is generally prepared based on an optical scan of the dental patient's oral cavity using complex imaging and reproduction hardware and software. The three-dimensional model can be electronically presented in a variety of different ways, similar to the various different ways used to present three-dimensional objects in various types of CAD / CAM and various imaging and solid modeling systems. The model can be electronically presented in a variety of different orientations and forms. For example, the upper and lower jaws 104 and 106 can be rotated around the rotation axis 108 so that the teeth associated with the two jaws are closed together and show a form similar to the form of a closed-mouthed patient. Each jaw and associated teeth can be presented as a separate three-dimensional model as an alternative.

[0030] Figure 2A-2B The relationship between the virtual camera position and the 3D model of the patient's teeth is shown. Figure 2A As shown, the three-dimensional model of the patient's teeth 202 can be located in a three-dimensional world coordinate system 204 with three mutually orthogonal axes X, Y, and Z in a translational and rotational manner. By simulating image capture using a virtual camera 208, a two-dimensional view of the three-dimensional model can be obtained from any position outside the three-dimensional model in the world coordinate system. The virtual camera 208 is associated with its own three-dimensional coordinate system 210 with mutually orthogonal axes x, y, and z. The world coordinate system and the camera coordinate system are of course mathematically related by the translation of the origin of the camera x, y, and z coordinate system relative to the origin of the world coordinate system 201 and by three rotation angles. When the three rotation angles are applied to the camera, the camera x, y, and z coordinate system is rotated relative to the world x, y, and z coordinate system. The origin of the camera x, y, and z coordinate system has coordinates (0, 0, 0) in the camera coordinate system and (X, Y, and Z) in the world coordinate system. c , Y c , Z c). The two-dimensional image captured by the virtual camera 216 can be considered to be located on the x, z plane of the camera coordinate system, with the center at the origin of the camera coordinate system, as shown in FIG. 2 .

[0031] Figure 2B The operations involved in orienting and positioning the camera x, y, z coordinate system to coincide with the world X, Y, Z coordinate system are shown. Figure 2B , the centers of the camera coordinate system 216 and the world coordinate system 204 are at two different origins 214, 212, respectively, and the orientation of the camera coordinate system is different from that of the world coordinate system. In order to orient and position the camera x, y, z coordinate system to coincide with the world X, Y, Z coordinate system, three operations are performed. The first operation 220 involves translating the camera coordinate system, the displacement of the translation represented by the vector t, so that the origins 214, 212 of the two coordinate systems coincide. The position of the camera coordinate system relative to the world coordinate system is shown by the dashed lines relative to the world coordinate system after the translation operation 220, including the dashed line 218. The second operation 222 involves rotating the camera coordinate system by an angle α (224) so ​​that the z axis of the camera coordinate system (referred to as the z' axis after the translation operation) coincides with the Z axis of the world coordinate system. In a third operation, the camera coordinate system is rotated about the Z / z' axis by an angle θ (228) so that all of the camera coordinate system axes coincide with their corresponding world coordinate system axes.

[0032] Figures 3A-3D A method of mapping points in a world coordinate system to corresponding points on a virtual camera image plane is shown. This process allows the virtual camera to be located anywhere in space with respect to a three-dimensional computer model of a patient's teeth and used to generate a two-dimensional image that corresponds to the two-dimensional image that would be taken by a real camera that has the same position and orientation relative to the three-dimensional representation of the patient's equivalent physical model. Figure 3A The image plane of a virtual camera, the aligned camera coordinate system and the world coordinate system, and a point in three-dimensional space, which is imaged on the image plane of the virtual camera, are shown. Figure 3A In, and later Figure 3B-3D In the case of the camera coordinate system including the x, y, and z axes, the camera coordinate system coincides with the world coordinate system X, Y, and Z. Figure 3A 302, y and Y axes 304, and z and Z axes 306. The imaged point 308 is shown having coordinates (X p , Y p , Z p The coordinates of the image 310 of this point on the virtual camera image plane are (x i ,y i). The center of the virtual lens of the virtual camera is at point 312, whose camera coordinates are (0, 0, l) and world coordinates are (0, 0, l). When point 308 is in focus, the distance l between origin 314 and point 312 is the focal length of the virtual camera. Figure 3A In , the z-axis is used as the symmetry axis of the virtual camera in addition to the y-axis, as Figure 2A A small rectangle is shown on the image plane, and its two corner vertices along a diagonal line are aligned with the origin 314 and the coordinates (x i ,y i ) coincides with point 310. The rectangle includes a length of x i The horizontal side and length are y i The horizontal side includes the horizontal side 316, and the vertical side includes the vertical side 318. A corresponding rectangle has a length of -X p , including a horizontal side of 320, and a length of -Y p , including the vertical side of the vertical side 322. The world coordinate is (X p , Y p , Z p ) and the world coordinates are (0, 0, Z p ) is located at the vertex of a diagonal of the corresponding rectangle. Note that the positions of the two rectangles are reversed on both sides of point 312. The length of the line segment 328 between point 312 and point 324 is Z p -1. The angles at which each straight line passing through point 312 intersects the z / Z axis are equal on both sides of point 312. For example, angle 330 and angle 332 are exactly the same. As a result, the world coordinates in three-dimensional space can be derived by using the correspondence principle between similar side lengths of similar triangles as (X p , Y p , Z p ) is the image plane coordinate (x i ,y i ) expression 334:

[0033]

[0034]

[0035]

[0036]

[0037] Of course, generally speaking, as mentioned above Figure 2A As described, the virtual camera coordinate system is not aligned with the world coordinate system. Therefore, some more complex analysis is required to develop a function or process that maps points in three-dimensional space to points on the virtual camera image plane. Figure 3B-3D The process of calculating the image of a point in three-dimensional space on a virtual camera image plane with arbitrary direction and position is shown. Figure 3B An arbitrarily positioned and oriented virtual camera is shown. Virtual camera 336 is mounted on bracket 337, which allows the virtual camera to be tilted by an angle α 338 relative to the vertical Z axis and rotated by an angle θ 339 about the vertical axis. Bracket 337 can be located anywhere in three-dimensional space, and its position is indicated by a position vector w pointing from the world coordinate system origin 341 to bracket 337. 0 340. The second vector r 342 represents the relative position of the center of the image plane 343 in the virtual camera 336 relative to the bracket 337. The direction and position of the origin of the camera coordinate system coincide with the center of the image plane 343 in the virtual camera 336. The image plane 343 is located in the x, y plane of the camera coordinate system axes 344-346. Figure 3B The camera shown in FIG. 3 images point w 347, and the image of point w appears as image point c 348 on the image plane 343 in the virtual camera. The vector w defining the position of the camera support 337 0 exist Figure 3B is displayed as a vector:

[0038]

[0039] Figure 3C-3D The process shown is used to map the coordinates of a point in three-dimensional space, such as a point corresponding to a vector w in a world coordinate system, to the image plane of a virtual camera of arbitrary orientation and position. Figure 3C The transformation between world coordinates and homogeneous coordinates h and the inverse transformation h are shown in expressions 350 and 351. -1 The forward transformation from world coordinates 352 to homogeneous coordinates h involves multiplying each coordinate component by an arbitrary constant k and adding a fourth coordinate component k. The vector w corresponding to the three-dimensional space point 347 imaged by the virtual camera is expressed as a cylindrical vector, such as Figure 3C The corresponding cylindrical coordinate vector W in homogeneous coordinate form is shown in expression 355 h . The matrix P is the perspective transformation matrix. Figure 3C The perspective transformation matrix is ​​used to perform the above reference Figure 3A The world-to-camera coordinate transformation discussed ( Figure 3A 334 in FIG. 334). The homogeneous coordinate form c of the vector c corresponding to the image 348 of point 347 is h , multiply w by the perspective transformation matrix h Calculate, such as Figure 3C As shown in Expression 357. Therefore, c in the homogeneous camera coordinate form h Expression and world coordinate form of ch The inverse homogeneous coordinate transformation 360 is used to transform c in world coordinate form h The expression is converted to a vector expression 361 in the world coordinate form of vector c 362. Comparing the camera coordinate expression 363 of vector c with the world coordinate expression 361 of the same vector reveals that the camera coordinates are obtained by referring to Figure 3A The transformation discussed ( Figure 3A 334) is related to the world coordinates. The inverse transformation of the perspective transformation matrix P -1 Shown in Figure 3C In expression 364 in . The inverse perspective transformation matrix can be used to correspond to the world coordinate point in the three-dimensional space corresponding to the image point expressed in camera coordinates, such as Figure 3C 366. Note that, in general, the Z coordinate of the three-dimensional point imaged by the virtual camera is not restored by the perspective transformation. This is because all points in front of the virtual camera along a straight line from the image point to the imaged point are mapped to the image point. In order to determine the Z coordinate of the three-dimensional point imaged by the virtual camera, other information is required, such as depth information obtained from a set of stereo images or depth information obtained from a separate depth sensor.

[0040] Figure 3D Three additional matrices are shown in , which represent the position and orientation of the virtual camera in the world coordinate system. w0 370 represents a camera stand ( Figure 3B 337) from its position in three-dimensional space to the origin of the world coordinate system ( Figure 3B 341 in ). The matrix R represents the rotation angles α and θ required to align the camera coordinate system with the world coordinate system 372. The transformation matrix C 374 represents the image plane of the virtual camera from the camera support ( Figure 3B 337) is transformed into the virtual camera by vector r( Figure 3B The image plane position represented by 342 in the figure. The vector w representing the three-dimensional space point h Converted to vector c representing the position of the image point on the virtual camera image plane h The entire expression, as expression 376 in Figure 3D The vector wh is first multiplied by the transformation matrix 37 to produce a first intermediate result, the first intermediate result is multiplied by the matrix R to produce a second intermediate result, the second intermediate result is multiplied by the matrix C to produce a third intermediate result, the third intermediate result is multiplied by the perspective matrix P to produce a vector c hExpression 378 shows the inverse transformation. In general there is a forward transformation from world coordinate points to image points 380, and when sufficient information is available, there is also an inverse transformation 381. It is this forward transformation 380 that is used to generate a two-dimensional image from a three-dimensional model or object corresponding to an arbitrary orientation and position of the virtual camera. Each point on the surface of the three-dimensional object or model is converted by the forward transformation 380 to a point on the virtual camera image plane.

[0041] Figure 4 A series of three-dimensional models of a dental patient's teeth are shown which may be electronically prepared to reflect the desired arrangement of the dental patient's teeth during a proposed treatment plan. Figure 4 The horizontal arrow below is the timeline of the treatment plan. At the initial starting point 404, Figure 4 The time t is specified in 0 , the arrangement of the patient's teeth is recorded in the initial three-dimensional model 406 of the patient's teeth. A time-planned three-dimensional model, such as three-dimensional model 408, can be prepared electronically as a prediction of a future time point t during the treatment or surgical procedure. 82 An estimate of how the arrangements are expected to be made. Figure 4 As shown, the final time-planned three-dimensional model 410 in the continuous series of three-dimensional models represents the time t 224 The desired treatment or surgical goal. Figure 4 Only four three-dimensional models are shown on the treatment or surgery timeline 402, but time-planned three-dimensional models for any point along the timeline can be electronically prepared using extrapolation and simulation-based methods.

[0042] Figures 5A-5D The present invention is partly concerned with a method for monitoring treatment, as illustrated in the figures. Figure 5A As shown, during the dental treatment or surgery, at a specific current time point t 82 Department Figure 5AIn one embodiment, the dentist examines the patient and takes n two-dimensional pictures of the patient's teeth 504. Alternatively, in some embodiments, the two-dimensional pictures may be taken by a friend or relative of the patient or the patient using a camera timer or a smartphone feature that facilitates the acquisition of user images. In the current example, n is equal to 3. In general, each picture or a subset of the pictures represents a certain standard view or image type. The dentist or other personnel is provided with instructions for taking images of a particular standard view or type. As shown in 5B, once the practitioner submits the two-dimensional images along with patient information, an identification of the time when the two-dimensional images were taken, or other such information, a treatment monitoring system to which the present invention relates in part determines camera parameters for virtual cameras 506-508, the orientation and position of which most likely correspond to the camera parameters of the dentist's camera at the time when each of the corresponding n two-dimensional pictures 510-512 was taken by the dentist or other personnel.

[0043] Next, if Figure 5C As shown, the determined camera parameters for the virtual camera are used to generate corresponding two-dimensional images 516-518 corresponding to the n two-dimensional images 510-512 taken by the dentist or other personnel. Figure 5D As shown, to generate the correlation coefficient, a comparison operation, such as comparison operation 520, is performed on each pair of the dentist-submitted image and the corresponding image generated from the three-dimensional model. In one implementation, the correlation coefficient is expressed as a floating point value between 0 and 1 ( Figure 5D In the presently described methods and systems, the correlation coefficients of the individual images are used to generate a single overall correlation value 524, which is also a floating point value between 0 and 1 ( Figure 5D 526 in). When the total correlation value calculated based on the submitted two-dimensional images and the planned three-dimensional model is greater than a threshold value, as determined in step 528, the treatment monitoring system stores in memory and returns an indication 530 that the treatment is proceeding as planned. Otherwise, the treatment monitoring system stores in memory and returns an indication 532 that the treatment is not proceeding as planned. It should be noted that in some implementations, multiple images for a given standard view can be used to generate several correlation coefficients and a total correlation coefficient. In other implementations, including the implementations discussed below, the best representative image for each standard view is selected for processing.

[0044] Figure 6 Provided are control flow charts illustrating the operation of the treatment monitoring system and treatment monitoring method according to the present invention. Other control flow charts discussed below are Figure 6Various steps in the top-level control flow diagram shown provide more details. In step 602, a treatment monitoring method and / or system ("treatment monitor") receives: (1) n two-dimensional photographs of a first jaw and m two-dimensional photographs of a second jaw of a dental patient being examined during a treatment or surgical procedure; (2) metadata in the Interchangeable Image File Format ("EXIF") for each of the n+m two-dimensional images; (3) text annotations for the two-dimensional images, some of which include identification of the standard type of view represented by each image, parameters of the digitally encoded image including image size, date and time information, camera settings including one or more of camera model, brand, camera orientation, aperture, shutter speed, focal length, metering mode, International Organization for Standardization (ISO) speed information; (4) patient ID and other information; and (5) the time and date t when the patient generated the two-dimensional photographs. In step 604, the treatment monitor checks and verifies the input data. When any error is detected, as determined in step 606, an error is returned in step 608 to allow various types of fine-tuning procedures to be taken, including requesting additional or alternative input information. Otherwise, in step 610, the treatment monitor determines that the virtual camera parameters for each input image have passed the initial check and verification step 604. In step 612, the corresponding two-dimensional input image is generated from the three-dimensional model according to the time plan using the determined set of virtual camera parameters for the two-dimensional image corresponding to each input image. In step 614, each input image is compared with the corresponding plan-based image to determine a correlation coefficient, and the total correlation coefficient is determined using this set of correlation coefficients. When the determined total correlation coefficient is greater than a certain threshold, as determined in step 616, an indication that the treatment or procedure is on track is returned in step 618, ON_TRACK. Otherwise, an indication that the treatment or procedure is off track is returned in step 620.

[0045] Figure 7A-7B The control flow diagram shows Figure 6Step 604. In the nested for loops of steps 702-713, each image associated with each of the first and second jaws of the dental patient is considered. In step 704, the inspection and verification method matches the image currently being considered in the nested for loop with the metadata extracted and collected for the image from the input data. In step 705, in order to adjust the image for further analysis, the inspection and verification method filters the image, removes inconsistent content and defects, and in some cases, can expand or reduce the contrast in the image, or change the color balance. In step 706, the inspection and verification method performs a qualitative analysis of the image and assigns a quality metric to the image. The quality metric reflects how well the image corresponds to the image type indicated by the metadata, as well as the sharpness, clarity and completeness of the image. When the image currently under consideration is equivalent to the image already processed in the nested for loop, as determined in step 707, and when the quality metric assigned to the image currently under consideration is better than that of the processed image, as determined in step 708, in step 709, the processed image is replaced with the image currently under consideration in the set of processed images prepared by the check and verification method. Otherwise, when the quality metric assigned to the image currently under consideration is not as good as that of the processed image, as determined in step 708, in step 710, the image currently under consideration is discarded. When the image currently under consideration is not equivalent to the processed image, as determined in step 707, in step 711, the image currently under consideration is placed in the set of processed images together with the calculated quality metric and various metadata. In step 714, the check and verification method verifies the patient ID contained in the input information. Go to Figure 7B , when the patient ID is verified as determined in step 715, and when the processed image set prepared in the nested for loops of steps 702-713 is deemed sufficient for further analysis as determined in step 717, the check and verify method returns in step 719 without reporting an error status. In this case, the processed image set contains the single best image of each type, and the image set contains sufficient information to proceed with the analysis process for the treatment. Otherwise, an error status is reported in steps 716 and 718. Of course, the check and verify method may perform other types of input data verifications and checks in various implementations, and may return other types of error statuses when these other checks and verifications fail.

[0046] Figure 8 Provides Figure 6 The control flow chart of step 610 is a method for determining virtual camera parameters. In the for loop of steps 802-812, consider the above reference Figure 7A-7BEach image in the set of processed images produced by the inspection and verification method discussed. In step 803, the method uses metadata associated with the image, including the image type, to initialize a set of virtual camera parameters associated with the image and select an iteration number N. Different types of images and images with different characteristics and qualities may require different numbers of adjustments and fine-tuning. Next, in the while loop of steps 801-810, the while loop iteration is used to perform N iterations to adjust and fine-tune the initial virtual camera parameters of the image. The adjustment and fine-tuning process is non-convex, and as a result, the process does not necessarily converge. In step 805, a first virtual camera parameter adjustment method is called to make a rough adjustment to the initial virtual camera parameters. In step 806, a second virtual camera parameter adjustment method is called to more finely adjust the virtual camera parameters of the image. In step 807, the first virtual camera parameter fine-tuning method is called to fine-tune the virtual camera parameters associated with the image, and then in step 808, the second virtual camera parameter fine-tuning method is called to fine-tune the virtual camera parameters associated with the image. In step 809, the iteration variable N is decremented. When N is greater than 0, as determined at step 810, the while loop of steps 804-810 then continues to iterate. Otherwise, at step 811, the final fine-tuning method is called to adjust the virtual camera parameters for the image. Note that, through the various steps and iterations, the virtual camera parameters associated with a particular image are generally continuously adjusted and fine-tuned toward a set of virtual camera parameters that best estimates the position, orientation, and focal length of the dentist's camera relative to the patient's teeth, which position, orientation, and focal length were used to initially capture the input image.

[0047] Fig. 9 Provides Figure 8 Control flow diagram of the initial virtual camera parameter adjustment method called in step 805. In step 902, the method uses the metadata associated with the image to select the value of the iteration variable N, similar to Figure 8In step 803, a value of the iteration variable N is selected, and then the record set is initialized to empty. In step 904, the initial virtual camera parameter adjustment method applies a threshold transformation to each pixel color / emphasis value in the image to generate a tooth mask TM for the image. When the pixel is encoded according to the Lab color model, the a color component of the pixel color / emphasis is thresholded. The Lab color model is a color object (color-opponent) space based on nonlinearly compressed color space coordinates such as the International Commission on Illumination ("CIE") XYZ color space coordinates, with a dimension L for brightness and color object dimensions a and b. When other color models are used to encode pixel color and intensity, other components or values ​​derived from one or more components of the pixel color / intensity value are thresholded. The elements in the tooth mask TM corresponding to the image pixels are associated with one of two binary values ​​to indicate whether the pixel or element should be in the tooth area or non-tooth area in the image. In step 906, a similar tooth mask TM′ is generated from the time-planned three-dimensional model using the current virtual camera parameters of the two-dimensional input image based on which the tooth mask TM was generated. At step 908, a distance transform mask TM of the tooth mask TM is generated. dt At step 910, the method searches the tooth mask TM′ to the distance transformation mask TM, TM dt The minimum loss coverage mask on the tooth mask TM′ is searched for relative distance transformation mask TM dt In step 912, in order to generate the TM′ to TM dt The new tooth mask TM′ covering the mask with the minimum loss and the new two-dimensional image from the three-dimensional model are used to calculate the adjustment of the virtual camera parameters of the two-dimensional image currently considered. In step 914, the new tooth mask TM′ is calculated relative to TM dt The adjusted virtual camera parameters and the calculated losses obtained in steps 912 and 914 are stored as the next record in the set variable resultSet in step 916. In step 917, the iteration variable M is decremented. When M is still greater than 0 as determined in step 918, control returns to step 906 for another iteration of virtual camera parameter adjustment. Otherwise, in step 920, the minimum loss record in resultSet is selected, and the virtual camera parameters associated with the two-dimensional image are set to the virtual camera parameters in the selected record.

[0048] Figures 10A-10C Provides information about Fig. 9 Details of steps 904, 908 and 912 in FIG. Fig. 10A Shows Fig. 9The threshold conversion step 904 in FIG. 1 is a small imaginary image portion 1002 as the original image in FIG. Fig. 10A . Each pixel in the original image is shown to have an intensity / color component value. In the threshold conversion part, those pixels whose intensity / color component value is less than a certain threshold value are assigned a value of 1, while all other pixels are assigned a value of 0. The threshold value is 5 in the current hypothetical case. In one implementation, the threshold value is determined using the well-known Otsu threshold conversion method. In this method, image pixels are separated into two classes based on the assumption that the intensity value distribution of the pixels can be modeled as a bimodal distribution. The Otsu method seeks a threshold value that minimizes the difference between classes, which is a weighted sum of the intensity value differences of the two classes. Pixels assigned a value of 1 are considered to be tooth region pixels, while other pixels are considered to be non-tooth region pixels. Thus, in the image 1004 after threshold conversion, a small internal area 1006 corresponds to the teeth. In addition, there are two small tooth-shaped areas 1008 and 1010 adjacent to the edge of the image. In the next step, any tooth area adjacent to the edge is backfilled with a value of 0, because according to the procedures and protocols for dentists to take photos, teeth should not appear at the border of the photo. Fig. 10A A final step not shown in FIG. 1 reconsiders the areas of backfill 0 to ensure that tooth-like areas are not accidentally backfilled. The result of this next step is a tooth mask 1012 having inner areas 1014 corresponding to teeth with pixel values ​​of 1 and outer areas 1016 corresponding to non-tooth areas with values ​​of 0.

[0049] Fig. 10B Shown is the calculation of a distance transform mask of one mask when calculating the loss of a second mask overlaid on the distance transform mask of the one mask. Fig. 10B 1020 is shown in FIG. 1030 . The exterior 1022 of the mask is shown with grid lines and has one of two binary values, while the interior 1024 of the mask is shown without grid lines and each element or pixel in the interior has the other of the two binary values. A second mask 1026 is shown below the first mask. Note that the dimensions of the interior region 1028 of the second mask are shaped differently than the interior region 1024 of the first mask 1024. The distance transform mask transforms the first mask 1020 into a first mask distance transform mask 1030. In this transformation, the values ​​of the elements or pixels in the first mask are replaced by the distance, in units of elements or pixels, that needs to be crossed from the element to the boundary between the interior and exterior of the first mask. For example, starting from pixel 1032, in order to reach the boundary 1034 of the inner and outer mask regions, no other pixels or elements need to be crossed, so the value of pixel or element 1032 in the distance transform mask is 0. There are many different types of distance measures that can be used, including Euclidean distance, city block distance, and so on. Fig. 10BAt the right-hand corner of , the second mask 1026 is rotated relative to the distance transformation mask 1030 of the first mask and is covered on the upper part of the distance transformation mask of the first mask, thereby generating a covering mask 1040. Fig. 9 In step 910, TM′ is searched for relative to TM dt When the minimum loss or optimal coverage mask is found, a state space search is performed in which various possible rotations and translations of the distance transform mask of the second mask relative to the first mask are considered for various different scales or sizes of the second mask. The loss for a specific coverage mask, such as the coverage mask 1040, is calculated as the sum of the values ​​of the elements of the distance transform mask that are located below the interior region 1028 of the second mask, and the loss ... Fig. 10B Indicated by 1042 below the covering mask 1040.

[0050] Fig. 10C Shown are some adjustments to the virtual camera parameters in order to generate a certain projection image from which a new TM′ corresponding to the minimum loss coverage mask of TM′ on the distance transformation mask can be prepared. These adjustments include adjusting the position or center of the virtual camera 1060, adjusting the roll angle 1062 of the virtual camera, adjusting the yaw angle 1064 of the virtual camera, and adjusting the focal length of the virtual camera or the distance from the 3D model surface 1066. These adjustments change one or more of the shape, size and orientation of the internal tooth area in the previous TM′, thereby obtaining the minimum loss coverage mask of the new TM′. First, the camera 1060 is rotated so that the centroids of the two masks TM and TM′ coincide. To achieve this goal, two vectors are constructed in a coordinate system whose origin coincides with the camera position. The two vectors include a first vector describing the position of the centroid of TM and a second vector describing the position of the centroid of TM′. The camera is rotated about an axis of rotation that coincides with the vector obtained as the cross product of the first and second vectors. For the front view or side view only, the linear relationship is determined by fitting a straight line through the two tooth masks and rotating around the z-axis ( Figure 3B 346) rotate the camera so that these lines are parallel, and adjust the camera roll angle 1062. To modify the area 1066 of the mask TM', move the camera closer to or farther from the three-dimensional model. The coefficient K controls this movement. When K is greater than 1, the camera moves away from the three-dimensional model. Otherwise, the camera moves closer to the three-dimensional model. The coefficient K is obtained using the following empirical formula:

[0051]

[0052] Where |TM′| and |TM| are the areas of the two masks TM and TM′ in pixels, respectively. The adjustment 1064 of the yaw angle is performed for the front view and the side view, but not for the occluded view. The yaw angle is calculated in radians using the parameters of the parabola fitted through the masks and using the empirical formula for calculating the yaw angle:

[0053] y=a 1 x 2 +b 1 x+c 1 , where x, y∈TM,

[0054] y=a 2 x 2 +b 2 x+c 2 , where x, y∈TM′,

[0055]

[0056] Fig.11 Shown in Figure 8 The second virtual camera parameter adjustment method is called in step 806 of . In step 1102, metadata associated with the image is used to determine the value of the iteration variable N. Next, in the for loop of steps 1104-1107, each of the two jaws of the dental patient and the teeth associated therewith are considered. In step 1105, the jaw and the associated teeth that are not currently under consideration are removed from the image and from the current consideration. Then, in step 1106, the virtual camera parameters associated with the image are optimized for the portion of the image corresponding to the jaw and its associated teeth that are currently under consideration.

[0057] Fig.12 Provides Fig.11 1204-1212, the optimization method is used to obtain an optimized set of virtual camera parameters for the image currently being considered. The while loop iterates through the number of iterations indicated by the iteration variable N. The iteration variable is in Fig.11In step 1205, a two-dimensional image is generated from the three-dimensional model using the virtual camera parameters q. In step 1206, a gradient-based loss is calculated for this generated two-dimensional image relative to the input image currently under consideration. When the new gradient-based loss is greater than the value stored in the variable cost, the virtual camera parameters p are returned in step 1208 because these parameters correspond to at least the local minimum found in the previous while loop. Otherwise, in step 1209, the value of the variable cost is set to the new loss calculated in step 1206, and the virtual camera parameters p are set to q. In step 1210, the virtual camera parameters q are perturbed in a direction that minimizes the loss within certain predetermined limits. In step 1211, the iteration variable N is decremented. When the iteration variable N is greater than or equal to 0 as determined in step 1212, control returns to step 1205 for another iteration of the while loop. Otherwise, the current virtual camera parameters p are returned in step 1213. Next, for the multiple steps described below, use Fig.12 The optimization method shown in .

[0058] In one implementation, the Nelder-Mead downhill simplex optimization method is used, with seven dimensions, including three rotations, three translations, and the virtual camera view. In this method, a simplex with n+1 n-dimensional vectors for the problem in the form of n variables is used, where the test points correspond to the vectors. The test points are replaced in a way that preserves the simplex volume but moves and deforms the simplex toward a local optimum. There are many variations of the Nelder-Mead downhill simplex optimization method, and there are many other optimization methods that can be used to optimize the virtual camera parameters.

[0059] Figures 13A-13C 13. A small number of pixels 1302 in an image are shown in the upper portion of FIG. 13. These pixels are cells in a grid, for example, cell 1302 has grid coordinates (x, y). The gradient at the location of pixel (x, y) in image f is vector 1306, which can be estimated based on discrete pixel intensity calculation method 1308. Thus, the gradient vector can be calculated for each pixel or element of the image Gradient vector 1310 for pixel f(x,y) 1304 is shown as an example of a gradient associated with a pixel element. The image can be converted into a vector field by calculating the gradient at each pixel in the image using a different method used to calculate certain boundary pixel gradient vectors. The two images can be compared by calculating a contrast value based at least in part on a comparison of the two image gradient vector fields.

[0060] Fig. 13BThe generation of a histogram for the intensity values ​​of two images to be compared is shown in FIG. The probability that a randomly selected pixel from an image has a particular value can be calculated from the pixel intensity histogram. Fig. 13B , two images 1320 and 1322 are shown. Image 1320 is referred to as image I 1 , image 1322 is referred to as image I 2 Typically, pixels in an image are displayed as rectangular or square cells that are associated with coordinates with respect to the x-axis 1324-1325 and the y-axis 1326-1327. Each pixel is associated with an intensity value. Depending on the color model used to encode the color and intensity of the pixel, the intensity value may have different components, or may be associated with a single intensity derived from multiple color / intensity components associated with a pixel. In one implementation, each pixel is associated with an average intensity that represents the average of the three color channel intensities. Image I 1 The intensity of the pixel with coordinates (x, y) is represented by I 1 (x, y). For example, the intensity of cell 1328 is I 1 (4, 12). Histograms 1330-1332 are obtained based on a single image I 1 and I 2 The pixel intensity values ​​of the image are prepared for the combination of the two images. For example, the histogram 1330 is based on the image I 1 The horizontal axis 1335 of the histogram is incremented according to increasing intensity, and the vertical axis 1336 represents the number of pixels in the image having a particular intensity value. Of course, the histogram can be represented in a computable manner by an array of pixel numbers indicated by intensity values. The histogram 1331 is similarly represented according to the image I 2 1322. The joint histogram 1332 is a histogram representing the number of matching or aligned pixels in the two images I1 and I2, which have a pair of intensity values ​​of a particular order. The horizontal axis 1338 of the joint image histogram 1332 increases in units of intensity value pairs, and the vertical axis represents the number of equivalent, aligned pixel pairs having a particular order of intensity pairs. One pixel in each pair of pixels is selected from the first of the two images, and the other pixel in each pair of pixels is selected from the second of the two images, and both pixels in each pair have the same (x, y) coordinates. These histograms can be thought of as discrete probability distributions. For example, from image I 1 The probability that a randomly selected pixel has an intensity value of 10 is 1340, which can be calculated according to the image I 1 The number of pixels with a medium intensity value of 10, 1342, divided by image I 1 The number of pixels in the image is 1344. In a similar manner, the two images I 1 and I 2The probability 1346 that two aligned and matching pixels in the image have a particular ordering of pairs of intensity values ​​can be calculated as the ratio of the number of pixels in the two images having pairs of intensity values ​​of that ordering divided by the total number of pixels in each of the two images 1348.

[0061] like Fig. 13C As shown, for image I 1 and I 2 The intensity probabilities computed for randomly selected pixels in the image, and the joint intensity profiles for matching pixel pairs selected from the two images, can be represented by another simpler notation 1350. Using this notation, expression 1352 shows the joint intensity profiles for the matching pixel pairs selected from the two images. 1 ,I 2 and image I 1 and I 2 The combination of Shannon entropy H 1 , H 2 and H 1,2 Next, as shown in expression 1354, the entropy of the two images I is calculated based on the entropy calculated in expression 1352. 1 and I 2 The mutual information MI(I 1 , I 2 ). MI in terms of entropy, joint entropy and conditional entropy (I 1 , I 2 ) can be used to calculate MI(I 1 , I 2 Then, the interaction information of the two images is multiplied by the calculated value G(I 1 , I 2 ), the opposite number of the two images I 1 and I 2 The loss function of the comparison value is calculated, as shown in expression 1356. As shown in expression 1358, the calculated value G(I 1 , I 2 ) is calculated as the sum of the function f() calculated over all pixels of the images. The function f() takes as arguments the gradient calculated for each pixel (x, y) of the two images and the value in the tooth mask calculated for the first of the two images. As shown in expression 1360, when the pixel to which the function f() is applied is a member of the tooth region in the mask, the function has the value Otherwise 0. By reference Fig.12 In the discussed optimization method, the loss function value is minimized.

[0062] Fig.14 Provided for Figure 8807 in step 1406. In step 1402, metadata associated with the currently considered input image is used to determine the value of the iteration variable N, and the collection variable resultSet is initialized to empty. In step 1404, the current virtual camera parameters of the image are stored. In the outer for loop of step 1406, each of the patient's two jaws and their associated teeth are considered separately in two iterations. In the inner for loop of steps 1407-1411, each tooth group of the currently considered jaw is considered. In step 1408, the rest of the image except for the currently considered tooth group is masked out from the image. Next, in step 1409, as described above with reference to Fig.12 An optimization procedure equivalent to the optimization procedure discussed is used to optimize the virtual camera parameters of the input image currently under consideration. In step 1410, the optimized virtual camera parameters are stored as the next record in the set resultSet generated by the inner and outer for loops of steps 1406-1413. Finally, in step 1411, the current virtual camera parameters are reset to the stored virtual camera parameters stored in step 1404. After terminating the two nested loops in steps 1406-1413, in step 1414, the fine-tuned virtual camera parameters are calculated based on the records stored in the set resultSet by the two nested loops. This calculation may include finding irrelevant results from the result set and then averaging or using a weighted average for the other results of the result set. Next, in step 1415, the virtual camera parameters of the input image currently under consideration are set to the fine-tuned virtual camera parameters determined in step 1414.

[0063] Fig.15 An example of a tooth group is shown, Fig.14 The inner loop of the method shown iterates over these groups of teeth. Fig.15 In the upper part of the figure, a chart 1502 is provided which identifies the universal numerical codes of human teeth. Below the chart is shown an example tooth group, for example, an example tooth group 1504 consisting of four teeth 7, 8, 9, 10. In order to derive the above reference Fig.14 Multiple iterations of the inner loop of the discussed nested for loop may employ various different grouping methods of grouping the teeth into tooth groups.

[0064] The current discussion has ended Figure 8 The second virtual camera parameter fine-tuning method called in step 808 uses the same method as the one in the reference above on all completed teeth. Fig.12 The methods discussed are similar to the optimization methods. Figure 8The final fine-tuning method called in step 811 is similar to the method called in step 808, except that the calculation of the loss is based on considering the second-order derivatives or Laplace vectors of the image at the image elements instead of the first-order derivatives or gradient vectors. Fig.16 Shown is an explanation of the Laplacian vector calculation for an image element and the generation of a Laplacian vector field using the same illustration convention as used in FIG. 13 .

[0065] As mentioned above Figure 5D and Fig. 9 As discussed, once all input images are associated with the final fine-tuned virtual camera parameters, a comparison is performed between each input image and the corresponding image generated from the three-dimensional model in order to generate a comparison value or correlation coefficient. Figures 17A-17B The calculation of the contrast value or correlation coefficient is explained. As shown in Figure 17, using the above reference Figure 8 The virtual camera parameters determined by the method discussed above are for the corresponding input image I p 1708, generating a two-dimensional image 1702I from the three-dimensional model 1704 s Next, a contour line 1706 is generated that surrounds the teeth in the image. This contour line is then overlaid or superimposed on the original input image 1708 to produce an overlaid image 1710. Fig.17A As shown in the middle illustration 1712, for each pixel (x, y) located on the contour line 1714, a small square, rectangle or other compact area 1716 can be constructed, ε x,y The contrast values ​​generated from the overlay image 1710 involve calculating the p and the image I generated from the 3D model s The loss of pixels on the contours in the two images. The loss of contour overlapping pixels (x, y) is calculated as Fig. 17B 1720. When the absolute value or magnitude of the gradient of a pixel (x, y) in the two images is greater than a certain threshold T, the loss of the pixel is the dot product of the gradient of the pixel in the two images divided by the product of the two gradient magnitudes. The value of threshold T in one implementation is 3. Otherwise, the loss is 0. The fit or similarity measure can be calculated as the loss of pixels located on the contour divided by the length of the contour in pixels, as shown in expression 1722. An alternative fit or similarity measure is shown in expression 1724. In this alternative fit value, the loss of the two images is calculated as the sum of the pixel losses along the contour divided by the area ε of each pixel (x, y) in the two images. x,yThe sum of the maximum losses of any pixel in . This alternative metric has greater reliability when the input image is a bit blurred by applying a Gaussian filter. The fitness value or a linear combination of two fitness values ​​can be used as a similarity measure or correlation coefficient calculated for comparing two images.

[0066] It should be noted that, in general, color images are processed by monitoring methods. When gradients and Laplacian vectors are calculated, they are calculated based on the overall illumination or intensity calculated from three different color values ​​in whatever color system is used to encode the image.

[0067] Fig.18 A general architectural diagram for various types of computers is provided, including computers used to implement a dental treatment monitoring system. The computer system contains: one or more central processing units ("CPUs") 1802-1805; one or more electronic memories 1808, which are interconnected to the CPU via a CPU / memory-subsystem bus 1810 or multiple buses; a first bridge 1812, which interconnects the CPU / memory-subsystem bus 1810 with other buses 1814, 1816, or other types of high-speed interconnect media including multiple high-speed serial interconnects. These buses or serial interconnects in turn connect the CPU and memory to a special processor such as a graphics processor 1818, and connect the CPU and memory to one or more other bridges 1820. The other bridges 1820 are interconnected with high-speed serial links or with multiple controllers 1822-1827. Controllers 1822-1827, such as controller 1827, provide access to various different types of mass storage devices 1828, electronic displays, input devices, and the like components, subcomponents, and computing resources. It should be noted that computer-readable data storage devices include optical disks, electromagnetic disks, electronic memories, and other physical data storage devices. Those familiar with modern science and technology understand that electromagnetic radiation and propagating signals cannot store data for later retrieval, but can "store" only one byte or less of information per mile in a transient form, far less than the information required to encode even the simplest routines.

[0068] Of course, there are many different types of computer system architectures that differ from one another in the number of different memories, including different types of cache hierarchies, the number of processors and the connections between the processors and other system components, the number of internal communication buses and serial links, and many other aspects. However, computer systems generally execute stored programs by retrieving instructions from memory and then executing those instructions in one or more processors.

[0069] Fig.19An example of a distributed computer system connected to the Internet is illustrated.With advances in communications and networking technology in terms of performance and accessibility, and with the steady and rapid growth in computing bandwidth, data storage capacity, and other performance and capabilities of all types of computer systems, much of modern computing now typically involves large distributed systems and computers interconnected by local area networks, wide area networks, wireless communications, and the Internet. Fig.19 A typical distributed system is shown in which a large number of PCs 1902-1905, a high-end distributed mainframe system 1910 with a large data storage system 1912, and a large computer center 1914 with a large number of rack-mounted servers or blade servers are interconnected through various communication and network systems, which together include the Internet 1916. This distributed computing system provides a wide array of functions. For example, a PC user sitting at home can access hundreds of millions of different websites provided by hundreds of thousands of different network servers around the world, and can access high computing bandwidth computing services from remote computer devices to run complex computing tasks.

[0070] Until recently, computing services were typically provided by computer systems and data centers that were purchased, configured, managed, and maintained by a service provider organization. For example, an e-commerce retailer typically purchases, configures, manages, and maintains a data center that includes numerous web servers, back-end computer systems, and data storage systems used to serve web pages to remote customers, receive orders through a web interface, process orders, track completed orders, and a variety of other tasks associated with an e-commerce business.

[0071] Fig. 20 shows cloud computing. In the recently developed cloud computing paradigm, computing cycles and data storage facilities are provided to organizations and individuals by cloud computing providers. In addition, large organizations may choose to establish private cloud computing facilities rather than subscribing to computing services provided by public cloud computing service providers. Fig. 20 In the example, the organization's system administrator uses PC 2002 to access the organization's private cloud 2004 through local network 2006 and private cloud interface 2008, and also accesses public cloud 2012 through cloud service interface 2014 through Internet 2010. The administrator can configure virtual computer systems or even entire virtual data centers in the case of private cloud 2004 or public cloud 2012, and start applications on the virtual computer systems and virtual data centers to perform any of many different types of computing tasks. As an example, a small organization can configure and run a virtual data center within a public cloud, which executes a network server to provide an e-commerce interface through the public cloud to the organization's remote customers (e.g., users who are viewing the organization's e-commerce webpage using remote user system 2016).

[0072] Cloud computing facilities are designed to provide computing bandwidth and data storage services, just as utility companies provide electricity and water to consumers. Cloud computing offers great advantages to small organizations that do not have the resources to purchase, manage, and maintain in-house data centers. Instead of purchasing enough computer systems in a physical data center to handle peak computing bandwidth and data storage needs, these organizations can dynamically add and remove virtual computer systems from virtual data centers in the public cloud in order to track computing bandwidth and data storage needs. Moreover, small organizations can completely avoid the daily expenses incurred in maintaining and managing physical computer systems, including hiring and regularly engaging information technology experts and constantly paying for operating system and database management system upgrades. Moreover, cloud computing interfaces allow for simple and straightforward construction of virtual computing facilities, flexibility in the types of applications and operating systems that can be constructed, and other features that are useful even to owners and managers of private cloud computing facilities used by a single organization.

[0073] Fig.21 Explains the hardware and software components of a general purpose computer system, such as a Figure 121. A computer system 2100 is generally considered to include three basic layers: (1) a hardware layer or bottom layer 2102; (2) an operating system layer or layer 2104; and (3) an application layer or layer 2106. The hardware layer 2102 includes one or more processors 2108, system memory 2110, various types of input-output ("I / O") devices 2110 and 2112, and mass storage devices 2114. Of course, the hardware layer may also include many other components, including power supplies, internal communication links and buses, application specific integrated circuits, a variety of different types of processor-controlled or microprocessor-controlled peripheral devices and controllers, and many other components. The operating system 2104 is coupled to the hardware layer 2102 via a low-level operating system and hardware interface 2116, which includes a set of non-privileged computer instructions 2118, a set of privileged computer instructions 2120, a set of non-privileged registers and memory addresses 2122, and a set of privileged registers and memory addresses 2124. In general, the operating system exposes non-privileged instructions, non-privileged registers, and non-privileged memory addresses 2126 and a system call interface 2128 as operating system interfaces to application programs 2132-2136, and application programs 2132-2136 run in an operating environment provided by the operating system to the application programs. Only the operating system can access privileged instructions, privileged registers, and privileged memory addresses. By retaining access to privileged instructions, privileged registers, and privileged memory addresses, the operating system can ensure that applications and other higher-level computing entities cannot interfere with each other's operations and cannot change the overall state of the computer system in a way that may adversely affect system operations. The operating system includes many internal components and modules, including a scheduler 2142, a memory manager 2144, a file system 2146, a device driver 2148, and a variety of other components and modules. To some extent, modern operating systems provide multiple abstraction layers on the hardware layer, including virtual memory, which provides each application and other computing entity with a separate, large, linear memory address space, which is mapped by the operating system to various electronic memories and mass storage devices. The scheduler coordinates the intertwined operation of various different applications and higher-level computing entities, providing each application with a virtual, independent system dedicated to the operation of that application. From the application's point of view, the application runs continuously without concern for the need to share processor resources and other system resources with other applications and higher-level computing entities. The device driver abstracts the details of the operation of the hardware components, allowing the application to use the system call interface to transmit data to and receive data from the communication network, mass storage devices and other I / O devices and subsystems.File system 2136 facilitates the abstraction of mass storage devices and memory resources as a high-level, easily accessible file system interface. Thus, the development and evolution of operating systems has led to the generation of various virtual operating environment types for application programs and other higher-level computing entities.

[0074] Although the operating environment provided by the operating system has proven to be a hugely successful abstraction layer within computer systems, the abstraction layer provided by the operating system is associated with difficulties and challenges faced by developers and users of applications and other higher-level computing entities. One difficulty arises from the fact that there are many different operating systems running on various different types of computers. In many cases, popular applications and computing systems are developed to run only on a subset of the available operating systems and therefore can only be executed on the subset of the various different types of computer systems on which the operating system was designed to run. Often, even when an application or other computing system is ported to another operating system, the application or other computing system can run more efficiently on the operating system for which the application or other computing system was originally targeted. Another difficulty arises from the increasingly distributed nature of computer systems. Although distributed operating systems are the subject of a large amount of research and development efforts, many popular operating systems are primarily designed to run on a single computer system. In many cases, it is difficult to move applications between different computer systems in a distributed computer system in real time for the purposes of high availability, fault tolerance, and load balancing. The problem is even greater in heterogeneous distributed computer systems that include different types of hardware and devices running different types of operating systems. Operating systems are continually evolving, and as a result, certain older applications and other computing entities may not be compatible with newer versions of the operating system for which they were designed.

[0075] For all of these reasons, and to address many of the difficulties and challenges associated with traditional computing systems, including the compatibility issues discussed above, higher abstraction layers known as "virtual machines" were developed and evolved to further abstract computer hardware. Figures 22A-22B Two types of virtual machines and virtual machine execution environments are explained. Figures 22A-22B use Fig.21 The same example conventions are used in . Fig.22A A first type of virtualization is shown. Fig.22A The computer system 2200 includes Fig.21 The same hardware layer 2202 as shown in the hardware layer 2102. However, instead of Fig.21 As in the example above, the operating system layer is directly placed on the hardware layer. Fig.22A The virtual computing environment illustrated in the example is characterized by a virtualization layer 2204, which is equivalent to Fig.21The virtualization layer / hardware layer interface 2206 of the interface 2116 in the virtualization layer is combined with the hardware. The virtualization layer provides a hardware-like interface 2208 to multiple virtual machines such as virtual machine 2210 running on top of the virtualization layer in the virtual machine layer 2212. Each virtual machine includes one or more application programs or other higher-level computing entities packaged together with an operating system called a "guest operating system", such as application 2214 and guest operating system 2216 are packaged together in virtual machine 2210. Each virtual machine is then equivalent to Fig.21 2206. The client operating system in each virtual machine is combined with the virtualization layer 2208, rather than the actual hardware interface 2206. The virtualization layer separates the hardware resources into several abstract virtual hardware layers, and the client operating system in each virtual machine is combined with the abstract virtual hardware layer. The client operating system in the virtual machine is generally unaware of the virtualization layer, and it is as if these client operating systems directly access the real hardware interface during operation. The virtualization layer ensures that each virtual machine currently running in the virtual environment receives a fair configuration of the basic hardware resources, and ensures that all virtual machines receive sufficient resources required for continued operation. The virtualization layer interface 2208 can be different from the client operating system. For example, the virtualization layer is generally capable of providing a virtual hardware interface for various different types of computer hardware. As an example, this allows virtual machines including client operating systems designed for a specific computer architecture to run on hardware of different architectures. The number of virtual machines does not have to be equal to the number of physical processors, or even a multiple of the number of processors.

[0076] The virtualization layer includes a virtual machine monitor module 2218 ("VMM"), which virtualizes the physical processors in the hardware layer and generates virtual processors on which each virtual machine runs. For operational efficiency, the virtualization layer attempts to allow virtual machines to directly run non-privileged instructions and directly access non-privileged registers and memories. However, when the client operating system in the virtual machine accesses virtual privileged instructions, virtual privileged registers, and virtual privileged memories through the virtualization layer interface 208, the access results in the execution of virtualization layer code for simulating or emulating privileged resources. The virtualization layer also includes a kernel module 2220, which manages memory, communication, and data storage resources ("VM kernel") on behalf of the running virtual machine. For example, the VM kernel maintains a shadow page table on each virtual machine so that the virtual memory facilities of the hardware layer can be used to process memory accesses. The VM kernel also includes routines for implementing virtual communication, and data storage devices and device drivers that directly control the operation of basic hardware communication and data storage devices. Similarly, the VM kernel virtualizes various other types of I / O devices including keyboards, optical drives, and other such devices. The virtualization layer mainly arranges the operation of virtual machines, much like the operating system arranges the operation of applications, so that each virtual machine runs in a complete and fully functional virtual hardware layer.

[0077] Fig. 22B The second type of virtualization is illustrated in Fig. 22B In the embodiment, the computer system 2240 includes Fig.21 The same hardware layer 2242 and software layer 2244 are shown in FIG. Several application programs 2246 and 2248 are shown running in the operating environment provided by the operating system. In addition, a virtualization layer 2250 is also provided in the computer 2240, but it is different from the reference layer 2250. Fig.22A The virtualization layer 2204 discussed above, the virtualization layer 2250 is a layer formed on the operating system 2244 referred to as the "main OS", and uses the operating system interface to access the functions and hardware provided by the operating system. The virtualization layer 2250 mainly includes the VMM and Fig.22A The hardware-like interface 2208 in the virtualization layer / hardware layer interface 2252 is equivalent to Fig.21 The interface 2116 in provides an operating environment for multiple virtual machines 2256-2258, each of which includes one or more application programs or other higher-level computing entities packaged with a client operating system.

[0078] The dental disease monitoring system can be implemented with several single PCs or servers, can be implemented in a distributed computing system, or can be implemented using cloud computing facilities. Similarly, the physician can communicate with the dental disease monitoring system using a PC, server, or a variety of other processing control devices including tablets, notebooks, and smart phones.

[0079] Although the present invention has been described with respect to specific embodiments, it is not intended that the present invention be limited to these embodiments. Multiple improvements within the spirit of the present invention will be clear to those skilled in the art. For example, any one of a variety of different designs and implementation parameters including operating systems, hardware platforms, programming languages, module organizations, control structures, data structures and other such parameters can be changed to produce various replacement implementations. As another example, the two-dimensional images obtained from the patient during treatment can be obtained by using a variety of different imaging devices, including two-dimensional digital cameras, three-dimensional digital cameras, film-based cameras of local digital conversion devices, and even non-optical imaging devices.

[0080] It is to be understood that the above description of the disclosed embodiments is provided to enable those skilled in the art to make or utilize the contents disclosed in the present application. Various improvements to these embodiments will be easily clear to those skilled in the art, and the general principles defined in the present application can be applied to other embodiments without departing from the spirit or scope of the present application's disclosure. Therefore, the present application's disclosure is not intended to be limited to the embodiments shown in the present application, but should be given the widest scope consistent with the principles disclosed in the present application and novel features.

Claims

1. A method performed in a system having: one or more processors; one or more electronic memories storing instructions and data; one or more mass storage devices storing encoded images and patient information; and a communication subsystem, wherein the system receives images and information from a remote computer system via the communication subsystem, the method comprising: receiving in the system via the communication subsystem one or more two-dimensional photographs of a patient's teeth; determining virtual camera parameters for each of the one or more two-dimensional photographs of the patient's teeth; using the virtual camera parameters for each of the one or more two-dimensional photographs of the patient's teeth to generate a two-dimensional image from a three-dimensional model of the patient's teeth having the expected positions of the patient's teeth at the point in time when the two-dimensional photographs were taken; and A comparison is made to determine how closely each of the one or more two-dimensional photographs of the patient's teeth corresponds to a corresponding two-dimensional image generated from the three-dimensional model of the patient's teeth.

2. The method of claim 1 further comprising notifying a dentist that a dental plan is not proceeding as planned if the one or more two-dimensional photographs are not closely contrasted with corresponding two-dimensional images generated from the three-dimensional model within a correspondence threshold level.

3. The method of claim 1, further comprising taking the one or more two-dimensional photographs of the patient's teeth with a digital camera.

4. The method according to claim 1, wherein: Comparing and determining includes determining an overall correlation coefficient.

5. The method according to claim 4, wherein: Determining how closely each of the one or more two-dimensional photographs of the patient's teeth corresponds to a corresponding two-dimensional image generated from the three-dimensional model of the patient's teeth includes determining whether a correlation coefficient is greater than a threshold value.

6. The method according to claim 1, wherein: Determining the virtual camera parameters includes describing the position, orientation, and one or more camera characteristics of a virtual camera that produces a generated image based on a three-dimensional model of the patient's teeth at a time point corresponding to the one or more two-dimensional photographs of the patient's teeth. The method of claim 1 , further comprising optimizing the virtual camera parameters.

8. The method according to claim 7, wherein: Optimizing the virtual camera parameters includes: For each of multiple iterations, the current virtual camera parameters of the image are used to generate a next corresponding image generated based on the three-dimensional model of the patient's teeth, the loss between the image and the next corresponding image is calculated, and the virtual camera parameters of the image are perturbed in a direction that minimizes the loss between the image and the next corresponding image.

9. The method according to claim 8, wherein: Optimizing the virtual camera parameters includes applying a Nelder-Mead downhill simplex optimization method with seven dimensions, including three rotations, three translations, and the virtual camera viewing angle.

10. The method according to claim 1, wherein: Receiving more than one two-dimensional photograph of the patient's teeth in the system includes receiving a two-dimensional photograph of an upper jaw of the patient and a two-dimensional photograph of a lower jaw of the patient.

11. The method according to claim 1, wherein: The comparison includes: for each two-dimensional photograph of the patient's teeth, preparing a tooth contour based on a corresponding two-dimensional image generated based on the three-dimensional model of the patient's teeth; overlaying the contour onto the two-dimensional photograph of the patient's teeth; and generating a fit metric for the two-dimensional photograph of the patient's teeth, the corresponding image, and the overlaid contour.

12. The method according to claim 11, wherein: The goodness of fit metric is based on a per-pixel loss equal to the normalized dot product of the gradient vectors of the image and the corresponding image computed for each pixel when the magnitude of the gradient vectors are both greater than 1.

13. The method according to claim 12, wherein: The goodness of fit metric is the sum of the per-pixel losses for pixels covering the contour divided by the length of the contour.

14. The method according to claim 12, wherein: The goodness of fit metric is the sum of per-pixel losses for pixels covering the contour divided by the maximum sum of losses of pixels in a compact neighborhood including each pixel covering the contour.

15. The method of claim 1, further comprising indicating whether treatment is proceeding according to a treatment plan or not proceeding according to a treatment plan based on the comparison.

16. A method performed in a dental treatment monitoring system, the dental treatment monitoring system having: one or more processors; one or more electronic memories storing instructions and data; one or more mass storage devices storing encoded images and patient information; and a communication subsystem, wherein the dental treatment monitoring system receives images and information from a remote computer system via the communication subsystem, the method comprising: At a time point during dental treatment, planning a three-dimensional model of the patient's teeth at the time point, wherein the three-dimensional model of the patient's teeth is prepared before the start of the dental treatment; receiving, in the dental treatment monitoring system, via the communication subsystem, one or more two-dimensional photographs of the patient's teeth taken at the time point; determining virtual camera parameters for each of the one or more two-dimensional photographs of the patient's teeth; generating a two-dimensional image from a three-dimensional model of the patient's teeth at the point in time using the virtual camera parameters for each of the one or more two-dimensional photographs of the patient's teeth; and Comparing and determining how closely each of the one or more two-dimensional photographs of the patient's teeth corresponds to a corresponding two-dimensional image generated from the three-dimensional model of the patient's teeth at the point in time.

17. The method of claim 16, further comprising taking the one or more two-dimensional photographs of the patient's teeth with a digital camera.

18. The method of claim 16, further comprising notifying a dentist when the dental treatment is not proceeding according to a treatment plan.

19. The method according to claim 16, wherein: Comparing and determining includes determining an overall correlation coefficient.

20. The method according to claim 19, wherein: Determining how closely each of the one or more two-dimensional photographs of the patient's teeth corresponds to a corresponding two-dimensional image generated from the three-dimensional model of the patient's teeth at the time point includes determining whether the correlation coefficient is greater than a threshold.

21. The method according to claim 16, wherein: Determining the virtual camera parameters includes describing the position, orientation and one or more camera characteristics of a virtual camera that produces a generated image based on the three-dimensional model of the patient's teeth at the time point corresponding to the one or more two-dimensional photographs of the patient's teeth.

22. The method of claim 16, further comprising optimizing the virtual camera parameters.

23. The method according to claim 22, wherein: Optimizing the virtual camera parameters includes: For each of multiple iterations, the current virtual camera parameters of the image are used to generate a next corresponding image generated based on the three-dimensional model of the patient's teeth at the time point, the loss between the image and the next corresponding image is calculated, and the virtual camera parameters of the image are perturbed in a direction that minimizes the loss between the image and the next corresponding image.

24. The method according to claim 23, wherein: Optimizing the virtual camera parameters includes applying a Nelder-Mead downhill simplex optimization method with seven dimensions, including three rotations, three translations, and the virtual camera viewing angle.

25. The method of claim 16, wherein: Receiving more than one two-dimensional photograph of the patient's teeth in the dental treatment monitoring system includes receiving a two-dimensional photograph of the patient's upper jaw and a two-dimensional photograph of the patient's lower jaw.

26. The method of claim 16, wherein: The comparison includes: for each two-dimensional photograph of the patient's teeth, preparing a tooth contour based on a corresponding two-dimensional image generated based on the three-dimensional model of the patient's teeth at the time point; overlaying the contour onto the two-dimensional photograph of the patient's teeth; and generating a fit metric for the two-dimensional photograph of the patient's teeth, the corresponding image, and the overlaid contour.

27. The method according to claim 26, wherein: The goodness of fit metric is based on a per-pixel loss equal to the normalized dot product of the gradient vectors of the image and the corresponding image computed for each pixel when the magnitude of the gradient vectors are both greater than 1.

28. The method according to claim 27, wherein: The goodness of fit metric is the sum of the per-pixel losses for pixels covering the contour divided by the length of the contour.

29. The method according to claim 27, wherein: The goodness of fit metric is the sum of per-pixel losses for pixels covering the contour divided by the maximum sum of losses of pixels in a compact neighborhood including each pixel covering the contour.

30. The method of claim 16, further comprising indicating whether the treatment is proceeding according to a treatment plan or not proceeding according to a treatment plan based on the comparison.

31. A dental treatment monitoring system, comprising: More than one processor; one or more electronic memories storing instructions and data; one or more mass storage devices storing the encoded images and patient information; a communication subsystem through which the dental treatment monitoring system receives images and information from a remote computer system; and Computer instructions encoded in one or more of the one or more electronic memories, the computer instructions controlling the dental treatment monitoring system to: storing a three-dimensional model of the patient's teeth in one or more data storage devices selected from the one or more electronic memories and the one or more mass storage devices, receiving, via the communication subsystem, one or more two-dimensional digital photographs of the patient's teeth taken at time t during a dental treatment procedure, time-planning the three-dimensional model of the patient's teeth to the time t to generate and store the time-planned three-dimensional model of the patient's teeth representing the desired configuration of the patient's teeth, determining virtual camera parameters for each input two-dimensional digital photo for the planned three-dimensional model, and generating a two-dimensional digital image corresponding to the two-dimensional digital photo from the time-planned three-dimensional model of the patient's teeth using the determined parameters, comparing one or more of the one or more two-dimensional digital photos with a corresponding two-dimensional digital image generated based on the time-planned three-dimensional model of the patient's teeth to generate one or more comparison values, determining whether the configuration of the patient's teeth is within a threshold level of correspondence corresponding to an expected configuration of the patient's teeth based on one or more comparison values ​​obtained by comparing the generated two-dimensional digital image with the two-dimensional digital photograph, and The determined identification is stored in an electronic memory of the one or more electronic memories.

32. A method performed in a dental treatment monitoring system having: one or more processors; one or more electronic memories storing instructions and data; one or more mass storage devices storing encoded images and patient information; A communication subsystem, wherein the dental treatment monitoring system receives images and information from a remote computer system via the communication subsystem, and the method comprises: storing a three-dimensional model of the patient's teeth in one or more data storage devices selected from the one or more electronic memories and the one or more mass storage devices; receiving, via the communication subsystem, one or more two-dimensional digital photographs of a patient's teeth taken at time t during a dental treatment procedure; time-planning the three-dimensional model of the patient's teeth to a time t to generate and store the time-planned three-dimensional model of the patient's teeth representing a desired configuration of the patient's teeth; determining virtual camera parameters for each input two-dimensional digital photo for the planned three-dimensional model, and generating a two-dimensional digital image corresponding to the two-dimensional digital photo according to the time-planned three-dimensional model of the patient's teeth using the determined parameters, comparing one or more of the one or more two-dimensional digital photographs with a corresponding two-dimensional digital image generated based on the time-planned three-dimensional model of the patient's teeth to generate one or more comparison values; determining whether the configuration of the patient's teeth is within a threshold level of correspondence corresponding to an expected configuration of the patient's teeth using one or more comparison values ​​generated by comparing the generated two-dimensional digital image with the two-dimensional digital photograph; and The determined identification is stored in an electronic memory of the one or more electronic memories.

33. A computer-implemented method for evaluating dental treatment of a patient's teeth, comprising: receiving a two-dimensional image of the patient's teeth taken at a specific time during a dental treatment procedure; generating a two-dimensional image from the three-dimensional model representing the desired configuration of the patient's teeth at a particular time; comparing the two-dimensional image to a two-dimensional image generated from a three-dimensional model representing an expected configuration of the patient's teeth at the particular time; determining a set of virtual camera parameters describing an estimated position and orientation of a virtual camera that produces the generated two-dimensional image based on the three-dimensional model; determining an iteration variable N based on metadata associated with the two-dimensional image data; iteratively modifying the set of virtual camera parameters N times to produce an improved generated image based on the three-dimensional model; generating a contrast value of the two-dimensional image; and A determination is made based on the one or more comparison values ​​whether the configuration of the patient's teeth is within a threshold level of correspondence with the desired configuration of the patient's teeth.

34. A dental treatment monitoring system comprising a computing device readable medium having instructions executable by one or more processors to cause a computing device to: receiving a two-dimensional image of a patient's teeth taken at a specific time during a dental treatment procedure; generating a two-dimensional image from the three-dimensional model representing the desired configuration of the patient's teeth at a particular time; comparing the two-dimensional image to the two-dimensional image generated from a three-dimensional model representing an expected configuration of the patient's teeth at the particular time; determining a set of virtual camera parameters describing an estimated position and orientation of a virtual camera that produces the generated two-dimensional image based on the three-dimensional model; determining an iteration variable N based on metadata associated with the two-dimensional image data; iteratively modifying the set of virtual camera parameters N times to produce an improved generated image based on the three-dimensional model; generating a contrast value of the two-dimensional image; and A determination is made based on the comparison value as to whether the configuration of the patient's teeth is within a threshold level of correspondence with the desired configuration of the patient's teeth.

35. A dental treatment monitoring system, comprising: More than one processor; one or more electronic memories storing instructions and data; one or more mass storage devices storing the encoded images and patient information; a communication subsystem through which the dental treatment monitoring system receives images and information from a remote computer system; and computer instructions encoded in one or more of the one or more electronic memories, the computer instructions controlling the dental treatment monitoring system to store a three-dimensional model of the patient's teeth in one or more data storage devices selected from the one or more electronic memories and one or more mass storage devices, receiving via the communication subsystem one or more two-dimensional digital photographs of the patient's teeth taken at time t during a dental treatment procedure, time-programming the three-dimensional model of the patient's teeth to time t to generate and store a time-programmed three-dimensional model of the patient's teeth representing a desired configuration of the patient's teeth, comparing one or more of the one or more two-dimensional digital photographs with a corresponding two-dimensional digital image generated from the time-programmed three-dimensional model of the patient's teeth, to generate one or more contrast values, and based on the received one or more two-dimensional digital photographs, generate a set of one or more processed images associated with metadata for analysis, for each image in the set of one or more processed images, determine a set of virtual camera parameters describing a position and orientation of a virtual camera that produces an image generated based on a time-planned three-dimensional model of the patient's teeth that is equivalent to an image in the set of one or more processed images, for each image in the set of one or more processed images, generate an initial set of virtual camera parameters for an image in the set of one or more processed images using a standard type view and additional metadata, and determine a value of an iteration variable using the standard type view and additional metadata; performing a plurality of optimization iterations, the number of optimization iterations being equal to the value of the iteration variable; and finally, fine-tuning virtual camera parameters for the images in the set of the one or more processed images, generating, for each image in the set of the one or more processed images, a generated image corresponding to the image based on the time-planned three-dimensional model of the patient's teeth and the virtual camera parameters determined for the image in the set of the one or more processed images, and comparing each image in the set of the one or more processed images with the corresponding generated image to generate one or more contrast values ​​for the image in the set of the one or more processed images, determining whether the configuration of the patient's teeth is within a correspondence threshold level corresponding to the desired configuration of the patient's teeth based on the one or more contrast values, and storing the determined identification in one of the one or more electronic memories.

36. A method performed in a dental treatment monitoring system having: one or more processors; one or more electronic memories storing instructions and data; one or more mass storage devices storing encoded images and patient information; and a communication subsystem, wherein the dental treatment monitoring system receives images and information from a remote computer system via the communication subsystem, the method comprising: storing a three-dimensional model of the patient's teeth in one or more data storage devices selected from the one or more electronic memories and one or more large-capacity storage devices; receiving through the communication subsystem one or more two-dimensional digital photographs of the patient's teeth taken at time t during the dental treatment process, and time-planning the three-dimensional model of the patient's teeth to the time t to generate and store the time-planned three-dimensional model of the patient's teeth representing the desired structure of the patient's teeth; comparing one or more two-dimensional digital photographs of the one or more two-dimensional digital photographs with corresponding two-dimensional digital images generated according to the time-planned three-dimensional model of the patient's teeth to generate one or more contrast values; generating a set of one or more processed images associated with metadata for analysis based on the received one or more two-dimensional digital photographs, and determining a set of virtual camera parameters for each image in the set of one or more processed images, the virtual camera parameters describing the position and orientation of the virtual camera, the virtual camera generating an image equivalent to the image in the set of one or more processed images according to the time-planned three-dimensional model of the patient's teeth the image generated by the time-planned three-dimensional model of the patient's teeth, for each image in the set of one or more processed images, using the standard type view and additional metadata to generate an initial set of virtual camera parameters for the image, using the standard type view and additional metadata to determine a value of an iteration variable; performing a plurality of optimization iterations, the number of optimization iterations being equal to the value of the iteration variable; finally, fine-tuning the virtual camera parameters for the image, for each image in the set of one or more processed images, generating a generated image corresponding to the image according to the time-planned three-dimensional model of the patient's teeth and the virtual camera parameters determined for the image in the set of one or more processed images, and comparing each image in the set of one or more processed images with the corresponding generated image to generate a contrast value for the image; determining whether the configuration of the patient's teeth is within a correspondence threshold level corresponding to the desired configuration of the patient's teeth using the one or more contrast values; and storing the determined identification in one of the one or more electronic memories.

37. A dental treatment monitoring system, comprising: More than one processor; one or more electronic memories storing instructions and data; one or more mass storage devices storing the encoded images and patient information; a communication subsystem through which the dental treatment monitoring system receives images and information from a remote computer system; and Computer instructions encoded in one or more of the one or more electronic memories, the computer instructions controlling the dental treatment monitoring system to: store a three-dimensional model of the patient's teeth in one or more data storage devices selected from the one or more electronic memories and one or more mass storage devices, receive through the communication subsystem one or more two-dimensional digital photographs of the patient's teeth taken at time t during the dental treatment process, time-plan the three-dimensional model of the patient's teeth to time t to generate and store the time-planned three-dimensional model of the patient's teeth representing the desired structure of the patient's teeth, compare one or more of the one or more two-dimensional digital photographs with a corresponding two-dimensional digital image generated based on the time-planned three-dimensional model of the patient's teeth to generate one or more comparison values, and based on the received one or more two-dimensional digital photographs, generating a set of one or more processed images associated with metadata for analysis, determining, for each image in the set of one or more processed images, a set of virtual camera parameters describing a position and orientation of a virtual camera that produces an image equivalent to an image in the set of one or more processed images and generated according to a temporally planned three-dimensional model of the patient's teeth, generating, for each image in the set of one or more processed images, a generated image corresponding to the image according to the temporally planned three-dimensional model of the patient's teeth and the virtual camera parameters determined for the image, and comparing each image in the set of one or more processed images with the corresponding generated image to generate a contrast value for the image, and, for each image, preparing a tooth contour according to the corresponding generated image; overlaying the tooth contour onto the image, generating a goodness of fit metric for the image, the corresponding image, and the overlaid contour, wherein the goodness of fit metric is based on a per-pixel loss equal to a normalized dot product of gradient vectors of the image and the corresponding image calculated for each pixel, determining whether the configuration of the patient's teeth is within a threshold level of correspondence corresponding to the expected configuration of the patient's teeth based on the one or more contrast values, and storing the determined identification in one of the one or more electronic memories.

Images