A real-time tooth movement measurement method based on image

Abstract

The present invention discloses a method for measuring tooth movement based on an image in real time. The method comprises: obtaining a target image, wherein the target image comprises a front view of a tooth bracket; inputting the target image into a trained YOLO model to detect the tooth bracket bounding box information, wherein the tooth bracket bounding box information comprises the coordinate information of the upper left and lower right diagonal points of the tooth bracket bounding box and the confidence of the tooth bracket surrounded by the bounding box; screening out the tooth bracket bounding boxes greater than a set confidence threshold, and arranging them according to the coordinate information of the bounding boxes, and then classifying them into maxillary brackets and mandibular brackets; arranging the screened tooth bracket bounding boxes in order from left to right, and measuring the distance change between teeth with reference to the length of the tooth bracket frame in the center of the picture, and then obtaining the displacement measurement result of the teeth. The present invention can measure the displacement of teeth under the action of force in a real-time and accurate manner during orthodontic process in a non-invasive manner, and does not cause additional harm to the patient.

Description

Technical Field

[0001] The present invention relates to the field of biomedical engineering technology, and more specifically, to a real-time tooth movement measurement method based on images. Background Art

[0002] Orthodontics is a dental specialty dedicated to the diagnosis, prevention, management and correction of misaligned teeth, jaws and bite patterns. Misaligned teeth and jaws are common symptoms that can be treated by long-term application of corrective forces. In order to adjust the corrective forces applied by brackets on the teeth, orthodontists need to measure the patient's interdental displacement to ensure that treatment is proceeding as planned and that the applied forces are the most appropriate for the target displacement of the teeth. However, the interdental displacement schemes currently used are usually invasive or harmful to the human body, prone to large human measurement errors or unable to be detected in real time.

[0003] At present, clinical orthodontic treatment is to install corrective appliances in the mouths of patients with malocclusion so that the teeth move under the action of force. Tooth displacement measurement requires measuring the initial position of the teeth and the current position of the teeth, then calculating the difference between the two and obtaining the final displacement through a series of conversions. In order to detect and adjust the force required for orthodontic treatment, determine the treatment status and predict the effect of further treatment, it is particularly important to design an accurate real-time tooth displacement measurement algorithm.

[0004] In the prior art, different types of scanned images have been used to achieve tooth displacement measurement. For example, patent application CN201911000420.9A describes a method for measuring the three-dimensional movement of teeth during orthodontic treatment. This solution isolates a single tooth for analysis, constructs a local three-dimensional coordinate system of the target tooth, marks three reference points at the crown bracket positions on the front and back intraoral models, calculates the spatial transformation matrix based on the reference point coordinates, and then calculates the three-dimensional translation and rotation of the tooth to obtain the three-dimensional tooth movement. This solution requires continuous acquisition of multiple intraoral CT scan models of the patient, which is easy to cause additional harm to the patient. For example, another solution uses a micro-light-emitting diode as a light source, uses a CCD chip to capture the spot changes, and digitizes the data through an acquisition card and finally outputs it as tooth displacement data. However, this solution is invasive, which will increase the patient's discomfort during the measurement process, and the cost of implementing the device is high. In short, the existing solutions mainly use invasive measurements or measurement methods based on X-rays and CT images, lack real-time measurement functions, require multiple measurements, and cause additional radiation damage to patients. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a method for measuring tooth movement based on images in real time. The method comprises the following steps:

[0006] Acquire a target image, wherein the target image includes a front view of a dental bracket;

[0007] Input the target image into the trained YOLO model to detect the bracket bounding box information, wherein the bracket bounding box information includes the coordinate information of the upper left and lower right diagonal points of the bracket bounding box and the confidence of the bracket enclosed by the bounding box;

[0008] Screen out the bracket bounding boxes that are greater than the set confidence threshold, arrange them according to the coordinate information of the bounding boxes, and then classify them into maxillary brackets and mandibular brackets;

[0009] The selected bracket boundary boxes are arranged in order from left to right, and the distance change between the teeth is measured with reference to the bracket frame size in the center of the picture, thereby obtaining the displacement measurement result of the teeth.

[0010] Compared with the prior art, the advantage of the present invention is that a tooth movement measurement method based on image detection is designed, which can measure the displacement of teeth under force during orthodontic process in real time and accurately in a non-invasive manner without causing additional harm to patients.

[0011] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0013] Figure 1 is a flow chart of a method for measuring tooth movement based on an image in real time according to an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of a process from raw orthodontic image data to motion measurement according to an embodiment of the present invention;

[0015] Figure 3 Schematic diagram of the measurement result of the distance between teeth according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0017] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0018] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0019] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0020] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0021] The present invention provides a real-time tooth displacement measurement solution, which uses the YOLO model to detect the brackets (or brackets for short) on the teeth from the image captured by the camera, and displays the measurement results in real time. In general, the bracket boundary box coordinates output by the YOLO model are first arranged and classified into maxillary brackets and mandibular brackets, so that the maxillary and mandibular brackets can be measured at the same time. Then, the bracket boundary boxes are arranged in order from left to right. Next, the distance change between each tooth is measured based on the bracket frame size in the center of the picture. Finally, the measurement results are converted into millimeters for display.

[0022] Combination Figure 1 and Figure 2 As shown, the provided image-based real-time tooth movement measurement method comprises the following steps:

[0023] Step S110, constructing a training set, wherein each sample reflects image data and a corresponding bracket bounding box.

[0024] First, a camera can be used to take images of the patient's teeth at different distances as sample images. Then, the sample images are cleaned. Images with clear bracket front view are screened out for use in training sets and test sets. For example, the final image needs to meet the following requirements: at least one bracket is a front view; the image is not too bright or too dark; the resolution is at least 640×640 pixels, etc. In one embodiment, a set of 125 sample images that meet these requirements are used, and the sample set is divided into three groups, namely 87, 25, and 13, which are used for training sets, validation sets, and test sets, respectively.

[0025] Subsequently, the sample images in the training set are annotated and the samples are expanded. The present invention only needs to identify and annotate one category, namely brackets. Then, a list of images and a text file corresponding to each image can be obtained, which contains the annotated bounding box information. Furthermore, a variety of basic image transformation techniques can be used to increase the training data to increase the accuracy of model detection. For example, image transformation methods include: vertical flipping, clockwise and counterclockwise rotation, and changing brightness (such as between -35% and +35%), etc.

[0026] In this step, by taking sample images at different distances and performing image transformation on the samples, sample images of different clarity and angles can be obtained, thereby improving the accuracy and robustness of subsequent model training. In addition, in the sample images, most patients can have brackets installed on both the maxillary and mandibular bones.

[0027] Step S120, training the YOLO model to detect the bracket boundary.

[0028] The YOLO model divides the image into a grid system, and then each grid is responsible for detecting the objects inside. In one embodiment of the present invention, the YOLO v5 model is selected as the target detection model to improve the detection speed and detection accuracy. The YOLO v5 model creates features based on the input image, and then sends these features to the prediction system to draw bounding boxes around the objects and predict their categories. The YOLO network includes three main parts: the backbone, the neck, and the head. The backbone is a convolutional neural network that is used to aggregate and form image features of different granularities. The neck is used to mix and combine image features and pass the features to the prediction layer. The head uses features from the neck to predict bounding boxes and categories.

[0029] In order to speed up model training, in practical applications, the training can be set to start from the YOLOv5l.pt pre-trained model and only used for labeling of one class (i.e. brackets).

[0030] In one embodiment, a pre-trained YOLOv5l model is used to minimize the loss during training, and during the training process, an Nvidia GTX 1070 GPU is used to increase the training speed. After 2000 iterations of training, the loss has stopped and significantly decreased. Finally, after model training, weight parameters and the like are obtained, which can be used to detect the bracket border. It should be understood that in order to ensure the accuracy of model training, during the training stage, the trained model can be verified using a validation set and a test set to determine whether the weight parameters of the model need to be further updated.

[0031] Step S130, for the target image, using the trained YOLO model to obtain the bracket bounding box information.

[0032] In practical applications, the trained YOLO model can be used to detect the bracket bounding box from the target image.

[0033] For example, when the target image is input, the final output obtained by using the trained YOLOv5 model has five elements, namely the coordinates of the upper left and lower right diagonal points of the bracket bounding box, and the confidence of the bracket enclosed by the bounding box. Only the bounding box with a confidence exceeding the set threshold will be displayed and used for measurement, and the obtained coordinate value can be a percentage of the length and width of the entire image.

[0034] Step S140: Classify the brackets into maxillary brackets and mandibular brackets according to the obtained bracket boundary box information.

[0035] The obtained bracket bounding box information can be used to detect the maxillary bracket and the mandibular bracket in the target image. In addition, when the same target image contains both maxillary and mandibular brackets, it is necessary to distinguish the maxillary and mandibular brackets and divide them into two groups for adjacent tooth displacement measurement.

[0036] For example, the designed classifier module is used to classify brackets into maxillary brackets and mandibular brackets. The classifier module takes the coordinate array of all detected bracket bounding boxes output by YOLOv5 as input and outputs two groups of bracket bounding boxes, namely maxillary and mandibular. First, the classifier loops through the coordinate array. Because the Y-axis is oriented downward in the YOLO algorithm, the bounding box closest to the upper boundary of the image is obtained by taking the bounding box with the smallest Y value, which indicates that it is one of the maxillary brackets, and the upper and lower Y values ​​of the bounding box of the bracket are used as reference limits. Therefore, the classifier module will classify according to the coordinates of each subsequent bounding box, and if the Y coordinate is between the reference limits, it will be placed in the maxillary bracket column. Any bracket that does not meet the maxillary bracket group is classified into the mandibular bracket group.

[0037] Step S150, sorting the brackets according to the actual relative positions of the teeth and brackets, and measuring the distance change between each tooth based on the size of the bracket frame in the center of the screen.

[0038] The input images of YOLOv5 are in the order in which the images of the brackets are captured by the camera, not in the order in which the teeth and brackets are in actual relative position. Therefore, the output of YOLOv5 is an array of bounding box information that is independent of the actual relative position, that is, when the bounding boxes are arranged in the order of the input, they are not arranged from left to right. To solve this problem, in one embodiment, a sorting model is created to arrange the brackets in the order in which they are located. For example, the sorting model arranges each bounding box information array from left to right according to its X coordinate.

[0039] For the distance measurement between bounding boxes, considering that the bracket facing the camera at a vertical angle in the center of the image is least affected by occlusion and rotation errors, in order to accurately measure the distance using the image when the camera position is constantly changing, in one embodiment, the bounding box size of the bracket in the center of the image is used as a reference size for measurement.

[0040] Specifically, the size of the bracket bounding box at the center of the image is calculated as follows:

[0041]

[0042] Where B is the size of the bracket bounding box at the center of the image, W is the X-axis coordinate of the far edge of the bracket, X is the X-axis coordinate of the near edge of the bracket, and n is the maximum number of brackets detected in the image, n = 0, 1, 2, ...

[0043] For example, if the distance between two bracket bounding boxes in the image is 40 pixels, and the width of the reference bounding box is 20 pixels, it can be concluded that the distance between the two brackets is equal to the width of the two reference bracket bounding boxes.

[0044] In one embodiment, to ensure that the measurement is not affected by the distance between the camera and the patient, the distance between adjacent brackets can be calculated as follows:

[0045] D(b i , b i+1 )=X i+1 -W i (2)

[0046] Where D represents the distance between two adjacent brackets, represents the i-th bracket, and b i+1 represents the i+1th bracket, X i+1 represents the X-axis coordinate of the proximal edge of the i+1th bracket, W i Represents the X-axis coordinate of the far edge of the i-th bracket.

[0047] Preferably, the measured distance finally displayed is converted into millimeters. Therefore, it is necessary to convert the distance expressed in the number of bounding boxes into a value in millimeters. For example, the conversion is completed by multiplying the number of bounding boxes contained in the distance between adjacent brackets by the actual width of the bracket. Specifically, in practice, when the width of the bracket is measured to be 4.25 mm, the measurement result finally displayed is calculated by the following formula:

[0048]

[0049] The above formula (3) can be generally expressed as:

[0050]

[0051] Among them, D w Indicates the measured width of the bracket.

[0052] During the application of force in orthodontic treatment, the measurement results of tooth displacement can be obtained by displaying the changes in the distance between teeth in real time.

[0053] Figure 3 It is the measurement result of the distance between teeth for a target image. A total of 14 brackets were detected, of which 7 correspond to the maxillary bones and the other 7 correspond to the mandibular bones.

[0054] To further verify the effect of the present invention, orthodontic tooth images of 6 patients were tested over an average of 3 years. The experimental results show that the present invention has high accuracy, including real-time measurement of continuous motion, a resolution better than 30μm, and displays the results at 25FPS. In actual application, orthodontists can wear a head-mounted camera to obtain target images while applying loads, successfully realizing real-time automatic measurement of tooth displacement based on images during orthodontic treatment.

[0055] In summary, compared with the prior art, the present invention has the following advantages:

[0056] 1) An algorithm was designed to divide the detected bracket frame into maxillary and mandibular parts, measure the movement of all teeth within the camera's viewing angle, and detect, measure and display the distance between teeth in real time to help measure the tooth displacement distance before, after and during the loading process. In addition, after detecting the bracket and measuring the distance between each tooth, the measurement results in millimeters can be obtained in real time.

[0057] 2) In actual application, a non-invasive head-mounted camera can be used to detect brackets, and the displacement of teeth under the action of force can be accurately measured in real time based on the YOLO model, reducing interference to patients during the measurement process.

[0058] 3) Provides an image-based, fully automatic, real-time tooth displacement measurement method that replaces the computed tomography and X-ray images currently used in measurement systems by utilizing images obtained with a non-invasive, lightweight head-mounted camera, significantly reducing the required setup time and health risks while maintaining measurement accuracy.

[0059] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0060] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0062] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0063] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0064] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0065] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0066] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0067] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for measuring tooth movement based on an image in real time, comprising the following steps: Acquire a target image, wherein the target image includes a front view of a dental bracket; Input the target image into the trained YOLO model to detect the bracket bounding box information, wherein the bracket bounding box information includes the coordinate information of the upper left and lower right diagonal points of the bracket bounding box and the confidence of the bracket enclosed by the bounding box; Screen out the bracket bounding boxes that are greater than the set confidence threshold, arrange them according to the coordinate information of the bounding boxes, and then classify them into maxillary brackets and mandibular brackets; The selected bracket boundary frames are arranged in order from left to right, and the distance change between the teeth is measured with reference to the size of the bracket boundary frame in the center of the picture, thereby obtaining the displacement measurement result of the teeth; The following formula is used to calculate the change in the distance between adjacent teeth: Among them, D mm Indicates the distance between adjacent teeth in millimeters, D(b i , b i+1 ) represents the i-th bracket b i and the i+1th bracket b i+1 The distance between them, B is the bounding box size of the bracket in the center of the image, and D w It means the width of the bracket. The size B of the bounding box of the central bracket of the image is expressed as: Where W is the X-axis coordinate of the distal edge of the bracket, X is the X-axis coordinate of the proximal edge of the bracket, and n is the maximum number of brackets detected in the image; Among them, the following formula is used to calculate D(b i , b i+1 ): D(b i ,b i+1 )=X i+1 -W i Among them, X i+1 represents the X-axis coordinate of the proximal edge of the i+1th bracket, W i Represents the X-axis coordinate of the far edge of the i-th bracket.

2. The method according to claim 1, characterized in that The training set for training the YOLO model is obtained according to the following steps: Screening out dental bracket images that meet the definition requirement from the sample images, wherein at least one bracket in the screened out sample images is a front view, and the sample images are taken at different distances; Cropping the screened sample image, wherein the resolution of the cropped image is at least 640×640 pixels; The sample images are labeled, and the labeled category is dental brackets, thereby obtaining an image list and a text file corresponding to each sample image as sample data, which contains the labeled dental bracket bounding box information; After the sample data is enhanced by image transformation, a training set is constructed, wherein the image transformation includes vertical flipping, clockwise and counterclockwise rotation, and brightness change.

3. The method according to claim 1, characterized in that The YOLO model is a YOLOv5 model, including a trunk, a neck and a head, wherein the trunk is used to aggregate and form image features of different granularities; the neck is used to mix and combine image features; The head uses the image features output by the neck to make bounding box and category predictions.

4. The method according to claim 1, characterized in that The maxillary bracket and the mandibular bracket are classified by the following steps: Using a classifier, taking the coordinate arrays of all bracket bounding boxes output by the YOLO model as input, obtaining the bounding box closest to the upper boundary of the image by taking the bounding box with the smallest Y value, classifying it as a maxillary bracket, and taking the upper and lower bounds of the maxillary bracket's bounding box Y values ​​as reference limits; For the remaining bracket bounding boxes, if the Y coordinate is within the reference limit, it is classified as a maxillary bracket, otherwise it is classified as a mandibular bracket.

5. The method according to claim 1, characterized in that The target image is captured by using a head-mounted camera.

6. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images