Information processing system for orthodontic appliances

Abstract

This concerns a system that automatically identifies the parts included in orthodontic appliances. [Solution] The information processing system of the present invention comprises: an image acquisition means for acquiring data relating to orthodontic appliances; an annotation means for assigning information to the features of parts included in the data acquired by the image acquisition means; a learning means for constructing a model for identifying parts by learning the features of parts based on the annotation data, using the data and information from the annotation means as annotation data; and an inference means for identifying parts included in new data relating to orthodontic appliances using the model constructed by the learning means, and outputting the part identification results in association with information.

Description

Technical Field

[0001] The present invention relates to a system using artificial intelligence (AI), and more particularly to a system used in the dental industry.

Background Art

[0002] In the production and management of orthodontic appliances, the identification of each part and the creation of invoices are important processes. Ideally, when a dental technician creates an invoice himself / herself, the accurate information of the parts used is directly reflected. However, dental technicians require a lot of time for the production of appliances, and invoice creation tends to be postponed. Due to the high priority of production work, it is difficult to allocate sufficient time for invoice creation, which has been a factor delaying the progress of the entire business.

[0003] Therefore, in order to smoothly proceed with invoice creation, clerical staff supported the work. The clerical staff input part information into an Excel sheet or dedicated invoice software while checking the actual items and receiving information from dental technicians to create invoices.

[0004] In addition, since the parts used are not fully described in the instructions from the dentist, the content of the instructions cannot be directly used as an invoice.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, because administrative staff lacked specialized knowledge of dental technology, it was difficult for them to accurately identify which parts were used in the finished orthodontic appliances. Understanding the types and placement of parts required additional learning and confirmation with dental technicians, which slowed down the creation of invoices. As a result, work efficiency decreased and delays occurred.

[0007] Furthermore, manual data entry into Excel spreadsheets and dedicated invoice software was prone to human errors such as incorrect or missing entries. These errors led to incomplete invoices and the provision of incorrect information to dental patients, resulting in decreased reliability and additional corrective work. In particular, incorrect entries of part names and quantities affected billing operations, impairing the overall efficiency and accuracy of the business.

[0008] To solve these problems, a new system was needed that would enable automatic part identification and automatic invoice generation, but such a system did not exist until then.

[0009] The invention described in Patent Document 1 provides a system that automatically calculates the billing amount based on image data of a prosthesis to solve the above problems. This system receives image data of a prosthesis and information from a manufacturing instruction sheet as input, and uses this to identify parts and materials. Furthermore, it analyzes this information using a learning model and automatically calculates the appropriate cost for manufacturing the prosthesis. The learning model is built based on past billing data and material usage history, and can clearly present the basis for the billing amount. In addition, the system verifies the consistency between the contents of the instruction sheet and the actual image data, reducing the risk of incorrect billing. This eliminates the inaccuracies of conventional manual data entry and billing, making it possible to improve operational efficiency and billing transparency.

[0010] Patent Document 1 describes an embodiment in which image data of a prosthesis is input, and a system is described that automatically identifies parts and materials based on that image data. First, the system acquires image data of the prosthesis using a 3D scanner or a high-resolution camera. Based on this data, a pre-learned model is applied according to the shape and material of the prosthesis, and the price is automatically calculated based on the materials and techniques used. The system also has a function to compare the contents of the instruction sheet with the image data and confirm that the information matches. This comparison function makes it possible to confirm whether the materials and techniques of the prosthesis were carried out as instructed, adding reliability to the billing. Furthermore, the system continuously learns, and even if new prosthesis patterns or techniques are added, the model is updated and highly accurate price calculations are performed.

[0011] However, the invention described in Patent Document 1 is a system relating to prosthetic devices and cannot be directly applied to orthodontic appliances.

[0012] Based on the above, the present invention aims to provide a system for automatically and efficiently recognizing parts used in orthodontic appliances. Furthermore, the system aims to provide a system that automatically generates invoices for recognized parts. [Means for solving the problem]

[0013] To solve the above problems, the present invention has the following features. The present invention is an information processing system for identifying parts included in an orthodontic appliance composed of at least one or more parts, An image acquisition method for acquiring data related to orthodontic appliances, An annotation means that adds information to the features of parts contained in the data acquired by the image acquisition means, A learning method that uses annotation data and information obtained through annotation means as annotation data, and uses the annotation data to learn the characteristics of parts and build a model for identifying parts, The system includes an inference means that uses a model constructed by a learning means to identify parts included in new data about orthodontic appliances, and outputs the part identification results in association with information.

[0014] Preferably, the system further includes a slip output means that outputs a slip relating to the parts included in the orthodontic appliance based on the output from the inference means.

[0015] Preferably, the annotation means provides information by indicating the parts included in the data acquired by the image acquisition means with bounding boxes.

[0016] Preferably, the annotation means provides information by indicating the parts included in the data acquired by the image acquisition means with polygons.

[0017] Preferably, the annotation means provides information by indicating the parts included in the data acquired by the image acquisition means with free curves.

[0018] Preferably, the image acquisition means acquires data from at least two viewpoints, such as a photograph from a first angle and a photograph from a second angle. The annotation means performs annotation on each of the photographs from the first and second angles. The learning method should involve learning features from a first perspective and features of the part from a second perspective, and then integrating the features obtained from at least two viewpoints to construct a model.

[0019] Preferably, the image acquisition means uses a 3D scanner to acquire 3D data of the orthodontic appliance. The annotation means should perform annotation on 2D images from multiple viewpoints generated from 3D data acquired by the image acquisition means.

[0020] Preferably, the image acquisition means uses a 3D scanner to acquire 3D data of the orthodontic appliance. The annotation means may perform annotation on the 3D data acquired by the image acquisition means using voxel data, point cloud data, or mesh data.

[0021] Preferably, the annotation means may automatically indicate parts with a bounding box, polygon, or free curve by utilizing the differences in color or texture of the parts included in the orthodontic appliance.

[0022] Preferably, when the annotation is corrected by the operator, the annotation means may correct the parameters for discriminating the differences in color or texture of the parts included in the orthodontic appliance.

[0023] Preferably, the correction of the parameters may be performed by correction using a feedback loop.

[0024] Preferably, the correction of the parameters may be performed by a deep learning model.

[0025] Preferably, the annotation means labels the parts of the orthodontic appliance by classifying them into large categories and small categories, and the learning means may identify the parts based on the hierarchy of large categories and small categories.

[0026] Preferably, the annotation means assigns information to small-scale image data, the learning means constructs a pre-trained model based on the small-scale dataset annotated by the annotation means, the annotation means automatically labels the information for the parts included in the orthodontic appliance using the pre-trained model, and when the annotation is corrected by the operator, the annotation means may use the corrected dataset for reconstructing the pre-trained model.

[0027] Preferably, the learning means further constructs an interrelationship model that learns the correlations between parts of the orthodontic appliance. The inference method should ideally output inference results that take into account the interrelationships between parts by applying an interrelationship model to the inference results of the model.

[0028] Furthermore, the present invention relates to an information processing system for generating a learning model for identifying parts included in an orthodontic appliance composed of at least one or more parts, An image acquisition method for acquiring data related to orthodontic appliances, An annotation means that adds information to the features of parts contained in the data acquired by the image acquisition means, The system includes annotation data and information obtained through annotation methods, and a learning method that uses this annotation data to learn the characteristics of parts and build a model for identifying parts.

[0029] Furthermore, the present invention relates to an information processing system for creating annotation data for a learning model for identifying parts included in an orthodontic appliance composed of at least one or more parts, An image acquisition method for acquiring data related to orthodontic appliances, The system includes annotation means for assigning information to the characteristics of parts contained in data acquired by an image acquisition means.

[0030] Furthermore, the present invention relates to an information processing system for identifying parts included in an orthodontic appliance composed of at least one or more parts, The system includes an inference means that learns the characteristics of parts and uses a model that has learned information about parts to identify parts included in new data about orthodontic appliances, and outputs the part identification results in association with the information.

[0031] Preferably, the system also includes a slip output means that outputs a slip relating to the parts included in the orthodontic appliance based on the output from the inference means. [Effects of the Invention]

[0032] According to the present invention, a system is provided for automatically and efficiently identifying parts of orthodontic appliances.

[0033] Furthermore, according to the present invention, by outputting the identification results to a slip, rapid recording and management become possible.

[0034] Furthermore, according to the present invention, by using bounding boxes, the area of ​​a part can be easily identified, enabling efficient annotation.

[0035] Furthermore, according to the present invention, complex-shaped parts can be precisely identified by using polygon-based or free-form curve annotation.

[0036] Furthermore, according to the present invention, the accuracy of part identification is improved by integrating photographs acquired from different angles.

[0037] Furthermore, according to the present invention, visual information can be utilized from multiple perspectives by converting 3D data to 2D and annotating it.

[0038] Furthermore, according to the present invention, by directly annotating 3D data, precise identification that reflects shape and depth becomes possible.

[0039] Furthermore, according to the present invention, automatic annotation reduces the burden of manual work and enables efficient data processing.

[0040] Furthermore, according to the present invention, the identification accuracy can be further improved by adjusting parameters based on feedback.

[0041] Furthermore, according to the present invention, annotation accuracy is gradually improved by parameter optimization utilizing feedback.

[0042] Furthermore, according to the present invention, adaptive and highly accurate annotation can be achieved by modifying parameters using deep learning.

[0043] Furthermore, according to the present invention, detailed and structural management becomes easier through part identification based on major and minor classifications.

[0044] Furthermore, by extending from a small-scale learning model to a large-scale learning model, it is possible to efficiently improve the recognition accuracy from the initial data.

[0045] Furthermore, according to the present invention, by applying an interrelationship model, it becomes possible to enable highly accurate identification based on the relationships between parts.

[0046] The purpose, features, structure, operation, and effects of the present invention and its embodiments will become even clearer from the following detailed description in reference to the accompanying drawings. [Brief explanation of the drawing]

[0047] [Figure 1] This is a block diagram showing the configuration of the orthodontic appliance recognition system 100. [Figure 2] This diagram shows the operation in the first embodiment. [Figure 3] This figure shows an example of the annotation work in the first embodiment. [Figure 4] This diagram shows the operation of the inference process in the first embodiment. [Figure 5] This figure shows an example of the annotation process in the second embodiment. [Figure 6] This figure shows the model generation process in the third embodiment. [Figure 7] This figure shows the operation of the inference process in the third embodiment. [Figure 8] This figure shows the operation of automatic bounding box generation in the sixth embodiment. [Figure 9] This figure shows the operation of automatic annotation in the seventh embodiment. [Figure 10] This figure shows an example of annotation for major and minor categories in the eighth embodiment. [Figure 11] This figure shows the operation of automatic annotation in the ninth embodiment. [Figure 12] This figure shows the operation using the CNN model and the interaction learning model in the 10th embodiment. [Modes for carrying out the invention]

[0048] First, an overview of the embodiments of the present invention described herein will be presented. (First embodiment: Annotation using bounding boxes) In the first embodiment, an annotation method using rectangular bounding boxes is employed for 2D planar images. For images of orthodontic appliances acquired using an image acquisition device, an operator manually creates bounding boxes and assigns labels of part names to them. A CNN model is trained using this data and automatically identifies parts in new images. This method is simple and efficient.

[0049] (Second embodiment: Polygon-based annotation) In the second embodiment, an annotation method using polygon-based bounding boxes is employed. This allows for more accurate outlining of wires and irregularly shaped parts, improving identification accuracy. Because annotation is applied along the polygon shape, the contours of complex parts can be precisely captured, minimizing background areas.

[0050] (Third embodiment: Feature map integration from multiple viewpoints) In the third embodiment, information obtained from two viewpoints, a plan view and a front view, is integrated to improve the accuracy of part identification. By training a CNN model with feature maps obtained from each viewpoint, the contours and detailed shapes of the parts are identified more accurately. Furthermore, the same feature map integration is performed during inference, resulting in improved identification accuracy.

[0051] (Fourth embodiment: 2D image generation from 3D data and multiview CNN) In the fourth embodiment, 3D data is acquired, 2D images are generated from different viewpoints, and identification is performed using a multi-view CNN. Based on the 3D data, 2D images obtained from multiple angles such as the front, side, and top are annotated, and the CNN model learns from these images. This method reflects depth and detailed shape information that cannot be obtained from 2D images alone, improving the identification accuracy.

[0052] (Fifth embodiment: Identification by direct use of 3D data) In the fifth embodiment, 3D data is handled directly, and part identification in 3D space is performed without converting it to a 2D image. The 3D data format can be voxel, point cloud, or mesh data, and annotation methods are adopted according to the characteristics of each. This allows for accurate identification of depth and subtle shape differences that cannot be captured with 2D data, enabling highly accurate identification.

[0053] (Sixth embodiment: Automatic bounding box generation using an image processing algorithm) In the sixth embodiment, bounding boxes are automatically generated using image processing algorithms such as edge detection, region segmentation, and contour detection. Corrections made by the operator are used as feedback to adjust the algorithm parameters and improve annotation accuracy. By repeating the feedback loop, the automatic annotation becomes increasingly refined.

[0054] (Seventh embodiment: Automatic parameter adjustment using deep learning) In the seventh embodiment, deep learning is used to adjust the parameters of automatic annotation. By utilizing a feedback loop, the CNN model automatically learns to correct errors, eliminating the need to manually adjust the parameters of edge detection and region segmentation, thus enabling more accurate and efficient automatic annotation.

[0055] (Eighth embodiment: Part identification by major and minor categories) In the eighth embodiment, parts are divided into major and minor categories, and a CNN model is used to identify each. The operator manually creates bounding boxes and generates image data with major and minor category labels for each part, and the CNN model is trained based on this data. The model learns the hierarchical relationships between the parts, thereby improving the identification accuracy.

[0056] (Ninth embodiment: Automated annotation using a pre-trained model and feedback loop) In the ninth embodiment, automatic annotation is performed using a pre-trained model, and accuracy is improved through a feedback loop of operator corrections. Pre-training is performed by annotating a small dataset of photographs and images found in textbooks and catalogs, and then the model is applied to a large dataset. By using the operator-corrected data as feedback and optimizing the model parameters, highly accurate automatic annotation is achieved.

[0057] (Tenth embodiment: Combination of CNN model and interaction model) The tenth embodiment relates to a system that learns the correlations between each part of an orthodontic appliance and improves the accuracy of part identification by a CNN. The system is configured to simultaneously train a reciprocal relationship model that learns the correlations between parts, in addition to extracting and identifying part features using a CNN model.

[0058] Based on the above overview, each embodiment will now be described in detail. (First embodiment) The configuration of the orthodontic appliance recognition system 100 in the first embodiment of the present invention will be described in detail with reference to Figure 1.

[0059] The orthodontic appliance recognition system 100 comprises an image acquisition device 101, a data processing device 102, a network-attached storage (NAS) 103, and a learning model computing device 104.

[0060] The image acquisition device 101 includes a high-resolution digital camera, a shooting table, and a lighting device. The data processing device 102 is a device for managing and annotating image data, and comprises at least a computer main unit, a display device, and an input device. Furthermore, it comprises a central processing unit (CPU), main memory, and auxiliary storage.

[0061] The learning model computing device 104 is a device for performing high-speed training and inference of deep learning models. The learning model computing device 104 comprises at least a computer main unit, a graphics processing unit (GPU), main memory, and auxiliary storage. The graphics processing unit (GPU) should preferably be a device capable of high-speed deep learning computation, but is not limited to that. The auxiliary storage is a high-speed SSD or NVMe storage, etc., and stores the training dataset and the data of the constructed learning model. The auxiliary storage of the learning model computing device is mainly used for training and saving the model.

[0062] The data processing unit 101 and the learning model computing unit 104 are connected to the NAS 103 via a LAN (Local Area Network). A shared folder is created on the NAS 103 to store annotation data, training datasets, and learning model data. This allows for smooth data transfer between the two devices. The learning model computing device 104 and / or NAS 103 may also be devices located in the cloud.

[0063] As described above, the data processing unit 101 and the learning model computing unit 104 each use independent auxiliary storage devices and primarily perform different roles. The auxiliary storage device of the data processing unit 101 is used for storing and managing annotation data and image data. On the other hand, the auxiliary storage device of the learning model computing unit 104 is used for storing the training dataset and the data of the constructed learning model.

[0064] Data sharing between the two devices is performed via NAS103. This allows for the efficient transfer of annotation data from the data processing device 101 to the learning model computing device 104. Furthermore, data safety and availability can be ensured through data backup and redundancy.

[0065] It should be noted that data sharing via NAS103 is merely one example; the data processing unit 101 and the learning model computing unit 104 could also exchange data directly, or use cloud storage for data exchange. Furthermore, the data processing unit 101 and the learning model computing unit 104 could be implemented using a single computer, or using three or more computers.

[0066] In the above configuration, the data flow from the image acquisition device 101, data processing device 102, NAS 103, to the generation of the trained model in the learning model computing device 104 will be explained with reference to Figure 2.

[0067] First, in step S101, the user takes a high-resolution planar photograph of the orthodontic appliance using the high-resolution digital camera of the image acquisition device 101. The captured image data is input to the data processing device 102 via data transfer means such as a USB cable, memory card, or wireless communication.

[0068] In step S102, the user performs annotation on the acquired image data in the data processing device 102, adding information corresponding to each part. This annotated image data is temporarily stored in the auxiliary storage device of the data processing device 102. Next, the data processing device 102 saves the annotated image data to the NAS 103 via the network.

[0069] The learning model computing device 104 accesses the NAS 103 and acquires annotated image data (step S103). Using the acquired data, the learning model computing device 104 performs training (step S104) and inference (step S105) using, for example, a CNN (Convolutional Neural Network).

[0070] The data used in this embodiment is summarized below. 1. Raw image data - Source: Images are captured by image acquisition device 101 (high-resolution digital camera). -Contents: High-resolution planar photographs of orthodontic appliances. -Storage location: Temporarily stored in the auxiliary storage device of the data processing device 102.

[0071] 2. Preprocessed image data - Source: Generated by the data processing device 102 by performing preprocessing such as resizing and normalization on the raw image data. -Content: Image data converted into a format suitable for inference processing. -Storage location: The data is temporarily stored in the data processing device 102 or the learning model computing device 104 as needed.

[0072] 3. Annotated image data - Source: Generated by performing annotation work on pre-processed image data in the data processing device 102. -Contents: Image data with rectangular bounding boxes and labels attached to each part. -Storage location: The data is temporarily stored in the auxiliary storage device of the data processing device 102. It is then stored in the NAS 103 and shared with the learning model computing device 104.

[0073] 4. Training dataset -Contents: This is a collection of annotated image data aggregated on NAS103. - Usage: Used for training the CNN model on the learning model computing device 104.

[0074] 5. Pre-trained model data - Source: Generated by training a CNN model using the training dataset on the learning model computing device 104. -Content: This data includes parameters and structure of a pre-trained CNN model. -Storage location: Primarily stored in the auxiliary storage device of the learning model computing unit 104. Also stored in NAS103, making it accessible from other devices.

[0075] 6. Inference result data - Source: Generated by the learning model computing device 104, which uses the trained model to perform inference on newly input preprocessed image data. -Contents: This data includes the identification results of each part, location information (bounding box), class label, etc. -Storage location: Temporarily stored in the auxiliary storage device of the learning model computing device 104. Transmitted and stored in the data processing device 102 and NAS 103.

[0076] 7. Invoice data - Source: Automatically generated by the data processing device 102 based on the inference result data. -Contents: This is data necessary for creating a document that includes information about each part. -Storage location: The data is stored in the auxiliary storage device of the data processing device 102, and printing or linking to other systems is performed as needed.

[0077] Next, the annotation process in the data processing device 102 will be explained in detail, referring to the annotation example in Figure 3.

[0078] First, a planar photograph of the orthodontic appliance is taken by the image acquisition device 101. The acquired image data is then transferred to the data processing device 102. The data processing device 102 is running software for performing annotation work.

[0079] The data processing device 102 displays the acquired image data on the screen. At this point, the brightness and contrast are adjusted manually or automatically as needed. The user performing the annotation work opens the annotation interface (software) and loads the image data.

[0080] As shown in Figure 3, annotation is the process of identifying each part on an image and assigning a corresponding label to it. In the first embodiment, a rectangular bounding box is used to enclose the target part. The rectangular bounding box is set to accurately cover the entire part and is adjusted to minimize the background area.

[0081] Users annotate rectangular bounding boxes by adding label information. Select the appropriate label for the part. Labels are predefined and can be selected from a list or entered manually. For example, part names such as retainers and clasps are assigned as labels corresponding to each bounding box. Any known or future data format can be used for annotation.

[0082] Once the annotation process is complete, the annotated image data is temporarily stored in the auxiliary storage device of the data processing device 102. After the annotation process is complete, the data processing device 102 allows the user to review the annotation results and make corrections as needed. Here, the user checks whether the annotation is done correctly and whether each bounding box accurately encloses the correct part. If inaccurate annotations are found, users can correct them by readjusting the bounding box or changing the label.

[0083] The finally verified and corrected annotated image data is saved to NAS103 (Network Attached Storage). This enables data sharing with the learning model computing device 104, allowing the learning process to proceed.

[0084] The learning model computing device 104 constructs a deep learning model (CNN: Convolutional Neural Network) based on annotated data and performs a learning process to automate the identification of orthodontic appliance parts. The details of this process are described below.

[0085] The learning model computing unit 104 retrieves the annotated data stored in the NAS 103. The retrieved data is appropriately divided into training, validation, and test sets. The division ratio is not limited, but for example, it may be set as follows: Training data (learning data): 70% Validation data: 15% Test data (evaluation data): 15%

[0086] The learning model computing device 104 performs preprocessing on the acquired data. This preprocessing is an important step to enable the CNN model to learn efficiently. For example, the following are examples of preprocessing steps, but they are not limited to these. Image resizing: Resizes all images to the input size required by the model (e.g., 224 x 224 pixels). Data normalization: Scaling pixel values ​​to a range of 0 to 1 allows for smoother learning. Data augmentation (as needed): Perform operations such as rotating, zooming, and flipping images to increase data variety and improve the versatility of the model.

[0087] Next, we move on to the step of building the learning model. In this embodiment, we will use a Convolutional Neural Network (CNN). On the learning model computing device 104, the architecture of the CNN model is defined using, for example, a well-known deep learning library such as TensorFlow or PyTorch. Note that the deep learning library used is not limited to this invention.

[0088] A CNN model generally consists of the following layers: Convolutional Layer: This layer extracts features from an image and detects image edges and patterns through filters. Pooling Layer: Reduces image size and computational cost while preserving important features. Fully Connected Layer: Classifies parts based on extracted features. Output Layer: This layer outputs labels for the parts, and the classification results are obtained.

[0089] To optimize the performance of the learning model, for example, the following parameters can be set. Note that the following parameters are examples and do not limit the present invention. Epoch count: Specifies the number of times the model will train on the entire dataset. For example, it is typically set to around 10 to 100 epochs. Batch size: Determines the number of data items to process at once. For example, the batch size can be set to 16, 32, 64, etc., and is selected according to the memory capacity and processing speed. Loss function: In classification tasks, for example, cross-entropy loss is used. Optimization algorithms: To optimize the model parameters, algorithms such as Adam or SGD (stochastic gradient descent) are used.

[0090] Once the learning model computing device 104 has completed the construction of the learning model, it moves on to the process of actually inputting data into the model and training the model. Training data is input into the model, and the learning process begins. In the convolutional layer, image features are extracted, the pooling layer reduces the image size, and this is then passed to the fully connected layer, from which the predicted part labels are output from the output layer. Any well-known technique can be used in the learning process of a CNN.

[0091] Furthermore, the learning model computing device 104 organizes models according to the learning conditions and dataset versions in order to manage multiple models. This allows for efficient management of models trained under specific conditions, and makes it easy to refer to and use those models when needed. For example, if a model trained on a certain version of the dataset excels at recognizing a particular part, that model can be reused to perform inference with high accuracy for that specific recognition.

[0092] Evaluating a trained model is a crucial step in objectively measuring the performance of a trained model and selecting a model suitable for operation. The trained model computing device 104 evaluates the trained CNN (Convolutional Neural Network) model, and the model's quality is confirmed based on this evaluation. The details of the trained model evaluation process are described below.

[0093] After training is complete, a test dataset is used to evaluate the model's performance. The test dataset consists of data not used for training or validation, and is used to assess how accurately the model can identify parts in unfamiliar data.

[0094] The test dataset is a portion of the dataset (generally about 15% of the total) that was divided at the start of training, and the learning model computer 104 uses this data to verify the generalization performance of the model. The test data includes labeled part information. The test dataset undergoes the same preprocessing (resizing, normalization, etc.) as during training. This allows the model to be evaluated in an environment consistent with the training environment.

[0095] The performance of the model is measured based on well-known, standard evaluation metrics. Each metric quantitatively indicates whether the model can accurately identify parts. The learning model calculator 104 compares the outputted prediction results with the correct labels (annotated labels) included in the test dataset. Based on this comparison, evaluation metrics (e.g., well-known evaluation metrics such as accuracy, recall, precision, and F1 score) are calculated.

[0096] The learning model computing device 104 outputs the calculation results for each evaluation metric, and the overall performance of the model is evaluated. Based on these evaluation results, it is determined whether the model is suitable for operation or whether retraining is necessary.

[0097] Based on the evaluation results, it is important to consider ways to improve the model. In particular, if the performance is poor for a specific part or if there are many false positives, consider the following improvement measures. If there are many false positives for a particular part, it may indicate that there is insufficient data related to that part. In this case, it is effective to add new annotations to the dataset and retrain the model. If the model cannot correctly identify complex parts, you can consider improving the structure of the CNN. For example, you can increase the number of convolutional layers or enhance feature extraction by using different filter sizes. By readjusting the parameters of data augmentation (image rotation, zoom, flip, etc.) performed during training, the generality of the model is improved, thereby improving performance on test data.

[0098] Based on the evaluation results performed in the learning model computing device 104, the model to be used in operation is ultimately selected. Models with high accuracy and recall are given priority, but the most suitable model is adopted considering the performance indicators required for the business.

[0099] The model deemed to have the best performance is stored in the learning model computing unit 104 and NAS103 and used in subsequent inference processes and actual operations.

[0100] Next, we will explain the inference process during actual operation in detail, referring to Figure 4. The inference process uses a trained model to automatically identify parts in new data (images of orthodontic appliances). The trained model stored in the training model computing device 104 is used to identify parts in newly acquired image data, and the results are then used for generating invoices and other tasks.

[0101] The data used for inference consists of images of orthodontic appliances newly acquired by the image acquisition device 101. To perform inference, this data is first properly prepared (step S201). New image data is acquired using the image acquisition device 101. The planar photographs are transferred to the data processing device 102, which performs preprocessing. The data processing unit 102 resizes images to match the input size of the model (e.g., 224 x 224 pixels), normalizes pixel values ​​by scaling them to a range of 0 to 1, and standardizes image formats in order to conform to the format expected by the learning model. The pre-processed image data is transferred from the data processing device 102 to the learning model computing device 104 via the NAS 103.

[0102] The learning model computing device 104 loads the saved trained model and prepares to perform inference on new data (step S202). When the learning model computing device 104 loads a trained model, it inputs the preprocessed image data into the model and executes the inference process (step S203).

[0103] Preprocessed image data is fed into the model's input layer. The convolutional layer receives the image data and extracts features. Within the model, important features (shape, positional relationships, color, texture, edges, contours, etc.) are extracted from the image through multiple convolutional and pooling layers. This allows the model to analyze and identify the features of the parts contained within the image. The extracted features are passed to the fully connected layer, where the parts are classified. The position (bounding box) and label of each part are predicted and obtained from the model's output layer.

[0104] As an inference result, identification results (labels) and positional information (coordinates of the bounding box) for each part are generated. These results are transmitted to the data processing device 102.

[0105] Since the inference results cannot be used as is, the learning model computing device 104 performs post-processing as needed to prepare them for use in actual operations (step S204). The model's inference results include the confidence level (probability) of the prediction for each part. Based on the set threshold, predictions with low confidence levels are excluded. Duplicate bounding boxes and misrecognitions are also corrected at this stage. The inference results are converted into a format suitable for document creation and visualization. Labels for each part, bounding box coordinates, and confidence levels are organized, making them easy to use in subsequent processes.

[0106] After the inference is complete and the results are obtained, the results are returned to the data processing device 102 and used where needed (step S205). The data processing device 102 stores the transmitted inference results in an appropriate format (e.g., JSON, CSV, etc.). The stored inference results are then used for subsequent document creation or as feedback to the system. The inference results are visualized on the data processing device 102 and displayed to the user. The position and labels of each part are overlaid on the image, allowing for an intuitive understanding of the inference results. Based on the inference results, the data processing device 102 sends identification information for each part to the invoice creation system. The data processing device 102 automatically inputs information such as the name, quantity, and unit price of the parts, and the invoice is automatically generated.

[0107] Before the inference results are used, the user can review and correct them as needed. They can then verify that the final document is accurate and provide feedback for future learning (step S206).

[0108] The user verifies whether the inference results are accurate and whether the bounding box and labels are correct. If there are misrecognitions, they can be manually corrected. The corrected inference results and new data are fed back into the system, becoming part of the training dataset, which is then used for future training and model updates. This feedback process allows the system to continuously improve, and the accuracy of the model also increases.

[0109] In the first embodiment described above, a system for automatically identifying parts used in orthodontic appliances was explained. Specifically, the process described in detail involved first acquiring a planar photograph of the orthodontic appliance using an image acquisition device 101, then performing annotation on the image using a data processing device 102, and finally generating a learning model using a convolutional neural network (CNN) in a learning model computing device 104. The annotated data is shared throughout the system via a NAS 103, and inference is performed using the learned model, thereby automating the identification of each part of the orthodontic appliance. Furthermore, invoices are automatically generated based on the identified part information, enabling efficient business operations.

[0110] (Second embodiment) In the second embodiment, a polygon-based bounding box annotation and a learning process using a CNN are employed for planar photographs. The main feature of this embodiment is that it can perform precise annotation along the shape of each part, and can identify parts with complex shapes with high accuracy.

[0111] Similar to the first embodiment, a planar photograph of the orthodontic appliance is acquired using the image acquisition device 101. The acquired image is high resolution, clearly capturing even the fine details of wires, resin, and metal parts. Next, each part is annotated using a polygon-based annotation tool in the data processing device 102. Polygon-based annotation can handle curved wires and irregularly shaped parts, and can accurately enclose complex parts that could not be captured with a rectangular bounding box. Figure 5 shows an example of polygon-based annotation.

[0112] On the data processing unit 102, the user freely places vertices along the contours of the parts to form polygons and enclose each part. This minimizes the background and ensures accurate labeling. Labels can be selected from a list or entered manually. The annotated data is then saved from the data processing unit 102 to the NAS 103 and shared with the learning model computing unit 104.

[0113] The learning model computing device 104 uses polygon-based annotation data to train a CNN model. By inputting polygon coordinate data, the model learns features corresponding to the shape of the parts, and the convolutional layer extracts the features of each part in detail. This allows for more accurate identification of the contours and detailed shapes of the parts.

[0114] Unlike the first embodiment, the second embodiment uses a polygon-based bounding box, so polygon regions are also generated during the inference process.

[0115] Polygon-based annotation enables highly accurate inference for complex and intricate parts. This is because, while rectangular bounding boxes can sometimes include background elements, the polygon-based approach allows for more precise contour-based identification, which is particularly effective for curved wires and irregularly shaped parts.

[0116] By introducing polygon-based annotation, although the annotation process is somewhat more time-consuming in the second embodiment, there is a significant advantage in that the recognition accuracy is improved. This is expected to reduce misrecognition and correction work when creating documents based on inference results, thereby improving operational efficiency.

[0117] Alternatively, you can use a free curve to enclose the area of ​​the part and then perform the annotation.

[0118] (Third embodiment) In the third embodiment, a method is employed to improve part identification accuracy by using image data (planar and frontal photographs) acquired from multiple viewpoints and integrating the feature maps obtained from each viewpoint. For example, planar photographs capture the overall outline of the device, the arrangement of parts, and their relative positions in detail, while frontal photographs more clearly show the detailed shape and characteristics of each component. By integrating this information, more accurate identification is achieved.

[0119] Referring to Figure 6, the generation and integration of feature maps in the model generation process of the third embodiment will be described. First, planar and frontal photographs are input to the learning model calculation device 104 (S301, S302). The planar photograph captures the overall shape of the device and the relative positions of its parts, while the frontal photograph shows the detailed shapes of each component.

[0120] The learning model computing device 104 first sends the input image to the convolutional layer and applies a filter (S303, S304). This filter plays a role in highlighting specific features from the image, such as edges, patterns, and part shapes, and extracting important parts.

[0121] The learning model computing device 104 generates a feature map from the input image by applying a filter (S305, S306). The feature map is data that visualizes the important features in the image, showing which parts have what features. The feature map used during training is used by the model to learn common patterns from the input image and plays a role in improving the accuracy of part identification.

[0122] The feature map generated by the learning model computing device 104 has its features summarized through a pooling layer (S307, S308). The pooling layer reduces the size of the feature map, decreasing the computational cost while retaining important features. This process makes the data passed to the next layer more abstract and allows for more effective information transfer.

[0123] The learning model computing device 104 integrates the contour and arrangement information of the parts obtained from the planar photographs with the detailed shape information obtained from the frontal photographs (S309). This integrated feature map is used as training data so that the learning model can identify parts with high accuracy. As a result, the model learns common features related to the shape and arrangement of the parts, improving its accuracy.

[0124] The learning model computing device 104 processes the integrated feature map through convolutional and pooling layers, and finally sends it to the fully connected layer (S310). The fully connected layer is the final stage of the CNN model, where it classifies and identifies parts based on the integrated feature map. The labels and positional information of each part are finally determined, enabling the model to achieve highly accurate part identification.

[0125] Next, we will explain the inference process with reference to Figure 7. During inference, new planar and frontal photographs are input to the trained model of the learning model computing device 104 (S401, S402).

[0126] The trained model of the learning model computing device 104 applies filters in the convolutional layer to a new input image, just as it did during training, to extract features (S403, S404). The features obtained here are based on the edges and shapes of the parts, and the features necessary for identification are emphasized.

[0127] The trained model of the learning model computing device 104 generates a feature map during inference based on features obtained from new images (S405, S406). The feature map generated during inference is used to identify parts from the input image based on the knowledge of the trained model.

[0128] The feature maps generated during inference by the trained model of the learning model computing device 104 are also summarized in the pooling layer, and only the important features are retained (S407, S408).

[0129] The trained model of the learning model computing device 104 integrates information on the arrangement and relative positions of parts obtained from planar photographs with detailed shape information of each part obtained from frontal photographs (S409). As a result, the model identifies parts more accurately based on the rich information obtained from both viewpoints.

[0130] The feature maps summarized and integrated in the pooling layer are passed to the connected layer, where the trained model in the learning model computing device 104 performs the final part classification and outputs the inference result (S410). In the connected layer, the model utilizes the feature maps obtained from new images based on the knowledge it gained during training and outputs the part labels.

[0131] Furthermore, bounding boxes and labels generated based on inference results may undergo post-processing. In particular, if results from multiple viewpoints do not match or if the feature map is insufficient, the final decision will be made based on highly reliable information. In post-processing, information obtained from planar and frontal photographs is adjusted and modified to obtain highly accurate classification results.

[0132] Furthermore, if necessary, additional side photographs can be acquired and integrated to further improve identification accuracy. Side photographs provide information about depth and shape, complementing information that cannot be obtained from plan and front photographs.

[0133] Thus, in the third embodiment, by integrating information from multiple perspectives, more accurate part identification is achieved, and the overall system performance is improved.

[0134] (Fourth embodiment) In the fourth embodiment, a method is employed that uses 3D data to generate 2D images obtained from multiple viewpoints and performs part identification using a multi-view CNN. This method allows for the utilization of depth of shape and detailed information from different angles that cannot be captured by 2D images alone.

[0135] In the fourth embodiment, the image acquisition device 101 includes a 3D scanner. First, 3D data of the orthodontic appliance is acquired by the 3D scanner. The 3D scanner scans the entire appliance and records its shape with high precision from multiple angles. Based on this 3D data, 2D images from different viewpoints are generated.

[0136] Next, the data processing device 102 generates 2D images rendered from various angles, such as the front, side, and top, based on the acquired 3D data. These images contain shape information obtained from the 3D data and reflect the features from each viewpoint in detail. For each 2D image, bounding boxes and polygon annotations are applied to each part, similar to the annotation in the first or second embodiment, and labels are assigned. This creates an annotation dataset that accurately reflects information from different angles.

[0137] The learning model computing device 104 is capable of executing a multi-view CNN. The multi-view CNN learns images from various angles, generating a trained model. The concept of the multi-view CNN is as described in the third embodiment.

[0138] The learning model computing device 104 inputs the newly created 2D image into the trained multi-view CNN model. The multi-view CNN extracts feature maps obtained from each viewpoint and integrates them to identify parts. This reflects detailed shape and arrangement information that cannot be obtained from a single viewpoint, improving the identification accuracy.

[0139] 2D images from multiple viewpoints based on 3D data encompass information from different angles, significantly improving recognition accuracy, especially for complex shapes and fine details. Furthermore, it allows for easy implementation using existing 2D CNN frameworks while leveraging the advantages of 3D data.

[0140] (Fifth embodiment) In the fifth embodiment, an approach is adopted that directly handles 3D data, performing part identification using 3D data without converting it to a 2D image. This method makes full use of the information in 3D space, enabling more precise identification.

[0141] First, the image acquisition device 101 acquires the shape of the orthodontic appliance using a 3D scanner and generates high-precision 3D data. The acquired 3D data contains detailed shape information of the entire appliance, reflecting information about the shape, arrangement, and depth of each component.

[0142] The acquired 3D data can be treated as voxel data (cubic grid data), point cloud data, or mesh data. Each format has different characteristics, and the appropriate format is selected depending on the shape of the data being analyzed and the purpose of the analysis.

[0143] Voxel data: This format divides 3D space into a grid, with each grid indicating the presence or absence of an object. Shapes are represented as cubes, making it easy to handle, but fine details are difficult to represent. Point cloud data: A format that represents the surface of an object as a point cloud, assigning attributes such as coordinates and color to each point. While it can represent the shape in detail, the amount of data is large. Mesh data: A format that represents the surface of an object using a polyhedron of triangles and quadrilaterals. It is widely used to efficiently represent detailed shapes.

[0144] Next, the data processing unit 102 performs annotation work in 3D space. Labels are assigned to each part of the 3D data, and the shape and position of each part are precisely defined. Unlike conventional 2D images, annotation on 3D coordinates is accurate as it takes into account spatial structure and depth information.

[0145] In the fifth embodiment, annotation is performed using 3D data. In this embodiment, multiple annotation methods are employed depending on the characteristics of the 3D data, but are not limited to bounding boxes. Each annotation method will be described below.

[0146] 1. Annotation using 3D bounding boxes This method involves setting a bounding box, which is a cube, around a target part in 3D space. Similar to 2D bounding boxes, it employs a format where the part is enclosed by the smallest possible rectangular prism.

[0147] 2. Voxel-based annotation This method divides the target part into voxels in 3D space and assigns a label to each voxel. A voxel is the smallest unit of a cube in 3D space, and it represents the structure of an object in a grid format. This method makes it possible to annotate complex shapes and fine parts with high accuracy.

[0148] 3. Point cloud-based annotation This annotation method uses point cloud data and assigns a label to each point. A point cloud consists of countless points representing the surface of a target part, making it possible to accurately identify the part to which each point belongs. This method is particularly effective for annotating parts with complex curved surfaces or intricate structures.

[0149] 4. Mesh-based annotation This method uses mesh data to label the triangular or quadrilateral polygons that make up the surface of an object. By annotating the part to which each polygon belongs, it is possible to accurately capture curved surfaces and complex shapes. This method is particularly suitable for parts with complex shapes or detailed surface structures.

[0150] As described above, in the fifth embodiment, 3D data annotation requires the selection of an appropriate method depending on the shape and characteristics of the part. This makes it possible to accurately identify parts with complex shapes, which was difficult with conventional 2D data.

[0151] The annotated 3D data is input into the 3D deep learning models (e.g., VoxelNet, PointNet, MeshCNN) of the learning model computing unit 104. These models are designed to directly handle 3D data and identify each part. VoxelNet: A model that uses voxel data as input to perform object detection and identification in 3D space. PointNet: A model that takes point cloud data as input and identifies shapes and parts from the point cloud. MeshCNN: A model that takes mesh data as input and performs shape recognition based on polyhedral structures. These models fully utilize shape information in 3D space to improve the accuracy of part identification. In particular, they can accurately identify complex shapes and minute parts.

[0152] In this way, the learning model computing device 104 generates a 3D deep learning model. When new 3D data is input to the generated 3D deep learning model, the parts used are inferred.

[0153] In the fifth embodiment, by directly handling 3D data, it is possible to learn in detail depth and subtle shape differences that cannot be captured with 2D data, thereby maximizing the recognition accuracy. Furthermore, it enables complex shape recognition that exceeds the limitations of existing 2D CNNs, resulting in the highest level of recognition performance.

[0154] (Sixth embodiment) This embodiment relates to a system for automatically generating bounding boxes (or polygons or free curves) to represent parts, in order to streamline the automatic annotation of orthodontic appliances. In particular, the sixth embodiment automatically generates bounding boxes using a conventional image processing algorithm and improves annotation accuracy by optimizing the algorithm parameters through a feedback loop.

[0155] Orthodontic appliances consist of a white plaster model of the teeth and metal or resin parts. Annotation can be performed by utilizing the clear difference in color and texture between the plaster and the parts. In this embodiment, conventional algorithms such as edge detection, region segmentation, and contour detection are used, and each parameter is adjusted based on feedback to generate an efficient and highly accurate bounding box.

[0156] Referring to Figure 8, the operation of the data processing device 102 for automatic bounding box generation will be explained.

[0157] First, an image acquisition device 101 is used to acquire images of the orthodontic appliance (planar photographs, frontal photographs, etc.). These images include white plaster models and metal or resin parts (S501).

[0158] Next, the data processing device 102 extracts the contours of metal and resin parts using edge detection algorithms (such as Sobel filters and Canny edge detection) (S502). Edge detection captures the boundary between the part and the background, which is the first step in generating the bounding box. By appropriately setting parameters that determine the strength and sensitivity of the edges (for example, the threshold of the Sobel filter and the threshold of Canny edge detection), highly accurate contours can be obtained.

[0159] The data processing device 102 separates the plaster parts from the metal / resin parts by utilizing differences in color and texture (S503). This distinguishes the areas where the parts are located from the background and identifies the areas that form the basis for annotation. Clustering methods or edge-based segmentation (such as the Watershed method) are used, and parameters such as color thresholds and edge strengths are adjusted based on feedback.

[0160] The data processing device 102 uses a contour detection algorithm to capture the contour of an object in detail (S504). For the regions of parts identified by region segmentation, contour detection is used to extract even more precise contours.

[0161] Then, the data processing unit 102 generates a final bounding box based on the contour (S505). The smoothness of the contour and the detection threshold are also adjusted by feedback.

[0162] The edge detection, region segmentation, and contour detection algorithms can be used individually or in combination. The optimal algorithm is selected based on the image characteristics, and the results are combined to generate a bounding box. <Example of standalone use> If there is a clear color difference between the part and the background, the contour can be extracted using simple edge detection, and a bounding box can be generated directly. <Example of combined use> When the shape of a part is complex, edge detection is used to obtain a rough boundary, region segmentation is used to separate the background using color difference, and contour detection is used to fill in the finer contours. This combination allows for the generation of highly accurate bounding boxes even for parts with complex shapes.

[0163] In this way, the data processing device 102 automatically generates bounding boxes from input planar images, frontal images, and images from various angles obtained from 3D data. The operator then annotates the parts to be associated with the automatically generated bounding boxes to the data processing device 102.

[0164] By the way, human operators check whether the automatically generated bounding boxes are appropriate and identify errors such as the following: 1. Over-annotation: When the bounding box is wide enough to include the background. 2. Under annotation: When the bounding box does not enclose the entire part. 3. Incorrect area annotation: When an area that is not a part is included in the bounding box. If the operator identifies one of the three errors described above, the data processing device 102 will correct the bounding box. Based on the correction made by the operator, the data processing device 102 can recognize which of the three errors it is.

[0165] Based on the feedback regarding this error, the data processing device 102 adjusts the parameters as follows: 1. Edge detection threshold The threshold used for edge detection is adjusted based on feedback to ensure that the contours of parts are accurately extracted. For example, if over-annotation occurs, the threshold is increased to remove unnecessary noise, and if under-annotation occurs, the threshold is decreased to recognize more edges. In the case of incorrect area annotation, the edge detection threshold is readjusted to exclude the misidentified area and optimize the threshold to avoid detecting edges of noise or unwanted parts.

[0166] 2. Color threshold for region segmentation To utilize the color difference between metal or resin parts and plaster models, the color threshold in area subdivision is adjusted. In the case of over-annotation, the threshold is tightened to remove unnecessary areas, and in the case of under-annotation, the threshold is loosened to cover more areas. In addition, in the case of incorrect area annotation, the color threshold is adjusted to more accurately reflect the color difference between the part and the background, preventing misidentification.

[0167] 3. Contour detection accuracy If the contours are too complex or lack smoothness, adjust the contour detection parameters to extract smooth and appropriate contours. In the case of over-annotation, increase the contour threshold to remove unwanted areas and suppress background detection by smoothing the contours. In the case of under-annotation, lower the contour threshold to broaden the detection range and improve contour approximation accuracy to accurately enclose the entire part. In the case of incorrect region annotation, adjust the threshold and contour smoothing to accurately remove the misidentified areas and suppress misidentification by performing accurate contour extraction.

[0168] Furthermore, parameter adjustment by the data processing unit 102 is not a one-time process; accuracy improves through repeated feedback loops. With each loop, the accuracy of the bounding box increases, ultimately achieving a level of automated annotation where manual correction is almost unnecessary.

[0169] In the sixth embodiment, the post-annotation learning and inference processes are the same as those described in the other embodiments.

[0170] This embodiment streamlines the automatic generation of bounding boxes by combining conventional algorithms with a feedback loop, significantly reducing the burden of manual annotation. It is particularly effective when there are clear differences in color or texture, such as with orthodontic appliances.

[0171] (Seventh Embodiment) The seventh embodiment relates to a system that utilizes deep learning to improve the accuracy of bounding box generation in the automated annotation of orthodontic appliances. In particular, while based on conventional image processing algorithms (edge ​​detection, region segmentation, contour detection), the system aims to improve the final annotation accuracy by introducing deep learning, which automatically adjusts parameters through a feedback loop.

[0172] Figure 9 illustrates the overview of the automated annotation process. First, the image acquisition device 101 acquires images of orthodontic appliances (such as planar or frontal photographs) and inputs them into the data processing device 102 (S601). These images include white plaster models and metal or resin parts.

[0173] Next, the data processing device 102 first generates an initial bounding box using conventional image processing algorithms (edge ​​detection, region segmentation, contour detection) (S602). In this initial generation stage, fixed default parameters are used, so it is expected that the bounding box will not be perfectly accurate.

[0174] The generated initial bounding box is evaluated by a human operator (S603). If any errors are found, the operator corrects the bounding box. Over-annotation: When the bounding box is wide enough to include the background. Under annotation: When the bounding box does not enclose the entire part. Incorrect region annotation: When a region that is not a part is included in the bounding box. The operator's corrections are input to the data processing device 102 as feedback, and based on this feedback, the parameters are adjusted using deep learning as described below.

[0175] (Automatic parameter tuning using deep learning) Based on feedback from the operator, the deep learning model in the data processing unit 102 (or the learning model computing unit 104) automatically adjusts the parameters (S604). This generates a more accurate bounding box in the next annotation process.

[0176] Here, we will explain how the parameters are adjusted. 1. Adjusting edge detection parameters The deep learning model learns from feedback data and automatically adjusts the edge detection threshold. If the threshold is too high, under-annotation occurs; if it's too low, over-annotation occurs. The model learns from these errors and sets the optimal threshold. 2. Adjusting the parameters for region division The threshold values ​​in the region segmentation algorithm are also dynamically adjusted using deep learning. The model learns the color difference and texture between parts and the background, and sets the optimal threshold value to accurately capture differences in color and brightness, thereby improving the accuracy of part identification. 3. Adjusting Contour Detection Parameters The contour detection algorithm is also optimized using a deep learning model. Parameters are adjusted to improve contour accuracy so that complex shapes and fine details are accurately enclosed.

[0177] After processing in S604, the data processing unit 102 returns to the operation of S601, and as a result, the following process is repeated by a feedback loop using deep learning. 1. Processing feedback through automated learning If the generated bounding box is not appropriate, the model automatically evaluates the result as feedback and reflects it in the next annotation. 2. Sequential parameter optimization By repeating the feedback loop, the parameters are automatically adjusted repeatedly, and the accuracy of bounding box generation improves step by step. 3. Reduced operator intervention As deep learning models are continuously improved, the need for manual corrections by operators gradually decreases, and the accuracy of automated annotation increases.

[0178] Thus, the seventh embodiment significantly improves the accuracy of bounding box generation by automating the feedback loop using deep learning and efficiently adjusting parameters. This also reduces the burden on human operators and ultimately enables highly accurate annotation automatically. As a result, the efficiency of annotation work is dramatically improved and the burden of manual annotation is greatly reduced.

[0179] (Eighth embodiment) The eighth embodiment relates to a system for identifying parts of orthodontic appliances by dividing them into major and minor categories, and for learning using a CNN model. In this embodiment, an operator manually encloses the parts with bounding boxes and generates image data that associates the part name with a major or minor category label. The CNN model is trained using this data, and ultimately becomes capable of automatically identifying and classifying parts in new images.

[0180] The parts of orthodontic appliances can be broadly divided into major and minor categories. Major categories are the main parts that support the overall structure of the appliance, such as retainers and bite plates. On the other hand, minor categories are parts that are attached to these major categories, such as clasps, labial wires, and elastic wires. It should be noted that the major and minor categories of parts are merely examples and do not limit the present invention. This classification is based on the shape and function of the parts, and in CNN models, it is used to identify whether smaller parts are associated with the locations where larger parts exist. This allows the model to make more accurate predictions than simply identifying individual parts.

[0181] Figure 10 shows an example of annotation for major and minor categories. First, the operator manually creates bounding boxes around the shapes of the parts based on the image of the orthodontic appliance. This clearly indicates the area in which each part is located. Then, the following information is associated with each bounding box: Name (bite plate, single clasp, Adams clasp, labial line, etc.) Labels for major or minor classifications (bite plates are major classifications, single clasps, Adams clasps, and labial lines are minor classifications) This manual process is performed by the operator based on the function and shape of the parts, and a verification process is incorporated to prevent misidentification.

[0182] Next, the manually generated image data with bounding boxes is input into a CNN (Convolutional Neural Network) model, and the model on the learning model computing device 104 undergoes training. The training flow of this model is as follows: (1) Feature extraction from input image CNN models extract features based on the shape and texture of parts from input image data. This feature extraction process considers the shape, edges, and color of parts contained within their bounding boxes, learning information that helps in part identification. (2) Distinguishing between major and minor categories The CNN model identifies major classification parts and their associated minor classification parts based on the results of feature extraction. Here, the position and shape information of the major classification parts influence the prediction of the existence of the minor classification parts. Through this process, the model learns the hierarchical relationships between parts, improving its identification accuracy.

[0183] Once the training model is complete, it can be used to perform part identification on images of new orthodontic appliances. From newly input images, the CNN model automatically detects the location of the parts and associates the part name and major / minor category labels with each part. This information is also used when reflecting the parts identification results on invoices. In particular, recording whether a part belongs to a major or minor category helps to streamline the design information and parts management of equipment.

[0184] Thus, the eighth embodiment provides a system that improves identification accuracy and annotation efficiency in the identification of orthodontic appliance parts by simultaneously learning major and minor classifications.

[0185] By identifying sub-categorized parts based on major category parts, it becomes possible to leverage the relationships between parts and achieve highly accurate part identification. In particular, even complex sub-categorized parts can be misidentified and identification reliability improved by being associated with the major category.

[0186] By having operators annotate both major and minor categories together, the accuracy of the learned model is improved, and manual annotation work becomes more efficient. This reduces the workload and ensures consistent annotation quality.

[0187] By managing parts hierarchically, categorized into major and minor groups, the precise positional relationships between parts can be understood during design and manufacturing, improving the efficiency of parts management. Hierarchical information is also reflected when creating invoices, allowing for the rapid and accurate creation of such invoices.

[0188] Models based on the relationship between major and minor categories are easy to correct when misrecognition occurs, and errors can be detected and corrected efficiently.

[0189] (Ninth embodiment) The ninth embodiment relates to a system that improves annotation accuracy by utilizing a small-scale pre-trained model using photographs from existing textbooks and catalogs to streamline the automated annotation of orthodontic appliance parts, and then improving the accuracy of the annotation through a feedback loop that combines automated annotation on a large dataset with operator corrections.

[0190] The automated annotation in the ninth embodiment will be described with reference to Figure 11. First, as a pre-training stage, the data processing device 102 manually performs annotation (including bounding boxes and classification labels) on small-scale examples of orthodontic appliances listed in textbooks and catalogs (S701).

[0191] Using this annotated data, the CNN model of the learning model computing device 104 is trained to create a pre-trained model (S702). This pre-trained model can learn to identify orthodontic appliances based on major and minor classifications, even with a small dataset, and can be applied to the initial stages of highly accurate automated annotation.

[0192] Specifically, in S701, during the pre-training phase, the operator uses the data processing unit 102 to manually draw bounding boxes from images in textbooks and catalogs, and assigns major and minor category labels to each part. For example, they manually annotate "retainer" as the major category and "single clasp" as the minor category, and then pre-train the CNN model of the learning model computing unit 104 based on this information. This pre-training lays the foundation for automated annotation in the initial stages.

[0193] The operator uses the pre-trained model of the learning model computer 104 to perform automated annotation on a large dataset of orthodontic appliances. As initial annotation, the pre-trained model of the learning model computer 104 automatically generates bounding boxes and classification labels based on major and minor categories for newly input images (S703).

[0194] After the initial annotation is complete, a human operator reviews the results and makes corrections as needed (S704). Corrections may include the following: Over-annotation: When the bounding box includes background area outside the part. Under annotation: When the bounding box does not enclose the entire part. Incorrect labeling: When the labels for major and minor categories are incorrect.

[0195] Annotation data corrected by the operator is stored as feedback data and used to adjust the model parameters. Specifically, thresholds for edge detection and region segmentation, contour accuracy for contour detection, etc., are automatically optimized based on the feedback and reflected in the next automated annotation. This creates a feedback loop, resulting in a gradual improvement in accuracy.

[0196] Alternatively, parameter tuning can be performed using deep learning. In particular, by utilizing feedback data, the CNN model itself can automatically learn error correction, further improving the accuracy of part identification and classification. This eliminates the need for manual parameter tuning of edge detection and region segmentation, resulting in more efficient and accurate automated annotation.

[0197] In the ninth embodiment, manual annotation work is streamlined by a pre-trained model, improving annotation accuracy for large datasets. Through repeated feedback loops, accuracy continuously improves, eventually refining automated annotation to a level where manual correction is virtually unnecessary. Furthermore, the identification of major and minor categories of parts facilitates detailed document creation and parts management, resulting in a more efficient business process.

[0198] Thus, in the ninth embodiment, by combining a pre-trained model and a feedback loop, highly accurate and efficient automated annotation becomes possible, which is particularly effective even for complex component configurations such as orthodontic appliances.

[0199] Furthermore, the annotated training set created in the ninth embodiment is trained using a CNN, and a trained model is generated by the training model computing device 104.

[0200] (Tenth embodiment) In the tenth embodiment, a system is described that learns the correlations between each part of an orthodontic appliance and improves the accuracy of part identification by a CNN. A CNN model is used for feature extraction and part identification, and a model for learning part correlations is trained in parallel. A tenth embodiment will be described with reference to Figure 12.

[0201] Using one of the methods already described, the data processing device 102 first annotates the image data of the orthodontic appliance with bounding boxes and polygons based on the shape of each part (S801). This clearly identifies the position and shape of the parts and obtains the data necessary for learning correlations. This annotation data is used to learn the correlations between parts, without necessarily depending on major or minor classifications.

[0202] The learning model computing device 104 uses annotation data to enable the CNN model to learn features based on the shape and texture of each part using one of the methods already described (S802). The CNN generates feature maps from the image through convolutional layers and identifies the parts. This process takes into account the contours and edge information of the parts to achieve high-precision identification.

[0203] In the tenth embodiment, the learning model computing device 104 simultaneously constructs a model for learning the correlations between parts (hereinafter referred to as the interrelationship model) (S803). Specifically, it uses one or a combination of the following algorithms. Graph Neural Networks (GNNs): These networks represent the relationships between parts as a graph structure and learn how each part relates to other parts. This makes it possible to model the probability of a particular part appearing simultaneously with other parts. Co-occurrence matrices and probabilistic models: A co-occurrence matrix of parts is created, and the frequency of part combinations is learned using statistical methods. Conditional probabilities between parts are estimated using probabilistic graphical models (e.g., Bayesian networks). Embedding-based approach: Each part is embedded in a vector space, and correlations are learned based on the distance and similarity between parts. This method contributes to improved recognition accuracy by learning to move similar parts closer together spatially.

[0204] After the CNN model and the interrelationship model are generated in this way, new images are input to the learning model computing device 104 (S804). The CNN in the learning model computing device 104 will execute the inference process and output parts (S805).

[0205] The interaction model performs an inference process (S806) and eliminates combinations with a low probability of occurrence. The correlation model selects combinations of parts with a high probability of occurrence from the remaining candidates. In this process, it utilizes pre-learned correlation information of parts to prioritize outputting the most reasonable combination (S807). Furthermore, the correlation model presents other candidate parts that are predicted to have a high probability of occurrence in place of the eliminated parts. This makes the output more accurate.

[0206] Thus, in the tenth embodiment, integrating the CNN with the parts correlation model significantly improves parts identification accuracy compared to using a standalone CNN model. Furthermore, because it enables flexible correlation learning that is independent of major and minor classifications, it can easily handle new parts configurations and unknown combinations. In addition, it is expected that misrecognition will decrease, improving reliability in invoice creation and parts management.

[0207] In all of the above embodiments, the image processing device 101 functioned as an image acquisition means for acquiring data related to orthodontic appliances (such as photographs, images, 2D data, and 3D data), but other devices or systems may also assume the role of image acquisition means. Similarly, while the data processing device 102 functioned as an annotation means for adding information to the characteristics of parts included in the data relating to orthodontic appliances, other devices or systems may also assume the role of annotation means. Furthermore, while the learning model computing device 104 functioned as a learning means to build a model for identifying parts by learning the characteristics of the parts based on annotation data, other devices or systems may also assume the role of learning means. Furthermore, while the learning model computing device 104 functioned as an inference means to identify parts included in new data about orthodontic appliances using the constructed model and output the part identification results in association with information, other devices or systems may also take on the role of inference means.

[0208] Furthermore, the present invention includes a computer program that implements the above means using a computer device.

[0209] Although the present invention has been described in detail above, the above description is merely illustrative in all respects and is not intended to limit its scope. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. Each constituent element of the invention disclosed herein shall stand as an independent and standalone invention. Inventions that combine each constituent element in any way shall also be included in the present invention. The specific expressions in this specification are merely illustrative, and the present invention shall also include conceptualizations of such illustrative expressions. [Industrial applicability]

[0210] This invention provides an information processing system that can automatically recognize parts included in orthodontic appliances, and is industrially applicable. [Explanation of Symbols]

[0211] 100 Orthodontic Appliance Recognition System 101 Image acquisition device 102 Data Processing Devices 103 Network Attached Storage (NAS) 104 Computing device for learning models

Claims

1. An information processing system for identifying a part included in an orthodontic appliance composed of at least one part, An image acquisition means for acquiring data related to the aforementioned orthodontic appliance, An annotation means for assigning information to the features of the parts included in the data acquired by the image acquisition means, A learning means that uses the data and information obtained by the annotation means as annotation data, and learns the characteristics of the parts based on the annotation data to construct a model for identifying the parts, An information processing system characterized by comprising: an inference means that uses the model constructed by the learning means to identify the parts included in new data relating to the orthodontic appliance, and outputs the identification result of the parts in association with the information.

2. An information processing system for generating a learning model for identifying parts included in an orthodontic appliance composed of at least one part, An image acquisition means for acquiring data related to the aforementioned orthodontic appliance, An annotation means for assigning information to the features of the parts included in the data acquired by the image acquisition means, An information processing system characterized by comprising: a learning means that uses the data and information obtained by the annotation means as annotation data, and uses the annotation data to learn the characteristics of the parts and construct a model for identifying the parts.

3. An information processing system for creating annotation data for a learning model for identifying parts included in an orthodontic appliance composed of at least one part, An image acquisition means for acquiring data related to the aforementioned orthodontic appliance, An information processing system characterized by comprising annotation means for assigning information to the features of the parts included in the data acquired by the image acquisition means.

4. An information processing system for identifying a part included in an orthodontic appliance composed of at least one part, An information processing system comprising: an inference means that learns the characteristics of the aforementioned parts and uses a model that has learned information about the aforementioned parts to identify the aforementioned parts included in new data relating to the aforementioned orthodontic appliance, and outputs the identification result of the aforementioned parts in association with the aforementioned information.

Images