Method for measuring opening size

Abstract

Disclosed according to an embodiment of the present disclosure is a method for measuring an opening size, which is performed by a computing apparatus. The method may comprise the steps of: obtaining a face image; preprocessing the face image; extracting a contour of teeth from the preprocessed face image; and measuring an opening size by using information generated through the preprocessing and information about the extracted contour of the teeth.

Description

How to measure aperture

[0001] The present invention relates to a method for measuring aperture.

[0002] Recent advances in computer vision and deep learning technologies have opened up the possibility of analyzing such unstructured data more efficiently and accurately. In particular, the analysis of various biometric data based on facial images has become possible, creating an environment where detailed measurements, such as mouth opening, can be performed with greater precision.

[0003] Furthermore, technology that accurately calculates mouth opening by analyzing human facial images plays a crucial role in the fields of medicine, dentistry, and psychology. This analysis is particularly useful for non-invasive and non-contact assessment of a patient's oral condition or monitoring changes during treatment. Existing methods for measuring mouth opening primarily rely on physical instruments and manual calculations, which can be time-consuming and inaccurate. Therefore, there is a growing need to develop a framework capable of calculating mouth opening through real-time image analysis.

[0004] Korean Patent No. 10-2000222 (July 9, 2019) discloses an aperture measuring device.

[0005] The present disclosure aims to provide a method for measuring mouth opening by extracting a tooth contour from a facial image and calculating the distance between the upper and lower teeth using the extracted contour.

[0006] Meanwhile, the technical task to be achieved by the present disclosure is not limited to the technical task mentioned above, and may include various technical tasks within a scope obvious to a person skilled in the art from the contents described below.

[0007] According to one embodiment of the present disclosure for achieving the aforementioned task, a method for measuring mouth opening, performed by a computing device, is disclosed. The method may include the steps of: acquiring a facial image; performing preprocessing on the facial image; extracting a tooth contour from the preprocessed facial image; and measuring the mouth opening using information generated by the preprocessing and information on the extracted tooth contour.

[0008] In one embodiment, the step of performing preprocessing on the facial image may include the step of extracting facial landmark points based on the facial image using an artificial intelligence model; the step of extracting a mouth area from the facial image based on the extracted facial landmark points; and the step of performing color correction on the extracted mouth area image.

[0009] In one embodiment, the step of performing color correction on the extracted mouth area image may include at least one of a first color correction step of removing color information from the mouth area image and converting it to grayscale; a second color correction step of adjusting the brightness contrast of the mouth area image using a histogram equalization technique; a third color correction step of enhancing a boundary of the mouth area image using a binarization technique; or a fourth color correction step of removing noise from the mouth area image using a morphology operation technique.

[0010] In one embodiment, the step of extracting a contour of a tooth from the preprocessed facial image may include the step of extracting a white area as a candidate contour based on the extracted mouth area image and the extracted facial landmark points; and the step of filtering a contour corresponding to a tooth area among the extracted candidate contours.

[0011] In one embodiment, the step of filtering contours corresponding to a tooth region among the extracted candidate contours may include: analyzing an area of ​​the extracted candidate contours; setting a minimum value and a maximum value based on the area of ​​the extracted candidate contours; and extracting a contour corresponding to a tooth region among the extracted candidate contours based on the set minimum value and maximum value.

[0012] In one embodiment, the step of extracting contours of teeth from the preprocessed facial image may further include a step of dividing the filtered contours into contours corresponding to the upper teeth area or contours corresponding to the lower teeth.

[0013] In one embodiment, the step of distinguishing upper teeth and lower teeth from among the filtered contours may include a step of distinguishing the filtered contours into first contours corresponding to the upper teeth area or second contours corresponding to the lower teeth area by utilizing landmark points of the face.

[0014] In one embodiment, the step of measuring the mouth opening may include the step of merging the extracted facial landmark points with the distinguished first contours and second contours; the step of selecting an upper tooth contour and a lower tooth contour to be used for calculating the mouth opening from the first contours and second contours merged with the facial landmark points; and the step of measuring the mouth opening based on the selected upper tooth contour and the selected lower tooth contour.

[0015] In one embodiment, the step of measuring the opening amount based on the selected upper tooth contour and the selected lower tooth contour may include the step of converting the selected upper tooth contour and the selected lower tooth contour from two-dimensional coordinates to three-dimensional coordinates; and the step of measuring the opening amount by projecting the upper tooth contour and the lower tooth contour converted into the three-dimensional coordinates onto a two-dimensional plane based on the z-axis.

[0016] In one embodiment, the step of selecting upper teeth contours and lower teeth contours to be used for calculating the opening amount from the first contours or second contours merged with the landmark points of the face may include the step of selecting a contour having the largest area among the separated first contours and closest to the first point of the extracted facial landmark as the upper teeth contour; and the step of selecting a contour having the largest area among the separated second contours and closest to the second point of the extracted facial landmark as the lower teeth contour.

[0017] In one embodiment, the step of converting the selected upper teeth contour and the selected lower teeth contour from two-dimensional coordinates to three-dimensional coordinates may include the step of determining the z-axis value of the selected upper teeth contour as the z-value of the lip landmark closest to the selected upper teeth contour among the extracted facial landmark points; and the step of determining the z-axis value of the selected lower teeth contour as the z-value of the lip landmark closest to the selected lower teeth contour among the extracted facial landmark points.

[0018] In one embodiment, the step of measuring the mouth opening amount based on the selected upper teeth contour and the selected lower teeth contour may further include, after the step of converting into three-dimensional coordinates: adjusting the scale of the coordinates of the third point and the fourth point and the coordinates of the selected upper teeth contour and lower teeth contour converted into three-dimensional coordinates so that the distance between the third point and the fourth point among the extracted facial landmark points becomes a preset value; and the step of measuring the mouth opening amount by projecting the upper teeth contour and lower teeth contour converted into three-dimensional coordinates onto a two-dimensional plane with respect to the z-axis may include the step of measuring the mouth opening amount by projecting the upper teeth contour and lower teeth contour converted into three-dimensional coordinates and the scaled upper teeth contour onto a two-dimensional plane with respect to the z-axis.

[0019] In one embodiment, the step of measuring the opening by projecting the upper teeth contour and lower teeth contour converted into the three-dimensional coordinates onto a two-dimensional plane based on the z-axis may calculate the minimum distance between the upper teeth contour and the lower teeth contour using the projected data.

[0020] According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, the computer program causes the one or more processors to perform the following operations for measuring an opening amount, which may include: acquiring a facial image; performing preprocessing on the facial image; extracting a contour of a tooth from the preprocessed facial image; and measuring the opening amount by utilizing information generated by the preprocessing and information on the extracted tooth contour.

[0021] In one embodiment, the operation of performing preprocessing on the facial image may include an operation of extracting facial landmark points based on the facial image using an artificial intelligence model; an operation of extracting a mouth area from the facial image based on the extracted facial landmark points; and an operation of performing color correction on the extracted mouth area image.

[0022] In one embodiment, the operation of extracting a contour of a tooth from the preprocessed facial image may include an operation of extracting a white area as a candidate contour based on the extracted mouth area image and the extracted facial landmark points; and an operation of filtering a contour corresponding to a tooth area among the extracted candidate contours.

[0023] In one embodiment, the operation of extracting contours of teeth from the preprocessed facial image may further include an operation of dividing the filtered contours into contours corresponding to the upper teeth area or contours corresponding to the lower teeth.

[0024] In one embodiment, the operation of distinguishing upper teeth and lower teeth from among the filtered contours may include an operation of distinguishing the filtered contours into first contours corresponding to the upper teeth area or second contours corresponding to the lower teeth area by utilizing landmark points of the face.

[0025] In one embodiment, the operation of measuring the mouth opening may include an operation of merging the extracted facial landmark points with the distinguished first contours and second contours; an operation of selecting an upper tooth contour and a lower tooth contour to be used for calculating the mouth opening from the first contours and second contours merged with the facial landmark points; and an operation of measuring the mouth opening based on the selected upper tooth contour and the selected lower tooth contour.

[0026] In one embodiment, the operation of measuring the opening amount based on the selected upper tooth contour and the selected lower tooth contour may include an operation of converting the selected upper tooth contour and the selected lower tooth contour from two-dimensional coordinates to three-dimensional coordinates; and an operation of measuring the opening amount by projecting the upper tooth contour and the lower tooth contour converted into the three-dimensional coordinates onto a two-dimensional plane based on the z-axis.

[0027] In one embodiment, the operation of converting the selected upper teeth contour and the selected lower teeth contour from two-dimensional coordinates to three-dimensional coordinates may include an operation of determining a z-axis value of the selected upper teeth contour as a z-value of a lip landmark closest to the selected upper teeth contour among the extracted facial landmark points; and an operation of determining a z-axis value of the selected lower teeth contour as a z-value of a lip landmark closest to the selected lower teeth contour among the extracted facial landmark points.

[0028] A computing device according to one embodiment of the present disclosure for achieving the aforementioned task is disclosed. The device comprises at least one processor; and a memory, wherein the at least one processor is configured to acquire a facial image; perform preprocessing on the facial image; extract a tooth contour from the preprocessed facial image; and measure an opening amount using information generated by the preprocessing and information on the extracted tooth contour.

[0029] In one embodiment, the at least one processor may be configured to extract facial landmark points based on the facial image using an artificial intelligence model; extract a mouth region from the facial image based on the extracted facial landmark points; and perform color correction on the extracted mouth region image.

[0030] In one embodiment, the at least one processor may be configured to extract a white area as a candidate contour based on the extracted mouth area image and the extracted facial landmark points; and filter a contour corresponding to a tooth area among the extracted candidate contours.

[0031] In one embodiment, the at least one processor may be configured to distinguish the filtered contours into contours corresponding to the upper teeth area or contours corresponding to the lower teeth area.

[0032] In one embodiment, the at least one processor may be configured to utilize landmark points of the face to segment the filtered contours into first contours corresponding to an upper teeth area or second contours corresponding to a lower teeth area.

[0033] In one embodiment, the at least one processor may be configured to merge the extracted facial landmark points with the distinguished first contours and second contours; select an upper tooth contour and a lower tooth contour to be used for calculating the mouth opening amount from the first contours and second contours merged with the facial landmark points; and measure the mouth opening amount based on the selected upper tooth contour and the selected lower tooth contour.

[0034] In one embodiment, the at least one processor may be configured to convert the selected upper tooth contour and the selected lower tooth contour from two-dimensional coordinates to three-dimensional coordinates; and to project the upper tooth contour and the lower tooth contour converted into the three-dimensional coordinates onto a two-dimensional plane with respect to the z-axis to measure the opening amount.

[0035] In one embodiment, the at least one processor may be configured to determine the z-axis value of the selected upper teeth contour as the z-value of the lip landmark that is closest to the selected upper teeth contour among the extracted facial landmark points; and to determine the z-axis value of the selected lower teeth contour as the z-value of the lip landmark that is closest to the selected lower teeth contour among the extracted facial landmark points.

[0036] The present disclosure measures the aperture based on a facial image without a physical device, thereby saving time and effort compared to manual measurement through automated image analysis, thereby improving efficiency in the diagnosis and treatment process.

[0037] In addition, the present disclosure utilizes computer vision and deep learning technologies to measure aperture in real time, thereby enabling more precise measurement of aperture through image analysis technology compared to conventional methods relying on manual calculation or physical instruments, thereby reducing errors and ensuring consistency in measurement.

[0038] Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within a range apparent to those skilled in the art from the contents described below.

[0039] FIG. 1 is a block diagram of a computing device for measuring aperture according to one embodiment of the present disclosure.

[0040] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0041] FIG. 3 is a flowchart illustrating a method for measuring an aperture according to one embodiment of the present disclosure.

[0042] FIG. 4 is a diagram for explaining a process of performing preprocessing on a facial image according to one embodiment of the present disclosure.

[0043] FIG. 5 is a diagram showing the result of performing color correction on an extracted mouth area image according to one embodiment of the present disclosure.

[0044] FIG. 6 is a diagram for explaining an operation of extracting a contour of teeth from a preprocessed facial image according to one embodiment of the present disclosure.

[0045] FIG. 7 is a drawing for explaining an operation of distinguishing upper and lower teeth among filtered contours according to one embodiment of the present disclosure.

[0046] FIG. 8 is a diagram exemplarily showing the result of merging first contours and second contours separated from extracted facial landmark points according to one embodiment of the present disclosure.

[0047] FIG. 9 is a drawing exemplarily showing selected upper teeth contour and lower teeth contour results according to one embodiment of the present disclosure.

[0048] FIG. 10 is a drawing exemplarily showing the result of converting a selected upper tooth contour and a selected lower tooth contour from two-dimensional coordinates to three-dimensional coordinates according to one embodiment of the present disclosure.

[0049] FIG. 11 is a drawing exemplarily showing the results of performing scale adjustment and origin movement for aperture calculation according to one embodiment of the present disclosure.

[0050] FIG. 12 is a drawing visualizing the results of selecting the end points of the upper and lower teeth contours to be used in calculating the opening amount according to one embodiment of the present disclosure.

[0051] FIG. 13 is a drawing exemplarily showing the result of calculating the minimum distance between the upper teeth contour and the lower teeth contour using projected data according to one embodiment of the present disclosure.

[0052] FIG. 14 is a flowchart illustrating a method for measuring an aperture according to one embodiment of the present disclosure.

[0053] FIG. 15 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0054] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific details.

[0055] As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and / or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and / or data transmitted to another system via a network such as the Internet via signals).

[0056] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

[0057] Additionally, the terms "comprises" and / or "comprising" should be understood to imply the presence of the features and / or components in question. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, components, and / or groups thereof. Furthermore, unless otherwise specified or clear from the context to refer to the singular form, the singular in the specification and claims should generally be construed to mean "one or more."

[0058] And, the term "at least one of A or B" should be interpreted to mean "if it includes only A", "if it includes only B", or "if it is combined in the composition of A and B".

[0059] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0060] The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not limited to the embodiments disclosed herein. The present invention is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

[0061] In the present disclosure, network function, artificial neural network and neural network can be used interchangeably.

[0062]

[0063] FIG. 1 is a block diagram of a computing device for measuring aperture according to one embodiment of the present disclosure.

[0064] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0065] A computing device (100) may include a processor (110), memory (130), and network unit (150).

[0066] The processor (110) may be configured with one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in the memory (130) to perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process learning of a network function. For example, a CPU and a GPGPU can jointly process network function learning and data classification using network functions. Furthermore, in one embodiment of the present disclosure, processors of multiple computing devices can be jointly used to process network function learning and data classification using network functions. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0067] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0068] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the internet. The description of the above-described memory is merely an example, and the present disclosure is not limited thereto.

[0069] The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0070] In addition, the network unit (150) presented in this specification can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

[0071] In the present disclosure, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a local area network (LAN), a personal area network (PAN), and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth.

[0072] The techniques described in this specification can be used in other networks as well as the networks mentioned above.

[0073]

[0074] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0075] Throughout this specification, the terms artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0076] A neural network can be composed of a set of interconnected computational units, generally referred to as nodes. These nodes can also be referred to as neurons. A neural network consists of at least one node. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

[0077] Within a neural network, one or more nodes connected via links can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, meaning that any node that is in an output node relationship with one node can also be in an input node relationship with another node, and vice versa. As described above, the relationship between input nodes and output nodes can be created based on links. One input node can be connected to one or more output nodes via links, and vice versa.

[0078] In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and output nodes can have a weight. The weight can be variable and can be varied by the user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values ​​input to the input nodes connected to the output node and the weight set on the link corresponding to each input node.

[0079] As described above, a neural network is a network in which one or more nodes are interconnected through one or more links, forming input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links within the network, the relationships between the nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weight values ​​for the links, the two neural networks can be perceived as different from each other.

[0080] A neural network can be composed of a set of one or more nodes. A subset of the nodes comprising the neural network can form a layer. Some of the nodes comprising the neural network can form a layer based on their distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links required to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by its distance from the final output node.

[0081] In one embodiment of the present disclosure, a set of neurons or nodes may be defined as a layer.

[0082] An initial input node may refer to one or more nodes within a neural network into which data is directly input without going through links with other nodes. Alternatively, within a neural network, it may refer to nodes that do not have other input nodes connected by links in the relationship between nodes based on links. Similarly, a final output node may refer to one or more nodes within a neural network that do not have output nodes in their relationship with other nodes. Furthermore, a hidden node may refer to nodes that constitute a neural network other than the initial input node and the final output node.

[0083] A neural network according to one embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the above-described neural networks.

[0084] An AI-based model according to one embodiment of the present disclosure may include a deep neural network (DNN). A DNN may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Using a DNN, it is possible to identify latent structures in data. That is, the latent structures of a photo, text, video, voice, protein sequence structure, gene sequence structure, peptide sequence structure, music (e.g., what objects are in a photo, what the content and emotion of a text are, what the content and emotion of a voice are, etc.), and / or the binding affinity between a peptide and MHC can be identified. Deep neural networks may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, generative adversarial networks (GANs), transformers, and the like. The description of the above-described deep neural networks is merely an example and the present disclosure is not limited thereto.

[0085] The artificial intelligence-based model of the present disclosure can be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

[0086] The neural network that can be used in the artificial intelligence-based model of the present disclosure may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of the neural network may be a process of applying knowledge to the neural network to perform a specific action.

[0087] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, training data with the correct answer for each training data is used (i.e., labeled training data). In unsupervised learning, the correct answer may not be labeled for each training data. For example, in supervised learning for data classification, the training data may be data with each category labeled. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the training data labels. Alternatively, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network training to quickly achieve a certain level of performance, thereby increasing efficiency. A lower learning rate can be used in the later stages of training to increase accuracy.

[0088] In neural network training, training data can typically be a subset of real-world data (i.e., the data to be processed using the trained neural network). Therefore, there can be a learning cycle where errors on the training data decrease but errors on the real-world data increase. Overfitting is a phenomenon where excessive training on the training data leads to increased errors on the real-world data. For example, a neural network trained on yellow cats may fail to recognize cats when shown non-yellow colors, a type of overfitting. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. These methods include increasing the training data, regularization, dropout, which disables some nodes in the network during the learning process, and the use of batch normalization layers.

[0089]

[0090] The present disclosure develops a model for calculating mouth opening using human facial images, enabling the model to provide stable and accurate results in real time. To this end, a framework for calculating mouth opening using multiple facial images captured under various conditions is established, and the reliability and accuracy of the model are verified. The present disclosure can provide a universal tool for calculating mouth opening that can be applied to all individuals, regardless of age or gender.

[0091] Specifically, the present disclosure automates the process of accurately extracting tooth contours from facial images and calculating the distance between the upper and lower teeth to calculate mouth opening. This process involves a series of tasks, including image preprocessing, tooth contour extraction, and differentiation of the upper and lower teeth, resulting in the creation of a precise 3D model. This model can provide real-time feedback in medical and dental diagnoses and contribute to the development of personalized treatment plans. Furthermore, the framework developed based on the research results goes beyond simply calculating mouth opening and establishes a scalable technical foundation that can be applied to various biometric data analyses in the future.

[0092]

[0093] FIG. 3 is a flowchart illustrating a method for measuring aperture size according to one embodiment of the present disclosure. Meanwhile, the method for measuring aperture size may be performed by a computing device (100).

[0094] For example, referring to FIG. 3, the computing device (100) can analyze in real time a facial image of a person with an open mouth, regardless of age or gender, and return the mouth opening amount thereof. In addition, the computing device (100) can perform preprocessing of the image data to extract tooth contours from the image data and improve reliability, and after performing the preprocessing, extract tooth contours through a tooth data production process, refine them, and distinguish between the upper and lower teeth. In addition, the computing device (100) can measure (calculate) the mouth opening amount using the finally generated tooth contour data.

[0095] Hereinafter, each step included in the method for measuring the aperture performed by the computing device (100) will be specifically described with reference to FIGS. 4 to 13.

[0096] According to one embodiment of the present disclosure, a computing device (100) may acquire a facial image (S10). For reference, the acquired facial image may be an image captured while the user has their mouth open. Furthermore, the facial image may include images captured while the user has their mouth open, such that at least a portion of the teeth are visible. For example, the computing device (100) may utilize various photographing devices to acquire the facial image. For example, the photographing device may include a high-resolution camera, a 3D scanner, a non-contact infrared camera, or a smartphone camera. The computing device (100) may acquire facial images from people of various ages and genders. For example, the computing device (100) may acquire facial images encompassing all population groups, not limited to a specific race or gender, to increase the versatility of mouth opening calculations. Furthermore, the computing device (100) may acquire facial data captured in various environments, such as a dental clinic, a hospital environment, or everyday situations. Alternatively, in order to measure the mouth opening more accurately, the computing device (100) may provide guidance messages such as, "Please look straight ahead as much as possible, take the picture with your back to the light source so that the teeth are visible as much as possible, and the inside of the mouth is not reflected in the light," through the application when the user takes a facial image using a smartphone camera.

[0097]

[0098] FIG. 4 is a diagram for explaining a process of performing preprocessing on a facial image according to one embodiment of the present disclosure, and FIG. 5 is a diagram showing a result of performing color correction on an extracted mouth area image according to one embodiment of the present disclosure.

[0099] According to one embodiment of the present disclosure, the computing device (100) can perform preprocessing on a facial image. In addition, the computing device (100) can perform preprocessing to convert the facial image (original image) into a form suitable for analysis. For example, the step of performing preprocessing on the facial image may include, in detail, an image RGB conversion step (S21), a facial landmark extraction step (S22), an mouth region extraction step (S23), a grayscale conversion step (S24), a histogram equalization (CLAHE) step (S25), a binarization step (S26), and a morphology operation step (S27).

[0100] According to one embodiment, the computing device (100) may perform RGB conversion on the acquired facial image (S21). For example, referring to (a) of FIG. 4, the computing device (100) may perform RGB conversion on the facial image, which is an original image acquired through various photographing devices. For example, image data acquired from a camera may exist as a RAW file or other format of data. At this time, each pixel of the image has a unique color value according to the intensity of light, and the facial image may not be converted to RGB in the initial state. The computing device (100) may perform RGB conversion by assigning R, G, and B values ​​to each pixel of the acquired facial image in order to input the facial image into an artificial intelligence model that extracts landmark points of the face.

[0101] According to one embodiment, the computing device (100) can extract facial landmark points based on a facial image by utilizing an artificial intelligence model (S22). For example, referring to (b) of FIG. 4, the computing device (100) can extract 478 facial landmark points based on an RGB-converted facial image by utilizing an artificial intelligence model. For example, the computing device (100) inputs the RGB-converted facial image into the artificial intelligence model in step S22, and the artificial intelligence model can extract facial landmark points. The artificial intelligence model can extract a total of 478 facial landmark points from the facial image. For example, the computing device (100) can detect a facial region from the facial image. For example, technologies for detecting facial regions can be applied, such as Haar Cascades, HOG (Histogram of Oriented Gradients), MTCNN (Multi-Task Cascaded Convolutional Networks), or YOLO, but are not limited thereto, and existing or future-developed algorithms can be applied. In addition, the detected facial region can be set as a Region of Interest (ROI) for subsequent landmark extraction. In addition, an AI model can predict major facial landmark points using the detected facial region as input. The AI ​​model can be a Convolutional Neural Network (CNN)-based model or Mediapipe's Face Landmarker model, but are not limited thereto, and existing or future-developed algorithms can be applied. In addition, the AI ​​model can use structures such as Stacked Hourglass Network and UNet to extract 478 facial landmark points, and can extract more accurate landmark points through post-processing and normalization.

[0102] In one embodiment, the computing device (100) can extract a mouth area from a facial image based on extracted facial landmark points (S23). For example, referring to FIG. 4 (c), the computing device (100) can extract a mouth area from a facial image including the entire facial area based on the extracted facial landmark points. In addition, the computing device (100) can crop only the mouth area image from the facial image based on the extracted mouth area. For example, the computing device (100) can extract the mouth area from the facial image using coordinate information of the extracted facial landmark points. Meanwhile, the computing device (100) can increase the reliability of tooth classification and contour detection by extracting only the mouth area from the facial image based on the coordinate information of the extracted facial landmark points. Alternatively, the computing device (100) can also extract the mouth area from the facial image using an artificial intelligence model that recognizes the mouth area.

[0103] In one embodiment, the computing device (100) may perform color correction on the extracted mouth area image. For example, the computing device (100) may perform a first color correction to remove color information from the mouth area image and convert it to grayscale (S24). For example, referring to (d) of FIG. 4, the computing device (100) may perform a first color correction to convert it to grayscale to maximize the contrast of the mouth area image in order to obtain more accurate results in a subsequent processing step. For reference, grayscale removes color information from the RGB color model and expresses each pixel as a brightness value (a value between 0 and 255). Here, 0 means black, 255 means white, and intermediate values ​​can be expressed as gray shades. For example, the computing device (100) may extract R, G, and B values ​​for each pixel in the extracted mouth area image. In addition, the computing device (100) can calculate the grayscale value by applying the weighted sum method or the maximum value method to each pixel. In addition, the computing device (100) can generate a "grayscale mouth area image" based on the calculated grayscale value. Meanwhile, the computing device (100) can form a clear boundary between the teeth and the lips by removing the color channel of the mouth area image and converting it into a grayscale image containing only brightness information. In other words, by performing the first color correction to convert to grayscale, the computing device (100) can improve the processing speed by performing simple calculations since only a single channel needs to be processed instead of processing each of the RGB channels in the color image.

[0104] In addition, the computing device (100) may perform a second color correction to adjust the brightness contrast of the mouth area image by utilizing a histogram equalization technique (S25). For example, referring to (e) of FIG. 4, the computing device (100) may perform a second color correction to improve the brightness of the mouth area image on which the first color correction has been performed. The computing device (100) may perform the second color correction to improve the brightness of the grayscale mouth area image by utilizing Contrast Limited Adaptive Histogram Equalization (CLAHE). For reference, CLAHE is one of the histogram equalization techniques and may improve the contrast of an image. CLAHE may perform this for each small area rather than adjusting the contrast of the entire image, thereby preventing excessive global contrast enhancement and effectively improving local contrast. For example, the computing device (100) may divide the grayscale mouth area image converted through step S24 into grids (tiles) of various sizes. This can maintain information locally and enable detailed processing. In addition, the computing device (100) can calculate a histogram for each tile. In addition, the computing device (100) can generate a histogram based on the frequency of pixel values ​​of each tile. In addition, the computing device (100) can readjust the pixel values ​​based on the histogram of each tile. For example, in CLAHE, when adjusting the brightness of each tile, the clip limit value is used to limit the maximum frequency of a specific histogram. Frequencies exceeding this value are shifted to the tail of the distribution, thereby preventing excessive contrast and reducing noise. For example, the clip limit value is an important parameter of CLAHE, and controls the degree of contrast enhancement. A higher value provides a stronger contrast effect, but may generate excessive noise.The computing device (100) can minimize noise while maintaining appropriate contrast by setting the clip limit value to a predetermined value (e.g., 2). The predetermined value can be adjusted according to the characteristics and needs of each image set. Meanwhile, referring to FIG. 5, (a) of FIG. 5 is an image of a mouth area in which only the first color correction has been performed, and (b) of FIG. 5 is an image of a mouth area in which the first and second color corrections have been performed. Referring to FIG. 5 (b), the computing device (100) can adjust the contrast of the grayscale mouth area image by performing the second color correction step, thereby making the boundary between the mouth and teeth more distinct. In addition, this can help the subsequent steps of mouth area extraction and mouth opening calculation to be performed more accurately.

[0105] In addition, the computing device (100) may perform a third color correction to enhance the boundary of the mouth area image by utilizing a binarization technique (S26). For example, referring to (f) of FIG. 4, the computing device (100) may perform a third color correction to clarify the boundary of the mouth area image on which the first and second color corrections have been performed. For reference, binarization refers to a process of dividing pixel values ​​into two classes based on a specific threshold in an image. Generally, pixels brighter than a specified threshold are converted to white (255), and pixels below that are converted to black (0). This allows for emphasizing specific structures or features of the image. For example, the computing device (100) may set a threshold value (thresh_min) to be used for binarization. The computing device (100) may convert pixels brighter than a preset threshold value (e.g., 135) to white (255). The computing device (100) can clarify the bright area by presetting a threshold value and make the shape of the teeth in the binarized image more recognizable. In other words, the computing device (100) can set the color to white (255) if the preset threshold value is greater than or equal to the threshold value, and set the color to black (0) if the value is less than the preset threshold value, thereby forming a clear boundary between the teeth and the background. Through the third color correction, the computing device (100) emphasizes the tooth area in white and processes the background in black to make the distinction clear, thereby making it easier to recognize the shape of the teeth and making the distinction between the mouth and the teeth clear.

[0106] In addition, the computing device (100) can perform a fourth color correction to remove noise from the mouth area image by utilizing a morphological operation technique (S27). For example, the computing device (100) can perform a fourth color correction using a morphological operation technique to remove unnecessary noise from a facial image on which the first to third color corrections have been performed and to better maintain important structures. The computing device (100) can remove small noises around the teeth by utilizing open (open=1) and close (close=1) operations. For example, open removes noise, and close fills small holes, and the computing device (100) can extract a clearer and more accurate boundary of the tooth area by combining these two operations.

[0107] Meanwhile, the computing device (100) can emphasize only the teeth area in the facial image by performing preprocessing on the facial image, thereby generating a teeth contour from the facial image itself with high accuracy.

[0108]

[0109] FIG. 6 is a drawing for explaining an operation of extracting a contour of teeth from a preprocessed facial image according to one embodiment of the present disclosure, and FIG. 7 is a drawing for explaining an operation of distinguishing upper teeth and lower teeth from among filtered contours according to one embodiment of the present disclosure.

[0110]

[0111] According to one embodiment of the present disclosure, a computing device (100) can generate dental data from a preprocessed facial image. The step of generating dental data may specifically include a contour extraction step (S31), a contour filtering step (S32), and an upper and lower teeth distinction step (S33). For reference, the step of generating dental data can identify and analyze an actual dental structure based on information acquired in the image data preprocessing. In addition, the step of generating dental data can extract information necessary to identify and analyze an actual dental structure from a facial image on which color correction has been performed.

[0112] According to one embodiment, the computing device (100) can extract a contour of a tooth from a preprocessed facial image (S31). In addition, the computing device (100) can extract a white area as a candidate contour based on the extracted mouth region image and the extracted facial landmark points. For example, the extracted candidate contour is represented by x, y coordinate values ​​in a two-dimensional planar coordinate system and may include an outline that accurately reflects the shape of the tooth. For reference, the extracted candidate contours are represented in the form of points and can be generated at the same scale as the coordinate system of the facial landmark points generated from the facial image (original image). In addition, the coordinate data can be utilized as basic data for 3D transformation and mouth opening calculation in a subsequent step. For example, referring to (a) of FIG. 6, the white area can be extracted as candidate contour points based on the extracted mouth region image and the extracted facial landmark points. For example, the computing device (100) can track a white area (e.g., a tooth area) based on the facial image preprocessed through steps S21 to S27 and the facial landmark points extracted in step S22, and extract the boundary as candidate contour points. Since the boundary between the teeth and the background has become clear in the mouth area image on which color correction has been performed, the computing device (100) can recognize the outline of the teeth along the boundary and extract candidate contour points. For example, the computing device (100) can use the cv2.findContours function of the OpenCV library to generate candidate contour points. For example, the function can identify and extract boundaries that share the same color or intensity value within the image. When extracting candidate contours, the computing device (100) maintains the hierarchical structure between overlapping objects by using the cv2.RETR_TREE option, and uses the cv2.The CHAIN_APPROX_SIMPLE option can be used to optimize memory usage by removing unnecessary duplicate points and retaining only key points along the boundary. The above is merely an example and the present disclosure is not limited thereto.

[0113] According to one embodiment, the computing device (100) may filter contours corresponding to the tooth region from among the extracted candidate contours (S32). For example, the computing device (100) may remove noise (e.g., light reflected from the tongue) or unnecessary small objects (e.g., small objects around the lips) from among the candidate contours extracted in step S31, leaving only the boundaries of meaningful teeth. By filtering the contours, the computing device (100) may select only contours of a similar size and shape to teeth, thereby increasing the accuracy of the analysis. For example, the computing device (100) may connect candidate contour points to generate a polygon. For example, the computing device (100) may use the 'cv2.approxPolyDP' function to approximate the contour to a polygon and remove unnecessary extracted candidate contour points. The 'epsilon' value in the function determines the accuracy of the approximation. If it is small, the contour may be extracted closer to the original shape of the image, and if it is large, the contour may be extracted in a simplified manner. For example, the computing device (100) may set the epsilon value to 0.00001 to extract detailed boundary line information for the accuracy of the aperture amount. The above is merely an example, and the present disclosure is not limited thereto. In addition, the computing device (100) may calculate the area of ​​candidate contours generated as polygons. In addition, the computing device (100) may analyze the area based on the boundary of the candidate contours generated as polygons. In addition, the computing device (100) may set a minimum value and a maximum value based on the area of ​​the extracted candidate contours. For example, the computing device (100) may set a minimum value (e.g., 10) to remove excessively small objects or noises among the candidate contours generated as polygons, and a maximum value (e.g., 10,000,000) to remove excessively large objects in order to identify contours corresponding to meaningful tooth regions based on the areas of the candidate contours generated as polygons.For example, the set minimum value can be used to remove small noise inside the oral cavity or light reflected on the tongue, and the maximum value can be used to filter a large area such as the entire lips. For example, the minimum and maximum values ​​can be adjusted to cover a wide range of areas because the size of teeth varies depending on the human oral structure or shooting conditions. In addition, the computing device (100) can extract a contour corresponding to the tooth area from among the extracted candidate contours based on the set minimum and maximum values. For example, the computing device (100) can extract a contour corresponding to the tooth area only if the area of ​​the candidate contour generated as a polygon is included in the set minimum and maximum values. In other words, if the area of ​​the candidate contour generated as a polygon is not included in the set minimum and maximum values, the candidate contour can be removed (deleted). For example, referring to (b) of FIG. 6, the computing device (100) can remove unnecessary candidate contour points based on the minimum and maximum values. Additionally, referring to (c) of FIG. 6, the computing device (100) may remove candidate contour points for non-teeth areas, such as lips, chin, skin, and outside of the mouth area, to distinguish between upper and lower teeth. Additionally, the contour corresponding to the extracted tooth area may be a polygonal shape created by connecting candidate contour points.

[0114] According to one embodiment, the computing device (100) may divide the filtered contours into contours corresponding to the upper teeth area or contours corresponding to the lower teeth (S33). For example, the computing device (100) may divide the filtered contours into contours corresponding to the upper teeth area or contours corresponding to the lower teeth by utilizing facial landmark points. For example, the computing device (100) may divide the filtered contours into contours corresponding to the upper teeth area or contours corresponding to the lower teeth by utilizing a first point and a second point among the facial landmark points. For example, the first point may correspond to facial landmark point 13, and the second point may correspond to facial landmark point 14. Facial landmark point 13 may correspond to the center of the upper lip, and facial landmark point 14 may correspond to the center of the lower lip. Additionally, the computing device (100) may distinguish the filtered contours based on the first point and the second point into first contours corresponding to the upper teeth area or second contours corresponding to the lower teeth area. For example, referring to FIG. 7, the computing device (100) may distinguish the filtered contours into first contours corresponding to the upper teeth area when they are close to the first point (e.g., landmark point 13 of the face). Additionally, the computing device (100) may distinguish the filtered contours into second contours corresponding to the lower teeth area when they are close to the second point (e.g., landmark point 14 of the face).

[0115] Alternatively, the computing device (100) may extract the centerline area of ​​the lips by utilizing facial landmark points. For example, the computing device (100) may extract the centerline area of ​​the lips by utilizing a first point and a second point among the facial landmark points. For example, the first point may correspond to facial landmark point 13, and the second point may correspond to facial landmark point 14. Facial landmark point 13 may correspond to the center of the upper lip, and facial landmark point 14 may correspond to the center of the lower lip. In addition, the computing device (100) may divide the filtered contours into first contours corresponding to the upper teeth area or second contours corresponding to the lower teeth area based on the extracted centerline area. For example, referring to FIG. 7, the computing device (100) may distinguish the filtered contours as first contours corresponding to the upper teeth area when the filtered contours are close to a first point (e.g., landmark point 13 of the face). Additionally, the computing device (100) may distinguish the filtered contours as second contours corresponding to the lower teeth area when the filtered contours are close to a second point (e.g., landmark point 14 of the face).

[0116] Meanwhile, the computing device (100) can extract the tooth contour by utilizing the tooth image emphasized through the preprocessing process of the image data. In this process, the computing device (100) can select optimal parameters and options to precisely detect the curvature of the tooth, thereby generating an accurate tooth contour. In addition, the purpose of the contour extraction process is to more precisely detect the tooth boundary in the image, and the computing device (100) can utilize various image filtering and segmentation algorithms to reflect important features of the tooth structure. In addition, the computing device (100) can preferentially remove unnecessary information, such as noise or light reflected from inside the mouth, from the extracted tooth contour in the contour filtering step. This filtering step is to extract valid tooth data that will be used in the final mouth opening calculation. This can increase the reliability of the results of the future mouth opening calculation. In addition, the tooth contour extracted through the filtering process can be divided into upper tooth contour and lower tooth contour by utilizing the face mesh landmark information No. 13 and 14. Through this, the computing device (100) can then calculate the aperture amount more precisely.

[0117]

[0118] According to one embodiment of the present disclosure, the computing device (100) can measure (calculate) the mouth opening amount by utilizing the information generated by the preprocessing and the information on the extracted tooth contour. The step of measuring (calculating) the mouth opening amount may include, in detail, a maximum contour selection step (S41), a tooth 3D transformation step (S42), a scale adjustment step (S43), an origin movement step (S44), a side 2D projection step (S45), an interdental distance calculation step (S46), and a visualization and analysis step (S47). According to one embodiment, the step of measuring the mouth opening amount is a step of measuring the mouth opening amount (interdental distance) between the upper and lower teeth and analyzing it. In addition, the computing device (100) can convert the extracted tooth contour into a three-dimensional space and compare it through scale adjustment and origin movement in the step of measuring the mouth opening amount. In addition, the computing device (100) can calculate the mouth opening amount by projecting it into two dimensions from the side.

[0119] FIG. 8 is a diagram exemplarily showing a result of merging first contours and second contours distinguished from extracted facial landmark points according to an embodiment of the present disclosure, and FIG. 9 is a diagram exemplarily showing a result of selecting upper teeth contours and lower teeth contours according to an embodiment of the present disclosure. For reference, (a) of FIG. 8 is a result of merging first contours and second contours distinguished from landmark points of the face of a first user, and (b) of FIG. 8 is a result of merging first contours and second contours distinguished from landmark points of the face of a second user. In addition, (a) of FIG. 9 is a diagram exemplarily showing a result of selecting upper teeth contours and lower teeth contours from the first contours and second contours of (a) of FIG. In addition, Fig. 9 (b) is a drawing exemplarily showing the result of selecting the upper tooth contour and the lower tooth contour from the first contours and the second contours of Fig. 8 (b). The selected upper tooth contour and the lower tooth contour described through Fig. 9 may include one or more teeth. Meanwhile, for the convenience of explanation, the operation of measuring the opening amount by selecting only one upper tooth contour and one lower tooth contour will be described below.

[0120] According to one embodiment, the computing device (100) can merge the extracted facial landmark points and the distinguished first contours and second contours. For example, referring to FIG. 8, the computing device (100) can merge the facial landmark points generated in the image data preprocessing step with the first contours corresponding to the upper teeth area and the second contours corresponding to the lower teeth area generated in the teeth data generation step. For reference, the facial landmark points are three-dimensional data, and each facial landmark point may have x, y, and z coordinate values. The coordinates for the facial landmark points are values ​​that reflect the original scale of the facial image, and may provide accurate structural information of the face within the facial image. In addition, the distinguished first contours and second contours may be two-dimensional planar data, and each point (point) may have x, y coordinate values. For example, the first contour and the second contour may represent the boundaries of the upper teeth and the lower teeth, respectively, and may refer to the shape of the tooth area extracted from the facial image. In addition, since the facial landmark points and the first contours and the second contours are data generated from the facial image (original image), the computing device (100) can merge the two data sets without separate scale adjustment. In other words, since the coordinates of the facial landmark points and the coordinates of the first contours and the second contours are defined by the same coordinate system on the same image, they can be merged immediately without the need to adjust the size or position between the data. For example, the computing device (100) can merge the facial landmark points and the first contours and the second contours based on the x and y coordinates in a two-dimensional planar coordinate system. Here, the z coordinate of the facial landmark point may be ignored or processed separately as needed.In addition, the landmark points of the face can provide overall shape information of the face, and the first contours and the second contours can provide boundary information of teeth inside the oral cavity. That is, the computing device (100) can generate analysis information that integrates the overall facial structure and tooth boundaries by merging the three-dimensional landmark points of the face and the two-dimensional first contours and the second contours extracted from the same facial image (original image) without scaling.

[0121] According to one embodiment, the computing device (100) can select upper tooth contours and lower tooth contours to be used for calculating the mouth opening amount from the first contours and second contours merged with the landmark points of the face (S41). For example, the first contours represent the tooth boundaries corresponding to the upper tooth area, and the second contours represent the tooth boundaries corresponding to the lower tooth area. For example, referring to (b) of FIG. 8, it can be confirmed that, even though noise is removed by setting a threshold value of the candidate contour area through step S32, noise such as the inside of the mouth reflecting light is not completely removed. The computing device (100) can select upper tooth contours and lower tooth contours to be used for calculating the mouth opening amount from the first contours and second contours to accurately calculate the mouth opening amount. For example, referring to FIG. 9, the computing device (100) can compare the areas of the distinguished first contours. In addition, the computing device (100) may select the contour with the largest area among the first contours as the upper teeth contour. For example, when a plurality of contours with the largest areas among the first contours are selected, the contour that is closest to the first point (e.g., landmark 13) of the extracted facial landmark may be selected as the upper teeth contour. The computing device (100) may select the contour that is closest to the first point (e.g., landmark 13) of the extracted facial landmark as the upper teeth contour by calculating the Euclidean distance between each point of the first contours and the first points. That is, the computing device (100) may select the contour that has the largest area among the identified first contours and is closest to the first point of the extracted facial landmark as the upper teeth contour. In addition, the computing device (100) may compare the areas of the identified second contours. Additionally, the computing device (100) can select the contour with the largest area among the second contours as the lower teeth contour.For example, when multiple contours with the largest area are selected from among the second contours, the contour closest to the second point (e.g., landmark 14) of the extracted facial landmark may be selected as the lower teeth contour. The computing device (100) may select the contour closest to the second point (e.g., landmark 14) of the extracted facial landmark as the lower teeth contour by calculating the Euclidean distance between each point of the second contours and the second points. That is, the computing device (100) may select the contour with the largest area among the identified second contours and closest to the second point of the extracted facial landmark as the lower teeth contour. Meanwhile, the computing device (100) may preferentially select contours with the largest area in order to reflect the tooth structure, which is important in calculating the mouth opening amount. In other words, the upper tooth contour may be one or more of the upper teeth with the largest area closest to landmark 13, and the lower tooth contour may be one or more of the lower teeth with the largest area closest to landmark 14.

[0122] Alternatively, the upper tooth contour may be one or more upper teeth having the largest area among the first contours, and the lower tooth contour may be one or more upper teeth having the largest area among the second contours. For example, the selected upper tooth contour and lower tooth contour may include one or more teeth. For example, if there are multiple first contours having the same area and closest to landmark 13, the computing device (100) may select multiple upper tooth contours. Additionally, if there are multiple second contours having the same area and closest to landmark 14, the computing device (100) may select multiple lower tooth contours.

[0123] FIG. 10 is a diagram exemplarily showing the result of converting a selected upper tooth contour and a selected lower tooth contour from two-dimensional coordinates to three-dimensional coordinates according to one embodiment of the present disclosure, and FIG. 11 is a diagram exemplarily showing the result of performing scale adjustment and origin shift for calculating an opening amount according to one embodiment of the present disclosure. For reference, (a) of FIG. 10 may be the result of converting the selected upper tooth contour and the selected lower tooth contour of FIG. 9 (a) from two-dimensional coordinates to three-dimensional coordinates. In addition, (b) of FIG. 10 may be the result of converting the selected upper tooth contour and the selected lower tooth contour of FIG. 9 (b) from two-dimensional coordinates to three-dimensional coordinates. In addition, (a) of FIG. 11 may be the result of performing scale adjustment and origin shift of FIG. 10 (a), and (b) of FIG. 11 may be the result of performing scale adjustment and origin shift of FIG. 11 (b).

[0124] According to one embodiment, the computing device (100) can convert the selected upper teeth contour and the selected lower teeth contour from two-dimensional coordinates to three-dimensional coordinates (S42). The selected upper teeth contour and the selected lower teeth contour are planar data that only include x, y coordinates and do not include information on the depth (z-axis) of the teeth. In addition, when a person opens his / her mouth so that the teeth are visible, the extreme end (inside the mouth) of the lips (soft tissue) surrounds the gums (hard tissue), making it difficult to measure an accurate mouth opening amount. Considering this, the computing device (100) can convert from two-dimensional coordinates to three-dimensional coordinates in order to more accurately calculate the actual mouth opening amount by reflecting the information on the depth (z-axis) of the teeth. For example, the computing device (100) can determine the value of the z-axis based on the lip landmark that is closest to the selected upper teeth contour or lower teeth contour among landmark points of the face. For example, the computing device (100) may determine the z-axis value of the selected upper teeth contour as the z-value of the lip landmark that is closest to the selected upper teeth contour from among the extracted facial landmark points. In addition, the computing device (100) may determine the z-axis value of the selected lower teeth contour as the z-value of the lip landmark that is closest to the selected lower teeth contour from among the extracted facial landmark points. For example, the computing device (100) may determine the z-axis value of the selected upper teeth contour by calculating the Euclidean distance between the points of the selected upper teeth contours and the facial landmark points. In addition, the computing device (100) may determine the z-axis value of the selected lower teeth contour by calculating the Euclidean distance between the points of the selected lower teeth contours and the facial landmark points. Referring to FIG. 10 as an example, (a) of FIG. 10 is the result of converting (a) of FIG. 9 from two-dimensional coordinates to three-dimensional coordinates, and (b) of FIG. 10 is the result of converting (b) of FIG. 9 from two-dimensional coordinates to three-dimensional coordinates.The converted three-dimensional coordinate information can express the upper and lower teeth contours in three dimensions more precisely, and can even include depth information of the tooth structure.

[0125] According to one embodiment, the computing device (100) may adjust the scale of the coordinates of the third point and the fourth point and the coordinates of the selected upper teeth contour and lower teeth contour converted into three-dimensional coordinates so that the distance between the third point and the fourth point among the extracted facial landmark points becomes a preset value (S43). For example, the computing device (100) may adjust the scale of the coordinates of the merged and three-dimensional facial landmark and the upper teeth contour and lower teeth contour in order to quantitatively compare the mouth opening amount for the same person. For example, the computing device (100) may normalize the dental data based on the third point (e.g., left tear duct no. 133) and the fourth point (e.g., right tear duct no. 362) among the extracted facial landmark points, thereby enabling the comparison of the mouth opening amount between different images for the same person. Referring to FIG. 11 as an example, the computing device (100) can adjust the scale of the coordinates of the third point and the fourth point and the coordinates of the selected upper and lower teeth contours converted into three-dimensional coordinates so that the distance between the third point (e.g., 133) and the fourth point (e.g., 362) becomes a preset value (e.g., 1). The distance between the third point and the fourth point can be an important reference for measuring the size of the face or the scale of the image. By adjusting the distance between the third point and the fourth point to become a preset value, the computing device (100) can convert the face of the same person in different images to the same scale, thereby maintaining consistency in the size and position of the teeth. In addition, by simultaneously adjusting the coordinates of the selected upper and lower teeth contours and the coordinates of the third and fourth points among the extracted facial landmark points, an accurate comparison of the mouth opening amount is possible.

[0126] According to one embodiment, the computing device (100) can perform a full coordinate translation so that the coordinate of the third point among the extracted facial landmark points becomes the origin (S44). The computing device (100) can move the third point (e.g., point 133) to the origin (0,0,0) based on the scaled coordinate system. Through this, all coordinates can be converted into a standardized three-dimensional space. In step S43, the scaled facial landmark points and the coordinates of the selected upper and lower tooth contours can be moved based on the third point. Through this, the three-dimensional coordinates of the upper and lower tooth contours and the coordinates of the facial landmark points can be aligned to the standardized three-dimensional space. Through the coordinate translation, a consistent coordinate system can be maintained between different images of the same person, which allows for accurate comparison of the tooth structure and mouth opening across multiple images.

[0127]

[0128] FIG. 12 is a drawing visualizing the result of selecting the end points of the upper teeth contour and the lower teeth contour to be used for calculating the mouth opening amount according to one embodiment of the present disclosure, and FIG. 13 is a drawing exemplarily showing the result of calculating the minimum distance between the upper teeth contour and the lower teeth contour using projected data according to one embodiment of the present disclosure. For reference, (a) of FIG. 13 may be the result of calculating the minimum distance between the upper teeth contour and the lower teeth contour of (a) of FIG. 11. In addition, (b) of FIG. 13 may be the result of calculating the minimum distance between the upper teeth contour and the lower teeth contour of (b) of FIG. 11. In other words, (a) of FIG. 13 is the result of calculating the mouth opening amount of a first user, and (b) of FIG. 13 is the result of calculating the mouth opening amount of a second user.

[0129] According to one embodiment, the computing device (100) can measure the mouth opening amount by projecting the upper teeth contour and lower teeth contour converted into three-dimensional coordinates onto a two-dimensional plane with respect to the z-axis (S45). In addition, the computing device (100) can measure the mouth opening amount by projecting the upper teeth contour and lower teeth contour converted into three-dimensional coordinates and scaled onto a two-dimensional plane with respect to the z-axis. For example, the upper teeth contour and lower teeth contour converted into three-dimensional coordinates in steps S42 to S44 have x, y, and z coordinate values, and the computing device (100) can convert this into a two-dimensional coordinate system by projecting it onto the yz plane or the xz plane with respect to the z-axis. The reason for projecting with respect to the z-axis is that it is an important axis for measuring the degree of mouth opening when calculating the mouth opening amount. Additionally, when projecting onto the yz plane, the x-axis information is removed and the computing device (100) can measure the aperture using only the y-axis and z-axis information.

[0130] According to one embodiment, the computing device (100) can measure the opening amount based on the selected upper teeth contour and the selected lower teeth contour (S46). In addition, the computing device (100) can calculate the minimum distance between the upper teeth contour and the lower teeth contour by utilizing the projected data. For example, the computing device (100) can calculate the minimum distance between the upper teeth contour and the lower teeth contour projected onto a two-dimensional plane. The upper teeth contour and the lower teeth contour may be projected onto a two-dimensional plane (e.g., yz plane) in step S45. The computing device (100) can find the two closest points between the upper teeth and the lower teeth and calculate the minimum distance. For example, referring to FIG. 12, the computing device (100) may first group and select points that are closest to the outside of the mouth (e.g., the lowest z-axis value) in the upper and lower tooth contours to calculate an accurate mouth opening amount. In addition, the computing device (100) may select a point (point) that is closest to the lower tooth contour, that is, a point with a minimum y value, within the group of selected upper tooth contour points. In addition, the computing device (100) may select a point that is closest to the upper tooth contour, that is, a point with a minimum y value, within the group of selected lower tooth contour points. In addition, the computing device (100) may calculate a Euclidean distance between the selected upper tooth contour points (points) and lower tooth contour points (points). For example, the computing device (100) may first select a point that is closest to the outside of the mouth (i.e., a point with a lowest z-axis value) in the upper and lower tooth contours to calculate an accurate mouth opening amount. From the group of upper and lower tooth contour points selected in this way, the point with the y value closest to the lower teeth can be selected for the upper teeth, and the point with the y value closest to the upper teeth can be selected for the lower teeth, and the Euclidean distance between these two points can be calculated.For example, referring to (a) of FIG. 13, the computing device (100) can calculate the distance between a line (Distance Line) connecting an upper tooth contour point (Upper Point) close to the lower tooth contour in the selected upper teeth contour and a lower tooth contour point (Lower Point) close to the upper teeth contour in the selected lower teeth contour, thereby performing calculation of the mouth opening amount using the distance between teeth. (b) of FIG. 13 can also measure the mouth opening amount using the same method.

[0131]

[0132] According to one embodiment of the present disclosure, the computing device (100) can perform verification to determine the accuracy of the measured opening. First, the computing device (100) can obtain a verification image. The verification image can include a facial image taken while biting an actual 1 cm, 2 cm, or 3 cm stick so that the teeth are visible. In addition, the computing device (100) can perform a comparison between the actual distance (true value) and the calculated distance (e.g., the distance predicted in step S46). For example, the absolute percentage error (APE), the mean absolute percentage error (MAPE), and the accuracy can be utilized for the accuracy calculation and comparison. The calculation method for the absolute percentage error (APE), the mean absolute percentage error (MAPE), and the accuracy can be expressed as in (Mathematical Formula 1), (Mathematical Formula 2 2), and (Mathematical Formula 3).

[0133] [Mathematical Formula 1]

[0134]

[0135] [Equation 2]

[0136]

[0137] [Equation 3]

[0138]

[0139] For example, the computing device (100) performed accuracy verification by calculating the aperture for a total of 12 verification images. Referring to Table 1, as a result, the absolute percentage error APE2 when the actual aperture is 2 cm was at least 0.00 and at most 8.33, and the absolute percentage error APE3 when the actual aperture is 3 cm was at least 2.30 and at most 9.38. Finally, the accuracy value calculated using the MAPE calculated for each verification image was at least 93.61 and at most 97.84.

[0140] [Table 1]

[0141]

[0142]

[0143] Below, we will briefly review the operating flow of the present invention based on the detailed description above.

[0144] FIG. 14 is a flowchart illustrating a method for measuring an aperture according to one embodiment of the present disclosure.

[0145] The method for measuring the aperture illustrated in FIG. 14 can be performed by the computing device (100) described above. Therefore, even if omitted below, the description of the computing device (100) can be equally applied to the description of the method for measuring the aperture.

[0146] Referring to FIG. 14 as an example, a method for measuring mouth opening may include a step of acquiring a facial image (S110), a step of performing preprocessing on the facial image (S120), a step of extracting a tooth contour from the preprocessed facial image (S130), and a step of measuring the mouth opening using information generated by the preprocessing and information on the extracted tooth contour (S140).

[0147] Step S110 is a step of acquiring a facial image. For example, the computing device (100) may acquire a facial image captured so that at least a portion of the teeth are visible while the user has his or her mouth open.

[0148] Step S120 is a step for performing preprocessing on the above facial image. For example, the computing device (100) may perform preprocessing such as image RGB conversion, facial landmark extraction using a face mesh, mouth region extraction, grayscale conversion, image contrast processing using CLAHE, binarization, and morphology operation processes, with the goal of converting the facial image into a form suitable for analysis.

[0149] Step S130 is a step for extracting a tooth contour from the preprocessed facial image. For example, step S130 is a step for identifying and analyzing an actual tooth structure based on information obtained from image data preprocessing. In this process, the computing device (100) can extract a tooth contour and, based on this, distinguish between upper and lower teeth to generate data. Step S130 is a step for extracting information necessary for identifying and analyzing an actual tooth structure from a binarized image obtained through the image data preprocessing step. The above step S130 can be divided into a total of three processes: contour extraction, contour filtering, and distinguishing between upper and lower teeth.

[0150] Step S140 is a step for measuring the mouth opening by utilizing the information generated by the above preprocessing and the information on the extracted tooth contour. For example, step S140 is a step for accurately measuring and analyzing the mouth opening (interdental distance) between the upper and lower teeth. Step S140 converts the extracted tooth contour into 3D space and enables comparison through scale adjustment and origin movement. In addition, the computing device (100) can calculate the mouth opening by projecting it from the side into 2D. Step S140 may include a geometric transformation for calculating the mouth opening from the tooth data.

[0151]

[0152] The steps described in the above description may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Furthermore, some steps may be omitted as needed, and the order of the steps may be changed.

[0153]

[0154] Meanwhile, a computer-readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

[0155] A computer-readable medium storing a data structure according to one embodiment of the present disclosure is disclosed. The aforementioned data structure can be stored in a storage unit in the present disclosure, executed by a processor, and transmitted and received by a communication unit.

[0156] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification. A data structure can refer to the organization of data to solve specific problems (e.g., data analysis, data retrieval, data storage, data modification). A data structure can also be defined as the physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements can include connections between user-defined data elements. Physical relationships between data elements can include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., persistent storage). Specifically, a data structure can include a collection of data, relationships between data, and functions or commands applicable to the data. An effectively designed data structure allows a computing device to perform operations while minimizing the use of its resources. Specifically, a computing device can improve the efficiency of operations, reading, inserting, deleting, comparing, exchanging, and searching through an effectively designed data structure.

[0157] Data structures can be categorized as linear or nonlinear, depending on their form. A linear data structure can be a structure in which only one piece of data is linked to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a series of data sets with an internal order. Lists can also include linked lists. A linked list is a data structure in which data is linked in a single line, each piece having a pointer. In a linked list, a pointer can contain information about the next or previous piece of data. Linked lists can be expressed as singly linked lists, doubly linked lists, or circular linked lists, depending on their form. A stack can be a data listing structure with limited data access. A stack can be a linear data structure in which data operations (e.g., insertion or deletion) can only be performed at one end of the data structure. Data stored in a stack can be a Last-in-First-out (LIFO) data structure. A queue is a data structure with limited access to data. Unlike a stack, it can be a first-in, first-out (FIFO) data structure, with later data being retrieved later. A deck can be a data structure that can process data at both ends.

[0158] A nonlinear data structure can be a structure in which multiple pieces of data are connected behind a single piece of data. Nonlinear data structures can include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structures can include tree data structures. A tree data structure can be a data structure in which there is only one path connecting two different vertices among multiple vertices included in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

[0159] Throughout this specification, the terms artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they are collectively referred to as neural networks. A data structure may include a neural network. And, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include preprocessed data for processing by a neural network, data input to a neural network, neural network weights, neural network hyperparameters, data obtained from a neural network, activation functions associated with each node or layer of a neural network, loss functions for neural network learning, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination of preprocessed data for processing by a neural network, data input to a neural network, neural network weights, neural network hyperparameters, data obtained from a neural network, activation functions associated with each node or layer of a neural network, loss functions for neural network learning, etc. In addition to the aforementioned configurations, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated in the computational process of the neural network, and is not limited to the aforementioned. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0160] The data structure may include data input to a neural network. The data structure including the data input to the neural network may be stored on a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and / or input data input to the neural network after training has been completed. The data input to the neural network may include data that has undergone preprocessing and / or data that is the target of preprocessing. Preprocessing may include a data processing process for inputting data to the neural network. Accordingly, the data structure may include data that is the target of preprocessing and data generated by the preprocessing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0161] The data structure may include weights of the neural network. (In this specification, the terms "weight" and "parameter" may be used interchangeably.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weights may be variable and may be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine a data value output from the output node based on the values ​​input to the input nodes connected to the output node and the weights set for the links corresponding to each input node. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0162] By way of example and not limitation, the weights may include weights that vary during the neural network training process and / or weights that have completed neural network training. The weights that vary during the neural network training process may include weights at the start of the training cycle and / or weights that vary during the training cycle. The weights that have completed neural network training may include weights that have completed the training cycle. Accordingly, a data structure including the weights of a neural network may include a data structure including weights that vary during the neural network training process and / or weights that have completed neural network training. Therefore, the above-described weights and / or combinations of each weight are included in the data structure including the weights of a neural network. The above-described data structures are merely examples and the present disclosure is not limited thereto.

[0163] A data structure including neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed and used. A computing device can serialize the data structure to transmit and receive data over a network. The serialized data structure including neural network weights can be reconstructed on the same computing device or another computing device through deserialization. The data structure including neural network weights is not limited to serialization. Furthermore, the data structure including neural network weights can include a data structure that increases computational efficiency while minimizing the use of computing device resources (e.g., a B-Tree, an R-Tree, a Trie, an m-way search tree, an AVL tree, a Red-Black Tree in nonlinear data structures). The foregoing is merely an example, and the present disclosure is not limited thereto.

[0164] The data structure may include hyperparameters of a neural network. Furthermore, the data structure including the hyperparameters of the neural network may be stored on a computer-readable medium. The hyperparameters may be variables that can be varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle repetitions, weight initialization (e.g., setting a range of weight values ​​to be subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0165]

[0166] FIG. 15 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0167] Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and / or other program modules that may be executed on one or more computers and / or as a combination of hardware and software.

[0168] Generally, program modules include routines, programs, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be implemented with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

[0169] The described embodiments of the present disclosure can also be practiced in distributed computing environments, where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0170] Computers typically include a variety of computer-readable media. Computer-readable media can be any media that can be accessed by a computer, and includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

[0171] Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

[0172] An exemplary environment for implementing various aspects of the present disclosure is illustrated, including a computer (1102), which includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

[0173] The system bus (1108) may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0174] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA) - which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0175] These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of the computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art will appreciate that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

[0176] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and / or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0177] A user may enter commands and information into the computer (1102) via one or more wired / wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is connected to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

[0178] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0179] The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), although for simplicity, only the memory storage device (1150) is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) (1152) and / or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

[0180] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means of establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules or portions thereof described for the computer (1102) may be stored in a remote memory / storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

[0181] The computer (1102) operates to communicate with any wireless device or object that is arranged and operates via wireless communication, such as a printer, a scanner, a desktop and / or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

[0182] Wi-Fi (Wireless Fidelity) enables connections to the Internet and other devices without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data anywhere within the coverage area of ​​a base station, both indoors and outdoors, similar to cell phones. Wi-Fi networks use wireless technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual-band).

[0183] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0184] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0185] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0186] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0187] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

[0188] As described above, the relevant contents have been described in the best form for carrying out the invention.

Claims

1. A method for measuring an aperture, performed by a computing device, Step of acquiring a facial image; A step of performing preprocessing on the above facial image; A step of extracting a tooth contour from the above preprocessed facial image; and A step of measuring the opening amount by utilizing the information generated by the above preprocessing and the information on the extracted tooth contour; Including, method.

2. In paragraph 1, The step of performing preprocessing on the above facial image is as follows: A step of extracting facial landmark points based on the facial image using an artificial intelligence model; A step of extracting a mouth area from the facial image based on the extracted facial landmark points; and Step of performing color correction on the above extracted mouth area image Including, method.

3. In paragraph 2, The step of performing color correction on the extracted mouth area image is as follows: A first color correction step of removing color information from the above mouth area image and converting it to grayscale; A second color correction step for adjusting the brightness contrast of the above-mentioned mouth area image by using a histogram equalization technique; A third color correction step for enhancing the boundary of the above-mentioned lip region image by using a binarization technique; or A fourth color correction step for removing noise from the image of the above-mentioned lip area by using a morphological operation technique; Containing at least one of: method.

4. In paragraph 2, The step of extracting the tooth contour from the above preprocessed facial image is as follows: A step of extracting a white area as a candidate contour based on the extracted mouth area image and the extracted facial landmark points; and A step of filtering contours corresponding to the tooth area among the candidate contours extracted above. Including, method.

5. In paragraph 4, The step of filtering the contours corresponding to the tooth area among the candidate contours extracted above is: A step of analyzing the area of ​​the above extracted candidate contour; A step of setting minimum and maximum values ​​based on the area of ​​the extracted candidate contour; and A step of extracting a contour corresponding to a tooth area from among the extracted candidate contours based on the minimum and maximum values ​​set above; Including, method.

6. In paragraph 4, The step of extracting the tooth contour from the above preprocessed facial image is as follows: A step of dividing the above filtered contours into contours corresponding to the upper teeth area or contours corresponding to the lower teeth area. Including more, method.

7. In paragraph 6, The step of distinguishing the upper and lower teeth among the above filtered contours is: A step of dividing the filtered contours into first contours corresponding to the upper teeth area or second contours corresponding to the lower teeth area by utilizing the landmark points of the above face; Including, method.

8. In paragraph 7, The step of measuring the above opening is: A step of merging the extracted facial landmark points with the separated first contours and second contours; A step of selecting upper teeth contours and lower teeth contours to be used for calculating the opening amount from the first contours and second contours merged with the landmark points of the above face; and A step of measuring the opening amount based on the selected upper teeth contour and the selected lower teeth contour. Including, method.

9. In paragraph 8, The step of measuring the opening amount based on the selected upper teeth contour and the selected lower teeth contour is as follows: A step of converting the above-mentioned selected upper tooth contour and the above-mentioned selected lower tooth contour from two-dimensional coordinates to three-dimensional coordinates; and A step of measuring the opening amount by projecting the upper teeth contour and lower teeth contour converted into the three-dimensional coordinates onto a two-dimensional plane based on the z-axis; Including, method.

10. In paragraph 8, The step of selecting the upper teeth contour and lower teeth contour to be used for calculating the opening amount from the first contours or second contours merged with the landmark points of the above face is as follows. A step of selecting the contour with the largest area among the above-mentioned first contours and closest to the first point of the extracted facial landmark as the upper teeth contour; and A step of selecting a contour having the largest area among the above-mentioned second contours and closest to the second point of the extracted facial landmark as the lower teeth contour; Including, method.

11. In paragraph 9, The step of converting the above-mentioned selected upper tooth contour and the above-mentioned selected lower tooth contour from two-dimensional coordinates to three-dimensional coordinates is: A step of determining the z-axis value of the selected upper teeth contour as the z-value of the lip landmark closest to the selected upper teeth contour among the extracted facial landmark points; and A step of determining the z-axis value of the selected lower teeth contour as the z-value of the lip landmark closest to the selected lower teeth contour among the extracted facial landmark points; Including, method.

12. In paragraph 9, The step of measuring the opening amount based on the selected upper teeth contour and the selected lower teeth contour is as follows: After the step of converting to the above 3D coordinates: A step of adjusting the scale of the coordinates of the third point and the fourth point and the coordinates of the selected upper tooth contour and lower tooth contour converted into the three-dimensional coordinates so that the distance between the third point and the fourth point among the extracted facial landmark points becomes a preset value. Including more, The step of measuring the opening amount by projecting the upper and lower teeth contours converted into the three-dimensional coordinates onto a two-dimensional plane based on the z-axis is as follows. Including a step of measuring the opening amount by projecting the upper tooth contour and lower tooth contour, which are converted into the three-dimensional coordinates and scaled, onto a two-dimensional plane based on the z-axis. method.

13. In paragraph 9, The step of measuring the opening amount by projecting the upper and lower teeth contours converted into the three-dimensional coordinates onto a two-dimensional plane based on the z-axis is as follows. A step of calculating the minimum distance between the upper teeth contour and the lower teeth contour by utilizing the above projected data. Including, method.

14. A computer program stored in a computer-readable storage medium, wherein when the computer program is executed on one or more processors, the computer program causes the one or more processors to perform the following operations for measuring an aperture, the operations being: The act of acquiring a facial image; An action for performing preprocessing on the above facial image; An operation of extracting a tooth contour from the above preprocessed facial image; and An operation of measuring the opening amount by utilizing the information generated by the above preprocessing and the information on the extracted tooth contour; Including, A computer program stored on a computer-readable storage medium.

15. As a computing device, at least one processor; and memory; Including, At least one processor of the above, Obtain a facial image; Perform preprocessing on the above facial image; Extracting the contour of the teeth from the above preprocessed facial image; and It is configured to measure the opening amount by utilizing the information generated by the above preprocessing and the information on the extracted tooth contour. device.

Images