Quantitative evaluation method based on the amount of movement of facial soft tissue

Abstract

This facial soft tissue evaluation method comprises: a step for capturing and recording moving images of facial motions of a subject using a three-dimensional moving image recording device 10, and creating homologous models 32 obtained by normalizing each frame data of three-dimensional facial moving images transmitted from the three-dimensional moving image recording device 10 by arithmetic processing by an analysis device 20; a step for identifying a plurality of landmarks in the respective homologous models 32; and a step for measuring movement quantities caused by the facial motions of the respective identified landmarks, and evaluates the motor function of facial soft tissue of the subject on the basis of the movement quantities.

Description

Technical Field

[0001] The present invention relates to a method and a system for quantitatively evaluating the state of the facial soft tissue or the facial motor function of a subject by performing detailed data analysis on the amount of movement and morphological changes of the soft tissue during facial movements such as smiling expression and chewing and swallowing.

Background Art

[0002] A person's face has a strong influence on obtaining psychological satisfaction that one is socially accepted. Also, facial expressions play an important function as a non-verbal communication means for transmitting emotions and thoughts in social life. In modern orthodontic treatment, from a social-psychological perspective, improving the morphology of the soft tissue of the face including expressions is recognized as one of the important treatment objectives.

[0003] Conventionally, as a method for quantitatively evaluating the morphology and motor function of the face, facial measurement using photographs is common and is also clinically applied. In recent years, facial evaluation by three-dimensional measurement using a scanning method or a stereo method has also been studied. For example, in the case of a patient with an expression disorder, it is known that a morphological distortion peculiar to the disease occurs in the face during a smiling expression movement. The inventors have already reported that three-dimensional measurement of the face at rest and during expression is effective for detecting such distortion (see, for example, Non-Patent Document 1).

[0004] In addition, orthodontic treatment aims not only at improving the appearance of facial expressions and occlusion, but also at restoring and improving the chewing function for eating. Note that chewing is performed by a series of coordinated movements of the masticatory muscles and facial muscles including the facial muscles in the facial soft tissue.

[0005] For example, in cases of skeletal crossbite, hypoplasia of the surrounding masticatory muscles and facial expression muscles is observed along with hypoplasia of the maxilla. In cases of anterior open bite, enhanced compensatory muscle activity in response to incomplete lip closure is observed. From these cases, it is expected that in patients with occlusal abnormalities, the patterns of dynamic displacement of the facial soft tissues during chewing are significantly different from those of normal individuals, and methods for objectively evaluating chewing function based on mandibular movement trajectories and activities of the oro-facial muscles have also been investigated (see, for example, Non-Patent Document 2).

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] In conventional facial measurements, a method has been adopted in which a plurality of anatomical feature points (landmarks) are found on the face of the subject, and markers are placed at the positions of those landmarks to photograph the face. However, there are at least 30 or more landmarks required for human facial measurement, and the task of marking while individually identifying such a large number of landmarks is not only specialized but also complicated and extremely time-consuming. In addition, the facial soft tissues in areas where no markers were installed could not be evaluated.

[0008] An object of the present invention is to provide a technique capable of quantitatively evaluating the state or motor function of a subject's facial soft tissue without using a marker.

Means for Solving the Problems

[0009] To solve the above problems, the present invention provides a method for evaluating a subject's facial soft tissue, comprising: using a three-dimensional video recording device to capture and record a moving image of the subject's facial movement; and the arithmetic processing by an analysis device creating a homologous model by normalizing each frame data of the moving image transmitted from the three-dimensional video recording device; identifying a plurality of landmarks in each of the homologous models; and measuring the respective movement amounts of the identified landmarks accompanying the facial movement, and evaluating the motor function of the subject's facial soft tissue based on the movement amounts.

[0010] The method for evaluating facial soft tissue preferably includes a step of the arithmetic processing by the analysis device calculating a jerk cost of the movement trajectory of each of the landmarks identified in each of the homologous models, and evaluating the movement smoothness of the subject's facial soft tissue based on the calculated jerk cost.

[0011] Further, the method for evaluating facial soft tissue includes a step of the arithmetic processing by the analysis device constructing a spring lattice model in which the form of each of the homologous models is simulated by a polyhedron, each of a plurality of vertices of the polyhedron corresponding to a coordinate point of any of the plurality of landmarks, and an edge connecting the vertices of the polyhedron being replaced by a spring element; and a step of calculating spring energy generated in each of the spring elements when the facial movement is reproduced by the movement of the spring lattice model, and preferably evaluating the movement flexibility of the subject's facial soft tissue based on the calculated spring energy.

[0012] Further, in the facial soft tissue evaluation method, it is preferable that the facial movement is a smile expression movement or a chewing movement.

Advantages of the Invention

[0013] According to the present invention, for example, by measuring the three-dimensional movement amounts of facial soft tissues during facial movements such as the expression of a smile and chewing over time, it is possible to quantitatively evaluate the state and motor function of facial soft tissues in a relatively wide range. Further, since it is not necessary to place markers on the face of the subject, facial measurement can be easily performed in a short time even by a non-expert.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 5A

Figure 5B

Figure 5C

Figure 6

Figure 7A

Figure 7B

[0015] FIG. 1 shows a schematic configuration of a facial soft tissue evaluation system according to an embodiment of the present invention. This system includes a three-dimensional video recording device 10 and an analysis device 20. As the three-dimensional video recording device 10, for example, a product named "3dMD dynamic SYSTEM" provided by 3dMD, USA can be used. The analysis device 20 is a computer system connected to the three-dimensional video recording device 10 via a communication line, and includes a homologous model creation means 21, a landmark identification means 22, a soft tissue movement amount measurement means 23, a motor function evaluation means 24, and the like. The motor function evaluation means 24 further includes a jerk cost calculation means 25, a spring energy conversion means 26, and the like. These means in the analysis device 20 are arithmetic processing means for realizing the target functions by arithmetic processing of a computer.

[0016] A method for quantitatively evaluating the facial soft tissue of a subject using the system of FIG. 1 will be specifically described below.

[0017] First, using the three-dimensional video recording device 10, while causing the subject to perform facial movements accompanied by changes in expression, such as a smiling expression movement or chewing and swallowing movements, a three-dimensional moving image of the facial head is captured. The moving image data recorded in the three-dimensional video recording device 10 is each converted into time-series frame data 31 having three-dimensional form information and transmitted to the analysis device 20. In one embodiment, the facial movement of the subject is captured at 10 frames per second, and the origin of the coordinate system is set based on the frame data 31 in the resting state at the 20th frame (about 2 seconds after the start of shooting).

[0018] Based on the three-dimensional facial moving image (time-series set of frame data) of the subject transmitted from the three-dimensional video recording device 10, the analysis device 20 automatically measures the amount of movement and morphological changes of the soft tissue during facial movement by the method described below.

[0019] The set of frame data 31 captured by the three-dimensional video recording device 10 represents the three-dimensional morphological changes during the facial movement of the subject in time series. The homologous model creation means 21 performs a process of creating a three-dimensional facial form model 32 normalized from each frame data 31. Specifically, as shown in FIG. 2, the homologous model creation means 21 extracts anatomical feature points (landmarks, semi-landmarks) from the facial form model 311 of each frame data 31, and relocates those feature points according to the scale of the standard template model 312 having the same number of points and the same topological geometric structure, thereby constructing a normalized homologous model 32. For this modeling process, for example, the HBM (Homologous Body Modeling) program provided by AIST (National Institute of Advanced Industrial Science and Technology) can be used.

[0020] The landmark identification means 22 performs a process of identifying landmarks, which are a plurality of representative anatomical feature points common to each homologous model 32. Here, FIG. 3 illustrates a time series of the homologous model 33 in which a plurality of representative landmarks are identified.

[0021] In addition, FIGS. 4A to 4C illustrate the results of comparing the measured values using markers with the identified values according to the present embodiment for representative landmarks Nos. 1 to 38. In these box-and-whisker plots, the vertical axis indicates the errors in the x-axis (left-right direction), y-axis (up-down direction), and z-axis (front-back direction) with respect to the measured values using markers (gold standard, i.e., correct values). As shown in FIGS. 4A to 4C, the errors with respect to the marker reference are within 2 mm for most landmarks, and it was confirmed that the accuracy of the facial soft tissue movement tracking method according to the present embodiment is good.

[0022] The soft tissue movement amount measuring means 23 performs a process of measuring the respective movement amounts accompanying the facial movement for each landmark identified in the homologous model 33. Then, the motor function evaluating means 24 quantitatively evaluates the actual state and motor function of the subject's facial soft tissue by calculating kinetic variables (such as jerk cost, peak velocity, movement time, etc.) based on the movement amounts of the respective landmarks.

[0023] A specific example of the method for evaluating facial soft tissue performed in the above-described system will be described.

[0024] (Example 1) An example of evaluating facial soft tissue during a smiling expression showing movement will be described. First, using the three-dimensional video recording device 10, a three-dimensional facial moving image is captured from the state where the subject's face is at rest until the subject shows a smiling expression and returns to the rest state again, and this step is repeated 7 times.

[0025] Next, in the analysis device 20, by installing landmarks and semi-landmarks on each frame 31 of the three-dimensional facial moving image, a time-series set of the homologous model 32 in the smiling expression showing movement is created. The data of the created homologous model group 32 is recorded in the database of the analysis device 20.

[0026] Subsequently, in each homologous model 32, 38 representative anatomical feature points (landmarks) required for measurement are identified. Then, the three-dimensional coordinate values of each individual landmark are plotted on the time axis, and by appropriately performing interpolation processing between frames, the three-dimensional movement trajectory of each landmark is obtained.

[0027] Then, the jerk cost calculation means 25 performs a process of calculating the jerk cost of the movement trajectory of each landmark identified in each homologous model 32. Here, the jerk cost is the sum of the squares of the second derivative of the movement trajectory (i.e., acceleration). The higher the value of the jerk cost, the more awkward the movement indicates, and the lower the value of the jerk cost, the smoother the movement indicates. That is, the movement smoothness of the facial soft tissue can be quantitatively evaluated by the value of the jerk cost. FIGS. 5A to 5C show examples of the change in the movement amount of the landmark and the total jerk cost JC during the expression movement of a smiling expression measured by the tracking method of this embodiment.

[0028] In addition to the jerk cost, the movement function evaluation means 24 can quantitatively evaluate the movement functionality of the facial soft tissue of the subject based on variables such as the movement speed, peak speed, and movement time of each landmark.

[0029] Further, the analysis device 20 may construct a spring lattice model (unstructured lattice model) 41 that simulates the shape of the facial soft tissue of the subject with a polyhedron from each homologous model 33 recorded in the database. For example, in the example of FIG. 6, each vertex (lattice point) of the polyhedron of this spring lattice model 41 coincides with the coordinate points of the landmarks, which are, for example, the above-described 38 anatomical feature points. In addition, the sides connecting the vertices (lattice points) of the polyhedron constituting the spring lattice model 41 are replaced with spring elements. The natural length of the spring element connecting two landmarks is determined based on the three-dimensional facial image obtained in the resting state immediately before the expression movement of a smile.

[0030] The spring energy conversion means 26 of the analysis device 20 reproduces the facial movement of the subject, that is, the movement from the state where the face is at rest to expressing a smiling face and then back to the rest state, using the spring lattice model 41, and performs a process of converting the movement of the soft tissue at that time into the variation of the spring energy generated in each spring element. In dynamics, the energy E generated in a spring is proportional to the square of the amount of expansion and contraction of the spring (L m -L0), as expressed by the following formula (1). Spring energy E = 1 / 2×k×(L0-L m ) 2 ···(1) Here, k: spring constant L0: natural length of the spring (initial value of the lattice point distance) L m : length of the spring after movement (lattice point distance) However, in formula (1), although the absolute amount of the spring energy E can be obtained, it is impossible to distinguish whether the soft tissue replaced by the spring is stretched or contracted. Therefore, for example, when the soft tissue is stretched (L0-L m <0), it is preferable to distinguish the expansion and contraction state by attaching a minus sign to the absolute amount of the spring energy E in formula (1). Also, it is preferable to visually display the movement state of the facial soft tissue by changing the color according to the variation amount of the spring energy E.

[0031] In this way, by converting the movement of the facial soft tissue of the subject into the variation of the spring energy E generated in each spring element of the spring lattice model 41, the movement flexibility of the facial soft tissue of the subject can be quantitatively evaluated.

[0032] Note that when creating the homologous model 32, a spring lattice model 42 modeled by a polygon mesh including sub-landmarks that are finer than the landmarks may be created to perform spring energy analysis of the facial soft tissue (see FIGS. 7A and 7B). According to this embodiment, the state and movement function of the facial soft tissue in a relatively wide range can be evaluated in more detail.

[0033] (Example 2) Next, an example of facial soft tissue evaluation during chewing movement will be described. First, using the three-dimensional video recording device 10, a three-dimensional facial moving image is taken of the subject with the face in a resting state for 5 seconds, air chewing for 30 seconds, and chewing chewing gum for 30 seconds. As a result, frame data 31R at rest, frame data 31E during air chewing, and frame data 31C during chewing chewing gum are obtained.

[0034] Next, the homologous model creation means 21 extracts anatomical feature points (landmarks, semi-landmarks) from each frame data 31R, 31E, 31C, and creates homologous model time series groups 32R, 32E, 32C during each movement. The data of the created homologous model groups 32R, 32E, 32C are recorded in the database of the analysis device 20.

[0035] Subsequently, the landmark identification means 22 identifies landmarks, which are a plurality of representative anatomical feature points common to each homologous model 32R, 32E, 32C. The soft tissue movement amount measuring means 23 performs a process of measuring the movement amount of each landmark during air chewing and chewing movement. Then, the motor function evaluation means 24 plots the three-dimensional coordinate values of each individual landmark on the time axis, and by appropriately performing interpolation processing between frames, obtains the three-dimensional movement trajectory of each landmark, and can calculate kinetic variables (for example, jerk cost, peak velocity, movement time, etc.) from the obtained data.

[0036] For example, the jerk cost calculation means 25 can perform a process of calculating the jerk cost of the movement trajectory of each landmark during air chewing and chewing movement. Further, the spring energy conversion means 26 may reproduce the movement of each landmark during air chewing and chewing movement with the spring lattice model 41, and perform a process of converting the movement of the soft tissue at that time into a change in spring energy generated in each spring element. By statistically comparing the differences in changes between each movement using the multidimensional feature vector having the above kinetic variables as elements, it is possible to contribute to the establishment of a method for quantitatively evaluating, for example, the decline in chewing function of the elderly, the degree of lip closure insufficiency, and the effect and recovery degree by rehabilitation.

[0037] According to the embodiments described above, even without installing markers on the face of the subject, the state or motor function of the facial soft tissue of the subject can be quantitatively evaluated. Since there is no need to install markers, facial measurement can be easily performed in a short time even by a non-expert.

[0038] More specifically, for example, in orthodontics, it can be used as an effective evaluation means for formulating treatment plans and confirming treatment effects. For example, by dynamically measuring the three-dimensional facial morphology from a resting state to a smiling expression and analyzing in detail the movement amount of the facial soft tissue including the facial muscles, morphological distortions specific to the disease can be detected. Further, by quantifying such morphological distortions, it can also be used for diagnosis such as case classification and disease severity.

[0039] Also, under the recognition that facial expressions are important non-verbal communication means in social life, it can be used, for example, as a tool for practicing making a good smile or confirming the effect of rehabilitation training. Furthermore, if this technology can be applied to a portable camera such as a smartphone, for example, facial expressions can be observed daily at home, and the practical application of an "expression diagnosis system" that supports early detection of depression, dementia, etc. and accurate medical intervention for such diseases can also be expected.

[0040] Also, by evaluating the facial motor function during chewing and swallowing movements, for example, the chewing and swallowing functions and the degree of recovery of patients who have undergone orthodontic treatment can be grasped. Also, for example, it is possible to predict and evaluate the degree of decline in the swallowing function of frail elderly people.

Explanation of Reference Numerals

[0041] 10 Three-dimensional video recording device 20 Analysis device 21 Homogeneous model creation means 22 Landmark identification means 23 Soft tissue movement amount measurement means 24 Motion function evaluation means 25 Jerk cost calculation means 26 Spring energy conversion means 31 Frame data of three-dimensional facial motion images 32 Homologous model with normalized three-dimensional facial form 33 Homologous model with representative landmarks identified 311 Three-dimensional facial form model 312 Standard template model 41, 42 Spring lattice model simulating three-dimensional facial form

Claims

1. A method for evaluating the facial soft tissue of a subject, comprising: using a three-dimensional video recording device to capture and record a moving image of the facial movement of the subject, wherein the arithmetic processing by an analysis device creates respective homologous models by normalizing each frame data of the moving image transmitted from the three-dimensional video recording device; identifying a plurality of predetermined landmarks in each of the homologous models; measuring the respective amounts of movement associated with the facial movement for each of the plurality of identified landmarks; constructing a spring lattice model in which the form of each of the homologous models is approximated by a polyhedron, wherein each of the plurality of vertices of the polyhedron corresponds to a coordinate point of any of the plurality of landmarks, and the edges connecting the vertices of the polyhedron are replaced by spring elements; calculating the variation in the spring energy generated in each of the spring elements when the facial movement is reproduced by the movement of the spring lattice model; and a method including the above steps.

2. The method for evaluating facial soft tissue according to claim 1, wherein the facial movement is a smiling expression movement.

3. The method for evaluating facial soft tissue according to claim 1, wherein the facial movement is a chewing movement.

Images