Face motion evaluation system and rehabilitation support system

Abstract

To disclose a face motion evaluation system for appropriately evaluating a short-time operation of an organ of a face.SOLUTION: A system includes an interface device and a processing circuit. The processing circuit operates a face recognition system that on the basis of data related to a face acquired via the interface device, sets a plurality of predetermined points in a three dimensional relative coordinate system configured by taking the face itself as a reference on a face surface. The processing circuit executes: a step (1) of executing, on a specific part of the face over all of variations measurable using one or more points with respect to the specific part of the face, a sub step (1-1) of measuring a variation in a predetermined short time on one or more points with respect to a specific part of the face and a sub step (1-2) of calculating left / right ratios of the maximum values of the measured variations by taking a health side of the specific part as a reference; and a step (2) of calculating an average left / right ratio of the left / right ratios calculated in the step (1).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a facial movement evaluation system that evaluates the movement of each facial organ, and a rehabilitation support system that supports the rehabilitation of the movement of each facial organ. [Background technology]

[0002] There are many diseases that cause changes in facial movement. A typical example is facial nerve paralysis, which is a disease in which facial expression on one side of the face is weakened or lost due to damage to the facial nerve on one side. For many years, subjective evaluation methods based on the experience of the evaluator have been used to evaluate the degree of paralysis. An example of a subjective evaluation method is the 40-point method (Yanagihara method).

[0003] In addition to facial paralysis, Parkinson's disease and depression can cause a lack of facial expression overall, a condition known as mask facies.

[0004] Regardless of whether or not a patient has a disease, facial movement varies from person to person, making it difficult to rely on machines to evaluate facial movement. On the other hand, quantitative and reliable means of evaluating facial movement have always been needed by both patients and clinicians. In light of this, various computer-based facial movement evaluation systems have been proposed in recent years, thanks to the development of hardware, including smartphones and tablet devices, and software, including image processing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-36734 A [Patent Document 2] Special Publication No. 2022-533646 Summary of the Invention [Problem to be solved by the invention]

[0006] However, most of the facial movement evaluation systems proposed to date evaluate the static appearance of the face, for example, by showing the relative distance and position of each organ, such as the nose, eyes, cheeks, and mouth. In contrast, in facial nerve diagnosis, a means is required that can evaluate the movement of each organ in a short period of time (e.g., a few seconds), or even at an instant. In addition, a means that can evaluate the specific movement of each organ in a short period of time is very useful in a rehabilitation support system for facial nerve paralysis.

[0007] The present disclosure presents a facial movement evaluation system that appropriately evaluates the short-term movement of each facial organ, and a rehabilitation support system that effectively performs rehabilitation of facial organs and parts by utilizing the function of appropriately evaluating the short-term movement of each facial organ. [Means for solving the problem]

[0008] The present invention provides a facial movement evaluation system that solves the above problems. The facial movement evaluation system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device. Furthermore, the processing circuit (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of a face using one or more points related to the specific part of the face; (1-1) A sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points related to a specific part of the face; (1-2) A sub-step (1-3) of calculating the maximum value of the measured variation value on the healthy side for the specific site; (2) a step (2) of calculating an average left-right ratio for the left-right ratio calculated in the step (1); The purpose of this project is to carry out the following:

[0009] The facial movement evaluation system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device. Furthermore, the processing circuit (1) measuring the variation of one or more points on a specific part of the face within a predetermined short period of time; The purpose of this project is to carry out the following:

[0010] The rehabilitation support system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired through the interface device. Furthermore, the processing circuit displays a message instructing the movement of the affected side of a specific part of the face through the interface device. Furthermore, the processing circuit (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of a face using one or more points related to the specific part of the face; (1-1) A sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points related to a specific part of the face; (1-2) a sub-step (1-2) of calculating a left-right ratio of the maximum value of the measured fluctuation value with respect to the non-movement side of the specific part; (2) A step (2) of calculating an average left / right ratio for the left / right ratio calculated in the step (1); (3) issuing a first notification indicating that motion is occurring if the calculated average left-right ratio is greater than a given motion threshold; and The purpose of this project is to carry out the following:

[0011] The rehabilitation support system of the present disclosure also includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device. Furthermore, the processing circuit displays a message instructing the movement of the affected side of a first specific part of the face via the interface device. Furthermore, the processing circuit (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured using one or more points on a second specific portion of the face, the second specific portion of the face being different from the first specific portion; (1-1) a sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points on a second specific part of the face; (1-2) A sub-step (1-2) of calculating a left-right ratio of the maximum value of the measured fluctuation value with respect to the non-moving side of the second specific part; (2) A step (2) of calculating an average left / right ratio for the left / right ratio calculated in the step (1); (4) issuing a second notification indicating non-stationary if the calculated average left-right ratio is greater than a given non-stationary threshold; and The purpose of this project is to carry out the following:

[0012] The rehabilitation support system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device. Furthermore, the processing circuit displays a message instructing the operation of a specific part of the face via the interface device. Furthermore, the processing circuit: (1) measuring a variation value within a predetermined short period of time for one or more points on a specific part of the face; (2) issuing a first notification indicating that the device is operating if the measured variation value is greater than a given operating threshold; The purpose of this project is to carry out the following:

[0013] Furthermore, the rehabilitation support system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference based on face-related data acquired through the interface device. Furthermore, the processing circuit indicates a message instructing an operation of a first specific part of the face through the interface device. Furthermore, the processing circuit: (1) measuring a variation value within a predetermined short period of time for one or more points on a second specific part of the face, the second specific part being different from the first specific part; (2) issuing a second notification of non-stationary if the measured variation value is greater than a given non-stationary threshold; The purpose of this project is to carry out the following: Effect of the Invention

[0014] By using the facial movement evaluation system of the present disclosure, the short-term movements of each facial organ and part can be properly evaluated from the viewpoint of left-right ratio and can be properly understood from the viewpoint of the amount of fluctuation. Furthermore, the rehabilitation support system of the present disclosure can effectively perform the rehabilitation of facial organs and parts by utilizing the function of properly evaluating the short-term movements of each facial organ and part from the viewpoint of left-right ratio and the function of properly understanding from the viewpoint of the amount of fluctuation. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a system configuration diagram of a facial movement evaluation system according to the first embodiment. [Diagram 2]FIG. 2 is a diagram showing multiple (e.g., 1,220) predetermined point clouds (hereinafter referred to as "point clouds generated by the face recognition system") in a three-dimensional relative coordinate system configured based on the face, set by the three-dimensional face recognition system used in this disclosure. [Diagram 3] FIG. 3 is a diagram showing a schematic diagram of an infrared sensor, a front camera, and an infrared irradiation unit in a smartphone, which are used in the face recognition system used in the present disclosure. [Figure 4] Figures 4(a-1) and 4(a-2) are diagrams showing distances between two pairs of predetermined points near the eyebrows in a point cloud obtained by a face recognition system. Figures 4(a-3) and 4(a-4) are diagrams showing predetermined points near the eyebrows in a point cloud obtained by a face recognition system. [Diagram 5] FIG. 5 shows the inside of the eyelid in a point cloud captured by a face recognition system. [Figure 6A] FIG. 6A is a diagram showing a number of predetermined points near the cheeks in a point cloud generated by a face recognition system. [Figure 6B] Fig. 6B(b-1) is a diagram showing angles at rest between the eyebrows and the nose tip on the midline and a specified point near the cheek in the point cloud obtained by the face recognition system. Fig. 6B(b-2) is a diagram showing angles during mouth corner lifting movement between the eyebrows and the nose tip on the midline and a specified point near the cheek in the point cloud obtained by the face recognition system. [Figure 7A] Fig. 7A(a-1) is a diagram showing a plurality of predetermined points near the corners of the mouth in a point cloud obtained by a face recognition system, and Fig. 7A(a-2) is an enlarged view of the vicinity of the plurality of predetermined points near the corners of the mouth in Fig. 7A(a-1). [Figure 7B] Fig. 7B(b-1) is a diagram showing the angle at rest between the eyebrows and the nose tip on the midline and a specified point near the corners of the mouth in the point cloud obtained by the face recognition system. Fig. 7B(b-2) is a diagram showing the angle during the movement of lifting the corners of the mouth between the eyebrows and the nose tip on the midline and a specified point near the corners of the mouth in the point cloud obtained by the face recognition system. [Figure 7C]Fig. 7C(c-1) is a diagram showing the angle at rest between the eyebrows and the nose tip on the midline and the lip point on the midline in the point cloud obtained by the face recognition system. Fig. 7C(c-2) is a diagram showing the angle during the mouth corner lift movement between the eyebrows and the nose tip on the midline and the lip point on the midline in the point cloud obtained by the face recognition system. [Figure 8] FIG. 8 is a diagram showing a plurality of predetermined points near the ala of the nose in a point cloud obtained by a face recognition system. [Figure 9] FIG. 9 is a system configuration diagram of a rehabilitation support system according to the second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a screen of the rehabilitation support system according to the second embodiment. [Figure 11A] FIG. 11A is a diagram showing a flowchart illustrating an evaluation value calculation process in the facial movement evaluation system according to the first embodiment. [Figure 11B] FIG. 11B is a diagram showing a flowchart illustrating a fluctuation value measurement process in the facial movement evaluation system according to the second embodiment. [Figure 11C] FIG. 11C is a diagram showing a flowchart illustrating an evaluation value calculation and movement determination process in the rehabilitation support system according to the third embodiment. [Figure 11D] FIG. 11D is a flowchart showing the evaluation value calculation and attention notification processing in the rehabilitation support system according to the third embodiment. [Figure 11E] FIG. 11E is a diagram showing a flowchart illustrating a variation measurement and movement determination process in the rehabilitation support system according to the fourth embodiment. [Figure 11F] FIG. 11F is a diagram showing a flowchart illustrating a variation measurement and attention notification process in the rehabilitation support system according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

[0017] The inventor provides the accompanying drawings and the following description so that those skilled in the art can fully understand the present disclosure, and does not intend for them to limit the subject matter described in the claims.

[0018] 1. [First embodiment] Hereinafter, a first preferred embodiment of the present disclosure will be described with reference to the accompanying drawings. The facial movement evaluation system according to the first embodiment is useful for evaluating facial movement in facial nerve paralysis.

[0019] 1.1. [System Configuration] 1 is a diagram showing the overall system configuration of a facial movement evaluation system 2 according to the first embodiment. The facial movement evaluation system 2 according to the first embodiment runs on a mobile terminal 6.

[0020] As shown in FIG. 9, a rehabilitation support system 2' according to the second embodiment of the present disclosure can also operate on the mobile terminal 6 in which the facial movement evaluation system 2 according to the first embodiment operates.

[0021] The mobile terminal 6 is a portable computing device such as a tablet terminal or a smartphone. The mobile terminal 6 includes an imaging element and an imaging control means for controlling the imaging element. That is, for example, as shown in FIG. 3, the mobile terminal 6 includes an infrared sensor 20, a front camera 22, and an infrared irradiation unit 24. The mobile terminal 6 is connected to an external network 8. Note that the mobile terminal 6 is not limited to a portable computing device such as a tablet terminal or a smartphone, and may be, for example, a PC (personal computer) or a workstation computer.

[0022] The external network 8 is, for example, the Internet, and is connected to the mobile terminal 6 via an interface device 10 (described below). The facial movement evaluation system 2 is based on (for example) a native app running on the mobile terminal 6, and therefore the mobile terminal 6 builds the facial movement evaluation system 2 in the mobile terminal 6 by downloading an application program related to the facial movement evaluation system from an app store server 4 connected to the external network 8. Note that the facial movement evaluation system 2 does not have to be based on a native app, and may be based on a web app or a hybrid app.

[0023] As shown in FIG. 1, the mobile terminal 6 includes an interface device 10, a processing circuit 12, and a memory .

[0024] The interface device 10 is an interface unit capable of sending and receiving data with the outside, including a network terminal, a video input / output terminal, a USB terminal, a camera, a pointing device, a keyboard, a mouse, etc. The data here is, for example, image data relating to a face. After being acquired, the data can be recorded in a storage unit such as a memory 14 inside the mobile terminal 6.

[0025] The processing circuit 12 is configured with a processor. The processor here includes a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Various processes of the facial movement evaluation system 2 according to the first embodiment are realized by the processing circuit 12 executing various programs.

[0026] The memory 14 is a rewritable storage unit within the mobile terminal 6, and is configured, for example, by a RAM (Random Access Memory) including a large number of semiconductor memory elements. The memory 14 temporarily stores specific computer programs, variable values, parameter values, etc., when the processing circuit 12 executes various processes. The memory 14 may include a so-called ROM (Read Only Memory). The ROM stores in advance application programs that realize various processes of the facial movement evaluation system 2 described below. The processing circuit 12 reads out the application programs from the ROM and loads them into the RAM, thereby enabling the processing circuit 12 to execute the application programs.

[0027] The facial movement evaluation system 2 according to the first embodiment is constructed using a computer language such as Swift. Computer languages ​​that can be used to construct the facial movement evaluation system 2 are not limited to these, and other computer languages ​​may be used as a matter of course.

[0028] Furthermore, the processing circuit 12 of the mobile terminal 6 according to the first embodiment operates a face recognition system that sets a number of predetermined points on the face surface in a three-dimensional relative coordinate system based on the face itself, based on image data related to the face acquired via the interface device 10. The face recognition system here is, for example, a system based on "ARFacegeometry" (Class name) used in FaceID, a three-dimensional face authentication system provided by Apple Inc., and in addition to performing image processing of the face using the front camera 22 (as shown in FIG. 3), infrared irradiation is performed by the infrared irradiation unit 24, and the face is three-dimensionally grasped using the infrared sensor 20, thereby performing real-time three-dimensional tracking. Note that "ARFacegeometry" is a development kit provided by Apple Inc. The face recognition system here is capable of acquiring the three-dimensional coordinates of 1220 points arranged on the face. Furthermore, in this face recognition system, the coordinates are determined by a relative coordinate system based on the face itself, so that the three-dimensional coordinates of the points arranged on the face surface do not change even if the position or orientation of the face changes. Therefore, the face recognition system of the present disclosure sets 1220 predetermined points on the face surface in a three-dimensional relative coordinate system configured with the face itself as the reference.

[0029] 2 is a diagram showing a plurality of (e.g., 1220) predetermined points (hereinafter, referred to as "points by the face recognition system") in a three-dimensional relative coordinate system configured with the face as a reference, which is set by the face recognition system used in the present disclosure. The 1220 points are displayed on the display surface of the mobile terminal 6 as a mirror image of the face of the operator facing the display surface of the mobile terminal 6. Although not shown in FIG. 2, each of the 1220 predetermined points is assigned a fixed number (see FIGS. 4 to 8).

[0030] The face recognition system to be used in the present disclosure is not limited to a system based on ARFacegeometry used in FaceID, a 3D face authentication system provided by Apple, Inc., but may be any face recognition system capable of setting a large number of points on the surface of the face in a relative coordinate system constructed with the face itself as the reference.

[0031] 1.2.[System Operation] The operation of the facial movement evaluation system 2 by the operator of the facial movement evaluation system 2, i.e., the person to be evaluated, and the action of the facial movement evaluation system 2 in response thereto will be described below.

[0032] An operator of the facial movement evaluation system 2 first starts the operation of the facial movement evaluation system 2 on the mobile terminal 6. Here, in the facial movement evaluation system 2, the healthy side (and the affected side) of the operator (subject) can be set automatically or manually.

[0033] The operator (evaluated person) performs facial movements such as closing both eyes, closing one eye at a time, raising eyebrows, and smiling with the corners of the mouth raised while looking at the display surface of the mobile terminal 6. At that time, the facial movement evaluation system 2 performs the following evaluation value calculation process based on real-time facial data acquired via the interface device 10 such as the front camera 22 and the infrared sensor 20, and further based on a point cloud generated by a face recognition system from the facial data. Fig. 11A is a diagram showing a flowchart showing the following evaluation value calculation process.

[0034] [Evaluation value calculation process] [1] Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of the face using one or more points on the specific part of the face. [1-1] Sub-step (1-1) of measuring the variation value within a predetermined short period of time for one or more points on a specific part of the face. [1-2] A sub-step (1-2) of calculating a left-right ratio of the maximum measured fluctuation value for the specific site, with the healthy side as the reference. [2] Step (2) of calculating an average left / right ratio for the left / right ratios calculated in step (1).

[0035] The average ratio calculated in [2] above is the evaluation value of the movement of that particular part. In other words, the average ratio of the part, which indicates the degree of movement of the affected side based on the healthy side of the part, is the "evaluation value."

[0036] In the facial movement evaluation system 2 according to the first embodiment, in the above-mentioned evaluation value calculation process, the specific parts of the face are, for example, eyebrows, eyes, cheeks, corners of the mouth, wings of the nose, etc. Moreover, the "predetermined short time" is a time sufficient for performing simple facial movements, for example, 3 seconds, 5 seconds, etc.

[0037] Further, examples of maximum variation values ​​that can be measured using one or more points on a particular part of the face are as follows: (1-1), (1-2), (2-1), (3-1), (3-2), (3-3), (4-1), (4-2), (4-3), (4-4), (5-1), (5-2), and (5-3). ((1) If the specific part is the eyebrows) (1-1) The total distance when the total value of the distance between two points of a specified set of two points near the eyebrows is maximum. (1-2) Maximum movement speed for a number of predetermined points in the vicinity of the eyebrows. ((2) When the specific part is the "eyes") (2-1) The difference in the area inside the eyelid when it is at its narrowest when the eyes are closed and when the eyes are open and at rest. ((3) If the specific part is the cheek) (3-1) The maximum total movement distance of a number of specified points in the vicinity of the cheek. (3-2) The maximum value of the average movement speed of a number of predetermined points in the vicinity of the cheek. (3-3) The difference in angle between the point between the eyebrows and the tip of the nose on the midline and a specified point near the cheek at rest and during the movement of lifting the corners of the mouth. ((4) If the specific area is the "corner of the mouth") (4-1) The maximum total movement distance of a number of specified points in the vicinity of the corners of the mouth. (4-2) The maximum value of the average movement speed of a number of predetermined points in the vicinity of the corners of the mouth. (4-3) The difference in angle between the point between the eyebrows and the point on the nose on the midline and a specified point near the corners of the mouth at rest and during the movement of lifting the corners of the mouth. (4-4) Difference in the angle between the eyebrow point and the nose point on the midline and the lip point on the midline at rest and during the movement of lifting the corners of the mouth ((5) If the specific part is the "wing of the nose") (5-1) The maximum total movement distance of a number of specified points in the vicinity of the nasal ala. (5-2) The maximum value of the average movement speed of a number of specified points in the vicinity of the nasal ala.

[0038] Examples of the (maximum) fluctuation values ​​that can be measured using one or more points on a specific part of the face, as described above in (1-1) to (5-2), will be described with reference to FIGS.

[0039] [Explanation of maximum fluctuation value example]

[0040] (1) When the specific body part is the "eyebrows" 4(a-1) and 4(a-2) are diagrams showing distances between two points of a set of two points near the eyebrows in a point cloud obtained by a face recognition system. FIG. 4(a-1) shows distances between two points of a set of two points near the eyebrows when the subject is at rest, and FIG. 4(a-2) shows distances between two points of a set of two points near the eyebrows on the left side. For example, five sets of two points, (329-1150), (328-1151), (327-1152), (335-1153), (197-1154), and (198-1155), are assumed as sets of two points near the left eyebrow. Points corresponding to these five sets of points are assumed for the right eyebrow. In FIG. 4(a-1) and FIG. 4(a-2), only two sets, (329-1150) and (328-1151), are connected by a line segment. FIG. 4(a-2) is a diagram of a point cloud generated by a face recognition system, which may show the "total distance when the sum of the distances between two points for a specified number of pairs of two points near the eyebrows is maximized."

[0041] Fig. 4(a-3) and Fig. 4(a-4) are diagrams showing a plurality of predetermined points near the eyebrows in a point cloud obtained by a face recognition system. Fig. 4(a-3) is for a subject at rest, and Fig. 4(a-3) is for a subject with an eyebrow raised. For example, five points 329, 328, 327, 335, 197, and 198 are assumed as multiple points near the left eyebrow. Points corresponding to these five points are assumed for the right eyebrow. In Fig. 4(a-3) and Fig. 4(a-4), only 329 is circled. Fig. 4(a-4) is a diagram of a point cloud obtained by a face recognition system that may indicate the "maximum movement speed for the plurality of predetermined points near the eyebrows".

[0042] (2) When the specific body part is the "eye" FIG. 5 is a diagram showing the inside of the eyelid in a point cloud obtained by a face recognition system. That is, the inside of the left eyelid and its area are shown by connecting 24 points, 1085-(serial number)-1108, around the edge of the left eye. For the right eye, the points corresponding to these 24 points are assumed to indicate the inside of the eyelid. The difference between when this area is at its smallest (i.e., when the eyes are closed) and when at rest (i.e., when the eyes are open) is the "difference in the area of ​​the inside of the eyelid between when the eyes are closed at their smallest and when the eyes are open at rest."

[0043] (3) When the specific part is the cheek FIG. 6A is a diagram showing a plurality of predetermined points near the cheeks in a point cloud obtained by a face recognition system. For example, in the vicinity of the right cheek shown in FIG. 6A, four points 600, 601, 785, and 821 are assumed as a plurality of points. For the left cheek, points corresponding to these four points are assumed. In FIG. 6A, 600, 601, 785, and 821 are each surrounded by a circle. The maximum value of the total movement distance for each of these four points is the "maximum value of the total movement distance for the predetermined plurality of points near the cheeks". Furthermore, the maximum value of the average movement speed for each of these four points is the "maximum value of the average movement speed for the predetermined plurality of points near the cheeks".

[0044] FIG. 6B(b-1) is a diagram showing the angle at rest, which is determined by the point between the eyebrows and the tip of the nose on the midline, and a specified point near the cheek, in the point cloud obtained by the face recognition system. FIG. 6B(b-2) is a diagram showing the angle during the mouth corner lifting movement, which is determined by the point between the eyebrows and the tip of the nose on the midline, and a specified point near the cheek, in the point cloud obtained by the face recognition system. The point between the eyebrows and the tip of the nose on the midline are points 15,7. On the right cheek, the specified point near the cheek is point 600. For the left cheek, a point corresponding to this point (600) is assumed. The difference between the angle formed by these three points for each cheek at rest and during the mouth corner lifting movement is the "difference between the angle at rest and during the mouth corner lifting movement, which is determined by the point between the eyebrows and the tip of the nose on the midline, and a specified point near the cheek."

[0045] (4) When the specific area is the "corner of the mouth" FIG. 7A(a-1) is a diagram showing a plurality of predetermined points in the vicinity of the corners of the mouth in a point cloud obtained by a face recognition system. FIG. 7A(a-2) is an enlarged view of the vicinity of the plurality of predetermined points in the vicinity of the corners of the mouth in FIG. 7A(a-1). In the vicinity of the corners of the mouth on the right side shown in FIG. 7A(a-1) and FIG. 7A(a-2), 12 points, 823, 824, 825, 826, 684, 638, 678, 639, 834, 835, 836, and 837, are assumed as a plurality of points. For the corners of the mouth on the left side, a point cloud corresponding to these 12 points is assumed. In FIG. 7A(a-1) and FIG. 7A(a-2), the point clouds 823, 824, 825, 826, 684, 638, 678, 639, 834, 835, 836, and 837 are surrounded by a line such as a circle. The maximum value of the total movement distance for each of these 12 points is the "maximum value of the total movement distance for a plurality of predetermined points near the corners of the mouth." Furthermore, the maximum value of the average movement speed for each of these 12 points is the "maximum value of the average movement speed for a plurality of predetermined points near the corners of the mouth."

[0046] FIG. 7B(b-1) is a diagram showing the angle at rest, which is determined by the point between the eyebrows and the tip of the nose on the midline, and a specified point near the corner of the mouth, in the point cloud obtained by the face recognition system. FIG. 7B(b-2) is a diagram showing the angle during the mouth corner lifting movement, which is determined by the point between the eyebrows and the tip of the nose on the midline, and a specified point near the corner of the mouth, in the point cloud obtained by the face recognition system. The point between the eyebrows and the tip of the nose on the midline are points 15,7. For the right corner of the mouth, the specified point near the corner of the mouth is point 825. For the left corner of the mouth, a point corresponding to this point (825) is assumed. The difference between the angle formed by these three points for each corner of the mouth at rest and during the mouth corner lifting movement is the "difference between the angle at rest and during the mouth corner lifting movement, which is determined by the point between the eyebrows and the tip of the nose on the midline, and a specified point near the corner of the mouth."

[0047] FIG. 7C(c-1) is a diagram showing angles at rest, formed by the point between the eyebrows and the tip of the nose on the midline, and the point of the lips on the midline, in the point cloud obtained by the face recognition system. FIG. 7C(c-2) is a diagram showing angles at rest, formed by the point between the eyebrows and the tip of the nose on the midline, and the point of the lips on the midline, in the point cloud obtained by the face recognition system, during the movement of raising the corners of the mouth, particularly during the movement of raising the corners of the right mouth. The difference between the angle formed by these three points for each corner of the mouth at rest and during the movement of raising the corners of the mouth, particularly for the right corner of the mouth, is the "difference between the angle formed by the point between the eyebrows and the tip of the nose on the midline, and the point of the lips on the midline, at rest and during the movement of raising the corners of the mouth". For the left corner of the mouth, the angle during the movement of raising the left corner of the mouth must be used.

[0048] (5) When the specific area is the "nasal ala" FIG. 8 is a diagram showing a plurality of predetermined points in the vicinity of the nasal ala in a point cloud obtained by a face recognition system. For example, in the vicinity of the right nasal ala shown in FIG. 8, eleven points, 652, 655, 732, 743, 729, 651, 653, 747, 760, 727, 748, are assumed as a plurality of points. For the left nasal ala, points corresponding to these eleven points are assumed. In FIG. 8, the point cloud of 652, 655, 732, 743, the point cloud of 729, 651, 653, 747, and the point cloud of 760, 727, 748 are each surrounded by a line. The maximum value of the total movement distance for these eleven points on the right and left sides is the "maximum value of the total movement distance of the predetermined plurality of points in the vicinity of the nasal ala." Furthermore, the maximum value of the average movement speed for these 11 points on the right and left sides is the "maximum value of the average movement speed for a plurality of predetermined points in the vicinity of the nasal ala."

[0049] [Explanation of maximum fluctuation value example - End]

[0050] [About output and display of evaluation values] The facial movement evaluation system 2 according to the first embodiment can display an "evaluation value" for each specific part of the face as a result of the movement evaluation on the display surface of the mobile terminal 6, for example as follows:

[0051] [Table 1] Here, it is shown that the corners of the mouth and the ala of the nose on the affected side are less mobile than other areas.

[0052] The facial movement evaluation system 2 can also calculate an evaluation value for the entire face by averaging the "evaluation values" for all specific parts of the face (e.g., eyebrows, eyes, cheeks, corners of the mouth, and wings of the nose). Therefore, the facial movement evaluation system 2 can display the evaluation value for the entire face on the display surface of the mobile terminal 6 as the result of the movement evaluation. In the example of Table 1, the result of the movement evaluation is "(90+90+90+60+60) / 5=78%."

[0053] Furthermore, the facial movement evaluation system 2 can also display the expected degree of facial paralysis depending on the evaluation value. For example, if the evaluation value of the entire face is close to approximately 100%, it can display "There is no problem with facial movement," and if the evaluation value of the entire face is less than a predetermined threshold value (e.g., 90%), it can display "There is a difference in facial movement between the left and right sides."

[0054] 1.3. [Summary] The facial movement evaluation system 2 according to the first embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device 10. Furthermore, the processing circuit 12 (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of a face using one or more points related to the specific part of the face; (1-1) A sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points related to a specific part of the face; (1-2) A sub-step (1-2) of calculating a left-right ratio of the maximum value of the measured variation value with respect to the healthy side for the specific site; (2) A step (2) of calculating an average left / right ratio for the left / right ratio calculated in the step (1); The purpose of this project is to carry out the following:

[0055] By using the facial movement evaluation system 2 according to the first embodiment, the movements of each organ or part of the face over a short period of time can be appropriately evaluated from the perspective of the left-right ratio.

[0056] 2. [Second embodiment] Following the first embodiment, a preferred second embodiment of the present disclosure will be disclosed. The facial movement evaluation system according to the second embodiment is substantially the same as the facial movement evaluation system according to the first embodiment. Therefore, the following description will focus on the differences between the two.

[0057] 2.1. [System Configuration] A facial movement evaluation system 2 according to the second embodiment has the same system configuration as the facial movement evaluation system according to the first embodiment shown in FIG.

[0058] 2.2.[System Operation]

[0059] The operation of the facial movement evaluation system 2 by the operator of the facial movement evaluation system 2 in the second embodiment, i.e., the person being evaluated, and the response of the facial movement evaluation system 2 will be described below, focusing on the differences between the first and second embodiments.

[0060] An operator of the facial movement evaluation system 2 first starts the operation of the facial movement evaluation system 2 on the mobile terminal 6 .

[0061] The operator (evaluatee) performs facial movements such as closing both eyes, closing one eye at a time, raising eyebrows, smiling with the corners of the mouth raised, opening the mouth, lowering the chin, etc. while looking at the display surface of the mobile terminal 6. At that time, the facial movement evaluation system 2 performs the following variation value measurement process based on real-time facial data acquired via the interface device 10 such as the front camera 22 and the infrared sensor 20, and further based on a point cloud generated by a facial recognition system from the facial data. Figure 11B is a diagram showing a flowchart showing the following variation value measurement process.

[0062] [Variation measurement processing] [1] Step (1) of measuring the variation of one or more points on a specific part of the face within a predetermined short period of time.

[0063] In the above-mentioned fluctuation value measurement process in the facial movement evaluation system 2 according to the second embodiment, the specific parts of the face are, for example, the right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, right corner of the mouth, left corner of the mouth, right nasal wing, left nasal wing, lips, neck, lower jaw, etc. Also, the "predetermined short period" is a period of time sufficient for performing simple facial movements, for example, 4 seconds, 6 seconds, etc.

[0064] Further, the variation value measured using one or more points on a particular part of the face may be, for example, the average movement distance of a plurality of predetermined points on the particular part of the face; - the average moving speed of a given number of points on a given specific part; Variation in the area formed by a number of predetermined points on a particular part of the face, Variation in the angle formed by three given points on a particular part of the face, etc.

[0065] As mentioned above, FIGS. 4 to 8 show examples of variations that can be measured using one or more points on a particular part of the face.

[0066] The facial movement evaluation system 2 according to the second embodiment can perform the "variation value measurement process" simultaneously for a plurality of specific facial regions in parallel. Furthermore, the facial movement evaluation system 2 according to the second embodiment can perform the "variation value measurement process" in parallel so as to simultaneously measure a plurality of variation values ​​(for example, "average movement distance of a plurality of predetermined points on a specific facial region" and "average movement speed of a plurality of predetermined points on a specific facial region") for a specific facial region.

[0067] [About output and display of fluctuation values] The facial movement evaluation system 2 according to the second embodiment can display the "variation value" for each specific part of the face on the display surface of the mobile terminal 6 as a result of the movement measurement.

[0068] Furthermore, for example, the facial movement evaluation system 2 can add up and average the "average movement distances of specified multiple points" for all specific parts of the face (e.g., right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, corners of the mouth, wings of the nose, lips, neck, lower jaw, etc.), calculate this as a value for the movement of the entire face, and display it on the display surface of the mobile terminal 6.

[0069] 2.3. [Summary] The facial movement evaluation system 2 according to the second embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device 10; Furthermore, the processing circuit 12 (1) measuring the variation of one or more points on a specific part of the face within a predetermined short period of time; The purpose of this project is to carry out the following:

[0070] By using the facial movement evaluation system 2 according to the second embodiment, the movements of each organ or part of the face over a short period of time can be properly understood from the viewpoint of a fluctuation value (amount of fluctuation).

[0071] 3. [Third embodiment] Following the first and second embodiments, a preferred third embodiment of the present disclosure will be disclosed. The rehabilitation support system according to the third embodiment is similar to the facial movement evaluation system according to the first and second embodiments. Therefore, the following description will focus on the differences between the two. The rehabilitation support system according to the third embodiment is useful for supporting rehabilitation for facial paralysis.

[0072] 3.1. [System configuration] 9 is a diagram showing the overall system configuration of a rehabilitation support system 2' according to the third embodiment. The rehabilitation support system 2' according to the third embodiment operates on a mobile terminal 6, similar to the facial movement evaluation systems 2 according to the first and second embodiments.

[0073] As in the first and second embodiments, the mobile terminal 6 is a mobile computing device such as a tablet terminal or a smartphone. The mobile terminal 6 includes an imaging element and an imaging control means for controlling the imaging element. That is, for example, as shown in FIG. 3, the mobile terminal 6 includes an infrared sensor 20, a front camera 22, and an infrared irradiation unit 24. The mobile terminal 6 is connected to an external network 8.

[0074] The external network 8 is, for example, the Internet, and is connected to the mobile terminal 6 via the interface device 10. The rehabilitation support system 2′ may also be a native application running on the mobile terminal 6. Thus, the mobile terminal 6 can build the rehabilitation support system 2′ in the mobile terminal 6 by downloading an application program related to the rehabilitation support system 2′ from an application store server 4 connected to the external network 8.

[0075] 9 also includes an interface device 10, a processing circuit 12, and a memory 14. The interface device 10, the processing circuit 12, and the memory 14 are all similar to those of the mobile terminals 6 in the first and second embodiments.

[0076] Furthermore, the processing circuit 12 of the mobile terminal 6 according to the third embodiment also operates a face recognition system that sets a number of predetermined points on the face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on image data related to the face acquired via the interface device 10, as in the first and second embodiments. The face recognition system here is also a system based on "ARFacegeometry" (Class name) used in FaceID, a three-dimensional face authentication system provided by Apple Inc., and performs real-time three-dimensional tracking using the front camera 22, the infrared irradiation unit 24, the infrared sensor 20, etc. Therefore, the face recognition system here also sets 1220 predetermined points on the face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference (see FIG. 2). The face recognition system here is not limited to one based on "ARFacegeometry", and may be any system for face recognition that can set a number of points on the face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference.

[0077] 3.2.[System Operation] The operation of the rehabilitation support system 2' by the operator of the rehabilitation support system 2' and the corresponding behavior of the rehabilitation support system 2' will be described below.

[0078] First, the operator of the rehabilitation support system 2 ′ starts operation of the rehabilitation support system 2 ′ on the mobile terminal 6 .

[0079] Next, the healthy side (and the affected side) of the operator's face can be set automatically or manually in the rehabilitation support system 2'. In rehabilitation using the rehabilitation support system 2', the focus is on moving only the affected side of the face. In other words, the affected side is the working side, and the healthy side is the non-working side.

[0080] The facial parts that will be rehabilitated are mainly the eyes (closed, open) and the corners of the mouth. The rehabilitation shown below is also performed on the eyes (closed, open) and the corners of the mouth as an example.

[0081] 3.2.1. [Movement determination in rehabilitation support systems] For example, when the affected side is the right side, the rehabilitation support system 2' displays "Please close only your right eye" on the display screen of the mobile terminal 6 (as shown in FIG. 10). The operator of the rehabilitation support system 2' tries to move in accordance with the display. If the rehabilitation support system 2' judges that the right eye is moving sufficiently, the display screen of the rehabilitation support system 2' displays, for example, "Your eyes are closed properly" (see FIG. 10(1)). In this way, when the movement of a specific part is sufficient and appropriate, the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' performs the following evaluation value calculation and movement judgment processing. The following "evaluation value calculation" processing is the same as the [evaluation value calculation processing] performed by the processing circuit 12 of the mobile terminal 6 in the facial movement evaluation system 2 according to the first embodiment. Therefore, the average left-right ratio calculated in the following [2] is the evaluation value of the movement of the specific part (however, the non-moving side of the part is used as the basis, not the healthy side of the part). A movement of a specific part is sufficient and appropriate when the calculated average left-right ratio is greater than a given movement threshold (step (3)). In the example shown in FIG. 10, the specific part is the "eyes." In accordance with the first notification issued in step [3], the processing circuit 12 displays via the interface device 10 that the movement is appropriate (for example, "eyes are closed appropriately"). FIG. 11C is a diagram showing a flowchart illustrating the following evaluation value calculation and movement determination process.

[0082] [Evaluation value calculation and operation determination process] [1] Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of the face using one or more points on the specific part of the face. [1-1] Sub-step (1-1) of measuring the variation value within a predetermined short period of time for one or more points on a specific part of the face. [1-2] A sub-step (1-2) of calculating the left-right ratio of the maximum measured fluctuation value with respect to the non-moving side for the specific part. [2] Step (2) of calculating an average left / right ratio for the left / right ratios calculated in step (1). [3] Step (3) of issuing a first notification indicating that movement is occurring if the calculated average left-right ratio is greater than a given movement threshold.

[0083] In the above-mentioned evaluation value calculation and movement determination process, the specific parts of the face are the eyes, the corners of the mouth, etc., as described above. Moreover, the "predetermined short time" is a time sufficient for performing simple facial movements, such as 3 seconds, 5 seconds, etc.

[0084] Furthermore, examples of maximum variation values ​​that can be measured using one or more points on a specific part of the face are as follows (2-1), (4-1), (4-2), (4-3), and (4-4), as described in the first embodiment. ((2) When the specific part is the "eyes") (2-1) The difference in the area inside the eyelid when it is at its narrowest when the eyes are closed and when the eyes are open and at rest. ((4) If the specific area is the "corner of the mouth") (4-1) The maximum total movement distance of a number of specified points in the vicinity of the corners of the mouth. (4-2) The maximum value of the average movement speed of a number of predetermined points in the vicinity of the corners of the mouth. (4-3) The difference in angle between the point between the eyebrows and the point on the nose on the midline and a specified point near the corners of the mouth at rest and during the movement of lifting the corners of the mouth. (4-4) Difference in the angle between the eyebrow point and the nose point on the midline and the lip point on the midline at rest and during the movement of lifting the corners of the mouth

[0085] Examples of the (maximum) fluctuation values ​​that can be measured using one or more points on a specific part of the face, as described above in (2-1), (4-1) to (4-4), are described in [Explanation of an example of the maximum fluctuation value] to [Explanation of an example of the maximum fluctuation value - end] in [First embodiment].

[0086] 3.2.2. [Caution notification in rehabilitation support systems] Also, when the operator of the rehabilitation support system 2' tries to make a movement in accordance with the display of "Please close only your right eye", if the rehabilitation support system 2' judges that the right corner of the mouth has moved (is linked), for example, "Your mouth is also moving at the same time" is displayed on the display surface of the rehabilitation support system 2' (see FIG. 10(2)). In this way, when a second specific part different from the first specific part that is the subject of rehabilitation is linked, the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' performs the following evaluation value calculation and attention notification processing. The "evaluation value calculation" processing below is the same as the evaluation value calculation of the [evaluation value calculation and movement judgment processing] executed by the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' according to the present embodiment (third embodiment) described above. Therefore, the average left-right ratio calculated in the following [2] is the evaluation value of the movement of the second specific part. The second specific body part is linked, i.e., a corresponding link is occurring, when the calculated average left-right ratio is greater than a given non-stationary threshold (step (4)). In the example shown in FIG. 10(2), the second specific body part is the "corner of the mouth." In accordance with the second notification issued in step [4], the processing circuit 12 displays a warning notification (e.g., "Your mouth is also moving at the same time") via the interface device 10 that linkage of other body parts is occurring. FIG. 11D is a diagram showing a flowchart illustrating the following evaluation value calculation and warning notification processing.

[0087] [Evaluation value calculation and warning notification processing] [1] Step (1) of carrying out the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured using one or more points on a second specific area of ​​the face, the second specific area being different from the first specific area. [1-1] Sub-step (1-1) of measuring the variation value for one or more points on a second specific part of the face within a predetermined short period of time. [1-2] A sub-step (1-2) of calculating a left-right ratio of the maximum measured fluctuation value with respect to the non-moving side for the second specific part. [2] Step (2) of calculating an average left / right ratio for the left / right ratios calculated in step (1). [4] Step (4) of issuing a second notification indicating non-stationary if the calculated average left-right ratio is greater than a given non-stationary threshold.

[0088] In the above-mentioned evaluation value calculation and attention notification process, when the first specific part of the face is the eye, the second part of the face, i.e., the part that is easily linked, is the corner of the mouth, eyebrows, etc. When the first specific part of the face is the corner of the mouth, the second part of the face, i.e., the part that is easily linked, is the eye, eyebrows, etc. Furthermore, the "predetermined short period of time" is a period of time sufficient for performing simple facial movements, such as 3 seconds, 5 seconds, etc.

[0089] Furthermore, examples of maximum variation values ​​that can be measured using one or more points on a second specific part of the face, as described in the first embodiment, are as follows: (1-1), (1-2), (2-1), (4-1), (4-2), (4-3), and (4-4). ((1) If the second specific part is the eyebrows) (1-1) The total distance when the total value of the distance between two points of a specified set of two points near the eyebrows is maximum. (1-2) Maximum movement speed for a number of predetermined points in the vicinity of the eyebrows. ((2) When the second specific part is the "eyes") (2-1) The difference in the area inside the eyelid when it is at its narrowest when the eyes are closed and when the eyes are open and at rest. ((4) If the second specific part is the "corner of the mouth") (4-1) The maximum total movement distance of a number of specified points in the vicinity of the corners of the mouth. (4-2) The maximum value of the average movement speed of a number of predetermined points in the vicinity of the corners of the mouth. (4-3) The difference in angle between the point between the eyebrows and the point on the nose on the midline and a specified point near the corners of the mouth at rest and during the movement of lifting the corners of the mouth. (4-4) Difference in the angle between the eyebrow point and the nose point on the midline and the lip point on the midline at rest and during the movement of lifting the corners of the mouth

[0090] Examples of the (maximum of) fluctuation values ​​that can be measured using one or more points on the (second) specific part of the face, as described above in (1-1) to (1-2), (2-1), and (4-1) to (4-4), are described in [Explanation of an example of the maximum of fluctuation values] to [Explanation of an example of the maximum of fluctuation values ​​- end] in [First embodiment].

[0091] [About output and display of result messages] The rehabilitation support system 2' according to the third embodiment can display an evaluation message as a result of the motion determination process and a warning message as a result of the warning notification process on the display surface.

[0092] As shown in Fig. 10(1), when the movement of the right eye, which is a specific part of the face, is sufficient and appropriate, the message "eye closing is appropriate" is displayed on the display. At the same time, a circle can be displayed around the right eye, which is performing the appropriate movement.

[0093] Also, as shown in Fig. 10(2), when a link occurs in the right corner of the mouth, which is a second specific part different from the specific part of the face, the message "Your mouth is also moving at the same time" is displayed on the display. At the same time, a circular area can be added to the entire right corner of the mouth where the link occurs.

[0094] 3.3. [Summary] The rehabilitation support system 2′ according to the third embodiment includes: The device includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on facial data acquired via the interface device 10. Furthermore, the processing circuit displays a message via the interface device 10 instructing the operation of the affected side of a specific part of the face. Furthermore, the processing circuit 12 (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of a face using one or more points related to the specific part of the face; (1-1) A sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points related to a specific part of the face; (1-2) a sub-step (1-2) of calculating a left-right ratio of the maximum value of the measured fluctuation value with respect to the non-movement side of the specific part; (2) A step (2) of calculating an average left / right ratio for the left / right ratio calculated in the step (1); (3) issuing a first notification indicating that motion is occurring if the calculated average left-right ratio is greater than a given motion threshold; and The purpose of this project is to carry out the following:

[0095] In addition, the rehabilitation support system 2′ according to the third embodiment includes: The device includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on facial data acquired via the interface device 10. Furthermore, the processing circuit 12 displays a message via the interface device instructing the operation of the affected side of a first specific part of the face. Furthermore, the processing circuit 12 (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured using one or more points on a second specific portion of the face, the second specific portion of the face being different from the first specific portion; (1-1) a sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points on a second specific part of the face; (1-2) A sub-step (1-2) of calculating a left-right ratio of the maximum value of the measured fluctuation value with respect to the non-moving side of the second specific part; (2) A step (2) of calculating an average left / right ratio for the left / right ratio calculated in the step (1); (4) issuing a second notification indicating non-stationary if the calculated average left-right ratio is greater than a given non-stationary threshold; and The purpose of this project is to carry out the following:

[0096] The rehabilitation support system 2' according to the third embodiment can effectively perform the rehabilitation of facial organs by utilizing a function for appropriately evaluating the short-term movement of each facial organ or part from the perspective of left-right ratio.

[0097] 4. [Fourth embodiment] Following the third embodiment, a fourth preferred embodiment of the present disclosure will be disclosed. The rehabilitation support system according to the fourth embodiment is substantially similar to the rehabilitation support system according to the third embodiment. Therefore, the following description will focus on the differences between the two.

[0098] 4.1. [System configuration] A rehabilitation support system 2' according to the fourth embodiment has the same system configuration as the rehabilitation support system according to the third embodiment shown in FIG.

[0099] 4.2.[System Operation] The operation of the rehabilitation support system 2' by an operator of the rehabilitation support system 2' according to the fourth embodiment, and the corresponding behavior of the rehabilitation support system 2' will be described below, focusing on the differences between the third and fourth embodiments.

[0100] First, the operator of the rehabilitation support system 2 ′ starts the operation of the rehabilitation support system 2 ′ on the mobile terminal 6 .

[0101] 4.2.1. [Movement determination in rehabilitation support systems] The rehabilitation support system 2' displays a message on the display screen of the mobile terminal 6 stating "Move a specific part of the face". The operator of the rehabilitation support system 2' tries to make a movement in accordance with the display. If the rehabilitation support system 2' judges that the specific part is moving sufficiently, the display screen of the rehabilitation support system 2' displays, for example, a message stating "A specific part of the face is moving sufficiently". In this way, when the movement of the specific part is sufficient and appropriate, the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' performs the following [variation value measurement and movement judgment processing]. FIG. 11E is a diagram showing a flowchart representing the following variation value measurement and movement judgment processing. The following "variation value measurement" processing is the same as the [variation value measurement processing] performed by the processing circuit 12 of the mobile terminal 6 in the facial movement evaluation system 2 according to the second embodiment. The movement of the specific part is sufficient and appropriate means that the measured measurement value is greater than a given movement threshold value (step (2)). The processing circuit 12 displays, via the interface device 10, a message that the movement is appropriate (for example, that "a specific part of the face is moving sufficiently") in accordance with the first notification issued in step (2).

[0102] [Variation measurement and operation determination processing] [1] Step (1) of measuring the variation of one or more points on a specific part of the face within a predetermined short period of time. [2] Step (2) of issuing a first notification indicating that operation is occurring if the measured variation value is greater than a given operation threshold.

[0103] In the above-mentioned fluctuation value measurement process in the rehabilitation support system 2' according to the fourth embodiment, the specific parts of the face are, for example, the right eyebrow, the left eyebrow, the right eye, the left eye, the right cheek, the left cheek, the right corner of the mouth, the left corner of the mouth, the right nasal wing, the left nasal wing, the lips, the neck, the lower jaw, etc. Also, the "predetermined short period" is a period of time sufficient for performing simple facial exercises, such as 4 seconds, 6 seconds, etc.

[0104] Further, the variation value measured using one or more points on a particular part of the face may be, for example, the average movement distance of a plurality of predetermined points on the particular part of the face; - the average movement speed of a number of predetermined points on a particular part of the face; Variation in the area formed by a number of predetermined points on a particular part of the face, Variation in the angle formed by three given points on a particular part of the face, etc.

[0105] As mentioned above, FIGS. 4 to 8 show examples of variations that can be measured using one or more points on a particular part of the face.

[0106] The rehabilitation support system 2' according to the fourth embodiment can perform the "variation value measurement process" simultaneously for a plurality of specific facial regions. Furthermore, the rehabilitation support system 2' according to the fourth embodiment can perform the "variation value measurement process" in parallel so as to simultaneously measure a plurality of variation values ​​(for example, "average movement distance of a plurality of predetermined points on a specific facial region" and "average movement speed of a plurality of predetermined points on a specific facial region") for a certain facial region.

[0107] 4.2.2. [Caution notification in rehabilitation support systems] In addition, when the operator of the rehabilitation support system 2' tries to move in accordance with the display of "Move a specific part of the face", if the rehabilitation support system 2' judges that another specific part has moved (is interlocked), for example, "Another specific part of the face has also moved" is displayed on the display screen of the rehabilitation support system 2'. When a second specific part different from the first specific part that is the subject of rehabilitation is interlocked, the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' performs the following [variation value measurement and attention notification processing]. FIG. 11F is a diagram showing a flowchart representing the following variation value measurement and attention notification processing. The following "variation value measurement" processing is the same as the variation value measurement of the [variation value measurement and motion judgment processing] executed by the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' according to the present embodiment (fourth embodiment) described above. The fact that the second specific part has been interlocked, i.e., that a corresponding interlocking has occurred, means that the measured variation value is greater than a given non-stationary threshold (step (2)). In accordance with the second notification issued in step (2), the processing circuit 12 displays a warning notification that linkage of other parts is occurring via the interface device 10 (for example, a message to the effect that "another specific part of the face has also moved").

[0108] [Variation measurement and warning notification processing] [1] Step (1) of measuring variation values ​​within a predetermined short period of time for one or more points on a second specific portion of the face, different from the first specific portion. [2] Step (2) of issuing a second notification indicating non-stationary if the measured variation value is greater than a given non-stationary threshold.

[0109] In the above-mentioned variation value measurement and attention notification process in the rehabilitation support system 2' according to the fourth embodiment, the second specific part of the face is a part different from the first specific part that is the subject of rehabilitation, and is, for example, the right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, right corner of the mouth, left corner of the mouth, right nasal wing, left nasal wing, lips, neck, lower jaw, etc. In addition, the "predetermined short period" is a period of time sufficient for performing simple facial exercises, such as 4 seconds, 6 seconds, etc.

[0110] Furthermore, the variation value measured using one or more points on a second specific part of the face may be, for example, an average movement distance of a plurality of predetermined points on a second particular part of the face; - an average movement speed of a plurality of predetermined points on a second specific part of the face; Variation in the area formed by a plurality of predetermined points on a second specific part of the face; Variation in the angle formed by three given points with respect to a second specific part of the face; etc.

[0111] As noted above, FIGS. 4-8 show examples of variations that may be measured using one or more points on the second particular region of the face.

[0112] The rehabilitation support system 2' according to the fourth embodiment can simultaneously perform "variation value measurement and attention notification processing" for a plurality of specific facial regions. Furthermore, the rehabilitation support system 2' according to the fourth embodiment can perform "variation value measurement and attention notification processing" in parallel so as to simultaneously measure a plurality of variation values ​​(for example, "average movement distance of a plurality of predetermined points on a second specific facial region" and "average movement speed of a plurality of predetermined points on a second specific facial region") for a second specific facial region.

[0113] [About output and display of result messages] The rehabilitation support system 2' according to the fourth embodiment can display an evaluation message as a result of the motion judgment process and a warning message as a result of the warning notification process on the display surface. First, for example, when the motion of a specific part of the face (hereinafter referred to as "part A") is sufficient and appropriate, "Part A is moving appropriately" is displayed on the display surface. Also, when a link occurs in a second specific part (hereinafter referred to as "part B") different from the specific part of the face (part A), "Part B is also moving at the same time" is displayed on the display surface.

[0114] 4.3. [Summary] The rehabilitation support system 2' according to the fourth embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device 10. Furthermore, the processing circuit 12 displays a message instructing the operation of a specific part of the face via the interface device 10. Furthermore, the processing circuit 12 (1) measuring a variation value within a predetermined short period of time for one or more points on a specific part of the face; (2) issuing a first notification indicating that the device is operating if the measured variation value is greater than a given operating threshold; The purpose of this project is to carry out the following:

[0115] Moreover, the rehabilitation support system 2' according to the fourth embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired via the interface device 10. Furthermore, the processing circuit displays a message instructing the operation of a first specific part of the face via the interface device 10. Furthermore, the processing circuit 12 (1) measuring a variation value within a predetermined short period of time for one or more points on a second specific part of the face, the second specific part being different from the first specific part; (2) issuing a second notification of non-stationary if the measured variation value is greater than a given non-stationary threshold; The purpose of this project is to carry out the following:

[0116] The rehabilitation support system 2' of the fourth embodiment can effectively perform the rehabilitation of facial organs by utilizing a function that appropriately grasps the short-term movement of each facial organ and part in terms of fluctuation value (amount of fluctuation).

[0117] 5. [Other embodiments] As described above, the first and second embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate.

[0118] The facial movement evaluation system 2 according to the second embodiment can also measure the movement distance and average movement speed for each of all points on the face (for example, 122 points in this disclosure) within a predetermined short period of time. By adding up the movement distances for all points on the face or taking an arithmetic average, the value of the movement of the entire face can be calculated.

[0119] The facial movement evaluation system 2 according to the first and second embodiments, and the rehabilitation support system 2' according to the third and fourth embodiments can store data related to the movements of facial parts over time. Using the data stored over time, the facial movement evaluation system 2 according to the first embodiment can evaluate changes in facial movements over time. Also, the rehabilitation support system 2' according to the second embodiment can use the data stored over time to determine whether or not rehabilitation is effective over time.

[0120] In addition, the accompanying drawings and detailed description are provided to explain the embodiments. Therefore, among the components described in the accompanying drawings and detailed description, not only components essential for solving the problem but also components that are not essential for solving the problem in order to illustrate the above technology may be included. Therefore, the fact that the non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately being essential.

[0121] Furthermore, since the above-described embodiments are intended to illustrate the technology in the present disclosure, various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Explanation of symbols]

[0122] 2···Facial movement evaluation system, 2'···Rehabilitation support system, 4···App ​​store server, 6···Mobile terminal, 8···External network, 10···Interface device, 12···Processing circuit, 14···Memory.

Claims

1. 1. A facial movement assessment system comprising: an interface device and a processing circuit; the processing circuit operates a face recognition system that sets a plurality of predetermined points on a face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired through the interface device; Furthermore, the processing circuitry (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of a face using one or more points related to the specific part of the face; (1-1) a sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points related to a specific part of the face; (1-2) a sub-step (1-2) of calculating a left-right ratio of the maximum value of the measured variation value with respect to the healthy side for the specific site; (2) A step (2) of calculating an average left-right ratio for the left-right ratio calculated in the step (1); The present invention is directed to Facial movement assessment system.

2. The specific part of the face is an eyebrow, The maximum value of the variation that can be measured using one or more points on a specific part of the face is A total distance between two points of a predetermined set of two points in the vicinity of the eyebrows when the total value of the distance between the two points is maximized; and The maximum moving speed for a number of predetermined points in the vicinity of the eyebrows, Including, The facial movement assessment system of claim 1 .

3. The specific part of the face is an eye, The maximum value of the variation that can be measured using one or more points on a specific part of the face is The difference in the area of ​​the inside of the eyelids when the eyes are closed (at their narrowest) and when they are open (at rest) Including, The facial movement assessment system of claim 1 .

4. The specific part of the face is the cheek, The maximum value of the variation that can be measured using one or more points on a specific part of the face is The maximum total movement distance of a plurality of predetermined points in the vicinity of the cheek; and The maximum value of the average movement speed of a plurality of predetermined points in the vicinity of the cheek; The difference in the angle between the eyebrow point and the nose point on the midline and a specified point near the cheek at rest and during the mouth corner lifting movement Including, The facial movement assessment system of claim 1 .

5. The specific part of the face is a corner of the mouth, The maximum value of the variation that can be measured using one or more points on a specific part of the face is The maximum value of the total movement distance of a plurality of predetermined points in the vicinity of the corners of the mouth; The maximum value of the average movement speed of a plurality of predetermined points in the vicinity of the corners of the mouth; The difference in the angle between the eyebrows and the nose tip on the midline and a specified point near the corner of the mouth when at rest and when the corner of the mouth is being raised; The difference in the angle between the eyebrows and nose on the midline and the lip on the midline at rest and during the movement of raising the corners of the mouth. Including, The facial movement assessment system of claim 1 .

6. The specific part of the face is a nose wing, The maximum value of the variation that can be measured using one or more points on a specific part of the face is The maximum value of the total movement distance of a plurality of predetermined points in the vicinity of the nasal ala; The maximum value of the average moving speed of a number of predetermined points near the nose wing and Including, The facial movement assessment system of claim 1 .

7. 1. A facial movement assessment system comprising: an interface device and a processing circuit; the processing circuit operates a face recognition system that sets a plurality of predetermined points on a face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired through the interface device; Furthermore, the processing circuitry (1) measuring the variation of one or more points on a specific part of the face within a predetermined short period of time; The present invention is directed to Facial movement assessment system.

8. The specific parts of the face include the right eyebrow, the left eyebrow, the right eye, the left eye, the right cheek, the left cheek, the right corner of the mouth, the left corner of the mouth, the right nasal wing, the left nasal wing, the lips, the neck, and the lower jaw; The variation measured using one or more points on a particular part of the face is - Average movement distance of a number of predetermined points on a particular part of the face, - Average moving speed of a plurality of predetermined points on a predetermined specific part; - Variation in the area formed by a number of predetermined points on a particular part of the face, and - Variation in the angle formed by three specific points on a particular part of the face The facial movement evaluation system of claim 7 , comprising:

9. A rehabilitation support system, comprising: an interface device and a processing circuit; the processing circuit operates a face recognition system that sets a plurality of predetermined points on a face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired through the interface device; Further, the processing circuitry is configured to present a message via the interface device instructing a movement of a particular part of the face on the affected side; Furthermore, the processing circuitry (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of a face using one or more points related to the specific part of the face; (1-1) a sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points related to a specific part of the face; (1-2) a sub-step (1-2) of calculating a right-left ratio of the maximum measured fluctuation value with respect to the non-moving side of the specific part; (2) A step (2) of calculating an average left / right ratio for the left / right ratio calculated in the step (1); (3) issuing a first notification indicating that a motion is occurring if the calculated average left-right ratio is greater than a given motion threshold; and The present invention is directed to Rehabilitation support system.

10. The specific part of the face is an eye, The maximum value of the variation that can be measured using one or more points on a specific part of the face is The difference in the area of ​​the inside of the eyelids when the eyes are closed (at their narrowest) and when they are open (at rest) Including, The rehabilitation support system according to claim 9.

11. The specific part of the face is a corner of the mouth, The maximum value of the variation that can be measured using one or more points on a specific part of the face is The maximum value of the total movement distance of a plurality of predetermined points in the vicinity of the corners of the mouth; The maximum value of the average movement speed of a plurality of predetermined points in the vicinity of the corners of the mouth; The difference in the angle between the eyebrows and the nose tip on the midline and a specified point near the corner of the mouth when at rest and when the corner of the mouth is being raised; The difference in the angle between the eyebrows and nose on the midline and the lip on the midline at rest and during the movement of raising the corners of the mouth. Including, The rehabilitation support system according to claim 9.

12. A rehabilitation support system, comprising: an interface device and a processing circuit; the processing circuit operates a face recognition system that sets a plurality of predetermined points on a face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired through the interface device; Further, the processing circuitry is configured to present a message via the interface device instructing a movement of a first particular portion of the face on the affected side; Furthermore, the processing circuitry (1) Step (1) of performing the following sub-steps (1-1) and (1-2) for all of the variation values ​​that can be measured using one or more points on a second specific portion of the face, the second specific portion of the face being different from the first specific portion; (1-1) a sub-step (1-1) of measuring a variation value within a predetermined short period of time for one or more points related to a second specific part of the face; (1-2) a sub-step (1-2) of calculating a left-right ratio of the maximum value of the measured fluctuation value with respect to the non-moving side of the second specific part; (2) A step (2) of calculating an average left / right ratio for the left / right ratio calculated in the step (1); (4) issuing a second notification indicating non-stationary if the calculated average left-right ratio is greater than a given non-stationary threshold; and The present invention is directed to Rehabilitation support system.

13. the first particular part of the face is an eye; the second specific part of the face is a corner of the mouth; The maximum variation value that can be measured using one or more points on the second specific part of the face is The maximum value of the total movement distance of a plurality of predetermined points in the vicinity of the corners of the mouth; The maximum value of the average movement speed of a plurality of predetermined points in the vicinity of the corners of the mouth; The difference in the angle between the eyebrows and the nose tip on the midline and a specified point near the corner of the mouth when at rest and when the corner of the mouth is being raised; The difference in the angle between the eyebrows and nose on the midline and the lip on the midline at rest and during the movement of raising the corners of the mouth. Including, The rehabilitation support system according to claim 12.

14. the first particular part of the face is an eye; the second specific part of the face is an eyebrow; The maximum variation value that can be measured using one or more points on the second specific part of the face is A total distance between two points of a predetermined set of two points in the vicinity of the eyebrows when the total value of the distance between the two points is maximized; and The maximum moving speed for a number of predetermined points in the vicinity of the eyebrows, Including, The rehabilitation support system according to claim 12.

15. the first specific part of the face is a corner of the mouth; the second specific part of the face is an eye; The maximum variation value that can be measured using one or more points on the second specific part of the face is The difference in the area of ​​the inside of the eyelids when the eyes are closed (at their narrowest) and when they are open (at rest) Including, The rehabilitation support system according to claim 12.

16. the first specific part of the face is a corner of the mouth; the second specific part of the face is an eyebrow; The maximum variation value that can be measured using one or more points on the second specific part of the face is A total distance between two points of a predetermined set of two points in the vicinity of the eyebrows when the total value of the distance between the two points is maximized; and The maximum moving speed for a number of predetermined points in the vicinity of the eyebrows, Including, The rehabilitation support system according to claim 12.

17. A rehabilitation support system, comprising: an interface device and a processing circuit; the processing circuit operates a face recognition system that sets a plurality of predetermined points on a face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired through the interface device; Further, the processing circuitry indicates, via the interface device, a message instructing a particular facial part to be moved; Furthermore, the processing circuitry (1) measuring a variation value within a predetermined short period of time for one or more points on a specific part of the face; (2) issuing a first notification indicating that the device is operating if the measured variation value is greater than a given operating threshold; The present invention is directed to Rehabilitation support system.

18. The specific parts of the face include the right eyebrow, the left eyebrow, the right eye, the left eye, the right cheek, the left cheek, the right corner of the mouth, the left corner of the mouth, the right nasal wing, the left nasal wing, the lips, the neck, and the lower jaw; The variation measured using one or more points on a particular part of the face is - Average movement distance of a number of predetermined points on a particular part of the face, - Average moving speed of a plurality of predetermined points on a predetermined specific part; - Variation in the area formed by a number of predetermined points on a particular part of the face, and - Variation in the angle formed by three specific points on a particular part of the face The rehabilitation support system according to claim 17, comprising:

19. A rehabilitation support system, comprising: an interface device and a processing circuit; the processing circuit operates a face recognition system that sets a plurality of predetermined points on a face surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on face-related data acquired through the interface device; Further, the processing circuitry is configured to present a message via the interface device instructing a movement of a first particular part of a face; Furthermore, the processing circuitry (1) measuring a variation value within a predetermined short period of time for one or more points on a second specific part of the face, the second specific part being different from the first specific part; (2) issuing a second notification indicating non-stationary if the measured variation value is greater than a given non-stationary threshold; The present invention is directed to Rehabilitation support system.

20. The specific parts of the face include the right eyebrow, the left eyebrow, the right eye, the left eye, the right cheek, the left cheek, the right corner of the mouth, the left corner of the mouth, the right nasal wing, the left nasal wing, the lips, the neck, and the lower jaw; The variation measured using one or more points on a particular part of the face is - Average movement distance of a number of predetermined points on a particular part of the face, - Average moving speed of a plurality of predetermined points on a predetermined specific part; - Variation in the area formed by a number of predetermined points on a particular part of the face, and - Variation in the angle formed by three specific points on a particular part of the face The rehabilitation support system according to claim 19, comprising:

Images