Facial motion evaluation system and rehabilitation assistance system

Abstract

Disclosed is a facial motion evaluation system for appropriately evaluating short-time movement of a facial organ. This system comprises an interface device and a processing circuit. The processing circuit operates a face recognition system that sets, on the basis of face-related data acquired via the interface device, a plurality of prescribed points in a three-dimensional relative coordinate system which is configured with reference to the face per se on the surface of the face. The processing circuit executes: (1) a step for performing, with respect to a specific part of the face and for all variable values that can be measured using one or more points relating to the specific part of the face, (1-1) a sub-step for measuring variable values within a predetermined short time with respect to the one or more points relating to the specific part of the face, and (1-2) a sub-step for calculating left-right ratios of the maximum values of the measured variable values, with reference to a healthy side of the specific part; and (2) a step for calculating an average left-right ratio of the left-right ratios calculated in step (1).

Description

Facial movement evaluation system and rehabilitation support system

[0001] The present disclosure relates to a facial movement evaluation system that evaluates the movement of each facial organ, and a rehabilitation support system that supports the rehabilitation of the movement of each facial organ.

[0002] There are many diseases that cause changes in facial movement. A typical example is facial nerve paralysis, which is a disease that usually results from damage to one side of the facial nerve, resulting in a weakening or loss of facial expression on one side. For many years, subjective evaluation methods based on the experience of the evaluator have been used to assess the degree of paralysis. Examples of subjective evaluation methods include the 40-point method (Yanagihara method).

[0003] In addition to facial paralysis, Parkinson's disease and depression can cause a lack of facial expression, known as mask facies.

[0004] Because facial movement varies from person to person, regardless of whether or not a patient has a disease, it has been difficult to rely on machines to evaluate facial movement. However, quantitative and reliable methods for evaluating facial movement have always been needed by both patients and clinicians. Based on this, various computer-based facial movement evaluation systems have been proposed in recent years, thanks to advances in hardware, including smartphones and tablet devices, and software, including image processing.

[0005] Japanese Patent Application Laid-Open No. 2018-36734 Japanese Patent Application Laid-Open No. 2022-533464

[0006] However, most of the facial movement evaluation systems proposed to date evaluate the static appearance of the face, for example, by indicating the relative distance and position of each organ, such as the nose, eyes, cheeks, and mouth. In contrast, in facial nerve diagnosis, a means is required that can evaluate the movement of each organ over a short period of time (e.g., a few seconds), or even over an instant. Furthermore, a means that can evaluate the specific movement of each organ over a short period of time would be very useful in a rehabilitation support system for facial nerve paralysis.

[0007] The present disclosure presents a facial movement evaluation system that appropriately evaluates the movement of each facial organ over a short period of time, and a rehabilitation support system that effectively performs rehabilitation of facial organs and parts by utilizing the function of appropriately evaluating the movement of each facial organ over a short period of time.

[0008] The present invention provides a facial movement evaluation system that solves the above-mentioned problems. The facial movement evaluation system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system configured with the face itself as a reference based on facial data acquired via the interface device. The processing circuit further executes the following steps (1) for a specific facial region: (1) performing the following substeps (1-1) and (1-2) for all of the variation values ​​that can be measured using one or more points related to the specific facial region; (1-1) measuring variation values ​​within a predetermined short period for one or more points related to the specific facial region; (1-2) calculating the maximum of the measured variation values ​​on the healthy side for the specific facial region; and (2) calculating an average left-right ratio for the left-right ratio calculated in step (1).

[0009] The facial movement evaluation system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device. The processing circuit also executes the following steps: (1) measuring fluctuations of one or more points related to specific parts of the face within a predetermined short period of time.

[0010] The rehabilitation support system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device. Furthermore, the processing circuit displays a message via the interface device instructing the movement of a specific part of the face on the affected side. Furthermore, the processing circuit executes the following steps (1) to (1) for performing the following substeps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of the face using one or more points related to the specific part of the face: (1-1) substep (1-1) of measuring variation values ​​for one or more points related to the specific part of the face within a predetermined short period of time; (1-2) substep (1-2) of calculating the left-right ratio of the maximum of the measured variation values ​​based on the non-moving side of the specific part; (2) step (2) of calculating the average left-right ratio of the left-right ratios calculated in step (1); and (3) step (3) of issuing a first notification indicating that movement is occurring if the calculated average left-right ratio is greater than a given movement threshold.

[0011] The rehabilitation support system of the present disclosure includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device. Furthermore, the processing circuit displays a message via the interface device instructing a movement of an affected side of a first specific facial region. Furthermore, the processing circuit executes the following steps (1) (1) for performing the following substeps (1-1) and (1-2) for all of the variation values ​​that can be measured for a second specific region of the face, different from the first specific region, using one or more points on the second specific region of the face; (1-1) substep (1-1) of measuring variation values ​​for one or more points on the second specific region of the face within a predetermined short period of time; (1-2) substep (1-2) of calculating the left-right ratio of the maximum of the measured variation values, based on the non-moving side of the second specific region; (2) step (2) of calculating the average left-right ratio of the left-right ratios calculated in step (1); and (4) step (4) of issuing a second notification indicating non-stationary if the calculated average left-right ratio is greater than a given non-stationary threshold.

[0012] The rehabilitation support system of the present disclosure also includes an interface device and a processing circuit. The processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device. Furthermore, the processing circuit displays a message via the interface device instructing a movement of a specific facial part. The processing circuit also executes the following steps: (1) measuring a variation value of one or more points related to the specific facial part within a predetermined short period of time; and (2) issuing a first notification indicating movement if the measured variation value is greater than a given movement threshold.

[0013] The rehabilitation support system of the present disclosure further includes an interface device and a processing circuit. The processing circuit operates a facial recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device. The processing circuit also displays a message via the interface device instructing a movement of a first specific facial region. The processing circuit also executes the following steps: (1) measuring fluctuation values ​​within a predetermined short period of time for one or more points on a second specific facial region different from the first specific facial region; and (2) issuing a second notification indicating immobility if the measured fluctuation value is greater than a given immobility threshold.

[0014] By using the facial movement evaluation system of the present disclosure, the movement of each facial organ or part over a short period of time can be appropriately evaluated from the perspective of left-right ratio and can be appropriately understood from the perspective of the amount of fluctuation. Furthermore, the rehabilitation support system of the present disclosure can effectively carry out rehabilitation of facial organs and parts by utilizing the function of appropriately evaluating the movement of each facial organ or part over a short period of time from the perspective of left-right ratio and the function of appropriately understanding from the perspective of the amount of fluctuation.

[0015] FIG. 1 is a system configuration diagram of a facial movement evaluation system according to a first embodiment. FIG. 2 is a diagram showing a plurality of (e.g., 1,220) predetermined point clouds (hereinafter referred to as "point clouds generated by the face recognition system") in a three-dimensional relative coordinate system configured with the face as a reference, which is set by a three-dimensional face recognition system used in the present disclosure. FIG. 3 is a diagram schematically showing an infrared sensor, a front camera, and an infrared irradiation unit of a smartphone used by the face recognition system used in the present disclosure. FIGS. 4(a-1) and 4(a-2) are diagrams showing the distance between two predetermined pairs of two points near the eyebrows in the point cloud generated by the face recognition system. FIGS. 4(a-3) and 4(a-4) are diagrams showing predetermined points near the eyebrows in the point cloud generated by the face recognition system. FIG. 5 is a diagram showing the inside of the eyelid in the point cloud generated by the face recognition system. FIG. 6A is a diagram showing a plurality of predetermined points near the cheeks in the point cloud generated by the face recognition system. FIG. 6B(b-1) is a diagram showing the angle at rest formed by a point between the eyebrows and a point on the nose on the midline and a predetermined point near the cheek in a point cloud obtained by a facial recognition system. FIG. 6B(b-2) is a diagram showing the angle during a mouth corner lift movement formed by a point between the eyebrows and a point on the nose on the midline and a predetermined point near the cheek in a point cloud obtained by a facial recognition system. FIG. 7A(a-1) is a diagram showing a plurality of predetermined points near the corners of the mouth in a point cloud obtained by a facial recognition system. FIG. 7A(a-2) is an enlarged view of the vicinity of the predetermined plurality of points near the corners of the mouth in FIG. 7A(a-1). FIG. 7B(b-1) is a diagram showing the angle at rest formed by a point between the eyebrows and a point on the nose on the midline and a predetermined point near the corners of the mouth in a point cloud obtained by a facial recognition system. Fig. 7B (b-2) is a diagram showing angles during a mouth corner lifting movement formed by a point between the eyebrows and a point on the tip of the nose on the midline and a predetermined point near the corners of the mouth in a point cloud obtained by a facial recognition system. Fig. 7C (c-1) is a diagram showing angles at rest formed by a point between the eyebrows and a point on the tip of the nose on the midline and a point on the lips on the midline in a point cloud obtained by a facial recognition system. Fig. 7C (c-2) is a diagram showing angles during a mouth corner lifting movement formed by a point between the eyebrows and a point on the tip of the nose on the midline and a point on the lips on the midline in a point cloud obtained by a facial recognition system. Fig. 8 is a diagram showing a plurality of predetermined points near the ala of the nose in a point cloud obtained by a facial recognition system.Fig. 9 is a system configuration diagram of a rehabilitation support system according to a second embodiment. Fig. 10 is a diagram showing an example of a screen of the rehabilitation support system according to the second embodiment. Fig. 11A is a diagram showing a flowchart illustrating evaluation value calculation processing in the facial movement evaluation system according to the first embodiment. Fig. 11B is a diagram showing a flowchart illustrating variation value measurement processing in the facial movement evaluation system according to the second embodiment. Fig. 11C is a diagram showing a flowchart illustrating evaluation value calculation and movement determination processing in the rehabilitation support system according to the third embodiment. Fig. 11D is a diagram showing a flowchart illustrating evaluation value calculation and attention notification processing in the rehabilitation support system according to the third embodiment. Fig. 11E is a diagram showing a flowchart illustrating variation value measurement and movement determination processing in the rehabilitation support system according to the fourth embodiment. Fig. 11F is a diagram showing a flowchart illustrating variation value measurement and attention notification processing in the rehabilitation support system according to the fourth embodiment.

[0016] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0017] The inventors have provided the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0018] 1. First Embodiment A preferred first embodiment of the present disclosure will now be described with reference to the accompanying drawings. The facial movement evaluation system according to the first embodiment is useful for evaluating facial movement in facial nerve paralysis.

[0019] 1 is a diagram showing the overall system configuration of a facial movement evaluation system 2 according to the first embodiment. The facial movement evaluation system 2 according to the first embodiment runs on a mobile terminal 6.

[0020] As shown in FIG. 9, the rehabilitation support system 2' according to the second embodiment of the present disclosure can also operate on the mobile terminal 6 on which the facial movement evaluation system 2 according to the first embodiment operates.

[0021] The mobile terminal 6 is a portable computing device such as a tablet terminal or a smartphone. The mobile terminal 6 includes an imaging element and imaging control means for controlling the imaging element. That is, for example, as shown in FIG. 3 , the mobile terminal 6 includes an infrared sensor 20, a front camera 22, and an infrared irradiation unit 24. The mobile terminal 6 is connected to an external network 8. Note that the mobile terminal 6 is not limited to portable computing devices such as a tablet terminal or a smartphone, and may be, for example, a PC (personal computer) or a workstation computer.

[0022] The external network 8 is, for example, the Internet, and is connected to the mobile terminal 6 via an interface device 10 (described below). The facial movement evaluation system 2 is based on (for example) a native app running on the mobile terminal 6, and therefore the mobile terminal 6 builds the facial movement evaluation system 2 on the mobile terminal 6 by downloading an application program related to the facial movement evaluation system from an app store server 4 connected to the external network 8. Note that the facial movement evaluation system 2 does not have to be based on a native app, and may be based on a web app or a hybrid app.

[0023] As shown in FIG. 1, the mobile terminal 6 includes an interface device 10, a processing circuit 12, and a memory 14.

[0024] The interface device 10 is an interface unit that can exchange data with the outside and includes a network terminal, a video input / output terminal, a USB terminal, a camera, a pointing device, a keyboard, a mouse, etc. The data here is, for example, image data relating to a face. After being acquired, this data can be recorded in a storage unit such as a memory 14 inside the mobile terminal 6.

[0025] The processing circuit 12 is configured by a processor. The processor here includes a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Various processes of the facial movement evaluation system 2 according to the first embodiment are realized by the processing circuit 12 executing various programs.

[0026] The memory 14 is a rewritable storage unit within the mobile terminal 6, and is configured, for example, by a RAM (Random Access Memory) including a large number of semiconductor memory elements. The memory 14 temporarily stores specific computer programs, variable values, parameter values, etc., used by the processing circuit 12 when executing various processes. The memory 14 may also include a so-called ROM (Read Only Memory). The ROM pre-stores application programs that realize various processes of the facial movement evaluation system 2 described below. The processing circuit 12 reads the application programs from the ROM and loads them into the RAM, thereby enabling the processing circuit 12 to execute the application programs.

[0027] The facial movement evaluation system 2 according to the first embodiment is constructed using a computer language such as Swift. However, computer languages ​​that can be used to construct the facial movement evaluation system 2 are not limited to these, and other computer languages ​​may also be used.

[0028] Furthermore, the processing circuit 12 of the mobile terminal 6 according to the first embodiment operates a face recognition system that sets multiple predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial image data acquired via the interface device 10. The face recognition system here is, for example, a system based on "ARFacegeometry" (Class name) used in FaceID, a three-dimensional facial authentication system provided by Apple Inc. In addition to processing facial images using a front camera 22 (as shown in FIG. 3 ), the system also irradiates infrared light from an infrared irradiator 24, which then uses an infrared sensor 20 to stereoscopically grasp the face, thereby performing real-time three-dimensional tracking. "ARFacegeometry" is a development kit provided by Apple Inc. The face recognition system here is capable of acquiring the three-dimensional coordinates of 1,220 points arranged on the face. Furthermore, in this face recognition system, coordinates are determined in a relative coordinate system that uses the face itself as the reference, so the three-dimensional coordinates of points placed on the face surface do not change even if the position or orientation of the face changes. Therefore, the face recognition system of the present disclosure sets 1,220 predetermined points on the face surface in a three-dimensional relative coordinate system that is configured using the face itself as the reference.

[0029] 2 is a diagram showing a plurality of (e.g., 1220) predetermined points (hereinafter referred to as "point clouds generated by the face recognition system") in a three-dimensional relative coordinate system configured with the face as a reference, which is set by the face recognition system used in the present disclosure. The 1220 points are displayed on the display surface of the mobile terminal 6 as a mirror image of the face of the operator facing the display surface of the mobile terminal 6. Although not shown in FIG. 2, each of the 1220 predetermined points is assigned a fixed number (see FIGS. 4 to 8).

[0030] The face recognition system to be used in the present disclosure is not limited to a system based on ARFacegeometry used in FaceID, a three-dimensional face authentication system provided by Apple, but may be any face recognition system that can set a large number of points on the surface of the face in a relative coordinate system constructed with the face itself as the reference.

[0031] 1.2 System Operation The operation of the facial movement evaluation system 2 by the operator of the facial movement evaluation system 2, i.e., the person to be evaluated, and the operation of the facial movement evaluation system 2 in response thereto will be described below.

[0032] The operator of the facial movement evaluation system 2 first starts operation of the facial movement evaluation system 2 on the mobile terminal 6. Here, the healthy side (and affected side) of the operator (subject) can be set automatically or manually in the facial movement evaluation system 2.

[0033] The operator (evaluated person) performs facial movements such as closing both eyes, closing one eye at a time, raising the eyebrows, and smiling with the corners of the mouth raised while looking at the display surface of the mobile terminal 6. At this time, the facial movement evaluation system 2 performs the following evaluation value calculation process based on real-time facial data acquired via the interface device 10, such as the front camera 22 and the infrared sensor 20, and further based on a point cloud generated by a face recognition system from the facial data. Fig. 11A is a diagram showing a flowchart illustrating the following evaluation value calculation process.

[0034] [Evaluation Value Calculation Process] [1] Step (1) of executing the following substeps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific facial region using one or more points related to the specific facial region. [1-1] Substep (1-1) of measuring variation values ​​for one or more points related to the specific facial region within a predetermined short period of time. [1-2] Substep (1-2) of calculating a left-right ratio of the maximum measured variation values ​​for the specific region, using the healthy side as the reference. [2] Step (2) of calculating an average left-right ratio for the left-right ratios calculated in step (1).

[0035] The average right-left ratio calculated in [2] above is the evaluation value of the movement of that specific part. In other words, the average right-left ratio for the part indicates the degree of movement of the affected side relative to the healthy side of the part, and is used as the "evaluation value."

[0036] In the facial movement evaluation system 2 according to the first embodiment, in the above-described evaluation value calculation process, specific facial parts are, for example, the eyebrows, eyes, cheeks, corners of the mouth, wings of the nose, etc. The "predetermined short period" is a period of time sufficient for performing simple facial movements, such as 3 seconds or 5 seconds.

[0037] Furthermore, examples of maximum values ​​of variation that can be measured using one or more points related to a specific part of the face are as follows: (1-1), (1-2), (2-1), (3-1), (3-2), (3-3), (4-1), (4-2), (4-3), (4-4), (5-1), (5-2), and (5-3). ((1) When the specific part is the "eyebrow") (1-1) The total distance when the sum of the distances between a predetermined number of pairs of two points near the eyebrow is maximized. (1-2) The maximum movement speed for a predetermined number of points near the eyebrow. ((2) When the specific part is the "eye") (2-1) The difference in the area inside the eyelid between when the eye is at its smallest when closed and when the eye is at rest when open. ((3) When the specific part is the "cheek") (3-1) The maximum value of the total movement distance for a predetermined number of points near the cheek. (3-2) The maximum value of the average movement speed of a plurality of specified points near the cheeks. (3-3) The difference in angle between the point between the eyebrows and the tip of the nose on the midline and a specified point near the cheeks, at rest and during the movement of lifting the corners of the mouth. ((4) When the specific part is the "corner of the mouth") (4-1) The maximum value of the total movement distance of a plurality of specified points near the corners of the mouth. (4-2) The maximum value of the average movement speed of a plurality of specified points near the corners of the mouth. (4-3) The difference in angle between the point between the eyebrows and the tip of the nose on the midline and a specified point near the corners of the mouth, at rest and during the movement of lifting the corners of the mouth. (4-4) The difference in the angle between the point between the eyebrows and the point on the nose on the midline and the point on the lips on the midline between at rest and during the movement of lifting the corners of the mouth. ((5) When the specific part is the "ala of the nose") (5-1) The maximum total moving distance of multiple specified points near the ala of the nose. (5-2) The maximum average moving speed of multiple specified points near the ala of the nose.

[0038] Examples of the (maximum) fluctuation values ​​that can be measured using one or more points relating to specific parts of the face, as described above in (1-1) to (5-2), will be described with reference to FIGS.

[0039] [Explanation regarding examples of maximum fluctuation values]

[0040] (1) Regarding the Case Where the Specific Body Part is the "Eyebrow" Figures 4(a-1) and 4(a-2) are diagrams showing distances between multiple pairs of two points near the eyebrows in a point cloud generated by a face recognition system. Figure 4(a-1) shows the distances when the subject is at rest, and Figure 4(a-2) shows the distances when the subject is raised. For example, five pairs of two points are assumed near the left eyebrow: (329-1150), (328-1151), (327-1152), (335-1153), (197-1154), and (198-1155). Points corresponding to these five pairs of points are assumed for the right eyebrow. In Figures 4(a-1) and 4(a-2), only the two pairs, (329-1150) and (328-1151), are connected by a line segment. Figure 4(a-2) is a diagram of a point cloud generated by a face recognition system that can show the "total distance when the sum of the distances between two points for a specified number of pairs of two points near the eyebrows is maximized."

[0041] 4(a-3) and 4(a-4) are diagrams showing a plurality of predetermined points near the eyebrows in a point cloud generated by a facial recognition system. FIG. 4(a-3) shows the subject at rest, while FIG. 4(a-3) shows the subject when the eyebrows are raised. For example, five points, 329, 328, 327, 335, 197, and 198, are assumed as multiple points near the left eyebrow. Points corresponding to these five points are assumed for the right eyebrow. In FIGS. 4(a-3) and 4(a-4), only 329 is circled. FIG. 4(a-4) is a diagram of a point cloud generated by a facial recognition system that may indicate the "maximum movement speed for the plurality of predetermined points near the eyebrows."

[0042] (2) Regarding the case where the specific body part is the "eye" Figure 5 is a diagram showing the inside of the eyelid in a point cloud generated by a face recognition system. That is, the inside of the left eyelid and its area are shown by connecting 24 points, 1085-(sequential number)-1108, around the edge of the left eye. For the right eye, the points corresponding to these 24 points are assumed to represent the inside of the eyelid. The difference between when this area is at its smallest (i.e., when the eye is closed) and when at rest (i.e., when the eye is open) is the "difference in the area of ​​the inside of the eyelid between when the eye is closed, i.e., when the eye is at its smallest, and when at rest, i.e., when the eye is open."

[0043] (3) Case where the specific part is "cheek" FIG. 6A is a diagram showing a plurality of predetermined points near the cheeks in a point cloud generated by a face recognition system. For example, in FIG. 6A, four points 600, 601, 785, and 821 are assumed as multiple points near the right cheek. For the left cheek, points corresponding to these four points are assumed. In FIG. 6A, 600, 601, 785, and 821 are each surrounded by a circle. The maximum value of the total movement distance for each of these four points is the "maximum total movement distance for the predetermined plurality of points near the cheeks." Furthermore, the maximum value of the average movement speed for each of these four points is the "maximum average movement speed for the predetermined plurality of points near the cheeks."

[0044] FIG. 6B (b-1) is a diagram showing the angle at rest formed by the point between the eyebrows and the tip of the nose on the midline and a predetermined point near the cheek in a point cloud obtained by a facial recognition system. FIG. 6B (b-2) is a diagram showing the angle during a mouth corner lifting movement formed by the point between the eyebrows and the tip of the nose on the midline and a predetermined point near the cheek in a point cloud obtained by a facial recognition system. The point between the eyebrows and the tip of the nose on the midline are points 15,7. For the right cheek, the predetermined point near the cheek is point 600. For the left cheek, a point corresponding to point (600) is assumed. The difference in the angle formed by these three points for each cheek between rest and during a mouth corner lifting movement is the "difference in the angle formed by the point between the eyebrows and the tip of the nose on the midline and a predetermined point near the cheek between rest and during a mouth corner lifting movement."

[0045] (4) Regarding the Case Where the Specific Part is the "Corner of the Mouth" Figure 7A(a-1) is a diagram showing a plurality of predetermined points near the corners of the mouth in a point cloud generated by a face recognition system. Figure 7A(a-2) is an enlarged view of the vicinity of the predetermined plurality of points near the corners of the mouth in Figure 7A(a-1). In the vicinity of the right corner of the mouth shown in Figures 7A(a-1) and 7A(a-2), 12 points, 823, 824, 825, 826, 684, 638, 678, 639, 834, 835, 836, and 837, are assumed as multiple points. For the left corner of the mouth, a point cloud corresponding to these 12 points is assumed. 7A(a-1) and 7A(a-2), point groups 823, 824, 825, 826, 684, 638, 678, 639, 834, 835, 836, and 837 are surrounded by lines such as circles. The maximum total movement distance for each of these 12 points is the "maximum total movement distance for a predetermined number of points near the corners of the mouth." Furthermore, the maximum average movement speed for each of these 12 points is the "maximum average movement speed for a predetermined number of points near the corners of the mouth."

[0046] FIG. 7B (b-1) is a diagram showing the angle at rest, determined by a point between the eyebrows and the tip of the nose on the midline, and a predetermined point near the corner of the mouth, in a point cloud obtained by a facial recognition system. FIG. 7B (b-2) is a diagram showing the angle during a mouth corner lifting movement, determined by a point between the eyebrows and the tip of the nose on the midline, and a predetermined point near the corner of the mouth, in a point cloud obtained by a facial recognition system. The points between the eyebrows and the tip of the nose on the midline are points 15,7. For the right corner of the mouth, the predetermined point near the corner of the mouth is point 825. For the left corner of the mouth, a point corresponding to point 825 is assumed. The difference in the angle formed by these three points for each corner of the mouth between rest and during a mouth corner lifting movement is the "difference in the angle between the point between the eyebrows and the tip of the nose on the midline, and a predetermined point near the corner of the mouth, between rest and during a mouth corner lifting movement."

[0047] FIG. 7C (c-1) is a diagram showing the angle at rest formed by the point between the eyebrows and the tip of the nose on the midline and the point on the lip on the midline in a point cloud obtained by a facial recognition system. FIG. 7C (c-2) is a diagram showing the angle formed by the point between the eyebrows and the tip of the nose on the midline and the point on the lip on the midline in a point cloud obtained by a facial recognition system during a mouth corner lifting movement, particularly during a right mouth corner lifting movement. The difference between the angle formed by these three points for each mouth corner at rest and during a mouth corner lifting movement is, particularly for the right mouth corner, the "difference between the angle formed by the point between the eyebrows and the tip of the nose on the midline and the point on the lip on the midline at rest and during a mouth corner lifting movement." Note that for the left mouth corner, the angle during the left mouth corner lifting movement must be used.

[0048] (5) Cases where the specific body part is the "ala of the nose" Figure 8 is a diagram showing a plurality of predetermined points near the ala of the nose in a point cloud generated by a facial recognition system. For example, in Figure 8, eleven points, 652, 655, 732, 743, 729, 651, 653, 747, 760, 727, and 748, are assumed as multiple points near the right ala of the nose. Points corresponding to these eleven points are assumed for the left ala of the nose. In Figure 8, the point clouds 652, 655, 732, and 743, the point clouds 729, 651, 653, and 747, and the point clouds 760, 727, and 748 are each surrounded by lines. The maximum total movement distance for these eleven points on the right and left sides is the "maximum total movement distance for the predetermined plurality of points near the ala of the nose." Furthermore, the maximum value of the average movement speed for these 11 points on each of the right and left sides is the "maximum value of the average movement speed for a predetermined number of points near the ala of the nose."

[0049] [Explanation of maximum fluctuation value example - end]

[0050] [Regarding output and display of evaluation values] The facial movement evaluation system 2 according to the first embodiment can display the "evaluation value" for each specific part of the face as a result of movement evaluation on the display surface of the mobile terminal 6, for example, as follows:

[0051] Here, it is shown that the corners of the mouth and ala of the nose on the affected side are less mobile than other areas.

[0052] The facial movement evaluation system 2 can also calculate an evaluation value for the entire face by averaging the "evaluation values" for all specific parts of the face (e.g., eyebrows, eyes, cheeks, corners of the mouth, and wings of the nose). Therefore, the facial movement evaluation system 2 can display the evaluation value for the entire face as the result of the movement evaluation on the display surface of the mobile terminal 6. In the example of Table 1, the result of the movement evaluation is "(90+90+90+60+60) / 5=78%."

[0053] Furthermore, the facial movement evaluation system 2 can also display the expected degree of facial paralysis depending on the evaluation value. For example, if the evaluation value of the entire face is close to approximately 100%, it can display "No problems with facial movement," and if the evaluation value of the entire face is less than a predetermined threshold (e.g., 90%), it can display "There is a difference in facial movement between the left and right sides."

[0054] 1.3. [Summary] The facial movement evaluation system 2 according to the first embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device 10. The processing circuit 12 further executes the following steps (1) (1) for performing the following substeps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific facial region using one or more points related to the specific facial region: (1-1) substep (1-1) measuring variation values ​​for one or more points related to the specific facial region within a predetermined short period of time; (1-2) substep (1-2) calculating a left-right ratio of the maximum of the measured variation values, with the healthy side of the specific facial region as the reference; and (2) substep (2) calculating an average left-right ratio for the left-right ratio calculated in step (1).

[0055] By using the facial movement evaluation system 2 according to the first embodiment, the movements of each organ or part of the face over a short period of time can be appropriately evaluated from the perspective of the left-right ratio.

[0056] 2. [Second Embodiment] Following the first embodiment, a preferred second embodiment of the present disclosure will be disclosed. Note that the facial movement evaluation system according to the second embodiment is substantially the same as the facial movement evaluation system according to the first embodiment. Therefore, the following description will focus on the differences between the two.

[0057] 2.1 System Configuration A facial movement evaluation system 2 according to the second embodiment has the same system configuration as the facial movement evaluation system according to the first embodiment shown in FIG.

[0058] 2.2 System Operation

[0059] The operation of the facial movement evaluation system 2 by the operator of the facial movement evaluation system 2 in the second embodiment, i.e., the person being evaluated, and the response of the facial movement evaluation system 2 will be explained below, focusing on the differences between the first and second embodiments.

[0060] The operator of the facial movement evaluation system 2 first starts the operation of the facial movement evaluation system 2 on the mobile terminal 6 .

[0061] The operator (evaluee) performs facial movements such as closing both eyes, closing one eye at a time, raising the eyebrows, smiling with the corners of the mouth raised, opening the mouth, and lowering the chin while looking at the display surface of the mobile terminal 6. At this time, the facial movement evaluation system 2 performs the following variation value measurement process based on real-time facial data acquired via the interface device 10, such as the front camera 22 and the infrared sensor 20, and further based on a point cloud generated by a face recognition system from the facial data. Figure 11B is a diagram showing a flowchart illustrating the following variation value measurement process.

[0062] [Variation Value Measurement Process] [1] Step (1) of measuring the variation value of one or more points on a specific part of the face within a predetermined short period of time.

[0063] In the above-described fluctuation value measurement process in the facial movement evaluation system 2 according to the second embodiment, specific parts of the face are, for example, the right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, right corner of the mouth, left corner of the mouth, right ala of the nose, left ala of the nose, lips, neck, lower jaw, etc. The "predetermined short period" is a period of time sufficient for performing simple facial movements, such as 4 seconds or 6 seconds.

[0064] Furthermore, the variation values ​​measured using one or more points on a specific part of the face include, for example, the average movement distance of a plurality of predetermined points on a specific part of the face, the average movement speed of a plurality of predetermined points on a specific part of the face, the variation in the area formed by a plurality of predetermined points on a specific part of the face, the variation in the angle formed by three predetermined points on a specific part of the face, etc.

[0065] As noted above, FIGS. 4-8 illustrate examples of variations that may be measured using one or more points on a particular part of the face.

[0066] The facial movement evaluation system 2 according to the second embodiment can simultaneously perform the "variation value measurement process" for a plurality of specific facial regions in parallel. Furthermore, the facial movement evaluation system 2 according to the second embodiment can simultaneously perform the "variation value measurement process" for a specific facial region in parallel, so as to simultaneously measure a plurality of variation values ​​(for example, "average movement distance of a plurality of predetermined points on a specific facial region" and "average movement speed of a plurality of predetermined points on a specific facial region").

[0067] [Regarding output and display of fluctuation values] The facial movement evaluation system 2 according to the second embodiment can display the "fluctuation values" for each specific part of the face on the display screen of the mobile terminal 6 as the result of movement measurement.

[0068] Furthermore, for example, the facial movement evaluation system 2 can add up and average the "average movement distances of specified multiple points" for all specific parts of the face (e.g., right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, corners of the mouth, wings of the nose, lips, neck, lower jaw, etc.), calculate this as a value for the movement of the entire face, and display it on the display surface of the mobile terminal 6.

[0069] 2.3. [Summary] The facial movement evaluation system 2 according to the second embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system configured with the face itself as the reference, based on facial data acquired via the interface device 10. Furthermore, the processing circuit 12 executes the following steps: (1) measure the fluctuation values ​​of one or more points related to specific parts of the face within a predetermined short period of time.

[0070] By using the facial movement evaluation system 2 according to the second embodiment, it is possible to properly grasp the movements of each organ or part of the face over a short period of time from the viewpoint of fluctuation values ​​(amounts of fluctuation).

[0071] 3. [Third Embodiment] Following the first and second embodiments, a preferred third embodiment of the present disclosure will be disclosed. Note that the rehabilitation support system according to the third embodiment is similar to the facial movement evaluation systems according to the first and second embodiments. Therefore, the following description will focus on the differences between the two. The rehabilitation support system according to the third embodiment is useful for supporting rehabilitation for facial nerve paralysis.

[0072] 3.1 System Configuration Fig. 9 is a diagram showing the overall system configuration of a rehabilitation support system 2' according to the third embodiment. The rehabilitation support system 2' according to the third embodiment operates on a mobile terminal 6, similar to the facial movement evaluation systems 2 according to the first and second embodiments.

[0073] As in the first and second embodiments, the mobile terminal 6 is a mobile computing device such as a tablet terminal or a smartphone. The mobile terminal 6 includes an image sensor and an image capture control unit that controls the image sensor. That is, for example, as shown in FIG. 3 , the mobile terminal 6 includes an infrared sensor 20, a front camera 22, and an infrared irradiation unit 24. The mobile terminal 6 is connected to an external network 8.

[0074] The external network 8 is, for example, the Internet, and is connected to the mobile terminal 6 via the interface device 10. The rehabilitation support system 2′ may also be a native application running on the mobile terminal 6. Thus, the mobile terminal 6 can build the rehabilitation support system 2′ on the mobile terminal 6 by downloading an application program related to the rehabilitation support system 2′ from an application store server 4 connected to the external network 8.

[0075] 9 also includes an interface device 10, a processing circuit 12, and a memory 14. The interface device 10, the processing circuit 12, and the memory 14 are all similar to those of the mobile terminals 6 in the first and second embodiments.

[0076] Furthermore, as in the first and second embodiments, the processing circuit 12 of the portable terminal 6 according to the third embodiment also operates a face recognition system that sets multiple predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on image data of the face acquired via the interface device 10. The face recognition system here is also based on "ARFacegeometry" (Class name), which is used in FaceID, a three-dimensional facial authentication system provided by Apple, and performs real-time three-dimensional tracking using a front camera 22, an infrared irradiation unit 24, an infrared sensor 20, and the like. Therefore, the face recognition system here also sets 1,220 predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself (see FIG. 2 ). The face recognition system here is not limited to one based on "ARFacegeometry," but may be any system for face recognition that can set multiple points on the facial surface in a three-dimensional relative coordinate system based on the face itself.

[0077] 3.2 System Operation The operation of the rehabilitation support system 2' by the operator of the rehabilitation support system 2' and the corresponding operation of the rehabilitation support system 2' will be described below.

[0078] First, the operator of the rehabilitation support system 2 ′ starts operation of the rehabilitation support system 2 ′ on the mobile terminal 6 .

[0079] Next, the healthy side (and affected side) of the operator's face can be set automatically or manually in the rehabilitation support system 2'. Rehabilitation using the rehabilitation support system 2' focuses on exercising only the affected side of the face. In other words, the affected side is the working side, and the healthy side is the non-working side.

[0080] The facial areas that will be rehabilitated are mainly the eyes (closed, open) and the corners of the mouth. The rehabilitation described below will also be performed on the eyes (closed, open) and the corners of the mouth as an example.

[0081] 3.2.1. [Movement Determination in the Rehabilitation Support System] For example, when the affected side is the right side, the rehabilitation support system 2' displays "Please close only your right eye" on the display surface of the mobile terminal 6 (as shown in FIG. 10). The operator of the rehabilitation support system 2' attempts to move in accordance with the display. If the rehabilitation support system 2' determines that the right eye is moving sufficiently, the display surface of the rehabilitation support system 2' displays, for example, "Your eye is closed properly" (see FIG. 10(1)). In this way, when the movement of a specific part is sufficient and appropriate, the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' performs the following evaluation value calculation and movement determination process. The "evaluation value calculation" process described below is the same as the "evaluation value calculation process" performed by the processing circuit 12 of the mobile terminal 6 in the facial movement evaluation system 2 according to the first embodiment. Therefore, the average left-right ratio calculated in [2] below is the evaluation value of the movement of that specific part (however, the non-moving side of the part, not the healthy side, is used as the basis). The movement of a specific body part is sufficient and appropriate when the calculated average left-right ratio is greater than a given movement threshold (step (3)). In the example shown in FIG. 10, the specific body part is the "eyes." In accordance with the first notification issued in step [3], the processing circuit 12 displays via the interface device 10 that the movement is appropriate (for example, "Your eyes are closed appropriately"). FIG. 11C is a diagram showing a flowchart illustrating the following evaluation value calculation and movement determination process.

[0082] [Evaluation Value Calculation and Movement Determination Process] [1] Step (1) of performing the following substeps (1-1) and (1-2) for all of the fluctuation values ​​that can be measured for a specific facial region using one or more points related to the specific facial region. [1-1] Substep (1-1) of measuring fluctuation values ​​for one or more points related to the specific facial region within a predetermined short period of time. [1-2] Substep (1-2) of calculating the left-right ratio of the maximum of the measured fluctuation values, based on the non-moving side of the specific region. [2] Step (2) of calculating the average left-right ratio of the left-right ratios calculated in step (1). [3] Step (3) of issuing a first notification indicating movement if the calculated average left-right ratio is greater than a given movement threshold.

[0083] In the evaluation value calculation and movement determination process described above, the specific facial parts are, as mentioned above, the eyes, the corners of the mouth, etc. The "predetermined short period" is a period of time sufficient for simple facial movements, such as 3 seconds or 5 seconds.

[0084] Furthermore, as explained in the first embodiment, examples of maximum values ​​of fluctuations that can be measured using one or more points related to a specific part of the face are as follows (2-1), (4-1), (4-2), (4-3), and (4-4). ((2) When the specific part is the "eyes") (2-1) The difference in the area inside the eyelids between when the eyes are closed and when they are at their narrowest point, and when the eyes are open and at rest. ((4) When the specific part is the "corners of the mouth") (4-1) The maximum value of the total movement distance of a plurality of predetermined points near the corners of the mouth. (4-2) The maximum value of the average movement speed of a plurality of predetermined points near the corners of the mouth. (4-3) The difference in the angle between a point on the midline between the eyebrows and a point on the tip of the nose and a predetermined point near the corners of the mouth at rest and during the movement of lifting the corners of the mouth. (4-4) Difference in the angle between the points between the eyebrows and the tip of the nose on the midline and the points on the lips on the midline at rest and during lip corner lifting

[0085] Examples of the fluctuation values ​​(maximum values) that can be measured using one or more points on specific parts of the face, as described above in (2-1), (4-1) to (4-4), are described in [Explanation of examples of maximum fluctuation values] to [Explanation of examples of maximum fluctuation values ​​- end] in [First embodiment].

[0086] 3.2.2. [Caution Notification in Rehabilitation Support System] Furthermore, when the operator of the rehabilitation support system 2' attempts to perform a movement in accordance with the display of "Please close only your right eye," if the rehabilitation support system 2' determines that the right corner of the mouth is moving (linked), the display screen of the rehabilitation support system 2' displays, for example, "Your mouth is also moving at the same time" (see FIG. 10(2)). In this case, when a second specific part, different from the first specific part targeted for rehabilitation, is linked, the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' performs the following evaluation value calculation and caution notification process. The following "evaluation value calculation" process is the same as the evaluation value calculation in the [evaluation value calculation and movement determination process] executed by the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' according to this embodiment (third embodiment). Therefore, the average left-right ratio calculated in [2] below is the evaluation value of the movement of the second specific part. The second specific body part is interlocked, i.e., a certain degree of interlocking has occurred, when the calculated average left-right ratio is greater than a given non-stationary threshold (step (4)). In the example shown in FIG. 10(2), the second specific body part is the "corners of the mouth." In accordance with the second notification issued in step [4], the processing circuit 12 displays a warning notice (e.g., "Your mouth is also moving at the same time") via the interface device 10 indicating that interlocking of other body parts has occurred. FIG. 11D is a flowchart showing the following evaluation value calculation and warning notice process.

[0087] [Evaluation Value Calculation and Caution Notification Processing] [1] Step (1) of performing the following substeps (1-1) and (1-2) for all of the variation values ​​that can be measured using one or more points on a second specific facial region different from the first specific facial region. [1-1] Substep (1-1) of measuring variation values ​​for one or more points on the second specific facial region within a predetermined short period of time. [1-2] Substep (1-2) of calculating the left-right ratio of the maximum of the measured variation values, based on the non-moving side of the second specific facial region. [2] Step (2) of calculating the average left-right ratio of the left-right ratios calculated in step (1). [4] Step (4) of issuing a second notification indicating non-stationary status if the calculated average left-right ratio is greater than a given non-stationary threshold.

[0088] In the evaluation value calculation and warning notification process described above, when the first specific facial part is the eye, the second facial part, i.e., the part that is easily linked, is the corners of the mouth, eyebrows, etc. When the first specific facial part is the corners of the mouth, the second facial part, i.e., the part that is easily linked, is the eyes, eyebrows, etc. Furthermore, the "predetermined short period" is a period of time sufficient for simple facial movements, such as 3 seconds or 5 seconds.

[0089] Furthermore, as explained in the first embodiment, examples of maximum values ​​of variation that can be measured using one or more points related to a second specific part of the face are as follows (1-1), (1-2), (2-1), (4-1), (4-2), (4-3), and (4-4). ((1) When the second specific part is the "eyebrows") (1-1) The total distance when the sum of the distances between a predetermined number of pairs of two points near the eyebrows is maximum. (1-2) The maximum movement speed for a predetermined number of points near the eyebrows. ((2) When the second specific part is the "eyes") (2-1) The difference in the area of ​​the inside of the eyelids between when the eyes are at their narrowest, which is when the eyes are closed, and when the eyes are at rest, which is when the eyes are open. ((4) When the second specific part is the "corners of the mouth") (4-1) The maximum value of the total movement distance for a predetermined number of points near the corners of the mouth. (4-2) The maximum value of the average movement speed of a plurality of predetermined points near the corners of the mouth. (4-3) The difference in the angle between the point between the eyebrows and the point on the nose on the midline and predetermined points near the corners of the mouth, between at rest and during the movement of lifting the corners of the mouth. (4-4) The difference in the angle between the point between the eyebrows and the point on the nose on the midline and the point on the lips on the midline, between at rest and during the movement of lifting the corners of the mouth.

[0090] Examples of the fluctuation values ​​(maximum values) that can be measured using one or more points on the (second) specific part of the face, as described above in (1-1) to (1-2), (2-1), and (4-1) to (4-4), are described in [Explanation of an example of the maximum fluctuation value] to [Explanation of an example of the maximum fluctuation value - end] in [First embodiment].

[0091] [Regarding Output and Display of Result Messages] The rehabilitation support system 2' according to the third embodiment can display an evaluation message as a result of the movement determination process and a warning message as a result of the warning notification process on the display surface.

[0092] As shown in Figure 10 (1), when the movement of the right eye, a specific part of the face, is sufficient and appropriate, the message "Eye closure is correct" is displayed on the display. At the same time, a circle can be displayed around the right eye, which is performing the correct movement.

[0093] 10(2), when a link occurs in the right corner of the mouth, which is a second specific part different from the specific part of the face, the message "Your mouth is also moving at the same time" is displayed on the display screen. At the same time, a circular area can be displayed around the entire right corner of the mouth where the link occurs.

[0094] 3.3. [Summary] The rehabilitation support system 2' according to the third embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system configured with the face itself as the reference, based on facial data acquired via the interface device 10. Furthermore, the processing circuit displays, via the interface device 10, a message instructing the movement of a specific part of the face on the affected side. Furthermore, the processing circuit 12 executes the following steps (1-1) and (1-2) for all of the variation values ​​that can be measured for a specific part of the face using one or more points on the specific part of the face: (1) step (1); (1-1) substep (1-1) of measuring variation values ​​for one or more points on the specific part of the face within a predetermined short period of time; (1-2) substep (1-2) of calculating the left-right ratio of the maximum value of the measured variation values ​​based on the non-moving side of the specific part; (2) step (2) of calculating the average left-right ratio of the left-right ratios calculated in step (1); and (3) step (3) of issuing a first notification indicating that movement is occurring if the calculated average left-right ratio is greater than a given movement threshold.

[0095] A rehabilitation support system 2′ according to the third embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system that is configured with the face itself as a reference, based on facial data acquired via the interface device 10. Furthermore, the processing circuit 12 displays a message via the interface device instructing the movement of the affected side of a first specific part of the face. Furthermore, the processing circuit 12 executes the following steps (1-1) and (1-2) for all of the variation values ​​that can be measured using one or more points on a second specific part of the face, different from the first specific part; (1-1) substep (1-1) of measuring variation values ​​for one or more points on the second specific part of the face within a predetermined short period of time; (1-2) substep (1-2) of calculating the left-right ratio of the maximum of the measured variation values, based on the non-moving side of the second specific part; (2) step (2) of calculating the average left-right ratio of the left-right ratios calculated in step (1); and (4) step (4) of issuing a second notification indicating non-stationary if the calculated average left-right ratio is greater than a given non-stationary threshold.

[0096] The rehabilitation support system 2' according to the third embodiment can effectively carry out the rehabilitation of facial organs by utilizing a function that appropriately evaluates the short-term movements of each organ and part of the face from the perspective of left-right ratio.

[0097] 4. [Fourth Embodiment] Following the third embodiment, a preferred fourth embodiment of the present disclosure will be disclosed. The rehabilitation support system according to the fourth embodiment is substantially similar to the rehabilitation support system according to the third embodiment. Therefore, the following description will focus on the differences between the two.

[0098] 4.1 System Configuration A rehabilitation support system 2' according to the fourth embodiment has the same system configuration as the rehabilitation support system according to the third embodiment shown in FIG.

[0099] 4.2 System Operation The operation of the rehabilitation support system 2′ by the operator of the rehabilitation support system 2′ according to the fourth embodiment and the corresponding operation of the rehabilitation support system 2′ will be described below, focusing on the differences between the third and fourth embodiments.

[0100] First, the operator of the rehabilitation support system 2 ′ starts operation of the rehabilitation support system 2 ′ on the mobile terminal 6 .

[0101] 4.2.1. [Movement Determination in the Rehabilitation Support System] The rehabilitation support system 2' displays a message on the display screen of the portable terminal 6 stating, "Move a specific part of your face." The operator of the rehabilitation support system 2' attempts to perform a movement in accordance with the message. If the rehabilitation support system 2' determines that the specific part is moving sufficiently, the display screen of the rehabilitation support system 2' displays, for example, a message stating, "A specific part of your face is moving sufficiently." In this manner, when the movement of the specific part is sufficient and appropriate, the processing circuit 12 of the portable terminal 6 in the rehabilitation support system 2' performs the following [variation value measurement and movement determination process]. FIG. 11E is a diagram showing a flowchart illustrating the following variation value measurement and movement determination process. The following "variation value measurement" process is the same as the [variation value measurement process] performed by the processing circuit 12 of the portable terminal 6 in the facial movement evaluation system 2 according to the second embodiment. The movement of the specific part being sufficient and appropriate means that the measured value is greater than a given movement threshold (step (2)). In accordance with the first notification issued in step (2), the processing circuit 12 displays via the interface device 10 that the movement is appropriate (for example, that "a specific part of the face is moving sufficiently").

[0102] [Variation Value Measurement and Motion Determination Process] [1] Step (1) of measuring the variation value of one or more points on a specific part of the face within a predetermined short period of time. [2] Step (2) of issuing a first notification indicating motion if the measured variation value is greater than a given motion threshold.

[0103] In the above-described fluctuation value measurement process in the rehabilitation support system 2' according to the fourth embodiment, specific parts of the face are, for example, the right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, right corner of the mouth, left corner of the mouth, right ala of the nose, left ala of the nose, lips, neck, lower jaw, etc. The "predetermined short period" is a period of time sufficient for performing simple facial exercises, such as 4 seconds or 6 seconds.

[0104] Furthermore, the variation values ​​measured using one or more points on a specific part of the face include, for example, the average movement distance of a plurality of predetermined points on a specific part of the face, the average movement speed of a plurality of predetermined points on a specific part of the face, the variation in the area formed by a plurality of predetermined points on a specific part of the face, the variation in the angle formed by three predetermined points on a specific part of the face, etc.

[0105] As noted above, FIGS. 4-8 illustrate examples of variations that may be measured using one or more points on a particular part of the face.

[0106] The rehabilitation support system 2' according to the fourth embodiment can simultaneously perform the "variation value measurement process" for a plurality of specific facial regions. Furthermore, the rehabilitation support system 2' according to the fourth embodiment can simultaneously perform the "variation value measurement process" for a specific facial region to measure a plurality of variation values ​​(for example, "average movement distance of a plurality of predetermined points on a specific facial region" and "average movement speed of a plurality of predetermined points on a specific facial region").

[0107] 4.2.2. [Caution Notification in Rehabilitation Support System] Furthermore, when the operator of the rehabilitation support system 2' attempts to perform a movement in accordance with a display indicating "Move a specific facial part," if the rehabilitation support system 2' determines that another specific facial part has moved (is linked), a message such as "Another specific facial part has also moved" is displayed on the display screen of the rehabilitation support system 2'. When a second specific facial part, different from the first specific facial part targeted for rehabilitation, is linked, the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' performs the following "variation value measurement and caution notification process." FIG. 11F is a diagram showing a flowchart illustrating the following variation value measurement and caution notification process. The following "variation value measurement" process is similar to the variation value measurement in the "variation value measurement and movement determination process" executed by the processing circuit 12 of the mobile terminal 6 in the rehabilitation support system 2' according to this embodiment (fourth embodiment) described above. The second specific part is moving, i.e., a corresponding movement is occurring, when the measured fluctuation value is greater than a given non-stationary threshold (step (2)). In accordance with the second notification issued in step (2), the processing circuit 12 displays a warning notification that another part is moving (e.g., "Another specific part of the face is also moving") via the interface device 10.

[0108] [Variation Value Measurement and Caution Notification Processing] [1] Step (1) of measuring the variation value within a predetermined short period of time for one or more points on a second specific part of the face that is different from the first specific part, and [2] Step (2) of issuing a second notification indicating immobility if the measured variation value is greater than a given immobility threshold.

[0109] In the above-described variation value measurement and attention notification process in the rehabilitation support system 2′ according to the fourth embodiment, the second specific part of the face is a part different from the first specific part that is the target of rehabilitation, and is, for example, the right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, right corner of the mouth, left corner of the mouth, right ala of the nose, left ala of the nose, lips, neck, lower jaw, etc. The “predetermined short period” is a period of time sufficient for performing simple facial exercises, such as 4 seconds or 6 seconds.

[0110] Furthermore, the variation values ​​measured using one or more points on the second specific region of the face include, for example: average movement distance of a plurality of predetermined points on the second specific region of the face; average movement speed of a plurality of predetermined points on the second specific region of the face; variation in the area formed by a plurality of predetermined points on the second specific region of the face; variation in the angle formed by three predetermined points on the second specific region of the face; etc.

[0111] As noted above, FIGS. 4-8 illustrate examples of variations that may be measured using one or more points on a second particular region of the face.

[0112] The rehabilitation support system 2' according to the fourth embodiment can simultaneously perform the "variation value measurement and attention notification process" for a plurality of specific facial regions. Furthermore, the rehabilitation support system 2' according to the fourth embodiment can simultaneously perform the "variation value measurement and attention notification process" for a second specific facial region, so as to simultaneously measure a plurality of variation values ​​(for example, "average movement distance of a plurality of predetermined points on the second specific facial region" and "average movement speed of a plurality of predetermined points on the second specific facial region").

[0113] [Regarding output and display of result messages] The rehabilitation support system 2' according to the fourth embodiment can display an evaluation message as a result of the movement determination process and a warning message as a result of the warning notification process on the display surface. First, for example, if the movement of a specific part of the face (hereinafter referred to as "part A") is sufficient and appropriate, "Part A is moving appropriately" is displayed on the display surface. Furthermore, if a linkage occurs in a second specific part of the face (hereinafter referred to as "part B") different from the specific part of the face (part A), "Part B is also moving at the same time" is displayed on the display surface.

[0114] 4.3. [Summary] The rehabilitation support system 2' according to the fourth embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device 10. Furthermore, the processing circuit 12 displays a message instructing the movement of a specific part of the face via the interface device 10. Furthermore, the processing circuit 12 executes the following steps: (1) measuring a variation value of one or more points related to the specific part of the face within a predetermined short period of time; and (2) issuing a first notification indicating that the movement is being performed if the measured variation value is greater than a given movement threshold.

[0115] A rehabilitation support system 2′ according to the fourth embodiment includes an interface device 10 and a processing circuit 12. The processing circuit 12 operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system based on the face itself, based on facial data acquired via the interface device 10. Furthermore, the processing circuit displays a message instructing a movement of a first specific facial region via the interface device 10. Furthermore, the processing circuit 12 executes the following steps: (1) measuring fluctuation values ​​within a predetermined short period of time for one or more points on a second specific facial region different from the first specific facial region; and (2) issuing a second notification indicating immobility if the measured fluctuation value is greater than a given immobility threshold.

[0116] The rehabilitation support system 2' of the fourth embodiment can effectively carry out the rehabilitation of facial organs by utilizing the function of appropriately grasping the short-term movements of each organ and part of the face from the perspective of fluctuation values ​​(amounts of fluctuation).

[0117] 5. [Other Embodiments] As described above, the first and second embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to these and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate.

[0118] The facial movement evaluation system 2 according to the second embodiment can also measure the movement distance and average movement speed for each of all points on the face (for example, 122 points in the present disclosure) within a predetermined short period of time. By adding up the movement distances for all points on the face or by taking an arithmetic average, the movement value of the entire face can be calculated.

[0119] The facial movement evaluation system 2 according to the first and second embodiments, and the rehabilitation support system 2' according to the third and fourth embodiments, can both store data related to the movements of facial parts over time. Using the data stored over time, the facial movement evaluation system 2 according to the first embodiment can evaluate changes in facial movements over time. Furthermore, the rehabilitation support system 2' according to the second embodiment can use the data stored over time to determine whether or not rehabilitation is effective over time.

[0120] Furthermore, the accompanying drawings and detailed description are provided to explain the embodiments. Therefore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately identifying these non-essential components as essential.

[0121] Furthermore, since the above-described embodiments are intended to illustrate the technology of the present disclosure, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

[0122] 2: Facial movement evaluation system, 2': Rehabilitation support system, 4: App store server, 6: Mobile terminal, 8: External network, 10: Interface device, 12: Processing circuit, 14: Memory.

Claims

1. A facial movement evaluation system comprising an interface device and a processing circuit, the processing circuit operating a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system formed with the face itself as a reference based on facial data acquired via the interface device, and further comprising: (1) a step (1) of executing the following sub-steps (1-1) and (1-2) for all of a plurality of variation values ​​that can be measured for a specific part of the face using one or more points on the specific part of the face; (1-1) a sub-step (1-1) of measuring variation values ​​for one or more points on the specific part of the face within a predetermined short period of time; (1-2) a sub-step (1-2) of calculating a left-right ratio of the maximum of the measured variation values, taking the healthy side of the specific part as the reference; and (2) a step (2) of calculating an average left-right ratio for all of the plurality of variation values ​​in the specific part of the face for the left-right ratio calculated in the step (1).

2. The facial movement evaluation system of claim 1, wherein the specific facial part is an eyebrow, and the maximum value of the variation value that can be measured using one or more points on the specific facial part includes: a total distance when the total value of the distance between two points of a predetermined set of two points in the vicinity of the eyebrows is maximum; and a maximum movement speed for a predetermined set of points in the vicinity of the eyebrows.

3. The facial movement evaluation system of claim 1, wherein the specific part of the face is the cheek, and the maximum value of the fluctuation value that can be measured using one or more points on the specific part of the face includes: the maximum value of the total movement distance of a predetermined number of points near the cheek; the maximum value of the average movement speed of a predetermined number of points near the cheek; and the difference in angle between a point on the midline between the eyebrows and a point on the tip of the nose and a predetermined point near the cheek at rest and during mouth corner lift movement.

4. The facial movement evaluation system of claim 1, wherein the specific facial part is the corner of the mouth, and the maximum fluctuation values ​​that can be measured using one or more points on the specific facial part include: the maximum total movement distance of a plurality of predetermined points near the corner of the mouth; the maximum average movement speed of a plurality of predetermined points near the corner of the mouth; the difference in angle between a point between the eyebrows and a point on the nose on the midline and a specific point near the corner of the mouth, at rest and during mouth corner lifting movement; and the difference in angle between a point between the eyebrows and a point on the nose on the midline and a point on the lips on the midline, at rest and during mouth corner lifting movement.

5. The facial movement evaluation system of claim 1, wherein the specific facial part is the ala of the nose, and the maximum value of the fluctuation value that can be measured using one or more points on the specific facial part includes: the maximum value of the total movement distance of a predetermined number of points in the vicinity of the ala of the nose; and the maximum value of the average movement speed of a predetermined number of points in the vicinity of the ala of the nose.

6. A facial movement evaluation system comprising an interface device and a processing circuit, the processing circuit operating a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system formed with the face itself as a reference based on facial data acquired through the interface device, and further comprising: (1) a step (1) of measuring the fluctuation values ​​of one or more points relating to specific parts of the face within a predetermined short period of time, the specific parts of the face comprising the right eyebrow, left eye, right eye, left eye, right cheek, left cheek, right corners of the mouth, left corners of the mouth, right nasal ala, left nasal ala, lips, and lower jaw, and the fluctuation values ​​measured using the one or more points relating to the specific parts of the face comprising: - an average movement speed of the predetermined plurality of points relating to the specific parts of the face.

7. A rehabilitation support system comprising an interface device and a processing circuit, the processing circuit operating a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system formed with the face itself as a reference based on facial data acquired via the interface device, the processing circuit further displaying a message instructing a movement of an affected side of a specific part of the face via the interface device, the processing circuit further comprising: (1) a step (1) of executing the following sub-steps (1-1) and (1-2) for all of a plurality of fluctuation values ​​that can be measured for the specific part of the face using one or more points on the specific part of the face; (1-1) a sub-step (1-1) of measuring a fluctuation value for one or more points on the specific part of the face within a predetermined short period of time; (1-2) a sub-step (1-2) of calculating a left-right ratio of the maximum measured fluctuation value for the specific part, using a non-moving side opposite to the affected side for which the movement is instructed as a reference; (2) A step (2) of calculating an average left / right ratio for all of the multiple variation values ​​in a specific part of the face for the left / right ratio calculated in the step (1); and (3) a step (3) of issuing a first notification indicating that a movement is being performed if the calculated average left / right ratio is greater than a given movement threshold.

8. The rehabilitation support system of claim 7, wherein the specific facial part is the corner of the mouth, and maximum values ​​of fluctuation that can be measured using one or more points on the specific facial part include: a maximum value of a total movement distance of a plurality of predetermined points in the vicinity of the corner of the mouth; a maximum value of an average movement speed of a plurality of predetermined points in the vicinity of the corner of the mouth; a difference in angle between a point between the eyebrows and a point on the nose on the midline and a specific point in the vicinity of the corner of the mouth, at rest and during the movement of lifting the corners of the mouth; and a difference in angle between a point between the eyebrows and a point on the nose on the midline and a point on the lips on the midline, at rest and during the movement of lifting the corners of the mouth.

9. A rehabilitation support system comprising an interface device and a processing circuit, the processing circuit operating a face recognition system for setting a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system formed with the face itself as a reference based on facial data acquired via the interface device, the processing circuit further displaying a message instructing a movement of an affected side of a first specific part of the face via the interface device, the processing circuit further comprising: (1) a step (1) of executing the following substeps (1-1) and (1-2) for all of a plurality of fluctuation values ​​that can be measured for a second specific part of the face different from the first specific part, using one or more points on the second specific part of the face; (1-1) a substep (1-1) of measuring fluctuation values ​​for one or more points on the second specific part of the face within a predetermined short period of time; (1-2) a substep (1-2) of calculating a left-right ratio of the maximum measured fluctuation values ​​for the second specific part, using a non-moving side opposite to the affected side for which the movement is instructed; (2) A step (2) of calculating an average left / right ratio for all of the variation values ​​in a second specific part of the face for the left / right ratio calculated in the step (1); and (4) a step (4) of issuing a second notification indicating that the second specific part is non-stationary if the calculated average left / right ratio is greater than a given non-stationary threshold.

10. The rehabilitation support system of claim 9, wherein the first specific part of the face is the eye, the second specific part of the face is the corner of the mouth, and maximum values ​​of fluctuation that can be measured using one or more points on the second specific part of the face include: a maximum value of a total moving distance of a predetermined number of points near the corner of the mouth, a maximum value of an average moving speed of a predetermined number of points near the corner of the mouth, a difference in angle between a point between the eyebrows and a point on the nose on the midline and a predetermined point near the corner of the mouth, at rest and during the movement of lifting the corner of the mouth, and a difference in angle between a point between the eyebrows and a point on the nose on the midline and a point on the lips on the midline, at rest and during the movement of lifting the corner of the mouth.

11. The rehabilitation support system of claim 9, wherein the first specific part of the face is the eye, the second specific part of the face is the eyebrow, and the maximum value of the fluctuation value that can be measured using one or more points on the second specific part of the face includes: a total distance when the total value of the distance between two points of a predetermined plurality of pairs of two points in the vicinity of the eyebrows is maximized; and a maximum movement speed for a predetermined plurality of points in the vicinity of the eyebrows.

12. The rehabilitation support system of claim 9, wherein the first specific part of the face is the corner of the mouth, the second specific part of the face is the eyebrows, and the maximum value of the fluctuation value that can be measured using one or more points on the second specific part of the face includes: a total distance when the total value of the distance between two points of a predetermined plurality of pairs of two points in the vicinity of the eyebrows is maximized; and a maximum movement speed for a predetermined plurality of points in the vicinity of the eyebrows.

13. A rehabilitation support system comprising: an interface device; and a processing circuit, the processing circuit operates a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system formed with the face itself as a reference based on facial data acquired through the interface device, and the processing circuit displays a message instructing a movement of a specific part of the face through the interface device, and the processing circuit executes the following steps: (1) measuring a fluctuation value of one or more points on the specific part of the face within a predetermined short period of time; and (2) issuing a first notification indicating that a movement is being made if the measured fluctuation value is greater than a given movement threshold value; the specific part of the face comprises the right eyebrow, left eyebrow, right eye, left eye, right cheek, left cheek, right corner of the mouth, left corner of the mouth, right nasal ala, left nasal ala, lips, and lower jaw, and the fluctuation value measured using the one or more points on the specific part of the face comprises: - an average movement speed of a plurality of predetermined points on the specific part of the face.

14. A rehabilitation support system comprising an interface device and a processing circuit, the processing circuit operating a face recognition system that sets a plurality of predetermined points on the facial surface in a three-dimensional relative coordinate system formed with the face itself as a reference based on facial data acquired through the interface device, the processing circuit further displaying a message via the interface device instructing a movement of a first specific part of the face, and the processing circuit further executing the steps of: (1) measuring a variation value within a predetermined short period of time for one or more points on a second specific part of the face different from the first specific part, and (2) issuing a second notification indicating that the second specific part is not stationary if the measured variation value is greater than a given non-stationary threshold value.

15. The rehabilitation support system of claim 14, wherein the specific facial part includes the right eyebrow, the left eyebrow, the right eye, the left eye, the right cheek, the left cheek, the right corner of the mouth, the left corner of the mouth, the right nasal ala, the left nasal ala, the lips, and the lower jaw, and the variation values ​​measured using one or more points on the specific facial part include: - an average movement distance of a predetermined number of points on the specific facial part; - an average movement speed of a predetermined number of points on the specific facial part; - a variation in the area formed by a predetermined number of points on the specific facial part; and - a variation in the angle formed by three predetermined points on the specific facial part.

Images