Support system, support method, and program

Abstract

To provide a support system, a support method, and a program for continuing correct rehabilitation even when it is difficult to receive guidance from an expert.SOLUTION: A rehabilitation support system 1 includes: an acquisition section that acquires, from a user terminal, a face image in which a face of a user is imaged; and a display control section that causes the user terminal to display a composite image obtained by superimposing, on the face image, a mask image generated on the basis of feature points extracted from the face image acquired by the acquisition section and capable of observing facial movements of the user in the face image and shielding the whole or part of the face of the user in the face image, and that guides the user terminal to an improvement method for improving facial muscle movements on the face.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a support system, a support method, and a program. [Background technology]

[0002] There are techniques for improving the movement of facial muscles. For example, when forcibly moving facial muscles during rehabilitation after the onset of facial nerve paralysis, pathological synkinesis, in which the eyelids and mouth move involuntarily together, often remains as a residual effect. Non-Patent Document 1 introduces specialized support for preventing facial sequelae after the onset of facial nerve paralysis. Non-Patent Document 2 discloses a mirror biofeedback therapy technique that prevents pathological synkinesis by using a mirror to make the patient aware of the involuntary contraction of facial muscles, which the patient is not aware of. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "What is facial nerve paralysis? Causes, treatment, and rehabilitation," [online], Japan Society of Otolaryngology-Head and Neck Surgery, [searched April 26, 2024], Internet<URL:https: / / www.jibika.or.jp / owned / contents1.html> [Non-patent document 2] Takahiro Azuma, "Rehabilitation for Sequelae of Facial Nerve Paralysis," [online], 120th Annual Meeting of the Oto-Rhino-Laryngological Society of Japan, Symposium, Review, [Retrieved April 26, 2024], Internet <URL:https: / / www.jstage.jst.go.jp / article / jibiinkoka / 123 / 6 / 123_430 / _pdf / -char / ja#:~:text=%E7%97%85%E7%9A%84%E5%85%B1%E5%90%8C%E9%81%8B%E5%8B%95%E3%81%AE%E4%BA%88%E9%98%B2%E3%81%AE%E3%81%9F%E3%82%81%E3%81%AE%E3%83%9F%E3%83%A9%E3%83%BC,%E9%96%89%20%E7%9E%BC%E3%82%92%E6%8A%91%E5%88%B6%E3%81%99%E3%82%8B%EF%BC%8E> Summary of the Invention [Problem to be solved by the invention]

[0004] As mentioned above, if rehabilitation is performed using the wrong method, sequelae due to pathological synkinesis may occur. Therefore, if signs of pathological synkinesis are observed, proper rehabilitation must be performed under the guidance of a specialist, such as by starting mirror biofeedback therapy. However, rehabilitation covered by health insurance has a set period for each illness or injury, and it can be difficult to receive professional rehabilitation for more than a certain period. For diseases that require long-term rehabilitation, such as facial paralysis, the difficulty of continuing proper rehabilitation has been an issue.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a support system, a support method, and a program that enable correct rehabilitation to be continued even when it is difficult to receive guidance from a specialist. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, one aspect of the present invention is an assistance system comprising: an acquisition unit that acquires a facial image of a user's face from a user terminal; and a display control unit that displays on the user terminal a composite image in which the mask image, generated based on feature points extracted from the facial image acquired by the acquisition unit, allows observation of the facial movement of the user in the facial image and superimposes the mask image that covers all or part of the user's face in the facial image, and guides the user terminal on a method for improving the movement of mimetic muscles in the face.

[0007] One aspect of the present invention is an assistance method performed by an assistance system that is a computer, in which an acquisition unit acquires a facial image of a user's face from a user terminal, and a display control unit displays on the user terminal a composite image in which the mask image, which is generated based on feature points extracted from the facial image acquired by the acquisition unit and allows observation of the facial movement of the user in the facial image, and which is superimposed on the facial image and which obscures all or part of the user's face in the facial image, and guides the user terminal through an improvement method for improving the movement of mimetic muscles in the face.

[0008] One aspect of the present invention is a program that causes a support system, which is a computer, to acquire a facial image of a user's face from a user terminal, display on the user terminal a composite image in which the mask image, which is generated based on feature points extracted from the acquired facial image and allows the user's facial movement in the facial image to be observed and which obscures all or part of the user's face in the facial image, is superimposed on the facial image, and guides the user terminal through a method for improving the movement of mimetic muscles in the face. [Effects of the Invention]

[0009] According to the present invention, it is possible to continue correct rehabilitation even when it is difficult to receive guidance from a specialist. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of a rehabilitation assistance system according to an embodiment. [Figure 2] FIG. 2 is a diagram for explaining processing performed by the rehabilitation support system according to the embodiment. [Figure 3] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support by massage. [Figure 4] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support by massage. [Figure 5] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support by massage. [Figure 6] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through training. [Figure 7] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through training. [Figure 8] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through training. [Figure 9] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through training. [Figure 10] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through training. [Figure 11] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through training. [Figure 12] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through training. [Figure 13] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through a game. [Figure 14] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through a game. [Figure 15] 10A and 10B are diagrams for explaining a process in which the rehabilitation support system according to the embodiment performs rehabilitation support through a game. [Figure 16] 10 is a flowchart showing a processing flow in which the support server according to the embodiment performs rehabilitation support by massage. [Figure 17] 10 is a flowchart showing a processing flow in which the support server according to the embodiment performs rehabilitation support through training. [Figure 18] 10 is a flowchart showing a processing flow in which the support server according to the embodiment performs rehabilitation support through a game. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the following description, a case where rehabilitation is supported to improve facial paralysis in a patient with facial paralysis will be described as an example, but the present invention is not limited to this. This embodiment can be applied not only to facial paralysis but also to cases where facial distortion is improved in healthy individuals who are interested in beauty and health.

[0012] <1. About the rehabilitation support system> The configuration of a rehabilitation support system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the rehabilitation support system according to this embodiment. As shown in Fig. 1, the rehabilitation support system 1 includes a user terminal 10 and a support server 20. The user terminal 10 and the support server 20 are communicably connected via a communication network NW. The rehabilitation support system 1 is a system that provides a service (hereinafter referred to as a rehabilitation support service) that supports rehabilitation to improve the movement of facial mimetic muscles. In the rehabilitation support system 1, all or part of the functions of the support server 20 may be provided in the user terminal 10. For example, an application (a rehabilitation support application, which will be described later) installed in the user terminal 10 may have the functions of the support server 20 (an imaging control unit 23, a display control unit 24, a feature point detection unit 25, a determination unit 26, and an output unit 27, which will be described later). In this case, the communication network and the support server 20 can be omitted.

[0013] The user terminal 10 is a terminal used by a user who uses the rehabilitation support service. The user terminal 10 is, for example, a smartphone, a tablet terminal, a PC (Personal Computer), etc. The user terminal 10 is communicably connected to the support server 20 via a communication network NW.

[0014] The support server 20 is a server that supports rehabilitation so that users who use the rehabilitation support service can freely open and close their mouths and make the facial expressions they intend. The support server 20 is composed of one or more servers, such as a cloud server.

[0015] In the rehabilitation support system 1, a dedicated application (hereinafter referred to as the "rehabilitation support app") is distributed to the user. The user installs the rehabilitation support app distributed from the rehabilitation support system 1 into the user terminal 10. The user terminal 10 performs user registration with the support server 20 via the rehabilitation support app. This allows the user to use the rehabilitation support service. Note that if a rehabilitation support app having the functions of the support server 20 (an imaging control unit 23, a display control unit 24, a feature point detection unit 25, a determination unit 26, and an output unit 27, which will be described later) is installed in the user terminal 10, user registration is not necessarily required.

[0016] In the above example, a rehabilitation support app service is provided by installing a rehabilitation support app on the user terminal 10, but the present invention is not limited to this. The rehabilitation support service may be provided by a Web system (i.e., a Web app). When the rehabilitation support app is provided by a Web system, the rehabilitation support service is provided via a Web browser.

[0017] In this embodiment, three types of rehabilitation support services are provided: "massage," "training," and "games." The "massage" rehabilitation support service explains how to massage the face and helps the user to perform the massage correctly. The purpose of the massage here is to prevent muscle contracture and circulatory insufficiency in the face. For example, the service helps the user recognize the location of the facial muscles to be massaged and helps the user to massage gently with weak pressure so as not to injure the target facial muscles. The "training" rehabilitation support service explains how to train the face and supports the patient in moving the face correctly. The purpose of training here is to enable the patient to move only the target facial muscles. For example, the service helps the patient recognize which facial muscles to move and which not to move, and supports the patient in moving only the target facial muscles. This can prevent the onset of pathological synkinesis, in which the eyelids and mouth move involuntarily together when trying to move the eyelids. The "game" rehabilitation support service incorporates gamification into rehabilitation support. For example, by incorporating game elements, such as "level up" and "score competitions," rehabilitation can be supported to help patients continue their rehabilitation.

[0018] <2. User Devices> As shown in FIG. 1, the user terminal 10 includes a communication unit 11, a storage unit 12, a control unit 13, an imaging unit 14, a display unit 15, and an operation input unit 16. The communication unit 11 communicates with the user terminal 10 . The storage unit 12 stores various types of information. The storage unit 12 is configured by a storage medium provided as hardware in the user terminal 10, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read-only memory (EEPROM), a random access read / write memory (RAM), a read-only memory (ROM), or any combination of these storage media. The storage unit 12 stores programs for realizing functions provided in the user terminal 10 (including functions related to the control unit 13, particularly the rehabilitation assistance app). The control unit 13 realizes the functions of the user terminal 10 by causing a CPU provided as hardware in the user terminal 10 to execute a program stored in the storage unit 12. For example, the control unit 13 controls the imaging unit 14, the display unit 15, and the operation input unit 16. For example, in response to a user's operation input to the operation input unit 16, the control unit 13 communicates with the support server 20, launches a rehabilitation support app, and displays a screen of the rehabilitation support app in response to the operation on the display unit 15. Furthermore, in response to a guide to capture an image of the user's facial expression via the rehabilitation support app, the control unit 13 launches the camera function of the imaging unit 14 and displays the captured image on the display unit 15. The imaging unit 14 includes a camera module and takes an image under the control of the control unit 13 . The display unit 15 includes a display device such as a liquid crystal display, and displays an image according to the control of the control unit 13 . The operation input unit 16 acquires an operation input from the user and outputs the acquired operation input to the control unit 13 .

[0019] <3. About the support server> As shown in FIG. 1, the support server 20 includes a communication unit 21, a storage unit 22, an imaging control unit 23, a display control unit 24, a feature point detection unit 25, a determination unit 26, and an output unit 27. The communication unit 21 communicates with the user terminal 10 . The storage unit 22 stores various types of information. The storage unit 22 is configured by a storage medium provided as hardware in the support server 20, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. The storage unit 22 stores programs for realizing the functions provided in the support server 20 (including, in particular, an imaging control unit 23, a display control unit 24, a feature point detection unit 25, and a determination unit 26, which are functions of the rehabilitation support application). The functions provided in the support server 20 are realized by causing a CPU provided as hardware in the support server 20 to execute the programs stored in the storage unit 22.

[0020] The imaging control unit 23 controls imaging by the user terminal 10. Triggered by an operation to start the rehabilitation support service via the rehabilitation support app, the imaging control unit 23 instructs the user terminal 10 to capture a facial image of the user (captured image TG).

[0021] The display control unit 24 controls the images displayed on the user terminal 10, thereby guiding the user terminal 10 on the correct rehabilitation method as a method for improving the movement of facial mimetic muscles.

[0022] The display control unit 24 displays an image (model image MG) that presents a correct rehabilitation method on the user terminal 10. By displaying the model image MG, the user can understand the correct rehabilitation method and perform the correct rehabilitation. When providing a rehabilitation support service through "massage," the model image MG may be, for example, a muscle diagram illustrating the location of facial muscles, an explanatory video explaining how to perform a massage correctly, and a sample video showing how to massage a face. When providing a rehabilitation support service through "training," the model image MG may be, for example, a muscle diagram showing which facial muscles are in motion and which are not, an explanatory video explaining the correct way to move the face, and a sample video capturing the movement of the face.

[0023] Furthermore, the display control unit 24 displays a facial image (captured image TG) of the user undergoing rehabilitation on the user terminal 10. By displaying the captured image TG on the user terminal 10, the user can perform rehabilitation while visually checking their own face. For example, in training to move facial muscles, it can be checked whether the eyelids and mouth are moving together when trying to move the eyelids.

[0024] Here, when the display control unit 24 displays the captured image TG on the user terminal 10, it may display a composite image in which the mask image KG is superimposed on the facial image on the user terminal 10. The mask image KG is an image that covers all or part of the user's face in the facial image.

[0025] It is generally known that facial nerve paralysis is often caused by viruses, such as viral infection or reactivation. Such viral facial nerve paralysis tends to occur suddenly and without warning, such as when you wake up in the morning and look in the mirror. Sudden onset of facial nerve paralysis can be psychologically damaging, making it difficult to perform rehabilitation while staring directly at a paralyzed face. Even healthy individuals without facial nerve paralysis may be distracted by blemishes, bruises, acne, or acne scars when looking directly at their own face, making it difficult to concentrate on facial movements. In response to this, in this embodiment, a composite image in which a mask image KG is superimposed on a facial image is displayed on the user terminal 10, thereby reducing the psychological burden of staring directly at the face and allowing the user to focus on facial movements without being distracted by blemishes or other imperfections. On the other hand, in order to reduce the psychological burden of looking directly at the face, one possible measure is to display only the mask image KG on the user terminal 10. However, if only the mask image KG is displayed, it becomes difficult for the user to tell whether the mask image KG is based on the user's own face or whether it is simply an illustration of a normal face depicted using wires or the like. As a result, even if the user moves their face, it becomes difficult for them to actually feel the movement in the mask image KG, which may reduce the effectiveness of rehabilitation.

[0026] From this perspective, in this embodiment, the mask image KG is configured to enable observation of the user's facial movement in the facial image and to partially occlude the user's face in the facial image. Specifically, the display control unit 24 generates, as the mask image KG, an image generated based on feature points extracted from the facial image. Because the positions of the feature points extracted from the facial image vary depending on the facial movement, the facial movement can be easily observed even when the mask image is superimposed on the facial image. For example, the display control unit 24 uses, as the mask image KG, a wire-represented image of the face generated by connecting adjacent feature points based on the feature points extracted from the facial image (see the mask image KG in FIG. 4). Alternatively, the mask image KG is configured to use, as the mask image KG, an image of a point cloud representing the feature points themselves extracted from the facial image (see the mask image KG in FIG. 5). In addition, the display control unit 24 may cause the user terminal 10 to display, as a mask image KG, an image showing an expression of an avatar corresponding to the user based on multiple feature points extracted from a facial image (see the mask image KG shown as a display icon (game character) in Figures 14 and 15).

[0027] The display control unit 24 may also superimpose a region image RG indicating the area to be massaged on the composite image (see region image RG in FIG. 4). The display control unit 24 may also superimpose a direction image DG indicating the direction in which the massage will be performed on the composite image (see region image RG in FIG. 4).

[0028] Furthermore, the display control unit 24 causes the user terminal 10 to display an image (response image RG) indicating whether or not the user undergoing rehabilitation is performing the correct rehabilitation. The determination unit 26, which will be described later, determines whether or not the correct rehabilitation is being performed, and the display control unit 24 causes the user terminal 10 to display the determination result. If the correct rehabilitation is not being performed, the display control unit 24 causes the user terminal 10 to display a response image indicating that fact (see response image RG in FIG. 8). On the other hand, if the correct rehabilitation is being performed, the display control unit 24 causes the user terminal 10 to display a response image indicating that fact (see response image RG in FIG. 9).

[0029] In addition to the response image, whether the rehabilitation is being performed correctly may be indicated by a voice, an alarm sound, or the like. In this case, the support server 20 transmits response sound information indicating the determination result of whether the rehabilitation is being performed correctly by a voice, an alarm sound, or the like to the user terminal 10 via the output unit 27 and the communication unit 21, which will be described later. The user terminal 10 receives the response sound information notified from the support server 20, and outputs a voice, an alarm sound, or the like corresponding to the received response sound information from a speaker (not shown) of the user terminal 10.

[0030] The feature point detection unit 25 detects feature points in the face image. Feature points are pixels in the face image that correspond to the contours of each of the eyebrows, eyes, nose, cheeks, and mouth, which are facial features. The feature point detection unit 25 detects multiple feature points for each feature. For example, for the "eyebrows" as a facial feature, feature points corresponding to the beginning, ridge, and end of the eyebrows, which correspond to the contours of the eyebrows, are detected. For the "eyes," feature points corresponding to the inner corner, center of the upper eyelid, center of the lower eyelid, and outer corner of the eye, which correspond to the contours of the eyes, are detected. Similarly, for the nose, cheeks, and mouth, multiple feature points corresponding to the contours of the nose, cheeks, and mouth are detected.

[0031] The determining unit 26 determines whether the face has been moved correctly based on the face image. In the rehabilitation support app, when a user uses the rehabilitation support service by "training" to perform facial movement training, the user terminal 10, under the control of the imaging control unit 23, captures an image of the user's facial movements in response to the training. The user terminal 10 transmits the captured image TG to the support server 20. The determination unit 26 acquires the captured image TG captured by the user terminal 10 via the communication unit 21. The determination unit 26 extracts a first region to be moved in training and a second region to be kept stationary from the facial image based on the feature points extracted from the facial image. The first region is the region containing muscles that move either the "eyes" or the "mouth" on the face. The second region is the region containing muscles that move a part of the face other than the part corresponding to the first region. For example, when training to move the eyes by closing and opening them, the first region is the region surrounding the eyes, where the orbicularis oculi muscle that moves the eyes is located. In this case, the second region is the region where the muscles that move parts of the face other than the eyes, such as the mouth and eyebrows, are located. The determination unit 26 determines whether the second region is moving in response to the movement of the first region during training. If the second region is moving in response to the movement of the first region, the determination unit 26 determines that the face is not moving correctly. On the other hand, if the second region is not moving in response to the movement of the first region, the determination unit 26 determines that the face is moving correctly. The determination unit 26 causes the determination result to be displayed on the user terminal 10 via the display control unit 24.

[0032] The output unit 27 outputs the determination result by the determination unit 26, and for example, causes the user terminal 10 to output the determination result as to whether or not the face is moving correctly by voice, alarm, or the like.

[0033] Here, the processing performed by the rehabilitation support system will be described with reference to Fig. 2 to Fig. 15. Fig. 2 to Fig. 15 are diagrams for explaining the processing performed by the rehabilitation support system according to the embodiment.

[0034] 2 shows an example of a screen displayed on the user terminal 10 when the rehabilitation support app is launched. This figure shows an example in which multiple menus are presented in response to the launch of the rehabilitation support app. The multiple menus are menus of services provided by the rehabilitation support app, including massage, training, recording laterality, history, and games (i.e., IWP games). As shown in this figure, the display may also show "Paralysis: Right Side" or "X Days Completed." In this figure, "Paralysis: Right Side" indicates which side of the face is paralyzed, the left or right side, and indicates the side on which rehabilitation massages are focused. "X Days Completed" indicates the number of days that rehabilitation such as massages, training, or games has been completed. By showing the number of days of rehabilitation, the user can maintain motivation to continue rehabilitation.

[0035] When a user operating the user terminal 10 selects the menu button MB1 or MB2 displayed corresponding to "massage," a rehabilitation support service using "massage" is executed. Menu button MB1 is a button for selecting a short massage of about 5 minutes called the "easy course," and menu button MB2 is a button for selecting a long massage of about 9 minutes called the "firm course." When the user operating the user terminal 10 selects the menu button MB3 displayed corresponding to "Training", a rehabilitation support service for "Training" is executed. When the user operating the user terminal 10 selects the menu button MB4 displayed in correspondence with "I-Woop Game", a rehabilitation support service using a "game" is executed.

[0036] Fig. 3 shows an example of a screen that is displayed on the user terminal 10 when the "Easy Course" corresponding to the menu button MB1 in Fig. 2 is selected. Note that although the "Easy Course" and the "Firm Course" differ in the execution time and the number of target areas to be massaged, there is no difference in the massage method. Therefore, in the following, only the "Easy Course" will be explained, and an explanation of the "Firm Course" will be omitted. As shown in FIG. 3, when menu button MB1 is selected in FIG. 2, an example image MG is displayed as an "easy course" to explain how to massage. Here, the example image MG displays an image showing that muscles are relaxed through massage, and that a massage is performed to relax three basic types of muscles (forehead, cheeks, and around the eyes). Also displayed is a muscle diagram showing the locations of the three basic types of muscles (forehead, cheeks, and around the eyes). In this muscle diagram, the facial muscles that form facial expressions are illustrated, and the locations of the three types of muscles (forehead, cheeks, and around the eyes) that are the target areas for massage are emphasized. In the example shown in this figure, the three basic types of muscles are the forehead, cheeks, and muscles around the eyes, but this is not limiting. Of course, the three basic types of muscles may also be massaged: the cheeks, the muscles around the eyes, and the muscles around the mouth. By checking the check button in the "Skip Explanation" display field in Figure 3, you can skip the explanatory video that will be displayed afterwards. The explanatory video will explain how to give a massage using audio and video, for example.

[0037] 4 and 5 show examples of screens displayed on the user terminal 10 when an actual massage is being performed. 4 and 5, the user terminal 10 displays a captured image TG, a mask image KG, a model image MG, and the like. The captured image TG is a facial image of the user captured by the imaging unit 14. This facial image may be a still image or a moving image. A moving image is more preferable in terms of being able to confirm the movements of the massaging hands. The mask image KG is an image superimposed on the face image by the display control unit 24. In the example of FIG. 4, a mask image KG is shown in which adjacent feature points extracted from a face image are connected to form a wire representation. In the example of FIG. 5, a mask image KG is shown in which feature points extracted from a face image are expressed as a point cloud. The model image MG is an image showing how a massage is performed correctly by a professional, etc. In Figures 4 and 5, the model image MG shows how the forehead muscles (frontalis muscles) are massaged in the vertical direction. 4 and 5, a region image RG indicating the area to be massaged is displayed superimposed on the composite image, and a direction image DG indicating the direction in which the massage will be performed is displayed superimposed on the composite image. It is also possible to switch between displaying a wire-represented image as shown in Fig. 4 and displaying an image represented by a point cloud as shown in Fig. 5 as the mask image KG. In this case, a switch button for alternately switching between the wire-represented image and the point cloud image for the mask image KG is displayed, for example, in the upper right corner of the display screen of the user terminal 10.

[0038] FIG. 6 shows an example of a screen that is displayed on the user terminal 10 when "Training" corresponding to the menu button MB3 in FIG. 2 is selected. As shown in FIG. 6, when menu button MB3 is selected in FIG. 2, model images MG are displayed as "Training" to explain the training method. Here, the model images MG show three types of exercises, such as moving facial muscles (facial muscles) and moving either the eye or mouth muscles. Also shown is a muscle diagram showing the positions of the facial muscles that move the eyes and mouth. In this muscle diagram, the facial muscles that form facial expressions are illustrated, and the positions of the facial muscles that move the eyes and mouth, which are the target areas to be massaged, are emphasized. In Figure 6, by checking the check button provided in the "Skip Explanation" display field, you can skip the explanatory video that will be displayed afterwards. The explanatory video will explain the training method using audio and video, for example.

[0039] 7 to 12 show examples of screens displayed on the user terminal 10 when actually performing training.

[0040] 7 to 9 show examples of images displayed on the user terminal 10 in training for moving the mouth by making the action of saying the word "eee." The user terminal 10 displays a captured image TG, a mask image KG, a model image MG, and the like. The captured image TG is a facial image of the user captured by the imaging unit 14. This facial image may be a still image or a moving image. A moving image is more preferable in terms of being able to confirm facial movement. The mask image KG is an image superimposed on the facial image by the display control unit 24. In Fig. 7 to Fig. 9, the mask image KG is shown in which adjacent feature points extracted from the facial image are connected to form wire representations. The model image MG is an image showing correct facial movements as determined by an expert or the like. In Figures 7 to 9, the model image MG shows the correct mouth movements when uttering the word "I." The model image MG also shows a muscle diagram that allows one to distinguish between facial muscles that move (here, the orbicularis oris muscle) and those that do not move (here, the orbicularis oculi muscle). 8 shows a response image RG that is displayed when it is determined that the user is not moving their face correctly. Here, an example is shown in which an image with the message "Do not close your eyes" is displayed as the response image RG when the user closes their eyes while moving their mouth to say the word "yes" in response to training. In this way, by specifying a specific part of the body (here, the eyes) and instructing the user not to move that part, the user can recognize the correct mouth movements. 9 shows a response image RG that is displayed when it is determined that the user is moving their face correctly. By displaying such a response image RG, the user can recognize that their movements are correct.

[0041] 10 shows an example of images displayed on the user terminal 10 in training to move the eyes by closing the eyes. The user terminal 10 displays a captured image TG, a mask image KG, a model image MG, and the like. The captured image TG is a facial image of the user captured by the imaging unit 14. This facial image may be a still image or a moving image. A moving image is more preferable in terms of being able to confirm facial movement. The mask image KG is an image superimposed on the facial image by the display control unit 24. Fig. 10 shows the mask image KG in which adjacent feature points extracted from the facial image are connected to form wire representations. The model image MG is an image showing correct facial movements as determined by experts, etc. In FIG. 10, the model image MG shows the correct eye movements when closing the eyes. The model image MG also shows a muscle diagram that distinguishes between facial muscles that move (here, the orbicularis oculi muscle) and those that do not (here, the orbicularis oris muscle). This diagram also shows an example in which a hand is placed over the mouth to prevent the mouth from moving in sync with closing the eyes, or to make any movement immediately noticeable even if it does.

[0042] 11 and 12 show examples of images displayed on the user terminal 10 in training to move the eyes by opening the eyes wide. The user terminal 10 displays a captured image TG, a mask image KG, a model image MG, and the like. The captured image TG is a facial image of the user captured by the imaging unit 14. This facial image may be a still image or a moving image. A moving image is more preferable in terms of being able to confirm facial movement. The mask image KG is an image superimposed on the facial image by the display control unit 24. 11 and 12 show the mask image KG in which adjacent feature points extracted from the facial image are connected to form wire representations. The model image MG is an image showing correct facial movements as determined by an expert or the like. In Figures 11 and 12, the model image MG shows the correct mouth movements when the eyes are wide open. The model image MG also shows a muscle diagram that allows distinction between facial muscles that move (here, the orbicularis oculi muscle) and those that do not move (here, the orbicularis oris muscle). 12 shows a response image RG that is displayed when it is determined that the user is not moving their face correctly. Here, an example is shown in which an image with the message "Do not move your eyebrows" is displayed as the response image RG when the user's eyebrows move when they open their eyes in response to training. In this way, by specifying a specific part (here, the eyes) and instructing the user not to move that part, the user can recognize the correct eye movements. It is also possible to switch between displaying a wire-represented image as the mask image KG as shown in Figures 7 to 12 and displaying an image represented by a point cloud as shown in Figure 5. In this case, a switch button for alternately switching between the wire-represented image and the point cloud image for the mask image KG is displayed, for example, in the upper right corner of the display screen of the user terminal 10.

[0043] FIG. 13 shows an example of a screen that is displayed on the user terminal 10 when "Games" corresponding to the menu button MB4 in FIG. 2 is selected. As shown in FIG. 13, when menu button MB4 is selected in FIG. 2, a model image MG is displayed as "Game" explaining the rules of the game. Here, the model image MG explains that facial muscles (facial muscles) should be moved by making facial expressions such as "eeek," "uuu," and "pooh" in accordance with the displayed icon (game character). Also shown are illustrations of the displayed icons (game characters) making the "eeek," "uuu," and "pooh" expressions. In this illustration, a toothbrush is attached to the "eeek" expression, a drink with a straw is attached to the "pooh" expression, and a balloon is attached to the "pooh" expression. This suggests that the "eeek" expression should be the expression when brushing teeth, the "pooh" expression should be the expression when drinking a drink through a straw, and the "pooh" expression should be the expression when blowing up a balloon, making it easier to create each expression. Other precautions include making sure your eyes don't get caught and move along with the movement, and not putting too much strain into making facial expressions. When the "Start" operation button in FIG. 13 is operated, the game starts.

[0044] 14 and 15 show examples of screens displayed on the user terminal 10 when the game is actually being played.

[0045] 14 shows an example of an image displayed on the user terminal 10 at the start of a game. The user terminal 10 displays a timer image JG, a captured image TG, a mask image KG, a score image SG, and the like. The timer image JG is an image that measures the time that the game is being played. By measuring the time, it is possible to prevent muscle damage caused by continuously moving the face for a long period of time. The captured image TG is a facial image of the user captured by the imaging unit 14. It is assumed that this facial image is a moving image. The mask image KG is an image in which the display control unit 24 displays an expression of an avatar corresponding to the user as a display icon (game character) based on a plurality of feature points extracted from a facial image. The mask image KG is an image superimposed on the facial image by the display control unit 24. In this figure, as shown in the lower left of the display screen of the user terminal 10, the mask image KG is represented by wires by connecting adjacent feature points extracted from the facial image. The score image SG is an image showing the score acquired in the game. In this figure, since it is the start of the game, the game score is shown to be "0 (zero)". Here, if the facial expression that corresponds to the character's movement can be maintained for a predetermined period of time (for example, about one second), a score can be acquired.

[0046] Fig. 15 shows an example of an image displayed on the user terminal 10 when the game score reaches "1" after the game has started. Fig. 15 displays the same timer image JG, captured image TG, mask image KG, and score image SG as Fig. 14. As time has passed since the game started, the remaining time in the timer image JG has decreased, and points have been added to the score image SG. The captured image TG and mask image KG are the same as those in Fig. 14.

[0047] Here, the flow of processing according to this embodiment will be described with reference to Fig. 16 to Fig. 18. Fig. 16 to Fig. 18 are sequence diagrams showing the flow of processing performed by the support server according to this embodiment.

[0048] FIG. 16 is a flowchart showing the flow of processing for providing rehabilitation support through massage. First, the support server 20 starts rehabilitation support by massage (step S10). When a rehabilitation support app is started on the user terminal 10 and the menu button MB1 or MB2 labeled "Massage" is selected, the support server 20 starts rehabilitation support by massage. The support server 20 causes the user terminal 10 to display a model image MG that explains how to massage. Next, the support server 20 determines whether or not to display an explanatory video explaining a massage method on the user terminal 10 (step S11). If an operation to check the check button provided in the display field for "Skip explanation" on the model image MG displayed on the user terminal 10 is performed, the support server 20 determines to omit and not display the explanatory video. On the other hand, if an operation to check the check button provided in the display field for "Skip explanation" is not performed, the support server 20 determines to display the explanatory video. If an explanatory video explaining a massage method is to be displayed, the support server 20 causes the user terminal 10 to display the explanatory video (step S12). Next, the support server 20 instructs the user terminal 10 to start capturing an image, and causes the user terminal 10 to capture a moving image of the user's face (step S13). The user terminal 10 transmits the captured face image to the support server 20. The support server 20 generates a mask image using the feature points of the face image based on the face image, and causes the user terminal 10 to display a composite image in which the generated mask image is superimposed on the face image (step S14). Next, the support server 20 displays a model image MG for performing a massage on the user terminal 10 (step S15). The model image MG displayed here shows how a massage is performed correctly by a professional or the like, along with the area to be massaged and the direction of the massage. In addition, a muscle diagram in which the muscles for moving the area to be massaged are marked may also be displayed. Next, the support server 20 displays a composite image in which the mask image is superimposed on the face image, and further superimposed on the composite image are an area image RG indicating the area to be massaged and a direction image DG indicating the direction in which the massage will be performed (step S16). Next, the support server 20 guides the user to have a massage (step S17). For example, the support server 20 guides the user to have a massage by displaying a text message such as "Let's have a massage" on the user terminal 10. Then, the support server 20 determines whether there are any parts that have not been massaged (step S18). If there are any parts that have not been massaged, the support server 20 returns to step S15 and continues the massage. On the other hand, if there are no parts that have not been massaged and all massages have been completed, the support server 20 ends the process.

[0049] FIG. 17 shows a flowchart showing the flow of processing for providing rehabilitation support through training. First, the support server 20 starts rehabilitation support through training (step S20). When a rehabilitation support app is launched on the user terminal 10 and the menu button MB3 labeled "Training" is selected, the support server 20 starts rehabilitation support through training. The support server 20 causes the user terminal 10 to display a model image MG that explains the training method. Next, the support server 20 determines whether or not to display an explanatory video explaining the training method on the user terminal 10 (step S21). If an operation to check the check button provided in the display field for "Skip explanation" on the model image MG displayed on the user terminal 10 is performed, the support server 20 determines to omit and not display the explanatory video. On the other hand, if an operation to check the check button provided in the display field for "Skip explanation" is not performed, the support server 20 determines to display the explanatory video. If an explanatory video explaining the training method is to be displayed, the support server 20 causes the user terminal 10 to display the explanatory video (step S22). Next, the support server 20 instructs the user terminal 10 to start capturing an image, and causes the user terminal 10 to capture a moving image of the user's face (step S23). The user terminal 10 transmits the captured face image to the support server 20. The support server 20 generates a mask image using the feature points of the face image based on the face image, and displays a composite image in which the generated mask image is superimposed on the face image on the user terminal 10 (step S24). Next, the support server 20 displays a model image MG for training on the user terminal 10 (step S25). The model image MG displayed here shows how an expert or the like moves their face correctly. Next, the support server 20 causes the user terminal 10 to display a diagram that distinguishes between areas that are moved by training and areas that are not moved, for example, a muscle diagram that shows the facial muscles that are moved (orbicularis oris) and the facial muscles that are not moved (orbicularis oculi) as shown in Figure 7 (step S26). Next, the support server 20 guides the user to do training (step S27). For example, the support server 20 guides the user to do training by displaying a text sentence such as "Make the facial expression when you say the word 'Eee'" on the user terminal 10. Next, the support server 20 determines whether the training is being performed correctly (step S28). The support server 20 determines whether the user is moving their face correctly in response to the training. The support server 20 determines that the user is moving their face correctly if the area that should be moved by the training is moving and the area that should not be moved is not moving in conjunction with that movement. On the other hand, if the area that should be moved by the training is moving but the area that should not be moved is moving in conjunction with that movement, the support server 20 determines that the user is not moving their face correctly. If the user is moving their face correctly, the support server 20 outputs a response indicating OK (step S29). For example, the support server 20 displays a response image RG indicating OK on the user terminal 10 or outputs a response voice indicating OK from the user terminal 10. On the other hand, if the user is not moving their face correctly, the support server 20 outputs a response indicating NG (step S30). For example, the support server 20 causes the user terminal 10 to display a response image RG indicating NG or causes the user terminal 10 to output a response voice indicating NG. Then, the support server 20 determines whether there are any parts that have not been trained (step S31). If there are any other parts that have not been trained, the support server 20 returns to step S25 and continues training. On the other hand, if there are no other parts that have not been trained and all training has been completed, the support server 20 ends the process.

[0050] FIG. 18 shows a flowchart showing the flow of processing for providing rehabilitation support through a game. First, the support server 20 starts rehabilitation support through a game (step S40). When a rehabilitation support app is launched on the user terminal 10 and the menu button MB4 labeled "Game" is selected, the support server 20 starts rehabilitation support through a game. The support server 20 causes the user terminal 10 to display a model image MG that explains the rules of the game (step S41). Next, the support server 20 instructs the user terminal 10 to start capturing an image, and causes the user terminal 10 to capture a moving image of the user's face (step S42). The user terminal 10 transmits the captured face image to the support server 20. The support server 20 generates a mask image using the feature points of the face image based on the face image, and causes the user terminal 10 to display a composite image in which the generated mask image is superimposed on the face image (step S43). Next, the support server 20 displays a model image MG for starting the game on the user terminal 10 (step S44). The model image MG displayed here shows an illustration of a game character making a specific facial expression. Next, the support server 20 prompts the user to start the game (step S45). For example, the support server 20 prompts the user to start the game by displaying a text message such as "Start the game" on the user terminal 10. Next, the support server 20 determines whether the game achievement conditions have been met (step S46). If the user can maintain a facial expression that corresponds to the character's movement for a predetermined time (for example, about one second), the support server 20 determines that the game achievement conditions have been met. On the other hand, if the user cannot maintain a facial expression that corresponds to the character's movement for a predetermined time (for example, about one second), the support server 20 determines that the game achievement conditions have not been met. If the game achievement conditions have been met, the support server 20 increments the game score (step S47). Then, the support server 20 determines whether a predetermined execution time has elapsed since the start of the game (step S48). If the predetermined execution time has not elapsed since the start of the game, the support server 20 returns to step S44 and continues the game. On the other hand, if the predetermined execution time has elapsed since the start of the game, the support server 20 ends the processing.

[0051] As described above, the support server 20 according to the embodiment is an example of a support device. The support server 20 includes a communication unit 21 and a display control unit 24. The communication unit 21 is an example of an acquisition unit and acquires a facial image of a user's face from the user terminal 10. The display control unit 24 displays a composite image in which a mask image is superimposed on the facial image on the user terminal 10, and guides the user terminal 10 on a method for improving the movement of facial mimic muscles. The mask image is an image generated based on feature points extracted from the facial image. The mask image allows the user's facial movement to be observed in the facial image and partially obscures the user's face in the facial image. This allows the support server 20 according to the embodiment to guide the user terminal 10 on a method for improving the movement of facial mimic muscles. Moreover, since the user's facial image can be displayed on the user terminal 10, the user can visually check their facial expressions while undergoing rehabilitation, allowing them to continue correct rehabilitation even when it is difficult for them to receive guidance from an expert. Furthermore, a composite image in which a mask image is superimposed on a face image can be displayed on the user terminal 10, which reduces the psychological burden of looking directly at the face, or allows the user to concentrate on the movement of their own face without being distracted by blemishes on the face.

[0052] Furthermore, in the support server 20 according to the embodiment, the mask image is an image in which adjacent feature points among a plurality of feature points extracted from the face image are connected. As a result, in the support server 20 according to the embodiment, the mask image expressed as wires can be superimposed on the face image, making it easier to observe the movement of the face and reducing the psychological burden of looking directly at the face.

[0053] In addition, in the support server 20 according to the embodiment, the mask image is an image showing a plurality of feature points extracted from the face image. As a result, in the support server 20 according to the embodiment, a point cloud showing the feature points of the face can be superimposed on the face image as a mask image, making it easier to observe the movement of the face and reducing the psychological burden of looking directly at the face.

[0054] Furthermore, in the support server 20 according to the embodiment, the display control unit 24 displays model images and facial images that explain improvement methods on the user terminal 10, and guides the user to carry out the method corresponding to the model image. As a result, in the support server 20 according to the embodiment, the user can compare the model image with the facial image and perform massage or training by imitating the model image, and can continue correct rehabilitation even when it is difficult to receive guidance from a professional.

[0055] In addition, in the support server 20 according to the embodiment, the improvement method is a method of massaging the face. The display control unit 24 extracts an area to be massaged in the facial image based on feature points extracted from the facial image, and superimposes an image showing the extracted area on the composite image. As a result, in the support server 20 according to the embodiment, the user can massage while checking the area to be massaged, and can continue correct rehabilitation even when it is difficult to receive guidance from a specialist.

[0056] Furthermore, in the support server 20 according to the embodiment, the improvement method is a training method for moving the face. The support server 20 includes a determination unit 26 and an output unit 27. The determination unit 26 determines whether the face is moving correctly in response to the training based on the facial image. The output unit 27 outputs the determination result by the determination unit 26. As a result, the support server 20 according to the embodiment can determine whether the training is being performed correctly and provide the determination result as feedback to the user, allowing the user to continue correct rehabilitation even when it is difficult to receive guidance from an expert.

[0057] Furthermore, in the support server 20 according to the embodiment, the determination unit 26 extracts a first region to be moved during training and a second region to be kept still from the facial image based on feature points extracted from the facial image. If the second region moves in response to the movement of the first region, the determination unit 26 determines that the facial movement is incorrect. This allows the support server 20 according to the embodiment to detect signs of the onset of involuntary movements such as pathological synkinesia, thereby enabling proper rehabilitation to be continued even when it is difficult to receive guidance from an expert.

[0058] Furthermore, in the support server 20 according to the embodiment, a region containing muscles that move either the eyes or the mouth on the face is the first region. A region containing muscles that move a part of the face other than the part corresponding to the first region is the second region. As a result, the support server 20 according to the embodiment can detect signs of pathological synkinesis, in which the eyelids and mouth move involuntarily together, and therefore, it is possible to continue correct rehabilitation even when it is difficult to receive guidance from a specialist.

[0059] Furthermore, in the support server 20 according to the embodiment, when the determination unit 26 determines that the second region is moving in response to the movement of the first region, the display control unit 24 causes the user terminal 10 to display a response image indicating that the face is not moving correctly. As a result, the support server 20 according to the embodiment can display a response indicating a sign of pathological synkinesis on the user terminal 10, thereby enabling the user to continue correct rehabilitation even when it is difficult to receive guidance from a specialist.

[0060] The above describes an embodiment of the present invention. Note that some or all of the functions of the rehabilitation support system 1 and the support server 20 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be for implementing some of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0061] The embodiments of the present invention have been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope of the gist of the present invention. [Explanation of symbols]

[0062] 1 Rehabilitation support system 10 User terminal 11 Communications Department 12 Storage section 13 Control Unit 14 Imaging unit 15 Display section 16 Operation input section 20 Support Server 21 Communications Department 22 Memory section 23 Imaging control unit 24 Display control unit 25 Feature point detection unit 26 Judgment section 27 Output section

Claims

1. an acquisition unit that acquires a facial image of a user from a user terminal; a display control unit that displays on the user terminal a composite image in which the mask image is generated based on feature points extracted from the face image acquired by the acquisition unit, the composite image being a mask image that allows observation of the facial movement of the user in the face image and that covers all or part of the user's face in the face image and that guides the user terminal to a method for improving the movement of mimetic muscles in the face; A support system comprising:

2. the mask image is an image obtained by connecting adjacent feature points among a plurality of feature points extracted from the face image; The assistance system according to claim 1 .

3. the mask image is an image showing a plurality of feature points extracted from the face image; The assistance system according to claim 1 .

4. the mask image is an image showing an expression of an avatar corresponding to the user based on a plurality of feature points extracted from the face image; The assistance system according to claim 1 .

5. the display control unit displays a model image explaining the improvement method and the facial image on the user terminal, and guides the user to carry out the method corresponding to the model image. The assistance system according to claim 1 .

6. The improvement method is a method of massaging the face, the display control unit extracts an area to be massaged in the facial image based on the feature points extracted from the facial image, and superimposes an image showing the extracted area on the composite image. The assistance system according to claim 1 .

7. The improvement method is a facial movement training method, a determination unit that determines whether the face is moving correctly in response to training based on the facial image; an output unit that outputs a determination result by the determination unit; The assistance system of claim 1 further comprising:

8. the determination unit extracts a first region to be moved in training and a second region to be kept still from the face image based on feature points extracted from the face image, and determines that the face is not being moved correctly when the second region is moving in accordance with the movement of the first region. The assistance system according to claim 7.

9. the determination unit extracts a first region to be moved in training and a second region to be kept still from the face image based on feature points extracted from the face image, and determines that the face is being moved correctly when the second region does not move in response to the movement of the first region. The assistance system according to claim 7.

10. The first region is a region where there is a muscle that moves either the eyes or the mouth on the face, and the second region is a region where there is a muscle that moves a part of the face other than the part corresponding to the first region. The assistance system according to claim 8 or claim 9.

11. When the determination unit determines that the second area is moving in accordance with the movement of the first area, the display control unit outputs a response to the user terminal indicating that the face is not moving correctly. The assistance system according to claim 8.

12. A support method performed by a computer in a support system, an acquisition unit acquires a facial image of the user from the user terminal; a display control unit displays on the user terminal a composite image in which the mask image, which is generated based on feature points extracted from the face image acquired by the acquisition unit, is superimposed on the face image, and which allows the user to observe facial movements in the face image and covers all or part of the user's face in the face image, and provides the user terminal with an improvement method for improving the movements of mimetic muscles in the face; How to help.

13. The support system computer A facial image of the user is acquired from the user terminal; a mask image generated based on feature points extracted from the acquired face image, which allows the user's facial movement in the face image to be observed, and which masks the entire or part of the user's face in the face image; a composite image superimposed on the face image, and the user terminal is prompted to provide an improvement method for improving the movement of mimetic muscles in the face; program.

Images