Parkinson'mask face 'rehabilitation training method

By using mobile phone-based facial motion recognition and a game format driven by opera music, the problem of high cost of dedicated equipment has been solved, enabling low-cost and efficient Parkinson's disease rehabilitation training, improving facial muscle flexibility and cognitive ability, and providing convenient data analysis tools.

CN119015672APending Publication Date: 2024-11-26ZHEJIANG UNIV +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410738996.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-07
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing technologies for Parkinson's disease rehabilitation training based on specialized equipment suffer from high costs and a small target audience, making them difficult to widely adopt in China.

Method used

This paper designs a Parkinson's "mask face" rehabilitation training method based on mobile devices. It utilizes the camera and microphone of the mobile device to provide basic and advanced training modes through facial motion recognition and games driven by opera music. It combines facial motion unit recognition and scoring feedback to achieve facial muscle rehabilitation training.

Benefits of technology

It enables low-cost and easy-to-use Parkinson's disease rehabilitation training, improves patients' facial expression flexibility and cognitive ability, and provides convenient data recording and analysis tools to assist in diagnosis and treatment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119015672A_ABST
    Figure CN119015672A_ABST
Patent Text Reader

Abstract

The invention discloses a Parkinson'mask face 'rehabilitation training method which comprises the following steps: step 1, a user enters an interface through a mobile device, and meanwhile, according to whether the user registers an account number or not, if the account number is registered, the next step is continued, and if the account number is not registered, the user registration step is entered, and then the next step is continued; step 2, logging in an account, entering a main interface, selecting a mode through the main interface, and providing corresponding service by a business layer according to the selected mode; 3, basic training is started, then a basic training game is operated, and the user carries out training according to the basic training game; and 5, entering advanced training, and then running an advanced training game. According to the Parkinson'mask face 'rehabilitation training method, rehabilitation training can be effectively achieved through the mobile device through setting of the first step to the fifth step, and compared with a mode in the prior art, the method does not depend on special equipment, and only a microphone and a camera of the mobile device need to be used.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a rehabilitation training method, and more particularly to a Parkinson's "mask face" rehabilitation training method. Background Art

[0002] In the field of digital therapy research for Parkinson's disease motor symptoms, most of the related work is based on games to add fun experience to the rehabilitation treatment of Parkinson's disease patients. Lin Zhicheng et al.

[24] used Microsoft's XBOX motion sensing game to conduct balance training for Parkinson's disease patients, which effectively improved the patients' limb coordination ability, ankle control ability and body balance ability. Dauvergne et al.

[25] developed a tablet game called "Rhythm Worker" to help Parkinson's disease patients improve their movement rhythm. Patients are required to tap the sound on the screen as accurately as possible to adapt to the beat of various rhythmic auditory stimuli. Fernández-González et al.

[26] developed six electronic games such as piano games and grasping games based on Ultraleap's extended reality device Leap MotionController, aiming to simulate common movements such as palm grasping, finger flexion and extension, or hand supination in Parkinson's disease rehabilitation training. Campo-Prieto et al.

[27] used virtual reality equipment to let subjects practice different boxing skills in digital space, such as defense, jab, cross, hook, uppercut, etc., to fully exercise the subjects' trunk, head and lower limbs and strengthen their coordination ability. The experimental results show that immersive virtual reality video games are feasible as auxiliary treatments for patients with mild to moderate Parkinson's disease, with high usability and patient satisfaction. Similarly, the above work relies too much on special equipment, and still has the problems of high cost of use and small audience group, which is not suitable for my country's national conditions. How to design digital therapy application software that is affordable and easy to use based on widely popular mobile devices such as mobile phones is a major challenge faced by the development of new technologies for early intervention treatment of Parkinson's disease. Summary of the invention

[0003] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a Parkinson's "mask face" rehabilitation training method based on a mobile phone mobile device.

[0004] To achieve the above object, the present invention provides the following technical solution: a Parkinson's "mask face" rehabilitation training method, characterized in that it includes the following steps:

[0005] Step 1: The user enters the interface through a mobile device and proceeds to the next step based on whether the user has a registered account. If the user has a registered account, the user proceeds to the next step. If the user has not registered an account, the user registration step is entered and then the next step is continued.

[0006] Step 2: Log in to your account and enter the main interface. Select a mode through the main interface. The business layer provides corresponding services based on the selected mode. The modes include basic training mode, face detection mode and advanced training mode. If you select the basic training mode, continue to step 3 or select the face detection mode to jump to step 4 or select the advanced training mode to jump to step 5.

[0007] Step 3: Enter basic training, then run the basic training game, the user trains according to the basic training game, and after the training is completed, the overall feedback returns to step 2;

[0008] Step 4: perform face detection, then return the face detection result, and then return to step 2;

[0009] Step 5: Enter advanced training, then run the advanced training game. The user trains according to the advanced training game, and after the training is completed, the overall feedback returns to step 2.

[0010] As a further improvement of the present invention, the basic training game in step 3 includes the following steps:

[0011] Step 31: Provide a training part selection interface for the user to choose. After the user completes the selection, the client displays the facial movements of the instructor corresponding to the training part, and plays the explanation voice in the form of pictures and texts;

[0012] Step 32: Play opera music, and the client performs facial action unit recognition on the face image captured by the camera to obtain the face anchor point, and overlays the opera filter image on the user's real face based on the face anchor point, and prompts the user to make corresponding expressions;

[0013] Step 33, determining whether the user's facial expression action is successful, if successful, playing a motivational graphic message, if unsuccessful, displaying a message prompting the user to adjust the facial expression;

[0014] Step 34: Repeat steps 31 to 33 to complete basic training.

[0015] As a further improvement of the present invention, the advanced training game in step 5 includes the following:

[0016] Step 51, providing a training difficulty selection interface for the user to select, and after the user completes the selection, playing a medley of opera music for training different parts of the body, prompting the user to make facial expressions and movements according to the medley of opera music;

[0017] Step 52, determining whether the user's expression action is continuous and successful, and if so, playing an incentive graphic or voice message;

[0018] Among them, during the playback of the opera music medley, the opera filter picture changes as the opera music changes.

[0019] As a further improvement of the present invention, the specific method of facial action unit recognition in step 32 is as follows: first, the patient's real-time facial information is obtained using the video track, and the face recognition model and facial features positioning model provided in the faceapi library are used to process the face of the current image. The face recognition model will first identify the faces in the incoming video stream, and select the face with the highest credibility, and input the face into the facial features positioning model to obtain the anchor points of various parts of the face. After obtaining the anchor points, the captured face is input into the facial unit recognition model, so that it can analyze the user's facial expressions in real time and accurately identify different facial action units.

[0020] As a further improvement of the present invention, the specific method of judging whether the user's expression action is successful in step three is: inputting the patient's expression into the facial unit detection model, scoring according to the result returned by the model, the model scores the corresponding action according to the difference between the current expression and the natural expression in a certain facial unit, and judging whether the user's expression action is successful according to the score.

[0021] The beneficial effects of the present invention are that interesting facial opera exercises can relieve depression and cognitive ability can be improved in the form of human-computer interactive games. In addition, during the entire training process, only the camera, microphone and speaker of the mobile device need to be used. Therefore, compared with the methods in the prior art, the present invention is affordable and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the Parkinson's "mask face" rehabilitation training method of the present invention;

[0023] Figure 2 Flowchart for facial action unit recognition;

[0024] Figure 3 a is the "Rolling Lantern" opera filter provided in the embodiment, Figure 3 b is the opera filter of "Heavenly Girl Scattering Flowers" and Figure 3 c is a schematic diagram of the opera filter for “The Case of the Chopping of the Beauty”. DETAILED DESCRIPTION

[0025] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0026] Reference Figures 1 to 3 As shown, a Parkinson's "mask face" rehabilitation training method of this embodiment is characterized by comprising the following steps:

[0027] Step 1: The user enters the interface through a mobile device and proceeds to the next step based on whether the user has a registered account. If the user has a registered account, the user proceeds to the next step. If the user has not registered an account, the user registration step is entered and then the next step is continued.

[0028] Step 2: Log in to your account and enter the main interface. Select a mode through the main interface. The business layer provides corresponding services based on the selected mode. The modes include basic training mode, face detection mode and advanced training mode. If you select the basic training mode, continue to step 3 or select the face detection mode to jump to step 4 or select the advanced training mode to jump to step 5.

[0029] Step 3: Enter basic training, then run the basic training game, the user trains according to the basic training game, and after the training is completed, the overall feedback returns to step 2;

[0030] Step 4: perform face detection, then return the face detection result, and then return to step 2;

[0031] Step 5: Enter advanced training, and then run the advanced training game. The user will train according to the advanced training game, and after the training is completed, the overall feedback will return to step 2. Through the settings of the above steps, it is possible to effectively provide user access, and the above access can be achieved using existing mobile devices and mobile phones. At the same time, facial detection and rehabilitation training can be achieved according to the user's selection, specifically:

[0032] In the facial detection process, the current face must first be identified to ensure the accuracy of subsequent detection. The system loads the model and feeds back the probability returned by the face recognition model faceapi to the patient, asking him to adjust the position of his head in the camera. If the recognition probability of the model is above 0.9, the adjustment is complete. Then, the "mask face" recognition model is used to obtain the predicted "mask face" result as the basis for scoring the game.

[0033] In the rehabilitation training game process, it is also necessary to first identify the current face and make its confidence greater than 0.9. After starting the game, the video and the corresponding opera music will be played. We use the timer and Map data structure to provide real-time feedback to the patient on the expression he needs to make at the moment, and input the patient's expression into the facial unit detection model, and score it according to the result returned by the model. The game scores the corresponding action based on the difference between the current expression and the natural expression in a certain facial unit. In the advanced training game, we will add filters to the patient's face through an algorithm, and the algorithm will calculate the coordinates that need to be added to the filter through the facial anchor coordinates. And the filter will be switched according to the duration of the game, that is, the opera.

[0034] As a specific implementation of the improvement, the basic training game in step 3 includes the following steps: step 31, providing a training part selection interface for the user to select, after the user completes the selection, the client displays the facial movements of the instructor corresponding to the training part, and at the same time plays the explanation voice in the form of pictures and texts;

[0035] Step 32: Play opera music, and the client performs facial action unit recognition on the face image captured by the camera to obtain the face anchor point, and overlays the opera filter image on the user's real face based on the face anchor point, and prompts the user to make corresponding expressions;

[0036] Step 33, determining whether the user's facial expression action is successful, if successful, playing a motivational graphic message, if unsuccessful, displaying a message prompting the user to adjust the facial expression;

[0037] Step 34, repeat step 31 to step 33 to complete the basic training. Through the above steps, the basic training includes training of different parts (eyebrow and nose training, lip training, swallowing pronunciation training, etc.) and combination training. First, the user will follow the instructor to practice facial movements. The teacher will demonstrate facial movements and provide graphic and voice explanations. Then, the opera music plays, and the opera filter will appear on the user's real face, and make corresponding expressions according to the prompts. If the action is successful, the system will give feedback, otherwise the user will be prompted to adjust the facial expression and try again. Then, the user can choose to practice another part and repeat the process of "exercise-doing actions-feedback".

[0038] As a specific implementation of the improvement, the advanced training game in step 5 includes the following:

[0039] Step 51, providing a training difficulty selection interface for the user to select, and after the user completes the selection, playing a medley of opera music for training different parts of the body, prompting the user to make facial expressions and movements according to the medley of opera music;

[0040] Step 52, determining whether the user's expression action is continuous and successful, and if so, playing an incentive graphic or voice message;

[0041] Among them, during the playback of the opera music medley, the opera filter pictures are changed as the opera music is changed. The advanced training of this embodiment is carried out in the form of facial opera exercises, and users can choose training content of different durations and difficulties according to their own needs. In order to facilitate users with cognitive difficulties, the system also provides a voice playback function. Training levels consisting of different types of opera medleys are designed in the game, and different operas correspond to different training contents. For example: the opera "Rolling Lantern" with a cheerful rhythm is suitable for training lip movements, and the "The Case of the Chopping of the Beauty" with a long tone is suitable for pronunciation training. In order to increase the fun of the game, we have also designed corresponding makeup filters for each opera, which presents an interactive effect of "face changing" during the switching of operas, bringing users a richer experience. When the user succeeds in performing the actions continuously, the user will be given graphic and voice incentives. For example, this embodiment mainly provides the following Figure 3 These filters.

[0042] Therefore, through this game, middle-aged and elderly people can exercise their facial muscles in a pleasant atmosphere, improve the flexibility of facial expressions, increase their enthusiasm for training, and achieve continuity of treatment. It also provides doctors with convenient data recording and analysis to assist in the diagnosis and treatment of patients.

[0043] As a specific implementation of the improvement, the specific method of facial action unit recognition in step three two is as follows: This embodiment uses the open source OpenGraphAU model in IJCAI-ECAI 2022 to perform facial action unit recognition. The method adopts a graph representation learning method of edge learning to achieve excellent results in multiple data sets. At the same time, it can also monitor and analyze data such as the average value of AU, the variance value of AU, the average training time, and the average training accuracy, which is convenient for doctors to diagnose and further treat the disease. This embodiment proposes to use a facial action unit relationship modeling method to learn a unique graph that can explicitly describe the relationship between each pair of facial action units. The method first encodes the activation state of each facial action unit and the connection between the unit and other units as a node feature. Then, the method learns a pair of multidimensional edge features through node features to represent multiple specific task relationships between each pair of facial action units. In short, each facial action unit (node) is connected by a multidimensional weight (edge) to strengthen the connection between facial action units. In addition to the methodological improvements, this embodiment also uses a mixture of multiple data sets and uses the mixed data sets to train the model in two stages to improve the model recognition accuracy.

[0044] In the facial action unit recognition process, there are three steps in general, namely face recognition, facial features positioning and facial action unit recognition. First, the video track is used to obtain the patient's real-time facial information, and the face recognition model and facial features positioning model provided in the faceapi library are used to process the face of the current image. The face recognition model will first identify the face in the incoming video stream, select the face with the highest credibility, and input the face into the facial features positioning model to obtain the anchor points of each part of the face. The first purpose of the anchor point is to locate the filter algorithm, and the second purpose is to intercept the face position in the picture to improve the accuracy of facial action unit recognition. After obtaining the anchor point, the intercepted face is input into the facial unit recognition model, so that it can analyze the user's facial expression in real time and accurately identify different facial action units. This provides users with accurate feedback and guidance, enhances their facial muscle coordination and complex movement ability, and improves the training and rehabilitation effects.

[0045] In summary, the training method of this embodiment, through the settings of steps one to five, can use the camera and microphone of a mobile device to achieve training for Parkinson's "mask face", without the need to set up special equipment, and is easier to use.

[0046] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A Parkinson's "mask face" rehabilitation training method, characterized by: The steps include: Step 1: The user enters the interface through a mobile device and proceeds to the next step based on whether the user has a registered account. If the user has a registered account, the user proceeds to the next step. If the user has not registered an account, the user registration step is entered and then the next step is continued. Step 2: Log in to your account and enter the main interface. Select a mode through the main interface. The business layer provides corresponding services based on the selected mode. The modes include basic training mode, face detection mode and advanced training mode. If you select the basic training mode, continue to step 3 or select the face detection mode to jump to step 4 or select the advanced training mode to jump to step 5. Step 3: Enter basic training, then run the basic training game, the user trains according to the basic training game, and after the training is completed, the overall feedback returns to step 2; Step 4: perform face detection, then return the face detection result, and then return to step 2; Step 5: Enter advanced training, then run the advanced training game. The user trains according to the advanced training game, and after the training is completed, the overall feedback returns to step 2.

2. The Parkinson's "mask face" rehabilitation training method according to claim 1, characterized in that: The basic training game in step 3 includes the following steps: Step 31: Provide a training part selection interface for the user to choose. After the user completes the selection, the client displays the facial movements of the instructor corresponding to the training part, and plays the explanation voice in the form of pictures and texts; Step 32: Play opera music, and the client performs facial action unit recognition on the face image captured by the camera to obtain the face anchor point, and overlays the opera filter image on the user's real face based on the face anchor point, and prompts the user to make corresponding expressions; Step 33, determining whether the user's facial expression action is successful, if successful, playing a motivational graphic message, if unsuccessful, displaying a message prompting the user to adjust the facial expression; Step 34: Repeat steps 31 to 33 to complete basic training.

3. The Parkinson's "mask face" rehabilitation training method according to claim 2, characterized in that: The advanced training game in step 5 includes the following: Step 51, providing a training difficulty selection interface for the user to select, and after the user completes the selection, playing a medley of opera music for training different parts of the body, prompting the user to make facial expressions and movements according to the medley of opera music; Step 52, determining whether the user's expression action is continuous and successful, and if so, playing an incentive graphic or voice message; Among them, during the playback of the opera music medley, the opera filter picture changes as the opera music changes.

4. The Parkinson's "mask face" rehabilitation training method according to claim 3, characterized in that: The specific method of facial action unit recognition in step 32 is as follows: first, the patient's real-time facial information is obtained using the video track, and the face recognition model and facial features positioning model provided in the faceapi library are used to process the face of the current image. The face recognition model will first identify the faces in the incoming video stream, and select the face with the highest credibility, and input the face into the facial features positioning model to obtain the anchor points of various parts of the face. After obtaining the anchor points, the captured face is input into the facial unit recognition model, so that it can analyze the user's facial expressions in real time and accurately identify different facial action units.

5. The Parkinson's "mask face" rehabilitation training method according to claim 4, characterized in that: The specific method of judging whether the user's expression action is successful in the step three is: inputting the patient's expression into the facial unit detection model, scoring according to the result returned by the model, the model scores the corresponding action according to the difference between the current expression and the natural expression in a certain facial unit, and judging whether the user's expression action is successful according to the score.

Citation Information

Patent Citations

  • Facial muscle rehabilitation training system on basis of mirror image visual feedback

    CN108211241A

  • Expression recognition training method and system for autistic patient

    CN110866434A

  • Parkinson'mask face 'interesting rehabilitation training method

    CN118142133A

  • Rhythm face apparatus

    KR101525821B1

  • Exercise assistance service providing robot and exercise assistance service providing method

    KR102669062B1