Training support device, training support method, and program

Abstract

To support patients in appropriately performing rehabilitation, such as swallowing function rehabilitation, on their own. [Solution] The training support device according to the embodiment comprises an acquisition unit, an analysis unit, a determination unit, a guidance information generation unit, and an output unit. The acquisition unit acquires user image data and user voice data captured by a user performing a training task in which bodily movements and vocalizations are defined. The analysis unit analyzes the state of the user's movements and vocalizations from the user image data and voice data. The determination unit determines the degree of achievement of the user regarding the bodily movements and vocalizations defined in the training task based on the user's movements and vocalizations analyzed by the analysis unit. The guidance information generation unit generates guidance information that guides the user's movements and vocalizations to the bodily movements and vocalizations defined in the training task based on the determination result by the determination unit. The output unit outputs the generated guidance information.

Description

Technical Field

[0001] The embodiments disclosed in the present specification and drawings relate to a training support apparatus, a training support method, and a program. Background Art

[0002] Conventionally, it has been known that a decline in swallowing function leads to deterioration of a patient's health condition. It is also known that for maintaining or improving swallowing function, rehabilitation that continues various multiple trainings related to breathing, coughing and pronunciation in addition to exercise movements of the neck, shoulders and tongue under the guidance of a specialist for a period of not less than a certain length is effective. However, it may be highly difficult for a patient to effectively perform appropriate training for rehabilitation alone. Furthermore, not only for swallowing function, but also for rehabilitation of other functions, it may be highly difficult for a patient to effectively perform appropriate training for rehabilitation alone. Prior Art Documents Patent Documents

[0003] Patent Document 1 International Publication No. 2019 / 225242 Summary of the Invention Problem to be Solved by the Invention

[0004] One of the problems to be solved by the embodiments disclosed in the present specification and drawings is to support a patient to appropriately perform rehabilitation for swallowing function and the like even when the patient is alone. However, the problems to be solved by the embodiments disclosed in the present specification and drawings are not limited to the above problem. Problems corresponding to the respective effects achieved by each configuration shown in the embodiments described later can be positioned as other problems. Means for Solving the Problem

[0005] The training support device according to the embodiment comprises an acquisition unit, an analysis unit, a determination unit, a guidance information generation unit, and an output unit. The acquisition unit acquires user image data and user voice data captured by a user performing a training task in which bodily movements and vocalizations are defined. The analysis unit analyzes the state of the user's movements and vocalizations from the user image data and voice data. The determination unit determines the degree of achievement of the user regarding the bodily movements and vocalizations defined in the training task based on the user's movements and vocalizations analyzed by the analysis unit. The guidance information generation unit generates guidance information that guides the user's movements and vocalizations to the bodily movements and vocalizations defined in the training task based on the determination result by the determination unit. The output unit outputs the generated guidance information. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example of a usage scenario for the training support device according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing the configuration of the training support system according to the first embodiment. [Figure 3] Figure 3 shows an example of the registered contents of the training method database according to the first embodiment. [Figure 4] Figure 4 shows an example of the registered contents of the patient information database according to the first embodiment. [Figure 5A] Figure 5A is a flowchart showing an example of the processing flow for swallowing function training according to the first embodiment. [Figure 5B] Figure 5B is a continuation of the flowchart in Figure 5A. [Figure 6] Figure 6 shows an example of the output of guidance information according to the first embodiment. [Figure 7] Figure 7 shows another example of the output of guidance information according to the first embodiment. [Figure 8] Figure 8 shows another example of the output of guidance information according to the first embodiment. [Figure 9] Figure 9 shows an example of a display of the team-based training status according to the second embodiment. [Figure 10] Figure 10 shows an example of a message between users according to the second embodiment. [Figure 11] Figure 11 shows an example of the display screen for individual training records according to the third embodiment. [Figure 12] Figure 12 shows another example of the display screen for individual training records according to the third embodiment. [Figure 13] Figure 13 shows another example of the display screen for individual training records according to the third embodiment. [Figure 14] Figure 14 is a block diagram showing the configuration of the training support system according to the fourth embodiment. [Figure 15] Figure 15 shows an example of the outline configuration of the training support system according to Modification Example 1. [Figure 16] Figure 16 shows an example of the outline configuration of the training support system according to Modification Example 2. [Modes for carrying out the invention]

[0007] The following describes in detail embodiments of the training support device, training support method, and program, with reference to the drawings.

[0008] (First embodiment) Figure 1 shows an example of a usage scenario for the training support device 1 according to the first embodiment. In the example shown in Figure 1, the training support device 1 is connected to the camera 2, microphone 3, and speaker 4 in a communicative manner. The configuration including the training support device 1, camera 2, microphone 3, and speaker 4 is referred to as the training support system S.

[0009] User U, who uses the training support device 1, is, for example, a patient undergoing rehabilitation to maintain or improve swallowing function. In this embodiment, swallowing function rehabilitation is also referred to as "training." In this embodiment, training includes physical movements and vocalizations for maintaining or improving swallowing function.

[0010] Generally, swallowing function tends to decline in elderly people. Deterioration of swallowing function can also cause aspiration pneumonia, which is one of the leading causes of death among the elderly. In addition, decreased swallowing function occurs as one of the symptoms even in patients with progressive diseases accompanied by movement disorders such as Parkinson's disease. Rehabilitation can maintain swallowing function or slow down the rate of decline in swallowing function. Further, rehabilitation may also be performed by patients whose swallowing function has declined due to the influence of surgery or treatment in order to recover the swallowing function.

[0011] As a feature of swallowing function training, it is important for enhancing the effect to implement training by appropriately combining physical movement tasks and phonation tasks in a well-balanced manner. In addition, swallowing function training includes training tasks implemented by interlocking physical movement and phonation.

[0012] The imaging device 2 is a camera that images a user U who performs training including body movement and phonation for maintaining or improving swallowing function. In the present embodiment, "phonation" is defined to include respiration. The imaging device 2 images at least the range from the head to the upper half of the body of the user U. Further, the imaging device 2 may be capable of imaging the entire body of the user U. The imaging device 2 transmits captured image data of the user U (hereinafter referred to as user image data) to the training support device 1. The user image data is, for example, moving image data, and is continuously captured during the implementation of training.

[0013] The microphone 3 acquires audio data of the user U. The microphone 3 transmits the acquired audio data of the user U to the training support device 1. The microphone 3 continuously acquires audio data during the implementation of training.

[0014] The speaker 4 is an audio output device that outputs audio for guiding the body movement and phonation of the user U under the control of the training support device 1.

[0015] The training support device 1 analyzes and evaluates the user U's movements and vocalizations based on user image data captured by the camera 2 and the voice data of user U acquired by the microphone 3. Based on the results of the analysis and evaluation, it guides user U with images or sounds so that they can perform appropriate movements and vocalizations in accordance with the training task.

[0016] The training support device 1 is installed, for example, in a nursing home, hospital, rehabilitation facility, or the user U's home. When the training support device 1 is installed in a facility with multiple users, such as a nursing home, hospital, or rehabilitation facility, the training support device 1 can be shared by multiple users.

[0017] Figure 2 is a block diagram showing the configuration of the training support system S according to the first embodiment. As shown in Figure 2, the training support device 1 includes, for example, a network interface 11, a device interface 12, a memory circuit 13, an input interface 14, a display 15, and a processing circuit 16.

[0018] The network interface 11 sends and receives information with other devices via a network such as the Internet.

[0019] The device interface 12 transmits and receives information with the imaging device 2, microphone 3, and speaker 4 via wired or wireless communication. Alternatively, the network interface 11 may be connected to the imaging device 2, microphone 3, and speaker 4.

[0020] The memory circuit 13 stores various programs and various data used in the processing of the training support device 1. The memory circuit 13 can be implemented using semiconductor memory elements such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or a hard disk or optical disc. The memory circuit 13 can also be used as a non-transient storage medium provided by hardware. The memory circuit 13 is an example of a storage unit.

[0021] Furthermore, the memory circuit 13 stores, for example, a training method database (DB) 13a, a training record database 13b, and a patient information database 13c.

[0022] The training method database 13a is, for example, a database that associates training tasks with sample operation data for each training task and the target person (user U) for each training task.

[0023] Figure 3 shows an example of the registered contents of the training method database 13a according to the first embodiment. In the example shown in Figure 3, the training tasks registered include swallowing, shoulder movements, neck movements, arm movements, facial muscle movements, and movements including posture. Furthermore, arm movements are movements that combine arm movements with breathing (more specifically, deep breathing). Furthermore, facial muscle movements include movements that combine tongue muscles with facial muscle movements and breathing (more specifically, pursed-lip breathing). Furthermore, movements including posture include movements that combine the abdomen, hands, mouth, and breathing, and include cough training, huffing, rehabilitation exercises, and vocal exercises.

[0024] Sample operation data is data that shows the specific actions required for a training task. Sample operation data may be image data or text data. Furthermore, the sample operation data may include predefined success criteria used by the judgment function 166 (described later) to determine whether user U has achieved each training task.

[0025] For example, a training task may define criteria for the exercise intensity and angle of movement of the user U's body as a prescribed physical movement. Exercise intensity is determined by the magnitude and speed of movement of the user U's body part (e.g., arm), and the angle of movement is determined by the difference in angle of the user U's body part (e.g., arm) before and after the movement.

[0026] Furthermore, the training tasks may define achievement criteria for at least one of the following as prescribed vocalizations: pronunciation, volume, pitch, speed, and vocalization duration.

[0027] For example, in the example shown in Figure 3, one example of an achievement criterion for movement, including posture, is "Is the mouth wide open at the same time as vocalization?". Other examples of achievement criteria include "Is the pronunciation correct?", "Is the volume above the specified level?", "Is the pitch specified?", "Is the vocalization speed specified?", "Is the vocalization duration specified?", and "Are the movement and breathing coordinated at the specified timing?". Such achievement criteria may be associated with each training task. Furthermore, achievement criteria are not limited to a simple "did" or "did not do," but may include tiered levels or other distinctions.

[0028] The content of the training tasks and the sample movement data registered in the training method database 13a are examples of the body movements and vocalizations defined in the training tasks in this embodiment.

[0029] Since the target user U differs depending on the training task, each training task has its target user U individually registered as a target user. For example, different training tasks are set according to the physical limitations of the user U.

[0030] User U's physical limitations refer to actions that are difficult to perform or body parts that are difficult to move due to User U's disability, illness, or injury. For example, physical limitations may include being unable to raise an arm due to paralysis caused by illness, or being unable to move the neck due to the after-effects of an injury or illness, or due to rest after surgery.

[0031] In the example shown in Figure 3, three users U are registered: Patient A, Patient B, and Patient C. Of these, Patient A is assigned a training task in which they cannot perform neck movements due to physical limitations. Patient B is assigned a training task in which they cannot perform a combination of facial muscle movements and breathing (more specifically, pursed-lip breathing). Patient C is assigned a training task in which they cannot perform shoulder movements.

[0032] The multiple training tasks set for User U are collectively referred to as User U's training menu. The training menu includes multiple training tasks that differ in the body parts used and the types of vocalizations produced. When performing training, User U can select and complete any of the training tasks from their own training menu.

[0033] Furthermore, the training method database 13a may register identifying information or a name that can identify each training task. Also, the content of the training tasks shown in Figure 3 is just an example and is not limited to these.

[0034] Furthermore, the training tasks and the target individuals for those training tasks may be associated with the patient information database 13c instead of the training method database 13a, or they may be associated with other independent databases.

[0035] The training record database 13b is a database in which the training history and evaluations of the training performed by user U are registered. When multiple users U share the training support device 1, the training record database 13b stores the training history and evaluations for each user U. The evaluation of the training is, for example, the result of the judgment function 166 described later, which determines whether or not user U achieved the actions and vocalizations defined in the training task.

[0036] The patient information database 13c is a database in which information about user U, who uses the training support device 1, is registered.

[0037] Figure 4 shows an example of the registered contents of the patient information database 13c according to the first embodiment. In the example shown in Figure 4, the patient information database 13c includes a patient ID that can identify user U, name, age, gender, functional evaluation score, and information about user U's physical limitations. The functional evaluation score is the result of a test that measures user U's swallowing function or other physical functions. In addition to or instead of the functional evaluation score, medical information such as user U's chief complaint and interview results may also be included in the patient information database 13c.

[0038] In Figure 4, the patient information database 13c sets whether each training task should be "something that user U should do on a daily basis," "something that user U should do when feeling well," or "something that should not be done," as information regarding user U's physical limitations.

[0039] In Figure 4, the names of each training task are abbreviated as "Neck 1," etc., but it is sufficient if they can be associated with the training tasks registered in the training method database 13a, and the identification information or name of each training task may also be registered.

[0040] Furthermore, the patient information database 13c may also store information about user U's physical limitations without associating it with training tasks. For example, the patient information database 13c may include information such as user U being unable to raise their arm due to paralysis caused by illness, or being unable to move their neck due to sequelae of injury or illness, as part of the information about user U's physical limitations.

[0041] In addition to the information shown in Figure 4, the patient information database 13c may also include a photograph of user U's face. Alternatively, the photograph of user U's face may be stored in the memory circuit 13 separately from the patient information database 13c.

[0042] Returning to Figure 2, the input interface 14 is implemented by, for example, a trackball, switch buttons, a mouse, a keyboard, etc. In this embodiment, the input interface 14 is not limited to those equipped with physical operating components such as a mouse or keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device separate from the training support device 1 and outputs these electrical signals to the processing circuit 16 is also included as an example of the input interface 14.

[0043] The input interface 14 is connected to the processing circuit 16 and converts various input operations received from the user U into electrical signals, which are then output to the processing circuit 16.

[0044] The display 15 displays various GUIs, etc., under the control of the processing circuit 16. The display 15 is an example of a display unit.

[0045] The processing circuit 16 is a processor that controls the entire training support device 1. The processing circuit 16 includes a reception function 161, an acquisition function 162, a specific function 163, an analysis function 164, a guidance information generation function 165, a judgment function 166, a display control function 167, and an audio output control function 168. The reception function 161 is an example of a reception unit. The acquisition function 162 is an example of an acquisition unit. The specific function 163 is an example of a specific unit. The analysis function 164 is an example of an analysis unit. The guidance information generation function 165 is an example of a guidance information generation unit. The judgment function 166 is an example of a judgment unit. The display control function 167 is an example of an output unit or display control unit. The audio output control function 168 is an example of an output unit or audio output control unit.

[0046] Here, for example, each processing function of the processing circuit 16, which is a component of the processing circuit 16, is stored in the memory circuit 13 in the form of a program that can be executed by a computer. In other words, the processing circuit 16, when each program has been read, will have the functions shown in the processing circuit 16 of Figure 2. In Figure 2, the processing functions performed by the reception function 161, acquisition function 162, identification function 163, analysis function 164, guidance information generation function 165, judgment function 166, display control function 167, and voice output control function 168 are explained as being realized by a single processor, but it is also possible to configure the processing circuit 16 by combining multiple independent processors, and each processor will realize the functions by executing a program. Furthermore, although Figure 2 describes a single memory circuit 13 that stores programs corresponding to each processing function, it is also possible to have multiple memory circuits distributed and have the processing circuit 16 read the corresponding programs from individual memory circuits.

[0047] The above description illustrates an example in which the "processor" reads and executes programs corresponding to each function from a memory circuit, but the embodiments are not limited to this. The term "processor" refers to circuits such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), Application Specific Integrated Circuit (ASIC), and Programmable Logic Device (e.g., Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), and Field Programmable Gate Array (FPGA)). If the processor is a CPU, for example, the processor realizes its functions by reading and executing programs stored in memory 41. On the other hand, if the processor is an ASIC, instead of storing programs in the memory circuit 13, the functions are directly incorporated as logic circuits within the processor's circuitry. In this embodiment, each processor is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor and realize its functions. Furthermore, the multiple components shown in Figure 2 may be integrated into a single processor to realize their functions.

[0048] The reception function 161 accepts various operations from user U via the input interface 14. For example, the reception function 161 accepts an operation from user U to select a training task.

[0049] The acquisition function 162 acquires user image data from the imaging device 2 via the device interface 12 or the network interface 11. The acquisition function 162 also acquires user U's voice data from the microphone 3 via the device interface 12 or the network interface 11.

[0050] Furthermore, the acquisition function 162 continuously acquires user image data and audio data during the training.

[0051] The identification function 163 identifies user U from user image data acquired by the acquisition function 162. More specifically, the identification function 163 identifies user U's identification information depicted in the acquired user image data based on the acquired user image data and the facial photographs of multiple users U registered in the memory circuit 13. The identification function 163 may identify user U by, for example, a known image recognition method.

[0052] The analysis function 164 analyzes the user U's actions and vocalization state from the user image data and audio data acquired by the acquisition function 162. The vocalization state includes, for example, the user U's breathing, pronunciation, and vocalization volume. The vocalization state may also further include pitch, speed, and vocalization length.

[0053] More specifically, the analysis function 164 detects the position, breathing, pronunciation, and volume of user U's body parts during training from user image data and audio data. The body parts to be detected include, for example, at least one of the head, neck, mouth, and shoulders. Based on the detected changes in the position, pronunciation, and volume of the body parts, the analysis function 164 identifies the user U's movements, pronunciation, and volume, as well as the state of coordination between the movements and the state of pronunciation. The analysis function 164 may also detect the user U's movements depicted in the user image data using known image recognition or other methods.

[0054] Furthermore, the analysis function 164 may analyze the physical limitations of user U from user image data and audio data. For example, the analysis function 164 may analyze user U's movements based on user image data and identify body parts that user U cannot move from the analyzed movements.

[0055] The guidance information generation function 165 generates guidance information that guides the movements and vocalizations of user U, which have been analyzed by the analysis function 164, to the body movements and vocalizations defined in the training task.

[0056] More specifically, the guidance information generation function 165 generates guidance information that guides user U to the body movements and sounds defined in the training task, according to the training task selected by user U. The body movements and sounds defined in the training task are, for example, the sample movement data registered in the training method database 13a, as described above. The guidance information generation function 165 also generates guidance information based on the degree of user U's achievement of the training task, which is determined by the judgment function 166 described later. The guidance information generation function 165 may also generate guidance information according to the training task before acquiring user U's movements and vocalizations.

[0057] Guidance information includes at least one of the following: audio, images superimposed on user image data, and text displays.

[0058] The guidance information generation function 165 may change the content of the guidance information in accordance with the actions and vocalizations of user U analyzed by the analysis function 164.

[0059] The guidance information generation function 165 may, for example, select and combine guidance information patterns that are suitable for the user U's actions and vocalizations from a plurality of guidance information patterns pre-stored in the memory circuit 13 or the like. For example, criteria that the user U has not achieved and the guidance information to be output to the user U may be pre-combined and stored in the memory circuit 13. Alternatively, guidance information may be output by a trained model that has learned the correspondence between the user U's actions and vocalizations and the guidance information using machine learning or the like. This trained model may be stored in the memory circuit 13 or the like, or it may be incorporated into the guidance information generation function 165.

[0060] For example, if the guidance information generation function 165 determines that user U's speech does not meet the achievement criteria, it generates guidance information to improve the user's speech using at least one of the following: audio, an image superimposed on user image data, and text display.

[0061] Furthermore, if the guidance information generation function 165 determines that the exercise intensity or exercise angle in user U's movements does not meet the achievement criteria, it generates guidance information to improve user U's exercise intensity or exercise angle using at least one of the following: audio, an image superimposed on user image data, and text display.

[0062] Furthermore, the guidance information generation function 165 generates guidance information that corresponds to the physical constraints of user U. For example, the guidance information generation function 165 generates guidance information that corresponds to the physical constraints of user U based on the physical constraints of user U analyzed by the analysis function 164, and at least one of the information indicating the physical constraints of user U that has been previously registered in the patient information database 13c or the training method database 13a.

[0063] The judgment function 166 determines the degree of achievement of user U for the actions and vocalizations defined in the training tasks, based on the actions and vocalizations of user U analyzed by the analysis function 164. As described above, since user U performs multiple training tasks set for themselves, the judgment function 166 determines the degree of achievement of each user U for each of the multiple user Us for each training task.

[0064] For example, the judgment function 166 determines whether the movements, pronunciation, and volume of User U analyzed by the analysis unit, as well as the state of coordination between the movements and vocalization, have achieved the physical movements and vocalization defined in the training task.

[0065] Furthermore, the judgment function 166 determines whether user U has achieved the achievement criteria based on the user U's vocalizations analyzed by the analysis function 164 and the achievement criteria registered in the training method database 13a.

[0066] Furthermore, the judgment function 166 determines whether user U has achieved the performance criteria for exercise intensity and exercise angle defined in the training task, based on the movements of user U analyzed by the analysis function 164.

[0067] The judgment function 166 stores the judgment results regarding user U's actions and vocalizations in the training record database 13b.

[0068] The display control function 167 controls the display 15 to display various images. More specifically, the display control function 167 displays the images included in the guidance information generated by the guidance information generation function 165 on the display 15.

[0069] Furthermore, the display control function 167 displays a training menu selection screen on the display 15, which allows user U to select a training task from multiple options. The training tasks displayed on the training menu selection screen are the training tasks set as targets for user U in the training method database 13a, and these differ depending on user U. The training menu selection screen is an example of a selection screen in this embodiment.

[0070] The audio output control function 168 controls the speaker 4 to output various sounds. More specifically, the audio output control function 168 outputs the sounds included in the guidance information generated by the guidance information generation function 165 to the speaker 4.

[0071] Next, we will explain the processing flow of swallowing function training performed by the training support device 1 configured as described above.

[0072] Figures 5A and 5B are flowcharts showing an example of the processing flow for swallowing function training according to the first embodiment.

[0073] First, the acquisition function 162 acquires user image data from the imaging device 2 via the device interface 12 or the network interface 11 (S1).

[0074] Then, the identification function 163 identifies user U from the user image data acquired by the acquisition function 162 (S2).

[0075] The display control function 167 displays a training menu selection screen corresponding to the identified user U (S3). The training menu selection screen displays selection buttons that allow the user to select a training task that has been set as the target of training for user U identified in S2 in the training method database 13a.

[0076] Here, using Figure 6, we will explain the images and sounds output during the swallowing function training process.

[0077] Figure 6 shows an example of the output of guidance information according to the first embodiment. The progression of the display 15 shown in Figure 6 follows the processing flow of swallowing function training shown in Figures 5A and 5B.

[0078] In the example shown in Figure 6, the display control function 167 first displays a first screen area 151 on the display 15 that displays user image data 90 acquired by the acquisition function 162, and a second screen area 152 that displays text, etc. In the leftmost screen of Figure 6, the display control function 167 displays the training menu selection screen for S3 in the second screen area 152. On the training menu selection screen, multiple training tasks set for user U in the training method database 13a are displayed for selection.

[0079] When user U selects a training task from the training menu selection screen, the reception function 161 accepts the operation (S4).

[0080] The guidance information generation function 165 then generates guidance information for actions corresponding to the selected training task (S5). For example, the guidance information generation function 165 generates audio, images superimposed on user image data, and text displays to guide the user U's actions and vocalizations, based on the sample action data of the training task registered in the training method database 13a. Note that the guidance information output immediately after a training task is selected may be the same regardless of the state of the user U's actions and vocalizations, and therefore may not be generated each time training is performed, but may be pre-associated with the training task and saved.

[0081] The display control function 167 and the audio output control function 168 then output guidance information from the display 15 and the microphone 3 (S6). In the second screen from the left in Figure 6, the guidance information includes a pointing image superimposed on the user image data 90 in the first screen area 151, and text displays such as "Feel the movement of your throat" and "Feel the movement of your stomach." The guidance information may also include text displays explaining the content of the vocalizations in the training task, such as "Adjust your posture to the guide and vocalize the following in time with the sound and rhythm: Yes, sir, yes, 'Ah, uh, ee, oh, oo'," as shown in the second screen area 152 of the second screen from the left in Figure 6. The guidance information may also include audio with the same content as the text display shown in the second screen area 152. The guidance information may also include the sound of a metronome or music indicating the rhythm of the actions or vocalizations in the training.

[0082] Then, the acquisition function 162 acquires new user image data 90 and audio data from the shooting device 2 and microphone 3 during the training (S7).

[0083] The analysis function 164 detects the movements and vocalizations of user U by analyzing the new user image data 90 and audio data acquired by the acquisition function 162 (S8). For example, the analysis function 164 detects the movements of user U's body parts such as chest, abdomen, mouth, arms, and shoulders from the user image data 90, and also detects the volume of user U's vocalizations, breathing sounds, etc. from the audio data.

[0084] The judgment function 166 determines the degree to which user U has achieved the actions and vocalizations defined in the training task, based on the actions and vocalizations of user U analyzed by the analysis function 164. For example, the judgment function 166 determines whether the actions and vocalizations of user U analyzed by the analysis function 164 meet the criteria defined in the training task (S9). This determination may be performed in real time during the training task, without waiting for the completion of the training task. Alternatively, the judgment function 166 may determine whether user U has performed the actions and vocalizations instructed in the guidance information, based on the instructions for actions or vocalizations included in the guidance information and the analyzed actions and vocalizations of user U, rather than the actions and vocalizations defined in the training task.

[0085] Then, if the guidance information generation function 165 determines that user U's actions or vocalizations do not meet the criteria defined for the training task (S9 "No"), it generates guidance information for improving user U's actions or vocalizations (S10).

[0086] For example, if a training task being conducted has a success criterion such as "Does the mouth open wide at the same time as vocalization?", and the judgment function 166 determines that user U has not met the success criterion, the guidance information generation function 165 generates guidance information such as "Open your mouth wider vertically" as text information or an audio message as guidance information.

[0087] Furthermore, if a training task being conducted has a success criterion of "is the pronunciation correct?", and the judgment function 166 determines that user U has not met the success criterion, the guidance information generation function 165 may generate textual information or audio messages as guidance information, providing advice on how to move the lips and tongue.

[0088] Furthermore, if a training task being conducted has a success criterion of "whether the volume of the voice is above a specified level," and the judgment function 166 determines that user U has not met the success criterion, the guidance information generation function 165 may generate text information or an audio message as guidance information that instructs the user to speak at "a volume of about n decibels (specified level)" or "a louder voice."

[0089] Furthermore, if a training task being conducted has a success criterion of "whether the sound is at the specified pitch," and the judgment function 166 determines that user U has not met the success criterion, the guidance information generation function 165 may generate text information or an audio message as guidance information instructing the user to speak in a "lower voice" or a "higher voice."

[0090] Furthermore, if a training task being conducted has a success criterion such as "is the speaking speed at the specified speed?", and the judgment function 166 determines that user U has not met the success criterion, the guidance information generation function 165 may generate guidance information such as music, text information, or an audio message to guide the user to speak in time with the rhythm or music.

[0091] Furthermore, if a training task being conducted has a success criterion such as "Is the length of the vocalization a specified length (e.g., m seconds or more)?", and the judgment function 166 determines that user U has not met the success criterion, the guidance information generation function 165 may generate text information or an audio message as guidance information that instructs the user to "repeat a sustained vocalization for m seconds."

[0092] Furthermore, if a training task being performed has a success criterion such as "Are the movements and breathing synchronized at the prescribed timing?", and the judgment function 166 determines that user U has not met the success criterion, the guidance information generation function 165 may generate guidance information such as textual information or audio messages of advice, such as "The timing of your inhalation and exhalation are reversed," or "Let's exhale a little more rhythmically."

[0093] The criteria that user U has not met in the above example and the guidance information output to user U may be pre-combined and stored in the memory circuit 13.

[0094] The display control function 167 and the voice output control function 168 then output guidance information from the display 15 and the microphone 3 (S11). For example, in the third screen from the left shown in Figure 6, the second screen area 152 displays guidance information, including text information indicating the sounds defined in the training task ("Ah, Eh, Ee, Oh, Uh") and a graph showing the user U's utterances analyzed from the voice data. This guidance information is updated in real time along with the user U's utterances.

[0095] Then, the acquisition function 162 acquires user image data 90 and voice data after guidance information for improving user U's actions or speech has been output (S12).

[0096] The analysis function 164 then analyzes the acquired user image data 90 and audio data (S13).

[0097] The judgment function 166 determines whether user U needs to repeat the training currently being conducted, based on the actions and vocalizations of user U analyzed by the analysis function 164 (S14). For example, if user U's level of achievement for the actions and vocalizations defined in the training task is lower than the prescribed repetition requirement criterion, the judgment function 166 determines that user U needs to repeat the training.

[0098] If the judgment function 166 determines that repetition is necessary (S14 "Yes"), the guidance information generation function 165 generates guidance information to encourage repetition (S15). The judgment function 166 may also identify or evaluate the difference between the physical movements and vocalizations defined in the training task and the movements and vocalizations of user U analyzed by the analysis function 164. The guidance information to encourage repetition may include feedback on user U's movements or vocalizations, such as "It was a sound of ● decibels," as shown in the fourth screen from the left in Figure 6. The guidance information to encourage repetition may also include textual information or audio that explains areas for improvement during repetition, such as "For the 'a' sound, next time, open your mouth a little wider vertically, hold the sound a little longer, and try to produce a sound of ● decibels."

[0099] Furthermore, if the guidance information generation function 165 determines in the S9 judgment process that user U's actions or vocalizations meet the criteria defined for the training task (S9 "Yes"), the process proceeds to the S14 determination of whether repetition is necessary. The judgment criteria in S9 and the judgment criteria in S10 may be different. For example, the judgment criteria in the S10 determination of whether repetition is necessary may be stricter than the judgment criteria in S9 for whether or not to provide guidance information for improving actions or vocalizations. Also, the judgment criteria in the S10 determination of whether repetition is necessary may be based on user U's training history, such as "having performed the training task n or more times."

[0100] After generating guidance information to encourage repetition in S15, the process returns to S6, and processes S6 to S15 are repeatedly executed until it is determined that repetition is no longer necessary. For example, the guidance information generation function 165 may generate new guidance information based on the judgment result of the judgment function 166 each time a judgment result is obtained, in order to improve the quality of training for user U.

[0101] Furthermore, if the judgment function 166 determines that repetition is unnecessary (S14 "No"), the processing of this flowchart ends.

[0102] Note that the content of the guidance information is not limited to the example shown in Figure 6. For example, Figures 7 and 8 show other examples of the output of guidance information according to the first embodiment.

[0103] In Figures 7 and 8, user U is performing a training task that combines body movements and deep breathing. In such cases, the guidance information may include arrow images on user image data 90 that indicate the "exhalation" and "inhalation" phases. The guidance information may also include pointing images that indicate raising and lowering the arms.

[0104] As described above, the training support device 1 of this embodiment analyzes the state of User U's movements and vocalizations from User image data 90 and User's voice data, which are captured by User U performing a training task in which body movements and vocalizations are defined. Based on the analyzed movements and vocalizations of User U, the device determines User U's degree of achievement with respect to the body movements and vocalizations defined in the training task. Furthermore, based on the determination result of User U's degree of achievement, the training support device 1 of this embodiment generates and outputs guidance information that guides User U's movements and vocalizations toward the body movements and vocalizations defined in the training task. For this reason, the training support device 1 of this embodiment can support patients in appropriately performing physical function rehabilitation even when performing it independently.

[0105] Furthermore, the training tasks used in the training support device 1 of this embodiment correspond to maintaining or improving swallowing function. Therefore, the training support device 1 of this embodiment can support patients in appropriately performing swallowing function rehabilitation on their own.

[0106] Generally, training to maintain or improve swallowing function is considered effective when it combines physical movement tasks and vocalization tasks in a balanced manner. Therefore, training tasks in swallowing rehabilitation include those that coordinate physical movement and vocalization. Furthermore, it is known that high frequency and continuous implementation, such as "three or more days a week for three months or more," is desirable for effective swallowing training. For this reason, it was sometimes difficult for individuals to effectively practice and maintain movement and vocalization through self-help alone. Specifically, it was sometimes difficult for individuals to plan and implement complex training menus on their own, or to receive constant / continuous guidance from a third party (professional). Also, it is known that training to maintain or improve swallowing function is effective when conducted in groups of patients to maintain motivation and receive feedback on their own movements and vocalization. However, it was sometimes difficult to implement such training due to patients living alone or geographical or time constraints.

[0107] Furthermore, relying solely on one-way information, such as viewing videos demonstrating training exercises, made it difficult to obtain feedback on the movements and vocalizations performed by the patients, sometimes hindering improvements in the quality of training.

[0108] In contrast, the training support device 1 of this embodiment can improve the quality of training by improving the movements and vocalizations of user U and improving the quality of training by generating and outputting guidance information that guides user U's movements and vocalizations to the physical movements and vocalizations defined in the training task, based on user image data 90 and user voice data.

[0109] Furthermore, the training support device 1 of this embodiment acquires and analyzes new user image data 90 and user U's voice data after outputting guidance information. Based on the newly analyzed movements and vocalizations of user U, it determines the user's level of achievement regarding the movements and vocalizations defined in the training task and generates guidance information according to the determination result. Therefore, according to the training support device 1 of this embodiment, if user U's movements and vocalizations improve with the guidance information, further guidance information can be provided to the improved movements and vocalizations of user U. Moreover, such feedback is useful not only for swallowing function but also for rehabilitation of any part of the patient's body.

[0110] Furthermore, the training support device 1 of this embodiment detects the position, breathing, pronunciation, and volume of at least one of the body parts of the user U undergoing training, including the head, neck, mouth, and shoulders, from the user image data 90 and the user's voice data. Based on the detected changes in the position of the body parts, pronunciation, and volume, it identifies the user U's movements, pronunciation, and volume, as well as the state of coordination between the movements and vocalizations, and determines whether the identified results achieve the body movements and vocalizations defined in the training task. Therefore, the training support device 1 of this embodiment can perform a comprehensive evaluation including movements and vocalizations.

[0111] Furthermore, the training support device 1 of this embodiment displays a training menu selection screen on the display 15, allowing the user U to select a training task from multiple options. In accordance with the training task selected by the user U, it generates guidance information to guide the user U to perform the physical actions and sounds defined in the training task. The guidance information includes at least one of the following: sound, an image superimposed on the user image data 90, and text display. Therefore, with the training support device 1 of this embodiment, the user U can easily grasp areas for improvement in their own actions or sounds through sight or hearing.

[0112] Furthermore, the training tasks of this embodiment define achievement criteria for at least one of the following as prescribed vocalizations: pronunciation, volume, pitch, speed, and length of vocalization. The training support device 1 of this embodiment determines whether user U has achieved the achievement criteria based on user U's vocalization and the achievement criteria. If it determines that user U has not achieved the achievement criteria, it generates guidance information to improve user U's vocalization using at least one of the following: audio, an image superimposed on user image data 90, and text display. Therefore, the training support device 1 of this embodiment can analyze information regarding user U's vocalization in detail and provide feedback on areas for improvement.

[0113] Furthermore, the training tasks of this embodiment define achievement criteria for the exercise intensity and movement angle of the user U's body as prescribed physical movements. The training support device 1 of this embodiment determines whether the user U has achieved the achievement criteria based on the exercise intensity and movement angle of the user U's body and the achievement criteria. If it determines that the user U has not achieved the achievement criteria, it generates guidance information to improve the user U's exercise intensity or movement angle using at least one of voice, an image superimposed on the user image data 90, and text display. Therefore, the training support device 1 of this embodiment can analyze the user U's exercise intensity and movement angle in detail and provide feedback on areas for improvement.

[0114] Furthermore, in this embodiment, a training menu is set for each user U, including multiple training tasks that differ in the body parts used and the types of vocalizations produced, according to each user U's physical limitations. The training support device 1 of this embodiment also determines the degree of achievement of each user U for each training task for each of the multiple users U. For example, if the training menu is set without considering physical limitations, the user U may be given training tasks that they cannot perform, which could lead to a decrease in the user U's motivation or cause the user U to perform unnatural movements. Also, if training tasks that cannot be performed due to physical limitations are included in the evaluation of the degree of achievement, the user U may not be able to improve their level of achievement through their own efforts, which could lead to a decrease in the user U's motivation. In contrast, the training support device 1 of this embodiment determines the degree of achievement for each training task that has been set in advance taking into account each user U's physical limitations, so it can determine the degree of achievement of each user U's training tasks while taking each user U's physical limitations into consideration.

[0115] (Second embodiment) In the first embodiment described above, each user U individually performed swallowing function training using the training support device 1. In this second embodiment, the training support device 1 further includes a function that allows teams composed of multiple users U to compete in training evaluation.

[0116] The training support system S of this embodiment includes a camera 2, a microphone 3, and a speaker 4, similar to the first embodiment. The training support device 1 includes a network interface 11, a device interface 12, a memory circuit 13, an input interface 14, a display 15, and a processing circuit 16, similar to the first embodiment.

[0117] The processing circuit 16 of the training support device 1 in this embodiment includes a reception function 161, an acquisition function 162, a specific function 163, an analysis function 164, a guidance information generation function 165, a judgment function 166, a display control function 167, and an audio output control function 168, similar to the first embodiment. The reception function 161, acquisition function 162, specific function 163, and analysis function 164 have the same functions as in the first embodiment.

[0118] In this embodiment, the memory circuit 13 stores in the training record database 13b the training history performed by multiple users U, and the determination results of the determination function 166, indicating whether or not the user U achieved the actions and vocalizations defined in the training task, for each of the multiple users U. The memory circuit 13 also stores various data similar to those in the first embodiment.

[0119] In addition to the same functions as in the first embodiment, the judgment function 166 of this embodiment calculates the evaluation of groups of multiple users U divided into predetermined numbers, based on the training history and judgment results performed by the users U belonging to each group. Groups of multiple users U divided into predetermined numbers are also called teams.

[0120] In addition to the same functions as in the first embodiment, the display control function 167 of this embodiment outputs evaluations for each group.

[0121] Figure 9 shows an example of a display of the team-based training status according to the second embodiment. In the example shown in Figure 9, four users U using the training support device 1 are divided into two groups of two: "Team A" and "Team B". This grouping may be set automatically by, for example, a specific function 163, or it may be set by a user U or an administrator.

[0122] The four users U participating in the team competition shown in Figure 9 may be, for example, users U who share the same training support device 1 at a nursing care facility or similar location.

[0123] The evaluation function 166 determines the points for each user U based on their performance on the training tasks, and calculates the "overall evaluation" by summing the points of all users U belonging to the same team.

[0124] The key points regarding the implementation status of training tasks are determined by the quantity, quality, and variety of training tasks performed by each user U. The quantity of training tasks refers to, for example, the number and frequency of training tasks performed. The quality of training tasks refers to, for example, the degree to which user U achieved the desired actions and vocalizations for the training tasks performed. Furthermore, a greater variety of training tasks is considered to be achieved when there is a large number of different types of training tasks performed, and when the targets of the training tasks differ, such as different body parts or types of vocalizations.

[0125] Furthermore, since each user U has different physical limitations, the training tasks they can perform will also differ. For this reason, the judgment function 166 may adjust the points of user U according to the types and number of training tasks that user U can perform.

[0126] In addition to the same functions as in the first embodiment, the guidance information generation function 165 of this embodiment generates guidance information to improve the achievement status of each team's training tasks. In the example shown in Figure 9, the guidance information generation function 165 generates guidance information that guides teams to improve their scores, such as, "Team A has been able to train more frequently than Team B. If both you and Shin-chan perform facial muscle exercises, you will get 30 bonus points."

[0127] Furthermore, the display control function 167 of this embodiment may display messages sent and received between multiple users U on the display 15.

[0128] Figure 10 shows an example of a message between users U according to the second embodiment. As shown in Figure 10, the training support device 1 of this embodiment has a message (message) function from user U to other users, and can send messages and training tasks. The message is displayed on the display 15 by the display control function 167 when the identification function 163 identifies that the user U to whom the message was sent is sitting in front of the training support device 1.

[0129] Furthermore, the judgment function 166 may award bonus points for training tasks sent by other users.

[0130] Thus, the training support device 1 of this embodiment has a function that allows multiple users U to compete in teams to see who is best at completing the training, and therefore contributes to improving the motivation of the users U to continue training.

[0131] (Third embodiment) In this third embodiment, the training support device 1 further displays the individual training record of user U.

[0132] The training support system S of this embodiment includes a camera 2, a microphone 3, and a speaker 4, similar to the first embodiment. The training support device 1 includes a network interface 11, a device interface 12, a memory circuit 13, an input interface 14, a display 15, and a processing circuit 16, similar to the first embodiment.

[0133] The processing circuit 16 of the training support device 1 in this embodiment includes a reception function 161, an acquisition function 162, a specific function 163, an analysis function 164, a guidance information generation function 165, a judgment function 166, a display control function 167, and an audio output control function 168, similar to the first embodiment. The acquisition function 162, specific function 163, analysis function 164, judgment function 166, and audio output control function 168 have the same functions as in the first embodiment.

[0134] In addition to the same functions as in the first embodiment, the display control function 167 of this embodiment displays individual training records for each user U on the display 15 based on the training history and judgment results of user U stored in the training record database 13b.

[0135] Furthermore, in addition to the same functions as in the first embodiment, the determination function 166 of this embodiment may also evaluate the training status based on, for example, the frequency of implementation of training tasks over a specified period in the past, the cumulative number of implementations, the quality of training tasks, the variations in training tasks, etc. The determination function 166 may also determine the next training task to recommend to user U based on user U's training history and the determination result. For example, the memory circuit 13 may store combinations of training tasks already implemented and the next recommended training task in association. In this case, the determination function 166 may refer to the memory circuit 13 according to the training tasks already implemented by user U to identify the next recommended training task.

[0136] Figure 11 shows an example of a display screen for individual training records according to the third embodiment. The display control function 167 displays on the display 15, for example, the date and time when user U performed training during a specified period in the past (for example, the most recent week), and an evaluation of the training status during that specified period. The evaluation of the training status may be expressed in multiple stages, such as "going well," "fairly good," or "slacking off too much." The evaluation of the training status may be determined by the judgment function 166 based on, for example, the frequency of performing training tasks during the specified period in the past, the cumulative number of times, the quality of the training tasks, the variations of the training tasks, etc.

[0137] Figure 12 shows another example of the display screen for individual training records according to the third embodiment. In the example shown in Figure 12, the implementation history of each training content (training task) and the evaluation of the training status for a specified period in the past (for example, the most recent week) are displayed in more detail than in Figure 11. As explained using Figure 4 in the first embodiment, since the training tasks that can be performed differ depending on the physical limitations of user U, the display control function 167 also displays whether user U can perform each training task.

[0138] Furthermore, in the example shown in Figure 12, the display control function 167 displays on the individual training record whether each training task is included in the recommended menu for the next specified period.

[0139] Figure 13 also shows yet another example of the display screen for individual training records according to the third embodiment. As shown in Figure 13, the display control function 167 may visually display the quantity, quality, and variation of training performed by user U.

[0140] Furthermore, the display control function 167 of this embodiment may output an evaluation screen for medical professionals to review, separate from the training record display screen for user U, based on the training history and evaluation results of user U stored in the training record database 13b. The evaluation screen for medical professionals to review may display, for example, the implementation history of each training content (training task) and the evaluation of the training status over a specified period in the past (for example, the most recent week), similar to the individual training record display screen shown in Figure 12. The evaluation screen for medical professionals to review may display evaluation details in more detail than the training record display screen for user U. In addition, the evaluation screen for medical professionals to review may display the training records of multiple users U in a list.

[0141] Healthcare professionals may use the evaluation screen to understand each user's training progress and revise their rehabilitation plan accordingly.

[0142] Furthermore, the reception function 161 of this embodiment, in addition to the same functions as in the first embodiment, accepts changes by medical professionals to the training menu, which includes multiple training tasks assigned to the user. Changes to the training menu include adding and deleting training tasks included in the training menu set for user U. For example, medical professionals may change the training menu if user U's physical limitations change or if user U's swallowing function changes.

[0143] Furthermore, the training support device 1 may communicate with a terminal used by a caregiver via the network interface 21, enabling the caregiver to output analysis, guidance, and judgment results. In this case, the display control function 167 may display the training menu selection screen and guidance information on the display of the caregiver's terminal.

[0144] (Fourth embodiment) In the first embodiment described above, the training menu was pre-set according to the physical limitations of user U. In this fourth embodiment, the training support device 1 generates a training menu for user U according to the physical limitations of user U.

[0145] Figure 14 is a block diagram showing the configuration of the training support system S according to the fourth embodiment.

[0146] The training support system S of this embodiment includes a camera 2, a microphone 3, and a speaker 4, similar to the first embodiment. The training support device 1 includes a network interface 11, a device interface 12, a memory circuit 13, an input interface 14, a display 15, and a processing circuit 16, similar to the first embodiment.

[0147] The processing circuit 16 of the training support device 1 of this embodiment includes a training menu generation function 169, in addition to the same reception function 161, acquisition function 162, identification function 163, analysis function 164, guidance information generation function 165, judgment function 166, display control function 167, and voice output control function 168 as in the first embodiment. The training menu generation function 169 is an example of a training menu generation unit. The reception function 161, acquisition function 162, identification function 163, analysis function 164, guidance information generation function 165, judgment function 166, display control function 167, and voice output control function 168 have the same functions as in the first embodiment.

[0148] The training menu generation function 169 generates a training menu that includes multiple training tasks to be assigned to each of the multiple users U, according to the physical constraints of each of the multiple users U. The physical constraints of each user U may be registered in the patient information database 13c, as in the first embodiment. Alternatively, the physical constraints of user U may be analyzed from the analysis function 164, user image data, and voice data.

[0149] The training menu generation function 169 generates a training menu for each user U by excluding training tasks that each user U cannot perform based on their individual physical limitations. The structure of the generated training menu is similar to, for example, the information registered in the training method database 13a of the first embodiment described in Figure 3.

[0150] According to the training support device 1 of this embodiment, a training menu tailored to the physical limitations of each user U can be automatically generated. Therefore, while retaining the effects of the first embodiment, the burden on medical professionals managing rehabilitation can be reduced.

[0151] Furthermore, the training support device 1 of this embodiment sets a training menu for each user U that includes multiple training tasks, each with different body parts to be moved and different types of vocalizations, according to the physical limitations of each user U. The training support device 1 of this embodiment also determines the degree of achievement of each user U for each training task for each of the multiple users U. Therefore, the training support device 1 of this embodiment can determine the degree of achievement of each user U's training tasks while taking into account the physical limitations of each user U.

[0152] (Variation 1) In the first to fourth embodiments described above, the case in which multiple users U take turns operating one training support device 1 was mainly explained, but each user U may also use the training support device 1 from their own user terminal.

[0153] Figure 15 shows an example of the outline configuration of the training support system S according to Modification 1. As shown in Figure 15, multiple user terminals 5a, 5b may be connected to the training support device 1 via a network N such as the Internet. User terminals 5a, 5b may be, for example, smartphones, PCs (Personal Computers), tablet devices, etc.

[0154] In this case, the displays of user terminals 5a and 5b serve as an example of a display unit instead of the display 15 of the training support device 1. Furthermore, the camera and microphone provided by user terminals 5a and 5b may be used instead of the camera 2 and microphone 3 in the first embodiment.

[0155] The camera may be dedicated to the training support system S, or it may be a monitoring camera for elderly people living alone, or an in-camera (sub-camera) installed on the display side of a smartphone or similar device.

[0156] (Modification 2) Furthermore, the training support system S may be configured such that multiple training support devices 1 are connected to each other in a way that allows them to communicate with one another.

[0157] Figure 16 shows an example of the outline configuration of the training support system S according to Modification 2. For example, in the team competition described in the second embodiment, multiple training support devices 1a to 1c may communicate via the network interface 11 and a network such as the Internet, and a user U using each training support device 1a to 1c may participate in the team competition. In this case, the training support devices 1a to 1c may be installed in the user U's home.

[0158] (Variation 3) Furthermore, in the first embodiment described above, Figures 1 and 2 show the camera 2, microphone 3, and speaker 4 as being provided separately from the training support device 1, but the camera 2, microphone 3, and speaker 4 may be included in the training support device 1.

[0159] Furthermore, although Figures 1 and 2 describe the display 15 as being included in the training support device 1, the display 15 may be provided separately from the training support device 1. For example, the display 15 may be the display of an information terminal such as a smartphone of user U, which is connected to the training support device 1 via a network or the like.

[0160] The various types of data discussed in this specification are typically digital data.

[0161] According to at least one embodiment described above, it is possible to support patients in appropriately performing rehabilitation, such as swallowing function rehabilitation, even on their own.

[0162] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments are possible without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0163] 1,1a,1b,1c Training support equipment 2. Imaging device 3 Microphone 4 speakers 5a, 5b User terminals 11 Network Interfaces 12 Device Interfaces 13 Memory circuit 13a Training Method Database 13b Training Record Database 13c Patient Information Database 14 Input Interfaces 15 displays 16 Processing Circuit 41 memory 90 User Image Data 151 First screen area 152 Second screen area 161 Reception function 162 Acquisition function 163 Specific Functions 164 Analysis Functions 165 Guidance information generation function 166 Judgment Function 167 Display Control Function 168. Audio output control function 169 Training Menu Generation Function N Network S Training Support System U User

Claims

1. An acquisition unit acquires user image data and voice data of a user who performs a training task in which body movements and vocalizations are defined. An analysis unit that analyzes the user's actions and vocalizations from the user image data and the voice data, A determination unit determines the degree of achievement of the user regarding the physical movements and vocalizations defined in the training task, based on the user's movements and vocalizations analyzed by the analysis unit. A guidance information generation unit generates guidance information that guides the user's actions and vocalizations to the physical actions and vocalizations defined in the training task, based on the judgment result of the judgment unit. An output unit that outputs the generated guidance information, A training support device equipped with the following features.

2. The aforementioned training tasks address the maintenance or improvement of swallowing function. The training support device according to claim 1.

3. The acquisition unit acquires new user image data and user voice data after outputting the guidance information. The analysis unit analyzes the user's actions and vocalizations from the new user image data and the user's voice data. The determination unit determines the degree of the user's achievement with respect to the actions and vocalizations defined in the training task, based on the user's actions and vocalizations newly analyzed by the analysis unit. The guidance information generation unit generates new guidance information based on the determination result by the determination unit. The training support device according to claim 1.

4. The aforementioned vocalization state includes the user's breathing, pronunciation, and vocalization volume. The analysis unit detects the position, breathing, pronunciation, and vocal volume of at least one of the user's body parts, including the head, neck, mouth, and shoulders, from the user image data and the audio data while the user is performing the training task, and identifies the user's actions, pronunciation, and vocal volume, as well as the state of coordination between actions and vocalization, based on the detected changes in the position of the body parts, pronunciation, and vocal volume. The determination unit determines whether the user's movements, pronunciation, and vocal volume, as well as the state of coordination between the movements and vocalization, as analyzed by the analysis unit, have achieved the physical movements and vocalization defined in the training task. The training support device according to claim 1.

5. The output unit displays a selection screen on the display unit that allows the user to select the training task from among multiple options. The guidance information generation unit generates guidance information that guides the user to the physical actions and sounds defined in the training task, according to the training task selected by the user, using at least one of the following: audio, an image superimposed on the user image data, and text display. The training support device according to claim 1.

6. The aforementioned training task defines achievement criteria for at least one of the following aspects of vocalization: pronunciation, volume, pitch, speed, and duration. The determination unit determines whether the user has achieved the achievement criteria based on the user's speech analyzed by the analysis unit and the achievement criteria. If the determination unit determines that the user has not met the achievement criteria, the guidance information generation unit generates guidance information to improve the user's speech using at least one of the following: audio, an image superimposed on the user image data, and text display. The training support device according to claim 1.

7. The aforementioned training task defines achievement criteria for the exercise intensity and angle of the user's body movements. The determination unit determines, based on the user's movements analyzed by the analysis unit, whether the user has achieved the achievement criteria for the exercise intensity and exercise angle defined in the training task. If the guidance information generation unit determines that the user has not met the achievement criteria, it generates guidance information to improve the user's exercise intensity or exercise angle using at least one of the following: audio, an image superimposed on the user image data, and text display. The training support device according to claim 1.

8. The analysis unit analyzes the user's physical limitations from the user image data and the voice data, The guidance information generation unit generates guidance information corresponding to the user's physical constraints based on at least one of the user's physical constraints analyzed by the analysis unit and information indicating the user's physical constraints that has been registered in advance. The training support device according to claim 1.

9. The system further includes a storage unit that stores the history of the training tasks performed by the user, and the determination result of the determination unit as to whether or not the user achieved the actions and vocalizations defined in the training tasks. The output unit outputs an evaluation screen for confirmation by a medical professional based on the user's training task history and the judgment result stored in the memory unit. The system further includes a reception unit that accepts changes made by the medical professional to a training menu that includes multiple training tasks assigned to the user. The training support device according to claim 1.

10. The system further includes a training menu generation unit that generates a training menu including multiple training tasks to be assigned to each of the multiple users, according to the physical constraints of each of the multiple users. The training support device according to claim 1.

11. The training menu, which includes multiple training tasks imposed on the user, includes multiple training tasks that differ in the body parts used and the type of vocalizations produced, depending on the physical constraints of each of the multiple users. The determination unit determines the degree of achievement of each of the users for each of the training tasks included in the training menu for each of the multiple users. The training support device according to claim 1.

12. The system further includes a storage unit that stores, for each of the multiple users, the history of the training tasks performed by the multiple users, and the determination result by the determination unit of whether or not the user achieved the actions and vocalizations defined in the training tasks. The determination unit calculates the evaluation of groups into which the multiple users are divided into predetermined numbers, based on the history of the training tasks performed by the users belonging to the group and the determination results. The output unit outputs the evaluation for each group. The training support device according to claim 1.

13. A training support method that uses a computer, An acquisition step of acquiring user image data and voice data of a user who performs a training task in which body movements and vocalizations are defined, An analysis step of analyzing the user's actions and vocalizations from the user image data and the voice data, A determination step is performed to determine the degree of achievement of the user with respect to the physical movements and vocalizations defined in the training task, based on the user's movements and vocalizations analyzed in the analysis step. A guidance information generation step generates guidance information that guides the user's actions and vocalizations to the physical actions and vocalizations defined in the training task, based on the determination result from the determination step. An output step which outputs the generated guidance information, Training support methods including

14. An acquisition step of acquiring user image data and voice data of a user who performs a training task including physical movements and vocalizations, An analysis step of analyzing the user's actions and vocalizations from the user image data and the voice data, A determination step is performed to determine the degree of achievement of the user with respect to the physical movements and vocalizations defined in the training task, based on the user's movements and vocalizations analyzed in the analysis step. A guidance information generation step generates guidance information that guides the user's actions and vocalizations to the physical actions and vocalizations defined in the training task, based on the determination result from the determination step. An output step which outputs the generated guidance information, A program that causes a computer to execute something.

Images