Exercise support device, exercise support system, exercise support method, and exercise support program

Abstract

To create an exercise support device that can provide more useful advice to users who exercise in an unsupervised environment. [Solution] The exercise support device (1) refers to the instruction content given to the target user performing the exercise in an instructional environment and selects a malfunction class corresponding to that instruction content; (2) refers to a video representing the target user's movements while performing the exercise in an uninstructed environment and selects a malfunction class corresponding to those movements from the malfunction classes selected by the first selection means; and (3) refers to the malfunction class selected by the second selection means and generates advice to be presented to the target user performing the exercise in an uninstructed environment.

Description

Technical Field

[0001] This disclosure relates to exercise assistance technology.

Background Art

[0002] Technologies for assisting exercises for rehabilitation are widely used. Patent Document 1 discloses a technology for providing prescription content including training videos, training explanations, motion analysis definitions, etc. to patients according to instructions from medical staff.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Exercises for rehabilitation are not sufficient to be carried out only in an environment with guidance by an instructor (e.g., a physical therapist), such as a medical institution, and it is also necessary to carry them out in an environment without guidance by an instructor (e.g., the patient's home). However, when a user (e.g., a patient) performs an exercise in an environment without guidance, sufficient effects may not be obtained due to, for example, the form collapsing.

[0005] According to the technology described in Patent Document 1, prescription content including training videos, training explanations, motion analysis definitions, etc. can be provided to users. However, even when using the technology described in Patent Document 1, users cannot grasp whether the guidance of an instructor in an environment with guidance is reflected in the actions of users in an environment without guidance. Therefore, the prescription content provided by the technology described in Patent Document 1 cannot be said to be sufficiently beneficial advice for users performing exercises in an environment without guidance.

[0006] This disclosure is made in view of the above-mentioned issues, and one exemplary purpose is to realize exercise support technology that can provide more useful advice to users who exercise in an unsupervised environment. [Means for solving the problem]

[0007] An exercise support device relating to an illustrative aspect of this disclosure includes: a first selection means that selects a malfunction class corresponding to a given instruction content from among a plurality of malfunction classes by referring to the instruction content given to a target user performing the exercise in an instructional environment; a second selection means that selects a malfunction class corresponding to a given action from among the malfunction classes selected by the first selection means by referring to a video representing the action of the target user performing the exercise in an uninstructed environment; and an advice generation means that generates advice to be presented to the target user performing the exercise in the uninstructed environment by referring to the malfunction class selected by the second selection means.

[0008] An exercise support system relating to an illustrative aspect of this disclosure is an exercise support system including a server and a terminal, wherein the server refers to the instruction content for a target user performing the exercise in a supervised environment and selects a malfunction class corresponding to the instruction content from among a plurality of malfunction classes, and uses machine learning with tuning data including video representing the actions of the target user performing the exercise in the supervised environment and the malfunction class selected by the first selection means to determine from a general model in which the input is video representing the actions of the user performing the exercise and the output is one of the plurality of malfunction classes, the input is video representing the actions of the user performing the exercise and the output is the malfunction selected by the first selection means The terminal comprises a tuning means for generating an individually optimized model which is one of the error classes, and the terminal comprises a second selection means for selecting an error class corresponding to an error from among the error classes selected by the first selection means, by referring to a video representing the actions of the target user performing the exercise in an unsupervised environment, the second selection means for selecting an error class output from the individually optimized model when a video representing the actions of the target user performing the exercise in the unsupervised environment is input to the individually optimized model, as the error class corresponding to the error, and an advice generation means for generating advice to be presented to the target user performing the exercise in the unsupervised environment by referring to the error class selected by the second selection means.

[0009] An exercise support system relating to an exemplary aspect of this disclosure is an exercise support system including a server and a terminal, wherein the server includes a first selection means that refers to the instruction content for a target user performing the exercise in a supervised environment and selects a malfunction class corresponding to the instruction content from among a plurality of malfunction classes, and the terminal uses machine learning with tuning data including video representing the actions of the target user performing the exercise in the supervised environment and the malfunction class selected by the first selection means to determine from a general model in which the input is video representing the actions of the user performing the exercise and the output is one of the plurality of malfunction classes, that the input is video representing the actions of the user performing the exercise and the output is The system includes: a tuning means for generating an individually optimized model which is one of the more selected malfunction classes; a second selection means for selecting a malfunction class corresponding to a target user's actions while the target user is performing the exercise in an unsupervised environment, by referring to a video of the target user's actions while the target user is performing the exercise in an unsupervised environment; the second selection means for selecting a malfunction class output from the individually optimized model when the video of the target user's actions while the target user is performing the exercise in an unsupervised environment is input to the individually optimized model; and an advice generation means for generating advice to be presented to the target user performing the exercise in an unsupervised environment by referring to the malfunction class selected by the second selection means.

[0010] An exercise support method relating to an illustrative aspect of this disclosure includes: a first selection process in which at least one processor refers to instruction content for a target user performing the exercise in a supervised environment and selects a malfunction class corresponding to the instruction content from among a plurality of malfunction classes; a second selection process in which at least one processor refers to video representing the actions of the target user performing the exercise in an unsupervised environment and selects a malfunction class corresponding to those actions from among the malfunction classes selected in the first selection process; and an advice generation process in which at least one processor refers to the malfunction class selected in the second selection process and generates advice to be presented to the target user performing the exercise in the unsupervised environment. [Effects of the Invention]

[0011] According to an illustrative aspect of this disclosure, it is possible to realize exercise support technology that can provide more useful advice to users who exercise in an environment without guidance. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing the configuration of the exercise support device related to this disclosure. [Figure 2] This flowchart shows the flow of the exercise support method related to this disclosure. [Figure 3] This figure shows a specific example of the instructional content referenced by the first selection unit of the exercise support device shown in Figure 1. [Figure 4] Figure 1 shows a specific example of a malfunction class that can be selected by the first selection unit of the exercise support device shown in Figure 1. [Figure 5] This figure shows a specific example of the instruction content generated by the advice generation unit of the exercise support device shown in Figure 1. [Figure 6] This block diagram shows an example configuration of the second selection unit of the exercise support device shown in Figure 1. [Figure 7]It is a block diagram showing the configuration of an exercise support device according to the present disclosure. [Figure 8] The upper part is a diagram showing a specific example of a guided environment video referred to by the first video division unit of the exercise support device shown in FIG. 7. The middle part is a diagram showing a specific example of an exercise video generated by the first video division unit of the exercise support device shown in FIG. 7. The lower part is a diagram showing a specific example of a unit exercise video generated by the second video division unit of the exercise support device shown in FIG. 7. [Figure 9] It is a diagram showing a specific example of the guidance content generated by the voice analysis unit of the exercise support device shown in FIG. 7. [Figure 10] It is a block diagram showing the configuration of an exercise support device according to the present disclosure. [Figure 11] It is a diagram showing a specific example of a summary sentence generated by the summary / extraction unit of the exercise support device shown in FIG. 10. [Figure 12] It is a diagram showing a specific example of an explanatory video generated by the explanatory video generation unit of the exercise support device shown in FIG. 10. [Figure 13] It is a diagram showing a specific example of multimedia content generated by the composition unit of the exercise support device shown in FIG. 10. [Figure 14] It is a block diagram showing the configuration of an exercise support system according to the present disclosure. [Figure 15] It is a block diagram showing a modified example of the exercise support system shown in FIG. 14. [Figure 16] It is a block diagram showing a modified example of the exercise support system shown in FIG. 14. [Figure 17] It is a block showing the configuration of a computer operating as an exercise support device according to the present disclosure.

Embodiments for Carrying Out the Invention

[0013] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0014] [Explanation of Terms] In this disclosure, exercise performed for healthcare purposes (such as recovery, maintenance, or improvement of motor function) is referred to as "exercise." The person performing the exercise (e.g., a patient) is referred to as the "target user," and the person who instructs the target user in the exercise (e.g., a medical professional such as a physical therapist) is referred to as the "instructor."

[0015] Furthermore, in this disclosure, the environment in which target users perform exercises under the guidance of an instructor is referred to as a "supervised environment." A supervisory environment includes both offline supervisory environments where instruction is received offline, and online supervisory environments where instruction is received online. Examples of offline supervisory environments include medical facilities and gyms. Another example of an offline supervisory environment is a web meeting.

[0016] Furthermore, in this disclosure, an environment in which target users perform exercises without instruction from an instructor is referred to as an "uninstructed environment." An uninstructed environment can also be described as an environment for self-training. An example of an uninstructed environment is the target user's home.

[0017] This disclosure primarily assumes a scenario where a target user performs exercises in a supervised environment, and then performs those exercises in an unsupervised environment. It then describes technologies for supporting unsupervised exercises by utilizing information obtained from the supervised exercises.

[0018] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0019] (Configuration of the exercise support device) The configuration of the exercise support device 1 will be explained with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the exercise support device 1.

[0020] The exercise support device 1 is a device for supporting exercise in an unsupervised environment. As shown in Figure 1, the exercise support device 1 comprises a first selection unit 11, a second selection unit 12, and an advice generation unit 13.

[0021] The first selection unit 11 is a means for selecting a malfunction class corresponding to the instruction content given to a target user TU performing exercise E in a supervised environment, from among a predetermined set of malfunction classes C1, C2, ..., Cn. Each malfunction class Ci corresponds to a typical malfunction that frequently occurs in exercise E. Here, n is a natural number greater than or equal to 2, and i is a natural number between 1 and n. Hereinafter, the malfunction classes selected by the first selection unit 11 will be referred to as malfunction classes C'1, C'2, ..., C'm, where m is a natural number between 1 and n. Note that malfunction classes C1, C2, ..., Cn may include classes that do not correspond to typical malfunctions.

[0022] The first selection unit 11, for example, when it receives input of instruction content (represented by text or audio) for a target user TU performing exercise E, uses a machine learning model M1 that outputs the malfunction class with the highest probability (likelihood) corresponding to that instruction content from among predetermined malfunction classes C1, C2, ..., Cn. In this case, the first selection unit 11 inputs the instruction content for the target user TU performing exercise E in a supervised environment into model M1, and at the same time selects the malfunction class output from model M1 as the malfunction class corresponding to that instruction content.

[0023] Alternatively, the first selection unit 11 may use keywords associated with each of the predetermined malfunction classes C1, C2, ..., Cn. In this case, the first selection unit 11 selects a malfunction class associated with a keyword that matches the instruction content (represented by text) for a target user TU performing exercise E in a supervised environment, as the malfunction class corresponding to that instruction content.

[0024] Note that the first selection unit 11 selects only one malfunction class for each instructional content. However, in a single exercise E, the target user TU may repeatedly perform unit exercises, and the instructor may provide instruction for each unit exercise. In this case, the first selection unit 11 may refer to the instructional content for each unit exercise and select a malfunction class corresponding to each unit exercise. This is why there may be multiple malfunction classes C'1, C'2, ..., C'm selected by the first selection unit 11.

[0025] The second selection unit 12 is a means for selecting a malfunction class corresponding to the actions of a target user TU performing exercise E in an unsupervised environment, from among the single or multiple malfunction classes C'1, C'2, ..., C'm selected by the first selection unit 11, by referring to a video representing the actions of that target user TU. Hereinafter, the malfunction classes selected by the second selection unit 12 will be referred to as malfunction classes C"1", C"2", ..., C"k," where k is a natural number between 1 and m (inclusive).

[0026] The second selection unit 12, for example, when it receives video footage representing the actions of a target user TU performing exercise E, uses a machine learning model M2 that outputs the error class with the highest probability (likelihood) corresponding to that action from among the error classes C'1, C'2, ..., C'm selected by the first selection unit 11. In this case, the second selection unit 12 inputs video footage representing each action of the target user TU performing exercise E in a supervised environment into model M2, and selects the error class output by model M2 at this time as the error class corresponding to that action. The model M2 used by the second selection unit 12 will be described later as an individually optimized model M2.

[0027] Note that the second selection unit 12 selects only one malfunction class for each video representing an action. However, in a single exercise E, the target user TU may repeatedly perform the unit exercises. Therefore, the second selection unit 12 may refer to the video representing the action for the target user TU in each unit exercise and select a malfunction class corresponding to each unit exercise. This is why there may be multiple malfunction classes C"1", C"2", ..., C'k selected by the second selection unit 12.

[0028] The advice generation unit 13 is a means for generating advice to be presented to a target user TU performing exercise E in an unsupervised environment, by referring to one or more malfunction classes C"1", C"2", ..., C"k selected by the second selection unit 12. As an example, the advice generation unit 13 generates advice that points out the contents of the malfunction classes C"1", C"2", ..., C"k selected by the second selection unit 12 as problems.

[0029] Furthermore, the advice generation unit 13 may be configured to generate advice by referring to the instruction content for the target user TU who is performing exercise E in an instructional environment, in addition to the malfunction classes C"1", C"2", ..., C"k selected by the second selection unit 12.

[0030] Furthermore, the advice generated by the advice generation unit 13 is displayed, for example, on the display of the target user TU's mobile terminal. In this case, the advice generated by the advice generation unit 13 is transmitted to the target user TU's mobile terminal via the network.

[0031] (Exercise support method flow) The flow of exercise support method S1 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of exercise support method S1.

[0032] Exercise support method S1 is a method for supporting exercise in an unsupervised environment. As shown in Figure 2, exercise support method S1 includes a first selection process S11, a second selection process S12, and an advice generation process S13. Exercise support method S1 is executed, for example, by exercise support device 1.

[0033] The first selection process S11 is a process for selecting a malfunction class corresponding to the instruction content given to the target user TU who is performing exercise E in a supervised environment, from among a predetermined set of malfunction classes C1, C2, ..., Cn. Each malfunction class Ci represents a typical malfunction that frequently occurs in exercise E. Here, n is a natural number greater than or equal to 2, and i is a natural number between 1 and n. Hereafter, the malfunction classes selected in the first selection process S11 will be referred to as malfunction classes C'1, C'2, ..., C'm, where m is a natural number between 1 and n. Note that malfunction classes C1, C2, ..., Cn may include classes that do not correspond to typical malfunctions. The first selection process S11 is performed, for example, by the first selection unit 11 of the exercise support device 1.

[0034] In the first selection process S11, for example, when instruction content (represented by text or audio) for a target user TU is input, a machine learning model M1 is used to output the malfunction class with the highest probability (likelihood) corresponding to that instruction content from among predetermined malfunction classes C1, C2, ..., Cn. In this case, in the first selection process S11, the instruction content for the target user TU, who is performing exercise E in a supervised environment, is input to model M1, and the malfunction class output by model M1 at this time is selected as the malfunction class corresponding to that instruction content.

[0035] In the first selection unit 11, or, keywords associated with each of the predetermined malfunction classes C1, C2, ..., Cn are used. In this case, in the first selection process S11, the malfunction class associated with the keyword that matches the instruction content (represented by the text) for the target user TU who is performing exercise E in the instruction environment is selected as the malfunction class corresponding to that instruction content.

[0036] Note that only one malfunction class is selected in the first selection process S11 for each instructional content. However, in a single exercise E, the target user TU may repeatedly perform unit exercises, and the instructor may provide instruction for each unit exercise. In this case, the first selection process S11 may refer to the instructional content for each unit exercise and select a malfunction class corresponding to each unit exercise. This is why there may be multiple malfunction classes C'1, C'2, ..., C'm selected in the first selection process S11.

[0037] The second selection process S12 is a process for selecting a malfunction class corresponding to the actions of the target user TU performing exercise E in an unsupervised environment, by referring to a video of the actions of the target user TU, and selecting one or more malfunction classes C'1, C'2, ..., C'm selected in the first selection process S11. Hereinafter, the malfunction classes selected in the second selection process S12 will be referred to as malfunction classes C"1", C"2", ..., C"k, where k is a natural number between 1 and m (inclusive). The second selection process S12 is performed, for example, by the second selection unit 12 of the exercise support device 1.

[0038] In the second selection process S12, for example, when a video representing the actions of the target user TU is input, a machine learning model M2 is used to output the malfunction class with the highest probability (likelihood) corresponding to that action from among the malfunction classes C'1, C'2, ..., C'm selected in the first selection process S11. In this case, in the second selection process S12, a video representing the actions of the target user TU performing exercise E in a supervised environment is input to model M2, and the malfunction class output by model M2 at this time is selected as the malfunction class corresponding to that action. The model M2 used in the second selection process S12 will be described later as an individually optimized model M2.

[0039] Note that only one malfunction class is selected in the second selection process S12 for each video representing an action. However, in a single exercise E, the target user TU may repeatedly perform unit exercises. Therefore, in the second selection process S12, the video representing the action for the target user TU in each unit exercise may be referenced, and a malfunction class corresponding to each unit exercise may be selected. This is why there may be multiple malfunction classes C"1, C"2, ..., C'k selected in the second selection process S12.

[0040] The advice generation process S13 is a process for generating advice to be presented to the target user TU who is performing exercise E in an unsupervised environment, by referring to the single or multiple malfunction classes C"1", C"2", ..., C"k selected in the second selection process S12. As an example, in the advice generation process S13, advice is generated that points out the contents of the malfunction classes C"1", C"2", ..., C"k selected in the second selection process S12 as problems.

[0041] In addition, in the advice generation process S13, in addition to the malfunction classes C"1", C"2", ..., C"k selected in the second selection process S12, the advice may also be generated by referring to the instruction content for the target user TU who is performing exercise E in the instructional environment.

[0042] Furthermore, the advice generated by the advice generation process S13 is displayed, for example, on the display of the target user TU's mobile terminal. In this case, the advice generated in the advice generation process S13 is transmitted to the target user TU's mobile terminal via the network.

[0043] (Effects of exercise support devices and exercise support methods) As described above, the exercise support device 1 or exercise support method S1 can generate advice to be presented to a target user TU performing exercise E in an unsupervised environment, which is in accordance with the content of the instruction given to the target user TU performing exercise E in a supervised environment.

[0044] Furthermore, the advice generated by the exercise support method S1 is generated in accordance with the error classes C'1, C'2, ..., C'k selected from the error classes C'1, C'2, ..., C'm selected by referring to the instruction content given to the target user TU performing exercise E in an instructional environment, and in particular by referring to the error classes C"1, C"2, ..., C"k selected by referring to the video representing the actions of the target user TU performing exercise E in an uninstructed environment. In other words, it is advice corresponding to the errors that occurred when performing exercise E in an uninstructed environment, in particular, among the errors that were instructed when performing exercise E in an instructional environment.

[0045] Therefore, the exercise support device 1 or exercise support method S1 can generate highly beneficial advice for the target user TU, which is focused on the points that the target user TU should concentrate on, as advice to be presented to the target user TU performing exercise E in an unsupervised environment.

[0046] (Specific examples of instruction content, malfunction classes, and advice) The instructional content referenced by the first selection unit 11, the malfunction class selected by the first selection unit 11, and specific examples of advice generated by the advice generation unit 13 will be explained with reference to Figures 3 to 5. Here, we consider "clamshell" as an example of exercise E.

[0047] The instructional content referenced by the first selection unit 11 is, for example, a part of the instructor's utterances in a teaching environment. The instructor's utterances in a teaching environment include (1) utterances corresponding to "explanations" to the target user TU before performing each unit exercise that constitutes Exercise E, and (2) utterances corresponding to "instruction" (feedback) to the target user TU during or after performing each unit exercise that constitutes Exercise E. The instructional content referenced by the first selection unit 11 is the utterances from the instructor's utterances in a teaching environment that correspond to "instruction" to the target user TU during or after performing each unit exercise that constitutes Exercise E.

[0048] Figure 3 shows a specific example of instructional content referenced by the first selection unit 11. Figure 3 illustrates five instructional contents in one exercise E. Each instructional item is marked with a timestamp indicating the time the instruction was given.

[0049] The first instruction, spoken 29 seconds after the start of the exercise, is: "Be careful not to open your body too much or lean too far forward." The second instruction, spoken 1 minute and 21 seconds after the start of the exercise, is: "Bend your hips slightly and consciously use the back of your buttocks." The third instruction, spoken 1 minute and 51 seconds after the start of the exercise, is: "Be conscious of using your gluteal muscles properly and make sure your body doesn't wobble." The fourth instruction, spoken 2 minutes and 26 seconds after the start of the exercise, is: "Be careful not to round or arch your back." The fifth instruction, spoken 2 minutes and 41 seconds after the start of the exercise, is: "Keep your body from swaying back and forth and stabilize the position of your pelvis."

[0050] Figure 4 shows specific examples of malfunction classes that can be selected by the first selection unit 11. Figure 4 illustrates eight malfunction classes C1 to C8. Each of these malfunction classes C1 to C8 represents a typical malfunction that frequently occurs in Exercise E.

[0051] The model M1 used by the first selection unit 11 is machine-trained to output the malfunction class with the highest probability (likelihood) among the malfunction classes C1 to C8 shown in Figure 4, when the instruction content (represented by text or audio) for the target user TU is input.

[0052] Of the instructional content shown in Figure 3, the first instructional content, "Be careful not to let your body open up or lean too far forward," is a typical instructional content that an instructor would point out when the malfunction "pelvic rolling" is observed. Therefore, when the first selection unit 11 receives the first instructional content, "Be careful not to let your body open up or lean too far forward," as input, the model M1 outputs a malfunction class C1 representing the malfunction "pelvic rolling."

[0053] Furthermore, among the instructional content shown in Figure 3, the second instructional content, "Bend your hip joint slightly and use the back of your buttocks carefully," is a typical instructional content that an instructor would point out when the malfunction "hip joint angle is not 135°" is observed. Therefore, when the model M1 used by the first selection unit 11 receives the second instructional content, "Bend your hip joint slightly and use the back of your buttocks carefully," it outputs a malfunction class C5 representing the malfunction "hip joint angle is not 135°."

[0054] Furthermore, among the instructional content shown in Figure 3, the third instructional content, "Be conscious of using your gluteal muscles properly and keep your body from wobbling," is a typical instructional content that an instructor would point out when the malfunction "pelvic rolling" is observed, similar to the first instructional content. Therefore, when the model M1 used by the first selection unit 11 receives the third instructional content, "Be conscious of using your gluteal muscles properly and keep your body from wobbling," it outputs a malfunction class C1 representing the malfunction "pelvic rolling."

[0055] Furthermore, among the instructional content shown in Figure 3, the fourth instructional content, "Be careful not to hunch or arch your back," is a typical instructional content that an instructor would point out when the malfunction "back is arched / body is not straight" is observed. Therefore, when the model M1 used by the first selection unit 11 receives the fourth instructional content, "Be careful not to hunch or arch your back," it outputs a malfunction class C4 representing the malfunction "back is arched / body is not straight."

[0056] Furthermore, among the instructional content shown in Figure 3, the fifth instructional content, "Keep your body from swaying back and forth and stabilize the position of your pelvis," is a typical instructional content that an instructor would point out when the malfunction "pelvic rolling" is observed, similar to the first and third instructional content. When the model M1 used by the first selection unit 11 receives the fifth instructional content, "Keep your body from swaying back and forth and stabilize the position of your pelvis," it outputs a malfunction class C1 representing the malfunction "pelvic rolling."

[0057] As described above, if the instructional content referenced by the first selection unit 11 in a single exercise E is the five instructional content shown in Figure 3, the first selection unit 11 selects three malfunction classes C1, C4, and C5. Therefore, in this case, the second selection unit 12 refers to the video representing the actions of the target user TU performing exercise E in an uninstructed environment and selects the malfunction class corresponding to that action from among the three malfunction classes C1, C4, and C5 selected by the first selection unit 11.

[0058] Figure 5 shows a specific example of advice generated by the advice generation unit 13. The advice shown in Figure 5 is an example of advice generated when the malfunction classes selected by the second selection unit 12 for the current exercise E are malfunction classes C4 and C5, and the malfunction classes selected by the second selection unit 12 for the previous exercise E were malfunction classes C1 and C5.

[0059] In the advice illustrated in Figure 5, the content of malfunction classes C4 and C5 selected by the second selection unit 12 for this exercise E is pointed out as a problem (see "This time's mistake" in Figure 6). Furthermore, in the advice illustrated in Figure 5, the content of malfunction class C1, which was selected by the second selection unit 12 for the previous exercise E but not selected by the second selection unit 12 for this exercise E, is pointed out as an improved problem (see "Improved mistake" in Figure 5). Also, in the advice illustrated in Figure 5, the content of malfunction class C5, which was selected by the second selection unit 12 for the previous exercise E but not selected by the second selection unit 12 for this exercise E, is pointed out as a problem that has not been improved (see "Mistake with little improvement" in Figure 5).

[0060] Furthermore, the advice illustrated in Figure 5 includes comments corresponding to malfunction class C1, which represents an improved malfunction, such as "One improvement is that the hip... has increased," and comments corresponding to malfunction class C5, which represents a malfunction that has not been improved, such as "On the other hand, attention is still needed regarding the buttocks... issue." These comments can be generated, for example, by referring to the instruction given to a target user TU performing exercise E in a supervised environment.

[0061] (Example of the configuration of the second selection section) An example of the configuration of the second selection unit 12 of the exercise support device 1 will be explained with reference to Figure 6. Figure 6 is a block diagram showing an example of the configuration of the second selection unit 12.

[0062] As shown in Figure 6, the second selection unit 12 can be composed of a tuning unit 121, a general model M0, and an individually optimized model M2.

[0063] In this specific example, a video representing a certain action of a target user TU performing exercise E in a supervised environment, and instructional content for that action are provided as a set. The first selection unit 11 refers to the instructional content that constitutes the set and selects a malfunction class corresponding to the action of the target user TU represented by the video that constitutes the set.

[0064] The general model M0 is a model that has been pre-trained to output the error class with the highest probability (likelihood) corresponding to a given error among the predetermined error classes C1, C2, ..., Cn when a video representing a user's actions is input. The tuning unit 121 generates an individually optimized model M2 from the general model M0 by machine learning (retraining or additional training) using tuning data that combines the videos constituting the above set and the error classes selected by the first selection unit 11 by referring to the instructional content constituting the above set. When a video representing the actions of the target user TU is input, the individually optimized model M2 outputs the error class with the highest probability (likelihood) corresponding to that error among the error classes C'1, C'2, ..., C'm selected by the first selection unit 11.

[0065] The second selection unit 12 inputs video footage representing the actions of the target user TU performing exercise E in an unsupervised environment into the individual optimization model M2, and selects the malfunction class output from the individual optimization model M2 as the malfunction class corresponding to that action.

[0066] By configuring the second selection unit 12 as described above, it is possible to appropriately select a malfunction class corresponding to the behavior of the target user TU using the individually optimized model M2 optimized for the target user TU.

[0067] [Second exemplary embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0068] (Configuration of the exercise support device) The exercise support device 1A will be described with reference to Figures 7, 8, and 9. Figure 7 is a block diagram showing the configuration of the exercise support device 1A. Figure 8 is a diagram showing the uninstructed environment video, exercise video, and unit exercise video used in the exercise support device 1A. Figure 9 is a diagram showing the configuration of the exercise text used in the exercise support device 1A.

[0069] Exercise support device 1A is an exercise support device optimized for situations where a target user TU sequentially performs multiple exercises E1, E2, ..., Ep in both an instructed and uninstructed environment. The sth exercise performed by the target user TU will be referred to as exercise Es below. Here, p is a natural number greater than or equal to 1, and s is a natural number between 1 and p. Each exercise Es is a unit exercise Es1, Es2, ..., Esq s It shall be assumed that it is composed of the following. In each exercise Es, the unit exercise that the target user TU performs as the tth unit exercise will be referred to below as unit exercise Est. Here, q s is a natural number greater than or equal to 1, which depends on s, and t is a natural number greater than or equal to 1, q s The following are natural numbers. Note that the unit exercise Est is sometimes called a "rep".

[0070] Exercise support device 1A refers to the following video with instruction, audio with instruction, and video without instruction. The video recording and audio recording of the instruction environment described below shall be performed simultaneously in the instruction environment.

[0071] Video of the instructional environment: Video recorded in an instructional environment, representing the actions of the target user TU as they sequentially perform exercises E1, E2, ..., Ep. The video of the instructional environment is recorded, for example, on the target user TU's mobile device. In this case, the exercise support device 1A acquires the video of the instructional environment from the target user TU's mobile device via the network.

[0072] Audio in a teaching environment: Audio recorded in a teaching environment, representing the speech of the target user TU performing exercises E1, E2, ..., Ep in order, and the speech of the instructor guiding the target user TU. The recording of the audio in a teaching environment is performed, for example, on the instructor's mobile device. In this case, the exercise support device 1A acquires the audio in a teaching environment from the instructor's mobile device via the network.

[0073] Unsupervised environment video: Video recorded in an unsupervised environment, representing the actions of the target user TU as they sequentially perform exercises E1, E2, ..., Ep. The recording of the unsupervised environment video is performed, for example, on the target user TU's mobile device. In this case, the exercise support device 1A acquires the unsupervised environment video from the target user TU's mobile device via the network.

[0074] As shown in Figure 7, the exercise support device 1A includes a first selection unit 11, a second selection unit 12, an advice generation unit 13, a first video splitting unit 14, a second video splitting unit 15, an audio splitting unit 16, an audio analysis unit 17, and an extraction unit 18.

[0075] The first selection unit 11, the second selection unit 12, and the advice generation unit 13 of the exercise support device 1A function in the same way as the first selection unit 11, the second selection unit 12, and the advice generation unit 13 of the advice support device 1 shown in Figure 1. The functions of the first video splitting unit 14, the second video splitting unit 15, the audio splitting unit 16, the audio analysis unit 17, and the extraction unit 18 of the exercise support device 1A will be described below.

[0076] The first video splitting unit 14 is a means for splitting a video of the instructional environment, recorded in the instructional environment, into exercise videos corresponding to each exercise Es performed in that instructional environment. Here, the exercise video corresponding to exercise Es is a video representing the actions of the target user TU while performing that exercise Es in the instructional environment. An example of the instructional environment video before splitting is shown in the upper part of Figure 8, and an example of the exercise videos after splitting is shown in the middle part of Figure 8.

[0077] Alternatively, instead of the first video splitting unit 14, which divides the instructional environment video into exercise videos corresponding to each exercise Es, an identification unit that identifies the start time τs and end time τ's of each exercise Es in the instructional environment video may be used.

[0078] The second video splitting unit 15 is a means for splitting the exercise video corresponding to each exercise Es obtained by the first video splitting unit 14 into unit exercise videos corresponding to each unit exercise Est that constitutes that exercise Es. Here, the unit exercise video corresponding to each unit exercise Est is a video representing the actions of the target user TU performing that unit exercise Est in a training environment. An example of an exercise video before splitting is shown in the middle section of Figure 8, and an example of a unit exercise video after splitting is shown in the lower section of Figure 8.

[0079] Furthermore, if the identification unit described above is used instead of the first video splitting unit 14, the second video splitting unit 15 refers to the start time τs and end time τ's of each exercise Es identified by the identification unit described above and splits the instructional environment video into unit exercise videos corresponding to each unit exercise Est.

[0080] The audio splitting unit 16 is a means for splitting the audio of the instructional environment, recorded in the instructional environment, into exercise audio corresponding to each exercise Es performed in that instructional environment. Here, the exercise audio corresponding to each exercise Es is audio data that includes the utterances of the target user TU performing that exercise Es and the utterances of the instructor guiding the target user TU.

[0081] The speech analysis unit 17 is a means for generating exercise text corresponding to each exercise Es obtained by the speech splitting unit 16 by performing speech analysis on the exercise audio corresponding to that exercise Es. Here, the exercise text corresponding to each exercise Es is text data that includes the utterances of the target user TU performing that exercise Es and the utterances of the instructor guiding the target user TU. The speech analysis unit 17 may associate each utterance with a timestamp indicating the start time of the utterance and an identifier indicating the speaker of the utterance. An example of exercise text corresponding to exercise E1 is shown in Figure 9.

[0082] In the exercise text corresponding to each exercise Es, the utterances of the instructor guiding the target user TU include (1) utterances that correspond to "explanation" to the target user TU before performing each unit exercise Est, and (2) utterances that correspond to "instruction" (feedback) to the target user TU during or after performing each unit exercise Est. In the exercise text corresponding to exercise E1 shown in Figure 9, the five underlined utterances correspond to "explanation," and the five utterances written in bold correspond to "instruction."

[0083] The extraction unit 18 is a means for extracting utterances corresponding to "instruction" from the exercise text corresponding to each exercise Es obtained by the speech analysis unit 17, as the instruction content for that exercise Es. For example, the extraction unit 18 extracts the five utterances written in bold from the exercise text corresponding to exercise E1 shown in Figure 9 as the instruction content for exercise E1. The instruction content shown in Figure 3 is an example of instruction content extracted in this way by the extraction unit 18.

[0084] In this exemplary embodiment, the first selection unit 11 refers to the instructional content corresponding to each unit exercise Est obtained by the extraction unit 18 and selects a malfunction class corresponding to that instructional content. This makes it possible to select a malfunction class corresponding to each unit exercise Est that constitutes each exercise Es.

[0085] Furthermore, in this embodiment, the tuning unit 121 (see Figure 6) of the second selection unit 12 generates tuning data by combining the unit exercise video corresponding to each unit exercise Est that constitutes exercise Es and the instruction content corresponding to that unit exercise Est from the instruction content extracted by the extraction unit 18. Then, the tuning unit 121 of the second selection unit 12 generates an individually optimized model from the general model M0 by machine learning (retraining or additional training) using this tuning data. As mentioned above, the instruction content extracted by the extraction unit 18 is given a timestamp. The tuning unit 121 of the second selection unit 12 can identify the instruction content corresponding to each unit exercise Est by referring to the value of this timestamp.

[0086] Furthermore, the first video splitting unit 14 also functions as a means for splitting the unsupervised environment video, recorded in an unsupervised environment, into exercise videos corresponding to each exercise Es performed in that unsupervised environment. Here, the exercise video corresponding to exercise Es is a video representing the actions of the target user TU while performing that exercise Es in an unsupervised environment.

[0087] Furthermore, the second video splitting unit 15 also functions as a means for splitting the exercise video corresponding to each exercise Es obtained by the first video splitting unit 14 into unit exercise videos corresponding to each unit exercise Est that constitutes that exercise Es. Here, the unit exercise video corresponding to each unit exercise Est is a video representing the actions of the target user TU while performing that unit exercise Est in an unsupervised environment.

[0088] In this exemplary embodiment, the second selection unit 12 refers to the unit exercise video corresponding to each unit exercise Est obtained by the second video division unit 15 and selects the malfunction class corresponding to that unit exercise Est. This makes it possible to select the malfunction class corresponding to each unit exercise Est that constitutes each exercise Es.

[0089] [Third Exemplary Embodiment] A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiments are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs.

[0090] (Configuration of the exercise support device) The configuration of the exercise support device 1B will be explained with reference to Figure 10. Figure 10 is a block diagram showing the configuration of the exercise support device 1B. Figure 11 is a diagram showing an example of explanatory text generated by the exercise support device 1B. Figure 12 is a diagram showing an example of explanatory video generated by the exercise support device 1. Figure 13 is a diagram showing an example of multimedia content generated by the exercise support device 1.

[0091] Exercise support device 1B, like exercise support device 1A, is an exercise support device optimized for situations where a target user TU sequentially performs multiple exercises E1, E2, ..., Ep in both an instructed and uninstructed environment.

[0092] As shown in Figure 10, the exercise support device 1B includes a first selection unit 11, a second selection unit 12, an advice generation unit 13, a first video splitting unit 14, a second video splitting unit 15, an audio splitting unit 16, an audio analysis unit 17, an extraction unit 18, an extraction / summarization unit 19, an explanatory video generation unit 21, and a synthesis unit 22.

[0093] The first selection unit 11, second selection unit 12, advice generation unit 13, first video splitting unit 14, second video splitting unit 15, audio splitting unit 16, audio analysis unit 17, and extraction unit 18 of the exercise support device 1B function in the same way as the first selection unit 11, second selection unit 12, advice generation unit 13, first video splitting unit 14, second video splitting unit 15, audio splitting unit 16, audio analysis unit 17, and extraction unit 18 of the exercise support device 1A. The functions of the extraction / summarization unit 19, explanatory video generation unit 21, and synthesis unit 23 of the exercise support device 1B will be described below.

[0094] The extraction / summarization unit 19 is a means for extracting utterances corresponding to "instruction" from the exercise text corresponding to each exercise Es obtained by the speech analysis unit 17, as the instruction content for that exercise Es. Furthermore, the extraction / summarization unit 19 is a means for generating explanatory text to be presented to the target user TU who performs the exercise Es in an uninstructed environment by summarizing the exercise text corresponding to each exercise Es obtained by the speech analysis unit 17.

[0095] Preferably, the explanatory text generated by the extraction / summarization unit 19 includes, for example, at least one of the purpose, method, and number of repetitions of Exercise Es. Preferably, this explanatory text also includes each instructional content of Exercise Es. Preferably, this explanatory text also includes the positive and / or negative aspects of Exercise Es. An example of the explanatory text generated by the extraction / summarization unit 19 is shown in Figure 11.

[0096] In this exemplary embodiment, the first video splitting unit 14 also functions as a means for splitting the video of the instructional environment, recorded in the instructional environment, into exercise videos corresponding to each exercise Es performed in that instructional environment.

[0097] The explanatory video generation unit 21 is a means for generating explanatory videos to be presented to target users TU who perform exercise Es in an unsupervised environment, by referring to the exercise videos corresponding to each exercise Es obtained by the first video splitting unit 14, and the instructional content in that exercise Es extracted by the extraction / summarization unit 19.

[0098] The explanatory video generated by the explanatory video generation unit 21 is preferably a video obtained by superimposing each instructional content in exercise Es onto the exercise video corresponding to exercise Es. In this case, it is preferable that the instructional content spoken t seconds after the start of exercise Es is superimposed onto the video at t seconds after the start of exercise Es. The explanatory video generation unit 21 can generate such an explanatory video by referring to the timestamp assigned to each instructional content. An example of an explanatory video generated by the explanatory video generation unit 21 is shown in Figure 12.

[0099] The synthesis unit 22 is a means for generating multimedia content to be presented to a target user TU who performs exercise Es in an unsupervised environment, by synthesizing the explanatory video generated by the explanatory video generation unit 21 and the explanatory text generated by the extraction / summarization unit 19.

[0100] The multimedia content generated by the synthesis unit 22 preferably includes explanatory text generated by the extraction / summarization unit 19 as text content and explanatory video generated by the explanatory video generation unit 21 as video content. The multimedia comments generated by the synthesis unit 22 may be delivered to the target user TU as an email or as a web page. An example of multimedia comments generated by the synthesis unit is shown in Figure 13.

[0101] The exercise support device 1B can provide the target user TU with explanatory text generated by the extraction / summarization unit 19. Furthermore, the exercise support device 1B can provide the target user TU with explanatory videos generated by the explanatory video generation unit 21. Therefore, the exercise support device 1B can provide the target user TU with even more beneficial information.

[0102] [Fourth exemplary embodiment] A fourth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiments are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical hindrance occurs.

[0103] (Configuration of the exercise support system) The configuration of the exercise support system 100 will be explained with reference to Figure 14. Figure 14 is a block diagram showing the configuration of the exercise support system 100. In the block diagram shown in Figure 14, each block of each device in the exercise support system 100 is arranged according to the processing flow. Therefore, blocks that are used multiple times at different timings are shown multiple times.

[0104] As shown in Figure 14, the exercise support system 100 consists of a server 101, a user terminal 102, and an instructor terminal 103. The server 101 and the user terminal 102 are configured to communicate with each other via a network. The server 101 and the instructor terminal 103 are also configured to communicate with each other via a network.

[0105] Server 101 includes a first selection unit 11, a second selection unit 12, an advice generation unit 13, a first video splitting unit 14, a second video splitting unit 15, an audio splitting unit 16, an audio analysis unit 17, and an extraction unit 18. The first selection unit 11, second selection unit 12, advice generation unit 13, first video splitting unit 14, second video splitting unit 15, audio splitting unit 16, audio analysis unit 17, and extraction unit 18 of Server 101 function in the same way as the first selection unit 11, second selection unit 12, advice generation unit 13, first video splitting unit 14, second video splitting unit 15, audio splitting unit 16, audio analysis unit 17, and extraction unit 18 of Exercise Support Device 1A, respectively.

[0106] User terminal 102 is configured to record video of the instructional environment and video of the non-instructional environment. Instructor terminal 103 is configured to record audio of the instructional environment.

[0107] The exercise support system 100 operates as follows:

[0108] When the exercise in the supervised environment is completed, the user terminal 102 transmits the recorded video of the supervised environment to the server 101 via the network. The server 101 uses the first video splitting unit 14 and the second video splitting unit 15 to generate unit exercise videos from the supervised environment video received from the user terminal 102, representing the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the supervised environment.

[0109] Furthermore, once the exercise in the supervised environment is completed, the instructor terminal 103 transmits the recorded audio of the supervised environment to the server 101 via the network. The server 101 uses the audio splitting unit 16, audio analysis unit 17, extraction unit 18, and first selection unit 11 to select a malfunction class from the supervised environment audio received from the instructor terminal 103 that corresponds to the behavior of the target user TU who performs each unit exercise Est that constitutes exercise Es in the supervised environment.

[0110] Then, the tuning unit 121 of the server 101 generates tuning data for each unit exercise Est that constitutes exercise Es, by combining a unit exercise video representing the actions of a user performing that unit exercise Est in a supervised environment, and a malfunction class corresponding to the actions of a user performing that unit exercise Est in a supervised environment. Then, the tuning unit 121 of the server 101 generates an individualized optimization model M2 through additional learning using the generated tuning data.

[0111] When the exercise in the unsupervised environment is completed, the user terminal 102 transmits the recorded unsupervised environment video to the server 101 via the network. The server 101 uses the first video splitting unit 14 and the second video splitting unit 15 to generate unit exercise videos from the unsupervised environment video received from the user terminal 102, representing the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the unsupervised environment.

[0112] Then, the second selection unit 12 of the server 101 uses the individual optimization model M2 generated by the tuning unit 121 to select a malfunction class corresponding to the behavior of the target user TU when performing each unit exercise Est that constitutes exercise Es in a guided environment. The advice generation unit 13 of the server 101 refers to the malfunction class selected by the second selection unit 12 and generates advice to present to the target user TU. The server 101 sends the advice generated by the advice generation unit 13 to the user terminal 102, and the user terminal 102 presents the received advice to the target user TU.

[0113] The server 101 may further include the extraction / summarization unit 19, the explanatory video generation unit 21, and the synthesis unit 22 described above. This makes it possible for the server 101 to generate the explanatory text, explanatory video, and multimedia content described above.

[0114] (First variation of the exercise support system) A first modified example of the exercise support system 100 (hereinafter referred to as exercise support system 100A) will be described with reference to Figure 15. Figure 15 is a block diagram showing the configuration of exercise support system 100A. In the block diagram shown in Figure 15, each block of each device in exercise support system 100A is arranged according to the processing flow. Therefore, blocks that are used multiple times at different timings are shown multiple times.

[0115] Server 101 comprises a first selection unit 11, an audio splitting unit 16, an audio analysis unit 17, an extraction unit 18, and a tuning unit 121. User terminal 102 is configured for recording video of an environment with instruction and video of an environment without instruction, and comprises a second selection unit 12, an advice generation unit 13, a first video splitting unit 14, and a second video splitting unit 15. Instructor terminal 103 is configured for recording audio of an environment with instruction.

[0116] The exercise support system 100A operates as follows:

[0117] When the exercise in the supervised environment is completed, the user terminal 102 uses the first video splitting unit 14 and the second video splitting unit 15 to generate unit exercise videos from the recorded supervised environment video, representing the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the supervised environment. The user terminal 102 then transmits the generated unit exercise videos to the server 101.

[0118] Furthermore, when the exercise in the supervised environment is completed, the instructor terminal 103 transmits the recorded audio of the supervised environment to the server 101 via the network. The server 101 uses the audio splitting unit 16, audio analysis unit 17, extraction unit 18, and first selection unit 11 to select a malfunction class from the supervised environment audio received from the instructor terminal 103 that corresponds to the behavior of the target user TU who performs each unit exercise Est that constitutes exercise Es in the supervised environment.

[0119] The tuning unit 121 of server 101 then generates tuning data for each unit exercise Est that makes up exercise Es, by combining a unit exercise video (obtained from user terminal 102) that represents the user's actions when performing that unit exercise Est in a supervised environment, and a malfunction class (generated by itself) that corresponds to the user's actions when performing that unit exercise Est in a supervised environment. The tuning unit 121 of server 101 then generates an individualized optimization model M2 through additional learning using the generated tuning data. Server 101 then transmits the generated individualized optimization model M2 to user terminal 102.

[0120] When the exercise in the unsupervised environment is completed, the user terminal 102 uses the first video splitting unit 14 and the second video splitting unit 15 to generate unit exercise videos from the recorded unsupervised environment video, representing the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the unsupervised environment.

[0121] Then, the second selection unit 12 of the user terminal 102 uses the individual optimization model M2 received from the server 101 to select a malfunction class corresponding to the behavior of the target user TU when performing each unit exercise Est that constitutes exercise Es in a guided environment. The advice generation unit 13 of the user terminal 102 refers to the malfunction class selected by the second selection unit 12 and generates advice to present to the target user TU. The user terminal 102 presents the advice generated by the advice generation unit 13 to the target user TU.

[0122] The server 101 may further include the extraction / summarization unit 19, the explanatory video generation unit 21, and the synthesis unit 22 described above. This makes it possible for the server 101 to generate the explanatory text, explanatory video, and multimedia content described above.

[0123] According to the exercise support system 100A, machine learning to generate an individually optimized model M2 is performed on the server 101. Therefore, exercise support can be provided without placing an excessive load on the user terminal 102. Furthermore, if there are multiple users, the individually optimized model M2 corresponding to each user can be centrally managed on the server 101.

[0124] (A second variation of the exercise support system) A second modified example of the exercise support system 100 (hereinafter referred to as exercise support system 100B) will be described with reference to Figure 16. Figure 16 is a block diagram showing the configuration of exercise support system 100B. In the block diagram shown in Figure 16, each block of each device in exercise support system 100B is arranged according to the processing flow. Therefore, blocks that are used multiple times at different timings are shown multiple times.

[0125] Server 101 includes a first selection unit 11. User terminal 102 is configured for recording video of an instructional environment and video of an uninstructed environment, and includes a second selection unit 12, an advice generation unit 13, a first video splitting unit 14, a second video splitting unit 15, and a tuning unit 121. Instructor terminal 103 is configured for recording audio of an instructional environment, and includes a first selection unit 11, an audio splitting unit 16, an audio analysis unit 17, and an extraction unit 18.

[0126] The exercise support system 100 operates as follows:

[0127] When the exercise in the supervised environment is completed, the user terminal 102 uses the first video splitting unit 14 and the second video splitting unit 15 to generate unit exercise videos from the recorded supervised environment video, representing the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the supervised environment.

[0128] Furthermore, when the exercise in the supervised environment is completed, the instructor terminal 103 uses the audio splitting unit 16, the audio analysis unit 17, and the extraction unit 18 to extract instructional content from the recorded audio of the supervised environment regarding the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the supervised environment. The instructor terminal 103 transmits the extracted instructional content to the server 101. The server 101 uses the first selection unit 11 to select a malfunction class from the received instructional content that corresponds to the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the supervised environment. The server 101 and the instructor terminal 103 transmit the selected malfunction class to the user terminal 102.

[0129] Then, the tuning unit 121 of the user terminal 102 generates tuning data for each unit exercise Est that makes up exercise Es, by combining a unit exercise video (generated by itself) that represents the user's actions when performing that unit exercise Est in a supervised environment, and a malfunction class (received from server 101) that corresponds to the user's actions when performing that unit exercise Est in a supervised environment.Then, the tuning unit 121 of the user terminal 102 generates an individualized optimization model M2 through additional learning using the generated tuning data.

[0130] When the exercise in the unsupervised environment is completed, the user terminal 102 uses the first video splitting unit 14 and the second video splitting unit 15 to generate unit exercise videos from the recorded unsupervised environment video, representing the actions of the target user TU who performs each unit exercise Est that constitutes exercise Es in the unsupervised environment.

[0131] Then, the second selection unit 12 of the user terminal 102 uses the generated individual optimization model M2 to select a malfunction class corresponding to the behavior of the target user TU when performing each unit exercise Est that constitutes exercise Es in a supervised environment. The advice generation unit 13 of the user terminal 102 refers to the malfunction class selected by the second selection unit 12 and generates advice to present to the target user TU. The user terminal 102 presents the advice generated by the advice generation unit 13 to the target user TU.

[0132] Furthermore, the instructor terminal 103 may also include the extraction / summarization unit 19 described above. This makes it possible for the instructor terminal 103 to generate the explanatory text described above. In addition, the user terminal 102 may further include an explanatory video generation unit 21 and a synthesis unit 22. This makes it possible for the server 101 to generate the explanatory video and multimedia content described above.

[0133] According to the exercise support system 100B, machine learning to generate an individually optimized model M2 is performed on the user terminal 102. Therefore, exercise support can be provided without placing an excessive load on the server 101. Furthermore, if there are multiple users, the individually optimized model M2 corresponding to each user can be managed in a distributed manner on each user's user terminal 102.

[0134] [Examples of implementation using software] Some or all of the functions of the exercise support devices 1, 1A, 1B and the devices constituting the exercise support system 100 (hereinafter also referred to as "the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0135] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as Computer C) is shown in Figure 17. Figure 17 is a block diagram showing the hardware configuration of Computer C, which functions as each of the above devices.

[0136] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0137] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0138] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0139] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0140] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0141] [Additional Note 1] This disclosure includes the technologies described in the following appendices. However, this disclosure is not limited to the technologies described in the following appendices and is subject to various modifications.

[0142] (Note 1) A first selection means that, by referring to the instruction content given to a target user performing an exercise in a supervised environment, selects a malfunction class corresponding to the said instruction content from among multiple malfunction classes, A second selection means selects a malfunction class corresponding to the actions of the target user performing the exercise in an unsupervised environment from among the malfunction classes selected by the first selection means, by referring to a video representing the actions of the target user. The system includes an advice generation means that generates advice to be presented to the target user performing the exercise in the unsupervised environment, by referring to the malfunction class selected by the second selection means. Exercise support device.

[0143] (Note 2) The second selection means is, The system further comprises a tuning means that generates an individually optimized model from a general model whose input is a video representing the actions of a user performing the exercise and whose output is one of the multiple error classes selected by the first selection means, by machine learning using tuning data which includes a video representing the actions of the target user performing the exercise in the supervised environment and a class of malfunctions selected by the first selection means, wherein the input is a video representing the actions of a user performing the exercise and whose output is one of the class of malfunctions selected by the first selection means, When a video representing the actions of the target user performing the exercise in the unsupervised environment is input to the individual optimization model, the malfunction class output from the individual optimization model is selected as the malfunction class corresponding to that action. The exercise support device described in Appendix 1.

[0144] (Note 3) The aforementioned exercise consists of repetitions of unit exercises, The first selection means, for each unit exercise, refers to the instruction content given to the target user while the exercise is being performed in the supervised environment, and selects a malfunction class corresponding to the malfunction of the target user in that unit exercise. The second selection means, for each unit exercise, refers to a video representing the actions of the target user performing the exercise in the unsupervised environment and selects a malfunction class corresponding to the target user's malfunction in that unit exercise. An exercise support device as described in either Appendix 1 or 2.

[0145] (Note 4) The system further comprises extraction means for extracting instruction content for the target user performing the exercise in the supervised environment from the content of speech of the instructor who is instructing the target user performing the exercise in the supervised environment, The first selection means selects a malfunction class corresponding to the instruction content extracted by the extraction means from among the plurality of malfunction classes, An exercise support device as described in any of the appendices 1 to 3.

[0146] (Note 5) The system further includes an explanatory text generation means that generates an explanatory text to be presented to the target user performing the exercise in the unsupervised environment by summarizing the content of the speech of the instructor who is guiding the target user while the target user is performing the exercise in the supervised environment. An exercise support device as described in any of the appendices 1 to 4.

[0147] (Note 6) An extraction means for extracting instruction content for the target user performing the exercise in the aforementioned supervised environment from the content of speech of an instructor who is instructing the target user performing the exercise in the aforementioned supervised environment, The system further comprises: a video representing the actions of the target user performing the exercise in the supervised environment; and an explanatory video generation means that generates an explanatory video to be presented to the target user performing the exercise in the unsupervised environment, by referring to the instructional content extracted by the extraction means. An exercise support device as described in any of the appendices 1 to 5.

[0148] (Note 7) An exercise support system including a server and a terminal, The aforementioned server, A first selection means that, by referring to the instruction content given to a target user performing an exercise in a supervised environment, selects a malfunction class corresponding to the said instruction content from among multiple malfunction classes, The system includes a tuning means that generates an individually optimized model from a general model whose input is a video representing the actions of a user performing the exercise and whose output is one of the multiple error classes, by machine learning using tuning data that includes a video representing the actions of the target user performing the exercise in the supervised environment and error classes selected by the first selection means, wherein the input is a video representing the actions of a user performing the exercise and whose output is one of the error classes selected by the first selection means, The aforementioned terminal is A second selection means that, by referring to a video representing the actions of the target user performing the exercise in an unsupervised environment, selects a malfunction class corresponding to said actions from among the malfunction classes selected by the first selection means, wherein the second selection means selects the malfunction class output from the individual optimization model when a video representing the actions of the target user performing the exercise in an unsupervised environment is input to the individual optimization model as the malfunction class corresponding to said actions, The system includes an advice generation means that generates advice to be presented to the target user performing the exercise in the unsupervised environment, by referring to the malfunction class selected by the second selection means. Exercise support system.

[0149] (Note 8) An exercise support system including a server and a terminal, The aforementioned server, The system includes a first selection means that refers to the instruction content given to a target user performing an exercise in a supervised environment and selects a malfunction class corresponding to that instruction content from among multiple malfunction classes. The aforementioned terminal is A tuning means that generates an individually optimized model from a general model whose input is a video representing the actions of a user performing the exercise and whose output is one of the multiple error classes selected by the first selection means, by machine learning using tuning data which includes a video representing the actions of the target user performing the exercise in the supervised environment and a class of malfunctions selected by the first selection means, wherein the input is a video representing the actions of a user performing the exercise and whose output is one of the class of malfunctions selected by the first selection means, A second selection means that, by referring to a video representing the actions of the target user performing the exercise in an unsupervised environment, selects a malfunction class corresponding to said actions from among the malfunction classes selected by the first selection means, wherein the second selection means selects the malfunction class output from the individual optimization model when a video representing the actions of the target user performing the exercise in an unsupervised environment is input to the individual optimization model as the malfunction class corresponding to said actions, The system includes an advice generation means that generates advice to be presented to the target user performing the exercise in the unsupervised environment, by referring to the malfunction class selected by the second selection means. Exercise support system.

[0150] (Note 9) At least one processor performs a first selection process in which it refers to the instruction content for a target user performing an exercise in an instructional environment and selects a malfunction class corresponding to that instruction content from among several malfunction classes. A second selection process in which at least one of the aforementioned processors refers to a video representing the actions of the target user performing the exercise in an unsupervised environment and selects a malfunction class corresponding to said actions from among the malfunction classes selected in the first selection process, Any of the at least one of the processors includes an advice generation process that, with reference to the malfunction class selected in the second selection process, generates advice to be presented to the target user performing the exercise in the unsupervised environment. Exercise support methods.

[0151] (Note 10) A program for operating a computer as an exercise support device as described in any of the appendices 1 to 6, wherein the computer functions as one of the means described above. Exercise support program.

[0152] [Additional Note 2] This disclosure includes the technologies described in the following appendices. However, this disclosure is not limited to the technologies described in the following appendices and is subject to various modifications.

[0153] (Note 1) Equipped with at least one processor, The aforementioned at least one processor is A first selection process involves referring to the instruction content given to a target user performing an exercise in a supervised environment, and selecting a malfunction class corresponding to that instruction content from among several malfunction classes. A second selection process involves referring to a video representing the actions of the target user performing the exercise in an unsupervised environment, and selecting a malfunction class corresponding to those actions from among the malfunction classes selected in the first selection process. An advice generation process is performed which, by referring to the malfunction class selected by the second selection process, generates advice to be presented to the target user performing the exercise in the unsupervised environment. Exercise support device.

[0154] (Note 2) The aforementioned at least one processor is By machine learning using tuning data including video representing the actions of the target user performing the exercise in the supervised environment and the malfunction class selected by the first selection process, a tuning process is further performed to generate an individually optimized model from a general model whose input is video representing the actions of the user performing the exercise and whose output is one of the multiple malfunction classes, where the input is video representing the actions of the user performing the exercise and the output is one of the malfunction classes selected by the first selection process. In the second selection process, the at least one processor selects the malfunction class output from the individual optimization model when it receives a video representing the actions of the target user performing the exercise in the unsupervised environment, as the malfunction class corresponding to said actions. The exercise support device described in Appendix 1.

[0155] (Note 3) The aforementioned exercise consists of repetitions of unit exercises, In the first selection process, the at least one processor, for each unit exercise, refers to the instruction content given to the target user while the exercise is being performed in the supervised environment, and selects a malfunction class corresponding to the malfunction of the target user in that unit exercise. In the second selection process, the at least one processor, for each unit exercise, refers to a video representing the actions of the target user performing the exercise in the unsupervised environment and selects a malfunction class corresponding to the target user's malfunction in that unit exercise. The exercise support device described in Appendix 1.

[0156] (Note 4) The aforementioned at least one processor is Further extraction processing is performed to extract the content of instruction given to the target user while the target user is performing the exercise in the supervised environment, from the content of the speech of the instructor who is giving the instruction to the target user while the target user is performing the exercise in the supervised environment. In the first selection process, the at least one processor refers to the instruction content extracted by the extraction process and selects a malfunction class corresponding to the instruction content from among the plurality of malfunction classes. The exercise support device described in Appendix 1.

[0157] (Note 5) The aforementioned at least one processor is Further, an explanatory text generation process is performed to generate explanatory text to be presented to the target user performing the exercise in the unsupervised environment, by summarizing the content of the utterances of the instructor who is guiding the target user while they are performing the exercise in the supervised environment. The exercise support device described in Appendix 1.

[0158] (Note 6) The aforementioned at least one processor is An extraction process to extract the content of instruction given to the target user while the target user is performing the exercise in the supervised environment, from the content of the speech of the instructor who is giving the instruction to the target user while the target user is performing the exercise in the supervised environment, Further, the process involves generating an explanatory video to be presented to the target user performing the exercise in the unsupervised environment, by referencing a video showing the actions of the target user performing the exercise in the supervised environment, and the instructional content extracted by the extraction process. The exercise support device described in Appendix 1.

[0159] (Note 7) An exercise support system comprising a server having at least one processor and a terminal having at least one processor, At least one processor of the server is A first selection process involves referring to the instruction content given to a target user performing an exercise in a supervised environment, and selecting a malfunction class corresponding to that instruction content from among several malfunction classes. The following is performed: a tuning process is performed using machine learning with tuning data including video representing the actions of the target user performing the exercise in the supervised environment and the malfunction class selected by the first selection process, thereby generating an individually optimized model from a general model whose input is video representing the actions of the user performing the exercise and whose output is one of the multiple malfunction classes, where the input is video representing the actions of the user performing the exercise and whose output is one of the malfunction classes selected by the first selection process, At least one processor of the terminal is A second selection process that refers to a video representing the actions of the target user performing the exercise in an unsupervised environment and selects a malfunction class corresponding to said actions from among the malfunction classes selected by the first selection process, wherein the second selection process selects the malfunction class output from the individual optimization model when a video representing the actions of the target user performing the exercise in an unsupervised environment is input to the individual optimization model as the malfunction class corresponding to said actions, An advice generation process is performed which, by referring to the malfunction class selected by the second selection process, generates advice to be presented to the target user performing the exercise in the unsupervised environment. Exercise support system.

[0160] (Note 8) An exercise support system comprising a server having at least one processor and a terminal having at least one processor, At least one processor of the server is The system performs a first selection process that refers to the instruction content given to the target user performing the exercise in a supervised environment, and selects the malfunction class corresponding to that instruction content from among multiple malfunction classes. At least one processor of the terminal is A tuning process that generates an individually optimized model from a general model whose input is a video representing the actions of a user performing the exercise and whose output is one of the multiple error classes selected by the first selection process, using machine learning with tuning data that includes a video representing the actions of the target user performing the exercise in the supervised environment and error classes selected by the first selection process, wherein the input is a video representing the actions of a user performing the exercise and whose output is one of the error classes selected by the first selection process, A second selection process that refers to a video representing the actions of the target user performing the exercise in an unsupervised environment and selects a malfunction class corresponding to said actions from among the malfunction classes selected by the first selection process, wherein the second selection process selects the malfunction class output from the individual optimization model when a video representing the actions of the target user performing the exercise in an unsupervised environment is input to the individual optimization model as the malfunction class corresponding to said actions, An advice generation process is performed which, by referring to the malfunction class selected by the second selection process, generates advice to be presented to the target user performing the exercise in the unsupervised environment. Exercise support system.

[0161] (Note 9) At least one processor performs a first selection process in which it refers to the instruction content for a target user performing an exercise in an instructional environment and selects a malfunction class corresponding to that instruction content from among several malfunction classes. A second selection process in which at least one of the aforementioned processors refers to a video representing the actions of the target user performing the exercise in an unsupervised environment and selects a malfunction class corresponding to said actions from among the malfunction classes selected by the first selection process, Any of the at least one of the processors includes an advice generation process that, with reference to the malfunction class selected by the second selection process, generates advice to be presented to the target user performing the exercise in the unsupervised environment. Exercise support methods.

[0162] (Note 10) A computer-readable, non-temporary, tangible recording medium, on which a program for operating a computer as the exercise support device described in Appendix 1 is recorded, the exercise support program for causing the computer to function as described above. [Explanation of symbols]

[0163] 1,1A,1B Exercise support device 11. First Selection Unit (First Selection Means) 12. Second Selection Unit (Second Selection Means) 13. Advice generation unit (advice generation means) 14. First video split section 15. Second video split section 16. Audio splitting section 17. Voice Analysis Unit 18 Extraction part (extraction means) 19 Extraction / Summary Unit (Extraction means, summarization means) 21 Explanation Video Generation Unit 22 Synthesis section 100, 100A, 100B Exercise Support System 101 Server (Server) 102 User terminal (terminal) 103 Instructor terminal

Claims

1. A first selection means that, by referring to the instruction content given to a target user performing an exercise in a supervised environment, selects a malfunction class corresponding to the said instruction content from among multiple malfunction classes, A second selection means refers to a video representing the actions of the target user performing the exercise in an unsupervised environment, and selects a malfunction class corresponding to those actions from among the malfunction classes selected by the first selection means. The system includes an advice generation means that generates advice to be presented to the target user performing the exercise in the unsupervised environment, by referring to the malfunction class selected by the second selection means. Exercise support device.

2. The second selection means is, The system further comprises a tuning means that generates an individually optimized model from a general model whose input is a video representing the actions of a user performing the exercise and whose output is one of the multiple error classes selected by the first selection means, by machine learning using tuning data which includes a video representing the actions of the target user performing the exercise in the supervised environment and a class of malfunctions selected by the first selection means, wherein the input is a video representing the actions of a user performing the exercise and whose output is one of the class of malfunctions selected by the first selection means, When a video representing the actions of the target user performing the exercise in the unsupervised environment is input to the individual optimization model, the malfunction class output from the individual optimization model is selected as the malfunction class corresponding to that action. The exercise support device according to claim 1.

3. The aforementioned exercise consists of repetitions of unit exercises, The first selection means, for each unit exercise, selects a malfunction class corresponding to the malfunction of the target user in that unit exercise by referring to the instruction content given to the target user while the exercise is being performed in the instructional environment. The second selection means, for each unit exercise, refers to a video representing the actions of the target user performing the exercise in the unsupervised environment and selects a malfunction class corresponding to the target user's malfunction in that unit exercise. The exercise support device according to claim 1.

4. The system further comprises extraction means for extracting instruction content for the target user performing the exercise in the supervised environment from the content of speech of the instructor who is instructing the target user performing the exercise in the supervised environment, The first selection means selects a malfunction class corresponding to the instruction content extracted by the extraction means from among the plurality of malfunction classes, The exercise support device according to claim 1.

5. The system further includes a summarization means for generating explanatory text to be presented to the target user performing the exercise in the unsupervised environment, by summarizing the content of speech given by the instructor guiding the target user while they are performing the exercise in the supervised environment. The exercise support device according to claim 1.

6. An extraction means for extracting instruction content for the target user performing the exercise in the aforementioned supervised environment from the content of speech of an instructor who is instructing the target user performing the exercise in the aforementioned supervised environment, The system further comprises: a video representing the actions of the target user performing the exercise in the supervised environment; and an explanatory video generation means that generates an explanatory video to be presented to the target user performing the exercise in the unsupervised environment, by referring to the instructional content extracted by the extraction means. The exercise support device according to claim 1.

7. An exercise support system including a server and a terminal, The aforementioned server, A first selection means that, by referring to the instruction content given to a target user performing an exercise in a supervised environment, selects a malfunction class corresponding to the said instruction content from among multiple malfunction classes, The system includes a tuning means that generates an individually optimized model from a general model whose input is a video representing the actions of a user performing the exercise and whose output is one of the multiple error classes selected by the first selection means, by machine learning using tuning data which includes a video representing the actions of the target user performing the exercise in the supervised environment and a class of malfunctions selected by the first selection means, wherein the input is a video representing the actions of a user performing the exercise and whose output is one of the class of malfunctions selected by the first selection means, The aforementioned terminal is A second selection means that, by referring to a video representing the actions of the target user performing the exercise in an unsupervised environment, selects a malfunction class corresponding to said actions from among the malfunction classes selected by the first selection means, wherein the second selection means selects the malfunction class output from the individual optimization model when a video representing the actions of the target user performing the exercise in an unsupervised environment is input to the individual optimization model as the malfunction class corresponding to said actions, The system includes an advice generation means that generates advice to be presented to the target user performing the exercise in the unsupervised environment, by referring to the malfunction class selected by the second selection means. Exercise support system.

8. An exercise support system including a server and a terminal, The aforementioned server, The system includes a first selection means that, by referring to the instruction content given to a target user performing an exercise in a supervised environment, selects a malfunction class corresponding to the said instruction content from among a plurality of malfunction classes. The aforementioned terminal is A tuning means that generates an individually optimized model from a general model whose input is a video representing the actions of a user performing the exercise and whose output is one of the multiple error classes selected by the first selection means, by machine learning using tuning data which includes a video representing the actions of the target user performing the exercise in the supervised environment and a class of malfunctions selected by the first selection means, wherein the input is a video representing the actions of a user performing the exercise and whose output is one of the class of malfunctions selected by the first selection means, A second selection means that, by referring to a video representing the actions of the target user performing the exercise in an unsupervised environment, selects a malfunction class corresponding to said actions from among the malfunction classes selected by the first selection means, wherein the second selection means selects the malfunction class output from the individual optimization model when a video representing the actions of the target user performing the exercise in an unsupervised environment is input to the individual optimization model as the malfunction class corresponding to said actions, The system includes an advice generation means that generates advice to be presented to the target user performing the exercise in the unsupervised environment, by referring to the malfunction class selected by the second selection means. Exercise support system.

9. At least one processor performs a first selection process in which it refers to the instruction content for a target user performing an exercise in an instructional environment and selects a malfunction class corresponding to that instruction content from among several malfunction classes. A second selection process in which at least one of the aforementioned processors refers to a video representing the actions of the target user performing the exercise in an unsupervised environment and selects a malfunction class corresponding to said actions from among the malfunction classes selected in the first selection process, The at least one of the processors includes an advice generation process that, with reference to the malfunction class selected in the second selection process, generates advice to be presented to the target user performing the exercise in the unsupervised environment. Exercise support methods.

10. An exercise support program for operating a computer as an exercise support device according to any one of claims 1 to 6, wherein the computer is made to function as each of the means described above. Exercise support program.

Images