Teaching Support System

Abstract

[Problem] To easily evaluate physical exercise. [Solution] Provided is a guidance support system which supports, by a supporter, guidance given to a user who performs physical exercise, and comprises: a reference value storage unit which stores a reference value related to the position of at least one part of a body or a portion of a tool; a part and portion specifying unit which specifies the part or the portion by analyzing an image of the user's physical exercise; an evaluation unit which determines an evaluation value of the physical exercise by comparing the position of the part or the portion in the image with the reference value; and a supporter terminal having an imaging function for capturing an image of the user's physical exercise and a display function for displaying the evaluation value.

Description

[Technical Field]

[0001] The present invention relates to a teaching support system. [Background technology]

[0002] Physical education instruction and coaching are widely used in a variety of areas, such as sports competitions, strength training, and rehabilitation.

[0003] Instruction and coaching are often provided by experienced and knowledgeable supporters (including instructors, coaches, physical therapists, care workers, etc.), and are useful for users performing physical exercise because they can hear objective opinions. However, they are often based on the supporter's intuition and experience, and sometimes the supporter is unable to put it into words, which reduces the effectiveness of the instruction and coaching.

[0004] Patent Document 1 discloses an invention that supports instruction by comparing the user's movements with the instructor's movements by juxtaposing them side by side, as a method of teaching sports skills. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-313362 Summary of the Invention [Problem to be solved by the invention]

[0006] However, while the technology in Patent Document 1 clearly identifies the difference between the user and the instructor, it relies on the instructor for knowledge of how to narrow that gap, what kind of training is required, etc. Furthermore, because video is taken and compared, it is not possible to provide guidance immediately after or during physical exercise, which limits its effectiveness.

[0007] The present invention has been made in consideration of this background, and aims to provide a technology that can support exercise by easily detecting and evaluating the movements of body parts and equipment from images of the user's physical exercise, and by providing advice aimed at further improving skills and improving the effectiveness of training, rehabilitation, etc. [Means for solving the problem]

[0008] The present invention is an instruction support system that supports a supporter in providing instruction to a user performing physical exercise, and is characterized by comprising: a reference value storage unit that stores a reference value related to the position of at least one body part or tool part; a part / portion identification unit that analyzes an image of the user's physical exercise to identify the part or part; and an evaluation unit that compares the position of the part or part in the image with the reference value to determine an evaluation value of the physical exercise, and a supporter terminal that has an imaging function for capturing an image of the user's physical exercise and a display function for displaying the evaluation value.

[0009] Other problems and solutions disclosed in this application will be made clear in the section on preferred embodiments of the invention and the drawings. [Effects of the Invention]

[0010] According to the present invention, it is possible to easily evaluate the skills and effects of physical exercise using tools and provide support that contributes to improving skills and effects. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a physical exercise support system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of a user terminal 10. [Figure 3] FIG. 2 is a diagram illustrating an example of the software configuration of the user terminal 10. [Figure 4] 10 is a diagram showing an example of the structure of physical information stored in a physical information storage unit 130. FIG. [Figure 5]10 is a diagram showing an example of the structure of a rating request that a rating request sending unit 112 sends to a server device 20. FIG. [Figure 6] 10 is a diagram showing an example of the structure of evaluation information received by an evaluation information receiving unit 113 from a server device 20. FIG. [Figure 7] 10 is a diagram showing an example of the structure of a remedy request that a remedy request sending unit 116 sends to the server device 20. FIG. [Figure 8] 10 is a diagram showing an example of the configuration of a reference value request that a reference value request sending unit 119 sends to a server device 20. FIG. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of improvement measure information. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device 20. [Figure 11] FIG. 2 is a diagram illustrating an example of the software configuration of the server device 20. [Figure 12] 10 is a diagram showing an example of the configuration of image information stored in an image data storage unit 231. FIG. [Figure 13] 10 is a diagram showing an example of the configuration of reference information stored in a reference information storage unit 232. FIG. [Figure 14] 10 is a diagram showing an example of the configuration of evaluation condition information stored in an evaluation condition information storage unit 233. FIG. [Figure 15] 10 is a diagram showing an example of the configuration of improvement measure information stored in an improvement measure information storage unit 234. FIG. [Figure 16] 10 is a diagram showing an example of the structure of group analysis information stored in a group analysis information storage unit 235. FIG. [Figure 17] FIG. 10 is a diagram showing a flow of processing executed in the training support server of the present embodiment. [Figure 18] FIG. 10 is a diagram showing an example of a video evaluation screen in weightlifting mode. [Figure 19] FIG. 10 is another diagram showing an example of the video evaluation screen in the weightlifting mode. [Figure 20] FIG. 10 is another diagram showing an example of the video evaluation screen in the weightlifting mode. [Figure 21]FIG. 10 is another diagram showing an example of the video evaluation screen in the weightlifting mode. [Figure 22] FIG. 10 is another diagram showing an example of the video evaluation screen in the weightlifting mode. [Figure 23] 10 is another diagram showing an example of the video evaluation screen in the squat mode presented on the user terminal 10. FIG. [Figure 24] 10 is a diagram showing an example of an evaluation screen for a video in group training, which is presented on the supporter terminal 40. FIG. [Figure 25] 10 is another diagram showing an example of an evaluation screen for a video in group training, which is presented on supporter terminal 40. FIG. [Figure 26] 10 is another diagram showing an example of an evaluation screen for a video in group training, which is presented on supporter terminal 40. FIG. [Figure 27] FIG. 10 is another diagram showing an example of an image viewed by a supporter through the supporter terminal 40. DETAILED DESCRIPTION OF THE INVENTION

[0012] The contents of the embodiment of the present invention will be described below: A training support server according to the embodiment of the present invention has the following configuration.

[0013] [Item 1] An instruction support system that supports an assistant in providing instruction to a user who is performing physical exercise, a reference value storage unit configured to store a reference value relating to the position of at least one body part or tool part; a part or region specifying unit that analyzes an image of the user's physical movement and specifies the part or region; an evaluation unit that compares the position of the site or portion in the image with the reference value to determine an evaluation value of the physical movement; Equipped with an imaging function for capturing an image of the user's physical exercise; a supporter terminal having a display function for displaying the evaluation value; A teaching support system comprising: [Item 2] The supporter terminal is a wearable computer; The teaching support system according to claim 1, [Item 3] There are a plurality of the reference values, a reference value determination unit that presents one or more reference values ​​to the user based on at least one of information of the user's purpose, physical information, and the evaluation value; 3. The training support system according to claim 1, further comprising: [Item 4] a reference value selection information transmission unit for allowing the user to select from candidate reference values; 4. The training support system according to claim 1, further comprising: [Item 5] An instruction support method in which a supporter supports instruction given to a user performing physical exercise, comprising: a reference value storage step of storing a reference value relating to the position of at least one body part or tool part; a part or region specifying step of analyzing an image of the user's physical movement to specify the part or region; an evaluation step of determining an evaluation value of the body movement by comparing the position of the site or portion in the image with the reference value; Equipped with an imaging function for capturing an image of the user's physical exercise; a supporter terminal having a display function for displaying the evaluation value; A teaching support method comprising:

[0014] The instruction support server according to one embodiment of the present invention is designed to assist a supporter in providing instruction to a user during physical exercise (including exercise using tools) by evaluating the user's physical movements and the movements of the tools.

[0015] Physical exercises performed by users include, for example, gymnastics, fitness, walking, running, yoga, bodyweight training, rehabilitation, and other exercises that do not use equipment; sports that use only a ball as an equipment, such as soccer, volleyball, and basketball; sports that use a ball and other equipment, such as baseball, tennis, table tennis, and golf; sports that use equipment other than a ball, such as kendo and fencing; physical exercises that use equipment, such as juggling; physical exercises that use barbells or training equipment or machines; various types of rehabilitation that use equipment; movements of the person being cared for using equipment such as a cane; playing musical instruments; etc., but are not limited to these.

[0016] The training support server of this embodiment identifies the body, body parts, tools, and tool parts from images (which may be still images or moving images, but are assumed to be moving images in this embodiment) of a user performing physical exercise, and evaluates the movement of the body and tool based on the absolute position of the part or part and the relative positional relationship between multiple different parts and parts. Note that the movement of the tool may also be evaluated based on the relative positional relationship between multiple different body parts and tool parts.

[0017] 1 is a diagram showing an example of the overall configuration of a training support server according to this embodiment. As shown in the figure, the training support server according to this embodiment is configured to include a user terminal 10 and a server device 20. It may further include an imaging terminal 50. The user terminal 10, the server device 20, and the imaging terminal 50 are communicably connected to each other via a communication network 30. The communication network 30 is, for example, the Internet or a LAN (Local Area Network), and is constructed using a public telephone line network, a dedicated telephone line network, a mobile phone line network, Ethernet (registered trademark), a wireless communication path, etc.

[0018] ==User terminal 10== The user terminal 10 is a computer operated by a user or their supporter performing physical exercise. The user terminal 10 may be, for example, a smartphone, tablet computer, personal computer, or wearable computer. The user terminal 10 is equipped with an imaging device such as a camera, which can capture images of the user's body during exercise. In this embodiment, video images of the user's body during exercise are transmitted from the user terminal 10 to the server device 20. The user terminal 10 may be a computer such as a glasses-type, contact lens-type, hat-type, or HMD (head-mounted display) that has an imaging function that captures images close to the user's field of view and an output function that allows the user to view text, images, videos, etc. using a virtual image projection method, a retinal projection method, or other methods. The imaging function and the output function may be performed by separate terminals. The user terminal 10 may also function as both the imaging terminal 50 and the output terminal 60. Alternatively, the imaging function of the user terminal 10 may also function as the imaging terminal 50. Alternatively, the output function of the supporter terminal 40 may also function as the output terminal 60. Furthermore, although only one user terminal 10 is shown in FIG. 1, it goes without saying that there may be a plurality of user terminals.

[0019] ==Server Device 20== The server device 20 is a computer that evaluates the movements of the body and tools. The server device 20 is, for example, a workstation, a personal computer, or a virtual computer logically realized by cloud computing. The server device 20 receives video images captured by the user terminal 10, analyzes the received video images, and evaluates the body movement. The server device 20 also proposes measures to improve the body movement. Details of the evaluation of the body movement and the proposal of measures to improve the body movement will be described later.

[0020] ==Supporter Terminal 40== The supporter terminal 40 is a computer operated by a person (supporter) such as a trainer, physical therapist, or caregiver who provides guidance, instruction, explanation, and support to a user performing physical exercise. The supporter terminal 40 may be, for example, a smartphone, tablet computer, personal computer, or wearable computer. The supporter terminal 40 may be a computer such as a glasses-type, contact lens-type, hat-type, or HMD (head-mounted display) that has an imaging function for capturing images close to the supporter's field of vision and an output function for visualizing text, images, videos, etc. using a virtual image projection method, a retinal projection method, or other methods. The imaging function and the output function may be performed by separate terminals. The supporter terminal 40 may also function as both the imaging terminal 50 and the output terminal 60. Alternatively, the imaging function of the supporter terminal 40 may also function as the imaging terminal 50, or the output function of the supporter terminal 40 may also function as the output terminal 60. Although only one supporter terminal 40 is shown in FIG. 1 , it goes without saying that multiple supporter terminals 40 may be used.

[0021] ==Imaging terminal 50== The imaging terminal 50 is a computer or a camera with communication capabilities that captures images of the user and is installed or attached to a location where the user performs physical exercise. The imaging terminal 50 may be, for example, a smartphone, tablet computer, personal computer, or wearable computer. The imaging terminal 50 is equipped with an imaging device such as a camera, which can capture images of the user's body during exercise. In this embodiment, video images of the user's body during exercise are transmitted from the imaging terminal 50 to the server device 20. The user, their supporter, or a business operator using the training support server may input the captured image data of the user stored in the imaging terminal 50 directly to the server device 20 or via the communication network 30. Although only one imaging terminal 50 is shown in FIG. 1 , it goes without saying that multiple imaging terminals 50 may be used. Furthermore, imaging terminals 50 may be installed in multiple locations and capture images of the subject simultaneously or sequentially. Furthermore, the imaging terminal 50 may also serve as both the user terminal 10 and the output terminal 60, or may perform the imaging functions of the user terminal 10 and the supporter terminal 40.

[0022] ==Output terminal 60== The output terminal 60 outputs information under the control of the server device 20 via the communication network 30. The output terminal 60 may be, for example, a smartphone, tablet computer, personal computer, or wearable computer. Alternatively, a computer may be connected to an output device such as a display, and the computer and the server device 20 may communicate with each other to output information to the output terminal 60 in response to instructions from the server device 20. The output terminal 60 may also include an audio output device, and the user may wear an audio output device such as headphones, earphones, or a neck speaker to output information as audio. Information may also be output to a user or a support person via an interface such as a glasses-type, contact lens-type, hat-type, or other output terminal, an HMD (head-mounted display), or a BMI (brain-machine interface), which utilizes brain activity such as electroencephalography to input text, images, videos, etc. by directly stimulating the brain without using sensory organs. 1, there may be more than one output terminal 60, and the video output, audio output, and other outputs may be performed by one output terminal or by different output terminals. Furthermore, the output terminal 60 may serve as both the user terminal 10 and the supporter terminal 40, or may take on the output functions of the user terminal 10 and the supporter terminal 40.

[0023] FIG. 2 is a diagram showing an example of the hardware configuration of the user terminal 10. The user terminal 10 includes a CPU 101, a memory 102, a storage device 103, a communication interface 104, a touch panel display 105, and a camera 106. The storage device 103 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 104 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The touch panel display 105 is a device for inputting and outputting data. The user terminal 10 may further include input devices such as a keyboard, a mouse, buttons, and a microphone, and output devices such as a speaker and a printer.

[0024] 3 is a diagram showing an example of the software configuration of the user terminal 10. The user terminal 10 includes functional units, namely, an imaging unit 111, an evaluation request transmitting unit 112, an evaluation information receiving unit 113, an evaluation display unit 114, a checkpoint display unit 115, an improvement measure request transmitting unit 116, an improvement measure information receiving unit 117, an improvement measure information display unit 118, a reference value request transmitting unit 119, and a reference value selection information transmitting unit 120, as well as storage units, namely, a physical information storage unit 130, an image storage unit 131, an evaluation information storage unit 132, and an improvement measure storage unit 133.

[0025] Each of the above functional units is realized by the CPU 101 of the user terminal 10 reading out a program stored in the storage device 103 into the memory 102 and executing it, and each of the above storage units is realized as part of the storage area provided by the memory 102 and storage device 103 of the user terminal 10.

[0026] The imaging unit 111 captures images, including video, of the user's physical exercise. The imaging unit 111 can acquire video images of the physical exercise by controlling the camera 106. The user or the user's supporter simply installs the user terminal 10 on a flat surface or a wall, points the optical axis of the camera 106 toward the location where the user is exercising, and issues an instruction to start video recording. In response, the imaging unit 111 operates the camera 106 to acquire the video. The imaging unit 111 stores the acquired video in the image storage unit 131.

[0027] The image storage unit 131 stores images captured by the imaging unit 111. In this embodiment, the images are moving images, but are not limited to this. The image storage unit 131 can store the moving images as files, for example.

[0028] The physical information storage unit 130 stores information (hereinafter referred to as physical information) relating to the user's body, physical abilities, factors that affect training effects, etc. FIG. 4 is a diagram showing an example of the configuration of physical information stored in the physical information storage unit 130. As shown in the figure, the physical information includes height, weight, sex, dominant hand, arm length, leg length, hand size, finger length, grip strength, muscle strength, flexibility, shoulder strength, genome, epigenome, genetic polymorphism, intestinal flora, diet, etc. In addition, the type of physical exercise performed and the number of times thereof, etc., obtained by image analysis by the image analysis unit 212, may also be stored.

[0029] The evaluation request sending unit 112 sends to the server device 20 a request to evaluate the physical exercise using the tool (hereinafter referred to as an evaluation request) based on the image captured by the imaging unit 111.

[0030] FIG. 5 is a diagram showing an example of the configuration of a rating request sent by rating request sending unit 112 to server device 20. As shown in the figure, the rating request includes a user ID, a mode, physical information, and image data. The user ID is information that identifies the user. The mode is information that indicates the physical exercise performed by the user. The mode can be, for example, "walking," "yoga: cat pose," "bench press," "tennis serve," or "walking rehabilitation." Note that the mode is selected from a predetermined selection. The physical information is physical information stored in physical information storage unit 130. The image data is data of a moving image acquired by imaging unit 111.

[0031] The evaluation information receiving unit 113 receives information relating to the evaluation of the physical exercise (hereinafter referred to as evaluation information) returned from the server device 20 in response to the evaluation request. The evaluation information receiving unit 113 registers the received evaluation information in the evaluation information storage unit 132.

[0032] 6 is a diagram showing an example of the configuration of evaluation information received by the evaluation information receiving unit 113 from the server device 20. As shown in the figure, the evaluation information includes a mode, a user ID, tool position information, body part position information, posture information, movement information, and checkpoint information.

[0033] The user ID and mode are the user ID and mode included in the evaluation request. The captured image shows the user's body performing the exercise indicated by the mode.

[0034] The tool position information indicates the position of each part of the tool in the image (for example, the entire baseball bat, both ends, the grip, the point where the ball is hit, the center of gravity, or any other part; the entire barbell, the shaft, the plates (weight parts), the gripped part, the center of the shaft, the center of gravity, or any other part; including the entire tool). The tool position information includes each part of the tool and its position, associated with a point on the timeline of the video. Based on the tool position information, the movement of the tool and its relationship with body parts can be displayed. That is, for example, a figure indicating a part (for example, a circle) can be superimposed on the image at the position indicated by the tool position information. Note that the positions of multiple parts may be included for one point in time. Furthermore, tool position information does not need to be included for a part connecting two parts (for example, the midpoint between the parts where a barbell is gripped in the right and left hands). In this case, the part connecting these two parts can be represented by connecting a pair of marks (for example, circles) indicating two specified parts. Tool part position information may be included for each frame constituting the video, for each key frame (including frames related to checkpoints, which will be explained later), for every arbitrary number of frames, or for a random point in time. If My Frame does not include position information, the figure can be displayed based on the position information of the most recent past point in time.

[0035] The body position information indicates the position of each body part (e.g., head, shoulders, elbows, hips, knees, ankles, etc.) in the image. The body position information includes the part and the position of the part, corresponding to a time point on the time axis of the video. The skeletal state (bones) of the body can be displayed based on the body position information. That is, for example, a figure indicating the part (e.g., a circle) can be superimposed on the image at the position indicated by the body position information. Note that the positions of multiple parts can be included for one time point. Note that position information does not need to be included for parts connecting two parts (e.g., the forearm connecting the wrist and elbow, or the thigh connecting the hip and knee). In this case, the part connecting these two parts can be expressed by connecting a pair of marks (e.g., a circle) indicating two specific parts with a line. Position information may be included for each frame constituting the video, for each key frame (including frames related to checkpoints, which will be described later), for every arbitrary number of frames, or for random time points. If My Frame does not include position information, bones can be displayed based on position information from the most recent past time point.

[0036] Tool orientation information is information related to the orientation of the tool used by the user and the direction in which parts of the tool are facing. Tool orientation information includes, in association with a point on the video timeline, the part of the tool to be evaluated, a tool movement value, an evaluation rank, and an evaluation comment. The tool orientation value is a value that represents the orientation of the tool. Examples of the tool orientation value include the distance from the ground to a certain part of the tool, the distance between two parts of the tool, the angle of the parts (the angle between a first part of the tool and, for example, the part where the user is holding the tool, and between a second part of the tool and, for example, the part where the user is holding the tool), and the movement of a certain part of the tool. The evaluation rank is a value that expresses the evaluation value as a rank. The evaluation rank is expressed, for example, on a 5-point scale (1 to 5) or A, B, C, or the like. The evaluation comment is a comment related to the evaluation of posture. For example, if the mode is "upright row" and the right and left ends of the barbell are different distances from the ground, an evaluation comment such as "different forces are being applied to the left and right sides" may be included.

[0037] The tool movement information is information related to the movement of the tool used by the user. The tool movement information includes, in association with a period on the time axis of the video, the part of the tool to be evaluated, a list of the tool orientation values, an evaluation rank, and an evaluation comment. The list of tool orientation values ​​is a chronological list of tool orientation values ​​within the period. The evaluation comment is a comment related to the evaluation of the tool movement. For example, if the mode is "upright row" and the up and down movement of the barbell is insufficient, an evaluation comment such as "The barbell is not lifted sufficiently" may be included.

[0038] The posture information is information related to the posture of the user's body. The posture information includes a part to be evaluated, a posture value, an evaluation rank, and an evaluation comment, all associated with a time point on the time axis of the video. The posture value is a value that represents the posture. Examples of the posture value include the distance from the ground to a part, the distance between two parts, and the angle of the joint (the angle formed by the line from the first end part to the joint part and the line from the second end part to the joint part). The evaluation rank is a value that expresses the evaluation value as a rank. The evaluation rank is expressed, for example, on a 5-point scale from 1 to 5, or A, B, C, or the like. The evaluation comment is a comment related to the evaluation of the posture. For example, if the mode is "lifting" and bending is insufficient, an evaluation comment such as "your hips are not lowered" may be included.

[0039] The body movement information is information related to the user's body movements. The body movement information includes, in association with a period on the time axis of the video, the body part to be evaluated, a list of posture values, an evaluation rank, and an evaluation comment. The list of posture values ​​is a time series of posture values ​​within the period. The evaluation comment is a comment related to the evaluation of the movement. For example, if the mode is "lifting" and the knee extension is not smooth, an evaluation comment such as "The knee movement is not smooth" may be included.

[0040] The relationship information indicates one or more pieces of positional relationship information for each of the tool and the body. The relationship information includes information on the relationship between the tool's position, orientation, and movement, etc., and information on the relationship between the tool's parts and the body's parts, posture, and movement, etc., in association with a point on the video's timeline. For example, the positional relationship between the two can be displayed based on the tool position information and the body position information. That is, for example, a figure representing a part (e.g., a circle) can be superimposed on the image at the position indicated by the tool position information, and a figure representing a part (e.g., a circle) can be displayed at the position indicated by the body position information. Note that the positions of multiple parts and parts can be included for one point in time. Note that position information does not need to be included for the point connecting the parts (e.g., the tip of the bat and the center point of the part where the user grips the bat). In this case, the point connecting these two parts and parts can be represented by connecting a pair of two specified parts and figures (e.g., circles) representing these parts with a line. The relationship information may be included for each frame that makes up the video, for each key frame, for each checkpoint (details will be described later), for every arbitrary number of frames, or for a random point in time. If My Frame does not include location information, shapes indicating parts or regions and lines connecting them can be displayed based on location information from the most recent past point in time.

[0041] Checkpoint information is information that indicates points (hereinafter referred to as checkpoints) where the orientation of the tool or the posture of the body should be checked in the movement of the tool used by the user or in a series of movements of the user's body. For example, if the mode is "weightlifting," checkpoints include when the barbell reaches its highest position, when it reaches its lowest position, and the moment it is lifted. If the mode is "pitching," checkpoints include when the foot is raised, when the raised foot is lowered and weight is transferred, and when the ball is released. The checkpoint information stores information that indicates the checkpoint (hereinafter referred to as checkpoint ID) in association with a time point on the time axis of the video. In other words, it is possible to identify the frame (still image) in the video in which the checkpoint indicated by the checkpoint ID is displayed.

[0042] The evaluation display unit 114 displays the evaluation information. For example, the evaluation display unit 114 can superimpose on the video a graphic representing a tool part or a body part (for example, a circle representing the edge of the tool or the center of gravity of the body, and a line connecting them) based on the tool position, tool orientation, tool movement information, body position, posture, body movement information, etc. included in the evaluation information, thereby superimposing the tool part or the movement of the body part on the video. Furthermore, the evaluation display unit 114 can, for example, graphically display the position, orientation, and movement of the tool part, as well as the position, posture, and movement of the body part over time.

[0043] Furthermore, the evaluation display unit 114 can display an evaluation rank and an evaluation comment along with the display of the video, based on the tool orientation information and tool movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and evaluation comment included in the tool orientation information when the playback time of the video reaches a point around the time point included in the tool orientation information (this can be any length, for example, around 5 seconds). The evaluation display unit 114 can also display the evaluation rank and evaluation comment included in the tool movement information when the playback time of the video reaches a period included in the tool movement information. The evaluation display unit 114 can also display the orientation values ​​included in the orientation information. The evaluation display unit 114 can also display a graph of the time-series conversion of the tool orientation values ​​based on a list of tool orientation values ​​included in the tool movement information.

[0044] Furthermore, the evaluation display unit 114 can display an evaluation rank and an evaluation comment in conjunction with the display of the video based on the posture information and movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and evaluation comment included in the posture information when the playback time of the video reaches a point around the time point included in the posture information (for example, a desired length, such as around 5 seconds). The evaluation display unit 114 can display the evaluation rank and evaluation comment included in the movement information when the playback time of the video reaches a period included in the movement information. The evaluation display unit 114 can also display the posture values ​​included in the posture information. The evaluation display unit 114 can also display a graph of the time-series conversion of the posture values ​​based on a list of posture values ​​included in the movement information. The evaluation rank and evaluation comment may be displayed together with the evaluation rank and evaluation comment displayed in conjunction with the display of the video based on the tool orientation information and tool movement information described in the previous paragraph.

[0045] The checkpoint display unit 115 can extract and display images of checkpoints from the video. The checkpoint display unit 115 can read frames corresponding to time points included in the checkpoint information from the video image data stored in the image storage unit 131 and display them as still images. Furthermore, the checkpoint display unit 115 may, for example, extract and display only body parts from the read frames.

[0046] The improvement measure request sending unit 116 sends a request for acquiring improvement measures related to tool handling and physical exercise (hereinafter referred to as an improvement measure request) to the server device 20. FIG. 7 is a diagram showing an example of the configuration of an improvement measure request. As shown in the figure, the improvement measure request includes a user ID, a mode, a purpose, etc. The purpose is the reason for the user to make an improvement. Examples of purposes include "increase ball speed," "increase muscle strength," and "stabilize the lower body." The purpose is also selected from a predetermined list of options.

[0047] The improvement measure information receiving unit 117 receives information about improvement measures (hereinafter referred to as improvement measure information) transmitted from the server device 20 in response to an improvement measure request. The improvement measure information receiving unit 117 stores the received improvement measure information in the improvement measure storage unit 133. FIG. 8 shows an example of the configuration of improvement measure information. As shown in FIG. 8, the improvement measure information includes a purpose, advice, and reference information. In this embodiment, the advice is assumed to be a character string representing the improvement measure. However, it may also be content that presents the improvement measure using an image or video. The reference information includes the preferred orientation and movement of the tool (the position, orientation, movement, speed, angle, etc. of each part), the body posture (the position, orientation, movement, speed, angle, etc. of each part), and the relationship between the tool and the body (the position, orientation, movement, speed, angle, etc. of each part and each part, and the relationship therebetween). Note that the improvement measure information receiving unit 117 may receive improvement measure information transmitted by the improvement measure information transmitting unit 216 based on the evaluation result and the reference value, even without an improvement measure request.

[0048] The improvement measure information display unit 118 displays the improvement measure. The improvement measure information display unit 118 displays advice included in the improvement measure information. Furthermore, if the improvement measure information includes a suitable position or angle for the part, these may be displayed superimposed on the image information.

[0049] The reference value request sending unit 119 sends a request (hereinafter referred to as a reference value request) to the server device 20 to acquire the user's target reference value or a candidate thereof. FIG. 8 is a diagram showing an example of the configuration of an improvement measure request. As shown in the figure, the improvement measure request includes a user ID, a mode, a purpose, the image information, the physical information, and the evaluation information. The purpose is the reason for the user to make an improvement. Examples of purposes include "increasing ball speed," "increasing muscle strength," and "stabilizing the lower body." The purpose is also selected from a predetermined list of options.

[0050] The reference value selection information transmitting unit 120 transmits information indicating the selection of one or more reference values ​​presented by the reference information determining unit 220 to the server device 20. The reference value selection information transmitting unit 120 may also transmit an option to the server device 20 requesting that other options be presented.

[0051] 10 is a diagram illustrating an example of the hardware configuration of the server device 20. The server device 20 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like for inputting data. The output device 206 is, for example, a display, a printer, a speaker, or the like for outputting data.

[0052] 11 is a diagram illustrating an example of the software configuration of the server device 20. As illustrated in the diagram, the server device 20 includes functional units, namely, an evaluation request receiving unit 211, an image analysis unit 212, an evaluation unit 213, an evaluation information transmitting unit 214, an improvement measure request receiving unit 215, an improvement measure information transmitting unit 216, a group analysis unit 217, a group analysis information presenting unit 218, a reference information request receiving unit 219, and a reference information determining unit 220, as well as storage units, namely, an image data storage unit 231, a reference information storage unit 232, an evaluation condition information storage unit 233, an improvement measure information storage unit 234, and a group analysis information storage unit 235.

[0053] Each of the above functional units is realized by the CPU 201 of the server device 20 reading out a program stored in the storage device 203 into the memory 202 and executing it, and each of the above storage units is realized as part of the storage area provided by the memory 202 and storage device 203 of the server device 20.

[0054] The rating request receiving unit 211 receives a rating request transmitted from the user terminal 10. The rating request receiving unit 211 registers information including image data included in the received rating request (hereinafter referred to as image information) in the image data storage unit 231. FIG. 12 is a diagram showing an example of the configuration of image information stored in the image data storage unit 231. As shown in the figure, the image information includes image data associated with a user ID indicating the user who captured the image. The image data was included in the rating request.

[0055] The reference information storage unit 232 stores information (hereinafter referred to as reference information) related to physical exercise using a tool, including tool position, tool movement (such as the orientation and movement of the tool), posture (such as the position and angle of a part), and reference values ​​related to the relationship derived from the relationship between the tool and the body. FIG. 13 is a diagram showing an example of the configuration of reference information stored in the reference information storage unit 232. As shown in the figure, the reference information includes, but is not limited to, the absolute position of parts of the tool when performing physical exercise using a tool, information on how parts of the tool moved (such as the speed, distance, and direction of movement), reference information on the absolute position of parts or their relative position with respect to other parts or other reference objects (hereinafter referred to as position reference information), reference information on the angles formed by lines connecting two parts and the joint parts for three parts including joint parts (hereinafter referred to as angle reference information), and information on the relationship between parts of the tool and parts of the body. In addition, a reference value may be prepared for each physical movement (for each mode), and for each physical movement (mode), there may be multiple reference values, such as those for each purpose, for each characteristic of the physical information, for each characteristic of the evaluation information, or even those that are standardized based on the physical movements of a specific individual (assuming, but not limited to, athletes who have achieved certain results, professional athletes, skilled athletes, experienced athletes, etc.).

[0056] The tool position reference information includes tool parts and the reference positions of those parts, associated with the mode and checkpoint ID. There may be multiple parts. The vertical position of a part may be, for example, the height from the ground or the distance from one of the toes. Furthermore, for example, if the mode is "weightlifting," the distance from a body part or a line connecting two parts, such as the distance between the shaft and the line connecting both shoulders, may be used. The horizontal position of a part may be the distance from a predetermined reference object (for example, a mound plate or a mark on the floor) or the distance from a reference part such as the shoulders, chest, or feet. The position reference information is assumed to be registered in advance.

[0057] The tool movement reference information includes reference values ​​for information such as the movement speed and movement distance of tool parts, the direction of movement at a certain point in time, and the trajectory of movement over a certain period of time, in association with the mode and checkpoint ID.

[0058] The position reference information includes a body part and a reference position for that body part, in association with the mode and checkpoint ID. There may be multiple body parts. Regarding the position, the vertical position may be, for example, the height from the ground or the distance from one of the toes. The horizontal position may be the distance from a predetermined reference object (for example, a mound plate or a mark on the floor) or the distance from a reference body part such as the shoulders, chest, or feet. The position reference information is assumed to be registered in advance.

[0059] The angle reference information includes, in association with the mode and checkpoint ID, two parts (part 1 and part 2), one joint part, and a reference value for the angle between a line connecting part 1 and the joint part, and a line connecting part 2 and the joint part.

[0060] The related reference information includes information related to the reference relationship between tool parts and body parts, associated with the mode and checkpoint ID. The related reference information includes information obtained from the movement speed, movement distance, angle, etc. for one or more parts and parts, associated with the mode and checkpoint ID. For example, if the mode is batting, the related reference information includes the movement speed of the tip of the bat at the time the ball is met, and the angle formed by the bat and the dominant arm holding the bat, etc.

[0061] The evaluation condition information storage unit 233 stores information for evaluation (hereinafter referred to as evaluation condition information). FIG. 14 is a diagram showing an example of the configuration of evaluation condition information stored in the evaluation condition information storage unit 233. The evaluation condition information includes a category, a condition, an evaluation rank, and a comment. The category is an evaluation category. Examples of categories include "muscle strength," "ball speed," and "control." The conditions are conditions for the position, orientation, or movement (change in position over time) of each part of the tool in the image, as well as conditions for the position or movement (change in position over time) of each part of the body. For example, when analyzing weightlifting movements, conditions for the elbow angle, the speed at which the arms are extended, and the shaft movement and up-and-down speed during the period of lifting and lowering the barbell can be set in the evaluation condition information for the checkpoint at which the ball is released. Furthermore, when analyzing pitching form, conditions for the elbow angle, the arm speed, and the like can be set in the evaluation condition information for the checkpoint at which the ball is released. The evaluation rank is an evaluation value when the above conditions are met. The comment is an explanation of the body posture and movement when the above conditions are met.

[0062] The image analysis unit 212 (part / region identification unit) analyzes the image data. The image analysis unit 212 analyzes the image data to extract the feature amounts of each part of the tool and each part of the body, and identifies the position of each part and each region in the image. The image analysis unit 212 also analyzes the image data to extract the feature amounts of each part of the tool and identifies the direction in which each part is facing. Note that the image analysis method used by the image analysis unit 212 is a general one, and a detailed description thereof will be omitted here. The image analysis unit 212 may analyze the image data for each frame or each key frame, may analyze the image data for each checkpoint, or may analyze the image data at random timing.

[0063] The image analysis unit 212 also compares the position of each part extracted from the image data with the position reference information stored in the reference information storage unit 232 for each checkpoint ID, and identifies the closest point in time as the checkpoint point in time.

[0064] The evaluation unit 213 evaluates the movement of the tool used by the user based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including conditions satisfied by the positions and movements of each part of the tool identified from the image data, and if there is evaluation condition information for which the conditions are satisfied, acquires the evaluation rank and comments included therein. Note that the evaluation unit 213 may evaluate the movement of the tool and count the number of physical exercises.

[0065] The evaluation unit 213 evaluates the user's body movements based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including conditions satisfied by the positions and movements of each body part identified from the image data, and if there is evaluation condition information for which the conditions are satisfied, acquires the evaluation rank and comments included therein. Note that the evaluation unit 213 may evaluate the body movements and count the number of body exercises.

[0066] The evaluation unit 213 evaluates the tool used by the user and the body movement based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition information storage unit 233 for evaluation condition information including the positions of each part of the tool and each part of the body identified from the image data, and conditions satisfied by the movements or relationships of the parts and the parts, and if there is evaluation condition information for which the conditions are satisfied, acquires the evaluation rank and comments included therein. Note that the evaluation unit 213 may evaluate the tool and the body movement and count the number of body movements.

[0067] The evaluation information transmission unit 214 transmits evaluation information to the user terminal 10. The evaluation information transmission unit 214 generates tool position information including the time point on the time axis of the video identified by the image analysis unit 212 and the position of each part of the tool. If the position of the tool part satisfies the conditions for the evaluation rank and comment acquired by the evaluation unit 213, the evaluation information transmission unit 214 generates posture information including the time point, part, and tool orientation value, as well as the evaluation rank and comment. If the movement of the part (change in position over time) satisfies the conditions, the evaluation information transmission unit 214 generates tool movement information including a list of the time point, part, and tool orientation value, as well as the evaluation rank and comment. The evaluation information transmission unit 214 also generates checkpoint information including the time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmission unit 214 creates evaluation information including the generated tool position information, tool orientation information, tool movement information, and checkpoint information, and transmits it to the user terminal 10. The evaluation unit 213 and the evaluation information transmission unit 214 may correspond to the comment output unit of the present invention.

[0068] The evaluation information transmitting unit 214 transmits the evaluation information to the user terminal 10. The evaluation information transmitting unit 214 generates position information including the time point on the time axis of the video identified by the image analysis unit 212 and the position of each part. Regarding the evaluation rank and comment acquired by the evaluation unit 213, if the position of the part satisfies the condition, the evaluation information transmitting unit 214 generates posture information including the time point, part, and posture value, as well as the evaluation rank and comment. If the movement of the part (change in position over time) satisfies the condition, the evaluation information transmitting unit 214 generates movement information including a list of time points, part, and posture values, as well as the evaluation rank and comment. The evaluation information transmitting unit 214 also generates checkpoint information including the time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmitting unit 214 creates evaluation information including the generated position information, posture information, movement information, and checkpoint information, and transmits it to the user terminal 10. The evaluation unit 213 and the evaluation information transmitting unit 214 may correspond to a comment output unit of the present invention.

[0069] The improvement measure information storage unit 234 stores information related to improvement measures (hereinafter referred to as improvement measure information). FIG. 15 is a diagram showing an example of the configuration of improvement measure information stored in the improvement measure information storage unit 234. As shown in the figure, the improvement measure information includes advice associated with objectives, categories, and conditions. The conditions may be conditions on the tool itself (such as the weight of a barbell), conditions on how to use the tool, conditions on physical conditions (such as flexibility), conditions on the position, orientation, and movement of a part of the tool, or conditions on the position and movement of a part of the body.

[0070] The remedy request receiving unit 215 receives a remedy request sent from the user terminal 10 .

[0071] The improvement measure information transmitting unit 216 searches for improvement measure information corresponding to the mode and purpose included in the improvement measure request that satisfies the conditions of the user's physical information included in the evaluation request and the position, orientation, movement, etc. of each part and each portion identified by the image analysis unit 212. The improvement measure information transmitting unit 216 acquires advice from the searched improvement measure information, creates improvement measure information in which the purpose and advice are set, and responds to the created improvement measure information to the user terminal 10. The improvement measure information transmitting unit 216 also transmits the position, orientation, speed, angle, etc. of each part and each portion included in the reference information, including the improvement measure information. Note that the improvement measure information transmitting unit 216 may search for improvement measures based on the evaluation information and reference values ​​even without an improvement measure request, and the improvement measure information transmitting unit 216 may transmit the improvement measure to the user terminal 10.

[0072] FIG. 17 is a diagram showing an example of the flow of processing executed in the training support server of this embodiment.

[0073] In the user terminal 10, the imaging unit 111 accepts the input of the mode, captures an image of the user's body during exercise, and acquires video data (S321). The evaluation request sending unit 112 sends an evaluation request including the user ID indicating the user, the accepted mode, physical information, and video data to the server device 20 (S322).

[0074] When the evaluation request receiving unit 211 in the server device 20 receives an evaluation request, the image analysis unit 212 analyzes the video data to extract features (S323) and identifies the position of each part and each region (S324). Here, the image analysis unit 212 may identify the position on the image, or may identify the actual position (height from the ground, distance from a reference point such as the center of gravity of the body, etc.) using body information. The evaluation unit 213 obtains an evaluation rank and comments from evaluation condition information that satisfies the conditions of the position of each part and each region and the movement of each part and region (time-series change in position) (S325). The evaluation information sending unit 214 creates evaluation information and sends it to the user terminal 10 (S326).

[0075] The evaluation display unit 114 in the user terminal 10 displays the position, posture, and movement of the body, and the position, orientation, and movement of the tool, etc. on the video data based on the received evaluation information (S327). Furthermore, the evaluation display unit 114 in the user terminal 10 may display the position (bones) of each part indicating the posture of the body, as well as an evaluation rank and comments (S327). Here, the evaluation display unit 114 may graphically display the position, orientation, and movement of parts, as well as time-series changes in the position and movement of parts. Furthermore, the checkpoint display unit 115 may extract and display images of checkpoints from the video. The improvement request sending unit 116 sends an improvement request to the server device 20 in response to an instruction from the user (S328).

[0076] In the server device 20, when the improvement request receiving unit 215 receives an improvement request sent from the user terminal 10, the improvement information sending unit 216 searches for improvement information that satisfies the conditions, acquires the advice included in the searched improvement information (S329), creates improvement information including the acquired advice, and sends it to the user terminal 10 (S330).

[0077] When the improvement information receiving unit 117 receives improvement information in the user terminal 10, the improvement information display unit 118 displays the advice contained in the received improvement information and can also display the appropriate way to use the tool by superimposing it on the video data (S331).

[0078] When the improvement information receiving unit 117 receives the improvement information in the user terminal 10, the improvement information display unit 118 displays the advice contained in the received improvement information and can also display an appropriate body posture in the form of bones superimposed on the video data (S331).

[0079] As described above, the training support server of this embodiment allows for easy evaluation of physical exercise. In particular, for physical exercise related to sports, the ability to evaluate the relative positions and movements of each part of equipment and each part of the body can lead to specific efforts to improve, which is expected to lead to improved performance. Furthermore, the training support server of this embodiment also provides comments and advice, allowing the user to easily understand the current situation and measures for improvement.

[0080] Fig. 18 shows an example of a screen displaying an evaluation of physical exercise using a tool. This figure illustrates a case where a video is captured in weightlifting mode. As shown in this figure, a screen 41 displays a mark 411 indicating the position of the barbell shaft. The movement of the barbell shaft is displayed by a line 412.

[0081] FIG. 19 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. This diagram illustrates a case where a video is captured in weightlifting mode. The diagram displays, as an example, the tilt of the barbell shaft, the distance traveled by the shaft, and the speed of travel (line 421) as evaluation results performed by the evaluation unit 213 in weightlifting mode. In this diagram, a reference value (line 422) is displayed. The actual measurement result (line 423) may be displayed, and the difference from the reference value may be displayed numerically or as a graph. The user can use this as a reference to consider which movements or postures need to be corrected.

[0082] FIG. 20 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. This diagram illustrates a case where a video is captured in weightlifting mode. As a result of evaluation by the evaluation unit 213 in weightlifting mode, the diagram displays, for example, an evaluation result (line 432) along with a reference value (line 431) such as the angle of the body joints at the lowest point of the barbell. The difference from the reference value may also be displayed numerically or as a graph. Furthermore, the evaluation result based on the relationship between the body and the tool may also be displayed. The user can use this as a reference to consider movements and postures that need to be corrected.

[0083] FIG. 21 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. This diagram illustrates a case where a video is captured in weightlifting mode. As a result of evaluation by the evaluation unit 213 in weightlifting mode, lines 441 are displayed as bones, which are lines connecting predetermined positions of tool parts and body parts identified from the image. Note that the bones may be displayed superimposed on the captured image. Furthermore, lines 442 show the acceleration of each body part. Furthermore, count results such as the number of times the weight was lifted may be displayed. Furthermore, as shown by line 443, the evaluation results and the next training results tailored to the objectives, etc. may be displayed.

[0084] 22 is another diagram showing an example of a screen displaying an evaluation of physical exercise using a tool. This diagram illustrates a case where a video is captured in weightlifting mode. As a result of evaluation performed by the evaluation unit 213 in weightlifting mode, an evaluation (line 441) obtained by comparison with various reference values ​​is displayed.

[0085] The group analysis unit 217 is intended for use in situations where multiple users use the system simultaneously and a small number of supporters provide remote instruction (for example, training instruction or remote rehabilitation support using online meeting tools or online communication tools). In such situations, there may be multiple users for each supporter. Although image information of multiple users captured in real time is displayed on the device used by the supporter, it is difficult to grasp the detailed status of the users, such as which users are training effectively and which users are fatigued, resulting in reduced effectiveness. In addition, there is also the issue that it is difficult to communicate between the supporter and users based on the training content performed, which reduces continuity. The group analysis unit 217 solves these problems.

[0086] The group analysis unit 217 analyzes the evaluation information provided by the evaluation unit 213 for each user's image and stores the analyzed information in the group analysis information storage unit 235. Based on the evaluation information provided by the evaluation unit 213 regarding the user's physical exercise (including exercise using tools), the group analysis unit 217 creates a ranking of users whose physical exercises (which may be analyzed in real time) were effective (or ineffective), such as the number of repetitions of the training or the order of closeness (or furthestness) to a reference value. The group analysis unit 217 may also categorize each user into a top or bottom group in the ranking. Furthermore, the user's level of fatigue may be determined by the deviation from the reference value of the position, movement speed, movement distance, joint angle, etc., of the tool or body part as the user continues to perform physical exercise. Furthermore, the level of fatigue may be estimated by the deviation from the reference value. Furthermore, the time required to complete a predetermined number of repetitions may be measured for each set, and the level of fatigue may be determined by the increase in the completion time. The level of fatigue may also be estimated by comparing the increased time with the completion time of the previous set.

[0087] Furthermore, the group analysis unit 217 may estimate the user's concentration level by evaluating the time until the actual start of physical exercise, such as a supporter's signal to start or end the exercise. Furthermore, the group analysis unit 217 may calculate a deviation score from the evaluation information among users participating in training, etc. at the same time, or may create a ranking based on the deviation score. Furthermore, the group analysis unit 217 may calculate a deviation score from the evaluation information among users who performed the same physical exercise at different times, or may create a ranking based on the deviation score. Furthermore, based on the ranking, fatigue level, concentration level, etc., the group analysis unit 217 may identify users who need guidance or users for whom direct comments are thought to be effective in terms of continuity, etc. Furthermore, the group analysis unit 217 may compare evaluation information from a user's previous similar physical exercise with evaluation information evaluated from an image of the newly performed physical exercise.

[0088] The group analysis information storage unit 235 stores the group analysis information performed by the group analysis unit 217. Fig. 16 is a diagram showing an example of the configuration of group analysis information stored in the group analysis information storage unit 235. The group analysis information broadly includes ranking information, situation information, and caution information. The ranking information includes rankings based on the evaluation information, and category information such as whether the ranking is in a higher or lower group. The situation information includes fatigue levels and concentration levels. The caution information includes information such as users who require special guidance.

[0089] The group analysis information presenting unit 218 presents the group analysis information stored in the group analysis information storage unit 235 to the user terminal 10 and the supporter terminal 40. The group analysis information presenting unit 218 presents to the user terminal 10 the ranking, deviation value, categories such as top and bottom, etc. of the user among users who are performing physical exercise at the same time as the user, and among users including users who performed similar physical exercise at different times. Furthermore, the group analysis information presenting unit 218 may also present information stored in the evaluation information storage unit 132. Furthermore, the group analysis information presenting unit 218 may present the ranking and deviation value information of each user to the supporter terminal 40, or may color-code users who are ranked high (or low) so that they are easily visible on the screen (for example, by displaying the screen showing the user larger or at the top).

[0090] 23 is a diagram showing an example of a screen created and presented by the group analysis information presenting unit 218 on the user terminal 10. The figure illustrates a case where a video is captured in squat mode. As shown in the figure, on screen 46, line 461 shows the user's ranking within the group. Line 462 indicates the number of exercises performed. Line 463 displays real-time video (or recorded video) of the trainer who is the model.

[0091] 24 is a diagram showing an example of a screen created by the group analysis information presenting unit 218 and presented to the supporter terminal 40. The supporter terminal 40 displays the evaluation results of each user, group analysis information, etc., together with the video of each user.

[0092] 25 is a diagram showing an example of a screen created by the group analysis information presenting unit 218 and presented on the supporter terminal 40. Screen 47 displays images of users, and based on evaluation and group analysis, displays users in the group who need more guidance (including support, etc.) in a way that is easy for the supporter to understand. As an example, screen 47 displays a mark in the upper right corner of the screen showing the user who needs guidance.

[0093] 26 is a diagram showing an example of a screen created by the group analysis information presenting unit 218 and presented on the supporter terminal 40. Screen 48 is a screen showing details of each user, and displays the evaluation results (bone and posture values, tool orientation values, number of repetitions of physical exercise, etc.), group analysis results (ranking within the group and its progress, etc.), advice sent to the user by the improvement information sending unit 216, a function to view recorded images (or a part of a video, such as a few seconds before and after the point when the evaluation dropped), etc. Also, screens 47 and 48 may be combined.

[0094] The reference information request receiving unit 219 receives a reference information request transmitted from the user terminal 10. The reference information request receiving unit 219 registers information including image data (hereinafter referred to as image information) included in the received reference information request in the image data storage unit 231. FIG. 12 is a diagram showing an example of the configuration of image information stored in the image data storage unit 231. As shown in the figure, the image information includes image data associated with a user ID indicating the user who captured the image. The image data was included in the reference information request. It is to be noted that it is also possible that the reference information request does not include image data.

[0095] The reference information determination unit 220 searches for reference information that meets the user's needs based on the mode, purpose, image information, the physical information, the evaluation information, etc. included in the reference information request, and transmits the searched reference information to the user terminal 10. The reference information determination unit 220 may also receive information that selects one of the reference information transmitted by the reference value selection information transmission unit 120, and the image analysis unit 212 may perform analysis based on the selected reference information. Furthermore, even without a reference information request, the reference information determination unit 220 may transmit candidate reference information to the user based on the purpose, physical information, basic information, etc. Furthermore, the reference information determination unit 220 may present an option that requests the user to submit another reference value along with the reference value options, and in that case, may also present options such as "increase intensity" or "can continue for a long time" as references for selecting a new reference value to be presented.

[0096] For example, when the reference information request includes a weightlifting mode and the objective includes "muscle hypertrophy of the upper pectoralis major," the reference information determination unit 220 selects reference information suited to training the upper pectoralis major from multiple pieces of reference information stored in the reference information storage unit. In this case, for example, reference information such as reference information 1 "for training the upper pectoralis major: push your arms diagonally upward," reference information 2 "for training the inner pectoralis major: close your arms in front," and reference information 3 "lower pectoralis major: push your arms diagonally downward" is stored in the reference information storage unit 232. The reference information determination unit 220 transmits reference information according to the objective to the user terminal 10, and when the user selects the reference information, the reference information determination unit 220 accepts the selection, and the reference information becomes the reference information used by the evaluation unit 213 for evaluation.

[0097] Furthermore, when the reference information request includes a weightlifting mode and the objective includes "muscle hypertrophy of the upper pectoralis major," the reference information determination unit 220 selects reference information suited to training the upper pectoralis major from multiple pieces of reference information stored in the reference information storage unit as reference information. Alternatively, the reference information may be selected based on the physical information. In this case, for example, reference information such as "Reference Information 1 for training the upper pectoralis major: push arms diagonally upward," "Reference Information 2 for training the inner pectoralis major: close arms in front," and "Reference Information 3 for training the lower pectoralis major: push arms diagonally downward" is stored. If there is a relationship between the respective pieces of reference information, such as the fact that training the upper pectoralis major will be less effective or may result in injury if the inner pectoralis major is not trained to a certain extent, the reference information determination unit 220 may first transmit reference information 2 "For training the inner pectoralis major: close arms in front" to the user terminal 10 based on the user's physical information at that time (e.g., muscle strength, inner pectoralis major muscle strength, etc.). In this case, training steps and advice such as "First train the inner pectoralis major (i.e., reference information 2), then train the upper pectoralis major (i.e., reference information 1)" may also be sent.

[0098] Furthermore, when the reference information request includes weightlifting mode and the objective includes "muscle hypertrophy of the upper pectoralis major", the reference information determination unit 220 selects reference information suited to training of the upper pectoralis major from multiple pieces of reference information stored in the reference information storage unit as reference information, and may further select reference information based on the evaluation information. In this case, for example, reference information such as reference information 1 "For training the upper pectoralis major: push your arms diagonally upward," reference information 2 "For training the inner pectoralis major: close your arms in front," and reference information 3 "For training the lower pectoralis major: push your arms diagonally downward" are stored, and if there is a relationship between the respective pieces of reference information, such as the fact that the movements for training the upper pectoralis major are difficult, and therefore it is better to first practice the movements for training the inner pectoralis major, the reference information determination unit 220 may first send reference information 2 "For training the inner pectoralis major: close your arms in front," which is less difficult, to the user terminal 10 based on information such as tool movement information (for example, whether the barbell can be raised and lowered smoothly) included in the user's evaluation information (which can be considered as skill) at that time (for this reason, the image analysis unit 212 may analyze image information included in the reference information request and the evaluation unit 213 may evaluate it). In this case, training steps and advice such as "First, practice the movements to train the inner pectoralis major (i.e., reference information 2), and then train the upper pectoralis major (i.e., reference information 1)" may also be sent.

[0099] Furthermore, when the reference information request includes weightlifting mode and the purpose includes "muscle hypertrophy of the upper pectoralis major", the reference information determination unit 220 selects reference information suited to training of the upper pectoralis major from multiple pieces of reference information stored in the reference information storage unit as reference information, and may further select reference information based on past training history information included in the physical information. In this case, for example, reference information such as reference information 1 "for training the upper pectoralis major: push your arms diagonally upwards," reference information 2 "for training the inner pectoralis major: close your arms in front," and reference information 3 "for training the lower pectoralis major: push your arms diagonally downwards" are stored, and if there is a relationship between the respective pieces of reference information such as the fact that if the inner pectoralis major is not trained to a certain extent, training the upper pectoralis major will be less effective or there is a risk of injury, the reference information determination unit 220 may further transmit reference information 1 "for training the upper pectoralis major: push your arms diagonally upwards" to the user terminal 10 when it is confirmed that the training history is greater than a certain number of times, for example, based on the user's past training history information (for example, the number of times inner pectoralis major training has been performed, or the number of times the improvement information transmission unit 216 has already sent a message to the user terminal 10 requesting the user to perform inner pectoralis major training). In this case, training steps and advice such as "You have already sufficiently trained the inner pectoralis major (i.e., reference information 2) so let's train the upper pectoralis major (i.e., reference information 1)" may also be sent.

[0100] Furthermore, when the reference information request includes, for example, weightlifting mode and the purpose includes "muscle hypertrophy of the upper pectoralis major", the reference information determination unit 220 selects reference information suited to training of the upper pectoralis major from multiple pieces of reference information stored in the reference information storage unit as reference information, but may also send a menu of physical exercises to the user terminal 10 to prompt the user to perform one or more physical exercises, and select reference information based on the user's evaluation of the physical exercises captured on camera. In this case, for example, reference information such as reference information 1 “Upper pectoral muscle training A: Push arms diagonally upward with a 20 kg barbell,” reference information 2 “Upper pectoral muscle training B: Push arms diagonally upward with a 40 kg barbell,” and reference information 3 “Basic chest muscle training: 30 push-ups” are stored. If there is a relationship between the respective reference information, such as a relationship indicating that the chest muscles are trained to a certain extent before proceeding to the next step, the reference information determination unit 220 first transmits “pull-ups” to the user terminal 10, has the user take images of the physical exercise, and the images are received by the image analysis unit and counts the number of pull-ups, etc. If a certain standard is exceeded, the reference information determination unit 220 transmits reference information 1 “Upper pectoral muscle training A: Push arms diagonally upward with a 20 kg barbell” to the user terminal 10 as the next step. If the certain standard is not exceeded, reference information 3 “Basic chest muscle training: 30 push-ups” may be transmitted to the user terminal 10 to satisfy the standard. In this case, training steps and advice such as "To train the upper pectoral muscles (i.e., reference information 1 or 2), first do basic chest muscle training with push-ups (reference information 3)" may also be sent.

[0101] Furthermore, the reference information determination unit 220 may also present the recommendation level of the reference information to be presented to the user based on the mode, purpose, and image information included in the reference information request, the physical information, the evaluation information, etc. In this case, the recommendation level may be indicated numerically, such as a number out of 100 or a percentage, or may be indicated as a level such as 3 out of 5, or may be indicated by marks such as ○, ×, or △, or may be indicated by a symbol or color indicating an impression of suitability, such as thumbs up or thumbs down, or by color, and the number of users who have selected that reference value in the past and the evaluations of that reference value obtained from those users may also be presented.

[0102] Furthermore, the reference information determination unit 220 may derive the recommendation level of the reference information to be presented to the user from the degree of match between the objective and the objective set in the reference value, or from the relationship between the evaluation information and the reference value (if the evaluation information is a fast bat swing speed, the standard value for people with a fast bat swing speed will have a high degree of suitability, etc.), or from the relationship between the physical information and the standard value (if the physical information is left-handed, the standard value for batting for left-handed batters will have a high degree of suitability, and furthermore, if the physical exercise is strong, the standard value indicating that strong muscle strength is more effective will have an even higher degree of suitability, etc.), or the recommendation level may be derived from any one of these alone or in combination.

[0103] Furthermore, the reference information determination unit 220 may derive the recommendation level of the reference information to be presented to the user from the degree of match between the objective and the objective set in the reference value, or from the relationship between the evaluation information and the reference value (if the evaluation information is a fast bat swing speed, the standard value for people with a fast bat swing speed will have a high degree of suitability, etc.), or from the relationship between the physical information and the standard value (if the physical information is left-handed, the standard value for batting for left-handed batters will have a high degree of suitability, and furthermore, if the physical exercise is strong, the standard value indicating that strong muscle strength is more effective will have an even higher degree of suitability, etc.), or the recommendation level may be derived from any one of these alone or in combination.

[0104] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0105] For example, in this embodiment, the image analysis is performed in the server device 20, but this is not limiting. The image analysis may be performed in the user terminal 10 to identify the positional relationship of each part and each region.

[0106] In addition, in this embodiment, the positions of the parts and regions are assumed to be positions on a two-dimensional image, but this is not limiting and the positions may be three-dimensional. For example, if the user terminal 10 is equipped with a depth camera in addition to the camera 106, the three-dimensional positions of the parts and regions can be identified based on the image from the camera 106 and the depth map from the depth camera. Alternatively, for example, the three-dimensional positions of the parts and regions may be identified by estimating the three-dimensionality from the two-dimensional image. Note that it is also possible to provide a depth camera instead of the camera 106 and identify the three-dimensional positions only from the depth map from the depth camera. In this case, the user terminal 10 can transmit the depth map to the server device 20 together with or instead of the image data, and the image analysis unit 212 of the server device 20 can analyze the three-dimensional positions.

[0107] Furthermore, in this embodiment, an image of the user's body while exercising using a tool is transmitted from the user terminal 10 to the server device 20, but this is not limited to this. The user terminal 10 may extract features from the image and transmit the features to the server device 20. Alternatively, the user terminal 10 may estimate the parts of the tool or the body parts based on the features, obtain the absolute positions of the parts or parts (which may be positions on the XY coordinate system of the image, or may be actual distances from a reference position (for example, the ground, the toes, the head, the center of gravity of the body, etc.), or may be positions in any other coordinate system), or the relative positional relationships between multiple parts, multiple parts, or multiple part parts, and transmit these absolute positions or relative positional relationships to the server device 20.

[0108] In addition, in this embodiment, the improvement measure information is provided as content prepared on the server device 20 side, but this is not limiting. For example, it is also possible to include reference values, and superimpose marks or bones that represent correct movements and postures (positions and orientations of each part, positions and angles of each part, etc.) based on the reference values ​​onto a video or a still image extracted from the video. This makes it easy to understand what movements and postures should be adopted.

[0109] In addition, in this embodiment, the evaluation is based on the part and orientation of the tool, the position or movement of a body part (position over time), etc., but this is not limited to this, and the evaluation may also be based on identifying the position of the tool worn by the user.

[0110] In this embodiment, content such as advice is provided as a measure for improvement. However, for example, tool recommendations may also be performed. In this case, the server device 20 stores tools and standard values ​​for the tool's size (e.g., length) in association with the user's physical information (e.g., height, weight), extracts features of the tool used by the user from image data to identify the tool's shape, estimates the tool's size based on the shape and the user's size (e.g., height) included in the physical information, and recommends a tool of the standard size if the difference between the estimated tool size and the standard value is equal to or greater than a predetermined threshold. Furthermore, tools according to the purpose may be recommended based on information such as the tool's conditions (e.g., the weight of a barbell), how the tool is used, physical conditions (e.g., flexibility), and the position, orientation, and movement of the tool's parts.

[0111] Furthermore, in this embodiment, content such as advice is provided as a remedial measure. However, for example, the physical exercise being performed may be interrupted. In this case, the server device 20 stores a reference value for when the physical exercise should be interrupted in association with the user's physical information (purpose, height, weight, etc.), and interrupts the physical exercise if the number of times or speed of the physical exercise being performed by the user (for example, if the speed at which the barbell is lifted drops drastically or if too many repetitions are performed at once) deviates from the reference value based on the image data. In this case, the server device 20 may issue a message to the user terminal 10 telling the user to stop, or may notify the user by changing the display, such as by turning off the screen, or by emitting a sound such as an alert, or by vibrating the user.

[0112] In addition, in this embodiment, content such as advice is provided regarding improvement measures, but for example, a diagnosis of a disease and physical exercises for improving the disease may be presented. In this case, the server device 20 extracts candidate diseases that the user is suspected to have from the symptoms and evaluation information entered by the user in the physical information, and presents a screening test to narrow down the candidate diseases. Once the user performs the screening test and narrows down the disease name, the server device 20 may recommend a doctor's consultation, physical exercises for improvement, or items such as tools and meals for physical exercise.

[0113] Furthermore, by estimating the positions of the tool parts, the server device 20 can estimate the speed, acceleration, movement distance, trajectory, etc. of the tool. Furthermore, by extracting a pattern of change in the tool position over time, the server device 20 can estimate the number of times a pattern occurs as the number of actions using the tool.

[0114] Furthermore, in this embodiment, the exercise is evaluated, but the present invention is not limited to this. When a certain posture or movement is detected, a task for that movement may be proposed. In this case, the server device 20 may store a task in association with one or a series of postures or movements, instead of an evaluation comment, and output the task.

[0115] Furthermore, in this embodiment, exercise is evaluated, but this is not limiting, and when a certain tool movement, tool orientation, posture, or body part movement is detected, content to improve physical exercise according to the purpose, etc. may be presented, such as training to be done, rehabilitation, musical performance, or preparatory steps such as stretching, strength training, posture, etc. In this case, the server device 20 may store the details of the training, etc., in association with one or a series of tool part movements, tool part orientations, body postures, or body part movements, instead of evaluation comments, and output the details.

[0116] Furthermore, in this embodiment, the exercise is evaluated, but the present invention is not limited to this and the movements performed by the user can also be automatically detected. In this case, the server device 20 stores the positions and postures of each part of the tool used to perform a predetermined movement such as shooting or passing (the positions of each body part) as reference information, and can identify the movement performed by the user in the image by comparing the positions of the tool parts and body parts analyzed from the image with the reference information.

[0117] In addition, in this embodiment, the motion is evaluated by analyzing images captured in the past, but this is not limiting. The analysis process may be performed in real time, and when a predetermined motion is detected, the next tactic to be adopted may be recommended. In this case, the tactic may be stored in association with the posture or motion instead of the evaluation comment, and the tactic may be output in real time.

[0118] In this embodiment, the supporter terminal 40 may also function as the imaging terminal 50 or the output terminal 60. In this case, the supporter terminal 40 may be in the form of, for example, glasses, contact lenses, a hat, or an HMD (head-mounted display). The supporter terminal 40 captures an image of a range close to the supporter's field of vision using its imaging function, and the captured image is sent to the server device 20 via the communication network 30. The evaluation unit 213 and the group analysis unit 217 process the image, and the results are sent to the supporter terminal 40 or the user terminal 10 via the communication network 30. As shown in FIG. 27 , the supporter terminal 40 outputs information to the supporter or user via an interface such as a virtual image projection system, a retinal projection system, or other system, or a brain-machine interface (BMI) that utilizes brain activity such as electroencephalography to input text, images, videos, etc. by directly stimulating the brain without using sensory organs. By performing this communication and processing at high speed, the supporter can view the user's physical movement and confirm the user's physical movement evaluation and group evaluation in near real time. In addition, the user can also check the evaluation of his / her own physical movement and the evaluation within the group in almost real time. Note that in the field of view of the supporter, the user's physical movement is seen as a real image, and the results of processing by the evaluation unit 213 and the group analysis unit 217 may be superimposed on the field of view seen by the supporter with the naked eye via the supporter terminal 40, so that the supporter can see them. Furthermore, if the supporter terminal 40 is an HMD or the like, the results of processing by the evaluation unit 213 and the group analysis unit 217 may be presented to the supporter in a form superimposed on the image captured by the supporter terminal 40.

[0119] In this embodiment, the user terminal 10 may also function as the imaging terminal 50 or the output terminal 60. In this case, the user terminal 10 may be, for example, in the form of glasses, contact lenses, a hat, or an HMD (head-mounted display). The imaging function of the user terminal 10 captures an image of a range close to the user's field of view, and the captured image is sent to the server device 20 via the communication network 30. The evaluation unit 213 and the group analysis unit 217 process the image, and the results are sent to the user terminal 10 or the supporter terminal 40 via the communication network 30. As shown in FIG. 25 , the supporter terminal 40 outputs information to the user or supporter via an interface such as a virtual image projection system, a retinal projection system, or other system, or a brain-machine interface (BMI) that utilizes brain activity such as electroencephalography to input text, images, videos, etc. by directly stimulating the brain without using sensory organs. By performing this communication and processing at high speed, the user can view their own physical movements through a mirror or the like, and check the evaluation of their own physical movements and the evaluation of their group in near real time.

[0120] Buying and selling of standard information In this embodiment, there may be a plurality of reference values, and the user may select a reference value for a fee.

[0121] Buying and selling of standard information In this embodiment, the reference values ​​may include values ​​created based on the physical exercise (including that performed using equipment) of skilled people such as professional athletes.

[0122] Buying and selling of standard information Furthermore, in this embodiment, the user terminal 10, the supporter terminal 40, the imaging terminal 50, and the output terminal 60 may be wearable terminals such as glasses, and the results of capturing images, analyzing, and evaluating them may be output to the wearable terminals. However, non-wearable user terminals 10, supporter terminals 40, imaging terminals 50, and output terminals 60 such as mobile terminals may also exist simultaneously, and the results of analysis and evaluation may also be displayed on the non-wearable terminals.

[0123] In addition, in this embodiment, the execution of a predetermined function and the storage of information are performed by the user terminal 10 or the server device 20, but this is not limiting, and the execution of the function and the storage of information may be performed by either device. Alternatively, the function unit and the storage unit may be provided separately in a form different from this embodiment.

[0124] In addition, in this embodiment, the execution of a predetermined function and the storage of information are performed by the user terminal 10 or the server device 20, but this is not limiting, and the execution of the function and the storage of information may be performed by either device. Alternatively, the function unit and the storage unit may be provided separately in a form different from this embodiment. [Explanation of symbols]

[0125] 10 User terminal 20 Server device 30 Communication Network 40 Supporter terminal 50 Imaging terminal 60 Output Terminal 101 CPU 102 memory 103 Storage device 104 Communication Interface 105 Output Device 106 Cameras and other input devices 111 Imaging unit 112 Evaluation request sending unit 113 Evaluation information receiving unit 114 Evaluation display section 115 Checkpoint display 116 Improvement Request Sending Unit 117 Improvement Information Receiving Department 118 Improvement measure display area 119 Reference value request transmission unit 120 Reference value selection information transmission unit 130 Physical information storage unit 131 Image storage unit 132 Evaluation information storage unit 133 Improvement plan storage section 201 CPU 202 memory 203 Storage device 204 Communication Interface 205 Output Device 206 Cameras and other input devices 211 Evaluation Request Receiving Unit 212 Image Analysis Unit 213 Evaluation Department 214 Evaluation information transmission unit 215 Remediation Request Reception Unit 216 Improvement Information Transmission Department 217 Group Analysis Department 218 Group Analysis Presentation Section 219 Reference value request receiver 220 Reference value determination unit 231 Image data storage unit 232 Standard information storage unit 233 Evaluation condition information storage unit 234 Improvement information storage unit 235 Group analysis information storage unit

Claims

1. An instruction support system that supports an assistant in providing instruction to a user who is performing physical exercise, an imaging unit that captures an image of the user's physical exercise; a reference value storage unit configured to store a reference value relating to the position of at least one body part or tool part; a region or part specifying unit that analyzes the image and specifies the region or part; an evaluation unit that compares the position of the site or portion in the image with the reference value to determine an evaluation value of the physical exercise while the user is performing the physical exercise; an evaluation display unit that displays information based on the evaluation value by superimposing it on the user who is performing the physical exercise; A teaching support system comprising:

2. A supporter terminal having the imaging unit and the evaluation display unit, The supporter terminal is a wearable computer. The teaching support system according to claim 1 .

3. There are a plurality of the reference values, a reference value determination unit that presents one or more reference values ​​to the user based on at least one of information of the user's purpose, physical information, and the evaluation value; 3. The training support system according to claim 1, further comprising:

4. a reference value selection information transmission unit for allowing the user to select from candidate reference values; The training support system according to claim 1 , further comprising:

5. An instruction support method in which a supporter supports instruction given to a user performing physical exercise, comprising: The processor: capturing an image of the user's physical exercise; a reference value storage step of storing a reference value relating to the position of at least one body part or tool part; a site or portion identification step of analyzing the image to identify the site or portion; an evaluation step of comparing the position of the site or portion in the image with the reference value to determine an evaluation value of the physical exercise while the user is performing the physical exercise; an evaluation display step of displaying information based on the evaluation value superimposed on the user performing the physical exercise; A teaching support method that implements the above.

Images