Exercise Support System

Abstract

To efficiently support body exercise.SOLUTION: A system for supporting user's exercise includes: a camera 21 for imaging a user in exercise; a display 22 installed in a site where a user exercises; an instruction output part for outputting an instruction about exercise on the display; a position specifying part for specifying a position of a user's body part by analyzing an image captured by the camera; and an output part for outputting the position of the part to the display while superposing the position on the image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an exercise support system. [Background technology]

[0002] Conventional analytical techniques in the field of physical movement include a technique (Patent Document 1) for extracting movement patterns that appear repeatedly in a video, comparing the movement patterns at multiple positions with different time positions in the extracted video from a predetermined viewpoint, and outputting the comparison results as a video or still image. Also disclosed in the field of sports, for example, is a technique for analyzing a golf wing using a motion sensor by attaching a device to the user's body in golf (Patent Document 2), and a technique for capturing images of each stage of a pitch (placement of the pivot foot, horizontal rotation of the trunk) with two cameras and calculating the transition time (Patent Document 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-229081 A [Patent Document 2] JP 2010-068947 A [Patent Document 3] JP 2018-114138 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, all of the technologies in Patent Documents 1 to 3 require equipment and resources such as multiple video images, specific devices, and multiple cameras, making it difficult for a single user to use these devices to analyze physical movements and understand the analysis results.

[0005] The present invention has been made in view of the above background, and has an object to provide a technique capable of efficiently supporting physical exercise. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problems is a system for assisting a user in exercising, comprising a camera that photographs the user while exercising, a display installed at the location where the exercise is performed, an explanation output unit that outputs an explanation of the exercise to the display, a body part identification unit that analyzes the image captured by the camera and identifies the position of a body part of the user, and an output unit that superimposes the position of the body part on the image and outputs it to the display.

[0007] Other problems and solutions disclosed in this application will be made clear in the description of the preferred embodiments of the invention and the drawings. Effect of the Invention

[0008] According to the present invention, physical exercise can be efficiently supported. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an exercise support system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device 20. [Diagram 3] FIG. 2 is a diagram illustrating an example of the software configuration of the server device 20. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a training information storage unit 231. [Diagram 5] 2 is a diagram illustrating an example of the configuration of a user information storage unit 232. FIG. [Figure 6] 13 is a diagram showing an example of an input screen 41 for inputting user information. [Figure 7] FIG. 2 is a diagram illustrating an example of the configuration of an image data storage unit 233. [Figure 8] 13 is a diagram illustrating an example of the configuration of a reference information storage unit 234. FIG. [Figure 9] 2 is a diagram illustrating an example of the configuration of an evaluation condition information storage unit 235. FIG. [Figure 10] FIG. 2 is a diagram illustrating an example of the configuration of a training plan storage unit 236. [Figure 11] FIG. 2 is a diagram illustrating an example of the configuration of a training history storage unit 237. [Figure 12] FIG. 2 is a diagram illustrating an example of the configuration of an improvement condition storage unit 238. [Figure 13] FIG. 4 is a diagram showing an example of an output screen 42 for training information. [Figure 14] 13 is a diagram showing an example of the configuration of evaluation information created by an evaluation unit 215. FIG. [Figure 15] 4 is a diagram showing an example of a screen 42 output by an evaluation information output unit 216. FIG. [Figure 16] 4 is a diagram showing an example of a screen 42 output by an evaluation information output unit 216. FIG. [Figure 17] 4 is a diagram showing an example of a screen 42 output by an evaluation information output unit 216. FIG. [Figure 18] FIG. 2 is a diagram showing an example of a training history displayed by the user terminal 10. [Figure 19] FIG. 4 is a diagram showing an example of an output screen 44 for training volume information. [Figure 20] FIG. 2 is a diagram showing a flow of processing executed in the exercise support system of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The contents of the embodiments of the present invention will be described below. An exercise support system according to an embodiment of the present invention has the following configuration.

[0011] [Item 1] A system for assisting a user in exercising, comprising: a camera that photographs the user while exercising; A display installed in a place where the exercise is performed; an explanation output unit that outputs an explanation regarding the exercise to the display; a body part identification unit that analyzes an image captured by the camera to identify a position of a body part of the user; an output unit that superimposes the position of the part on the image and outputs the image to the display; An exercise support system comprising: [Item 2] The exercise support system according to item 1, a reference value storage unit that stores a reference value relating to the position of at least one of the parts; an evaluation unit that compares the position of the part in the image with the reference value to determine an evaluation value of the movement; Further equipped with The output unit further outputs the evaluation value to the display. An exercise support system characterized by the above. [Item 3] The exercise support system according to item 2, the evaluation unit determines the evaluation value according to a distance from the reference value to the position; An exercise support system characterized by the above. [Item 4] The exercise support system according to item 2, the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion; the region identifying unit identifies each of the first to third regions from the image; the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with the reference value to determine the evaluation value; An exercise support system characterized by the above. [Item 5] The exercise support system according to item 1, A number counting unit that counts the number of times of the exercise based on the position of the part is further provided, The output unit further outputs the number of times of the exercise to the display. An exercise support system characterized by the above. [Item 6] The exercise support system according to claim 2, The output unit further outputs the evaluation value by voice. An exercise support system characterized by the above. [Item 7] The exercise support system according to item 1, The output unit further outputs remedial action information by voice based on the identified body position of the user; An exercise support system characterized by the above.

[0012] An exercise support system according to one embodiment of the present invention is intended to support a user's training in a training gym. The exercise support system of this embodiment captures with a camera the user exercising using training equipment, and counts the number of training sessions from the captured images (which may be still images or moving images, but in this embodiment, moving images are used), and evaluates the user's body posture and movement during training.

[0013] 1 is a diagram showing an example of the overall configuration of an exercise support system according to this embodiment. As shown in the figure, in the exercise support system of this embodiment, a server device 20, a camera 21, and a display 22 are communicably connected to each other via a communication network 23. The communication network 23 is, for example, the Internet or a LAN (Local Area Network), and is constructed by a public telephone line network, a dedicated telephone line network, a mobile phone line network, Ethernet (registered trademark), a wireless communication path, etc.

[0014] The camera 21 captures images of the user 1 during training. For example, a surveillance camera may be used as the camera 21, or the camera 21 may be installed at the training location, such as by attaching the camera 21 to training equipment. The camera 21 can transmit the captured images to the server device 20 using protocols such as TCP / IP and Bluetooth (registered trademark). The camera 21 may capture images continuously, may start capturing images when it detects that the user 1 has come close, or may start capturing images in response to an instruction from the user 1.

[0015] The display 22 is installed so as to be visible to the user 1 before and during training. The display 22 outputs information under the control of the server device 20 via the communication network 23. Note that by connecting a computer to the display 22 and having the computer communicate with the server device 20, information can be output to the display 22 in response to instructions from the server 20. In addition to or instead of the display 22, the user 1 can be made to wear an audio output device such as headphones, earphones, or a neck speaker, and information can be output as audio.

[0016] The user terminal 10 is a computer operated by the user 1 or his / her supporter. The user terminal 10 is, for example, a smartphone, a tablet computer, a personal computer, or the like. The user terminal 10 may be equipped with an imaging device such as a camera, which may be used to capture images of the body of the user 1 during exercise. The information displayed on the display 22 is also provided to the user terminal 10 from the server device 20. The user 1 can view information related to the training by checking the user terminal 10 later.

[0017] The server device 20 is a computer that supports training. The server device 20 is, for example, a workstation, a personal computer, or a virtual computer logically realized by cloud computing. The server device 20 receives moving images captured by the user terminal 10, analyzes the received moving images, and evaluates the physical exercise. The server device 20 also makes suggestions regarding measures to improve the physical exercise. Details of the evaluation of the physical exercise and the suggestions of measures to improve the physical exercise will be described later.

[0018] The administrator terminal 30 is a computer operated by an administrator of the training gym. The administrator terminal 30 may be, for example, a smartphone, a tablet computer, or a personal computer.

[0019] FIG. 2 is a diagram showing an example of a hardware configuration of the server device 20. The server device 20 includes an arithmetic device 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The arithmetic device 201 performs various arithmetic processing, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, and the like. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, and the like for inputting data. The output device 206 is, for example, a display, a printer, a speaker, and the like for outputting data.

[0020] 3 is a diagram showing an example of a software configuration of the server device 20. As shown in the figure, the server device 20 includes functional units, namely, a user authentication unit 210, a training information output unit 211, an image acquisition unit 212, an image analysis unit 213, a frequency count unit 214, an evaluation unit 215, an evaluation information output unit 216, a training history transmission unit 217, and an improvement measure information output unit 218, and storage units, namely, a training information storage unit 231, a user information storage unit 232, an image data storage unit 233, a reference information storage unit 234, an evaluation condition information storage unit 235, a training plan storage unit 236, a training history storage unit 237, and an improvement condition storage unit 238.

[0021] Each of the above-mentioned functional units is realized by the arithmetic unit 201 of the server device 20 reading out a program stored in the storage device 203 into the memory 202 and executing it, and each of the above-mentioned storage units is realized as part of the storage area provided by the memory 202 and storage device 203 of the server device 20.

[0022] The training information storage unit 231 stores information about training. The information about training is, for example, a description of training. The description of training includes how to use training equipment, and an effective way of training using the training equipment. FIG. 4 is a diagram showing an example of the configuration of the training information storage unit 231. As shown in the figure, the training information storage unit 231 stores training information and display information. The training information is information about training. The training information includes information for identifying a training menu (menu ID) and video data showing the content of training, the number of times, and the way of training in association with the purpose of training. The purpose of training indicates the results desired from training, such as resolving lack of exercise and increasing muscle mass. The content of training can be set to any information, such as how to use training equipment and points to note about training. The video data is, for example, a video of a person demonstrating how to train, a tutorial video showing how to use a training machine, and the like. The content of training and the video data are linked to the purpose of training, and are prepared to suggest what kind of training should be done and how many times to do it in order to achieve the purpose using the training equipment. The display information is information for setting a training menu to be performed while watching the display 22. The display information includes a menu ID associated with a display ID that identifies the display 22.

[0023] The user information storage unit 232 stores information about the user 1 (hereinafter, referred to as user information). FIG. 5 is a diagram showing an example of the configuration of the user information storage unit 232. As shown in the figure, the user information stored in the user information storage unit 232 includes authentication information, height, weight, muscle mass, flexibility, and the like, in association with a user ID that identifies the user 1. The authentication information is information that can be used to authenticate the user 1. The authentication information may be, for example, a feature amount of the face part of the user 1 for use in face authentication, a device ID of the user terminal 10, a combination of a user name and a password, or a public key of the user 1, but is not limited to these as long as it is information that can authenticate the user 1. In addition to the above items, the user information may include various information about the user's body that can be measured or evaluated.

[0024] 6 is a diagram showing an example of an input screen 41 for inputting user information. For example, the input screen 41 as shown in the figure may be displayed on the user terminal 10, and the user 1 may input each item 411 of the user information. In this case, each item of the user information may be transmitted from the user terminal 10 to the server device 20 and registered in the user information storage unit 232.

[0025] The image data storage unit 233 stores images captured by the camera 21. Fig. 7 is a diagram showing an example of the configuration of the image data storage unit 233. As shown in the figure, the image data storage unit 233 stores image information, and the image information includes image data associated with a camera ID that identifies the camera 21 that captured the image. The image data is data for displaying the image captured by the camera 21. The image data can be data in any format, such as MP4, MPEG2, AVI, or MOV.

[0026] The reference information storage unit 234 stores information (hereinafter referred to as reference information) including reference values ​​of postures (positions and angles of body parts) related to the physical movement of the user 1. Fig. 8 is a diagram showing an example of the configuration of the reference information storage unit 234. As shown in the figure, the reference information storage unit 234 stores reference information (hereinafter referred to as position reference information) related to absolute positions of parts or relative positions with respect to other parts or other reference objects, and, for three parts including a joint part, reference information (hereinafter referred to as angle reference information) of angles formed by straight lines connecting each of two parts with the joint part.

[0027] The position reference information includes a part and a reference position of the part in association with a mode and a checkpoint ID. The checkpoint ID is information indicating a point (hereinafter, referred to as a checkpoint) where posture, etc. should be checked in a series of movements of the user's body. There may be a plurality of parts. Regarding the position, the vertical position may be, for example, the height from the ground or the distance from one of the toes. The horizontal position may be the distance from a predetermined reference object (for example, a mark on the floor or an installed instrument) or the distance from a reference part such as the shoulder, chest, or foot. It is assumed that the position reference information is registered in advance.

[0028] The angle reference information corresponds to a mode and a checkpoint ID and includes two parts (part 1 and part 2), one joint part, and a reference value of the angle between a line connecting part 1 and the joint part, and a line connecting part 2 and the joint part.

[0029] The evaluation condition information storage unit 235 stores information for performing evaluation (hereinafter, referred to as evaluation condition information). FIG. 9 is a diagram showing an example of the configuration of the evaluation condition information storage unit 235. The evaluation condition information includes a category, a condition, an evaluation rank, and a comment. The category is an evaluation category. For example, the category can be "arm curl", "clean", "press", and the like. The condition is a condition for the position or movement (change in position in time series) of each part of the body in the image. For example, when analyzing the movement of weightlifting, conditions for the angle of the elbow, the angle of the waist, and the like can be set in the evaluation condition information for the checkpoint for cleaning the weight. The evaluation rank is an evaluation value when the above condition is met. The comment is an explanation of the posture and movement of the body when the above condition is met.

[0030] The training plan storage unit 236 stores a training plan set by the user 1 or his / her supporter. FIG. 10 is a diagram showing an example of the configuration of the training plan storage unit 236. The training plan storage unit 236 stores training plans, and each training plan includes a menu ID indicating a training menu, settings of training equipment and the like (e.g., weight, etc.), and the number of training sessions (e.g., repeated actions such as lifting weights or doing squats), in association with a user ID indicating the user 1 and a date. It is assumed that the training plan is set in advance by the user 1 or his / her supporter.

[0031] The training history storage unit 237 stores the training history of the user 1. Fig. 11 is a diagram showing an example of the configuration of the training history storage unit 237. As shown in the figure, the training history storage unit 237 stores training history, and each training history includes a menu ID indicating the training menu performed by the user 1, setting information indicating the setting contents of the training equipment, etc., the number of training sessions, the time for which the training was performed, etc., associated with a user ID indicating the user 1 and the date and time when the user 1 performed the training.

[0032] The improvement condition storage unit 238 stores information related to improvement measures for training (hereinafter, referred to as improvement condition information). FIG. 12 is a diagram showing an example of the configuration of the improvement condition storage unit 238. As shown in the figure, the improvement condition information stored in the improvement condition storage unit 238 includes advice associated with a menu ID, a purpose, a category, and a condition. The condition may be a condition for a body condition (such as flexibility), or a condition for the position or movement of a body part. In this embodiment, the advice is assumed to be a character string representing the improvement measure, but it may also be content that presents the improvement measure by an image, video, or the like.

[0033] The user authentication unit 210 authenticates the user 1 who is looking at the display 22. The user authentication unit 210 can authenticate the user 1 using authentication information of the user information. In this embodiment, it is assumed that the authentication information of the user information is a feature amount of the face part of the user 1, and the user authentication unit 210 authenticates the user 1 by extracting the feature amount from the image captured by the camera 21. Note that the user authentication unit 210 may, for example, communicate with the user terminal 10 to authenticate the user 1, or the display 22 may authenticate the user 1 by communicating with the user terminal 10, and the result may be transmitted from the display 22 to the user authentication unit 210.

[0034] The training information output unit 211 outputs training information to the display 22. The training information output unit 211 identifies a menu ID corresponding to a display ID indicating the display 22 from the display information, and outputs the contents of the training information and video data corresponding to the identified menu ID to the display 22. The training information output unit 211 can output training information to the display 22 when the user 1 comes near the display 22 (for example, the display 22 is provided with a human sensor, and the human sensor detects the user 1, or the display 22 is provided with a communication function for communicating with the user terminal 10, and the display 22 detects that it is able to communicate with the user terminal 10, thereby detecting that the user 1 is in the vicinity). The user 1 can refer to the training contents and video output to the display 22 before or during training by viewing the training contents and video output to the display 22.

[0035] 13 is a diagram showing an example of a training information output screen 42. A training objective 421, training content 422, and video 423 are displayed on the output screen 42. A plurality of video data may be included in the training information, and a plurality of videos may be displayed on the output screen 42.

[0036] The training information output unit 211 may correct the content of the training information based on the physical information. For example, the training information output unit 211 can change the height of the chair of the training equipment according to the height, or set the recommended weight according to the height and weight.

[0037] Furthermore, the training information output unit 211 may correct the contents of the training information according to the past training history of the user 1. For example, the training information output unit 211 may read out the training history corresponding to the user ID indicating the user 1 from the training history storage unit 237, and depending on the setting information (weight of the weight, height of the chair, etc.) and the number of times included in the read training history, include in the training information the same information as the content of the previous setting information, or a weight that is a predetermined value heavier than the previous weight, or a number of times that is a predetermined value more than the previous number of times, for example.

[0038] Image acquisition unit 212 acquires an image captured by camera 21. In this embodiment, camera 21 transmits captured image data to server device 20 by communication, and image acquisition unit 212 registers the image data transmitted from camera 21 in image data storage unit 233 in association with a camera ID indicating camera 21. Note that image acquisition unit 212 may transmit an instruction to camera 21 when user authentication unit 210 has successfully authenticated user 1, causing camera 21 to start capturing images.

[0039] The image analysis unit 213 (part identification unit) analyzes the image data. The image analysis unit 213 analyzes the image data to extract the feature amount of each part of the body and identifies the position of each part in the image. Note that a general method of image analysis is adopted by the image analysis unit 213, and detailed explanations are omitted here. The image analysis unit 213 may analyze the image data for each frame or each key frame, or may analyze at random timing.

[0040] The image analysis unit 213 also compares, for each checkpoint ID, the position of each part extracted from the image data with the position reference information stored in the reference information storage unit 232, and identifies the closest time as the checkpoint time.

[0041] The image analysis unit 213 can also recognize the body movements of the user 1. In this case, the image analysis unit 213 can create a model capable of recognizing body movements (such as squats and lift-ups) from video data by machine learning, for example, and recognize the movements using the model. In this case, the image analysis unit 213 may be realized by performing parallel processing using multiple GPUs, for example.

[0042] The number of times counting unit 214 counts the number of training sessions. The number of times counting unit 214 can count the number of times the user 1 changes to a predetermined posture (from another posture) using the position of each part analyzed by the image analysis unit 213. For example, the number of times counting unit 214 can count up when the distance between the position of each part analyzed by the image analysis unit 213 and the position included in the position reference information is within a predetermined value. Also, the number of times counting unit 214 may be configured to count up when the angle between two parts with a joint part included in the angle reference information as the vertex is within a predetermined value. Also, the number of times counting unit 214 may prepare in advance a model created by machine learning using the posture of the body as teacher data and the position (relative positional relationship) of each part as input data, identify the posture of the user 1 using the model, and count up when the user 1 takes a predetermined posture.

[0043] The repetition counting unit 214 can also count the number of training sessions and the number of training sets, multiply these values ​​by the weights, and calculate the training volume to calculate the workload of the user 1. This training volume can be used as an index for measuring the fatigue level of the user 1. By storing this training volume in the training history storage unit 237, the user 1 can display and check this training volume and its progress on the display 22 or the like at any desired time.

[0044] The evaluation unit 215 evaluates the body movement of the user 1 based on the image data. In this embodiment, the evaluation unit 215 searches the evaluation condition storage unit 235 for evaluation condition information including conditions satisfied by the position of each part and the movement of each part identified from the image data, and if there is evaluation condition information for which the conditions are satisfied, acquires the evaluation rank and comments included therein.

[0045] 14 is a diagram showing an example of the configuration of evaluation information created by the evaluation unit 215. The evaluation information includes a menu ID, a user ID, position information, posture information, movement information, and checkpoint information.

[0046] The position information indicates the position of each part of the body (for example, head, shoulders, elbows, waist, knees, ankles, etc.) in the image. The position information includes the part and the position of the part in association with a time point on the time axis of the video. The state of the body skeleton (bones) can be displayed based on the position information. That is, for example, a figure indicating the part (for example, a circle, etc.) can be displayed by superimposing it on the image at the position indicated by the position information. Note that the positions of multiple parts can be included for one time point. Note that the position information does not need to be included for a part connecting two parts (for example, a forearm connecting a wrist and an elbow, or a thigh connecting a waist and a knee, etc.). In this case, the part connecting these two parts can be expressed by connecting a pair of marks (for example, a circle, etc.) indicating two predetermined parts with a line. The position information may be included for each frame constituting the video, may be included for each key frame, may be included for every arbitrary number of frames, or may be included for a random time point. If the position information is not included in My Frame, the bones can be displayed based on the position information of the most recent past time point.

[0047] The posture information is information related to the posture of the user's body. The posture information includes a part to be evaluated, a posture value, an evaluation rank, and an evaluation comment, which are associated with a time point on the time axis of the video. The posture value is a value that represents the posture. The posture value is, for example, the distance from the ground to a part, the distance between two parts, the angle of the joint part (the angle formed by a straight line from the part at the first end to the joint part and a straight line from the part at the second end to the joint part), etc. The evaluation rank is a value that represents the evaluation value by a rank. The evaluation rank is expressed, for example, from 1 to 5 on a 5-point scale, A, B, C, etc. The evaluation comment is a comment related to the evaluation regarding the posture. For example, when the mode is "lifting" and bending is not sufficient, an evaluation comment such as "your hips are not lowered" may be included.

[0048] The movement information is information related to the movement of the user's body. The movement information includes the part to be evaluated, a list of posture values, an evaluation rank, and an evaluation comment, which are associated with a period on the time axis of the video. The list of posture values ​​is a time series of posture values ​​within a period. The evaluation comment is a comment related to the evaluation of the movement. For example, when the mode is "lifting" and the knee extension is not smooth, an evaluation comment such as "the knee movement is not smooth" may be included.

[0049] Checkpoint information is information about a checkpoint. For example, in the case of a weightlifting training menu, checkpoints include gripping the weight bar, standing upright while holding the weight, and raising the weight to chest height. The checkpoint information stores the checkpoint in association with a time point on the time axis of the video. In other words, it is possible to identify the frame (still image) in the video in which the checkpoint indicated by the checkpoint ID is displayed.

[0050] The evaluation unit 215 can generate position information including the time point on the time axis of the video identified by the image analysis unit 212 and the position of each part. If the position of the part satisfies the conditions for the acquired evaluation rank and comment, the evaluation unit 215 generates posture information including the time point, part, and posture value, as well as the evaluation rank and comment, and if the movement of the part (change in position in time series) satisfies the conditions, the evaluation unit 215 generates movement information including a list of the time point, part, and posture value, as well as the evaluation rank and comment. The evaluation unit 215 also generates checkpoint information including the time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint.

[0051] The evaluation information output unit 216 outputs the evaluation information to the display 22. For example, the evaluation display unit 114 can display a figure representing a body part (for example, a circle representing an end or joint and a line connecting them) superimposed on the moving image based on the position information included in the evaluation information, thereby superimposing the bone movement on the moving image. Furthermore, the evaluation information output unit 216 can display, for example, a graph of the time-series change in the position of the part.

[0052] Furthermore, the evaluation display unit 114 can display an evaluation rank and an evaluation comment together with the display of the video based on the posture information and the movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and the evaluation comment included in the posture information when the playback time of the video reaches a point around the time point included in the posture information (for example, it can be any length such as around 5 seconds). Furthermore, the evaluation display unit 114 can display the evaluation rank and the evaluation comment included in the movement information when the playback time of the video reaches a period included in the movement information. Furthermore, the evaluation display unit 114 can display the posture values ​​included in the posture information. Furthermore, the evaluation display unit 114 can display a graph of the time-series conversion of the posture values ​​based on a list of the posture values ​​included in the movement information.

[0053] 15, 16, and 17 are diagrams showing an example of a screen 42 output by the evaluation information output unit 216. In FIG. 15, bones 422 are displayed on the screen 42 by superimposing them on image data 421 of the user 1. In the example of FIG. 15, an evaluation comment 423 of the posture information and an evaluation comment 424 of the movement information are displayed on the screen 42. Furthermore, in the example of the screen 42, a number of times 425 of the training information and a number of times 426 counted by the number counting unit 214 are displayed. In the example of FIG. 16, an example of a graph 427 showing the speed of the shoulders, wrists, ankles, etc. is shown. In the example of FIG. 17, video data 423 of the training information is displayed. By displaying such a screen 42 on the display 22, the user 1 can correct the posture and movement of the body during training and perform efficient and effective training.

[0054] The evaluation information output section 216 can also transmit the evaluation information to the user terminal 10. The user 1 can use the user terminal 10 to review the training during and after the training.

[0055] The evaluation information output unit 216 registers the history of training by the user 1 in the training history storage unit 237. The evaluation information output unit 216 can register in the training history storage unit 237 a training history including a user ID indicating the user 1, the date and time when the user authentication unit 210 authenticated the user 1, a menu ID corresponding to the display 22, setting information, the number of times counted by the number of times counting unit 214, and the time from the time when the user was authenticated to the end of training. Regarding the setting information, the display 22 may accept an input of the setting information from the user 1 (for example, the display 22 may be a touch panel display, and options or the like may be displayed on the touch panel display to accept the input) and receive the setting information from the display 22, or the user terminal 10 may accept an input of the setting information and receive the setting information from the user terminal 10, or the training equipment may be provided with a communication function and acquire the setting information from the training equipment.

[0056] The training history transmission unit 217 transmits the training history to the user terminal 10. The training history transmission unit 217 may transmit training information to the user terminal 10 every time training is completed, or may respond with the training history in response to a request from the user terminal 10. The user terminal 10 can display the training history. FIG. 18 is a diagram showing an example of the training history displayed by the user terminal 10. On a screen 43 shown in FIG. 18, details 431 of the training performed by the user 1 and a video 432 of the user 1 during the training are displayed. In this way, the user 1 can look back on the state during the training later, which enables effective training.

[0057] The training history transmission unit 217 transmits the training volume to the user terminal 10. An example of a training volume information output screen 44 is shown in a screen 44 in FIG. 19. As shown in FIG. 19, the training volume can be displayed along with the user's daily physical information such as height, weight, etc., over a daily, weekly, or monthly period. Also, as shown in FIG. 19, by simultaneously displaying the training volume information along with the user's fat percentage, the user can compare the fat percentage change against the workload.

[0058] Furthermore, the training history transmission unit 217 can also transmit the training history to the manager terminal 30. The training history transmission unit 217 may transmit training information to the manager terminal 10 every time training ends, or may respond with the training history in response to a request from the manager terminal 30.

[0059] The improvement information output unit 218 outputs the improvement measures for the training to the display 22. The improvement information output unit 218 searches for improvement condition information corresponding to the menu ID corresponding to the display 22, in which the physical information of the user 1 and the position and movement of each part specified by the image analysis unit 213 satisfy the conditions. The improvement information output unit 218 acquires advice for the searched improvement information, and outputs the acquired advice to the display 22. The improvement information output unit 218 can also output the position and angle of each part included in the reference information to the display 22. For example, the improvement information output unit 218 can display the state of the body skeleton (bones) based on the position and angle of each part included in the reference position. That is, for example, a figure (such as a circle) indicating a part can be displayed superimposed on an image at a position indicated by the position information.

[0060] FIG. 20 is a diagram showing the flow of processing executed in the exercise support system of this embodiment.

[0061] The user authentication unit 210 of the server device 20 detects that the user 1 has come near the training equipment (S501). The user 1 may be detected, for example, by installing a human sensor near the display 22, or by monitoring an image from the camera 21 and detecting a human from the image.

[0062] Next, the user authentication unit 210 authenticates the user 1 (S502). The authentication of the user 1 can be performed, for example, by extracting features of the face part from the image and comparing them with the features included in the authentication information of the user information. The user authentication unit 210 may also authenticate the user 1 by, for example, communication with the user terminal 10 (for example, by accepting input of authentication information such as a user ID and password or a public key from the user terminal 10). An IC card reader may also be installed near the display 22 to perform authentication using a membership card.

[0063] When the user 1 is authenticated, the image acquisition unit 212 starts acquiring an image from the camera 21 and registering it in the image data storage unit 233 (S503). The training information output unit 211 identifies a menu ID corresponding to the display 22 from the training information storage unit 231, reads out training information corresponding to the identified menu ID from the training information storage unit 231 (S504), and modifies the read out training information using user information corresponding to the user 1 (S505). The training information output unit 211 can modify the weight and the number of repetitions according to, for example, height, weight, muscle mass, and the like. The training information output unit 211 may also modify the training information according to the training history of the user 1. The training information output unit 211 outputs the modified training information to the display 22 (S506). The training information output unit 211 may accept an input of an objective from the user 1 here, and read out training information corresponding to the objective and the menu ID. An example of the output of training information is, for example, the screen 42 shown in FIG. 13.

[0064] The image analysis unit 213 analyzes the image data acquired from the camera 21 to identify the position of each part of the body of the user 1 (S507). The number of times counting unit 214 counts the number of times the user 1 has performed the training based on the positions of the parts (S508). For example, the number of times counting unit 214 can count one leg bending and straightening session when the knee angle is within a predetermined angle, or count one squat session when the position of the buttocks is within a predetermined distance above the floor.

[0065] The evaluation information output unit 216 displays the position of each part and its connection relationship superimposed on the captured image of the user 1, that is, displays the so-called bones (S509). The evaluation unit 215 also evaluates the training based on the positions and movements of the parts (time-series changes in position or angle), and the evaluation information output unit 216 outputs the evaluation results to the display 22 (S510). Here, if the user 1 is wearing an audio output device such as earphones, headphones, or a neck speaker, the evaluation results can also be output as audio.

[0066] The improvement measure information output unit 218 searches the improvement condition storage unit 238 for improvement condition information that satisfies the conditions of the physical information of the user 1 and the position and movement of each part identified by the image analysis unit 213, and outputs advice of the searched improvement condition information to the display 22 (S511). Note that the improvement measure information output unit 218 may output other items included in the improvement condition information to the display 22. Furthermore, if the user 1 is wearing an audio output device such as earphones, headphones, or a neck speaker, information such as advice included in the improvement condition information may be output as audio.

[0067] Finally, the number counting unit 214 can register the training history in the training history storage unit 237 (S512). The number counting unit 214 may register the history every time it counts the number of training sessions, or when it detects the end of training due to the lapse of a predetermined time after it becomes impossible to count the training sessions, etc.

[0068] As described above, according to the exercise support system of this embodiment, information such as an explanation of the training is displayed on the display 22 installed on each training machine, and the body of the user 1 photographed by the camera 21 is analyzed to evaluate the posture and movement and provide improvement measures. Therefore, since the system can explain the equipment and evaluate and give advice on the movement of the body, which have conventionally been done manually by personal trainers, it is possible to make the instruction of the user 1 more efficient.

[0069] Moreover, according to the exercise support system of this embodiment, physical exercise can be easily evaluated. Since the positional relationship and movement of each part of the body can be evaluated for physical exercise related to training, it is easy to lead to concrete efforts for improvement, and it is expected that the training effect will be improved. Furthermore, the exercise support system of this embodiment also provides comments and advice, so that the user can easily understand the current situation and improvement measures.

[0070] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

[0071] For example, in this embodiment, the image analysis is performed in the server device 20, but this is not limiting. The image analysis may be performed in the camera 21 or the user terminal 10 to identify the positional relationship of each part.

[0072] In addition, in this embodiment, the position of the part is assumed to be a position on a two-dimensional image, but it is not limited to this and may be a three-dimensional position. For example, in addition to the camera 21, a depth camera is placed at the installation location of the training equipment, and the three-dimensional position of the part can be specified based on the image from the camera 21 and the depth map from the depth camera. It is also possible to provide a depth camera instead of the camera 21 and specify the three-dimensional position only from the depth map from the depth camera.

[0073] In this embodiment, it is assumed that the parts of the user's body photographed by the camera are parts seen from the front or side of the user, but this is not limiting, and it is also possible to take advantage of the characteristics of a camera installed above the user and photograph parts seen from above the user's head. In this case, the two-dimensional positions of the center of the user's head and both shoulders as viewed from above can be identified and analyzed.

[0074] In addition, in this embodiment, an image of the user's body during exercise is transmitted from camera 21 to server device 20, but this is not limited to the above. A computer may be placed at the location where the training equipment is installed, and the computer may extract features from the image and transmit the features to server device 20. Alternatively, the computer may estimate body parts based on the features and obtain the absolute position of the part (which may be the position on the XY coordinate system of the image, or the actual distance from a reference position (for example, the ground, the toes, the head, the center of gravity of the body, etc.), or the position in any other coordinate system) or the relative positional relationship between multiple parts, and transmit these absolute positions and relative positional relationships to server device 20.

[0075] In addition, in this embodiment, the improvement information is provided by the content prepared by the server device 20, but it is not limited to this. For example, it may include a reference value, and the bones that will result in the correct posture (position and angle of each part) based on the reference value may be superimposed on a video or a still image extracted from the video. This makes it easy to understand what posture the user should take.

[0076] In addition, in this embodiment, the position or movement (position over time) of a body part is evaluated, but this is not limiting, and the position of a tool held or worn by the user may be identified and evaluated. In this case, reference information and evaluation condition information regarding the tool position may be registered, and the feature amount of the tool may be extracted to identify the position, and the evaluation may be performed in the same manner as for the position of a body part.

[0077] In addition, in this embodiment, the content such as advice is provided as a remedy, but for example, equipment recommendations may be made. In this case, the server device 20 stores the equipment and the standard value of the size (length, etc.) of the equipment in association with the user's physical information (height, weight, etc.), extracts the feature amount of the equipment used by the user from the image data to identify the shape of the equipment, estimates the size of the equipment based on the shape and the user's size (height, etc.) included in the physical information, and if the difference between the estimated equipment size and the standard value is equal to or greater than a predetermined threshold, recommends equipment of the standard size.

[0078] Furthermore, by estimating the position of the tool, the server device 20 can estimate the speed of the tool. Furthermore, by extracting a pattern of the change in the position of the tool over time, the server device 20 can estimate the number of times the pattern occurs as the number of actions using the tool.

[0079] Also, when a certain posture or movement is detected, a task for that movement may be proposed. In this case, the server device 20 may store a task in association with one or a series of postures or movements, instead of an evaluation comment, and output the task.

[0080] Also, when a certain posture or movement is detected, a training to be performed may be presented. In this case, the server device 20 may store training contents in association with one or a series of postures or movements, instead of evaluation comments, and output the contents.

[0081] It is also possible to automatically detect the action performed by the player. In this case, the server device 20 stores the posture (position of each body part) of a player performing a predetermined action such as shooting or passing as reference information, and compares the positions of the body parts analyzed from the image with the reference information to identify the action performed by the player in the image.

[0082] In addition, in this embodiment, the execution of a predetermined function and the storage of information are performed by the user terminal or the server device, but this is not limited thereto, and the execution of the function and the storage of information may be performed by either one of the devices. Alternatively, the function unit and the storage unit may be provided in a distributed manner in a form different from that of this embodiment.

[0083] In addition, in this embodiment, the reference information (position reference information and angle reference information) is set in advance, but this is not limiting, and the reference information may be created from image data obtained by photographing a plurality of users 1. In this case, for example, input as to whether the posture or movement of the user 1 is correct or not may be received, and the reference information may be created based only on those evaluated as correct.

[0084] In addition, in this embodiment, information about the training, evaluation results, and the like are displayed on the display 22, or are output as audio from an audio output device worn by the user 1, but a message encouraging the training may be displayed on the display 22, or a voice encouraging the training may be output as audio from an audio output device worn by the user 1. This makes it possible to improve the motivation of the user 1 for training. [Explanation of symbols]

[0085] 10 User terminal 20 Server device 21 Camera 22 Display 23 Communication Network Sp 30 Administrator terminal 210 User authentication section 211 Training information output section 212 Image acquisition unit 213 Image Analysis Unit 214 Number of times counting unit 215 Evaluation Department 216 Evaluation Information Transmission Unit 231 Training information storage unit 232 User information storage unit 233 Image data storage unit 234 Standard information storage unit 235 Evaluation condition information storage unit 236 Training Plan Memory Unit 237 Training History Memory Unit 238 Improvement condition storage section

Claims

1. A system for assisting a user in exercising, comprising: a camera that photographs the user while exercising; A display installed in a place where the exercise is performed; an explanation output unit that outputs an explanation regarding exercise to the display; a body part identification unit that analyzes an image captured by the camera to identify a position of a body part of the user; an output unit that superimposes the position of the part on the image and outputs the image to the display; an improvement information storage unit in which improvement information including advice consisting of character strings excluding character strings consisting of numerical values ​​is stored in association with a condition of a position or movement of the body part of the user; Equipped with the output unit acquires, from the remedy information storage unit, the remedy information including advice that the position or movement of the part identified by the identification unit satisfies a condition, and presents the remedy information; An exercise support system characterized by the above.

2. The exercise support system according to claim 1 , the explanation output unit outputs, to the display, an explanation including at least video data showing a method of training; An exercise support system characterized by the above.

3. The exercise support system according to claim 2, a reference value storage unit that stores a reference value related to the position of at least one of the parts; an evaluation unit that compares the position of the part in the image with the reference value to determine an evaluation value of the movement; Equipped with the output unit displays, as improvement measure information, a state of the body skeleton based on the reference value by superimposing it on the image; An exercise support system characterized by the above.

4. The exercise support system according to claim 3, the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion; the part identification unit identifies each of the first to third parts from the image; the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with a reference value to determine the evaluation value; An exercise support system characterized by the above.

5. 5. The exercise support system according to claim 1, A number counting unit that counts the number of times of the exercise based on the position of the part is further provided, The output unit further outputs the number of times of the exercise to the display. An exercise support system characterized by the above.

6. The exercise support system according to claim 3 or 4, The output unit further outputs the evaluation value by voice. An exercise support system characterized by the above.

7. 5. The exercise support system according to claim 1, the output unit further outputs the remedy information by voice based on the position of the user's body identified by the part identification unit; An exercise support system characterized by the above.

8. A camera that photographs a user while the user is exercising; A display installed in the exercise area; A processor; A program for supporting the user's exercise by an exercise support system comprising: The processor, an explanation output step of outputting an explanation regarding the exercise to the display; a part identification step of analyzing an image captured by the camera to identify a position of a part of the user's body; an output step of superimposing the position of the part on the image and outputting the image to the display; an improvement information storage step of storing improvement information including advice consisting of character strings excluding character strings consisting of numerical values, in association with the condition of the position or movement of the body part; Run the command, the output step includes acquiring, from the remedy information storage step, the remedy information including advice that the position or movement of the part identified in the identification step satisfies a condition, and presenting the remedy information; An exercise support program that features:

9. A camera that photographs a user while the user is exercising; A display installed in the exercise area; A processor; A method for supporting the exercise of the user by an exercise support system comprising: The processor, an explanation output step of outputting an explanation regarding the exercise to the display; a part identification step of analyzing an image captured by the camera to identify a position of a part of the user's body; an output step of superimposing the position of the part on the image and outputting the image to the display; an improvement information storage step of storing improvement information including advice consisting of character strings excluding character strings consisting of numerical values, in association with the condition of the position or movement of the body part; Run the command, the output step includes acquiring, from the remedy information storage step, the remedy information including advice that the position or movement of the part identified in the identification step satisfies a condition, and presenting the remedy information; An exercise support method comprising:

Images