Exercise evaluation system

Abstract

To easily evaluate physical exercise. [Solution] An exercise evaluation system that evaluates physical exercise includes a reference value storage unit that stores a reference value related to the position of at least one body part. The body is imaged while exercising. The exercise evaluation system includes a part identification unit that analyzes the image of the body and identifies the body part in the image. The exercise evaluation system also includes an evaluation unit that compares the position of the part in the image with the reference value to determine an evaluation value for the exercise.

Description

[Technical field]

[0001] The present invention relates to an exercise evaluation system. [Background technology]

[0002] Conventional analytical techniques in the field of physical movement include a technique (Patent Document 1) for extracting movement patterns that appear repeatedly in a video, comparing the movement patterns at multiple positions with different time positions in the extracted video from a predetermined viewpoint, and outputting the comparison results as a video or still image. Also disclosed in the field of sports, for example, is a technique for analyzing a golf wing using a motion sensor by attaching a device to the user's body in golf (Patent Document 2), and a technique for capturing images of each stage of a pitch (placement of the pivot foot, horizontal rotation of the trunk) with two cameras and calculating the transition time (Patent Document 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-229081 A [Patent Document 2] JP 2010-068947 A [Patent Document 3] JP 2018-114138 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, all of the technologies in Patent Documents 1 to 3 require equipment and resources such as multiple video images, specific devices, and multiple cameras, making it difficult for a single user to use these devices to analyze physical movements and understand the analysis results.

[0005] The present invention has been made in view of the above background, and has an object to provide a technique that can easily evaluate physical exercise. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problems is a system for evaluating body movement, comprising a reference value memory unit that stores a reference value relating to the position of at least one of the body parts, a body part identification unit that analyzes an image of the body to identify the body part, and an evaluation unit that compares the position of the body part in the image with the reference value to determine an evaluation value of the movement.

[0007] Other problems and solutions disclosed in this application will be made clear in the description of the preferred embodiments of the invention and the drawings. Effect of the Invention

[0008] According to the present invention, physical exercise can be easily evaluated. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an exercise evaluation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal 10. [Diagram 3] FIG. 2 is a diagram illustrating an example of the software configuration of the user terminal 10. [Figure 4] 4 is a diagram showing an example of the configuration of physical information stored in a physical information storage unit 130. FIG. [Diagram 5] 13 is a diagram showing an example of the configuration of a rating request that a rating request sending unit 112 sends to a server device 20. FIG. [Figure 6] 10 is a diagram showing an example of the structure of evaluation information received by an evaluation information receiving unit 113 from a server device 20. FIG. [Figure 7] 13 is a diagram showing an example of the configuration of a remedy request that a remedy request sending unit 116 sends to a server device 20. FIG. [Figure 8] FIG. 13 is a diagram illustrating an example of a configuration of improvement measure information. [Figure 9] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device 20. [Figure 10] FIG. 2 is a diagram illustrating an example of the software configuration of the server device 20. [Figure 11] 13 is a diagram showing an example of the configuration of image information stored in an image data storage unit 231. FIG. [Figure 12] 13 is a diagram showing an example of the configuration of reference information stored in a reference information storage unit 232. FIG. [Figure 13] 13 is a diagram showing an example of the configuration of evaluation condition information stored in an evaluation condition information storage unit 233. FIG. [Figure 14] 13 is a diagram showing an example of the configuration of improvement condition information stored in an improvement condition storage unit 234. FIG. [Figure 15] FIG. 2 is a diagram showing a flow of processes executed in the exercise evaluation system of the present embodiment. [Figure 16] 13A and 13B are diagrams illustrating a case where a moving image is captured in a pitching mode. [Figure 17] FIG. 13 is a diagram showing an example of displaying an image of a checkpoint. [Figure 18] FIG. 13 is a diagram showing an example of a video evaluation screen in a weightlifting mode. [Figure 19] FIG. 13 is a diagram showing an example of a video evaluation screen in a weightlifting mode. [Figure 20] FIG. 13 is a diagram showing an example of a video evaluation screen in a weightlifting mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The contents of the embodiments of the present invention will be described below. An exercise evaluation system according to an embodiment of the present invention has the following configuration.

[0011] [Item 1] A system for evaluating physical exercise, comprising: a reference value storage unit that stores a reference value relating to the position of at least one of the parts; a part identification unit that identifies a part of the body by analyzing an image of the body; an evaluation unit that compares the position of the part in the image with the reference value to determine an evaluation value of the movement; An exercise evaluation system comprising: [Item 2] The exercise evaluation system according to item 1, the evaluation unit determines the evaluation value according to a distance from the reference value to the position; An exercise evaluation system comprising: [Item 3] The exercise evaluation system according to item 1, the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion; the region identifying unit identifies each of the first to third regions from the image; the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with the reference value to determine the evaluation value; An exercise evaluation system comprising: [Item 4] The exercise evaluation system according to item 1, a comment storage unit that stores a comment about the part in association with a condition for the part and the position; a comment output unit that outputs the comment corresponding to the condition satisfied by the position; The exercise evaluation system further comprising: [Item 5] The exercise evaluation system according to item 1, an improvement information storage unit that stores advice on an improvement measure related to the exercise in association with a goal of improvement related to the exercise and a condition for the position; an advice output unit that outputs the advice corresponding to the condition satisfied by the position; The exercise evaluation system further comprising:

[0012] An exercise evaluation system according to one embodiment of the present invention is intended to evaluate a user's physical exercise. Examples of the user's physical exercise include baseball pitching and weightlifting. The exercise evaluation system of this embodiment identifies body parts from images (which may be still images or moving images, but are assumed to be moving images in this embodiment) captured of the user performing physical exercise, and evaluates the posture and movement of the body based on the absolute position of the part and the relative positional relationship between multiple parts.

[0013] 1 is a diagram showing an example of the overall configuration of an exercise evaluation system according to this embodiment. As shown in the figure, the exercise evaluation system according to this embodiment includes a user terminal 10 and a server device 20. The user terminal 10 and the server device 20 are communicably connected to each other via a communication network 30. The communication network 30 is, for example, the Internet or a LAN (Local Area Network), and is constructed by a public telephone line network, a dedicated telephone line network, a mobile phone line network, Ethernet (registered trademark), a wireless communication path, or the like.

[0014] The user terminal 10 is a computer operated by a user performing physical exercise or by his / her supporter. The user terminal 10 is, for example, a smartphone, a tablet computer, a personal computer, or the like. The user terminal 10 is equipped with an imaging device such as a camera, which can capture images of the user's body during exercise. In this embodiment, a video image of the user's body during exercise is transmitted from the user terminal 10 to the server device 20.

[0015] The server device 20 is a computer that evaluates physical exercise. The server device 20 is, for example, a workstation, a personal computer, or a virtual computer logically realized by cloud computing. The server device 20 receives moving images captured by the user terminal 10, analyzes the received moving images, and evaluates the physical exercise. The server device 20 also makes suggestions regarding measures to improve the physical exercise. Details of the evaluation of physical exercise and the suggestions of measures to improve the physical exercise will be described later.

[0016] FIG. 2 is a diagram showing an example of a hardware configuration of the user terminal 10. The user terminal 10 includes a CPU 101, a memory 102, a storage device 103, a communication interface 104, a touch panel display 105, and a camera 106. The storage device 103 is, for example, a hard disk drive, a solid state drive, or a flash memory that stores various data and programs. The communication interface 104 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, or the like. The touch panel display 105 is a device for inputting and outputting data. The user terminal 10 may further include input devices such as a keyboard, a mouse, buttons, and a microphone, and output devices such as a speaker and a printer.

[0017] 3 is a diagram showing an example of the software configuration of the user terminal 10. The user terminal 10 includes functional units, namely, an imaging unit 111, an evaluation request transmitting unit 112, an evaluation information receiving unit 113, an evaluation display unit 114, a checkpoint display unit 115, an improvement request transmitting unit 116, an improvement information receiving unit 117, and an improvement display unit 118, as well as storage units, namely, a physical information storage unit 130, an image storage unit 131, an evaluation information storage unit 132, and an improvement storage unit 133.

[0018] Each of the above functional units is realized by the CPU 101 of the user terminal 10 reading out a program stored in the storage device 103 into the memory 102 and executing it, and each of the above storage units is realized as part of the storage area provided by the memory 102 and storage device 103 of the user terminal 10.

[0019] The imaging unit 111 captures an image of the user's body while exercising. The imaging unit 111 can acquire moving images of the body by controlling the camera 106. The user or the user's supporter sets up the user terminal 10 on a flat surface, points the optical axis of the camera 106 toward the place where the user is exercising, and issues an instruction to start video shooting. In response to this, the imaging unit 111 operates the camera 106 to acquire the moving images. The imaging unit 111 stores the acquired moving images in the image storage unit 131.

[0020] The image storage unit 131 stores images captured by the imaging unit 111. In this embodiment, the images are moving images. The image storage unit 131 can store the moving images as files, for example.

[0021] The physical information storage unit 130 stores information about the user's body (hereinafter referred to as physical information). Fig. 4 is a diagram showing an example of the configuration of physical information stored in the physical information storage unit 130. As shown in the figure, the physical information includes height, weight, flexibility, and the like.

[0022] The evaluation request sending unit 112 sends to the server device 20 a request to evaluate the physical exercise based on the image captured by the imaging unit 111 (hereinafter, referred to as an evaluation request).

[0023] FIG. 5 is a diagram showing an example of the configuration of an evaluation request transmitted by evaluation request transmission unit 112 to server device 20. As shown in the figure, the evaluation request includes a user ID, a mode, physical information, and image data. The user ID is information for identifying the user. The mode is information indicating the exercise performed by the user. The mode can be, for example, "pitching" or "lifting". It should be noted that the mode is selected from a predetermined selection. The physical information is physical information stored in physical information storage unit 130. The image data is data of a moving image acquired by imaging unit 111.

[0024] The evaluation information receiving unit 113 receives information regarding the evaluation of the physical exercise (hereinafter referred to as evaluation information) returned from the server device 20 in response to the evaluation request. The evaluation information receiving unit 113 registers the received evaluation information in the evaluation information storage unit 132.

[0025] 6 is a diagram showing an example of the configuration of evaluation information received by the evaluation information receiving unit 113 from the server device 20. As shown in the figure, the evaluation information includes a mode, a user ID, position information, posture information, movement information, and checkpoint information.

[0026] The user ID and mode are the user ID and mode included in the evaluation request. The captured image shows the user's body performing the exercise indicated by the mode.

[0027] The position information indicates the position of each part of the body (for example, head, shoulders, elbows, waist, knees, ankles, etc.) in the image. The position information includes the part and the position of the part in association with a time point on the time axis of the video. The state of the body skeleton (bones) can be displayed based on the position information. That is, for example, a figure indicating the part (for example, a circle, etc.) can be displayed by superimposing it on the image at the position indicated by the position information. Note that the positions of multiple parts can be included for one time point. Note that the position information does not need to be included for a part connecting two parts (for example, a forearm connecting a wrist and an elbow, or a thigh connecting a waist and a knee, etc.). In this case, the part connecting these two parts can be expressed by connecting a pair of marks (for example, a circle, etc.) indicating two predetermined parts with a line. The position information may be included for each frame constituting the video, may be included for each key frame, may be included for every arbitrary number of frames, or may be included for a random time point. If the position information is not included in My Frame, the bones can be displayed based on the position information of the most recent past time point.

[0028] The posture information is information related to the posture of the user's body. The posture information includes a part to be evaluated, a posture value, an evaluation rank, and an evaluation comment, which are associated with a time point on the time axis of the video. The posture value is a value that represents the posture. The posture value is, for example, the distance from the ground to a part, the distance between two parts, the angle of the joint part (the angle formed by a straight line from the part at the first end to the joint part and a straight line from the part at the second end to the joint part), etc. The evaluation rank is a value that represents the evaluation value by a rank. The evaluation rank is expressed, for example, from 1 to 5 on a 5-point scale, A, B, C, etc. The evaluation comment is a comment related to the evaluation regarding the posture. For example, when the mode is "lifting" and bending is not sufficient, an evaluation comment such as "your hips are not lowered" may be included.

[0029] The movement information is information related to the movement of the user's body. The movement information includes the part to be evaluated, a list of posture values, an evaluation rank, and an evaluation comment, which are associated with a period on the time axis of the video. The list of posture values ​​is a time series of posture values ​​within a period. The evaluation comment is a comment related to the evaluation of the movement. For example, when the mode is "lifting" and the knee extension is not smooth, an evaluation comment such as "the knee movement is not smooth" may be included.

[0030] The checkpoint information is information indicating a point (hereinafter referred to as a checkpoint) where posture, etc. should be checked in a series of movements of the user's body. For example, in the "pitching" mode, checkpoints include when the foot is raised, when the raised foot is lowered and weight is transferred, and when the ball is released. The checkpoint information stores information indicating the checkpoint (hereinafter referred to as a checkpoint ID) in association with a time point on the time axis of the video. In other words, it is possible to identify the frame (still image) in the video in which the checkpoint indicated by the checkpoint ID is displayed.

[0031] The evaluation display unit 114 displays the evaluation information. For example, the evaluation display unit 114 can display bone movements superimposed on the moving image by displaying figures representing body parts (for example, circles representing end parts or joints and lines connecting them) superimposed on the moving image based on the position information included in the evaluation information. In addition, the evaluation display unit 114 can display, for example, a graph showing the time-series changes in the positions of the parts.

[0032] Furthermore, the evaluation display unit 114 can display an evaluation rank and an evaluation comment together with the display of the video based on the posture information and the movement information included in the evaluation information. For example, the evaluation display unit 114 can display the evaluation rank and the evaluation comment included in the posture information when the playback time of the video reaches a point around the time point included in the posture information (for example, it can be any length such as around 5 seconds). Furthermore, the evaluation display unit 114 can display the evaluation rank and the evaluation comment included in the movement information when the playback time of the video reaches a period included in the movement information. Furthermore, the evaluation display unit 114 can display the posture values ​​included in the posture information. Furthermore, the evaluation display unit 114 can display a graph of the time-series conversion of the posture values ​​based on a list of the posture values ​​included in the movement information.

[0033] The checkpoint display unit 115 can extract and display images of checkpoints from the video. The checkpoint display unit 115 can read frames corresponding to time points included in the checkpoint information from the image data of the video stored in the image storage unit 131 and display them as still images. Furthermore, the checkpoint display unit 115 may, for example, extract and display only body parts from the read frames.

[0034] The improvement measure request sending unit 116 sends a request for acquiring improvement measures related to physical exercise (hereinafter referred to as an improvement measure request) to the server device 20. FIG. 7 is a diagram showing an example of the configuration of an improvement measure request. As shown in the figure, the improvement measure request includes a user ID, a mode, and a purpose. The purpose is the purpose for which the user makes an improvement. For example, the purpose may be "increase the ball speed," "increase muscle strength," "stabilize the lower body," etc. The purpose is also selected from a predetermined selection.

[0035] The improvement measure information receiving unit 117 receives information about improvement measures (hereinafter referred to as improvement measure information) transmitted from the server device 20 in response to an improvement measure request. The improvement measure information receiving unit 117 stores the received improvement measure information in the improvement condition storage unit 133. An example of the configuration of improvement measure information is shown in FIG. 8. As shown in the figure, the improvement measure information includes a purpose, advice, and reference information. In this embodiment, the advice is assumed to be a character string representing the improvement measure, but it may also be content that presents the improvement measure using an image, video, or the like. The reference information is an ideal body posture (the position and angle of each part).

[0036] The improvement measure display unit 118 displays the improvement measures. The improvement measure information display unit 118 displays advice included in the improvement measure information. In addition, when the improvement measure information includes a suitable position or angle for a part,

[0037] 9 is a diagram showing an example of a hardware configuration of the server device 20. The server device 20 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to the communication network 30, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. for inputting data. The output device 206 is, for example, a display, a printer, a speaker, etc. for outputting data.

[0038] 10 is a diagram showing an example of the software configuration of the server device 20. As shown in the diagram, the server device 20 includes functional units, namely, an evaluation request receiving unit 211, an image analysis unit 212, an evaluation unit 213, an evaluation information transmitting unit 214, an improvement request receiving unit 215, and an improvement information transmitting unit 216, and storage units, namely, an image data storage unit 231, a reference information storage unit 232, an evaluation condition information storage unit 233, and an improvement condition storage unit 234.

[0039] Each of the above-mentioned functional units is realized by the CPU 201 of the server device 20 reading out a program stored in the storage device 203 into the memory 202 and executing it, and each of the above-mentioned storage units is realized as part of the storage area provided by the memory 202 and storage device 203 of the server device 20.

[0040] The rating request receiving unit 211 receives a rating request transmitted from the user terminal 10. The rating request receiving unit 211 registers information including image data included in the received rating request (hereinafter referred to as image information) in the image data storage unit 231. FIG. 11 is a diagram showing an example of the configuration of image information stored in the image data storage unit 231. As shown in the figure, the image information includes image data associated with a user ID indicating the user who captured the image. The image data was included in the rating request.

[0041] The reference information storage unit 232 stores information (hereinafter referred to as reference information) including reference values ​​of postures (positions and angles of parts) related to physical movement. Fig. 12 is a diagram showing an example of the configuration of reference information stored in the reference information storage unit 232. As shown in the figure, the reference information includes reference information regarding the absolute position of a part or a relative position with respect to other parts or other reference objects (hereinafter referred to as position reference information), and, for three parts including a joint part, reference information of the angle formed by a straight line connecting each of two parts with the joint part (hereinafter referred to as angle reference information).

[0042] The position reference information includes a part and a reference position of the part in association with a mode and a checkpoint ID. There may be a plurality of parts. Regarding the position, the vertical position may be, for example, the height from the ground or the distance from one of the toes. The horizontal position may be the distance from a predetermined reference object (for example, a mound plate or a mark on the floor) or the distance from a reference part such as the shoulder, chest, or foot. The position reference information is assumed to be registered in advance.

[0043] The angle reference information corresponds to a mode and a checkpoint ID and includes two parts (part 1 and part 2), one joint part, and a reference value of the angle between a line connecting part 1 and the joint part, and a line connecting part 2 and the joint part.

[0044] The evaluation condition information storage unit 233 stores information for performing evaluation (hereinafter, referred to as evaluation condition information). FIG. 13 is a diagram showing an example of the configuration of the evaluation condition information stored in the evaluation condition information storage unit 233. The evaluation condition information includes a category, a condition, an evaluation rank, and a comment. The category is an evaluation category. For example, the category can be "ball speed" or "control". The condition is a condition for the position or movement of each part of the body in the image (change in position in time series). For example, when analyzing a pitching form, conditions for the elbow angle, arm line speed, etc. can be set in the evaluation condition information for the checkpoint where the ball is released. The evaluation rank is an evaluation value when the above condition is satisfied. The comment is an explanation of the posture and movement of the body when the above condition is satisfied.

[0045] The image analysis unit 212 (part identification unit) analyzes the image data. The image analysis unit 212 analyzes the image data to extract the feature amount of each part of the body and identify the position of each part in the image. Note that a general method of image analysis is adopted by the image analysis unit 212, and detailed description thereof will be omitted here. The image analysis unit 212 may analyze the image data for each frame or each key frame, or may analyze at random timing.

[0046] The image analysis unit 212 also compares, for each checkpoint ID, the position of each part extracted from the image data with the position reference information stored in the reference information storage unit 232, and identifies the closest time as the checkpoint time.

[0047] The evaluation unit 213 evaluates the user's body movement based on the image data. In this embodiment, the evaluation unit 213 searches the evaluation condition storage unit 233 for evaluation condition information including conditions satisfied by the position and movement of each part identified from the image data, and acquires the evaluation rank and comments included in the evaluation condition information if there is evaluation condition information that satisfies the conditions.

[0048] The evaluation information transmission unit 214 transmits the evaluation information to the user terminal 10. The evaluation information transmission unit 214 generates position information including the time point on the time axis of the video identified by the image analysis unit 212 and the position of each part. For the evaluation rank and comment acquired by the evaluation unit 213, if the position of the part satisfies the condition, the evaluation unit 214 generates posture information including the time point, part, and posture value, and the evaluation rank and comment, and if the movement of the part (change in position in time series) satisfies the condition, the evaluation condition transmission unit 214 generates movement information including a list of the time point, part, and posture value, and the evaluation rank and comment. In addition, the evaluation condition transmission unit 214 generates checkpoint information including the time point corresponding to each checkpoint analyzed by the image analysis unit 212 and a checkpoint ID indicating the checkpoint. The evaluation information transmission unit 214 creates evaluation information including the generated position information, posture information, movement information, and checkpoint information, and transmits it to the user terminal 10. The evaluation unit 213 and the evaluation information transmission unit 214 may correspond to the comment output unit of the present invention.

[0049] The improvement condition storage unit 234 stores information related to improvement measures (hereinafter referred to as improvement condition information). Fig. 14 is a diagram showing an example of the configuration of improvement condition information stored in the improvement condition storage unit 234. As shown in the figure, the improvement condition information includes advice associated with a purpose, a category, and a condition. The condition may be a condition for a body condition (such as flexibility), or a condition for the position or movement of a body part.

[0050] The remedy request receiving unit 215 receives a remedy request sent from the user terminal 10 .

[0051] The improvement information transmitting unit 216 searches for improvement condition information corresponding to the mode and purpose included in the improvement request, which satisfies the conditions of the user's physical information included in the evaluation request and the positions and movements of each part identified by the image analysis unit 212. The improvement information transmitting unit 216 acquires advice from the searched improvement information, creates improvement information in which the purpose and advice are set, and responds with the created improvement information to the user terminal 10. The improvement information transmitting unit 216 also transmits the improvement information including the positions and angles of each part included in the reference information.

[0052] FIG. 15 is a diagram showing the flow of processes executed in the exercise evaluation system of this embodiment.

[0053] In the user terminal 10, the imaging unit 111 accepts the input of the mode, images the user's body during exercise, and acquires video data (S321). The evaluation request sending unit 112 sends an evaluation request including the user ID indicating the user, the accepted mode, physical information, and video data to the server device 20 (S322).

[0054] When the evaluation request receiving unit 211 in the server device 20 receives an evaluation request, the image analysis unit 212 analyzes the video data to extract features (S323) and identifies the position of each part (S324). Here, the image analysis unit 212 may identify the position on the image, or may identify the actual position (height from the ground, distance from a reference point such as the center of gravity of the body, etc.) using body information. The evaluation unit 213 obtains an evaluation rank and a comment from evaluation condition information that satisfies the conditions of the position of each part and the movement of the part (change in position over time) (S325). The evaluation information transmission unit 214 creates evaluation information and transmits it to the user terminal 10 (S326).

[0055] The evaluation display unit 114 in the user terminal 10 displays the positions (bones) of each body part indicating the posture of the body on the video data based on the received evaluation information, and also displays an evaluation rank and comments (S327). Here, the evaluation display unit 114 may display a graph of the time-series changes in the positions and movements of the body parts. Also, the checkpoint display unit 115 may extract and display images of checkpoints from the video. The improvement measure request sending unit 116 sends an improvement measure request to the server device 20 in response to an instruction from the user (S328).

[0056] In the server device 20, when the improvement request receiving unit 215 receives an improvement request sent from the user terminal 10, the improvement information sending unit 216 searches for improvement condition information that satisfies the conditions, acquires advice contained in the searched improvement condition information (S329), creates improvement information including the acquired advice, and sends it to the user terminal 10 (S330).

[0057] When the improvement information receiving unit 117 in the user terminal 10 receives improvement information, the improvement display unit 118 displays the advice contained in the received improvement information and can also display an appropriate body posture in the form of bones superimposed on the video data (S331).

[0058] As described above, the exercise evaluation system of this embodiment allows easy evaluation of physical exercise. In particular, for physical exercise related to sports, the positional relationship and movement of each part of the body can be evaluated, which is likely to lead to concrete efforts for improvement and improve performance. In addition, the exercise evaluation system of this embodiment also provides comments and advice, allowing the user to easily understand the current situation and improvement measures.

[0059] 16 to 20 are diagrams showing examples of screens that display the evaluation of physical exercise or measures for improvement.

[0060] Fig. 16 illustrates a case where a video is captured in pitching mode. As shown in the figure, on a screen 41, the posture of the body is displayed by bones 411 (which may be defined as lines connecting 17 feature points such as eyes, ears, and joints, for example) indicating the position of each part of the body. Also, a comment according to the deviation from the reference value is displayed in a comment field 412. The user can refer to this to consider posture corrections, etc.

[0061] FIG. 17 shows an example of displaying an image of a checkpoint. A checkpoint selection field 421 is displayed on screen 42, from which a checkpoint can be selected. In the example of FIG. 17, checkpoint 422, which is the release point, is selected, and image 423 in the video corresponding to that checkpoint is displayed. This allows the user to easily grasp the posture and movement of his or her body at each checkpoint. As a method of estimating a checkpoint, for example, it can be determined that the release point has been reached when conditions are met, such as when the pitcher's upper body faces forward (toward the pitch), when the coordinates of the pitcher's right hand (dominant hand) are highest, and when the distance between the pitcher's shoulders is the widest (when the distance is longest), etc.

[0062] 18 to 20 are diagrams showing an example of an evaluation screen of a moving image in a weightlifting mode. In the example of the screen 43, in addition to the predetermined position of the part identified from the image (the position where the user's buttocks reach the lowest point), a suitable posture set as reference information is also displayed on the bone 431. Furthermore, on the screen 43, information 432 related to the movement of the shaft is displayed. In this way, it is possible to identify the position and movement of not only the part of the body but also the tool used by the user. This makes it possible to count, for example, the number of times the weight has been lifted from the image. Also, as shown in FIG. 19, comments or advice 433 related to evaluation or improvement measures can be displayed on the screen 43 together with the moving image. Also, in the example of FIG. 20, a time series graph 434 of the bending angle of the knee is displayed. By further analyzing the changes in the graph 434, an abnormal value 435 can be detected. Also, although not shown, a time series graph of the bending speed of the knee can be displayed together with the graph 434. These graphs can present the appropriate weight and number of times for the user.

[0063] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

[0064] For example, in this embodiment, the image analysis is performed in the server device 20, but this is not limiting. The image analysis may be performed in the user terminal 10 to identify the positional relationship of each part.

[0065] In addition, in this embodiment, it is assumed that the position of the part is a position on a two-dimensional image, but this is not limited to this, and the position may be a three-dimensional position. For example, when the user terminal 10 is equipped with a depth camera in addition to the camera 106, the three-dimensional position of the part can be specified based on the image from the camera 106 and the depth map from the depth camera. It is also possible to provide a depth camera instead of the camera 106 and specify the three-dimensional position only from the depth map from the depth camera. In this case, the user terminal 10 can transmit the depth map to the server device 20 together with the image data or instead of the image data, and the image analysis unit 212 of the server device 20 can analyze the three-dimensional position.

[0066] In addition, in this embodiment, an image of the user's body during exercise is transmitted from the user terminal 10 to the server device 20, but this is not limited to the above. The user terminal 10 may extract features from the image and transmit the features to the server device 20. Alternatively, the user terminal 10 may estimate body parts based on the features, obtain the absolute position of the part (which may be the position on the XY coordinate system of the image, or the actual distance from a reference position (for example, the ground, the toes, the head, the center of gravity of the body, etc.), or the position in any other coordinate system) or the relative positional relationship between multiple parts, and transmit these absolute positions and relative positional relationships to the server device 20.

[0067] In addition, in this embodiment, the improvement information is provided by the content prepared by the server device 20, but it is not limited to this. For example, it may include a reference value, and the bones that will result in the correct posture (position and angle of each part) based on the reference value may be superimposed on a video or a still image extracted from the video. This makes it easy to understand what posture the user should take.

[0068] In addition, in this embodiment, the position or movement (position over time) of a body part is evaluated, but this is not limiting, and the position of a tool held or worn by the user may be identified and evaluated. In this case, reference information and evaluation condition information regarding the tool position may be registered, and the feature amount of the tool may be extracted to identify the position, and the evaluation may be performed in the same manner as for the position of a body part.

[0069] In addition, in this embodiment, the content such as advice is provided as a remedy, but for example, equipment recommendations may be made. In this case, the server device 20 stores the equipment and the standard value of the size (length, etc.) of the equipment in association with the user's physical information (height, weight, etc.), extracts the feature amount of the equipment used by the user from the image data to identify the shape of the equipment, estimates the size of the equipment based on the shape and the user's size (height, etc.) included in the physical information, and if the difference between the estimated equipment size and the standard value is equal to or greater than a predetermined threshold, recommends equipment of the standard size.

[0070] Furthermore, by estimating the position of the tool, the server device 20 can estimate the speed of the tool. Furthermore, by extracting a pattern of the change in the position of the tool over time, the server device 20 can estimate the number of times the pattern occurs as the number of actions using the tool.

[0071] In addition, in this embodiment, the exercise is evaluated, but the present invention is not limited to this. When a certain posture or movement is detected, a task for that movement may be proposed. In this case, the server device 20 may store a task in association with one or a series of postures or movements instead of an evaluation comment, and output the task.

[0072] In addition, in this embodiment, the exercise is evaluated, but the present invention is not limited to this, and when a certain posture or movement is detected, a training to be performed may be presented. In this case, the server device 20 may store the training content in association with one or a series of postures or movements, instead of the evaluation comment, and output the content.

[0073] In addition, in this embodiment, the motion is evaluated, but the present invention is not limited to this, and the motion performed by the player can be automatically detected. In this case, the server device 20 stores the posture (position of each body part) of a player performing a predetermined motion such as shooting or passing as reference information, and can identify the motion performed by the player in the image by comparing the positions of the body parts analyzed from the image with the reference information.

[0074] In addition, in this embodiment, the image captured in the past is analyzed to evaluate the movement, but the present invention is not limited to this. The analysis process may be performed in real time, and when a predetermined movement is detected, the next tactic to be taken may be recommended. In this case, the tactic may be stored in place of the evaluation comment in association with the posture or movement, and the tactic may be output in real time.

[0075] In addition, in this embodiment, the execution of a predetermined function and the storage of information are performed by the user terminal or the server device, but this is not limited thereto, and the execution of the function and the storage of information may be performed by either one of the devices. Alternatively, the function unit and the storage unit may be provided in a distributed manner in a form different from that of this embodiment. [Explanation of symbols]

[0076] 10 User terminal 20 Server device 30 Communication Network 111 Imaging unit 112 Evaluation request sending unit 113 Evaluation information receiving unit 114 Evaluation display section 115 Checkpoint display section 116 Improvement Request Submission Unit 117 Improvement Measures Information Receiving Department 118 Improvement measure display area 130 Physical information storage unit 131 Image storage unit 132 Evaluation information storage unit 133 Improvement plan storage section 211 Evaluation request receiving unit 212 Image Analysis Unit 213 Evaluation Department 214 Evaluation information transmission unit 215 Remediation Request Reception Unit 216 Improvement Measures Information Transmission Department 231 Image data storage unit 232 Standard information storage section 233 Evaluation condition information storage unit 234 Improvement condition information storage unit

Claims

1. A system for evaluating a user's physical exercise, comprising: a reference value storage unit that stores a reference value relating to a relative position of at least one body part with respect to another reference object at a checkpoint, which is a timing for checking posture, that exists in a plurality of times during the predetermined exercise when the exercise is performed appropriately; a part identification unit that analyzes an image of the body captured by a camera of an information terminal used by the user, and identifies the part and the position of the part in the image of the body by extracting a feature amount of each part of the body; an evaluation unit that determines an evaluation value of the movement by comparing a position of the part with respect to the other reference object in the image and the reference value; Equipped with the evaluation unit compares a position of the part relative to the other reference object with the reference value in the images of the plurality of consecutive checkpoints; An exercise evaluation system comprising:

2. The exercise evaluation system according to claim 1 , the evaluation unit determines the evaluation value according to a distance obtained when comparing the reference value with a position of the part relative to the other reference object in the image; An exercise evaluation system comprising:

3. The exercise evaluation system according to claim 1 , the reference value storage unit stores the reference value for an angle between a straight line connecting each of the first and second portions and a third portion; the part identification unit identifies the first to third parts and positions of the parts from the image; the evaluation unit calculates the angles for the identified first to third portions, and compares the calculated angles with the reference value to determine the evaluation value; An exercise evaluation system comprising:

4. The exercise evaluation system according to claim 1 , a comment storage unit that stores a comment about the part in association with a condition for the part and the position; a comment output unit that outputs the comment corresponding to the condition satisfied by the position; The exercise evaluation system further comprising:

5. The exercise evaluation system according to claim 1 , an improvement information storage unit that stores advice on an improvement measure related to the exercise in association with a goal of improvement related to the exercise and a condition for the position; an advice output unit that outputs the advice corresponding to the condition satisfied by the position; The exercise evaluation system further comprising:

Images