Posture estimation device, posture estimation system, and posture estimation method

Abstract

Provided is a posture estimation device for estimating a user's posture, characterized by comprising: an analysis unit which identifies parts of a body from a video image including a user's motion, and analyzes the rotation of the parts; a reference value storage unit which stores a reference value relating to the rotation of at least one of the parts; and an evaluation unit which compares the rotation of the parts in the image with the reference value to determine an evaluation value for said motion.

Description

Technical Field

[0001] The present disclosure relates to a posture estimation device, a posture estimation system, and a posture estimation method.

Background Art

[0002] Techniques for analyzing postures are known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The above-described technique aims to easily identify image data of a video representing an operation including a form in which a predetermined result is likely to occur from among image data of a large number of videos, and images a series of operations of an athlete with a video camera for analysis. However, the actual body movements are three-dimensional, and even when analyzing planar movements, it is difficult to capture the rotation of joints, and the technique described in Patent Document 1 cannot accurately capture the movements in this regard and utilize them for training.

[0005] Therefore, the present disclosure has been made in view of the above problems, and an object thereof is to provide a technique capable of easily and accurately analyzing body movements.

Means for Solving the Problems

[0006] According to the present disclosure, there is provided a posture estimation device that estimates a user's posture, comprising: an analysis unit that identifies a part of the body from a moving image including the user's movement and analyzes the rotation of the part; a reference value storage unit that stores a reference value related to the rotation of at least one of the parts; and an evaluation unit that compares the rotation of the part in the image with the reference value and determines an evaluation value of the movement.

Effect of the Invention

[0007] According to the present disclosure, the movement of the body can be accurately analyzed.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiment for Carrying Out the Invention

[0009] The contents of the embodiments of the present invention will be listed and described. One embodiment of the present invention has the following configuration. [Item 1] A posture estimation device that estimates the user's posture, An analysis unit that identifies the body part from a moving image including the user's movement and analyzes the rotation of the part, A reference value storage unit that stores a reference value related to the rotation of at least one of the parts, An evaluation unit that compares the rotation of the part in the image with the reference value and determines an evaluation value of the movement A posture estimation device, characterized by comprising: [Item 2] The part includes a joint, A preprocessing unit that generates a bone connecting the joints, takes a midpoint between the joints, and generates a plurality of virtual coordinate points on a plane that passes through the midpoint and is perpendicular to the bone, Comprising: The analysis unit derives the three-dimensional rotation coordinates of the bone by analyzing the positions of the virtual coordinate points, The posture estimation device according to claim 1, characterized by: [Item 3] The analysis unit analyzes an image including a series of movements including a sign arranged on the user's body, The sign includes a line, The analysis unit analyzes the line and analyzes the twist of the part, The posture estimation device according to claim 1 or 2, characterized by: [Item 4] The preprocessing unit generates four virtual coordinate points, The posture estimation device according to claim 2, characterized by: [Item 5] A posture estimation system that estimates the user's posture, An analysis function that identifies a part of the user's body from a moving image including the user's actions and analyzes the rotation of the part, a reference value storage function that stores a reference value related to the rotation of at least one of the parts, and an evaluation function that compares the rotation of the part in the image with the reference value to determine an evaluation value of the action A posture estimation system characterized by comprising the above. [Item 6] A posture estimation method for estimating a user's posture, an analysis step of identifying a part of the user's body from a moving image including the user's actions and analyzing the rotation of the part, a reference value storage step of storing a reference value related to the rotation of at least one of the parts, and an evaluation step of comparing the rotation of the part in the image with the reference value to determine an evaluation value of the action A posture estimation method characterized by comprising the above.

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0011] The server device 1 analyzes the user's actions, generates advice that can be utilized for skill improvement, training to obtain correct effects, rehabilitation, etc., and presents it to the user. The server device 1 in the present embodiment performs machine learning using, as teacher data, 6D coordinates obtained by motion capture or the like on an image including a series of the user's actions, such as an image acquired using a device that the user can easily capture, and generates a prediction model. Using this prediction model, it attempts to provide a model for six-dimensionally analyzing the user's actions from an image that simply captures the user's actions. In the present embodiment, convolutional neural networks (CNNs) and a gradient boosting tree algorithm are used for machine learning, but it is not limited to these.

[0012] ==Overview== FIG. 1 is a diagram showing the overall configuration of the motion analysis system. As shown in FIG. 1, the motion analysis system includes a server device 1, a user terminal 3, and a photographing terminal 4. The server device 1 is connected to the user terminal 3 and the photographing terminal 4 via a network 2. Although only one user terminal 3 and one photographing terminal 4 are shown, it goes without saying that more may exist. Also, the specific devices of the user terminal 3 and the photographing terminal 4 are not limited to mobile terminals and personal computers, and may be, for example, smartphones, tablet computers, wearable terminals, and other electronic devices.

[0013] ==Server Device 1== The server device 1 is a computer that evaluates body movements. The server device 1 is, for example, a workstation, a personal computer, a virtual computer logically realized by cloud computing, or the like. The server device 1 receives a moving image photographed by the user terminal 3, analyzes the received moving image, and evaluates the body movement. Also, the server device 1 makes proposals regarding improvement measures for body movements. Details of the evaluation of body movements and the proposals for improvement measures will be described later.

[0014] ==User Terminal 3== The user terminal 3 is a computer operated by a user who performs physical movements or their supporter. The user terminal 3 is, for example, a smartphone, a tablet computer, a personal computer, etc. The user terminal 3 is equipped with an imaging device such as a camera, and thereby can image the user's body during movement. In this embodiment, it is assumed that the moving image of the user's body during movement is transmitted from the user terminal 3 to the server device 1. The user can access the server device 1, for example, through an application executed on the user terminal 3 or a web browser.

[0015] ==Shooting Terminal 4== The shooting terminal 4 is a device for acquiring details of the user's actions. The shooting terminal 4 is, for example, motion capture, etc., and may be of methods such as optical, magnetic, mechanical, and inertial sensor types, but is not limited thereto. The shooting terminal 4 of this embodiment is of the optical type, uses a plurality of cameras and reflection markers as trackers, and installs cameras around the shooting and measurement space for use.

[0016] Hereinafter, the configuration of the server device 1 will be described.

[0017] FIG. 2 is a diagram showing an example of the hardware configuration of the server device 1 of the present embodiment. The server device 1 includes a processor 101, a memory 102, a storage device 103, a communication interface 104, an input device 105, and an output device 106. The processor 101 is an arithmetic device that controls the operation of the entire server device 1, controls the transmission and reception of data between each element, and performs information processing necessary for the execution and authentication processing of applications. For example, the processor 101 is a processor such as a CPU (Central Processing Unit), and executes programs and the like stored in the storage device 103 and expanded in the memory 102 to perform each information processing. The memory 102 includes a main memory composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory composed of a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). The memory 102 is used as a work area of the processor 101, and stores a BIOS (Basic Input / Output System) executed at the time of startup of the server device 1 and various setting information. The storage device 103 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 104 is an interface for connecting to the network 2, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for performing wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 105 is a device that accepts input of data through, for example, a keyboard, a mouse, a touch panel, a button, or a microphone. The output device 106 outputs data and includes, for example, a display, a printer, or a speaker.

[0018] FIG. 3 is a block diagram showing the functional configuration of the server device 1. As shown in FIG. 3, the server device 1 includes each processing unit of a user information acquisition unit 111, an image information acquisition unit 112, a preprocessing unit 113, an analysis unit 114, an evaluation unit 115, an evaluation information presentation unit 116, and a learning unit 117, and each storage unit of a user information storage unit 131, an image information storage unit 132, a preprocessing data storage unit 133, a reference value information storage unit 134, an evaluation condition information storage unit 135, and an improvement condition storage unit 136.

[0019] Note that each of the above processing units is realized by the processor 101 provided in the server device 1 reading a program stored in the storage device 103 into the memory 102 and executing it, and each of the above storage units is realized as a part of the storage area provided by the memory 102 and the storage device 103 provided in the server device 1.

[0020] Here, in the present embodiment, the data configurations of each of the storage units of the user information storage unit 131, the image information storage unit 132, the preprocessing data storage unit 133, the reference value information storage unit 134, the evaluation condition information storage unit 135, and the improvement condition storage unit 136 are shown.

[0021] The user information storage unit 131 stores the user information received by the user information acquisition unit 111 and shown as an example in FIG. 4. As shown in FIG. 4, the user information is information indicating the attributes of the user and the state the user is aiming for, and is composed of, for example, information such as name, date of birth, gender, height, weight, exercise history, needs (such as skill improvement and effect verification), etc., linked to a user ID.

[0022] The image information storage unit 132 stores the information of images capturing a series of operations of the user received by the image information acquisition unit 112. There are two types of images, including the images captured by the user using the user terminal 3 and the images acquired by the imaging terminal 4.

[0023] The preprocessing data storage unit 133 stores the data obtained by the preprocessing unit 113 preprocessing the images stored in the image information storage unit 132 for learning.

[0024] The reference value information storage unit 134 stores information (hereinafter referred to as reference information) including reference values related to relationships derived from the position, movement, rotation, torsion, posture, etc. of body parts related to body movement, and the relationships between parts. FIG. 5 is a diagram showing a configuration example of the reference information stored in the reference value information storage unit 134. As shown in the figure, the reference information includes the absolute position of the body part when performing body movement, information on how the body part has moved (such as moving speed, moving distance, moving direction, etc.), reference information related to the absolute position of the part or the relative position with respect to other parts or other reference objects (hereinafter referred to as position reference information), and for three parts including a joint part, reference information on the angle formed by a straight line connecting each of the two parts and the joint part (hereinafter referred to as angle reference information), the angle at which the part itself rotates around a bone or the like, the rotation speed, the time required for rotation, the relationship between the time when the part starts to move and the time when the part starts or ends rotating, rotation combinations such as rotating back after rotating, etc. rotation information, and torsion reference information of the part, but is not limited thereto. Further, the reference values are prepared for each body movement (for each mode), and for each body movement (mode), there may be a plurality of reference values such as for each purpose, for each feature of the body information, for each feature of the evaluation information, and furthermore, based on the body movement of a specific individual (assuming skilled persons such as athletes or professional athletes who have achieved certain results, skilled persons, experienced persons, etc., but not limited thereto).

[0025] The position reference information includes, in association with the mode and the checkpoint ID, the body part and the position serving as the reference for the part. There may be a plurality of parts. For the position, the vertical position may be, for example, the height from the ground or the distance from any toe. Also, for example, when the mode is "weightlifting", it may be the distance between the line connecting both shoulders and the shaft, or the distance from a line connecting body parts or between body parts. The horizontal position of the part may be the distance from a predetermined reference object (such as a weight plate or a mark on the floor) or the distance from a reference part such as the shoulder, chest, or foot. The position reference information is assumed to be registered in advance.

[0026] The motion reference information includes reference values such as the moving speed, moving distance, moving direction at a certain point in time, and moving trajectory during a certain period of time of a part, etc., which are associated with the mode and the checkpoint ID.

[0027] The angle reference information includes reference values of the angle between two parts (part 1 and part 2), one joint part, the straight line connecting part 1 and the joint part, and the straight line connecting part 2 and the joint part, which are associated with the mode and the checkpoint ID.

[0028] The rotation reference information includes a body part and information on the reference rotation of the part, which are associated with the mode and the checkpoint ID. Regarding the rotation of the part, reference values such as the rotation angle with the bone as the axis of rotation, the rotation speed, and the rotation time (information on the time of the start and end points of the rotation) are included.

[0029] The torsion reference information includes a body part and information on the reference torsion of the part, which are associated with the mode and the checkpoint ID. Regarding the torsion, reference values such as the rotation angle with the bone as the axis of rotation, the rotation speed, and the rotation time (information on the time of the start and end points of the rotation) are included.

[0030] In addition, the content described as the body part so far may be replaced with the part of the tool. For example, the reference value information storage unit 134 stores information (hereinafter referred to as reference information) including reference values regarding the position, movement, rotation, torsion of the part of the tool, and the relationship derived from the relationship between the parts related to the body movement using the tool. In the present disclosure, replacing the content described as the body part with the part of the tool may be applied to the content described in all paragraphs.

[0031] The relationship reference information includes information regarding a reference represented by the relationship between a part of a tool and a part of the body, associated with a mode and a checkpoint ID. The relationship reference information includes information obtained from movement speed, movement distance, angle, etc. in one or more parts and sites, associated with a mode and a checkpoint ID. For example, when the mode is batting, the relationship reference information includes, as reference information, the movement speed of the tip of the bat at the time of hitting the ball and the angle formed by the bat and the dominant arm holding the bat.

[0032] The evaluation condition information storage unit 135 stores information for performing an evaluation (hereinafter referred to as evaluation condition information). FIG. 6 is a diagram showing a configuration example of the evaluation condition information stored in the evaluation condition information storage unit 135. The evaluation condition information includes a category, a condition, an evaluation rank, and a comment. The category is an evaluation category. Examples of the category can include "muscle strength", "ball speed", "control", etc. The condition is a condition for the position, orientation, or movement (change in position over time) of each part of the tool in the image, or for the position or movement (change in position over time) of each part of the body. For example, when analyzing the movement of weightlifting, for the checkpoint at the moment of lifting the barbell, conditions for the angle of the elbow and the speed of extending the arm, etc., and conditions for the movement of the barbell shaft and the speed of going up and down during the period of lifting and lowering the barbell can be set in the evaluation condition information. Also, when analyzing the pitching form, for the checkpoint of releasing the ball, conditions for the angle of the elbow and the rotational speed of the arm, etc. can be set in the evaluation condition information. The evaluation rank is an evaluation value when the above conditions are satisfied. The comment is an explanation about the body posture and movement when the above conditions are satisfied.

[0033] The improvement condition storage unit 136 stores information related to improvement measures (hereinafter referred to as improvement measure information). FIG. 7 is a diagram showing a configuration example of the improvement measure information stored in the improvement condition storage unit 136. As shown in the figure, the improvement measure information includes advice associated with the purpose, category, and conditions. The conditions may be conditions for the tool itself (such as the weight of the barbell), the way of using the tool, conditions for physical conditions (such as flexibility), or conditions for the position, movement, angle, rotation, and torsion of body parts.

[0034] The above is the description of the data configuration of the server device 1.

[0035] Here, in the present embodiment, the functions of each processing unit of the user information acquisition unit 111, the image information acquisition unit 112, the preprocessing unit 113, the analysis unit 114, the evaluation unit 115, the evaluation information presentation unit 116, and the learning unit 117 are shown.

[0036] The user information acquisition unit 111 acquires information about the user from the user terminal 3 via the network 2. The communication in the transmission and reception may be either wired or wireless, and any communication protocol may be used as long as the mutual communication can be executed. Note that the user information may be information collected by an operator who conducts business using the server device 1 through a hearing or questionnaire for the user, and input from the operator's terminal to the server device 1 via the network 2, or the operator may directly input it to the server device 1.

[0037] The image information acquisition unit 112 receives information of an image capturing a series of operations of the user from the user terminal 3 or the imaging terminal 4 via the network 2. The communication in the transmission and reception may be either wired or wireless, and any communication protocol may be used as long as the mutual communication can be executed. Also, an operator using the server device 1 may move the image information to the server device 1 from the user terminal 3 or the imaging terminal 4 through a removable storage medium.

[0038] The preprocessing unit 113 performs preprocessing on the image acquired by the imaging terminal 4 so that the analysis unit 114 can analyze it. Note that the preprocessing data may be used by the learning unit 117 to generate a posture prediction model.

[0039] Here, the image captured by the imaging terminal 4 will be described. The imaging terminal 4 arranges markers as signs on the joints etc. of the user, and captures a series of actions of the user with a plurality of cameras provided in the imaging terminal 4. The imaging terminal 4 detects the position of the marker and identifies the position of each joint (the part where the marker is attached). The data is transmitted to the server device 1. Note that the motion capture system may include a depth sensor and a color camera, and may automatically extract the position information of the user's joint points from the video and detect the actions of the subject. In this case, the user does not need to arrange markers on the body.

[0040] In addition, the images captured by the imaging terminal 4 may include images of a series of actions of the user with lines described below arranged on the user's body, captured by a plurality of cameras provided in the imaging terminal 4. As shown in an example in FIG. 8, the lines are arranged on the user's arms, legs, torso, etc. and used as signs. The lines may be directly attached to the skin, or the lines may be arranged by wearing tights or the like with the lines attached, but are not limited to these methods. The tights or the like are preferably sized to fit the body so that the movement of the epidermis of the user wearing them is transmitted, and further, a material such as silicon having a certain degree of adhesion or stickiness may be arranged on the inside. The locations on the body where the lines are arranged are locations for observing the torsion of the parts. For example, between joints (in the case of the arm, the parts including the upper arm, forearm, and the part beyond the wrist joint including the fingers), around the neck, around the waist, etc., but are not limited to these parts. The color of the lines may be any color as long as it is easily recognizable by general image processing techniques, and the color of the lines may be changed every other line, or the thickness of the lines themselves may be changed every other line. It is desirable to arrange the lines at at least four locations, including the outside, inside, and intermediate points of the parts, but not limited to these. A plurality of cameras included in the imaging terminal 4 capture a series of actions of the user with the lines arranged. FIG. 8 shows, as an example, the state of bending the arm, and FIG. 9 shows the state of making a bump, showing how the lines look. In FIG. 8, although not shown, the palm of the hand faces forward, and in FIG. 9, since the palm of the hand faces the user's body to make a bump, a twisting motion of the wrist is added, and the state can be observed as the torsion of the lines (the lines 151, 152, 153 near the wrist turn to the back side and disappear from the image, and a line 154 that has not appeared in the image until now appears). The imaging terminal 4 detects the positions of the lines and identifies the torsion of each part. The data is transmitted to the server device 1.

[0041] Furthermore, the images captured by the imaging terminal 4 may include images of a series of actions of the user with markers and lines arranged on the body together.

[0042] The preprocessing unit 113 generates bones connecting each joint based on the data including the coordinates of each marker in a series of user operations, which is transmitted from the imaging terminal 4 and stored in the image information storage unit 132. Next, as shown in FIG. 10, the preprocessing unit 113 takes the midpoints (1003a, 1003b) of the bones (1002a, 1002b) connecting each joint (1001a, 1001b, 1001c), and generates a plurality of virtual coordinate points (1004a, 1004b, 1004c, 1004d, 1004e, 1004f, 1004g, 1004h) on the plane passing through the midpoint and perpendicular to the bone. In FIG. 10, as an example, four virtual coordinate points are generated at a time.

[0043] The analysis unit 114 analyzes the image information stored in the image information storage unit 132 or the preprocessing data stored in the preprocessing data storage unit 133, and analyzes the user's posture.

[0044] The analysis unit 114 analyzes the image information to extract feature amounts of each part of the body and specifies the position of each part in the image. Note that a general method for image analysis by the analysis unit 114 is adopted, and detailed description is omitted here. The analysis unit 114 may analyze the image information for each frame or each key frame, or may analyze the image information for each checkpoint, or may analyze it at random timing. Thus, the analysis unit 114 analyzes the movement of each part three-dimensionally.

[0045] The analysis unit 114 also compares the position of each part extracted from the image information with the position reference information and the like stored in the reference value information storage unit 134 for each checkpoint ID, and specifies the time point closest to the checkpoint as the time point of the checkpoint.

[0046] In addition, the analysis unit 114 analyzes the three-dimensional rotation coordinates (Roll, Pitch, Yaw) of the body part by analyzing the position of the virtual coordinate points pre-processed by the pre-processing unit 113, and analyzes the user's posture from the image information. For example, the analysis unit 114 analyzes how the part from the elbow to the wrist rotates immediately before and after the ball leaves the hand in pitching. Thereby, the rotational movement (three-dimensional rotation) of each part can be analyzed when each part is regarded as a rigid body.

[0047] In addition, the analysis unit 114 may derive the twist of each part by analyzing the line. For example, the analysis unit 114 analyzes how the part near the elbow and the part near the wrist twist in the part from the elbow to the wrist immediately before and after the ball leaves the hand in pitching. Thereby, each part will be analyzed not as a mere rigid body but as a body part composed of muscles and bones.

[0048] In addition, the analysis unit 114 may use the posture prediction model generated by the learning unit 117 described later, take the image information captured by the user terminal 3 as input information, and analyze the three-dimensional rotation of each part or the twist of each part.

[0049] In addition, the analysis unit 114 may specify the position coordinates of the part and analyze the three-dimensional rotation coordinates from the image acquired by the image information acquisition unit 112 without going through the processing of the pre-processing unit 113. For example, when the user terminal 3 is an RGB camera and the imaging terminal 4 is a motion capture, the analysis unit 114 analyzes the image information captured by the user terminal 3, extracts the feature amounts of each part of the body, and specifies the position coordinates of each part in the image. The analysis unit 114 calibrates the position coordinate information of the part with the position coordinate information of the part analyzed from the image obtained from the imaging terminal 4, derives the depth of each part in the image coordinate system, and analyzes the three-dimensional rotation coordinates.

[0050] The evaluation unit 115 evaluates the movement of the user's body part based on the image information. In the present embodiment, the evaluation unit 115 searches the evaluation condition information storage unit 135 for evaluation condition information including conditions satisfied by any of the position of each part of the body part specified from the image information, the movement of the part, the angle formed by the parts, the rotation of the part, and the twist of the part. If there is evaluation condition information for which the condition is satisfied, the evaluation rank and comment included therein are acquired. Note that the evaluation unit 115 may evaluate the movement, rotation, twist, etc. of the part and count the number of body movements.

[0051] The evaluation unit 115 evaluates the movement of the user's body part based on the image information. In the present embodiment, the evaluation unit 115 searches the evaluation condition information storage unit 135 for evaluation condition information including conditions satisfied by any of the position of each part of the body or each part of the tool specified from the image information, the movement of the part or part, the angle formed by the parts or between the parts and the part, the rotation of the part or part, and the twist in the relationship between the part, part, and part. If there is evaluation condition information for which the condition is satisfied, the evaluation rank and comment included therein are acquired.

[0052] The evaluation information presentation unit 116 transmits the evaluation information to the user terminal 3. The evaluation information presentation unit 116 generates position information including the time point on the time axis of the video identified by the analysis unit 114, the position of each part, the movement of each part, the angle formed by the parts, the rotation of each part, and the twist of each part.

[0053] When the position of the part satisfies the condition for the evaluation rank and comment acquired by the evaluation unit 115, the evaluation information presentation unit 116 generates posture information including the time point, part, and posture value, and the evaluation rank and comment. When the movement of the part (change in position in time series) satisfies the condition for the evaluation rank and comment acquired by the evaluation unit 115, the evaluation information presentation unit 116 generates movement information including a list of the time point, part, and posture value, and the evaluation rank and comment. When the movement of the angle formed by the parts (change in position in time series) satisfies the condition for the evaluation rank and comment acquired by the evaluation unit 115, the evaluation information presentation unit 116 generates angle information including a list of the time point, part, and posture value, and the evaluation rank and comment. When the rotation of the part satisfies the condition for the evaluation rank and comment acquired by the evaluation unit 115, the evaluation information presentation unit 116 generates rotation information including a list of the time point, part, and posture value, and the evaluation rank and comment. When the torsion of the part satisfies the condition for the evaluation rank and comment acquired by the evaluation unit 115, the evaluation information presentation unit 116 generates torsion information including a list of the time point, part, and posture value, and the evaluation rank and comment.

[0054] In addition, the evaluation information presentation unit 116 generates checkpoint information including the time point corresponding to each checkpoint analyzed by the analysis unit 114 and the checkpoint ID indicating the checkpoint. The evaluation information presentation unit 116 creates evaluation information including the generated position information, posture information, movement information, angle information, opening information, evaluation information, and checkpoint information and transmits it to the user terminal 3. Note that the evaluation unit 115 and the evaluation information presentation unit 116 may correspond to the comment output unit of the present invention.

[0055] The evaluation information presentation unit 116 transmits either or both of the evaluation information and the improvement measure information to the user terminal 3. The evaluation information presentation unit 116 receives a request for either or both of the evaluation information and the improvement measure information from the user terminal 3, and searches for those that satisfy the conditions among the improvement measure information corresponding to the mode and purpose included in those requests, such as the user's physical information included in the user information, the positions, orientations, movements, etc. of each part and each site identified by the analysis unit 114. The evaluation information presentation unit 116 acquires the advice of the retrieved improvement measure information, creates improvement measure information with the purpose and advice set, and responds to the user terminal 3 with the created improvement measure information. The evaluation information presentation unit 116 also includes the positions, orientations, speeds, angles, etc. of each part and each site included in the reference information in the improvement measure information and transmits it. Note that the evaluation information presentation unit 116 may search for improvement measures based on the evaluation information and the reference value even without the request, and the evaluation information presentation unit 116 may transmit the improvement measures to the user terminal 3.

[0056] The evaluation information presentation unit 116 transmits the evaluation information to the user terminal 3. The evaluation information presentation unit 116 generates position information including the time point on the time axis of the video identified by the analysis unit 114 and the positions of each part. Regarding the evaluation rank and comment acquired by the evaluation unit 213, when the position of the part satisfies the conditions, posture information including the time point, the part, and the posture value, and the evaluation rank and the comment is generated, and when the movement of the part (change in position in time series) satisfies the conditions, movement information including a list of the time point, the part, and the posture value, and the evaluation rank and the comment is generated. In addition, the evaluation information presentation unit 116 generates checkpoint information including the time point corresponding to each checkpoint analyzed by the analysis unit 114 and the checkpoint ID indicating the checkpoint. The evaluation information presentation unit 116 creates evaluation information including the generated position information, posture information, movement information, and checkpoint information and transmits it to the user terminal 3. Note that the evaluation unit 115 and the evaluation information presentation unit 116 may correspond to the comment output unit of the present disclosure.

[0057] The learning unit 117 generates a learning model for estimating the user's posture. The learning unit 117 uses, as teacher data, the video including a series of actions of the user captured by the imaging terminal 4 and the data of the position, movement, angle, rotation, and twist of each part analyzed by the analysis unit 114, and uses the teacher data to input, as input information, an image including a series of actions of the user captured by the user terminal 3, and generates a learning model that outputs values related to the user's posture (the position, movement, angle, rotation, and twist of each part).

[0058] Using FIG. 11, the flow of typical processing of this embodiment will be described. The user information acquisition unit 111 receives user information (1001). The image information acquisition unit 112 receives image information (1003). The preprocessing unit 113 generates virtual coordinate points by preprocessing the image information (1004). The analysis unit 114 identifies parts from the image information and analyzes the movement of the parts (1004). Further, the analysis unit 114 analyzes the rotation of the parts based on the information of the lower-layer coordinate points (1005). Further, the analysis unit 114 analyzes the twist of the parts from the image information (1006). The evaluation unit 115 evaluates the position, movement, rotation, twist, etc. of the parts by comparing them with reference values (1007). The evaluation information presentation unit 116 presents the evaluation result to the user (1008).

[0059] As described above, the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that those having ordinary knowledge in the technical field of the present disclosure can conceive of various modification examples or correction examples within the scope of the technical idea described in the claims, and it is naturally understood that these also belong to the technical scope of the present disclosure.

[0060] In addition, although this embodiment has been described, the above embodiment is for facilitating the understanding of the present invention and is not for limiting the interpretation of the present invention. The present invention can be changed and improved without departing from its gist, and equivalents of the present invention are also included therein.

[0061] For example, in this embodiment, the server device 1 analyzes images. However, the present invention is not limited to this, and the user terminal 3 may analyze images to identify the positional relationship, angle, rotation, and twist of each part.

[0062] Further, in this embodiment, it is assumed that the position of the body part is a position on a two-dimensional image. However, the present invention is not limited to this, and it may also be a three-dimensional position. For example, when the user terminal 3 is provided with a depth camera in addition to a camera, the server device 1 can identify the three-dimensional position of the body part based on the image from the camera and the depth map from the depth camera. Further, for example, the server device 1 may estimate the three dimensions from the two-dimensional image to identify the three-dimensional position of the body part. Note that instead of the camera provided in the user terminal 3, a depth camera may be provided, and the server device 1 can also identify the three-dimensional position only from the depth map from the depth camera. In this case, the depth map can be transmitted to the server device 1 together with the image data or instead of the image data from the user terminal 3, and the analysis unit 114 of the server device 1 can analyze the three-dimensional position.

[0063] Also, in this embodiment, it is assumed that an image of the user's body during movement is transmitted from the user terminal 3 to the server device 1. However, the present invention is not limited to this, and the user terminal 3 may extract feature amounts from the image and transmit the feature amounts to the server device 1, or the user terminal 3 may estimate the body part based on the feature amounts, and the absolute position of the part (which may be the position on the XY coordinates of the image, or the distance in actual size from a reference position (such as the ground, the tip of the foot, the head, the center of gravity of the body, etc.), or the position in any other coordinate system).) or the relative positional relationship between a plurality of parts may be obtained, and these absolute positions and relative positional relationships may be transmitted to the server device 1.

[0064] Also, in this embodiment, it is assumed that the improvement measure information includes content prepared on the server device 1 side. However, the present invention is not limited to this. For example, the server device 1 may superimpose and display marks or bones that represent correct movements and postures (positions, angles, rotations, twists, etc. of each part) based on a reference value, including the reference value, on a moving image or a still image extracted from the moving image. This makes it possible to easily understand what kind of movements and postures should be taken.

[0065] Also, in this embodiment, the server device 1 is assumed to evaluate the position or movement of a body part (change in position over time), the angle formed between parts, rotation, twist, etc. However, the present invention is not limited to this. The server device 1 may evaluate the position or movement of a part of a tool used in exercise (change in position over time), the angles formed between parts of the tool, or the angles formed between a body part and a part of the tool, the rotation, twist, etc. of the part of the tool. Further, the server device 1 may identify and evaluate the position of the tool worn by the user.

[0066] Also, in this embodiment, regarding the improvement measures, content such as measures for improving operations and advice on training methods is provided. However, for example, the server device 1 may perform tool recommendations. In this case, the server device 1 stores reference values of tools and the sizes (lengths, etc.) of the tools in association with the user's body information (height, weight, etc.). The server device 1 extracts feature amounts of the tool used by the user from the image data to identify the shape of the tool, estimates the size of the tool based on the shape and the user's size (e.g., height, etc.) included in the body information, and if the difference between the estimated size of the tool and the reference value is equal to or greater than a predetermined threshold value, the server device 1 can recommend a tool with a size equal to the reference value. Further, tools suitable for the purpose may be recommended based on information such as conditions for the tool itself (weight of the barbell, etc.), how to use the tool, physical conditions (flexibility, etc.), the position, orientation, movement, etc. of the part of the tool.

[0067] Also, in this embodiment, it is assumed that content such as advice is provided regarding improvement measures. For example, the server device 1 may interrupt the ongoing body movement. In this case, the server device 1 stores a reference value for interrupting the body movement in association with the user's body information (purpose, height, weight, etc.), and when the number of times or speed of the body movement being performed by the user from the image data (for example, the speed of lifting the barbell drops extremely, or the number of times performed at once is too large, etc.) deviates from the reference value, the body movement is interrupted. In this case, a comment may be issued to the user terminal 3 to stop, or the user may be notified by changing the display of the display such as turning off the screen, or a sound such as an alert sound may be emitted, or the user may be notified by vibration.

[0068] Also, in this embodiment, it is assumed that content such as advice is provided regarding improvement measures. For example, the server device 1 may present body movements for the determination of illness or injury and improvement thereof. In this case, the server device 1 extracts candidates for illnesses or injuries that the user is assumed to have developed from the symptoms input by the user in the body information and the evaluation information, and presents a screening test for narrowing down. When the user performs the screening test and the disease name, location, and degree of the injury can be narrowed down, the server device 1 may recommend seeing a doctor, body movements for improvement, tools for performing body movements, or items such as food.

[0069] Also, by estimating the position of the tool part, the server device 1 can estimate the speed, acceleration, moving distance, trajectory, etc. of the tool. Further, the server device 1 can estimate the number of times of the pattern as the number of times of the operation using the tool by extracting the pattern of the change in the position of the tool in time series.

[0070] In addition, in this embodiment, although the movement is evaluated, the present invention is not limited to this. When a certain posture or movement is detected, problems with the movement may be proposed. In this case, the server device 1 may store problems in association with one or a series of postures or movements, instead of evaluation comments, and output the problems.

[0071] In addition, in this embodiment, although the movement is evaluated, the present invention is not limited to this. When the server device 1 detects the movement of a certain tool, the orientation of the tool, the posture, or the movement of a body part, the server device 1 may present content for improving body movement according to the purpose, such as training, rehabilitation, performance, or stretching and muscle strength training, which are the preparation stages thereof, and the posture. In this case, the server device 1 may store the content of implementation such as training in association with the movement of one or a series of parts of the tool, the orientation of the parts of the tool, the posture of the body, or the movement of the body part, instead of the evaluation comment, and output the content.

[0072] In addition, in this embodiment, although the movement is evaluated, the present invention is not limited to this. The server device 1 can also automatically detect the actions performed by the user. In this case, the server device 1 stores, as reference information, the positions and postures (positions of each part of the body) of each part of a tool that performs a predetermined action such as a shoot or a pass, compares the positions of the parts of the tool and the body part analyzed from the image with the reference information, and can identify the actions performed by the user in the image.

[0073] In addition, in this embodiment, the server device 1 analyzes an image captured in the past to evaluate the movement. However, the present invention is not limited to this. Real-time analysis processing may be performed, and when a predetermined action is detected, a tactic to be taken next may be recommended. In this case, the server device 1 may store the tactic in association with the posture or movement, instead of the evaluation comment, and output the tactic in real time.

[0074] In this embodiment, the evaluation information presenting unit 116 may present the result of evaluation by the evaluation unit 115 to a supporter terminal used by the supporter of the user (which may be a trainer, coach, or instructor for training, or may include a caregiver, physical therapist, medical professional, or other rehabilitation-related person). The user terminal 3 and the supporter terminal may be in the form of, for example, glasses, contact lenses, hats, HMD (head mounted display), or the like, and an image capturing function provided in the user terminal 3 or the supporter terminal captures an image of a range close to the field of view of the user or supporter, and the captured image is sent to the server device 1 via the communication network 2. The evaluation unit 115 processes the image, and the result is sent to the supporter terminal or the user terminal 3 via the communication network 2. The supporter terminal outputs information to the supporter or user via an interface such as a virtual image projection method, a retinal projection method, or other method, or a BMI (brain machine interface) that uses brain activity such as brain waves to input characters, images, videos, etc. by direct stimulation of the brain without going through sensory organs. By performing this communication and processing at high speed, when the supporter sees the user's physical movement, the supporter can check the evaluation of the user's physical movement almost in real time. In addition, the user can also check the evaluation of his / her own physical movement and the evaluation within the group almost in real time. Note that in the supporter's field of vision, the supporter sees a real image of the user's physical movement, and the result of the processing performed by the evaluation unit 115 may be superimposed on the field of vision seen by the supporter with the naked eye through the supporter's terminal, so that the supporter can visually recognize it. Furthermore, in the case where the supporter's terminal is an HMD or the like, the result of the processing performed by the evaluation unit 115 may be superimposed on the image captured by the supporter's terminal and presented to the supporter.

[0075] In this embodiment, there may be a plurality of reference values, and the user may select a reference value for a fee.

[0076] In this embodiment, the reference value may include a value created based on the physical exercise (including that performed using tools) of an expert such as a professional athlete.

[0077] In addition, in this embodiment, the user terminal 3, the imaging terminal 4, and the supporter's terminal may be wearable terminals such as glasses. The results of imaging, analysis, and evaluation performed by the wearable terminal may be output to the wearable terminal. However, the user terminal 3, the imaging terminal 4, and the supporter's terminal that are not wearable devices such as mobile terminals may also exist simultaneously, and the results of analysis and evaluation may also be displayed on the non-wearable terminals.

[0078] The devices described in this specification may be implemented as a single device, or may be implemented by a plurality of devices (such as a cloud server) partially or entirely connected by a network. For example, the processor 101 and the storage device 103 of the server device 1 may be implemented by different servers connected to each other by a network.

[0079] A series of processes by the devices described in this specification may be implemented using any of software, hardware, and combinations of software and hardware. It is possible to create a computer program for realizing each function of the server device 1 according to this embodiment and install it on a PC or the like. In addition, a computer-readable recording medium storing such a computer program can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

[0080] Also, the processes described using flowcharts in this specification do not necessarily have to be executed in the order shown. Some processing steps may be executed in parallel. In addition, additional processing steps may be adopted, and some processing steps may be omitted.

[0081] Also, the effects described in this specification are illustrative or exemplary only and not limiting. That is, the technology according to the present disclosure may exhibit other effects that are apparent to those skilled in the art from the description of this specification, together with or instead of the above effects.

Explanation of Reference Numerals

[0082] 1 Server device 2 Network 3 User terminal 4 Photographing terminal 101 Processor 102 Memory 103 Storage device 104 Communication interface 105 Input device 106 Output device 111 User information acquisition unit 112 Image information acquisition unit 113 Preprocessing unit 114 Analysis unit 115 Evaluation unit 116 Evaluation information presentation unit 117 Learning unit 131 User information storage unit 132 Image information storage unit 133 Learning data storage unit

Claims

1. A posture estimation device for estimating a user's posture, comprising: an analysis unit that identifies a part of the user's body from a moving image including the user's movement and analyzes the rotation of the part; a reference value storage unit that stores a reference value related to the rotation of at least one of the parts; an evaluation unit that compares the rotation of the part in the moving image with the reference value and determines an evaluation value of the movement A posture estimation device, characterized by comprising the above.

2. The part includes joints, a preprocessing unit that generates bones connecting the joints, takes a midpoint between the joints, and generates a plurality of virtual coordinate points on a plane that passes through the midpoint and is perpendicular to the bone; comprising The analysis unit derives a three-dimensional rotation coordinate of the bone by analyzing the positions of the virtual coordinate points. The posture estimation device according to claim 1, characterized by the above.

3. The analysis unit analyzes an image including a series of movements including a sign arranged on the user's body, the sign includes a line, the analysis unit analyzes the line and analyzes the twist of the part. The posture estimation device according to claim 1 or 2, characterized by the above.

4. The preprocessing unit generates four virtual coordinate points. The posture estimation device according to claim 2, characterized by the above.

5. A posture estimation system for estimating a user's posture, comprising: an analysis function that identifies a part of the user's body from a moving image including the user's movement and analyzes the rotation of the part; a reference value storage function that stores a reference value related to the rotation of at least one of the parts; an evaluation function that compares the rotation of the part in the moving image with the reference value and determines an evaluation value of the movement A posture estimation system, characterized by comprising the above.

6. A posture estimation method for estimating a user's posture, comprising: an analysis step of identifying a part of the user's body from a moving image including the user's movement and analyzing the rotation of the part; a reference value storage step of storing a reference value related to the rotation of at least one of the parts; an evaluation step of comparing the rotation of the part in the moving image with the reference value and determining an evaluation value of the movement A posture estimation method, characterized by comprising the above.

Images