Activity amount calculation device and activity amount calculation method

Abstract

To provide an activity amount calculation device and activity amount calculation method capable of calculating an amount of activity, parameters for evaluating physical activity of a user based on a moving image of the user.SOLUTION: An activity amount calculation device calculates, for each of calculation reference joint points in an exercise state acquired based on a captured moving image of a user during a calculation target period, muscle power during the exercise of a corresponding body part in a periphery of a joint point in the calculation target period; calculates an integral value of the total muscle power during the exercise in the calculation period, as energy consumption during the exercise; and calculates whole body energy consumption based on the energy consumption during the exercise.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an activity amount calculation device and an activity amount calculation method. [Background technology]

[0002] Physical activity refers to any movement that consumes more energy than a resting state, and there are various methods for quantifying (quantifying) physical activity (hereinafter referred to as "activity amount") to measure it. In recent years, to evaluate (measure) the amount of activity in daily life, activity amount has been measured using small devices worn on the body, such as pedometers and activity monitors. Activity amount can be expressed as parameters that quantify physical activity, such as energy expenditure (energy consumed), METs, exercise (METs x exercise time), and calories burned.

[0003] On the other hand, the use of images of a user to calculate the amount of activity without wearing such a small device has been studied. For example, Patent Document 1 discloses a muscle activity visualization system (hereinafter referred to as "conventional system") that estimates the amount of muscle activity of a subject (user) from data measured by a range imaging camera that measures the three-dimensional posture of the subject (user). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-86184 Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional systems are limited to estimating muscle activity and are unable to measure a user's activity. The present invention has been made to solve the above-mentioned problem. That is, one of the objects of the present invention is to provide an activity amount calculation device and an activity amount calculation method that can calculate an activity amount, which is a parameter that quantifies a user's physical activity, based on a video of the user. [Means for solving the problem]

[0006] The activity calculation device of the present invention is an activity calculation device including an information processing device that processes data based on a moving image of a user, and the information processing device calculates angular velocities of each of a plurality of calculation reference joint points among a plurality of joint points based on skeletal data of the user acquired based on the moving image of the user during a calculation target period, and calculates torque for each of the calculation reference joint points in a moving state by multiplying a rotational component of estimated gravity acting on the corresponding body part, which is obtained by multiplying an estimated mass of the corresponding body part around the joint point by gravitational acceleration, by a torque arm length which is the length between the calculation reference joint point and the center of gravity of the corresponding body part. and calculating a muscle power during exercise of the corresponding body part around the joint point during the calculation period by multiplying the calculated torque by the angular velocity in the rotational direction; calculating an integral of the total muscle power during exercise over the calculation period as whole-body energy expenditure; calculating whole-body energy expenditure based on at least the whole-body energy expenditure; and calculating an activity amount, which is a parameter quantifying the physical activity of the user, as the whole-body energy expenditure and at least one of the whole-body energy expenditure and a parameter derived from the whole-body energy expenditure.

[0007] The activity amount calculation method of the present invention is a method using an information processing device that processes data based on a video captured by a user, and includes calculating angular velocities of a plurality of calculation reference joint points among a plurality of joint points based on skeletal data of the user acquired based on the video captured by the user during a calculation target period, and multiplying, for each of the calculation reference joint points in a moving state, a rotational component of estimated gravity acting on the corresponding body part, which is obtained by multiplying an estimated mass of a corresponding body part around the joint point by gravitational acceleration, by a torque arm length which is the length between the calculation reference joint point and the center of gravity of the corresponding body part. The apparatus performs a torque calculation to calculate torque, and a muscle power calculation during exercise to calculate the muscle power during exercise of the corresponding body part around the joint point during the calculation period by multiplying the calculated torque by the angular velocity in the rotational direction, calculates the integral of the total muscle power during exercise over the calculation period as whole-body energy expenditure, calculates whole-body energy expenditure based on at least the whole-body energy expenditure, and calculates at least one of the whole-body energy expenditure and the parameter derived from the whole-body energy expenditure as an activity amount, which is a parameter that quantifies the physical activity of the user. [Effects of the Invention]

[0008] According to the present invention, it is possible to calculate the amount of activity, which is a parameter that quantifies the physical activity of a user, based on a video of the user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing an example of the schematic configuration of a system including an activity amount calculation device according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a specific example of the configuration of the activity amount calculation device. [Figure 3] FIG. 3 is a diagram for explaining user information. [Figure 4] FIG. 4 is a diagram for explaining the exercise menu data. [Figure 5]Figure 5 shows an image of the skeletal data. [Figure 6] FIG. 6 is a diagram for explaining the result data. [Figure 7] FIG. 7 is a block diagram showing a specific example of the configuration of a terminal device. [Figure 8] FIG. 8 is a diagram showing an example of a GUI screen displayed on the terminal device. [Figure 9] FIG. 9 is a flowchart showing a processing flow executed by the activity amount calculation device. [Figure 10] FIG. 10 is a diagram for explaining a method for calculating the torque applied to a joint point and a method for calculating the angular velocity. [Figure 11A] FIG. 11A is a diagram for explaining the calculation formula. [Figure 11B] FIG. 11B is a diagram for explaining the calculation formula. [Figure 12] FIG. 12 is a diagram showing an example of the measurement screen. [Figure 13] FIG. 13 is a diagram showing an example of the result screen. [Figure 14] FIG. 14 is a diagram showing an example of the activity amount presentation screen for this week. [Figure 15] FIG. 15 is a diagram illustrating an example of time-series data of joint torque, angular velocity, muscle power during exercise, and muscle power at rest for the waist center during the period from the start time ts of the exercise menu (abdominal muscles) to a certain time t2. [Figure 16] FIG. 16 is a diagram for explaining the calculation formula. [Figure 17] FIG. 17 is a flowchart showing a processing flow executed by the activity amount calculation device according to the second embodiment of the present invention. [Figure 18] FIG. 18 is a flowchart showing a processing flow executed by the activity amount calculation device according to the third embodiment of the present invention. [Figure 19] FIG. 19 is a diagram for explaining the calculation formula. [Figure 20] FIG. 20 is a flowchart showing a processing flow executed by an activity amount calculation device according to the fourth embodiment of the present invention. [Figure 21]FIG. 21 is a diagram for explaining the calculation formula. [Figure 22] FIG. 22 is a diagram for explaining calculation reference joint points corresponding to trunk muscles. [Figure 23] FIG. 23 is a diagram showing an example of the activity amount calculation method setting screen. [Figure 24] FIG. 24 is a diagram for explaining an example of a method for calculating the mass (estimated mass) and torque arm length of the corresponding body part. [Figure 25] FIG. 25 is a diagram for explaining an example of a method for calculating the mass (estimated mass) of the corresponding body part. [Figure 26] FIG. 26 is a schematic diagram of a system including an activity amount calculation device as an application example. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. In the following description, various types of information may be described using expressions such as "table," "row," and "record," but the various types of information may be expressed using data structures other than these. Furthermore, in the following description, when describing identification information, expressions such as "identification number," "identification information," and "name" are used, but these are interchangeable, and identification information may also be expressed using expressions other than these.

[0011] <<First Embodiment>> <Configuration> FIG. 1 is a schematic diagram showing an example of the schematic configuration of a system including an activity amount calculation device 100 according to a first embodiment of the present invention. As shown in FIG. 1, the system includes the activity amount calculation device 100 and a terminal device 200. In this example, the terminal device 200 is, for example, a tablet-type terminal device equipped with an imaging device, and functions as an exercise teaching and measurement device that displays images (videos) for teaching exercise to a user Us1 and acquires images (videos of exercise execution) necessary for measuring an activity amount (amount of physical activity), which is a parameter that quantifies the physical activity of the user Us1. The activity amount calculation device 100 and the terminal device 200 are connected to each other via a network NW1 so as to be able to communicate with each other.

[0012] As described above, the amount of activity is a parameter that quantifies the physical activity of the user Us1, and includes whole-body energy expenditure E, METs A, exercise Aw, whole-body calorie expenditure Ec, and the like.

[0013] The whole body energy expenditure E is a parameter (index) that represents the energy consumed by the physical activity of the user Us1 in joules.

[0014] The whole body calorie expenditure Ec is a parameter (index) that represents the energy consumed by the physical activity of the user Us1 in units of calories (kilocalories).

[0015] METs A is a parameter (index) that indicates the intensity of physical activity relative to a sitting and resting state, and is expressed in units called METs.

[0016] Exercise Aw is the product of METs A and exercise time, and is a parameter (indicator) that represents the amount of physical activity, expressed as "METs·hours."

[0017] These can be converted into each other using well-known conversion formulas. Whole-body calorie expenditure Ec, METs A, exercise Aw, etc. can be derived from whole-body energy expenditure E, and for convenience, they are also called "parameters derived from whole-body energy expenditure E."

[0018] Fig. 2 is a block diagram showing a specific example of the configuration of the activity amount calculation device 100. As shown in Fig. 2, the activity amount calculation device 100 includes a non-volatile storage device 110 that can read and write data, a processor 120 (e.g., a CPU), a memory 130 (e.g., a RAM), an input interface 140, an output interface 150, and a communication interface 160. These are connected via a bus 170. For convenience, a device including the bus 170 that connects the storage device 110, the processor 120, the memory 130, the input interface 140, the output interface 150, the communication interface 160, etc. is also referred to as an "information processing device." The information processing device may be a plurality of information processing devices or a virtual information processing device built on the cloud.

[0019] The storage device 110 holds (memorizes, stores) a database 111. The database 111 includes user information 112, exercise menu data 113, skeletal data 114, motion stillness determination data 115, activity amount data 116, and result data 117. Details of the data stored in the database 111 will be described later.

[0020] The processor 120 operates as a data processing unit 121. The data processing unit 121 includes a user information management unit 122, an exercise menu instruction unit 123, a skeleton data extraction unit 124, a motion stillness determination unit 125, an activity amount calculation unit 126, and a result processing unit 127.

[0021] The user information management unit 122 executes the user information registration process to update (register) the user information 112 stored in the database 111. The user information management unit 122 refers to the user information 112. Details of the user information registration process will be described later.

[0022] The exercise menu instruction unit 123 refers to the exercise menu data 113 stored in the database 111, and provides (instructs) the terminal device 200 with an exercise menu and an exercise instruction video.

[0023] The skeleton data extraction unit 124 extracts the skeleton data 114 based on the exercise video of the user Us1 captured by the terminal device 200, and stores the skeleton data 114 in the database 111.

[0024] The motion stillness determination unit 125 stores, in the database 111, motion stillness determination data 115 indicating whether each joint point (joint) of the user Us1 is in a moving state or a still state, based on the skeleton data 114.

[0025] The activity amount calculation unit 126 executes an activity amount calculation process, which will be described later, to calculate activity amount data 116 based on the skeleton data 114 and the motion stillness determination data 115, and stores the activity amount data 116 in the database 111. The result processing unit 127 calculates result data 117 based on the skeleton data 114 and the activity amount data 116, and stores the result data 117 in the database 111.

[0026] The processor 120 loads the program stored in the storage device 110 into the memory 130. The processor 120 executes the program loaded into the memory 130 to realize various functions of the data processing unit 121 (user information management unit 122, exercise menu instruction unit 123, skeletal data extraction unit 124, motion stillness determination unit 125, activity amount calculation unit 126, and result processing unit 127).

[0027] The input interface 140 is an interface for connecting operation devices such as a keyboard and a mouse.

[0028] The output interface 150 is an interface for connecting a display device or the like.

[0029] The communication interface 160 is an interface for connecting the activity amount calculation device 100 to the network NW1.

[0030] The above-mentioned user information 112, exercise menu data 113, skeleton data 114, motion stillness determination data 115, activity amount data 116, and result data 117 will be described in detail below.

[0031] The user information 112 includes personal attribute values, usage history information, user setting information, etc. FIG. 3 is a diagram illustrating the user information 112 represented as a table. As shown in FIG. 3, the user information 112 includes, as columns for storing information (values), a user ID 112a, a facility ID 112b, an in-facility user ID 112c, a gender 112d, an age 112e, a height 112f, a weight 112g, an exercise habit 112h, an exercise consciousness 112i, a stiff shoulder 112j, a lower back pain 112k, and history information 112l. In the user information 112, information corresponding to each column related to the user Us1 is associated with each other and stored as information (records) in units of rows.

[0032] Specifically, the user ID 112a stores an identification number for identifying the user Us1. The facility ID 112b stores identification information for identifying the facility. The in-facility user ID 112c stores an identification number for identifying the user Us1 in the facility. The gender 112d stores information indicating the gender of the user Us1 (male or female). The age 112e stores information indicating the age of the user Us1. The height 112f stores information indicating the height of the user Us1. The weight 112g stores information indicating the weight of the user Us1. Exercise habit 112h stores information indicating the frequency of exercise. Exercise consciousness 112i stores information indicating exercise consciousness. Stiff shoulders 112j stores information indicating whether or not there is stiff shoulders. Lower back pain 112k stores information indicating whether or not there is lower back pain. History information 112l stores exercise history information of user Us1. The exercise history information includes identification information of user Us1's exercise history, exercise content (exercise menu data, exercise duration data, skeletal data, activity amount data, etc.).

[0033] FIG. 4 is a diagram illustrating exercise menu data 113 represented as a table. As shown in FIG. 4, exercise menu data 113 includes an exercise ID 113a, an exercise name 113b, and an exercise instruction video 113c as columns for storing information (values). In exercise menu data 113, information corresponding to each column related to the exercise menu is associated with each other and stored as information (records) in units of rows. Specifically, exercise ID 113a stores identification information for identifying the exercise menu. Exercise name 113b stores the name of the exercise performed in the exercise menu. Exercise instruction video 113c stores the exercise instruction video.

[0034] The skeletal data 114 is information about the skeleton and joints (joint points) of the user Us1, extracted (recognized) based on an image of the user Us1. FIG. 5 is a diagram showing the skeletal data 114 at a certain time in the form of an image. The skeletal data 114 at a certain time includes information representing the skeleton and joints of the user Us1, acquired (extracted) based on an image (video) of the user Us1 (e.g., position information of the skeleton represented by the line L10, position information of each joint point Ji represented by the black dot Pt, position information indicating the body contour, etc.). When there is no need to distinguish between the joint points, each joint point is referred to as a "joint point Ji." When there is no need to distinguish between the joint points, each corresponding body part defined when calculating the joint torque Ti applied to each joint point Ji (described later) is referred to as a "corresponding body part BJi." The position information is expressed, for example, as xyz coordinates (values) (= x, y, z) in a predetermined three-dimensional coordinate space. Every time a predetermined time elapses, skeleton data 114 at a certain time is acquired, and the skeleton data 114 as time-series data is stored in the database 111.

[0035] The motion stillness determination data 115 is information indicating whether each joint point Ji is in a moving state or a stationary state. For example, if the angular velocity wi of the joint point Ji is not "zero," the motion stillness determination unit 125 determines that the joint point Ji is in a moving state. If the angular velocity wi of the joint point Ji is "zero," the motion stillness determination unit 125 determines that the joint point Ji is in a stationary state. Note that if the absolute value of the angular velocity wi of the joint point Ji is equal to or greater than a predetermined threshold, the motion stillness determination unit 125 may determine that the joint point Ji is in a moving state, and if the absolute value of the angular velocity wi of the joint point Ji is smaller than the predetermined threshold, the motion stillness determination unit 125 may determine that the joint point Ji is in a stationary state.

[0036] The activity amount data 116 includes data necessary for calculating an activity amount in an activity amount calculation process that is executed every time a predetermined time period described below elapses, the calculated activity amount, etc. Specifically, for example, the activity amount data 116 includes the following information: Torque Ti (also called "joint torque Ti") applied to each calculated reference joint point Ji based on skeletal data. In this example, the calculation reference joint points Ji are the bending joint points Ji (joints), namely the right elbow, right shoulder, left shoulder, left elbow, shoulder center, back, waist center, right waist, left waist, right knee, and left knee. Angular velocity wi of each reference joint point Ji The angular velocity wi is defined as being positive in the clockwise direction and negative in the counterclockwise direction. -Mass (estimated mass) of the corresponding body part BJi for each calculation reference joint point Ji Right elbow: The area between the right elbow and the tip of the right hand Left elbow: The area between the left elbow and the tip of the left hand Right shoulder: The area between the right shoulder and the tip of the right hand Left shoulder: The area between the left shoulder and the tip of the left hand ·Shoulder center: head Back: The area between the back and the center of the waist ·Waist center: lower body and upper body ·Right hip: left lower body ·Left hip: Right lower body Right knee: From the right knee to the tip of the right foot Left knee: The area between the left knee and the tip of the left foot -Center of gravity of each corresponding body part BJi For each calculation reference joint point Ji, the torque arm length Li between the calculation reference joint point Ji and the center of gravity of the corresponding body part BJi Muscle power (muscle power during exercise) of the muscle of the corresponding body part BJi corresponding to each calculation reference joint point Ji Energy consumption during whole-body exercise (Em) Whole body energy consumption E Parameters derived from whole-body energy expenditure E (METs A, whole-body calorie expenditure Ec, exercise Aw, etc.) These parameters will be described in detail later.

[0037] Fig. 6 is a diagram illustrating the result data 117 expressed as a table. As shown in Fig. 6, the result data 117 includes first result data 117a and second result data 117b. These are created based on the activity amount data 116 and the like.

[0038] The first result data 117a includes, as columns for storing information (values), an exercise ID 117a1, an exercise name 117a2, and a joint 117a6. The joint 117a6 includes a shoulder center 117a3, a right shoulder 117a4, a right knee 117a5, and the like.

[0039] In the first result data 117a, information corresponding to each column regarding the exercise results is associated with each other and stored as information (records) on a row-by-row basis. Specifically, the exercise ID 117a1 stores identification information for identifying the exercise. The exercise name 117a2 stores the name of the exercise (exercise menu). The joint 117a6 includes, as columns for storing information (values), a shoulder center 117a3, a right shoulder 117a4, and a right knee 117a5. Each column included in the joint 117a6 stores the activity amount (exercise Aw in this example) for each calculation reference joint point Ji when the user Us1 performs the corresponding exercise menu. Note that the first result data 117a may also store activity amounts other than exercise Aw.

[0040] The second result data 117b includes columns (columns) for storing information (values): implementation date / time 117b1, exercise ID 117b2, exercise name 117b3, time 117b4, and activity amount 117b5. The second result data 117b stores information corresponding to each column related to exercise results in association with one another. Specifically, the implementation date / time 117b1 stores the date and time the exercise menu was implemented. The exercise ID 117b2 stores identification information for identifying the exercise. The exercise name 117b3 stores the name of the exercise (exercise menu). The time 117b4 stores the implementation time of the corresponding exercise menu. The activity amount 117b5 stores the activity amount (in this example, exercise Aw) calculated as a result of implementing the corresponding exercise menu. Note that the second result data 117b may store activity amounts other than exercise Aw.

[0041] Fig. 7 is a block diagram showing a specific example configuration of the terminal device 200. As shown in Fig. 7, the terminal device 200 includes a non-volatile storage device 210 that can read and write data, a memory 220 (e.g., RAM), a processor 230, a camera 240, a display device 250, and a communication interface 260. These are connected by a bus 270.

[0042] The processor 230 operates as an exercise application execution unit 231 and a measurement unit 232. The exercise application execution unit 231 executes exercise menu presentation, which displays an exercise menu on the display screen 251, exercise instruction video playback, which plays back an exercise instruction video, exercise execution video recording, which records (stores in the storage device 210) an exercise execution video, skeletal data presentation, which displays an image representing skeletal data 114 on the display screen 251, and activity amount data presentation, which displays result data 117 based on the activity amount data on the display screen 251, etc.

[0043] The measurement unit 232 includes an exercise video measurement unit 232a. The exercise video measurement unit 232a acquires an exercise video (exercise execution video) of the user captured (pictured) by the camera 240. Furthermore, the exercise video measurement unit 232a stores the exercise execution video captured by the camera 240 in the storage device 210. Furthermore, the exercise video measurement unit 232a transmits the exercise execution video captured by the camera 240 to the activity amount calculation device 100.

[0044] Processor 230 loads a program (not shown) stored in storage device 210 into memory 220. Processor 230 executes the program loaded into memory 220 to realize various functions of exercise application execution unit 231 and measurement unit 232 (exercise video measurement unit 232a).

[0045] The storage device 210 holds (memorizes, stores) data 211 and the like used when the exercise application execution unit 231 executes each process. The data 211 includes exercise menu data, exercise instruction video data, exercise execution video data, skeletal data, and activity amount data. These are transmitted from the processor 230 and the activity amount calculation device 100. The camera 240 is an imaging device capable of capturing video, and captures (photographs) the user Us1 present within the imaging range of the camera 240. The display device 250 includes a display screen 251 and a touch sensor 252. An example of the display device 250 is, for example, a touch panel. The communication interface 260 is an interface for connecting the terminal device 200 to the network NW1.

[0046] <Operation of the activity amount calculation device 100> A description will now be given of the operation of the activity amount calculation device 100. The activity amount calculation device 100 executes a user information registration process and an activity amount calculation service process.

[0047] (User information registration process) The user information registration process will now be described. In order to receive the activity amount calculation service, the user Us1 registers user information in the activity amount calculation device 100 via the terminal device 200. The activity amount calculation device 100 displays a GUI screen on the display screen 251 of the display device 250 of the terminal device 200 as a GUI (Graphical User Interface) used when registering the user information. FIG. 8 shows an example of the GUI screen displayed on the terminal device 200.

[0048] The terminal device 200 displays a subject registration screen GM10 as a GUI screen on a display screen 251 of a display device 250. A user Us1 operates the subject registration screen GM10 (inputs information into the subject registration screen GM10) via an operating device (in this example, a touch panel including the display screen 251 and a touch sensor 252).

[0049] The subject registration screen GM10 includes a user ID input field 801, a facility ID input field 802, an in-facility user ID input field 803, a gender input field 804, an age input field 805, an exercise habit input field 806, an exercise awareness input field 807, a stiff shoulder input field 808, a lower back pain input field 809, a height input field 810, a weight input field 811, a first pull-down button 812, a second pull-down button 813, a third pull-down button 814, a fourth pull-down button 815, a fifth pull-down button 816, a sixth pull-down button 817, a seventh pull-down button 818, an eighth pull-down button 819, and a registration button Bt11.

[0050] The user ID input field 801, facility ID input field 802, in-facility user ID input field 803, gender input field 804, age input field 805, exercise habit input field 806, exercise awareness input field 807, stiff shoulder input field 808, lower back pain input field 809, height input field 810, and weight input field 811 are referred to as "input fields" when there is no need to distinguish between them. The first pull-down button 812, the second pull-down button 813, the third pull-down button 814, the fourth pull-down button 815, the fifth pull-down button 816, the sixth pull-down button 817, the seventh pull-down button 818, and the eighth pull-down button 819 are referred to as "pull-down buttons" when there is no need to distinguish between them. When a pull-down button is operated, a pull-down list (not shown) showing options for information to be entered in the input field is displayed. By selecting an option for information, information is entered into the input field.

[0051] A user ID is input into the user ID input field 801. In this example, "0001" is input as the user ID. A facility ID (identification information (name)) is input into the facility ID input field 802. In this example, "H001 (XXX Corporation)" is input as the facility ID. An in-facility user ID input field 803 is input as an in-facility user ID (identification number). In this example, "HU0001" is input as the in-facility user ID. Gender is input into the gender input field 804. In this example, "male" is input as the gender. Age is input into the age input field 805. In this example, "55" is input as the age. Exercise habit input field 806 is input as exercise habit (information indicating frequency of exercise). In this example, "once a week" is input as the exercise habit. Information representing exercise awareness is input into the exercise consciousness input field 807. In this example, "exercise" is input as the information representing exercise awareness. In the stiff shoulders input field 808, information indicating whether or not the user has stiff shoulders is input. In this example, "none" is input as information indicating whether or not the user has stiff shoulders. In the lower back pain input field 809, information indicating whether or not the user has lower back pain is input. In this example, "none" is input as information indicating whether or not the user has lower back pain. In the height input field 810, the user's height is input. In this example, "160 cm" is input as the height. In the weight input field 811, the user's weight is input. In this example, "60 kg" is input as the weight.

[0052] When the user Us1 operates the registration button Bt11, the activity amount calculation device 100 associates the information entered in each input field with each other and registers (stores, memorizes) the information in the database 111 of the storage device 110 as user information 112 (line-by-line information (record) of the user information 112).

[0053] (Activity calculation service processing) An outline of the activity amount calculation service process (service provided to the user) executed by the activity amount calculation device 100 will be described.

[0054] When the terminal device 200 requests execution of an activity calculation service process, the activity amount calculation device 100 displays an exercise menu selection screen (not shown) as a GUI screen on the display screen 251 of the terminal device 200. The exercise menu selection screen (not shown) includes a list of exercise menus that the activity amount calculation device 100 can provide, corresponding to the exercise menu data 113 (a list of exercise names of the exercise menus).

[0055] User Us1 selects an exercise menu to be performed from a list of exercise menus by operating an exercise menu selection screen (not shown). At this time, user Us1 can select one exercise menu or multiple exercise menus to be performed consecutively.

[0056] When an exercise menu is selected, the activity amount calculation device 100 refers to the exercise menu data 113 via the exercise menu instruction unit 123 to obtain a row (each row's exercise ID, exercise name, and exercise instruction video (hereinafter also referred to as "exercise menu information") that corresponds to the selected exercise menu) and transmits the exercise menu information to the terminal device 200.

[0057] The terminal device 200 displays an exercise start instruction screen (not shown) as a GUI screen using the exercise menu information and other necessary information. The exercise start instruction screen (not shown) includes, for example, a start button (not shown) for the exercise menu and the name of the exercise (not shown) for the selected exercise menu.

[0058] When the user Us1 operates a start button (not shown) of the exercise menu, a measurement screen GM12 (see FIG. 12 described later) including an exercise instruction video and the like is displayed. Details of this measurement screen GM12 will be described later. Furthermore, when the start button (not shown) of the exercise menu is operated, the terminal device 200 starts capturing an image of the user Us1 with the camera 240, and transmits the captured video (exercise execution video) of the user Us1 to the activity amount calculation device 100.

[0059] The activity amount calculation device 100 executes an activity amount calculation process based on the exercise video. Details of this activity amount calculation process will be described later.

[0060] When the selected exercise menu or menus are completed, the activity amount calculation device 100 stores the calculation results of the activity amount calculation process as result data 117 in the database 111 of the storage device 110.

[0061] When one or more selected exercise menus are completed, the terminal device 200 transmits an exercise record (exercise status, exercise date and time, etc.) of the user Us1 who is exercising to the activity amount calculation device 100. The activity amount calculation device 100 stores the exercise record as result data 117 in the database 111 of the storage device 110.

[0062] When there is a display request for a result screen GM13 (see FIG. 13 described later) of the activity amount calculation result from the terminal device 200, the activity amount calculation device 100 transmits to the terminal device 200 data required for displaying the result screen GM13 reflecting the result data 117 on the terminal device 200, and displays the result screen GM13 on the display screen 251 of the terminal device 200. Details of this result screen GM13 will be described later. The above is an overview of the activity amount calculation service processing executed by the activity amount calculation device 100.

[0063] (Activity calculation process) The activity amount calculation process described above will now be described in detail. FIG. 9 is a flowchart showing the process flow of the activity amount calculation process executed by the activity amount calculation device 100. The activity amount calculation device 100 starts the process from step 900 and sequentially executes the processes of steps 901 to 906 described below, and then proceeds to step 995 to temporarily end this process flow. The activity amount calculation device 100 executes this process flow every time a predetermined time elapses between an exercise menu start time tst (also referred to as "start time tst") and an exercise menu end time tend. The exercise menu start time tst is, for example, the time when a start button (not shown) for an exercise menu is operated, and the exercise menu end time tend is, for example, the time when one or more selected exercise menus end.

[0064] Step 901: The activity amount calculation device 100 causes the skeleton data extraction unit 124 to extract (acquire) the skeleton data 114 from the exercise video (images during exercise) of the user Us1.

[0065] Step 902: The activity amount calculation device 100 calculates, by the activity amount calculation unit 126, joint torques Ti and angular velocities wi in the rotational direction at the current time tp for a plurality of calculation reference joint points Ji among the joint points Ji.

[0066] 10 is a diagram for explaining a method for calculating joint torque Ti and angular velocity wi. The activity calculation device 100 calculates the angular velocity wi in the rotation direction of the joint point Ji from the skeleton data 114 acquired based on the exercise video. The activity calculation device 100 acquires the angle θi between the vertical direction of the position of the center of gravity of the body part BJi corresponding to the calculation reference joint point Ji and the rotation direction (the tangent direction of the rotating center of gravity) based on the skeleton data 114.

[0067] The activity calculation device 100 calculates the gravity F acting on the corresponding body part BJi by multiplying the mass Mi (estimated mass Mi) of the corresponding body part BJi by the gravitational acceleration g [m / s 2] is multiplied by (applied to calculation formula (1) in FIG. 10). In this example, the estimated mass Mi is a predetermined value set in advance for each corresponding body part BJi.

[0068] The activity calculation device 100 calculates the rotational component F of gravity acting on the corresponding body part BJi using the calculation formula (2) in FIG. Ri The activity amount calculation device 100 calculates the joint torque Ti applied to the joint point Ji using calculation formula (3) in Fig. 10. Note that the torque arm length Li is the length from the calculation reference joint point Ji to the position of the center of gravity c i.

[0069] The activity amount calculation device 100 performs the above calculation for all calculation reference joint points Ji, thereby calculating the joint torque Ti and the angular velocity wi in the rotation direction at the current time tp for each calculation reference joint point Ji of the user Us1.

[0070] Step 903: The activity amount calculation device 100 calculates the during-motion muscle power Pmi(tp) of each corresponding body part BJi of each calculation reference joint point Ji determined to be in a moving state at the current time tp by the activity amount calculation unit 126, and stores the calculated muscle power Pmi(tp) in the memory 130. That is, the activity amount calculation device 100 applies the joint torque Ti and the angular velocity wi in the rotational direction calculated in step 902 to the calculation formula (4) in FIG.

[0071] Step 904: The activity amount calculation device 100 calculates, using the activity amount calculation unit 126, the whole-body exercise energy expenditure Em for the period from the start time ts to the current time tp (also referred to as the "calculation target period"), based on the calculation formula (5) and the exercise muscle power Pmi(t) of each corresponding body part BJi of each calculation reference joint point Ji calculated for the period from the start time ts to the current time tp, which is stored in the memory 130. That is, when the activity amount calculation device 100 calculates, using the activity amount calculation unit 126, the whole-body exercise energy expenditure Em for the period from the time tst to the current time tp, the activity amount calculation device 100 calculates, as the whole-body exercise energy expenditure Em, the integral of the total exercise muscle power Pmi(t) for each calculation reference joint point Ji for the calculation target period.

[0072] Step 905: The activity amount calculation device 100 calculates, by the activity amount calculation section 126, the whole-body exercise energy expenditure Em calculated in step 904 as the whole-body energy expenditure E for the period from the start time ts to the current time tp.

[0073] Step 906: The activity amount calculation device 100 calculates whole-body calorie expenditure Ec by applying the whole-body energy expenditure E to calculation formula (6) of FIG. 11A using the activity amount calculation unit 126, and calculates METs A (whole-body activity amount A per hour) from the whole-body energy expenditure E by applying the whole-body calorie expenditure Ec, “H (= the time (hours) between the start time tst and the current time tp)” and “W (= the weight of the user Us1 (acquired from the user information 112))” to calculation formula (7A) of FIG. 11B. That is, the activity amount calculation device 100 converts the whole-body energy expenditure E into whole-body calories consumed Ec by multiplying the whole-body energy expenditure E by a first conversion coefficient (0.000239), and then calculates METs A as a parameter derived from the whole-body energy expenditure E by adding "1" to the result obtained by dividing the whole-body calories consumed Ec by the product of the second conversion coefficient (1.05), the time (H) corresponding to the calculation period, and the weight (W) of the user Us1. Furthermore, the activity amount calculation device 100 calculates the exercise Aw by applying the METs A and H to the calculation formula (7B) in FIG. 11B. That is, the activity amount calculation device 100 calculates the exercise Aw by multiplying the METs A and H. The parameters calculated in the activity amount calculation process are stored in the database 111 as result data 117.

[0074] Furthermore, in the activity amount calculation process, the activity amount calculation device 100 may further calculate the energy consumption for each calculation reference joint point Ji by integrating the muscle power during exercise Pmi of the body part BJi corresponding to each calculation reference joint point Ji during the calculation period using the activity amount calculation unit 126. Furthermore, the activity amount calculation device 100 may calculate the METs, calories consumed, exercise, etc. for each calculation reference joint point Ji based on the energy consumption for each calculation reference joint point Ji.

[0075] (Measurement screen display) 12 shows an example of the measurement screen GM12. As described above, when the user Us1 operates the start button (not shown) on the exercise start instruction screen (not shown), the activity amount calculation device 100 displays the measurement screen GM12 that reflects the exercise menu information, the skeletal data 114, and the exercise Aw calculated in the activity amount calculation process.

[0076] The measurement screen GM12 includes an exercise menu display section MN1, a user ID display field 1201, an energy consumption display field 1202, a progress display bar 1203, an exercise instruction video 1204, a user skeleton video 1205, a cancel button Bt21, and a next button Bt22.

[0077] The exercise menu display section MN1 displays the name of the exercise menu (exercise). The user ID display field 1201 displays the user ID of user Us1. The energy expenditure display field 1202 displays the current activity amount. The activity amount displayed is the value of exercise Aw calculated in the activity amount calculation process described above. The exercise instruction video 1204 is a sample video of the exercise menu (in this example, a video in which a picture representing a trainer (also referred to as a "trainer icon") performs the exercise menu (squats)). The user skeletal video 1205 includes a picture representing user Us1 (a picture of the outline of user Us1) corresponding to the captured video of user Us1 and a thick line representing the skeleton corresponding to skeletal data 114 of user Us1. The exercise instruction video 1204 may be a captured video of a person (trainer) performing a sample exercise menu. The user skeletal video 1205 may be a captured video of user Us1.

[0078] (Displays the results screen and this week's activity amount display screen) 13 shows an example of the result screen G13. As described above, when one or more selected exercise menus are completed, the activity amount calculation device 100 displays the result screen GM13 reflecting the result data 117 on the display device 250 of the terminal device 200.

[0079] 13, the results screen GM13 includes a user ID display field 1301, a list of exercises performed 1302, an image of energy expenditure of each muscle 1303, a button Bt31 for displaying this week's activity amount presentation screen, and a button Bt32 for ending the display. The list of performed exercises 1302 includes the exercise menus performed by user Us1 (the names of the exercise menus), the time spent performing the exercise menus, and the amount of activity (exercise Aw). Furthermore, the list of performed exercises 1302 includes the time spent performing each exercise menu, the total time spent performing the exercises, and the total amount of activity.

[0080] Each muscle energy expenditure image 1303 includes a drawing of a human outline and thick lines representing the skeleton. The thickness of the thick lines representing the skeleton changes depending on the amount of activity of the corresponding body part BJi corresponding to the skeleton. Specifically, the thicker the thick lines representing the skeleton corresponding to the corresponding body part BJi, the greater the amount of activity of the corresponding body part BJi corresponding to the skeleton, and the thinner the thick lines representing the skeleton corresponding to the corresponding body part BJi, the smaller the amount of activity of the corresponding body part BJi corresponding to the skeleton.

[0081] When the user Us1 operates the this week's activity amount presentation screen display button Bt31, the activity amount calculation device 100 displays the this week's activity amount presentation screen GM14 shown in Fig. 14 (described later) on the display device 250 of the terminal device 200. When the user Us1 operates the display end button Bt32, the activity amount calculation device 100 ends the display of the result screen GM13.

[0082] FIG. 14 shows an example of this week's activity amount presentation screen GM14. As shown in FIG. 14, the this week's activity amount presentation screen GM14 includes this week's activity amount list 1401, this week's activity amount target value 1402, this week's remaining activity amount 1403, and a back button Bt41. The this week's activity amount list 1401 includes details of exercise performed this week (1 week) and the corresponding exercise time and cumulative activity amount. These values ​​are displayed based on the result data 117. The back button Bt41 is a button operated to switch the this week's activity amount presentation screen GM14 to the result screen GM13. Therefore, when the back button Bt41 is operated by the user Us1, the activity amount calculation device 100 switches the this week's activity amount presentation screen GM14 to the result screen GM13 and displays the result screen GM13 on the display device 250 of the terminal device 200.

[0083] <Effects> As described above, the activity amount calculation device 100 according to the first embodiment of the present invention calculates the torque Ti applied to each joint point Ji and the muscle power during exercise for each corresponding body part BJi based on the torque Ti applied to each joint point Ji, based on a video (image) of the user Us1 performing exercise. Furthermore, the activity amount calculation device 100 according to the first embodiment calculates the whole-body exercise energy expenditure Em based on the muscle power during exercise for each corresponding body part BJi. Then, the activity amount calculation device 100 according to the first embodiment calculates an activity amount, which is a parameter that quantifies the physical activity of the user Us1 during exercise (measurement period), such as the whole-body energy expenditure E, METs A, and exercises Aw, based on the whole-body exercise energy expenditure Em. This allows the activity amount calculation device 100 according to the first embodiment to accurately measure (estimate) the activity amount, which is a parameter that quantifies the physical activity of the user Us1, based on the video (image) of the user Us1 performing exercise.

[0084] One prior art technology is an activity monitor that uses an acceleration sensor attached to the waist or chest to calculate whole-body energy expenditure. This prior art calculates the magnitude of the acceleration vector from acceleration data in the three axes (X, Y, and Z). However, this prior art technology does not detect movements of the arms, upper body, or other parts of the body other than the area where the sensor is attached, so the whole-body energy expenditure is inevitably an approximate value, which is an issue. Also, the measured values ​​vary depending on the area where the sensor is attached. Furthermore, the acceleration data measured by the acceleration sensor contains a lot of noise, making it difficult to convert it into speed or distance by integration. Therefore, the prior art technology can only calculate METs from acceleration data alone.

[0085] <<Second embodiment>> An activity amount calculation device 100 according to a second embodiment of the present invention will be described. In the activity amount calculation process, the activity amount calculation device 100 according to the second embodiment calculates the whole-body stationary energy expenditure Es in addition to the whole-body motion energy expenditure Em. Furthermore, the activity amount calculation device 100 according to the second embodiment calculates the whole-body energy expenditure E by summing the whole-body motion energy expenditure Em and the whole-body stationary energy expenditure Es. The activity amount calculation device 100 according to the second embodiment differs from the activity amount calculation device 100 according to the first embodiment only in the above-mentioned points. Therefore, the following description will mainly focus on these differences.

[0086] (Outline of activity amount calculation process) FIG. 15 shows an example of time series data of joint torque Ti, angular velocity wi, in-motion muscle power Pmi, and resting muscle power Psi for the waist center during the period from start time ts to a certain time t2 when user Us1 is performing an exercise menu (abdominal muscles).

[0087] 15, during the period from time ts to a certain time t2, while the user Us1 is performing an action (exercise) of lifting his / her back, the motion-still determination unit 125 determines that the waist center is in a state of motion, the angular velocity wi in the rotation direction of the waist center is not zero (the absolute value of the angular velocity wi is greater than zero), and the joint torque Ti (the absolute value of the joint torque Ti) acting on the waist center is greater than zero. During this period, the activity amount calculation device 100 calculates the muscle power during exercise Pmi(t) and calculates the energy expenditure Em during whole-body exercise, in the same manner as in the first embodiment.

[0088] During the period from immediately after time t1 to time t2, when user Us1 maintains a lifted back, the motion-still determination unit 125 determines that the waist center is stationary. The angular velocity wi in the rotational direction of the waist center is zero. Therefore, if the in-motion muscle power Pmi(t) is calculated using equation (4) in FIG. 11A, the in-motion muscle power Pmi(t) is zero. However, because user Us1 uses the muscles of the body part BJi corresponding to the joint point (waist center) to maintain this state, it is conceivable that muscle power is being generated even during this period.

[0089] Therefore, the activity calculation device 100 according to the second embodiment calculates the muscle power of the calculation reference joint point Ji determined to be in a stationary state as the stationary muscle power Psi (Psi(t)) using equation (8) in FIG. 16 . Note that Ci in equation (8) is a coefficient for converting the joint torque Ti (torque at rest) into the stationary muscle power Psi. Ci is set in advance, and is, for example, a constant value obtained by conducting an experiment or the like in advance. For example, Ci may be calculated based on the ratio of the energy consumption at rest measured by another method (for example, a measurement method based on the maximum oxygen uptake obtained by an activity meter or expired gas analysis) to the sum of the torques Ti applied to each calculation reference joint point Ji in a stationary state measured by this method.

[0090] Furthermore, the activity amount calculation device 100 calculates the whole-body energy expenditure Es at rest based on the resting muscle power Psi(t) (the resting muscle power Psi(t) calculated for each calculation reference joint point Ji) using equation (9) in Fig. 16. That is, for example, when calculating the whole-body energy expenditure Es at rest for a period from time tst to a certain time, the total integral of the resting muscle power Psi(t) for each calculation reference joint point Ji for that period is calculated as the whole-body energy expenditure Es at rest.

[0091] 16, the activity amount calculation device 100 calculates the total energy (total energy) obtained by adding up the whole-body energy expenditure Em during movement and the whole-body energy expenditure Es when the whole body is stationary, as the whole-body energy expenditure E. By calculating the whole-body energy expenditure E in this way, the activity amount calculation device 100 can calculate an activity amount that also reflects physical activity when stationary.

[0092] (Activity amount calculation process) 17 is a flowchart showing a processing flow executed by the activity amount calculation device 100 according to the second embodiment. The activity amount calculation device 100 executes this processing flow every time a predetermined time elapses from the exercise menu start time tst to the exercise menu end time tend.

[0093] The activity amount calculation device 100 starts the process from step 1700, and after sequentially executing the processes of step 1701 to step 1708 described below, proceeds to step 1795 and temporarily ends this processing flow.

[0094] Step 1701: The activity amount calculation device 100 executes the same process as that of step 901 in FIG. 9 described above.

[0095] Step 1702: The activity amount calculation device 100 executes the same process as the process of step 902 in FIG. 9 described above.

[0096] Step 1703: The activity amount calculation device 100 calculates the during-motion muscle power Pmi(tp) of each corresponding body part BJi of each calculation reference joint point Ji determined to be in a moving state at the current time tp using the activity amount calculation unit 126, and stores the calculated value in the memory 130. The activity amount calculation device 100 calculates the during-motion muscle power Pmi(tp) of the corresponding body part BJi of each calculation reference joint point Ji that is in a moving state by applying the torque Ti and angular velocity wi calculated in step 1702 to equation (4) in Fig. 11A.

[0097] Step 1704: The activity amount calculation device 100 calculates, by the activity amount calculation unit 126, the energy expenditure Em during whole-body exercise during the period from the start time ts to the current time tp, based on the calculation formula (5) in FIG. 11A and the muscle power during exercise Pmi(t) of each corresponding body part BJi of each calculation reference joint point Ji calculated during the period from the start time ts to the current time tp, which is stored in the memory 130.

[0098] Step 1705: The activity amount calculation device 100 calculates the resting muscle power Psi(tp) of each corresponding body part BJi of the calculation reference joint point Ji determined to be in a resting state at the current time tp by the activity amount calculation unit 126, and stores the calculated muscle power Psi(tp) in the memory 130. That is, the activity amount calculation device 100 applies the torque Ti calculated in step 1702 to the calculation formula (8) in FIG.

[0099] Step 1706: The activity amount calculation device 100 calculates, by the activity amount calculation unit 126, the whole-body stationary energy expenditure Es for the period from the start time ts to the current time tp, based on the calculation formula (9) and the stationary muscle power Psi(tp) of each corresponding body part BJi of each calculation reference joint point Ji calculated for the period from the start time ts to the current time tp, which is stored in the memory 130. That is, when the activity amount calculation device 100 calculates, by the activity amount calculation unit 126, the whole-body stationary energy expenditure Es for the period from the time tst to the current time tp, it calculates, as the whole-body energy expenditure Em during exercise, the integral of the total stationary muscle power Psi(t) for each calculation reference joint point Ji for the calculation period.

[0100] Step 1707: The activity amount calculation device 100 calculates the sum (total energy) of the whole-body energy expenditure Em during movement and the whole-body energy expenditure Es when stationary, as the whole-body energy expenditure E, using the activity amount calculation unit 126 and equation (10) in FIG. 16 .

[0101] Step 1708: The activity amount calculation device 100 calculates whole-body calorie expenditure Ec by applying the whole-body energy expenditure E to calculation formula (6) in FIG. 11A using the activity amount calculation unit 126, and calculates METs A (hourly whole-body activity amount A) from the whole-body energy expenditure E by applying the whole-body calorie expenditure Ec, "H (= the time between the start time and the current time (hours (hours))")," and "W (= the weight of the user Us1)" to calculation formula (7A) in FIG. 11B. Furthermore, the activity amount calculation device 100 calculates exercise Aw by applying METs A and H to calculation formula (7B) in FIG. 11B. The parameters calculated in the activity amount calculation process are stored in the database 111 as result data 117. Furthermore, in the activity amount calculation process, the activity amount calculation device 100 may further calculate the energy consumption for each calculation reference joint point Ji by adding together the energy consumption calculated by integrating the in-motion muscle power Pmi of the body part BJi corresponding to each calculation reference joint point Ji during the calculation period and the energy consumption calculated by integrating the resting muscle power Psi of the body part BJi corresponding to each calculation reference joint point Ji during the calculation period, using the activity amount calculation unit 126. Furthermore, the activity amount calculation device 100 may calculate METs, calories burned, exercise, etc. for each calculation reference joint point Ji based on the energy consumption for each calculation reference joint point Ji.

[0102] <Effects> As described above, the activity calculation device 100 according to the second embodiment of the present invention calculates the torque Ti applied to each joint point Ji and the muscle power during exercise and the muscle power at rest for each corresponding body part BJi based on the torque Ti applied to each joint point Ji, based on the muscle power during exercise and the muscle power at rest for each corresponding body part BJi. Furthermore, the activity calculation device 100 according to the second embodiment calculates the whole-body energy expenditure Em during exercise and the whole-body energy expenditure Es at rest based on the muscle power during exercise and the muscle power at rest for each corresponding body part BJi. The activity calculation device 100 according to the second embodiment then calculates an activity amount, which is a parameter that quantifies the physical activity of the user Us1 during exercise (measurement period), such as the whole-body energy expenditure E, METs A, and exercises Aw, based on the whole-body energy expenditure Em during exercise and the whole-body energy expenditure Es. This allows the activity calculation device 100 according to the second embodiment to more accurately measure (estimate) the activity amount, which is a parameter that quantifies the physical activity of the user Us1, based on the video (image) of the user Us1 performing exercise.

[0103] <<Third Embodiment>> An activity amount calculation device 100 according to a third embodiment of the present invention will be described. The activity amount calculation device 100 according to the third embodiment differs from the activity amount calculation device 100 according to the first embodiment only in that, in the activity amount calculation process, in addition to the whole-body exercise energy expenditure Em, a whole-body acceleration-based energy expenditure Eacc is calculated from the acceleration of a representative joint point Ji (also referred to as a "representative joint point Ji"), and the whole-body exercise energy expenditure E is calculated based on the whole-body exercise energy expenditure Em and the whole-body acceleration-based energy expenditure Eacc. Therefore, the following description will mainly focus on this difference.

[0104] (Activity amount calculation process) 18 is a flowchart showing a processing flow executed by the activity amount calculation device 100 according to the third embodiment. The activity amount calculation device 100 executes this processing flow every time a predetermined time elapses from the exercise menu start time tst to the exercise menu end time tend.

[0105] The activity amount calculation device 100 starts the process from step 1800, and after sequentially executing the processes of step 1801 to step 1807 described below, proceeds to step 1895 and temporarily ends this processing flow.

[0106] Step 1801: The activity amount calculation device 100 executes the same process as that of step 901 in FIG. 9 described above.

[0107] Step 1802: The activity amount calculation device 100 executes the same process as the process of step 902 in FIG. 9 described above.

[0108] Step 1803: The activity amount calculation device 100 executes the same process as the process of step 903 in FIG. 9 described above.

[0109] Step 1804: The activity amount calculation device 100 executes the same process as the process of step 904 in FIG. 9 described above.

[0110] Step 1805: The activity amount calculation device 100 calculates, using the activity amount calculation unit 126, whole-body acceleration-based energy expenditure Eacc for the period from the start time ts to the current time tp, based on the acceleration of a predetermined joint point (representative joint point). In this example, the representative joint point Ji is the waist center. More specifically, the activity amount calculation device 100 calculates the whole-body acceleration-based energy expenditure Eacc by applying the waist-center acceleration to a well-known calculation algorithm. Note that the representative joint point Ji may be a joint point Ji other than the waist center, or may be multiple joint points Ji.

[0111] Step 1806: The activity amount calculation device 100 calculates the sum (total energy) of the whole-body exercise energy expenditure Em and the whole-body acceleration-based energy expenditure Eacc as the whole-body energy expenditure E using the activity amount calculation unit 126 and equation (11) in FIG. 19 .

[0112] Step 1807: The activity amount calculation device 100 calculates the whole-body calorie expenditure Ec by applying the whole-body energy expenditure E to the calculation formula (6) of FIG. 11A, and calculates METs A (hourly whole-body activity amount A) from the whole-body energy expenditure E by applying the whole-body calorie expenditure Ec, "H (= the time between the start time and the current time (hours (hours))")," and "W (= the weight of the user Us1)" to the calculation formula (7A) of FIG. 11B. Furthermore, the activity amount calculation device 100 calculates the exercise Aw by applying the METs A and H to the calculation formula (7B) of FIG. 11B. The parameters calculated in the activity amount calculation process are stored in the database 111 as result data 117.

[0113] <Effects> As described above, the activity calculation device 100 according to the third embodiment of the present invention calculates the torque Ti applied to each joint point Ji and the muscle power during exercise for each corresponding body part BJi based on the torque Ti applied to each joint point Ji based on the muscle power during exercise for each corresponding body part BJi. Furthermore, the activity calculation device 100 according to the third embodiment calculates the whole-body exercise energy expenditure Em based on the muscle power during exercise for each corresponding body part BJi, and further calculates the whole-body acceleration-based energy expenditure Eacc based on the acceleration of the representative joint point. The activity calculation device 100 according to the third embodiment then calculates the activity amount, which is a parameter quantifying the physical activity of the user Us1 during exercise (measurement period), such as the whole-body energy expenditure E, METs A, and exercises Aw, based on the whole-body exercise energy expenditure Em and the whole-body acceleration-based energy expenditure Eacc. This allows the activity calculation device 100 according to the third embodiment to more accurately measure (estimate) the activity amount, which is a parameter quantifying the physical activity of the user Us1, based on the exercise video (image) of the user Us1.

[0114] <<Fourth Embodiment>> An activity amount calculation device 100 according to a fourth embodiment of the present invention will be described. The activity amount calculation device 100 according to the fourth embodiment differs from the activity amount calculation device 100 according to the first embodiment only in that, in the activity amount calculation process, the activity amount calculation device 100 calculates whole-body stationary energy expenditure Es in addition to whole-body motion energy expenditure Em, further calculates whole-body acceleration-based energy expenditure Eacc from the acceleration of the representative joint point Ji, and calculates whole-body acceleration-based energy expenditure E based on the whole-body motion energy expenditure Em, the whole-body stationary energy expenditure Es, and the whole-body acceleration-based energy expenditure Eacc. Therefore, the following description will mainly focus on this difference.

[0115] (Activity amount calculation process) 20 is a flowchart showing a processing flow executed by the activity amount calculation device 100 according to the fourth embodiment. The activity amount calculation device 100 executes this processing flow every time a predetermined time elapses from the exercise menu start time tst to the exercise menu end time tend.

[0116] The activity amount calculation device 100 starts the process from step 2000, and after sequentially executing the processes of step 2001 to step 2009 described below, proceeds to step 2095 and temporarily ends this processing flow.

[0117] Step 2001: The activity amount calculation device 100 executes the same process as that of step 901 in FIG. 9 described above.

[0118] Step 2002: The activity amount calculation device 100 executes the same process as the process of step 902 in FIG. 9 described above.

[0119] Step 2003: The activity amount calculation device 100 executes the same process as the process of step 1703 in FIG. 17 described above.

[0120] Step 2004: The activity amount calculation device 100 executes the same process as the process of step 1704 in FIG. 17 described above.

[0121] Step 2005: The activity amount calculation device 100 executes the same process as the process of step 1705 in FIG. 17 described above.

[0122] Step 2006: The activity amount calculation device 100 executes the same process as the process of step 1706 in FIG. 17 described above.

[0123] Step 2007: The activity amount calculation device 100 executes the same process as the process of step 1805 in FIG. 18 described above.

[0124] Step 2008: The activity amount calculation device 100 calculates the sum (total energy) of the whole-body energy expenditure during movement Em, the whole-body energy expenditure at rest Es, and the whole-body acceleration-based energy expenditure Eacc as the whole-body energy expenditure E using the calculation formula (12) in FIG. 21 by the activity amount calculation unit 126.

[0125] Step 2009: The activity amount calculation device 100 calculates whole-body calorie expenditure Ec by applying the whole-body energy expenditure E to calculation formula (6) in FIG. 11A using the activity amount calculation unit 126, and calculates METs A (hourly whole-body activity amount A) from the whole-body energy expenditure E by applying the whole-body calorie expenditure Ec, "H (= the time between the start time and the current time (hours (hours))")," and "W (= the weight of the user Us1)" to calculation formula (7A) in FIG. 11B. Furthermore, the activity amount calculation device 100 calculates exercise Aw by applying METs A and H to calculation formula (7B) in FIG. 11B. The parameters calculated in the activity amount calculation process are stored in the database 111 as result data 117.

[0126] <Effects> As described above, the activity amount calculation device 100 according to the fourth embodiment of the present invention calculates the torque Ti applied to each joint point Ji and the muscle power during exercise and the muscle power at rest for each corresponding body part BJi based on the torque Ti applied to each joint point Ji, based on a video (image) of the user Us1 performing exercise. Furthermore, the activity amount calculation device 100 according to the fourth embodiment calculates the whole-body energy expenditure Em during exercise and the whole-body energy expenditure Es at rest based on the muscle power during exercise and the muscle power at rest for each corresponding body part BJi, and further calculates the whole-body acceleration-based energy expenditure Eacc based on the acceleration of the representative joint point. The activity amount calculation device 100 according to the fourth embodiment then calculates the whole-body energy expenditure E, METs A, exercises Aw, and other parameters quantifying the physical activity of the user Us1 during exercise (measurement period), based on the whole-body energy expenditure Em, the whole-body energy expenditure Es, and the whole-body acceleration-based energy expenditure Eacc. As a result, the activity amount calculation device 100 according to the fourth embodiment can more accurately measure (estimate) the activity amount, which is a parameter that quantifies the physical activity of the user Us1, from the exercise video (image) of the user Us1.

[0127] <<First Modification>> In each of the above embodiments, in the activity amount calculation process, the whole-body consumed energy E and the like may be calculated by calculating the joint torques Ti and angular velocities wi for only some of all the calculation reference joint points Ji. Examples of some of the calculation reference joint points Ji include the calculation reference joint points Ji (right shoulder, shoulder center, left shoulder, back, waist center, right waist, and left waist) indicated by black dots corresponding to the trunk muscles corresponding to the skeleton indicated by the thick lines in FIG. 22. In this case, the first modified example executes step A described below instead of steps 902, 1702, 1802, and 2002 in each of the above embodiments.

[0128] Step A: The activity amount calculation device 100 calculates, by the activity amount calculation unit 126, joint torques Ti and angular velocities wi at the current time tp for some of the calculation reference joint points Ji of the user Us1.

[0129] According to the first variant, even if a part of the body (e.g., part of an arm, part of a leg, etc.) is not recognized as an image, the amount of activity, which is a parameter that quantifies the physical activity of user Us1, can be measured (estimated) from a video (image) of user Us1 performing exercise.

[0130] <<Second Modification>> In a second modified example of the activity amount calculation device 100, the calculation processing method of the activity amount calculation processing, the unit of activity amount, etc. are changed based on input information on a setting screen. Fig. 23 shows an example of an activity amount calculation method setting screen GM24. The activity amount calculation method setting screen GM24 includes an exercise movement evaluation setting field 2301, a stationary movement evaluation setting field 2302, a trunk muscle setting field 2303, a representative joint point acceleration consideration setting field 2304, an activity amount unit setting field 2305, a ninth pull-down button 2306, a tenth pull-down button 2307, an eleventh pull-down button 2308, a twelfth pull-down button 2309, and a thirteenth pull-down button 2310.

[0131] The exercise movement evaluation setting field 2301, the stationary movement evaluation setting field 2302, the trunk muscle setting field 2303, the representative joint point acceleration consideration setting field 2304, and the activity amount unit setting field 2305 are referred to as the "setting fields" when there is no need to particularly distinguish between them.

[0132] When each of the ninth pull-down button 2306, the tenth pull-down button 2307, the eleventh pull-down button 2308, and the twelfth pull-down button 2309 is operated, a pull-down list containing "Yes" and "No" to be input in the respective setting fields is displayed. When the thirteenth pull-down button 2310 is operated, a pull-down list containing "METs," "kcal," and "J" (or "kJ"), the units of activity amount to be input in the activity amount unit setting field 2305, is displayed. When the register button Bt51 is operated, an activity amount calculation method is set based on the information input in the respective setting fields.

[0133] Specifically, for example, when only the exercise action evaluation setting field 2301 is set to "Yes" among the exercise action evaluation setting field 2301 and the stationary action evaluation setting field 2302, and the register button Bt51 is operated, the activity amount in the unit set in the activity amount unit setting field 2305 is calculated by the activity amount calculation process shown in Fig. 9. Then, the calculated activity amount is displayed, for example, in the energy expenditure display field 1202 of the measurement screen GM12 in Fig. 12.

[0134] When the register button Bt51 is operated with both the exercise action evaluation setting field 2301 and the stationary action evaluation setting field 2302 set to "Yes," the activity amount in the unit set in the activity amount unit setting field 2305 is calculated by the activity amount calculation process shown in Fig. 17. Then, the calculated activity amount is displayed, for example, in the energy expenditure display field 1202 of the measurement screen GM12 in Fig. 12.

[0135] When the registration button Bt51 is operated while the representative joint point acceleration consideration setting field 2304 is set to "Yes," the activity amount in the unit set in the activity amount unit setting field 2305 is calculated by processing (for example, the activity amount calculation processing shown in either FIG. 18 or FIG. 20) that takes into account the acceleration reference energy calculated based on the acceleration of the representative joint point, in accordance with the input information in the exercise action evaluation setting field 2301 and the stationary action evaluation setting field 2302. Then, the calculated activity amount is displayed, for example, in the energy expenditure display field 1202 of the measurement screen GM12 in FIG. 12.

[0136] <<Third Modification>> In each of the above embodiments, the estimated mass Mi and torque arm length Li of the corresponding body part BJi may be calculated as described below. Fig. 24 is a diagram for explaining a method for calculating the estimated mass Mi and torque arm length Li of the corresponding body part BJi in a third modified example. A reference mass Mib of each corresponding body part BJi is set in advance with respect to a reference height D of a standard build. Similarly, a reference torque arm length Lib of each corresponding body part BJi is set in advance with respect to a reference height D of a standard build.

[0137] The third modified example calculates the estimated mass Mi of each corresponding body part BJi based on the height D' of the user Us1 using equation (13) in Fig. 24. That is, the third modified example calculates the estimated mass Mi of the corresponding body part BJi by multiplying a preset reference mass Mib by a ratio that is the cube of the value obtained by dividing the height D' of the user Us1 by a preset reference height D (D (= 170 cm) in this example).

[0138] In the third modification, the torque arm length Li of each corresponding body part BJi is calculated based on the height D' of the user Us1 using the calculation formula (14) in Fig. 24. That is, in the third modification, the torque arm length Li is calculated by multiplying the preset reference torque arm length Lib by a ratio obtained by dividing the height D' of the user Us1 by the preset reference height D. <<Fourth Modification>> In each of the above embodiments, the estimated mass Mi of the corresponding body part BJi may be calculated as follows. Fig. 25 is a diagram for explaining a method for calculating the estimated mass Mi of the corresponding body part BJi in a fourth modified example. A reference area Sb of the body contour of a standard body type is set in advance.

[0139] The fourth modification calculates the estimated mass Mi of each corresponding body part BJi based on the contour area S' of the user Us1 using equation (15) in Fig. 25. That is, the fourth modification obtains the contour area S' of the user Us1 based on the exercise video, and calculates the estimated mass Mi of the corresponding body part by multiplying a preset reference mass Mib by a ratio that is the square of the value obtained by dividing the contour area S' of the user Us1 by a preset reference area Sb.

[0140] <<Fifth Modification>> There is a case where the user Us1 exercises using equipment (for example, dumbbells). In this case, in each of the above embodiments, the mass Mi of the corresponding body part BJi on which the mass of the equipment acts may be replaced with the mass Mi obtained by adding the mass of the equipment to the mass Mi (Mi = Mi + equipment mass) to calculate the activity amount. Note that the mass of the equipment may be an estimated mass based on a video of the exercise or a mass of the equipment input in advance to the activity amount calculation device 100.

[0141] <<Sixth Modification>> The sixth modification executes a calibration measurement for calibration. For the calibration, the energy consumption of the user Us1 during exercise is measured both by the method based on the exercise video (image) described above and by a method using a known activity meter. For example, assume that during a calibration measurement (a certain calibration measurement period), the output value of METs (activity amount) measured using a method using an activity meter is P, and the METs (activity amount) calculated from the exercise video (image) is Q. In this case, the sixth modification calculates the final METs (activity amount) by multiplying the METs (activity amount) calculated from the exercise video (image) by a correction value (=P / Q).

[0142] <<Seventh Modification>> The seventh modified example performs a calibration measurement for calibration. For the calibration, the energy consumption during exercise of user Us1 is measured using both the method based on the exercise video (image) described above and the method using heart rate. For example, assume that during a calibration measurement (a certain calibration measurement period), the output value of METs (activity amount) measured using a known method using heart rate is P, and the METs (activity amount) calculated from the image is Q. In this case, the seventh modified example calculates the final METs (activity amount) by multiplying the METs (activity amount) calculated from the exercise video (image) by a correction value (=P / Q).

[0143] <<Application Example>> FIG. 26 is a diagram illustrating a system as an application example of the activity amount calculation device 100. As shown in FIG. 26, the system includes a health management system 2600, a terminal device 200, and a wearable device 2700 (e.g., a smart watch) having a function as an activity meter. These are communicably connected to each other via a network NW1. The health management system 2600 is configured on the cloud and includes the activity amount calculation device 100. As described above, the activity amount calculation device 100 can acquire and calculate data related to the exercise of the user Us1. Furthermore, the activity amount calculation device 100 can acquire and calculate data related to the walking, running, and other movements of the user Us1 collected by the wearable device 2700. The activity amount calculation device 100 stores an activity amount based on the data related to the exercise of the user Us1 and an activity amount based on the data related to the walking, running, and other movements of the user Us1 collected by the wearable device 2700. The health management system 2600 provides services (e.g., an activity amount calculation service, a service that calculates and provides the total amount of activity of user Us1 over a week, etc.) to user Us1 (terminal device 200 and / or wearable device 2700) based on data calculated and collected by the activity amount calculation device 100.

[0144] <<Other variations>> The present invention is not limited to the above-described embodiments, modifications, and applications, and various modifications may be adopted within the scope of the present invention. Furthermore, the above-described embodiments, modifications, and applications may be combined with each other without departing from the scope of the present invention. For example, in the above-described embodiments, the joint points Ji, the calculation reference joint points Ji, and the corresponding body parts BJi are not limited to those described above. For example, in the above-described embodiments, the torque arm length Li of each corresponding body part BJi is calculated for each calculation reference joint point Ji based on the exercise execution video. However, the torque arm length Li of each corresponding body part BJi may be a preset torque arm length Li. For example, in the above-described embodiments, part of the processing performed by the activity amount calculation device 100 may be performed by the terminal device 200. For example, in the above-described embodiments, skeletal data may be extracted on the terminal device 200 side based on the exercise execution video. [Explanation of symbols]

[0145] 100...activity amount calculation device, 110...storage device, 111...database, 112...user information, 113...exercise menu data, 114...skeleton data, 115...motion stillness determination data, 116...activity amount data, 117...result data, 120...processor, 121...data processing unit, 122...user information management unit, 123...exercise menu instruction unit, 124...skeleton data extraction unit, 125...motion stillness determination unit, 127...result processing unit, 200...terminal device, Us1...user

Claims

1. An activity amount calculation device including an information processing device that processes data based on a video captured by a user, The information processing device includes: calculating angular velocities of a plurality of calculation reference joint points among a plurality of joint points based on skeletal data of the user acquired based on video images of the user during a calculation target period; For each of the calculated reference joint points in motion, a torque calculation that calculates torque by multiplying a rotational component of estimated gravity acting on the corresponding body part, which is obtained by multiplying an estimated mass of the corresponding body part around the joint point by gravitational acceleration, by a torque arm length that is the length between the calculation reference joint point and the center of gravity of the corresponding body part; and a muscle power calculation that calculates muscle power during exercise of the corresponding body part around the joint point during the calculation period by multiplying the calculated torque by the angular velocity in the rotational direction. calculating an integral value of the total muscle power during exercise for the calculation period as the energy expenditure during whole-body exercise; Calculating whole-body energy expenditure based on at least the whole-body exercise energy expenditure; calculating at least one of the whole-body energy expenditure and a parameter derived from the whole-body energy expenditure as an activity amount, which is a parameter quantifying the physical activity of the user; It was configured as follows: Activity amount calculation device.

2. The activity calculation device according to claim 1 , The information processing device includes: For each of the calculated reference joint points in a rest state, performing the torque calculation and a resting muscle power calculation for calculating resting muscle power of the corresponding body part around the joint point by multiplying the calculated torque by a predetermined coefficient; calculating an integral value of the total resting muscle power for the calculation period as whole-body resting energy expenditure; The whole-body energy expenditure is calculated by adding up the whole-body motion energy expenditure and the whole-body stationary energy expenditure. It was configured as follows: Activity amount calculation device.

3. The activity calculation device according to claim 1 , The information processing device includes: further calculating an acceleration-based whole-body acceleration-based energy expenditure for the calculation period based on the acceleration of a representative joint point that is one or more representative joint points among the plurality of joint points; calculating the whole-body energy expenditure by adding up the whole-body exercise energy expenditure and the whole-body acceleration-based energy expenditure; It was configured as follows: Activity amount calculation device.

4. The activity calculation device according to claim 1 , The information processing device includes: For each of the calculated reference joint points in a rest state, performing the torque calculation and a resting muscle power calculation for calculating resting muscle power of the corresponding body part around the joint point by multiplying the calculated torque by a predetermined coefficient; calculating an integral value of the total resting muscle power for the calculation period as whole-body resting energy expenditure; further calculating an acceleration-based whole-body acceleration-based energy expenditure for the calculation period based on the acceleration of a representative joint point that is one or more representative joint points among the plurality of joint points; calculating the whole-body energy expenditure by adding up the whole-body motion energy expenditure, the whole-body stationary energy expenditure, and the whole-body acceleration-based energy expenditure; It was configured as follows: Activity amount calculation device.

5. The activity calculation device according to claim 1 , The information processing device includes: converting the whole body energy expenditure into whole body calories by multiplying the whole body energy expenditure by a first conversion coefficient; The whole body calorie consumption is divided by a value obtained by multiplying a second conversion coefficient by the time corresponding to the calculation period and the user's weight, and then adding "1" to the result. METs are calculated as a parameter derived from the whole-body energy expenditure. It was configured as follows: Activity amount calculation device.

6. The activity calculation device according to claim 1 , The information processing device includes: Among the plurality of joint points, only the joint points corresponding to the trunk muscles of the user are set as the calculation reference joint points. It was configured as follows: Activity amount calculation device.

7. The activity calculation device according to claim 1 , The information processing device acquires the height of the user, calculating the estimated mass of the corresponding body part by multiplying a predetermined reference mass by a ratio that is the cube of a value obtained by dividing the height of the user by a predetermined reference height; It was configured as follows: Activity amount calculation device.

8. The activity calculation device according to claim 1 , The information processing device acquires the height of the user, calculating the torque arm length by multiplying a predetermined reference torque arm length by a ratio obtained by dividing the height of the user by a predetermined reference height; It was configured as follows: Activity amount calculation device.

9. The activity calculation device according to claim 1 , the information processing device acquires an area of ​​a contour of the user based on the captured video; calculating the estimated mass of the corresponding body part by multiplying a predetermined reference mass by a ratio that is the square of a value obtained by dividing an area of ​​the user's outline by a predetermined reference area; It was configured as follows: Activity amount calculation device.

10. The activity calculation device according to claim 1 , The information processing device includes: When the user is using an appliance, the amount of activity is calculated using a mass obtained by adding a mass of the appliance used by the user to the estimated mass of the corresponding body part, instead of the estimated mass of the corresponding body part. It was configured as follows: Activity amount calculation device.

11. The activity calculation device according to claim 1 , The information processing device includes: During a given calibration measurement period, acquiring a calibration activity amount calculated in a similar manner to the calculation of the activity amount based on skeletal data of the user acquired based on the captured video of the user, and an activity meter-measured activity amount when the exercise of the user is measured by an activity meter; calculating a correction value based on the calibration activity amount and the activity amount measured by the activity meter; calibrating the amount of activity by applying the correction value to the amount of activity; It was configured as follows: Activity amount calculation device.

12. The activity calculation device according to claim 1 , The information processing device includes: During a given calibration measurement period, acquiring a calibration activity amount calculated in a similar manner to the calculation of the activity amount based on skeletal data of the user acquired based on the captured video of the user, and a heart rate-based activity amount measured based on the heart rate of the user; calculating a correction value based on the calibration activity amount and the heart rate reference activity amount; calibrating the amount of activity by applying the correction value to the amount of activity; It was configured as follows: Activity amount calculation device.

13. The activity calculation device according to claim 1 , the information processing device acquires the captured video of the user or skeletal data of the user from a terminal device equipped with an external imaging device through communication; calculating the amount of activity based on the captured video or the skeletal data of the user; transmitting the calculated amount of activity to the terminal device; It was configured as follows: Activity amount calculation device.

14. 1. A method for calculating an activity amount using an information processing device that processes data based on a video captured by a user, comprising: By the information processing device, calculating angular velocities of a plurality of calculation reference joint points among a plurality of joint points based on skeletal data of the user acquired based on video images of the user during a calculation target period; For each of the calculated reference joint points in motion, a torque calculation that calculates torque by multiplying a rotational component of estimated gravity acting on the corresponding body part, which is obtained by multiplying an estimated mass of the corresponding body part around the joint point by gravitational acceleration, by a torque arm length that is the length between the calculation reference joint point and the center of gravity of the corresponding body part; and a muscle power calculation that calculates muscle power during exercise of the corresponding body part around the joint point during the calculation period by multiplying the calculated torque by the angular velocity in the rotational direction. calculating an integral value of the total muscle power during exercise for the calculation period as the energy expenditure during whole-body exercise; Calculating whole-body energy expenditure based on at least the whole-body exercise energy expenditure; calculating at least one of the whole-body energy expenditure and a parameter derived from the whole-body energy expenditure as an activity amount, which is a parameter quantifying the physical activity of the user; Activity amount calculation method.

Images