Display control device, display control system, display control method, and program

Abstract

Provided are a display control device, a display control system, a display control method, and a display control program with which is is possible to precisely visualize movement of a body on a terminal device. A subject data storage unit (31) of the display control device (13) stores, for a subject, a body size of a prescribed region in the vertical direction in a standing posture. A three-dimensional human body model storage unit (32) stores a three-dimensional human body model displayed in a moving image and a set body size. A position calculation unit (38) determines center-of-gravity position time-series data and a ground-contact position on the basis of three-dimensional joint position time-series data which was acquired. A correction unit (37) corrects both a three-dimensional joint position and a center-of-pressure position on the basis of a correction factor. A display control unit (41) generates motion data showing movement of the three-dimensional human body model, and analysis data for at least any one among the ground reaction force, the center-of-gravity position, and a ground-contact marker shown together with the three-dimensional human body model in the moving image.

Description

Display control device, display control system, display control method and program

[0001] The present invention relates to a display control device, a display control system, a display control method, and a display control program.

[0002] Understanding a person's physical condition through their movements has long been practiced in various settings, such as medical care and sports. Body movements can be grasped visually or using measuring equipment, but in recent years, many techniques for analyzing them using motion capture have been proposed.

[0003] For example, Patent Document 1 describes an information processing device including an acquisition unit and a conversion unit. The acquisition unit acquires position dimension data of an object expressed in a generalized coordinate system. The conversion unit acquires at least acceleration dimension data from multiple inertial sensors attached to the object, and converts the acquired acceleration data into acceleration dimension data expressed in the generalized coordinate system based on the position dimension data of the object expressed in the generalized coordinate system. The object includes multiple segments and joints connecting two or more segments, and the generalized coordinate system includes variables representing rotation angles around one or more axes for each joint. This information processing device further includes an estimation unit. The estimation unit further includes an estimation unit that estimates at least one of an external force acting on the object and a torque generated at a joint based on the position dimension data expressed in the generalized coordinate system, velocity dimension data of the object obtained by differentiating the position dimension data, and acceleration dimension data converted by the conversion unit. The estimation unit estimates at least one of the external forces acting on the object and the torques generated at the joints by performing either or both of forward dynamics calculations and inverse dynamics calculations. Analysis techniques using three-dimensional joint positions (the three-dimensional positions of the joints), three-dimensional joint angles (the angles of the joints), and external forces such as ground reaction forces are effective in understanding the movements of the body.

[0004] In addition to the above-mentioned method using an inertial sensor (IMU, Inertial Measurement Unit), there are also methods using optical motion capture and multiple video cameras for detecting movement.

[0005] Japanese Patent Application Laid-Open No. 2020-201138

[0006] In medical practice, for example, when checking body movements during rehabilitation or when checking form in sports, there is a demand for checking body movements using a terminal device immediately after the subject moves. However, even if body movements are analyzed using the information processing device or optical motion capture described in Patent Literature 1, when the analysis results are displayed on a display unit such as a display of the terminal device, they often result in misalignment between the floor and the body, and the movement is not precisely visualized.

[0007] Therefore, an object of the present invention is to provide a display control device, a display control system, a display control method, and a display control program that can precisely visualize body movements on a terminal device.

[0008] The display control device of the present invention includes a subject data storage unit, a three-dimensional human body model storage unit, a correction factor calculation unit, an acquisition unit, a position calculation unit, a correction unit, and a display control unit. The subject data storage unit stores the body size of a predetermined region in the vertical direction in an upright posture of a subject whose movement is to be analyzed. The three-dimensional human body model storage unit stores a three-dimensional human body model displayed in a video as a subject who has performed a movement, and a body size set for the three-dimensional human body model. The correction factor calculation unit calculates a correction factor for correcting the subject's body size to the body size set for the three-dimensional human body model. The acquisition unit acquires time-series data for three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center positions for the subject during movement. The position calculation unit calculates time-series data for the subject's center of gravity position based on the time-series data for the three-dimensional joint positions, and calculates the contact positions on the floor where the feet touch the ground in chronological order. The correction unit corrects the three-dimensional joint positions and the foot pressure center positions based on the correction factors. The display control unit generates motion data indicating the movement of the three-dimensional human body model based on time-series data of the three-dimensional joint angles, floor reaction forces, corrected three-dimensional joint positions, corrected foot pressure center position, center of gravity position, and ground contact position, and generates analysis data of at least one of the floor reaction forces and ground contact markers indicating the center of gravity and ground contact position, which are shown together with the three-dimensional human body model in the video. The position calculation unit calculates the center of gravity position and ground contact position based on the three-dimensional joint positions before and after correction by the correction unit. When the center of gravity position is calculated based on the three-dimensional joint positions before correction, the correction unit corrects the center of gravity position based on a correction factor and sends the corrected center of gravity position to the display control unit. When the ground contact position is calculated based on the three-dimensional joint positions before correction, the correction unit corrects the ground contact position based on the correction factor and sends the corrected ground contact position to the display control unit.

[0009] When the body size of the subject is SO and the body size of the three-dimensional human body model is SM, the correction factor is preferably calculated as SM / SO.

[0010] The body size is preferably the size of a predetermined area set on the lower body, and more preferably the leg length.

[0011] It is preferable that the display control unit transmits the three-dimensional human body model to the display terminal that displays the video before transmitting the motion data and the analysis data.

[0012] The display control system of the present invention includes a client terminal and the above-described display control device. The client terminal displays a moving subject in a video using a three-dimensional human body model displayed in a video as the subject whose motion is to be analyzed. The display control device sends motion data indicating the motion of the three-dimensional human body model to the client terminal.

[0013] The display control method of the present invention causes a computer to execute a subject data storage step, a three-dimensional human body model storage step, a correction factor calculation step, an acquisition step, a position calculation step, a correction step, and a display control step. The subject data storage step stores the body size of a predetermined region in the vertical direction in an upright posture of a subject whose movement is to be analyzed. The three-dimensional human body model storage step stores a body size set for a three-dimensional human body model displayed in a video as a subject who has performed a movement. The correction factor calculation step calculates a correction factor for correcting the subject's body size to the body size set for the three-dimensional human body model. The acquisition step acquires time-series data for three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center positions for the subject during movement. The position calculation step calculates time-series data for the subject's center of gravity position based on the time-series data for the three-dimensional joint positions, and calculates the foot contact positions on the floor where the feet touch the ground in chronological order. The correction step corrects the three-dimensional joint positions and foot pressure center positions based on the correction factors. The display control step generates motion data indicating the movement of the three-dimensional human body model based on time-series data of the three-dimensional joint angles, floor reaction forces, corrected three-dimensional joint positions, corrected foot pressure center position, center of gravity position, and ground contact position, and generates analysis data of at least one of the floor reaction forces, the center of gravity position, and a ground contact marker indicating the ground contact position, which are shown together with the three-dimensional human body model in the video. The position calculation step determines the center of gravity position and the ground contact position based on the three-dimensional joint positions before and after correction by the correction step. The correction step, when the center of gravity position is determined based on the three-dimensional joint positions before correction, corrects the center of gravity position based on a correction factor and sends the corrected center of gravity position to the display control step, and when the ground contact position is determined based on the three-dimensional joint positions before correction, corrects the ground contact position based on the correction factor and sends the corrected ground contact position to the display control step.

[0014] The display control program of the present invention causes a computer to execute the above steps.

[0015] According to the present invention, body movements can be visualized in detail on a terminal device.

[0016] It is an explanatory diagram of a display control system which is an embodiment. It is an explanatory diagram of an image displayed on a client terminal. It is a configuration diagram of a display control system of a first embodiment. It is an explanatory diagram of body size. It is a processing flow of a display control system. It is a configuration diagram of a display control system of a second embodiment. It is a processing flow of a display control system.

[0017] [First Embodiment] The display control system 10 shown in FIG. 1 is an example of a display control system of the present invention. It is a motion analysis display control system that analyzes the body movements of an analysis subject (hereinafter simply referred to as "subject") and displays the analysis results on a display unit (display) of a terminal device. The display control system 10 includes a detection unit 11, a display control device 13, and a client terminal 15, which is a terminal device. The detection unit 11, the client terminal 15, and the display control device 13 communicate with each other via a communication network CN, i.e., transmit and receive various data. For example, the detection unit 11 detects the body movements of the subject while they are moving and transmits data indicating the detection results to the display control device 13 via the communication network CN. The display control device 13 analyzes the body movements of the subject based on the received detection results and transmits data indicating the analysis results to the client terminal 15, which is an example of a terminal device. As a result, the analysis results are displayed on the display unit (display) of the client terminal 15, allowing the client using the client terminal 15 to understand the analysis results.

[0018] The detection unit 11 detects the body position in space and the positions of each part of the body of a moving subject as body movement. One detection unit 11 is sufficient for one subject. Although only one detection unit 11 is depicted in FIG. 1, there may be multiple detection units 11 so that multiple subjects each use one detection unit. Furthermore, multiple subjects may share one detection unit 11.

[0019] The client terminal 15 is a terminal used by a client, such as a physical therapist or a trainer who provides physical training to a subject. The client is not limited to these and may also be the subject being analyzed. The client terminal 15 is an example of a display terminal that displays video. It is composed of a display unit that displays analysis results, etc. transmitted from the display control device 13, an input unit that allows input operations such as a pause button B1 and a stop button B2 (see FIG. 2 ), which will be described later. The client terminal 15 may be any of a personal computer, a mobile terminal, a smartphone, etc., and is not particularly limited. While FIG. 1 illustrates only one client terminal 15, the number of client terminals 15 may be multiple, each used by a client. The client terminal 15 may also be used as a management terminal that manages various settings and / or data of the display control device 13. The management terminal may be provided separately from the client terminal 15.

[0020] The display control device 13 performs predetermined processing in response to input of each piece of information from the detection unit 11 and the client terminal 15. For example, the display control device 13 performs an analysis process on body movements based on the detection results acquired from the detection unit 11, and generates various data for displaying the analysis results as images on the client terminal 15. Details of the display control device 13 will be described later using another drawing.

[0021] The display control device 13 and the client terminal 15 are each composed of a computer. The client terminal 15 may operate on a browser in response to a program that runs on the browser sent from the display control device 13, or may be equipped with predetermined application software and operate by executing the program. The display control device 13 is equipped with a predetermined program, and by executing this program, it functions as each unit described below and performs predetermined processing.

[0022] The program incorporated into the display control device 13 causes the computer to execute a subject data storage step, a three-dimensional human body model storage step, a correction factor calculation step, an acquisition step, a position calculation step, a correction step, and a display control step. The subject data storage step stores the body size of a predetermined area in the vertical direction in an upright posture for a subject whose movement is to be analyzed. The three-dimensional human body model storage step stores a body size set for a three-dimensional human body model displayed in a video as the subject performing a movement. The correction factor calculation step calculates a correction factor for correcting the subject's body size to the body size set for the three-dimensional human body model. The acquisition step acquires time-series data for three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center positions for the subject performing a movement. The position calculation step calculates time-series data for the subject's center of gravity position based on the time-series data for the three-dimensional joint positions, and chronologically calculates the contact positions on the floor where the feet touch the ground. The correction step corrects the three-dimensional joint positions and foot pressure center positions based on the correction factors. The display control step generates motion data indicating the movement of the three-dimensional human body model based on time-series data of the three-dimensional joint angles, floor reaction forces, corrected three-dimensional joint positions, corrected foot pressure center position, center of gravity position, and ground contact position, and generates analysis data of at least one of the floor reaction forces, the center of gravity position, and a ground contact marker indicating the ground contact position, which are shown together with the three-dimensional human body model in the video. The position calculation step determines the center of gravity position and the ground contact position based on the three-dimensional joint positions before and after correction by the correction step. The correction step, when the center of gravity position is determined based on the three-dimensional joint positions before correction, corrects the center of gravity position based on a correction factor and sends the corrected center of gravity position to the display control step, and when the ground contact position is determined based on the three-dimensional joint positions before correction, corrects the ground contact position based on the correction factor and sends the corrected ground contact position to the display control step.

[0023] As shown in FIG. 2 , a video including a three-dimensional human body model MA, designated as a subject, is displayed on the display unit 18 of the client terminal 15. The movement of the three-dimensional human body model MA in the video reflects the body movement of the subject. In the example shown in FIG. 2 , the subject is represented by the three-dimensional human body model MA walking on a flat floor. A video of the three-dimensional human body model MA walking from right to left on the page of FIG. 2 is displayed in the video image section Ga formed as a partial area of ​​the image G. The video shows the start and end of walking detection by the detection unit 11 (see FIG. 1 ). The progress of the video is indicated by a progress bar BP extending linearly from "START" indicating the start of the video to "END" indicating the end of the video. The image G includes a pause button B1 for pausing the video in progress, a stop button B2 for stopping the video, and a progress button (not shown) for progressing the video. The progress of the video is turned on and off depending on the operation of these buttons. In addition, the orientation of the three-dimensional human body model MA can be changed by a swipe operation in which the client touches and moves their finger on the moving image section Ga of the display unit 18, which in this example is a touch panel display.

[0024] In this example, only the three-dimensional human body model MA whose position and posture correspond to the progression of the video is shown in the video image section Ga (drawn with a solid line), and the three-dimensional human body models MA before and after this one in time are not displayed, as shown by the virtual two-dot chain lines in Figure 2. That is, in this example, one three-dimensional human body model MA is displayed in time with the progression of the video. However, three-dimensional human body models MA before and after this one in time may also be displayed in the video image section Ga in addition to the three-dimensional human body model MA whose position and posture correspond to the progression, so that they can be distinguished from the three-dimensional human body model MA. In this way, the client terminal 15 displays the analysis results showing the subject's body movements using a video including the three-dimensional human body model MA.

[0025] In this example, the floor reaction force, the center of gravity position, and the foot's contact position on the floor are also displayed on image G as analysis results, although either one may be displayed. The floor reaction force is displayed both as a numerical display D1a using a numerical value and unit such as "XX N" and as a vector display D1b using a vector, although only one of these may be displayed. The starting point of the vector indicates the origin of the floor reaction force, the length indicates the magnitude of the floor reaction force, and the direction indicates the direction in which the floor reaction force acts, and the numerical display indicates the magnitude of the floor reaction force.

[0026] The three-dimensional human body model MA in this example is composed of skeletal structure information indicating the skeleton; mesh part information indicating mesh parts that schematically show the shapes of muscles, skin, bones, etc.; texture information indicating texture (feel) using color (hue, saturation, brightness), line type, line thickness, etc.; and link information associating the skeletal structure parts with the mesh parts. The three-dimensional human body model MA is a rigid link model with the pelvis as the root joint. Note that the three-dimensional human body model MA displayed in the animation in this example is a schematic representation of muscles, skin, etc., but in Figure 2, to avoid cluttering the illustration, only the outline of the body surface is shown. The link information is information that associates which mesh parts move in response to the movement of any joint in the skeletal structure.

[0027] The ground contact position is the contact position between the foot and the floor surface, and is therefore displayed on the floor surface FS. The shape of the ground contact position marker D2 indicating the ground contact position is not particularly limited, and in this example, the ground contact position marker D2 is a line in a video in which the floor surface FS is viewed from the side as shown in FIG. 2, and a foot shape is used as an example of the shape of an area with an area in a video in which the floor surface FS is viewed from above or below. Other shapes such as an oval, rectangle, or irregular shape may be used instead of a foot shape. In this example, the ground contact position marker D2 is displayed at all of the determined ground contact positions so that the chronological history can be understood, but it is also possible to display only some of them.

[0028] In FIG. 3 , the detection unit 11 includes a first detection device 21 that determines a three-dimensional joint angle (hereinafter referred to as a three-dimensional joint angle) and a three-dimensional joint position (hereinafter referred to as a three-dimensional joint position), and a second detection device 22 that determines a ground reaction force and a foot pressure center position. The first detection device 21 is not particularly limited as long as it determines the three-dimensional joint angle and the three-dimensional joint position. Examples include a device that determines the three-dimensional joint angle and the three-dimensional joint position by performing inverse kinematics based on marker data obtained by optical motion capture, an inertial sensor, or the like, and a device that determines the three-dimensional joint angle and the three-dimensional joint position by three-dimensional posture estimation using AI. In this way, the first detection device 21 does not need to directly detect the three-dimensional joint angle and the three-dimensional joint position, but may instead determine the three-dimensional joint angle and the three-dimensional joint position based on the detected elements. The joint angle may be expressed using any expression, such as a rotation vector, Euler angle, rotation matrix, or quaternion. Three-dimensional information such as three-dimensional joint angles and three-dimensional joint positions has the origin at (x, y, z) = (0, 0, 0) in three-dimensional space.

[0029] In this example, three-dimensional joint angles and three-dimensional joint positions are expressed in a generalized coordinate system. The generalized coordinate system is a coordinate system capable of expressing a three-dimensional posture using variables corresponding to the degrees of freedom of a model in which a subject is modeled as having rigid segments and joints connecting the segments. The generalized coordinate system in this embodiment, like the generalized coordinate system described in the aforementioned Patent Document 1, defines segments SG, such as the head, chest, abdomen, pelvis, left and right thighs, left and right shins, and left and right feet, and defines joints JT(i) connecting the segments (where i = 1 to N). Of the segments SG, the segment SG(B) corresponding to the pelvis is defined as the base segment SG(B). The six-degree-of-freedom displacements of segments SG other than the base segment SG(B) relative to the origin of the absolute coordinate system are not variables in the generalized coordinate system; only the base segment SG(B) is treated as displaceable with six degrees of freedom. The six degrees of freedom include three translational directions (XYZ) and three rotational directions (yaw, roll, and pitch). In this example, the three translational directions are time series data of three-dimensional joint positions of identification information (ID) associated with the pelvis, and the three rotational directions are joint rotation vectors as joint angles of the ID associated with the pelvis.

[0030] The detection unit 11 in this example includes a subject skeletal model generation unit (not shown), which generates a subject skeletal model by applying at least the subject's height to the incorporated general skeletal model. The general skeletal model is expressed in the generalized coordinate system, and the segments and joints, the center of gravity of each segment (hereinafter referred to as the segment center of gravity), and the mass ratio of the segments are set for a given height and weight. When at least the subject's height is input, the subject skeletal model generation unit performs scaling transformation based on the input data to generate a subject skeletal model tailored to the subject's skeleton. As a result, three-dimensional joint angles, three-dimensional joint positions, and the center of gravity of each segment can be calculated based on the subject skeletal model, for example, by performing inverse kinematics using marker data obtained by optical motion capture. However, the detection unit 11 may also calculate three-dimensional joint angles and three-dimensional joint positions using other methods. The mass ratio of the segments of the subject skeletal model may be the same as that of the general skeletal model. When the segment center of gravity positions and the mass proportions of the segments are determined by the detection unit 11 as in this example, the detection unit 11 sends the segment center of gravity positions and the mass proportions of the segments in addition to the three-dimensional joint angles and three-dimensional joint positions to the position calculation unit 38. The segment center of gravity positions and the mass proportions of the segments may be determined by the position calculation unit 38.

[0031] The skeletal structure of the general skeletal model and the three-dimensional human body model MA is the same, meaning that the number, configuration, and degrees of freedom of the joints are the same, and the positions (relative positions) of the joints on the body are also the same.

[0032] For example, a commercially available force plate can be used as the second detection device 22. The force plate includes, for example, a plate and four load cells arranged below the plate, and measures the force generated when the subject moves on the plate, thereby enabling the position of the center of foot pressure to be determined.

[0033] A detection device that estimates the floor reaction force and the center of foot pressure in addition to estimating the three-dimensional posture using AI may be used instead of the detection unit 11. Also, although the detection unit 11 is connected to the display control device 13, it may be made a part of the display control device 13.

[0034] The display control device 13 includes a subject data storage unit 31, a three-dimensional human body model storage unit 32, a correction factor calculation unit 33, an acquisition unit 36, a correction unit 37, a position calculation unit 38, and a display control unit 41. In addition to generating various data as described below, the display control unit 41 also functions as a control unit that comprehensively controls each unit of the display control device 13. The subject data storage unit 31 stores the body size of a predetermined area in the vertical direction of the subject in an upright posture. Details of the body size will be described later using another drawing. The subject's body size is entered, for example, by an input operation on the client terminal 15, and is stored in the subject data storage unit 31 via the display control unit 41 in response to this input.

[0035] The three-dimensional human body model storage unit 32 stores the three-dimensional human body model MA (see FIG. 2) displayed in the video and the above-mentioned body size set for the three-dimensional human body model MA.

[0036] The correction factor calculation unit 33 calculates a correction factor C for correcting the subject's body size to the body size set for the three-dimensional human body model MA. The correction factor C is preferably calculated using the formula SM / SO, where SO is the subject's body size and SM is the body size of the three-dimensional human body model MA. The correction factor calculation unit 33 in this example has a correction factor storage unit (not shown) that stores the calculated correction factor C, and stores the calculated correction factor C in this correction factor storage unit. However, the correction factor calculation unit 33 may also send the calculated correction factor C to the correction unit 37 for storage. In this case, the correction unit 37 may be configured to include a correction factor storage unit (not shown) that stores the correction factor C.

[0037] The acquisition unit 36 ​​acquires time-series data of the three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center position for the subject during movement from the detection unit 11. The acquisition unit 36 ​​sends the time-series data of the three-dimensional joint angles and floor reaction forces to the display control unit 41, and sends the time-series data of the three-dimensional joint positions and foot pressure center position to the correction unit 37.

[0038] The correction unit 37 corrects the three-dimensional joint positions and foot pressure center position that make up the time-series data based on the correction factors. In the following description, the corrected three-dimensional joint positions will be referred to as corrected three-dimensional joint positions, and the corrected foot pressure center position will be referred to as corrected foot pressure center position. The correction unit 37 outputs the time-series data of the corrected three-dimensional joint positions and the time-series data of the corrected foot pressure center position to the position calculation unit 38.

[0039] The position calculation unit 38 calculates time-series data of the center of gravity position of the subject based on the time-series data of the corrected three-dimensional joint positions input from the correction unit 37. The position calculation unit 38 calculates a multiplication value by multiplying the segment center of gravity position by the mass fraction of the segment for each segment of the skeletal model described above when the detection unit 11 calculated the three-dimensional joint positions, based on the segment center of gravity positions and mass fractions of the segment input from the correction unit 37. The position calculation unit 38 then calculates the center of gravity position of the entire body by summing up the multiplication values ​​calculated for all segments.

[0040] The position calculation unit 38 further calculates the contact position of the foot on the floor surface in chronological order based on the time-series data of the corrected three-dimensional joint positions input from the correction unit 37. In this example, when the position of at least one of the multiple markers associated with the heel, toe, etc. and set as being in contact with the floor surface FS (see FIG. 2) is within a range of 0.05 m from the floor surface FS and the speed of the marker is 0.8 m / s or less, the marker is defined as being in contact with the floor surface FS.

[0041] The position calculation unit 38 outputs the time series data of the corrected three-dimensional joint positions, the corrected foot pressure center position, and the center of gravity position, as well as the ground contact position, to the display control unit 41 .

[0042] The display control unit 41 generates motion data indicating the movement of the three-dimensional human body model MA based on time-series data on the three-dimensional joint angles, floor reaction forces, corrected three-dimensional joint positions, corrected foot pressure center positions, center of gravity positions, and ground contact positions, and also generates analysis data on at least one of the floor reaction forces, center of gravity positions, and ground contact markers to be shown together with the three-dimensional human body model MA (see FIG. 2 ) in the video. Specifically, the display control unit 41 generates motion data and ground contact markers based on time-series data on the three-dimensional joint angles input from the acquisition unit 36 ​​and the corrected three-dimensional joint positions, corrected foot pressure center positions, center of gravity positions, and ground contact positions input from the position calculation unit 38, and generates the floor reaction forces and center of gravity positions based on the floor reaction forces input from the acquisition unit 36 ​​and the center of gravity positions input from the position calculation unit 38. The display control unit 41 sends the generated motion data and analysis data to the client terminal 15. The display control unit 41 also sends the three-dimensional human body model MA to the client terminal 15 before transmitting the motion data and analysis data.

[0043] The client terminal 15 includes a display unit 18, an input unit 51, a three-dimensional human body model storage unit 52 that stores the acquired three-dimensional human body model MA, and a control unit 53. The input unit 51 is used by the client to perform various input operations, such as a login operation, input operations for various buttons such as the pause button B1 (see FIG. 2), the stop button B2 (see FIG. 2), and the progress button, as well as a login operation when starting use. The input unit 51 can be a keyboard, a mouse, or the like. For example, if the display unit 18 is configured with a touch panel display, the touch panel display can be used as the input unit 51.

[0044] The three-dimensional human body model storage unit 52 of the client terminal 15 stores the three-dimensional human body model MA transmitted from the display control unit 41 of the display control device 13. It is preferable that the three-dimensional human body model MA be stored in the three-dimensional human body model storage unit 52 before analysis is performed, that is, before the analysis data is transmitted from the display control unit 41. For example, if the client terminal 15 operates on a browser as described above, the display control unit 41 may transmit the three-dimensional human body model MA to the client terminal 15 in response to a login operation and store it therein, or if the client terminal 15 operates by executing a predetermined application software program as described above, the three-dimensional human body model MA may be stored when the software is installed.

[0045] The control unit 53 comprehensively controls each unit of the client terminal 15, such as the display unit 18, the input unit 51, and the three-dimensional human body model storage unit 52. The control unit 53 may generate an image to be displayed on the display unit 18 based on the analysis data transmitted from the display control unit 41 of the display control device 13 and the three-dimensional human body model MA stored in the three-dimensional human body model storage unit 52, and display the image on the display unit 18, and this is also done in this example.

[0046] As shown in FIG. 4 , the subject's body size SO is the size of a predetermined region (area) in the vertical direction when the subject is in an upright posture (standing posture). The subject's body size SO when taken as the entire vertical region is height SO1. In this case, the body size SM of the three-dimensional human body model MA is also height SM1, which is the entire vertical region. By calculating the correction factor C using the body sizes SO and SM of the predetermined vertical regions, the corrected foot pressure center position calculated by the correction unit 37 is easily determined as a suitable position in the three-dimensional human body model MA. Furthermore, the corrected three-dimensional joint positions calculated by the correction unit 37 are suitable positions in the three-dimensional human body model MA, and as a result, the center of gravity position and ground contact position are determined as suitable positions in the three-dimensional human body model MA by the position calculation unit 38.

[0047] In order to generate an image showing the three-dimensional human body model MA with better corrected foot pressure center position and ground contact position, and corrected three-dimensional joint positions and center of gravity positions, the body size SO is preferably the size of a predetermined region set in the lower body, and even more preferably the length SO2 from the sole of the foot to the pelvis in the lower body, and particularly preferably the leg length, i.e., the length SO3 from the sole of the foot to the hip joint. In this example, the body size SO is the length SO3, and the length SO3 stored in the subject data storage unit 31 is 0.80 m. When the subject's body size SO is set in the lower body, the body size SM of the three-dimensional human body model MA is also set in the lower body. When the length SO2 is set, the length SM2 from the sole of the foot to the pelvis is also set, and when the length SO3 is set, the leg length is also set, SM3. In this example, the length SM3 stored in the three-dimensional human body model storage unit 32 is 0.88 m. The pelvis and hip joints may be set as three-dimensional joint positions for the subject, and as positions for the three-dimensional human body model MA in the skeletal structure described above.

[0048] The operation of the above configuration will be described with reference to FIG. 5 . When a client logs in to the client terminal 15, the client terminal 15 determines whether the three-dimensional human body model MA is already stored in the client terminal 15 (S1). If the three-dimensional human body model MA is not stored, the client terminal 15 sends a transmission request for the three-dimensional human body model MA to the display control device 13. When the display control device 13 receives the transmission request from the client terminal 15, it transmits the three-dimensional human body model MA to the client terminal 15 in response to the transmission request (S2). When the three-dimensional human body model MA is input, the client terminal 15 stores the three-dimensional human body model MA in the three-dimensional human body model storage unit 52 under the control of the control unit 53. When the client terminal 15 determines that the three-dimensional human body model MA is already stored, it sends a start request to the display control device 13 to request the start of analysis. However, as described above, the processing flow up to the storage of the three-dimensional human body model MA in the client terminal 15 is not limited to this example.

[0049] When the display control unit 41 of the display control device 13 receives the start request, it determines whether the subject's body size SO is stored in the subject data storage unit 31, in this example, whether the length (leg length) SO3 is stored (S3). If it determines that the length (leg length) SO3 is not stored, it sends a body size transmission request to the client terminal 15, requesting transmission of the length (leg length) SO3. When the length (leg length) SO3 is transmitted from the client terminal 15, the display control unit 41 stores the input length (leg length) SO3 in the subject data storage unit 31 as the body size SO (S4) and causes the correction factor calculation unit 33 to calculate the correction factor C (S5). If the display control unit 41 determines that the length (leg length) SO3 is stored, it causes the correction factor calculation unit 33 to calculate the correction factor C (S5). In this example, the subject's length (leg length) SO3 is stored as 0.80 m in the subject data storage unit 31, the length (leg length) SM3 of the three-dimensional human body model MA is stored as 0.88 m in the three-dimensional human body model storage unit 32, and the correction coefficient C is calculated by the correction calculation unit 33 as 0.880 / 0.80 = 1.1.

[0050] When the correction factor C is calculated, for example, the correction factor calculation unit 33 sends a calculation notification indicating that the correction factor has been calculated to the acquisition unit 36. For example, in response to the input of the calculation notification from the correction factor calculation unit 33, the acquisition unit 36 ​​acquires time-series data on the three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center position from the detection unit 11 (S6). In response to the acquisition of the time-series data on the three-dimensional joint positions and foot pressure center position, the acquisition unit 36 ​​outputs these time-series data to the correction unit 37, and in response to the acquisition of the three-dimensional joint angles and floor reaction forces, outputs these to the display control unit 41. Step S6 may be performed between steps S1 and S3, or between steps S3 and S5.

[0051] In response to the input of the time-series data of the three-dimensional joint positions and the foot pressure center position, the correction unit 37 reads out the correction factor C from the correction factor storage unit provided in the correction factor calculation unit 33, and uses this correction factor C to calculate the corrected three-dimensional joint positions from the three-dimensional joint positions (S7a), and also calculates the corrected foot pressure center position from the foot pressure center position (S7b). For example, ｊ１ , y ｊ１, z ｊ１ If the coordinates (x, y, and z) of the center of foot pressure are (0.5, 0.3, 1.2), the correction coefficient C is multiplied by each of the x, y, and z coordinates, and the coordinates (0.55, 0.33, 1.32) are obtained as the corrected three-dimensional joint position. ｃ１ , y ｃ１ , z ｃ１ ) is (1.2, 0.2, 0.0), the x-coordinate, y-coordinate, and z-coordinate are each multiplied by the correction factor C, and the coordinates of (1.32, 0.22, 0.0) are obtained as the corrected foot pressure center position. In the correction step S7 performed by the correction unit 37, either the first step S7a of obtaining the corrected three-dimensional joint positions or the second step S7b of obtaining the corrected foot pressure center position may precede the correction step S7. The correction unit 37 outputs the time series data of the obtained corrected three-dimensional joint positions and the time series data of the obtained corrected foot pressure center position to the position calculation unit 38.

[0052] In response to the input of the time-series data of the corrected three-dimensional joint positions, the position calculation unit 38 generates time-series data of the center of gravity positions (S8a), and in this example, further calculates the trajectory of the center of gravity positions based on this time-series data. In response to the input of the time-series data of the corrected three-dimensional joint positions, the position calculation unit 38 identifies the timing of the ground contact positions and calculates the ground contact positions at each of these timings as coordinates (S8b). The position calculation step S8 performed by the position calculation unit 38 may be preceded by either the first step S8a of calculating the center of gravity positions or the second step S8b of calculating the ground contact positions. The position calculation unit 38 outputs the calculated time-series data of the center of gravity positions, the trajectory of the center of gravity positions, and the ground contact positions to the display control unit 41, along with the corrected three-dimensional joint positions and the corrected foot pressure center positions.

[0053] The correction factor C is calculated using the subject's body size SO in the vertical direction and the body size SM of the three-dimensional human body model MA, and the corrected foot pressure center position is calculated based on this correction factor C. Therefore, the corrected foot pressure center position obtained is determined as a good position in the three-dimensional human body model MA. Furthermore, the corrected three-dimensional joint positions calculated by the correction unit 37 are good positions in the three-dimensional human body model MA, and as a result, the center of gravity position and ground contact position are calculated as good positions in the three-dimensional human body model MA by the position calculation unit 38. As a result, analysis data including a precise video in which the floor surface FS and the soles of the feet are aligned is obtained.

[0054] The following explains the differences in effectiveness between using the correction unit 37 with the SO1 reference correction based on height, the SO2 reference correction based on pelvic height, and the SO3 reference correction based on leg length, and when no correction is performed. Assume that the subject's body size SO is SO1 = 1.60 m, SO2 = 0.84 m, and SO3 = 0.80 m, respectively, and the body size SM of the three-dimensional human body model MA is SM1 = 1.80 m, SM2 = 0.97 m, and SM3 = 0.88 m, respectively. If the correction factors for each reference are C1, C2, and C3, then C1 = 1.125, C2 ≒ 1.155, and C3 = 1.1. As such, when the ratios of height, pelvic height, and leg length differ between the subject and the three-dimensional human body model MA, the correction factor C may differ depending on the body size SO used. The position and rotation of the pelvis and the angles of the lower limb joints are input into the three-dimensional human body model MA, and the position of the feet is determined. Because the angle of the lower limb joints is larger than the movement of the pelvis, leg length has a significant effect. Therefore, in this example, a correction factor C3 is used, which more closely reflects leg length. Without correction, the subject's pelvis height is input into the three-dimensional human body model MA, and the pelvis of the three-dimensional human body model MA is positioned 0.84 m above the floor. However, because the length from the pelvis to the soles of the feet of the three-dimensional human body model MA is 0.97 m, the three-dimensional human body model MA is sunk 0.13 m below the floor. Furthermore, the center of gravity and the ground contact position are shifted by the amount of correction factor C, resulting in a mismatch between the three-dimensional human body model MA and the display position.

[0055] The display control unit 41 receives the three-dimensional joint angles and floor reaction forces from the acquisition unit 36, and receives the time-series data of the center of gravity position, the trajectory of the center of gravity position, the ground contact position, the corrected three-dimensional joint position, and the corrected foot pressure center position from the display control unit 41. In response to these inputs, the display control unit 41 generates a display of a ground contact marker and motion data (S9a), and also generates a numerical display and vector display of the floor reaction force and a display showing the center of gravity position and its trajectory D3 (see FIG. 2) as analysis data (S9b). The display control step S9 performed by the display control unit 41 may be preceded by either the first step S9a of generating motion data or the second step S9b of generating analysis data.

[0056] The display control unit 41 sends the generated data to the client terminal 15 (S10), and the client terminal 15 merges the data with the pre-stored three-dimensional human body model MA and displays an image G including a moving image portion Ga on the display unit 18 (S11). Because the analysis data sent from the display control unit 41 is precise, the client terminal 15 displays the analysis results, including a moving image in which the floor surface FS and the soles of the feet are aligned. In other words, the subject's body movements are visualized in detail on the client terminal 15. Because the client terminal 15 stores the three-dimensional human body model MA before the analysis data is input, the analysis results regarding the body movements are displayed more quickly on the client terminal 15, and as a result, the client can more quickly understand the analysis results.

[0057] Second Embodiment In the first embodiment described above, the correction unit 37 is connected to the acquisition unit 36, and the position calculation unit 38 is connected to this correction unit 37, but the connection between the acquisition unit 36, correction unit 37, and position calculation unit 38 is not limited to this. For example, in a display control device 61 of a second embodiment shown in Fig. 6, the position calculation unit 38 is connected to the acquisition unit 36, and the correction unit 37 is connected to this position calculation unit 38. The acquisition unit 36 ​​in this example sends time-series data on the three-dimensional joint positions and the foot pressure center position to the position calculation unit 38. In this second embodiment, the same effects as in the first embodiment described above can be obtained.

[0058] The position calculation unit 38 in this example calculates the center of gravity position based on the three-dimensional joint positions and the ground contact position based on the foot pressure center position. The position calculation unit 38 sends the time-series data of the calculated center of gravity position and the ground contact positions calculated in chronological order to the correction unit 37, together with the time-series data of the three-dimensional joint positions and the foot pressure center position input from the acquisition unit 36.

[0059] The correction unit 37 corrects each of the three-dimensional joint positions, the foot pressure center position, the center of gravity position, and the ground contact position that constitute the time-series data based on the correction factor C. The correction unit 37 outputs these to the display control unit 41, similar to the position calculation unit 38 in the first embodiment.

[0060] The operation of the above configuration will be described with reference to FIG. 7 , but explanations of steps that are the same as those shown in FIG. 5 will be omitted. In response to acquisition of time-series data on three-dimensional joint positions and foot pressure center positions, the acquisition unit 36 ​​outputs the time-series data to the position calculation unit 38. In response to input of the time-series data on three-dimensional joint positions, the position calculation unit 38 generates time-series data on the center of gravity position (S21a). In this example, the position calculation unit 38 further calculates the trajectory of the center of gravity position based on this time-series data. In response to input of the time-series data on three-dimensional joint positions, the position calculation unit 38 identifies the timing of the contact point and calculates the contact point at each of these timings as coordinates (S21b). The position calculation step S21 performed by the position calculation unit 38 may be preceded by either the first step S21a for calculating the center of gravity position or the second step S21b for calculating the contact point. However, the calculated center of gravity position and contact point are coordinates in the subject's body size SO (in this example, length (leg length) SO3). The position calculation unit 38 outputs the obtained time series data of the center of gravity position, the trajectory of the center of gravity position, and the ground contact position to the correction unit 37 together with the time series data of the three-dimensional joint positions and the foot pressure center position.

[0061] The correction unit 37 corrects the time-series data of the three-dimensional joint positions and foot pressure center positions, the time-series data of the center of gravity position and the trajectory of the center of gravity position, and the ground contact position using the correction factor C in response to the input. As a result, corrected three-dimensional joint positions are calculated from the three-dimensional joint positions (S22a), and corrected foot pressure center positions are calculated from the foot pressure center positions (S22b). Furthermore, as calculated by the position calculation unit 38 in the first embodiment, the time-series data of the center of gravity position, the trajectory of the center of gravity position, and the ground contact position are calculated (S22c, S22d) corrected to correspond to the body size SM of the three-dimensional human body model MA. In this way, the time-series data of the corrected three-dimensional joint positions and the corrected foot pressure center position are generated, and the corrected time-series data of the center of gravity position, the trajectory of the center of gravity position, and the ground contact position are also obtained. In the correction step S22 performed by the correction unit 37, the first step S22a of determining the corrected three-dimensional joint positions, the second step S22b of determining the corrected foot pressure center position, the third step S22c of correcting the center of gravity position, and the fourth step S22d of correcting the ground contact position may be performed in any order. The correction unit 37 outputs each piece of data obtained in this manner to the display control unit 41, and the display control step S9, the transmission step S10, and the display step S11 are performed in this order, as in the first embodiment.

[0062] As described above, the position calculation unit 38 in this example determines the center of gravity position and the ground contact position based on the three-dimensional joint positions before or after correction by the correction unit 37. Because the center of gravity position and the ground contact position have been determined based on the three-dimensional joint positions before correction, the correction unit 37 corrects the center of gravity position and the ground contact position based on the correction factor C and sends the corrected center of gravity position and the ground contact position to the display control unit 41.

[0063] REFERENCE SIGNS LIST 10 Display control system 11 Detection unit 13, 61 Display control device 15 Client terminal 18 Display unit 31 Subject data storage unit 32 Three-dimensional human body model storage unit 33 Correction factor calculation unit 36 ​​Acquisition unit 37 Correction unit 38 Position calculation unit 41 Display control unit 51 Input unit 52 Three-dimensional human body model storage unit 53 Control unit

Claims

1. A subject data storage unit that stores the body size of a specified area in the vertical direction in a standing posture for a subject whose movement is to be analyzed; a three-dimensional human body model storage unit that stores a three-dimensional human body model displayed in a video as the subject moving, and the body size set for the three-dimensional human body model; a correction factor calculation unit that calculates a correction factor for correcting the body size of the subject to the body size set for the three-dimensional human body model; an acquisition unit that acquires time series data for three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center positions for the subject during movement; a position calculation unit that calculates time series data for the subject's center of gravity position based on the time series data of the three-dimensional joint positions, and calculates the contact positions on the floor surface where the feet are touching the ground in chronological order; and a correction unit that corrects the three-dimensional joint positions and the foot pressure center positions based on the correction factors. a display control unit that generates motion data indicating the movement of the three-dimensional human body model based on time series data of the three-dimensional joint angle, the floor reaction force, the corrected three-dimensional joint position, the corrected center of foot pressure position, the center of gravity position, and the ground contact position, and generates analysis data of at least one of the floor reaction force, the center of gravity position, and a ground contact marker indicating the ground contact position, which are shown together with the three-dimensional human body model in the video, wherein the position calculation unit calculates the center of gravity position and the ground contact position based on the three-dimensional joint position before or after correction by the correction unit, and when the center of gravity position is calculated based on the three-dimensional joint position before correction, the correction unit corrects the center of gravity position based on the correction rate and sends the corrected center of gravity position to the display control unit, and when the ground contact position is calculated based on the three-dimensional joint position before correction, corrects the ground contact position based on the correction rate and sends the corrected ground contact position to the display control unit.

2. The display control device according to claim 1, wherein the correction factor is calculated as SM / SO, where SO is the body size of the subject and SM is the body size of the three-dimensional human body model.

3. A display control device according to claim 1 or 2, wherein the body size is the size of the specified area set in the lower body.

4. The display control device according to claim 3, wherein the body size is leg length.

5. A display control device according to claim 1 or 2, wherein the display control unit transmits the three-dimensional human body model to a display terminal that displays the video before transmitting the motion data and the analysis data.

6. A client terminal for displaying a moving subject in a video using a three-dimensional human body model displayed in the video as a subject whose movement is to be analyzed, and a display control device for sending motion data indicating the movement of the three-dimensional human body model to the client terminal, the display control device comprising: a subject data storage unit for storing a body size of a predetermined area in the vertical direction in an upright posture for the subject; a three-dimensional human body model storage unit for storing the three-dimensional human body model and the body size set for the three-dimensional human body model; a correction factor calculation unit for calculating a correction factor for correcting the body size of the subject to the body size set for the three-dimensional human body model; an acquisition unit for acquiring time series data of three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center positions for the subject during movement; and a position calculation unit for calculating time series data of the center of gravity position of the subject based on the time series data of the three-dimensional joint positions, and for chronologically calculating ground contact positions on the floor surface on which the feet are placed. a display control unit that generates motion data indicating a movement of the three-dimensional human body model based on time-series data of the three-dimensional joint angle, the floor reaction force, the corrected three-dimensional joint position, the corrected foot pressure center position, the center of gravity position, and the ground contact position, and generates analysis data of at least one of the floor reaction force, the center of gravity position, and a ground contact marker indicating the ground contact position, which are shown together with the three-dimensional human body model in the video, and the ground contact position; wherein the position calculation unit determines the center of gravity position and the ground contact position based on the three-dimensional joint position before or after being corrected by the correction unit, and when the center of gravity position is determined based on the three-dimensional joint position before being corrected, the correction unit corrects the center of gravity position based on the correction rate and sends the corrected center of gravity position to the display control unit, and when the center of gravity position is determined based on the three-dimensional joint position before being corrected, the correction unit corrects the center of gravity position based on the correction rate and sends the corrected center of gravity position to the display control unit.

7. A subject data storage step for storing the body size of a predetermined area in the vertical direction in a standing posture for a subject whose movement is to be analyzed; a three-dimensional human body model storage step for storing a three-dimensional human body model displayed in a video as the subject who has moved and the body size set for the three-dimensional human body model; a correction factor calculation step for calculating a correction factor for correcting the body size of the subject to the body size set for the three-dimensional human body model; an acquisition step for acquiring time series data for three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center positions for the subject during movement; a position calculation step for calculating time series data for the subject's center of gravity position based on the time series data for the three-dimensional joint positions, and for calculating the contact positions on the floor surface where the feet are in contact in chronological order; and a correction step for correcting the three-dimensional joint positions and the foot pressure center positions based on the correction factors. a display control step of generating motion data indicating the movement of the three-dimensional human body model based on time-series data of the three-dimensional joint angle, the floor reaction force, the corrected three-dimensional joint position, the corrected center of foot pressure position, the center of gravity position, and the ground contact position, and generating at least one of analysis data of the floor reaction force, the center of gravity position, and a ground contact marker indicating the ground contact position, which are shown together with the three-dimensional human body model in the video, wherein the position calculation step calculates the center of gravity position and the ground contact position based on the three-dimensional joint position before or after correction by the correction step, and the correction step, when the center of gravity position is calculated based on the three-dimensional joint position before correction, corrects the center of gravity position based on the correction rate and sends the corrected center of gravity position to the display control step, and when the ground contact position is calculated based on the three-dimensional joint position before correction, corrects the ground contact position based on the correction rate and sends the corrected ground contact position to the display control step.

8. A subject data storage step for storing the body size of a predetermined area in the vertical direction in a standing posture for a subject whose movement is to be analyzed; a three-dimensional human body model storage step for storing a three-dimensional human body model displayed in a video as the subject who has moved and the body size set for the three-dimensional human body model; a correction factor calculation step for calculating a correction factor for correcting the body size of the subject to the body size set for the three-dimensional human body model; an acquisition step for acquiring time series data of three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, and foot pressure center positions for the subject during movement; a position calculation step for calculating time series data of the subject's center of gravity position based on the time series data of the three-dimensional joint positions, and for calculating the contact positions on the floor surface where the feet are in contact in chronological order; and a correction step for correcting the three-dimensional joint positions and the foot pressure center positions based on the correction factors. a display control step of generating motion data indicating the movement of the three-dimensional human body model based on time series data of the three-dimensional joint angle, the floor reaction force, the corrected three-dimensional joint position, the corrected center of foot pressure position, the center of gravity position, and the ground contact position, and generating analysis data of at least one of the floor reaction force, the center of gravity position, and a ground contact marker indicating the ground contact position, which are shown together with the three-dimensional human body model in the video, wherein the position calculation step calculates the center of gravity position and the ground contact position based on the three-dimensional joint position before or after being corrected by the correction step, and the correction step, when the center of gravity position is calculated based on the three-dimensional joint position before being corrected, corrects the center of gravity position based on the correction rate and sends the corrected center of gravity position to the display control step, and when the ground contact position is calculated based on the three-dimensional joint position before being corrected, corrects the ground contact position based on the correction rate and sends the corrected ground contact position to the display control step.

Images