Motion visualization system and motion visualization method

Abstract

To provide a motion visualization system that can compare multiple measurement data.SOLUTION: A motion visualization system 1 includes a skeletal recognition program 2 for acquiring the joint coordinates of a subject, a pitch extraction program 6 for extracting the walking cycle of the subject on the basis of the joint coordinates, and in one extracted walking cycle, a storage unit 106 for converting the coordinates of the joint coordinates into the reference coordinate system that the direction in which a predetermined part of the subject moves is used as an axis of the traveling direction and for storing a skeletal normalization program 3 for converting the joint coordinates on the basis of the information of the subject.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a motion visualization system and a motion visualization method, and more particularly to a motion visualization system and a motion visualization method for visualizing the walking dynamics of a subject. [Background technology]

[0002] The aging of the population is progressing. In an aging society, the increase in locomotive syndrome caused by musculoskeletal diseases has become a social problem. In order to extend healthy life expectancy even in an aging society, it is desirable to detect early decline in the motor function of subjects (elderly and middle-aged people) and provide appropriate exercise instruction at fitness clubs, rehabilitation facilities, etc. to improve the motor function of the subjects.

[0003] A motion information processing device that displays a movement trajectory of a subject is described, for example, in Patent Document 1. Patent Document 1 shows that the footprints of the subject and the movement trajectory of a predetermined part are displayed based on the coordinates of each part of the subject extracted from an image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-42241 A Summary of the Invention [Problem to be solved by the invention]

[0005] According to Patent Document 1, it is possible to display the movement trajectory of, for example, a specific part of a subject. However, if the environment in which the subject is measured (observed), for example, the direction in which the subject's image is captured, is different, the measurement data will be different, and it is difficult to obtain a meaningful result, for example, even if the movement trajectories are compared.

[0006] For example, even if the subject is the same, if the measurement environments are different, it is difficult to grasp the improvement over time or due to the instruction intervention even if the movement trajectories are compared before and after the passage of time or before and after the instruction intervention such as exercise instruction. Also, even if the movement trajectories are compared between multiple different subjects observed in different environments, it is difficult to grasp the state of the movement trajectory of a specific subject based on multiple subjects.

[0007] An object of the present invention is to provide an exercise visualization system and an exercise visualization method that are capable of comparing multiple pieces of measurement data.

[0008] Other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0009] A brief outline of a representative embodiment of the present invention will be described below.

[0010] That is, the motion visualization system includes a skeletal recognition unit that acquires skeletal coordinate information of the subject, a period extraction unit that extracts the walking period of the subject based on the skeletal coordinate information, and a conversion unit that converts the value of the skeletal coordinate information into a reference coordinate system whose axis of progression is the direction in which a specific part of the subject moves during one extracted walking period, and converts the skeletal coordinate information based on the subject's information. Effect of the Invention

[0011] To briefly explain the effect obtained by a representative embodiment of the invention disclosed in this application, it is possible to provide an exercise visualization system and an exercise visualization method that are capable of comparing multiple pieces of measurement data. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an exercise visualization system according to an embodiment. FIG. [Diagram 2]FIG. 4 is a flowchart illustrating the operation of the exercise visualization system according to the embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a relationship between a depth camera and a subject according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating a relationship between a depth camera and a subject according to an embodiment. [Diagram 5] FIG. 13 is a diagram for explaining skeleton recognition according to an embodiment. [Figure 6] FIG. 13 is a diagram for explaining extraction of one walking cycle according to an embodiment. [Figure 7] 13A and 13B are diagrams illustrating normalization and coordinate transformation according to an embodiment. [Figure 8] 13A and 13B are diagrams illustrating normalization and coordinate transformation according to an embodiment. [Figure 9] 13A and 13B are diagrams illustrating coordinate transformation according to an embodiment. [Figure 10] 1A to 1C are diagrams for explaining gait features acquired by the exercise visualization system according to the embodiment. [Figure 11] 1A to 1C are diagrams for explaining gait features acquired by the exercise visualization system according to the embodiment. [Figure 12] 1A to 1C are diagrams for explaining gait features acquired by the exercise visualization system according to the embodiment. [Figure 13] 13(A) and 13(B) are diagrams illustrating a method for displaying walking characteristics according to an embodiment. [Figure 14] 13(A) and 13(B) are diagrams illustrating a method for displaying walking characteristics according to an embodiment. [Figure 15] FIG. 2 is a diagram showing a display screen of the exercise visualization system according to the embodiment. [Figure 16] FIG. 11 is a diagram showing a relationship between a depth camera and a subject according to the first modification of the embodiment. [Figure 17] FIG. 11 is a block diagram showing a configuration of an exercise visualization system according to a second modification of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention. (Embodiment) <Configuration of the motion visualization system>

[0014] FIG. 1 is a block diagram showing a configuration of a motion visualization system according to an embodiment. In FIG. 1, 1 indicates the motion visualization system. The motion visualization system 1 includes, but is not limited to, a computer 100, a depth camera 200, an Internet line 301, and a server 300 connected to the computer 100 via the Internet line 301. Here, an example in which the motion visualization system 1 includes the Internet line 301 and the server 300 will be described, but the motion visualization system 1 is not limited to this. For example, the motion visualization system 1 may be configured by the computer 100 and the depth camera 200, excluding the Internet line 301 and the server 300.

[0015] The computer 100 includes a calculation unit 101, a control unit 102, an operation input unit 103, a display unit 104, a memory 105, a storage unit 106, an external input / output unit 107, and a network communication unit 108, all of which are connected to a bus. The control unit 102 reads out a program or the like previously stored in the storage unit 106, for example, into the memory 105, and executes the program read out into the memory 105. Of course, the control unit 102 may execute a program previously stored in the storage unit 106 without reading it out into the memory 105. In this case, the memory 105 is used, for example, as a working memory when executing the program.

[0016] The calculation unit 101 is used to perform calculations when the control unit 102 executes a program. The external input / output unit 107 is also used when the control unit 102 executes a program. In the embodiment, the depth camera 200 is connected to the external input / output unit 107. By executing the program, the control unit 102 controls the depth camera 200 using the external input / output unit 107, and acquires image data captured by the depth camera 200 via the external input / output unit 107.

[0017] The operation input unit 103 includes, for example, a keyboard and a mouse, and inputs are made to the computer 100 using the keyboard and the mouse. The display unit 104 includes, for example, a monitor such as a liquid crystal display for displaying images, and displays images and the like generated by the computer 100. The network communication unit 108 is connected between the bus Bus and the Internet line 301, and performs communication between the computer 100 and the server 300. For example, data generated by the computer 100 is transmitted to the server 300 via the network communication unit 108 and the Internet line 301, and is stored in the server 300. In addition, the data stored in the server 300 is supplied to the computer 100 via the network communication unit 108 and the Internet line 301, and is used when executing a program.

[0018] The storage unit 106 is configured, for example, by a hard disk or an SSD (Solid State Drive). A plurality of programs are stored in the storage unit 106. Data generated by the execution of the programs and / or data used when the programs are executed are also stored in the storage unit 106. As described above, a plurality of programs and data are stored in the storage unit 106, but only the programs and data necessary for explaining the embodiment are shown in FIG. 1.

[0019] In Fig. 1, 2 indicates a skeleton recognition program, 3 indicates a skeleton normalization program, 4 indicates a screen display program, 5 indicates a gait feature calculation program, and 6 indicates a pitch (gait cycle) extraction program. Also in Fig. 1, 7 indicates skeleton and gait feature data stored in the memory unit 106.

[0020] The control unit 102 executes the above-mentioned programs 2 to 6, whereby functional units (functional sections) constituting the motion visualization system 1 are realized in the computer 100. That is, the skeleton recognition section is constituted in the computer 100 by executing the skeleton recognition program 2, the conversion section is constituted in the computer 100 by executing the skeleton normalization program 3, the period extraction section is constituted in the computer 100 by executing the pitch extraction program 6, and the gait feature calculation program 5 is constituted in the computer 100.

[0021] By executing the screen display program 4, for example, an image generated by the gait feature calculation program 5 is displayed on the display unit 104. In addition, for example, data generated by executing the skeleton recognition program 2, the skeleton normalization program 3, the gait feature calculation program 5, and the pitch extraction program 6 is stored in the storage unit 106 as skeleton / gait feature data 7.

[0022] The skeleton / walking characteristic data 7 stored in the storage unit 106 is supplied to the server 300, for example, via the network communication unit 108 and the Internet line 301. Alternatively, the data stored in the server 300 is supplied to the storage unit 106 via the Internet line 301 and the network communication unit 108, and stored as the skeleton / walking characteristic data 7.

[0023] The exercise visualization system 1 of the embodiment captures the state of a person (subject) to be measured while walking using a depth camera 200, and based on the images obtained by capturing the images, observes changes in the subject's skeleton that accompany walking, and displays the walking characteristics of the subject, for example, on a display unit 104. <Operation of the motion visualization system>

[0024] 2 is a flowchart for explaining the operation of the exercise visualization system according to the embodiment. The operation of the exercise visualization system 1 shown in FIG. 1 will be explained below with reference to FIG.

[0025] In step S0, the operation of the motion visualization system 1 is started. By starting the operation, the control unit 102 starts executing the skeleton recognition program stored in the storage unit 106. This causes image (depth) capture in step S1 and skeleton recognition of each frame in step S2 to be executed. Note that, hereinafter, steps S1 and S2 may be collectively referred to as a skeleton recognition process.

[0026] In the image capture of step S1, images of a walking subject are continuously captured using the depth camera 200. The state of capturing the subject by the depth camera 200 will be described with reference to the drawings. FIG. 3 and FIG. 4 are diagrams showing the relationship between the depth camera and the subject according to the embodiment. Here, FIG. 3 is a diagram showing the depth camera 200 and the subject from the side, and FIG. 4 is a diagram showing the depth camera 200 and the subject from above. In FIG. 3 and FIG. 4, 500 indicates the subject, and the subject 500 walks on the floor surface 400 of the room in the direction of travel indicated by the arrow. The depth camera 200 is installed on the floor surface 400 by a fixing member 201 such as a tripod, although not particularly limited. 3 and 4, 200R indicates the shooting range of the depth camera 200, the Z axis indicates the depth direction of the depth camera 200 (the direction along the optical axis of the lens of the depth camera 200), the Y axis indicates the up-down direction (Y axis perpendicular to the Z axis) along the lens surface of the depth camera 200, and the X axis indicates the left-right direction (X axis perpendicular to the Z axis and Y axis) along the lens surface. Although not particularly limited, the center of the lens surface of the depth camera 200 is set as the origin (0, 0, 0) of the X axis, Y axis, and Z axis. Of course, the origin of the X axis, Y axis, and Z axis is not limited to this. For example, the origin (0) of the Y axis may be the floor surface 400. In addition, 401 indicates the ceiling surface of the room.

[0027] For example, if the subject is instructed to walk toward the lens surface of the depth camera 200, the subject may become nervous and walk in a different manner than usual. Therefore, as shown in Fig. 4, the subject 500 is instructed to walk in a direction that is different from the lens surface of the depth camera 200. As a result, as shown in Fig. 4, the moving direction of the subject 500 and the Z axis, which is the depth direction of the depth camera 200, do not coincide with each other.

[0028] A target person walking in the direction of travel is continuously photographed by the depth camera 200. A plurality of frames (image data) obtained by the continuous photographing are temporarily stored in, for example, the storage unit 106 via the external input / output unit 107.

[0029] In step S2, the control unit 102 executes a skeleton recognition process for recognizing the skeleton of the subject for each of the multiple frames temporarily stored in the storage unit 106, for example, using the calculation unit 101. FIG. 5 is a diagram for explaining skeleton recognition according to an embodiment. By executing step S2, the joint coordinates of the subject 500 are acquired as shown in FIG. 5. In FIG. 5, the acquired joint coordinates (skeletal coordinates) are shown as J1 to J21 as an example. For example, J3 to J5 indicate shoulder joint coordinates, and J13 to J15 indicate pelvis joint coordinates. The values ​​of the joint coordinates J1 to J21 at this time are values ​​of a coordinate system related to the depth camera 200. That is, the values ​​(x, y, z) of the joint coordinates J1 to J21 are values ​​on the X-axis, Y-axis, and Z-axis. By executing step S2, the joint coordinates as shown in FIG. 5 are acquired for each frame. The acquired joint coordinates are stored in the storage unit 106 as, for example, skeleton / walking feature data 7.

[0030] Following step S2, one walking cycle is extracted (cycle extraction step) in step S3. That is, the control unit 102 executes the pitch extraction program 6 stored in the storage unit 106. In the pitch extraction program 6, one walking cycle of the subject is extracted using the joint coordinates acquired for each frame. FIG. 6 is a diagram for explaining the extraction of one walking cycle according to the embodiment. In the pitch extraction program 6, the walking states 500_1 to 500_6 of the subject for each frame are determined based on the acquired joint coordinates. In the embodiment, although not particularly limited, a walking state 500_1 in which the right foot is forward and both feet are open is set as the start point and end point of walking, and the period between the start point and the end point is extracted as one walking cycle (one pitch). Of course, the start point and the end point are not limited to this, and any walking state may be set as the start point and the end point. In the following description, the walking state at the start point is indicated by the symbol 500_1S, and the walking state at the end point is indicated by the symbol 500_1E.

[0031] In the next step S4, the control unit 102 obtains a straight line connecting the walking state 500_1S of the start point and the walking state 500_1E of the end point, and extracts the obtained straight line as an axis of the proceeding direction (advancement axis). For example, the control unit 102 extracts a straight line connecting the pelvis joint coordinate J13 in the walking state 500_1S and the pelvis joint coordinate J13 in the walking state 500_1E as the proceeding axis. Of course, it is not limited to the pelvis joint coordinate. Although not particularly limited, the extraction of the proceeding axis in step S4 is performed as a part of the pitch extraction program 6. Of course, the extraction of one walking cycle and the extraction of the proceeding axis may be executed by separate programs.

[0032] Following step S4, in the embodiment, step S5 is performed to normalize the lengths of the XYZ axes based on the unit length (torso length), step S6 is performed to perform coordinate conversion so that the Z axis overlaps with the moving axis, and step S7 is performed to perform coordinate conversion so that the ground contact surfaces of both feet are parallel to the X axis. These steps S5 to S7 are realized by the control unit 102 executing the skeleton normalization program 3 stored in the storage unit 106. The program for extracting the moving axis may be a part of step S6. In this case, step S6 can be considered as a coordinate conversion step. Next, steps S5 to S7 will be described in detail with reference to the drawings. <<Normalization and coordinate transformation>>

[0033] 7 and 8 are diagrams for explaining normalization and coordinate conversion according to the embodiment. Here, Fig. 7 is a diagram of the walking of the subject 500 viewed from the side (X-axis direction), and Fig. 8 is a diagram of the walking of the subject 500 viewed from above (Y-axis direction).

[0034] In step S5, the value of each joint coordinate of the subject 500 is converted based on information (reference data) of the subject 500. The height, stride length, or length of a predetermined part of the subject 500 can be used as the reference data. Here, a case where the length of a predetermined part of the subject 500 is used as the reference data will be described. Also, the torso length of the subject 500 is taken as an example of the predetermined part.

[0035] The torso length of the subject 500 is the length between the shoulder joint coordinates J3 to J5 and the pelvis joint coordinates J13 to J15. The length (torso length) between the shoulder joint coordinates and the pelvis joint coordinates in one frame may be used as the reference data, but in this embodiment, the average value of the torso length in one walking cycle is used as the reference data. In step S5, the values ​​(x, y, z) of each joint coordinate on the X-axis, Y-axis, and Z-axis are converted to values ​​when the reference data is set to a unit length (for example, 1). That is, by executing step S5, the value of each joint coordinate is normalized with the torso length of the subject 500 as the reference.

[0036] As described in FIG. 4, the Z-axis in the depth direction is different from the moving direction of the subject 500. That is, the direction of the Z-axis is different from the direction of the moving axis obtained in step S4. In step S6, the values ​​of the coordinates of each joint are transformed so that the moving axis extracted in step S4 overlaps with the Z-axis. That is, the values ​​of the coordinates of each joint are transformed into a reference coordinate system in which the moving direction represented by the moving axis is the Z-axis.

[0037] Next, the state before and after normalization and coordinate transformation will be described with reference to Fig. 7 and Fig. 8. Fig. 7(A) shows the state before normalization and coordinate transformation, and Fig. 7(B) shows the state after normalization and coordinate transformation.

[0038] In FIG. 7(A), in the walking states 500_1S, 500_4, and 500_1E, the joint coordinates are indicated by circles, as in FIG. 5. Also, as an example, the shoulder joint coordinates and the pelvis joint coordinates are indicated by symbols J3 to J5 and J13 to J15. Since this is before normalization, the unit of the Z axis and the Y axis is meters (m). Also, for the pelvis joint coordinates, the progress axis connecting the start point and the end point is indicated by symbol 501. The progress direction is indicated by the arrow of the progress axis 501. In FIG. 7(A), the progress axis 501 is oriented in the direction in which the values ​​of the Y axis and the Z axis become smaller. That is, the subject 500 is walking in the direction in which the values ​​of the Y axis and the Z axis become smaller.

[0039] Since normalization is performed in step S5, the joint coordinates (e.g., J3-J5, J13-J15) in the walking states 500_1 and 500_4 are converted into joint coordinates (J3C-J5C, J13C-J15C) in which the torso length is used as the reference data (unit length) as shown in FIG. 7(B). Of course, in FIG. 7(B), the units of the Y-axis and Z-axis are torso length. Also, since coordinate conversion is performed in step S6, the progression axis 501 and the Z-axis overlap. In FIG. 7(B), 500_1SC, 500_4C, and 500_1EC show the walking states after conversion. Since the progression axis 501 overlaps with the Z-axis, the change in the walking state accompanying walking is along the Z-axis.

[0040] Fig. 8(A) shows the state before normalization and coordinate transformation are performed, and Fig. 8(B) shows the state after normalization and coordinate transformation are performed. As can be seen from Fig. 8(A), when viewed from above, the subject 500 is walking in a direction in which the Z-axis value is small and the X-axis value is large. By performing steps S5 and S6, the changes in the walking states 500_1SC, 500_4C, and 500_1EC accompanying walking are along the Z-axis, and the subject's joint coordinates are normalized in units of torso length. <<Coordinate transformation for tilt>>

[0041] 9A and 9B are diagrams for explaining coordinate conversion according to the embodiment, in which Fig. 9A shows the state before step S7 is performed, and Fig. 9B shows the state after step S7 is performed.

[0042] In the embodiment, as shown in FIG. 3, the depth camera 200 is installed on the floor surface 400 by the fixing member 201. Depending on how the depth camera 200 and / or the fixing member 201 are installed, it is considered that the depth camera 200 is inclined to the left or right with respect to the floor surface 400. In this case, in the frame acquired by the depth camera 200, the subject is inclined with respect to the X-axis as shown in FIG. 9(A). As a result, the X-coordinate of each joint coordinate recognized in step S2 is not aligned with the X-axis of the floor surface 400. Taking the pelvis joint coordinates J13 to J15 as an example, the X-coordinates of these joint coordinates are not aligned with the X-axis as shown in FIG. 9(A).

[0043] In step S7, the X coordinate of each joint of the subject is converted so as to be parallel to the floor surface 400 (X axis). As a result, as shown in FIG. 9(B), the pelvis joint coordinates J13 to J15 are converted to joint coordinates J13C to J15C having X coordinates parallel to the floor surface 400. Although the pelvis joint coordinates have been described as an example, the other joint coordinates are also converted in a similar manner in step S7. This is realized by converting the Y axis value of each joint coordinate so that the difference in the Y axis value between, for example, joint coordinates J14C and J15C becomes smaller.

[0044] In the embodiment, normalization and coordinate transformation are performed in the order of normalization (step S5) and coordinate transformation (steps S6 and S7) as shown in Fig. 2. However, this is not limited thereto, and for example, normalization in step S5 may be performed after the coordinate transformation in steps S6 and S7.

[0045] Returning to FIG. 2, the operation of the motion visualization system will be described.

[0046] After step S7, gait feature calculation (gait feature calculation step) is performed in step S8. That is, the control unit 102 shown in Fig. 1 executes the gait feature calculation program 5 stored in the storage unit 106. In this gait feature calculation program 5, calculation is performed using each joint coordinate acquired by executing steps S5 to S7, that is, each joint coordinate that has been normalized and coordinate transformed.

[0047] By using the gait feature calculation program 5, it is possible to acquire various features of the subject 500 associated with walking. Here, as examples of gait features, swaying from side to side while walking, vertical movement while walking, and rotation of a predetermined part will be described. <<Walking characteristics>>

[0048] Fig. 10 to Fig. 12 are diagrams for explaining gait features acquired by the motion visualization system according to the embodiment. Here, Fig. 10 is a diagram for explaining the left-right wobble of the subject 500, Fig. 11 is a diagram for explaining the rotation of a predetermined part of the subject 500, and Fig. 12 is a diagram for explaining the up-down movement of the subject. In Fig. 10 to Fig. 12, J3C to J5C, J8C, J11C, J13C to J15C, J18C, and J21C are joint coordinates acquired by applying the above-mentioned normalization and coordinate conversion to the joint coordinates J3 to J5, J8, J11, J13 to J15, J18, and J21 shown in Fig. 5. <<<Wobble>>>

[0049] In FIG. 10, 502 indicates a movement trajectory (unsteadiness trajectory) in the X direction of a predetermined part when the subject 500 walks one cycle. This movement trajectory 502 is calculated by executing the walking feature calculation program 5. Here, the pelvis joint coordinate J13C is adopted as the predetermined part. Therefore, the movement trajectory 502 represents the transition of movement when the subject 500 walks only one walking cycle. When the subject 500 walks one walking cycle, if the subject 500 does not wobble left and right, the movement trajectory 502 is parallel to the Z axis or overlaps with the Z axis. However, if the subject 500 wobbles left and right, as shown in FIG. 10, the movement trajectory 502 becomes a curve that changes left and right on either side of the Z axis. Of course, there are also cases where the movement trajectory 502 changes only left and right. This makes it possible to specify the presence or absence of wobbling of the subject 500 and the magnitude (value) of the wobbling.

[0050] Here, the movement of the pelvis joint coordinate J13C between the walking states 500_1SC and 500_1EC is used to calculate the movement trajectory 502, but this is not limited to this. For example, the movement of the shoulder joint coordinate J4C may be used to calculate the movement trajectory 502. <<<Rotation of specific body parts>>>

[0051] In FIG. 11, 503 is a rotation line. In FIG. 11, a straight line connecting the X coordinates of the pelvis joint coordinates J13C to J15C is the rotation line 503. The angle between this rotation line 503 and a virtual line VL perpendicular to the Z axis is calculated as the rotation angle of the pelvis. This rotation line 503 and the rotation angle of the pelvis are calculated by executing the walking feature calculation program 5. By calculating the rotation angle of the pelvis at each timing during the period in which the subject 500 walks one walking cycle, it is possible to grasp the transition of the rotation of the pelvis.

[0052] 11, the rotation angle of the pelvis is taken as an example, but the present invention is not limited to this. For example, the rotation angle of the shoulders may be calculated and its transition may be grasped.

[0053] <<<Up and down movement>>> In FIG. 12, 504 is a vertical movement trajectory. The vertical movement trajectory 504 indicates a movement trajectory in the Y direction of a predetermined part when the subject 500 walks for one walking cycle. This vertical movement trajectory 504 is calculated by executing the walking feature calculation program 5. Here, the pelvis joint coordinate J13C is adopted as the predetermined part. Therefore, the vertical movement trajectory 504 represents the transition of the vertical movement when the subject 500 walks for only one walking cycle. If the subject 500 does not move up and down when walking for one walking cycle, the vertical movement trajectory 504 has a constant value on the Y axis. However, if the subject 500 changes up and down while walking, as shown in FIG. 12, the vertical movement trajectory 504 becomes a curve that changes in the Y axis direction. This makes it possible to grasp the presence or absence of vertical movement of the subject 500 and the magnitude of the vertical movement. In FIG. 12, the pelvis joint coordinate J13C is used, but is not limited thereto. For example, the shoulder joint coordinate J4C may be used.

[0054] The various walking characteristics calculated in step S8 are stored in the storage unit 106 as skeleton / walking characteristics data 7.

[0055] In step S9 following step S8, the process up to the calculation of the gait features is completed. After this, the user operates, for example, the operation input unit 103 (FIG. 1) to instruct the motion visualization system 1 to display the gait features. In response to this instruction, the control unit 102 (FIG. 1) executes the screen display program 4 (FIG. 1).

[0056] Next, examples of gait features displayed on the display unit 104 (FIG. 1) by executing the screen display program 4 (display step) will be described with reference to the drawings. FIG. 13 and FIG. 14 are diagrams showing a method of displaying gait features according to an embodiment.

[0057] FIG. 13 shows a display method for displaying the subject's left-right sway based on the skeleton / walking feature data 7. In FIG. 13, 104_1 shows a part of the display area on the display screen of the display unit 104. The screen display program 4 displays the transition of the joint coordinates related to the left-right sway and the sway trajectory 502 on the display area 104_1 like an animation. That is, the joint coordinates and the sway trajectory 502 that change with time are displayed on the display area 104_1. The joint coordinates in the first walking state 500_1SC of one walking cycle are displayed as shown in FIG. 13(A). Thereafter, the joint coordinates and the sway trajectory 502 that change with time are sequentially displayed on the display area 104_1, and as shown in FIG. 13(B), the joint coordinates and the sway trajectory 502 in the last walking state 500_1EC of one walking cycle are displayed.

[0058] Although an example in which the change in joint coordinates is also displayed on the display area 104_1 has been described, the present invention is not limited to this. For example, only the joint coordinates of the first walking state 500_1SC and the last walking state 500_1EC and the sway trajectory 502 may be displayed on the display area 104_1. By performing a display as shown in FIG. 13, it is possible to present the left-right sway of the subject.

[0059] Fig. 14 is similar to Fig. 13. The difference is that in Fig. 14, the pelvis joint coordinates J14C, J15C and the rotation line are displayed together with the wobble trajectory 502. That is, the pelvis joint coordinates J14C, J15C and the rotation line (the straight line connecting the joint coordinates J14C and J15C) in one walking cycle are displayed continuously in time series. This makes it possible to present the change in the rotation of the subject's pelvis in one walking cycle.

[0060] In Fig. 14, the rotation of the pelvis is shown, but the present invention is not limited to this. For example, instead of the pelvis joint coordinates, the shoulder joint coordinates J3C, J5C and the rotation line (a straight line connecting the joint coordinates J3C and J5C) may be continuously displayed. This makes it possible to present the change in shoulder rotation.

[0061] Although not particularly limited, in Figs. 13 and 14, the number of walking states to be displayed on the display area 104_1 is set to a predetermined number, for example, 10, and walking states at the same time intervals are displayed.

[0062] In the above, an example of displaying gait features based on joint coordinates acquired by walking of a subject has been shown, but the present invention is not limited to this. For example, gait features obtained from joint coordinates acquired by measuring multiple subjects, or gait features obtained from joint coordinates acquired by measuring the same subject in different environments or at different times may be used for display. Next, an example of display using gait features obtained from joint coordinates acquired by measuring multiple subjects will be described with reference to the drawings. Here, too, the gait features of multiple subjects are assumed to have been subjected to the processing shown in FIG. 2 for the acquired joint coordinates.

[0063] Fig. 15 is a diagram showing a display screen of the exercise visualization system according to the embodiment. In Fig. 15, 104H shows a display screen displayed on the display unit 104 by executing the screen display program 4. The display screen 104H has a plurality of display areas 104_1 to 104_3, and different contents are simultaneously displayed in each display area. In the example shown in Fig. 15, the display area 104_1 displays the contents described in Fig. 14.

[0064] A radar chart using the walking characteristics is displayed in the display area 104_2. The items of the radar chart are the walking speed (velocity), stride length, vertical movement, rotation, and left-right wobble of the subject. The value of each item is set based on the walking characteristics of multiple subjects, and the walking characteristics of the subject (specific subject) measured this time are indicated by a characteristic line 505.

[0065] In addition, in the display area 104_3, the lateral wobble, which is one of the walking characteristics, is displayed by a bar graph. Depending on the number of subjects whose lateral wobble values ​​fall within a predetermined range, a distribution diagram of the lateral wobble is formed and displayed in the display area 104_3. In addition, which distribution the lateral wobble of the subject (specific subject) measured this time belongs to is clearly indicated by, for example, a color (dots in FIG. 15).

[0066] In this way, by displaying a comparison using a plurality of subjects as a standard, it is possible to present the current subject's position among the plurality of subjects.

[0067] Also, the previous walking characteristics of the current subject may be displayed superimposed on the radar chart (display area 104_2). Similarly, the previous lateral wobble may be displayed in a different color on the bar graph (display area 104_3). This makes it possible to present the current subject with, for example, improvements over time and / or through instructional intervention.

[0068] In order to make it possible to compare multiple subjects, it is preferable to make the number and intervals of walking states displayed in the display area 104_1 the same for each subject. When displaying the same subject multiple times, it is preferable to make the number and intervals of walking states displayed in the display area 104_1 the same.

[0069] The gait features of multiple subjects, previous gait features of the same subject, and / or gait features of the same subject in different environments may be stored in, for example, the storage unit 106 shown in Fig. 1, or may be stored in the server 300. When stored in the server 300, the gait features can be imported into the motion visualization system 1 via the Internet line 301 as necessary, making it possible to display as shown in Fig. 15. In this case, even if the storage capacity of the storage unit 106 is limited, comparison with many subjects is possible.

[0070] <Variation 1> FIG. 16 is a diagram illustrating a relationship between a depth camera and a subject according to the first modification of the embodiment.

[0071] In the first modification, the depth camera 200 is not installed on the floor surface 400 but is fixed to the ceiling surface 401. In this case, it is possible to instruct the subject 500 to walk toward the lens surface of the depth camera 200. As a result, when the moving direction of the subject 500 overlaps with the Z axis, it is possible to omit steps S4 and S6 shown in FIG. 2.

[0072] <Variation 2> FIG. 17 is a block diagram showing a configuration of an exercise visualization system according to the second modification of the embodiment. As shown in FIG.

[0073] FIG. 17 is similar to FIG. 2. The difference is that the depth camera 200 is changed to stereo cameras 200_1 and 200_2. By using the two cameras 200_1 and 200_2, it is possible to grasp the distance between the subject 500 and the cameras, and so on, so there is no need to use a depth camera. Also, compound eye cameras can be used as the two cameras 200_1 and 200_2.

[0074] According to the first embodiment, the values ​​of the joint coordinates are normalized based on the subject's information (reference data). By normalizing the joint coordinates for multiple subjects based on their respective information (reference data), it is possible to make a meaningful comparison of the movement trajectories between multiple subjects. In addition, even for the same subject, even when the environment changes and / or time has passed, it is possible to compare the movement trajectories without the influence of the change in the environment and / or the passage of time by normalizing the joint coordinates based on the same reference data. This makes it possible to grasp the improvement of motor skills over time and / or the improvement due to instruction intervention.

[0075] Furthermore, in the embodiment, coordinate conversion of the joint coordinates is performed so that the subject's axis of movement overlaps with the Z axis. This makes it possible to standardize the Z axis among multiple subjects. Also, when the same subject is measured in different environments, it becomes possible to standardize the Z axis.

[0076] That is, it is possible to provide a motion visualization system and a motion visualization method that are capable of comparing multiple pieces of measurement data (joint coordinates, walking characteristics, etc.).

[0077] The exercise visualization system and the exercise visualization method according to the first embodiment are particularly useful when used by an exercise instructor who instructs a subject in, for example, a fitness club, a rehabilitation facility, or the like. That is, by using the exercise visualization system and the exercise visualization method according to the first embodiment, an exercise instructor can compare a plurality of subjects with the subject measured this time, and can intuitively and quantitatively understand the characteristics of the subject measured this time from the displayed walking characteristics, leading to improvement. In addition, the exercise instructor can intuitively and quantitatively understand the improvement over time or due to instruction intervention. Furthermore, the exercise instructor can explain the improvement status, etc. to the subject measured this time while presenting the information shown in FIG. 15 to the subject measured this time.

[0078] 15 shows an example in which the gait features are displayed as radar charts and distributions, but the present invention is not limited to this. For example, statistical processing such as average and variance may be performed on the gait features, and the processing results may be displayed.

[0079] Furthermore, if the inclination shown in FIG. 9A is within an allowable range, step S7 shown in FIG. 2 may be omitted.

[0080] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention. [Explanation of symbols]

[0081] 1. Movement visualization system 2 Skeleton Recognition Program 3 Skeleton normalization program 4-screen display program 5. Gait characteristic calculation program 6. Pitch Extraction Programs 100 Computers 101 Arithmetic section 102 Control section 103 Operation input section 104 Display section 105 Memory 106 Storage section 200 Depth Camera 300 Servers 500 Target J1~J21 joint coordinates S0~S9 steps

Claims

1. A motion visualization system including a storage unit, a control unit, a calculation unit, and a display unit, The storage unit includes a program used by the calculation unit, and acquires and stores a plurality of image data successively capturing images of a subject walking in a direction in which the camera is installed; The control unit uses the program in the calculation unit, For each of the plurality of image data, a plurality of parts of the subject's skeleton are recognized, and a first axis is aligned with the optical axis of the camera lens, a second axis is perpendicular to the first axis and is the top and bottom of the lens surface of the camera, and a third axis is perpendicular to the first axis and the second axis and is the left and right of the lens surface of the camera. The plurality of parts of the skeleton are obtained as a plurality of skeleton coordinates using the first, second, and third axes, and stored in the storage unit; determining a walking state of the subject from the plurality of skeletal coordinates, and extracting, using the plurality of skeletal coordinates acquired for each of the plurality of image data, a time point at which the walking state changes from a specific state to the specific state again as one walking cycle having a walking start point and an end point; extracting, as a progression axis, a straight line connecting the skeletal coordinate of the start point of the walking and the skeletal coordinate of the end point of the walking in the one walking cycle, using at least one of the plurality of skeletal coordinates; normalizing the values ​​of the plurality of skeleton coordinates using a length between two different ones of the plurality of skeleton coordinates or an average value thereof in the plurality of image data as reference data; converting values ​​of the plurality of skeletal coordinates so that the progression axis of the plurality of skeletal coordinates overlaps with the first axis; 4. The motion visualization system, comprising: a display unit that displays gait characteristic information using the converted plurality of skeletal coordinates.

2. 2. The exercise visualization system according to claim 1, wherein the calculation unit converts the plurality of skeletal coordinates so that ground contact surfaces of both feet of the subject become parallel to the third axis.

3. 2. The motion visualization system according to claim 1, characterized in that for a given coordinate of the plurality of skeletal coordinates obtained by the normalization and the coordinate transformation, a transition of movement within one walking cycle in the third axis direction on the first axis is obtained as a movement trajectory, and characteristic information of staggering is specified based on a magnitude of movement in the third axis direction.

4. 2. The motion visualization system according to claim 1, characterized in that for a given coordinate of the plurality of skeletal coordinates that have been normalized and coordinate transformed, a transition of movement within one walking cycle in the second axis direction on the first axis is obtained as a movement trajectory, and characteristic information of up-and-down movement is identified based on a magnitude of movement in the second axis direction.

5. 4. The exercise visualization system according to claim 3, wherein the control unit sequentially displays the normalized and coordinate-converted plurality of skeletal coordinates and the movement trajectory on the display unit so as to change over time.

6. The exercise visualization system according to claim 3 , wherein the control unit compares the unsteadiness of a plurality of subjects with each other as a reference and displays the comparison results on the display unit.

7. 7. The exercise visualization system according to claim 3, wherein the control unit displays on the display unit a comparison result of the unsteadiness of the subject at different times or before and after an instruction intervention.

Images