Motion visualization system and motion visualization method

Abstract

To provide a motion visualization system that can compare multiple measurement data.SOLUTION: A motion visualization system 1 includes a skeletal recognition program 2 for acquiring the joint coordinates of a subject, a pitch extraction program 6 for extracting the walking cycle of the subject on the basis of the joint coordinates, and in one extracted walking cycle, a storage unit 106 for converting the coordinates of the joint coordinates into the reference coordinate system that the direction in which a predetermined part of the subject moves is used as an axis of the traveling direction and for storing a skeletal normalization program 3 for converting the joint coordinates on the basis of the information of the subject.SELECTED DRAWING: Figure 1

Description

Technical Field

[0005]

[0001] The present invention relates to a motion visualization system and a motion visualization method, and particularly to a motion visualization system and a motion visualization method for visualizing the walking behavior of a subject.

Background Art

[0002] The aging process is advancing. In an aging society, the increasing prevalence of locomotive syndrome due to musculoskeletal diseases has become a social problem. In order to extend the healthy life expectancy even in an aging society, it is desirable to detect the decline in the motor function of a subject (elderly, middle-aged and elderly people) at an early stage, provide appropriate exercise guidance in a fitness club, rehabilitation facility, etc., and improve the motor function of the subject.

[0003] An operation information processing device that displays the movement trajectory of a subject is described in, for example, Patent Document 1. Patent Document 1 shows that based on the coordinates of each part of the subject extracted from an image, the footprint of the subject and the movement trajectory of a predetermined part are displayed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0007] The object of the present invention is to provide a motion visualization system and a motion visualization method that enable the comparison of multiple measurement data.

[0008] Other objects and novel features of the present invention will become apparent from the description herein and the accompanying drawings. [Means for solving the problem]

[0009] A brief overview of some of the representative embodiments disclosed in this application is as follows.

[0010] In other words, the motion visualization system comprises a skeletal recognition unit that acquires skeletal coordinate information of a subject, a period extraction unit that extracts the subject's walking cycle based on the skeletal coordinate information, and a conversion unit that converts the values ​​of the skeletal coordinate information into a reference coordinate system in which the direction in which a predetermined part of the subject moves during one extracted walking cycle is the axis of the direction of movement, and converts the skeletal coordinate information based on the subject's information. [Effects of the Invention]

[0011] To briefly explain the effects obtained by a representative embodiment of the invention disclosed in this application, it is possible to provide a motion visualization system and motion visualization method that enable the comparison of multiple measurement data. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing the configuration of the motion visualization system according to the embodiment. [Figure 2]It is a flowchart for explaining the operation of the motion visualization system according to the embodiment. [Figure 3] It is a diagram showing the relationship between the depth camera and the subject according to the embodiment. [Figure 4] It is a diagram showing the relationship between the depth camera and the subject according to the embodiment. [Figure 5] It is a diagram for explaining the skeleton recognition according to the embodiment. [Figure 6] It is a diagram for explaining the extraction of one walking cycle according to the embodiment. [Figure 7] (A) and (B) are diagrams for explaining the normalization and coordinate transformation according to the embodiment. [Figure 8] (A) and (B) are diagrams for explaining the normalization and coordinate transformation according to the embodiment. [Figure 9] (A) and (B) are diagrams for explaining the coordinate transformation according to the embodiment. [Figure 10] It is a diagram for explaining the walking characteristics obtained by the motion visualization system according to the embodiment. [Figure 11] It is a diagram for explaining the walking characteristics obtained by the motion visualization system according to the embodiment. [Figure 12] (A) and (B) are diagrams for explaining the walking characteristics obtained by the motion visualization system according to the embodiment. [Figure 13] (A) and (B) are diagrams showing the display method of the walking characteristics according to the embodiment. [Figure 14] (A) and (B) are diagrams showing the display method of the walking characteristics according to the embodiment. [Figure 15] It is a diagram showing the display screen of the motion visualization system according to the embodiment. [Figure 16] It is a diagram showing the relationship between the depth camera and the subject according to the first modification example of the embodiment. [Figure 17] It is a block diagram showing the configuration of the motion visualization system according to the second modification example of the embodiment.

Embodiments of the Invention

[0013] The embodiments will be described with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the elements and their combinations described in the embodiments are essential for the solution of the invention.

[0014] (Embodiment) <Configuration of the Motion Visualization System> FIG. 1 is a block diagram showing the configuration of a motion visualization system according to an embodiment. In FIG. 1, reference numeral 1 denotes a motion visualization system. The motion visualization system 1 includes, without particular limitation, a computer 100, a depth camera 200, an Internet line 301, and a server 300 connected to the computer 100 via the Internet line 301. Here, an example in which the motion visualization system 1 includes the Internet line 301 and the server 300 will be described, but the present invention is not limited thereto. For example, the motion visualization system 1 may be constituted by the computer 100 and the depth camera 200 excluding the Internet line 301 and the server 300.

[0015] The computer 100 includes an arithmetic unit 101, a control unit 102, an operation input unit 103, a display unit 104, a memory 105, a storage unit 106, an external input / output unit 107, and a network communication unit 108 connected to a bus Bus. The control unit 102 reads a program or the like stored in advance in the storage unit 106 into, for example, the memory 105, and executes the program read into the memory 105. Of course, the control unit 102 may execute the program stored in advance in the storage unit 106 without reading it into the memory 105. In this case, the memory 105 is used as, for example, a work memory when the program is executed.

[0016] The arithmetic unit 101 is used to perform calculations when the program is executed by the control unit 102. The external input / output unit 107 is also used when the program is executed by the control unit 102. In this embodiment, the depth camera 200 is connected to the external input / output unit 107. By executing the program, the control unit 102 uses the external input / output unit 107 to control the depth camera 200 and acquires image data captured by the depth camera 200 via the external input / output unit 107.

[0017] The operation input unit 103 is equipped with, for example, a keyboard and mouse, and input is made to the computer 100 using the keyboard and mouse. The display unit 104 is equipped with, for example, a monitor such as an LCD that displays images, and displays images generated by the computer 100. The network communication unit 108 is connected between the bus and the internet line 301 and communicates between the computer 100 and the server 300. For example, data generated by the computer 100 is transmitted to the server 300 via the network communication unit 108 and the internet line 301 and stored in the server 300. Also, data stored in the server 300 is supplied to the computer 100 via the network communication unit 108 and the internet line 301 and used when executing programs.

[0018] The storage unit 106 is composed of, for example, a hard disk or an SSD (Solid State Drive). Multiple programs are stored in the storage unit 106. The storage unit 106 also stores data generated by the execution of programs and / or data used during program execution. As described above, multiple programs and data are stored in the storage unit 106, but in Figure 1, only the programs and data necessary to explain the embodiment are shown.

[0019] In Figure 1, 2 represents the skeleton recognition program, 3 the skeleton normalization program, 4 the screen display program, 5 the gait feature calculation program, and 6 the pitch (gait cycle) extraction program. Also in Figure 1, 7 represents the skeleton and gait feature data stored in the memory unit 106.

[0020] The control unit 102 executes the programs 2 to 6 described above, thereby realizing the functional units (function sections) that constitute the motion visualization system 1 in the computer 100. Specifically, when the skeleton recognition program 2 is executed, the skeleton recognition section is configured in the computer 100; when the skeleton normalization program 3 is executed, the conversion section is configured in the computer 100; when the pitch extraction program 6 is executed, the period extraction section is configured in the computer 100; and when the gait feature calculation program 5 is executed, the gait feature calculation section is configured in the computer 100.

[0021] When the screen display program 4 is executed, for example, an image generated by the gait feature calculation program 5 is displayed on the display unit 104. In addition, for example, data generated by the execution of the skeleton recognition program 2, the skeleton normalization program 3, the gait feature calculation program 5, and the pitch extraction program 6 is stored in the storage unit 106 as skeleton / gait feature data 7.

[0022] The skeletal and gait feature data 7 stored in the memory unit 106 is supplied to the server 300, for example, via the network communication unit 108 and the internet line 301. Alternatively, data stored in the server 300 is supplied to the memory unit 106 via the internet line 301 and the network communication unit 108 and stored as skeletal and gait feature data 7.

[0023] The motion visualization system 1 according to this embodiment captures the state of the subject (test subject) while they are walking using a depth camera 200, observes the changes in the subject's skeleton associated with walking based on the images obtained from the capture, and displays the subject's walking characteristics on, for example, a display unit 104.

[0024] <How the motion visualization system works> Figure 2 is a step-by-step diagram illustrating the operation of the motion visualization system according to the embodiment. The operation of the motion visualization system 1 shown in Figure 1 will be described below using Figures 1 and 2.

[0025] In step S0, the motion visualization system 1 begins operation. Upon starting operation, the control unit 102 begins executing the skeleton recognition program stored in the memory unit 106. This executes the image (depth) acquisition in step S1 and the skeleton recognition of each frame in step S2. In the following, steps S1 and S2 together may be referred to as the skeleton recognition process.

[0026] In step S1, during image acquisition, the depth camera 200 is used to continuously capture images of the subject as they walk. The state of capturing the subject by the depth camera 200 will be explained using diagrams. Figures 3 and 4 show the relationship between the depth camera and the subject according to the embodiment. Here, Figure 3 is a view of the depth camera 200 and the subject from the side, and Figure 4 is a view of the depth camera 200 and the subject from above. In Figures 3 and 4, 500 indicates the subject, and the subject 500 walks on the floor surface 400 of the room in the direction of travel indicated by the arrow. The depth camera 200 is installed on the floor surface 400 by a fixing member 201 such as a tripod, although this is not particularly limited. In Figures 3 and 4, 200R indicates the shooting range of the depth camera 200, the Z-axis indicates the depth direction of the depth camera 200 (along the optical axis of the lens of the depth camera 200), the Y-axis indicates the vertical direction along the lens surface of the depth camera 200 (the Y-axis perpendicular to the Z-axis), and the X-axis indicates the horizontal direction along the lens surface (the X-axis perpendicular to the Z-axis and Y-axis). Although not particularly limited, the center of the lens surface of the depth camera 200 is set as the starting point (0,0,0) of the X, Y, and Z axes. Of course, the starting points of the X, Y, and Z axes are not limited to this. For example, the starting point (0) of the Y-axis may be the floor surface 400. Note that 401 indicates the ceiling surface of the room.

[0027] For example, if a subject is instructed to walk toward the lens surface of the depth camera 200, the subject may become tense and exhibit an unusual gait. Therefore, as shown in Figure 4, the subject 500 is instructed to walk in a direction offset from the lens surface of the depth camera 200. As a result, as shown in Figure 4, the direction of movement of the subject 500 does not coincide with the Z-axis, which is the depth direction of the depth camera 200.

[0028] The depth camera 200 continuously captures images of a walking subject in the direction of travel. Multiple frames (image data) obtained through continuous shooting are temporarily stored, for example, in the storage unit 106 via the external input / output unit 107.

[0029] In step S2, the control unit 102, for example, using the calculation unit 101, performs a skeletal recognition process to recognize the subject's skeleton for each of the multiple frames temporarily stored in the storage unit 106. Figure 5 is a diagram illustrating the skeletal recognition according to the embodiment. As step S2 is executed, the joint coordinates of the subject 500 are obtained as shown in Figure 5. In Figure 5, as an example, the obtained joint coordinates (skeletal coordinates) are shown as J1 to J21. For example, J3 to J5 represent the shoulder joint coordinates, and J13 to J15 represent the pelvic joint coordinates. The values ​​of the joint coordinates J1 to J21 at this time are values ​​in the coordinate system related to the depth camera 200. That is, the values ​​(x, y, z) of the joint coordinates J1 to J21 are the values ​​in the X, Y, and Z axes as described above. As step S2 is executed, joint coordinates as shown in Figure 5 are obtained for each frame. The joint coordinates obtained here are stored in the storage unit 106 as, for example, skeletal / gait feature data 7.

[0030] Following step S2, step S3, the extraction of one gait cycle (cycle extraction process), is performed. That is, the control unit 102 executes the pitch extraction program 6 stored in the memory unit 106. In the pitch extraction program 6, the period of one gait of the subject is extracted using the joint coordinates acquired for each frame. Figure 6 is a diagram illustrating the extraction of one gait cycle according to the embodiment. In the pitch extraction program 6, the gait state 500_1 to 500_6 of the subject for each frame is determined based on the acquired joint coordinates. In the embodiment, although not particularly limited, the gait state 500_1 with the right foot forward and both feet spread apart is used as the start and end point of the gait, and the period between the start and end points is extracted as one gait cycle (one pitch). Of course, the start and end points are not limited to these, and any gait state may be used as the start and end points. In the following description, the gait state at the start point is indicated by the reference numeral 500_1S, and the gait state at the end point is indicated by the reference numeral 500_1E.

[0031] In the next step S4, the control unit 102 finds a straight line connecting the starting walking state 500_1S and the ending walking state 500_1E, and extracts the found straight line as the axis of movement (axis of movement). For example, the control unit 102 extracts the straight line connecting the pelvic joint coordinates J13 in walking state 500_1S and the pelvic joint coordinates J13 in walking state 500_1E as the axis of movement. Of course, it is not limited to the pelvic joint coordinates. Although not particularly limited, the extraction of the axis of movement in step S4 is performed as part of the pitch extraction program 6. Of course, the extraction of one gait cycle and the extraction of the axis of movement may be performed by separate programs.

[0032] Following step S4, in this embodiment, step S5 is performed to normalize the lengths of the XYZ axes based on the unit length (torso length), step S6 is performed to perform a coordinate transformation so that the Z axis coincides with the axis of movement, and step S7 is performed to perform a coordinate transformation so that the contact surfaces of both feet are parallel to the X axis. These steps S5 to S7 are realized by the control unit 102 executing the skeletal normalization program 3 stored in the memory unit 106. The program for extracting the axis of movement described above may be part of step S6. In this case, step S6 can be considered as a coordinate transformation process. Next, steps S5 to S7 will be described in detail with reference to the drawings.

[0033] <<Normalization and Coordinate Transformation>> Figures 7 and 8 are diagrams illustrating the normalization and coordinate transformation according to the embodiment. Here, Figure 7 is a view of subject 500 walking from the side (X-axis direction), and Figure 8 is a view of subject 500 walking from above (Y-axis direction).

[0034] In step S5, the values ​​of each joint coordinate of the subject 500 are transformed based on the subject 500's information (reference data). The reference data can include the subject 500's height, stride length, or the length of a predetermined body part. Here, we will explain the case where the length of a predetermined body part of the subject 500 is used as the reference data. The predetermined body part is, for example, the subject 500's torso length.

[0035] The torso length of subject 500 is the length between the shoulder joint coordinates J3-J5 and the pelvic joint coordinates J13-J15. As reference data, the length between the shoulder joint coordinates and the pelvic joint coordinates in one frame (torso length) may be used, but in this embodiment, the average value of the torso length in one gait cycle is used as reference data. In step S5, the values ​​of each joint coordinate (x, y, z) on the X, Y, and Z axes are converted to values ​​when the reference data is set to a unit length (e.g., 1). In other words, by executing step S5, the values ​​of each joint coordinate are normalized with respect to the torso length of subject 500.

[0036] As shown in Figure 4, the Z-axis in the depth direction is different from the direction of movement of subject 500. That is, the direction of the Z-axis is different from the direction of the movement axis determined in step S4. In step S6, the coordinate values ​​of each joint are transformed so that the movement axis extracted in step S4 coincides with the Z-axis. In other words, the coordinate values ​​of each joint are transformed into a reference coordinate system in which the direction of movement represented by the movement axis is the Z-axis.

[0037] Next, Figures 7 and 8 will be used to explain the state before and after normalization and coordinate transformation. Figure 7(A) shows the state before normalization and coordinate transformation, and Figure 7(B) shows the state after normalization and coordinate transformation.

[0038] In Figure 7(A), the joint coordinates for walking states 500_1S, 500_4, and 500_1E are indicated by circles, similar to Figure 5. Additionally, as an example, the shoulder and pelvic joint coordinates are labeled with symbols J3-J5 and J13-J15, respectively. Since the data is not yet normalized, the units for the Z and Y axes are meters (m). Regarding the pelvic joint coordinates, the axis of movement connecting the start and end points is indicated by symbol 501. The direction of movement is indicated by the arrow on the axis of movement 501. In Figure 7(A), the axis of movement 501 points in the direction where the values ​​of the Y and Z axes decrease. That is, subject 500 is walking in the direction where the values ​​of the Y and Z axes decrease.

[0039] In step S5, normalization is performed, so the joint coordinates (e.g., J3~J5, J13~J15) in walking states 500_1 and 500_4 are converted to joint coordinates (J3C~J5C, J13C~J15C) with torso length as the reference data (unit length), as shown in Figure 7(B). Of course, in Figure 7(B), the units of the Y and Z axes are torso length. Also, since the coordinate transformation is performed in step S6, the axis of movement 501 and the Z axis overlap. In Figure 7(B), 500_1SC, 500_4C, and 500_1EC show the walking states after the transformation. Since the axis of movement 501 overlaps with the Z axis, the changes in walking states associated with walking are along the Z axis.

[0040] Figure 8(A) shows the data before normalization and coordinate transformation, and Figure 8(B) shows the data after normalization and coordinate transformation. As can be seen from Figure 8(A), when viewed from above, subject 500 is walking in a direction where the Z-axis value is small and the X-axis value is large. With steps S5 and S6 performed, the changes in walking states 500_1SC, 500_4C, and 500_1EC associated with walking are aligned along the Z-axis, and the subject's joint coordinates are normalized to the unit of torso length.

[0041] <<Coordinate transformation related to slope>> Figure 9 is a diagram illustrating the coordinate transformation according to the embodiment. Here, Figure 9(A) shows the state before step S7 is performed, and Figure 9(B) shows the state after step S7 is performed.

[0042] In this embodiment, as shown in Figure 3, the depth camera 200 is installed on the floor surface 400 by a fixing member 201. Depending on how the depth camera 200 and / or the fixing member 201 are installed, the depth camera 200 may be tilted to the left or right with respect to the floor surface 400. In this case, in the frame acquired by the depth camera 200, the subject will be tilted with respect to the X-axis, as shown in Figure 9(A). As a result, the X-coordinates of each joint coordinate recognized in step S2 will not be aligned with the X-axis of the floor surface 400. Taking the pelvic joint coordinates J13 to J15 as an example, the X-coordinates of these joint coordinates are not aligned with the X-axis, as shown in Figure 9(A).

[0043] In step S7, the X-coordinate of each joint coordinate of the subject is transformed so that it is parallel to the floor plane 400 (X-axis). As a result, as shown in Figure 9(B), the pelvic joint coordinates J13~J15 are transformed into joint coordinates J13C~J15C, which have X-coordinates parallel to the floor plane 400. Although the pelvic joint coordinates were used as an example, other joint coordinates are transformed similarly in step S7. This is achieved by transforming the Y-axis values ​​of each joint coordinate so that, for example, the difference in Y-axis values ​​between joint coordinates J14C and J15C is minimized.

[0044] In this embodiment, as shown in Figure 2, normalization (step S5) and coordinate transformation (steps S6 and S7) are performed in that order. However, the embodiment is not limited to this, and for example, the normalization in step S5 may be performed after the coordinate transformation in steps S6 and S7.

[0045] Returning to Figure 2, let's explain how the motion visualization system works.

[0046] Following step S7, the gait feature calculation (gait feature calculation process) of step S8 is performed. That is, the control unit 102 shown in Figure 1 executes the gait feature calculation program 5 stored in the memory unit 106. This gait feature calculation program 5 performs calculations using the joint coordinates obtained by the execution of steps S5 to S7, i.e., the joint coordinates that have been normalized and transformed.

[0047] The gait feature calculation program 5 makes it possible to acquire various characteristics of 500 subjects related to walking. Here, as examples of gait features, we will explain lateral swaying during walking, vertical movement during walking, and rotation of a predetermined body part.

[0048] <<Walking Characteristics>> Figures 10 to 12 are diagrams illustrating the gait characteristics acquired by the motion visualization system according to the embodiment. Here, Figure 10 illustrates the lateral sway of the subject 500, Figure 11 illustrates the rotation of a predetermined part of the subject 500, and Figure 12 illustrates the vertical movement of the subject. In Figures 10 to 12, J3C to J5C, J8C, J11C, J13C to J15C, J18C, and J21C are joint coordinates acquired by applying the normalization and coordinate transformation described above to the joint coordinates J3 to J5, J8, J11, J13 to J15, J18, and J21 shown in Figure 5.

[0049] <<<Dizziness>>> In Figure 10, 502 shows the movement trajectory (sway trajectory) in the X direction of a predetermined part when subject 500 walks for one cycle. This movement trajectory 502 is calculated by executing the gait characteristic calculation program 5. Here, the joint coordinate J13C of the pelvis is used as the predetermined part. Therefore, the movement trajectory 502 represents the transition of movement when subject 500 walks for only one gait cycle. When subject 500 walks for one gait cycle, if there is no lateral swaying, the movement trajectory 502 is parallel to the Z axis or overlaps with the Z axis. However, if subject 500 sways from side to side, as shown in Figure 10, the movement trajectory 502 becomes a curve that changes on both sides of the Z axis. Of course, the movement trajectory 502 may also change only on one side. This makes it possible to identify whether subject 500 is swaying and the magnitude (value) of the swaying.

[0050] Here, the movement of the pelvic joint coordinate J13C between walking state 500_1SC and 500_1EC was used to calculate the movement trajectory 502, but this is not the only method. For example, the movement of the shoulder joint coordinate J4C may be used to calculate the movement trajectory 502.

[0051] <<<Rotation of the specified part>>> In Figure 11, 503 is the rotation line. In Figure 11, the rotation line 503 is the straight line connecting the X coordinates of the pelvic joint coordinates J13C to J15C. The angle between this rotation line 503 and the virtual line VL perpendicular to the Z axis is calculated as the pelvic rotation angle. This rotation line 503 and the pelvic rotation angle are calculated by executing the gait characteristic calculation program 5. By calculating the pelvic rotation angle at each timing during the period when subject 500 is walking one gait cycle, it is possible to understand the progression of pelvic rotation.

[0052] Figure 11 uses the pelvic rotation angle as an example, but it is not limited to this. For example, one could calculate the shoulder rotation angle and track its changes over time.

[0053] <<<Up and down movement>>> In Figure 12, 504 represents the vertical movement trajectory. The vertical movement trajectory 504 shows the Y-direction movement trajectory of a predetermined part when the subject 500 walks one gait cycle. This vertical movement trajectory 504 is calculated by executing the gait feature calculation program 5. Here, the pelvic joint coordinate J13C is used as the predetermined part. Therefore, the vertical movement trajectory 504 represents the transition of vertical movement when the subject 500 walks for only one gait cycle. If the subject 500 does not move vertically during one gait cycle, the vertical movement trajectory 504 will be a constant value in the Y-axis. However, if the subject 500 changes vertically during walking, the vertical movement trajectory 504 will be a curve that changes in the Y-axis direction, as shown in Figure 12. This makes it possible to understand whether or not the subject 500 moves vertically and the magnitude of the vertical movement. In Figure 12, the pelvic joint coordinate J13C is used, but it is not limited to this. For example, the shoulder joint coordinate J4C may be used.

[0054] The various gait characteristics calculated in step S8 are stored in the memory unit 106 as skeletal / gait characteristic data 7.

[0055] In step S9, following step S8, the process up to calculating gait characteristics is completed. After this, the user, for example, operates the operation input unit 103 (Figure 1) to instruct the motion visualization system 1 to display the gait characteristics. In response to this instruction, the control unit 102 (Figure 1) executes the screen display program 4 (Figure 1).

[0056] Next, an example of walking characteristics displayed on the display unit 104 (Figure 1) by the execution of the screen display program 4 (display step) will be explained using drawings. Figures 13 and 14 are diagrams showing a method for displaying walking characteristics according to an embodiment.

[0057] Figure 13 shows a display method for showing the lateral sway of a subject based on skeletal and gait characteristic data 7. In Figure 13, 104_1 shows a part of the display area of ​​the display screen of the display unit 104. The screen display program 4 displays the changes in joint coordinates and the sway trajectory 502 related to lateral sway on the display area 104_1 like an animation. That is, the joint coordinates and sway trajectory 502 that change over time are displayed on the display area 104_1. The joint coordinates in the first walking state 500_1SC of one gait cycle are displayed as shown in Figure 13(A). Subsequently, the joint coordinates and sway trajectory 502 that change over time are sequentially displayed on the display area 104_1, and the joint coordinates and sway trajectory 502 in the last walking state 500_1EC of one gait cycle are displayed as shown in Figure 13(B).

[0058] An example has been described in which changes in joint coordinates are also displayed on the display area 104_1, but this is not the only way to do so. For example, the display area 104_1 may only display the joint coordinates of the initial walking state 500_1SC and the final walking state 500_1EC, as well as the sway trajectory 502. By displaying the data as shown in Figure 13, it is possible to show the subject's lateral sway.

[0059] Figure 14 is similar to Figure 13. The difference is that in Figure 14, the pelvic joint coordinates J14C, J15C, and rotation line are displayed together with the sway trajectory 502. That is, the pelvic joint coordinates J14C, J15C, and rotation line (a straight line connecting joint coordinates J14C and J15C) are displayed continuously in time series during one gait cycle. This makes it possible to show the changes in the subject's pelvic rotation during one gait cycle.

[0060] Figure 14 shows pelvic rotation, but it is not limited to this. For example, instead of pelvic joint coordinates, the shoulder joint coordinates J3C and J5C and the rotation line (a straight line connecting joint coordinates J3C and J5C) could be displayed continuously. This would make it possible to show changes in shoulder rotation.

[0061] While not particularly restricted, in Figures 13 and 14, the number of walking states displayed on the display area 104_1 is predetermined, for example, set to 10, and walking states with the same time interval are displayed.

[0062] The above examples illustrate how to display gait characteristics based on joint coordinates obtained from a subject's walking, but the method is not limited to this. For example, gait characteristics obtained from joint coordinates acquired by measuring multiple subjects, or gait characteristics obtained from joint coordinates acquired by measuring the same subject in different environments or at different times, may also be used for display. Next, an example of displaying gait characteristics obtained from joint coordinates acquired by measuring multiple subjects will be explained using diagrams. Here, too, the gait characteristics of multiple subjects are assumed to have been processed as shown in Figure 2 for the acquired joint coordinates.

[0063] Figure 15 shows the display screen of the motion visualization system according to an embodiment. In Figure 15, 104H indicates the display screen that is displayed on the display unit 104 when the screen display program 4 is executed. The display screen 104H has multiple display areas 104_1 to 104_3, and different content is displayed in each display area simultaneously. In the example shown in Figure 15, the content described in Figure 14 is displayed in display area 104_1.

[0064] Display area 104_2 shows a radar chart using gait characteristics. The radar chart items are the subject's walking speed (velocity), stride length, vertical movement, rotation, and lateral sway. Based on the gait characteristics of multiple subjects, values ​​for each item are set, and the gait characteristics of the subject measured this time (a specific subject) are shown by the feature line 505.

[0065] Furthermore, in display area 104_3, lateral sway, one of the gait characteristics, is displayed as a bar graph. A distribution map of lateral sway is formed based on the number of subjects whose lateral sway values ​​fall within a predetermined range, and this map is displayed in display area 104_3. In addition, the distribution to which the lateral sway of the subject measured in this study (a specific subject) belongs is clearly indicated, for example, by color (dots in Figure 15).

[0066] In this way, by displaying comparisons based on multiple subjects, it is possible to show the subject their position among multiple subjects.

[0067] Furthermore, the subject's previous gait characteristics may be overlaid on the radar chart (display area 104_2). Similarly, previous lateral sway may be displayed in a different color on the bar graph (display area 104_3). This makes it possible to present to the subject, for example, improvements over time and / or due to instructional interventions.

[0068] To enable comparisons between multiple subjects, it is desirable to keep the number and interval of walking states displayed in display area 104_1 the same for all subjects. Even for the same subject, when displaying multiple times, it is desirable to keep the number and interval of walking states displayed in display area 104_1 the same.

[0069] The gait characteristics of multiple subjects, the previous gait characteristics of the same subject, and / or the gait characteristics of the same subject in different environments may be stored, for example, in the memory unit 106 shown in Figure 1, or they may be stored in the server 300. If stored in the server 300, the gait characteristics can be imported into the motion visualization system 1 via the internet line 301 as needed, and displayed as shown in Figure 15. In this case, even if the storage capacity of the memory unit 106 is limited, it becomes possible to compare with many subjects.

[0070] <Example 1> Figure 16 shows the relationship between the depth camera and the subject according to a modified example 1 of the embodiment.

[0071] In Modification 1, the depth camera 200 is fixed to the ceiling surface 401 instead of being installed on the floor surface 400. In this case, the subject 500 can be instructed to walk toward the lens surface of the depth camera 200. As a result, if the direction of movement of the subject 500 coincides with the Z-axis, steps S4 and S6 shown in Figure 2 can be omitted.

[0072] <Modification 2> Figure 17 is a block diagram showing the configuration of a motion visualization system according to a modified example 2 of the embodiment.

[0073] Figure 17 is similar to Figure 2. The difference is that the depth camera 200 has been replaced with stereo cameras 200_1 and 200_2. By using two cameras 200_1 and 200_2, it is possible to determine the distance between the subject 500 and the cameras, thus eliminating the need for a depth camera. Alternatively, a compound eye camera could be used instead of the two cameras 200_1 and 200_2.

[0074] According to Embodiment 1, the joint coordinate values ​​are normalized based on the subject's information (reference data). By normalizing the joint coordinates for multiple subjects based on their respective information (reference data), it is possible to compare significant movement trajectories among multiple subjects. Furthermore, even for the same subject, by normalizing the joint coordinates based on the same reference data when the environment changes or / or time passes, it is possible to compare movement trajectories while excluding the effects of environmental changes or / or the passage of time. This makes it possible to grasp the improvement in motor ability over time or / or improvement due to instructional intervention.

[0075] Furthermore, in this embodiment, the joint coordinates are transformed so that the subject's axis of movement coincides with the Z-axis. This makes it possible to standardize the Z-axis among multiple subjects. It also makes it possible to standardize the Z-axis even when measuring the same subject in different environments.

[0076] In other words, it is possible to provide a motion visualization system and motion visualization method that allows for the comparison of multiple measurement data (joint coordinates, gait characteristics, etc.).

[0077] The exercise visualization system and method according to Embodiment 1 are particularly beneficial for use by exercise instructors who guide individuals in places such as fitness clubs and rehabilitation facilities. Specifically, by using the exercise visualization system and method according to Embodiment 1, exercise instructors can compare multiple individuals with the individual measured in this instance. They can intuitively and quantitatively understand the characteristics of the individual measured in this instance from the displayed gait characteristics and use this understanding to facilitate improvement. Furthermore, exercise instructors can intuitively and quantitatively understand improvements made over time or through instructional interventions. In addition, exercise instructors can explain the progress of improvement to the individual measured in this instance while presenting them with information such as that shown in Figure 15.

[0078] Figure 15 shows an example of displaying gait characteristics using radar and distributions, but it is not limited to this. For example, statistical processing such as mean and variance could be performed on the gait characteristics, and the results of the processing could be displayed.

[0079] Furthermore, if the slope shown in Figure 9(A) is within an acceptable range, step S7 shown in Figure 2 may be omitted.

[0080] Although the present invention has been specifically described above based on embodiments, it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from its essence. [Explanation of symbols]

[0081] 1. Motion Visualization System 2. Skeleton Recognition Program 3. Skeletal Normalization Program 4. Screen display program 5. Walking Feature Calculation Program 6. Pitch Extraction Program 100 Computers 101 Arithmetic section 102 Control Unit 103 Operation Input Section 104 Display section 105 memory 106 Storage section 200 Depth Camera 300 servers 500 target individuals J1~J21 Joint Coordinates S0~S9 Steps

Claims

1. A depth camera is installed on the floor surface and photographs a person walking on the floor surface, A skeleton recognition unit that acquires the skeletal coordinate information of the subject based on the image data of the subject captured by the depth camera, A period extraction unit that extracts the gait cycle of the subject based on the skeletal coordinate information, A transformation unit transforms the skeletal coordinate information so that the direction of movement of a predetermined part of the subject in one walking cycle extracted by the period extraction unit coincides with the depth direction of the depth camera, and normalizes the skeletal coordinate information based on the subject's height, stride length, or length information of a predetermined part of the subject's body. Equipped with, A motion visualization system in which, when the depth camera is tilted with respect to the floor surface, the conversion unit converts the skeletal coordinate information so that the contact surfaces of both feet of the subject are parallel to the axis of the floor surface.

2. In the motion visualization system according to claim 1, A gait characteristic calculation unit calculates the gait characteristics of the subject based on the skeletal coordinate information obtained by the conversion unit, A display unit that displays the walking characteristics of the subject, A motion visualization system that further incorporates these features.

3. In the motion visualization system according to claim 2, The aforementioned display unit shows the movement trajectory of a predetermined part of the subject as they walk, in this motion visualization system.

4. In the motion visualization system according to claim 3, The aforementioned display unit shows the change in rotation of a predetermined part of the subject during the walking cycle and the subject's unsteadiness during one walking cycle, in this motion visualization system.

5. The process involves using a depth camera installed on the floor to photograph a person walking on the floor and capturing image data of the person, A skeletal recognition step is performed to acquire changes in the skeletal coordinates of the subject based on the aforementioned image data, A period extraction step for extracting the gait cycle of the subject based on the changes in the skeletal coordinates, A coordinate transformation step is performed to transform the skeletal coordinates so that the direction of movement of a predetermined part of the subject in one walking cycle extracted in the period extraction step coincides with the depth direction of the depth camera. A skeletal normalization step in which the skeletal coordinates are normalized based on information such as the height, stride length, or length of a predetermined part of the subject, Equipped with, A motion visualization method comprising the step of transforming the skeletal coordinates so that when the depth camera is inclined with respect to the floor surface, the contact surfaces of both feet of the subject are parallel to the axis of the floor surface.

6. In the motion visualization method described in claim 5, A gait characteristic calculation step calculates the gait characteristics of the subject based on the skeletal coordinates obtained in the coordinate transformation step and the skeletal normalization step, A display step that displays the characteristics of the gait of the subject, A method for visualizing movement that further enhances this.

7. In the motion visualization method described in claim 6, The method for visualizing movement, wherein the display step displays the movement trajectory of a predetermined part of the subject accompanying walking.

8. In the motion visualization method described in claim 7, A method for visualizing motion, wherein the display step displays changes in the rotation of a predetermined part of the subject during the walking cycle and the subject's unsteadiness during the walking cycle.

Images