Posture display system and posture display method

Abstract

To provide a posture display system capable of intelligibly displaying a change in a posture.SOLUTION: A posture display system 1 includes a storage unit 500_S for storing time-series data of skeleton information, a keyboard 100_K for inputting setting of a reference axis to a prescribed part of the skeleton information, and an arithmetic unit 500_O. The arithmetic unit 500_O extracts, from the time-series data, skeleton information when the prescribed part of the subject is present in a first state away from the reference point on the reference axis in a first direction, extracts, from the time-series data, skeleton information when the prescribed part of the subject is present in a second state away from the reference point in a second direction opposite to the first direction, and superimposedly displays the skeleton information in the first state with the skeleton information in the second state.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a posture display system and a posture display method, and more particularly to a posture display system and a posture display method suitable for displaying, for example, characteristics of a subject's walking.

Background Art

[0002] With the progress of aging, in an aging society, locomotive diseases such as locomotive syndrome and decline in motor function have become social problems. In order to prevent and improve locomotive diseases and decline in motor function, technologies have been developed to photograph a subject's walking with sensors or cameras, record data related to the walking posture, and analyze the walking posture to visualize characteristics of the walking and the level of motor function.

[0003] For example, Patent Document 1 discloses a technique for extracting and visualizing a subject's walking cycle based on the subject's skeletal coordinate information. Further, Patent Document 2 discloses a technique for displaying temporal changes in the knee joint based on angle data obtained by a knee joint angle sensor.

[0004] Furthermore, although it is a technique related to sports learning, Patent Document 3 discloses a technique for superimposing and displaying two skeletons (the skeletons of a learner and a model) during an operation.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] Patent Documents 1 and 2 display the time-series changes of a predetermined part of a target person, and Patent Document 3 displays the superposition of the skeletons of different people. In any case, the extraction of the posture to be displayed is left to the user. Therefore, there has been a problem that some users cannot extract characteristic postures and it is difficult to understand what is being displayed.

[0007] An object of the present invention is to provide a posture display system and a posture display method capable of extracting characteristic postures and easily displaying changes in postures and the like.

[0008] Other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

Means for Solving the Problems

[0009] The outline of typical embodiments disclosed in the present application will be briefly described as follows.

[0010] That is, the posture display system includes a storage unit that stores time-series data of skeleton information, a user interface that inputs the setting of a reference axis for a predetermined part of the skeleton information, and a calculation unit. Here, the calculation unit extracts the skeleton information when the predetermined part exists in a first state separated from the reference point in the first direction from the time-series data, extracts the skeleton information when the predetermined part exists in a second state separated from the reference point in the second direction opposite to the first direction from the time-series data, and superimposes the skeleton information in the first state and the skeleton information in the second state.

[0011] According to the posture display system according to the embodiment, characteristic movements are extracted from the skeletal information that changes over time, and the skeletal information related to the characteristic movements is displayed. For example, in the posture display system according to a typical embodiment, the change (movement) between the first state that is farthest from the reference point of the reference axis in the first direction and the second state that is farthest from the first direction (second direction) is regarded as a characteristic movement, this change is extracted, and the posture of the first state (first posture) and the posture of the second state (second posture) are displayed superimposed. Further, in the first posture and the second posture, the regions formed by specific bones are filled with a predetermined color. Thereby, it becomes possible to display characteristic movements in an easy-to-understand manner, and even a user with little experience can easily confirm, for example, how much the characteristic movement (movement from the first state to the second state) has changed by training or the like.

Effect of the Invention

[0012] Among the inventions disclosed in the present application, the effects obtained by typical embodiments will be briefly described. It is possible to provide a posture display system capable of easily displaying changes in posture and the like.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0014] The embodiments will be described with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the elements and combinations thereof described in the embodiments are essential for the solution means of the invention.

[0015] (Embodiment 1) <Configuration of Posture Display System> FIG. 1 is a block diagram showing the configuration of the posture display system according to Embodiment 1. In FIG. 1, 1 indicates the posture display system. The posture display system 1 includes, but is not particularly limited to, a personal computer (hereinafter also simply referred to as a computer) 100, a camera 200, a smartphone 300, a network line 400, and a server 500. The posture display system 1 does not necessarily need to include both the computer 100 and the smartphone 300, and may include only one of them.

[0016] The server 500, the computer 100, and the smartphone 300 are connected to the network line 400. The server 500 is accessed by the computer 100 or the smartphone 300 via the network line 400 and operates according to instructions from the computer 100 or the smartphone 300. Also, by operating, the data generated by the server 500 is supplied to the computer 100 or the smartphone 300 via the network line 400 and, for example, displayed.

[0017] Also, the camera 200 is connected to the network line 400, and for example, the image data captured by the camera 200 is supplied to the server 500 via the network line 400.

[0018] In the posture display system 1 according to the first embodiment, the functions related to posture display are realized by the server 500. Also, the settings and instructions related to posture display are performed by the computer 100 or the smartphone 300. For example, the settings and instructions related to posture display are given to the server 500 by the computer 100 or the smartphone 300 via the network line 400. When an instruction related to posture display is given, the server 500 controls the camera 200 to photograph the subject via the network line 400. When the photographed image data of the subject is supplied from the camera 200 to the server 500 via the network line 400, the server 500 stores the image data of the subject, executes the process related to posture display, generates the data related to posture display, and supplies it to the computer 100 or the smartphone 300 via the network line 400. Thereby, posture display is performed on the computer 100 or the smartphone 300.

[0019] Taking as an example the case where the computer 100 makes settings and instructions related to posture display, the settings and instructions related to posture display are input into the computer 100 by the mouse 100_M and keyboard 100_K connected to the computer 100, and are supplied to the server 500 via the network line 400. Further, the data related to posture display generated in the server 500 is supplied to the computer 100 and is displayed on the display unit (not shown) of the computer 100.

[0020] The smartphone 300 is the same as the computer 100. That is, the settings and instructions related to posture display are input by the touch panel (not shown) of the smartphone 300, and the data related to posture display is displayed on the screen of the smartphone 300.

[0021] The mouse 100_M, keyboard 100_K, and touch panel can be regarded as user interfaces for the user to make settings and instructions related to posture display. In addition, the user interface may be a numeric keypad, scanner, microphone, sensor, display, printer, or speaker.

[0022] <<Function related to posture display>> The posture display is realized by a combination of a plurality of functional units (functions). Each functional unit is realized by a program corresponding to the functional unit being executed in the server 500. Before explaining the program corresponding to the functional unit, an example of the configuration of the server 500 will be described with reference to FIG. 1.

[0023] In FIG. 1, the server 500 includes an arithmetic unit 500_O, a control unit 500_C, a memory 500_M, a network communication unit 500_N, and a storage unit 500_S, which are surrounded by a dashed-dotted line. The arithmetic unit 500_O, control unit 500_C, memory 500_M, network communication unit 500_N, and storage unit 500_S are connected to a bus 500_B, and data and the like are transmitted and received to and from each other via the bus 500_B.

[0024] The network communication unit 500_N is connected to the network line 400 and performs transmission and reception of data and the like between the network line 400 and the bus 500_B.

[0025] The storage unit 500_S is a storage medium in which a plurality of programs related to posture display are stored in advance, which will be described later. The storage unit 500_S includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a flash memory. The programs stored in the storage unit 500_S are read out to the arithmetic unit 500_O via the bus 500_B. The arithmetic unit 500_O realizes functions related to posture display by executing the read programs. The memory 500_M is used, for example, for temporarily storing data when the arithmetic unit 500_O executes a program. Also, the control unit 500_C controls the arithmetic unit 500_O, the memory 500_M, the network communication unit 500_N, and the storage unit 500_S.

[0026] In the storage unit 500_S according to the first embodiment, a skeleton recognition program 501, a walking cycle extraction program 502, a reference axis setting program 503, a state extraction program 504, a posture display program 505, and a UI (User Interface) program 506 are stored in advance. Further, the storage unit 500_S includes a skeleton coordinate data (hereinafter also referred to as a data storage area) 500_SA for storing various data such as the skeleton coordinate data of the subject.

[0027] When the subject walks, the walking state is photographed by the camera 200, and the time-series image data of the subject obtained by the photographing is stored in the data storage area 500_SA by the network communication unit 500_N. Thereafter, the control unit 500_C causes the arithmetic unit 500_O to execute the programs 501 to 506, thereby realizing functions corresponding to the programs.

[0028] That is, when the skeleton recognition program 501 is executed by the arithmetic unit 500_O, a skeleton recognition function unit that recognizes the skeleton coordinates of the subject based on the image data of the subject is realized.

[0029] Also, when the walking cycle extraction program is executed by the arithmetic unit 500_O, a walking cycle extraction function unit that extracts the cycle (pitch) of the subject's walking is realized.

[0030] Furthermore, when the state extraction program is executed by the arithmetic unit 500_O, a state extraction function unit that extracts that a predetermined part of the subject is in a first state and a second state that are farthest from the reference point on the reference axis is realized. Also, when the posture display program is executed by the arithmetic unit 500_O, a posture display function unit that superimposes and displays the first state and the second state extracted by the state extraction function unit is realized.

[0031] The UI program 506 is an input / output unit that realizes the input / output of the computer 100 and the smartphone 300 when executed by the arithmetic unit 500_O. Although an example will be described later, a function of displaying the posture of the subject to the user is realized by this UI program 506. Also, when the reference axis setting program 503 is executed by the arithmetic unit 500_O, a reference axis setting function unit for the user to set the above-mentioned reference axis is realized, for example.

[0032] <Operation of the Posture Display System> FIG. 2 is a flowchart for explaining the operation of the posture display system according to the first embodiment. The operation of the posture display system 1 will be mainly described with reference to FIGS. 1 and 2.

[0033] <<Steps S0, S1>> In step S0, the operation of the posture display system 1 starts (processing starts). By starting the operation, the control unit 500_C executes step S1 of acquiring skeletal data. In step S1, the image data of the subject is captured and the skeletal recognition of the subject is performed.

[0034] First, the capture of the subject's image data will be described. The control unit 500_C uses the camera 200 shown in FIG. 1 to continuously capture the walking subject and stores it as the image data of the subject during walking in the data storage area 500_SA via the network line 400 and the network communication unit 500_N.

[0035] The state of capturing the subject by the camera 200 will be described with reference to the drawings. FIG. 3 is a diagram showing the relationship between the camera and the subject according to Embodiment 1. Here, FIG. 3(A) is a view of the camera 200 and the subject from the side, and FIG. 3(B) is a view of the camera 200 and the subject from above. In FIG. 3, 600 indicates the subject, and the subject 600 walks on the floor surface DFL of the room in the advancing direction indicated by the arrow. The camera 200 is installed on the floor surface DFL by a fixing member 201 such as a tripod, although not particularly limited. Also, in FIG. 3, 200R indicates the shooting range of the camera 200, the Z-axis indicates the depth direction of the camera 200 (the direction along the optical axis of the lens of the camera 200), the Y-axis indicates the vertical (Y-axis perpendicular to the Z-axis) direction along the lens surface of the camera 200, and the X-axis indicates the horizontal (X-axis perpendicular to the Z-axis and the Y-axis) direction along the lens surface. Although not particularly limited, the center of the lens surface of the camera 200 is set as the origin (0, 0, 0) of the X-axis, Y-axis, and Z-axis. Of course, the origin of the X-axis, Y-axis, and Z-axis is not limited to this. For example, the origin (0) of the Y-axis may be the floor surface DFL. Note that UFL indicates the ceiling surface of the room.

[0036] The subject 600 walks towards the lens surface of the camera 200. As a result, the camera 200 continuously captures the subject 600 walking in the advancing direction. The time-series image data, which are a plurality of frames obtained by continuous shooting, are stored in the data storage area 500_SA of the storage unit 500_S.

[0037] Next, the control unit 500_C causes the arithmetic unit 500_O to execute the skeleton recognition program 501. As a result, for each of the plurality of frames stored in the data storage area 500_SA, a skeleton recognition process for recognizing the skeleton of the subject is executed. FIG. 4 is a diagram for explaining the skeleton recognition according to the first embodiment. When the skeleton recognition program 501 is executed, joint coordinates of the subject 600 are acquired as shown in FIG. 4. In FIG. 4, as an example, the acquired joint coordinates (skeleton coordinates) are shown as J1 to J21. For example, J3 to J5 indicate the joint coordinates of the shoulder, and J13 to J15 indicate the joint coordinates of the pelvis. The values of the joint coordinates J1 to J21 at this time are values in the coordinate system related to the camera 200. That is, the values (x, y, z) of the joint coordinates J1 to J21 are the values on the X-axis, Y-axis, and Z-axis described above. When the skeleton recognition program is executed by the arithmetic unit 500_O, joint coordinates as shown in FIG. 4 are acquired for each frame. The joint coordinates acquired here are stored in the data storage area 500_SA as skeleton coordinate (skeleton information) data, as shown in FIG. 1. Since the posture of the subject is represented by the joint coordinates J1 to J21, it can also be considered that the posture of the subject corresponding to each frame is stored in the data storage area 500_SA.

[0038] <<Steps S2, S3>> Next, in step S2, extraction of one walking cycle (cycle extraction step) is performed. That is, the control unit 500_C causes the arithmetic unit 500_O to execute the walking cycle extraction program 502. By executing the walking cycle extraction program 502, the cycle of one step of the subject is extracted using the joint coordinates of each frame stored in the data storage area 500_SA. FIG. 5 is a diagram for explaining the extraction of one walking cycle according to the first embodiment. In the walking cycle extraction program 502, the walking states 600_1 to 600_6 of the subject for each frame are discriminated based on the acquired joint coordinates. In the first embodiment, although not particularly limited, the walking state 600_1 with the right foot forward and both feet apart is used as the start point and end point of walking, and the period between the start point and the end point is extracted as one walking cycle (one pitch). Of course, the start point and the end point are not limited to this, and any walking state may be used as the start point and the end point.

[0039] In the following description, the walking state of the start point is indicated by the symbol 600_1S, and the walking state of the end point is indicated by the symbol 600_1E. The skeletal coordinate data corresponding to each of the walking states 600_1S to 600_1E in the extracted one walking cycle is also stored in the data storage area 500_SA.

[0040] In the embodiment, in step S3 following step S2, one or more reference axes are set. The one or more reference axes set here are used to extract two postures in subsequent steps S4 and S5. The setting of one or more reference axes is realized by the control unit 500_C shown in FIG. 1 causing the arithmetic unit 500_O to execute the reference axis setting program 503. During the execution of the reference axis setting program 503, the user operates, for example, the keyboard 100_K to input the setting of one or more reference axes to the computer 100. The input setting of one or more reference axes is stored in the data storage area 500_SA via the network line 400 or the like.

[0041] The one or more reference axes input by the user are not particularly limited, but the coordinate axes of the joint coordinate system (which also applies to the coordinate system related to the camera 200 in Embodiment 1), which is a coordinate system for representing the joint coordinates of the subject, are used. Also, a predetermined point on the coordinate axis specified as the reference axis is used as the reference point. For example, as the reference axis, the user designates and inputs the X-axis, Y-axis, Z-axis, or composite axis, which are the coordinate axes of the joint coordinate system, and also designates and inputs the reference point on the designated coordinate axis. Here, the composite axis is generated by the user designating at least two of the X-axis, Y-axis, and Z-axis of the joint coordinate system. That is, the composite axis is generated by vector synthesis of at least two axes designated by the user.

[0042] Also, the reference point is not particularly limited, but indicates the value of a predetermined joint coordinate of the subject. The joint coordinates are, for example, J1 to J21 shown in FIG. 4, and the reference point is the value on the reference axis specified by one of the coordinate values of joint coordinates J1 to J21.

[0043] For example, the user designates the X-axis as the reference axis and designates the X coordinate value of joint coordinate J13 as the reference point on the reference axis, and inputs in step S3.

[0044] In Embodiment 1, an example is shown in which the user sets the reference axis including the reference point using the user interface, but it is not limited thereto. For example, the reference axis including the reference point may be set in the posture display system in advance.

[0045] Next, in steps S4 and S5, the control unit 500_C causes the arithmetic unit 500_O to execute the state extraction program 504. During the execution of this state extraction program 504, the settings (reference axis and reference point) of one or more reference axes stored in the data storage area 500_SA are used.

[0046] <<Steps S4, S5>> By executing the state extraction program 504 in step S4, on the reference axis, the walking state corresponding to the state (first state) of the skeletal coordinates where the coordinates of a predetermined part of the subject are maximum in the first direction is extracted as the first posture from the walking states (the walking states 600_1S to 600_1E of one walking cycle shown in FIG. 5) stored in the data storage area 500_SA. "Maximum in the first direction" means that the distance between the coordinates of the predetermined part and the reference point is maximum in the first direction with respect to the reference axis.

[0047] Next, by executing the state extraction program 504 in step S5, on the reference axis, the walking state corresponding to the state (second state) of the skeletal coordinates where the coordinates of a predetermined part of the subject are maximum in the second direction is extracted as the second posture from the walking states (the walking states 600_1S to 600_1E of one walking cycle shown in FIG. 5) stored in the data storage area 500_SA. Here, the second direction indicates a direction different from the first direction on the reference axis. For example, it may be the opposite direction or the reverse direction. Also, "maximum in the second direction" means that the coordinates of the predetermined part are such that the distance from the reference point is maximum in the second direction of the reference axis.

[0048] By steps S4 and S5, one first posture represented by the skeletal coordinates of the first state and one second posture represented by the skeletal coordinates of the second state are respectively extracted from one walking cycle.

[0049] An example of the operations in steps S4 and S5 will be described below with reference to the drawings. FIG. 6 is a diagram for explaining state extraction according to the first embodiment. Here, FIG. 6(A) is a diagram for explaining the extraction of the first state (first posture) in step S4, and FIG. 6(B) is a diagram for explaining the extraction of the second state (second posture) in step S5. In FIG. 6, in step S3, the X-axis is specified by the user as the reference axis, and the X coordinate of the joint coordinate J13 of the pelvis is specified as the reference point. Also, the joint coordinate J4 connecting the shoulder and the neck is specified by the user as a predetermined part of the subject 600. This predetermined part may be determined in advance, for example, or the user may input it together with the reference axis in step S3.

[0050] In FIG. 6, with reference to the X coordinate of the joint coordinate J13, which is the reference point, the direction in which the X coordinate increases is defined as the first direction DR1, and the direction in which the X coordinate decreases, that is, the opposite direction of the first direction DR1, is defined as the second direction DR2.

[0051] As the subject 600 walks as shown in FIG. 5, the joint coordinate J4 connecting the shoulder and the neck changes, for example, vertically and horizontally. Therefore, time-series skeleton coordinate (skeleton information) data in which the coordinates of the joint coordinate J4 change with walking is stored as the walking state in the data storage area 500_SA.

[0052] By executing step S4, the arithmetic unit 500_O extracts the skeleton information when the joint coordinate J4, which is the predetermined part, is in the first state at the position farthest from the X coordinate of the joint coordinate J13, which is the reference point, in the first direction DR1, from the time-series skeleton information stored in the data storage area 500_SA. As a result, as shown in FIG. 6(A), the first posture 600_S1 of the walking state tilted to the right is extracted.

[0053] Also, by executing step S5 after step S4, the arithmetic circuit 500_O extracts, from the time-series skeletal information stored in the data storage area 500_SA, the skeletal information in the second state where the joint coordinate J4 at a predetermined part is at the position farthest from the X coordinate of the joint coordinate J13 which is the reference point in the second direction DR2. As a result, as shown in FIG. 6(B), the second posture 600_S2 in a walking state tilted to the left is extracted.

[0054] As the first state and the second state (the first posture and the second posture), the case where the distance from the reference point to the predetermined part is maximum has been described, but it is not limited to the maximum. However, in order to extract characteristic movements, the maximum is desirable. With this configuration, even a user who has no experience of observing walking postures can easily extract a state showing a characteristic posture from the time-series data.

[0055] Also, as an example of specifying a predetermined joint coordinate as the reference point, although it has been described, it is not limited thereto, and a predetermined coordinate on the specified reference axis may be used as the reference point. For example, referring to FIG. 6, the X axis may be specified as the reference axis and the point J0 on the X axis may be used as the reference point. Further, the first state and the second state (the first posture and the second posture) may be such that the case where the distance from the reference point to the predetermined part is maximum is the first state, and the case where the distance from the reference point to the predetermined part is minimum is the second state. For example, as shown in FIG. 6, when the coordinates of the reference point J0 are (xn, y0), in step S4, in the first direction DR1, the state where the joint coordinate J4 exists at the position farthest from the X coordinate (xn) of the reference point J0 is defined as the first state, and the first posture 600_S1 corresponding to the first state is extracted. On the other hand, in step S5, in the first direction DR1, the state where the joint coordinate J4 exists at the position closest to the X coordinate (xn) of the reference point J0, that is, the position where the distance is minimum, is defined as the second state, and the second posture 600_S2 corresponding to the second state is extracted.

[0056] Also, instead of the farthest and closest positions, in the first direction DR1, a predetermined first distance from the reference point J0 may be set as a first threshold value, and in the first direction DR1, a distance closer to the reference point J0 than the first distance may be set as a second threshold value. In this case, in step S4, in the first direction, the posture when the distance between the joint coordinate J4 and the reference point J0 is longer than the first threshold value is extracted as the first posture 600_S1, and in step S5, in the first direction, the posture when the distance between the joint coordinate J4 and the reference point J0 is shorter than the second threshold value is extracted as the second posture 600_S2. Furthermore, the second threshold value may be the same value as the first threshold value. Also, the first threshold value and the second threshold value may be input from the user via the computer 100, or may be preset.

[0057] <<Steps S6 to S8>> The positioning of the skeleton is performed in step S6 so that the first posture 600_S1 represented by the skeleton information of the first state extracted by the execution of step S4 and the second posture 600_S2 represented by the skeleton information of the second state extracted by the execution of step S5 overlap. Then, by the execution of step S7, the first posture and the second posture with the skeleton positioned are displayed superimposed. Furthermore, in a subsequent step S8, in the first posture and the second posture that are displayed superimposed, a process of filling the area connecting between predetermined bones with a predetermined color is performed. As a result, the area between the predetermined bones is made explicit by the color between the first posture and the second posture that are displayed superimposed.

[0058] In Embodiment 1, although not particularly limited, steps S6 to S8 are realized by the control unit 500_C causing the arithmetic unit 500_O to execute the posture display program 505. Of course, a part of steps S6 to S8 may be realized, for example, by the execution of the state extraction program 504.

[0059] In addition, the display of the first posture and the second posture in steps S7 and S8 is performed on the display unit of the computer 100 or the screen of the smartphone 300 shown in FIG. 1. Therefore, in steps S7 and S8 and the like, the network communication unit 500_N and the network line 400 and the like are also used. Further, in order to perform the display on the computer 100 or the smartphone 300, the UI program 506 is also executed by the arithmetic unit 500_O in steps S7 and S8.

[0060] An example of the operation in step S6 will be described below. FIG. 7 is a diagram for explaining the positioning of the superimposed skeletons according to the first embodiment. Here, a case where two positionings, that is, the vertical positioning S6-1 (step S6-1) and the horizontal positioning S6-2 (step S6-2), are performed will be described. In FIG. 7, FIG. 7(A) is a diagram for explaining the vertical positioning step S6-1, and FIG. 7(B) is a diagram for explaining the horizontal positioning step S6-2. Here, a case where the vertical positioning step S6-1 is performed first and then the horizontal positioning step S6-2 is performed will be described, but the order is not limited to this.

[0061] The calculation unit 500_O executes the posture display program 505 to, in step S6 (S6-1), among the joint coordinates J18 and J21 of the left and right feet included in the skeleton information in the first state, use the foot with the joint coordinate closer to the floor surface DFL as the pivot foot (in FIG. 7(A), the foot with the joint coordinate J21), and convert the skeleton information in the first state so that the pivot foot (J21) touches the floor surface DFL. For example, the skeleton information in the first state is converted so that the distance between the coordinates of the pivot foot (J21) and the floor surface DFL becomes 0. Similarly, the calculation unit 500_O executes the posture display program 505 to, in step S6 (S6-1), among the joint coordinates J18 and J21 of the left and right feet included in the skeleton information in the second state, use the foot with the joint coordinate closer to the floor surface DFL as the pivot foot (in FIG. 7(A), the foot with the joint coordinate J18), and convert the skeleton information in the second state so that the pivot foot (J18) touches the floor surface DFL. As a result, as shown in FIG. 7(A), with the floor surface DFL as the reference horizontal plane (reference surface), the first posture 600_S1 and the second posture 600_S2 are aligned vertically.

[0062] Next, the calculation unit 500_O executes the posture display program 505 to, in step S6 (S6-2), convert the skeleton information in the first state and / or the skeleton information in the second state so that the difference in distance between the coordinates of the same part in the first posture 600_S1 and the second posture 600_S2 becomes the minimum when they are overlaid later. FIG. 7(B) shows a case where the bone part J12-13 connecting the joint coordinate J12 and the joint coordinate J13 becomes the same part when overlaid. In FIG. 7(B), J12-13_1 indicates the part to be overlaid in the first posture 600_S1, and J12-13_2 indicates the part to be overlaid in the second posture 600_S2. In step S6 (S6-2), the calculation unit 500_O converts the skeleton information in the first state and / or the skeleton information in the second state so that the difference in distance between the coordinates of the part J12-13_1 to be overlaid and the coordinates of J12-13_2 becomes the minimum. As a result, the left and right differences between the first posture 600_S1 and the second posture 600_S2 are minimized and they are aligned.

[0063] By executing step S6, the skeletal information is converted so that the difference between the first posture 600_S1 and the second posture 600_S2 becomes smaller vertically and horizontally.

[0064] Next, in step S7, the arithmetic unit 500_O executes the posture display program 505 and the UI program 506 to display the first posture 600_S1 and the second posture 600_S2 superimposed in step S6 on the display unit of the computer 100 or the screen of the smartphone 300. FIG. 8 is a diagram for showing the superimposed display of the skeleton according to the first embodiment. By executing step S7, as shown in FIG. 8, the first posture 600_S1 based on the skeletal information of the first state and the second posture 600_S2 based on the skeletal information of the second state are superimposed and displayed, for example, on the display unit of the computer 100.

[0065] Subsequent to step S7, in step S8, the arithmetic unit 500_O executes the posture display program 505 and the UI program 506. As a result, in the postures 600_S1 and 600_S2 displayed superimposed in step S7, the region DFA connecting between the specific (predetermined) bone represented by the skeletal information of the first state and the specific (predetermined) bone represented by the skeletal information of the second state is filled with a predetermined color and made explicit. FIG. 9 is a diagram for showing the filling of the region according to the first embodiment. In FIG. 9, the bone connecting the joint coordinates J13 and J1 (J1-J13 is the specific bone. The region DFA connecting the specific bone (J1-J13) of the first posture 600_S1 and the specific bone (J1-J13) of the second posture 600_S2 is filled with a predetermined color (indicated by dots).

[0066] In step S9 after step S8, the operation of the posture display system 1 ends (the process ends).

[0067] When considering the operations shown in FIG. 2 as a posture display method, step S1 can be regarded as a skeleton recognition step of acquiring the skeleton information of the subject in time series. Also, step S3 can be regarded as a reference axis setting step of setting a reference axis and the like. Further, step S4 can be regarded as a first state extraction step of extracting the skeleton information in the first state in which a predetermined part has changed in the first direction from the reference point, and step S5 can be regarded as a second state extraction step of extracting the skeleton information in the second state in which a predetermined part has changed in the second direction from the reference point, and step S7 can be regarded as a display step of superimposing and displaying the skeleton information in the first state and the skeleton information in the second state. Also, step S2 in FIG. 2 can be regarded as a walking cycle extraction step of extracting the walking cycle of the subject, and step S6 can be regarded as an alignment step of aligning the skeleton information in the first state and the skeleton information in the second state. According to this embodiment, it is possible to extract characteristic postures and display the changes in postures in an easy-to-understand manner.

[0068] <Modification Example 1> In step S6, as shown in FIG. 7, an example of aligning the top and bottom and left and right by two steps S6-1 and S6-2 has been described, but it is not limited to this. In Modification Example 1, in the first posture 600_S1 and the second posture 600_S2, a common predetermined part (common predetermined part) is defined, and at the common predetermined part, the first posture and the second posture are aligned. That is, in Modification Example 1, the skeleton information in the first state and / or the skeleton information in the second state is converted so that the difference between the common predetermined part in the first posture 600_S1 and the common predetermined part in the second posture 600_S2 becomes small.

[0069] For example, as the common predetermined part, the joint coordinate J13 may be specified as the reference point. In this case, in step S6, the skeleton information in the first state and / or the skeleton information in the second state is converted so that the difference between the joint coordinate J13 which is the reference point in the first state and the joint coordinate J13 which is the reference point in the second state becomes, for example, the minimum. As a result, in step S7, the first posture 600_S1 and the second posture 600_S2 are displayed so as to overlap at their respective reference points (J13).

[0070] According to Modification 1, since step S6 is constituted by one step, it is possible to shorten the processing time related to the posture display.

[0071] <Modification 2> FIG. 10 is a block diagram showing the configuration of a posture display system according to Modification 2 of Embodiment 1. FIG. 10 is similar to FIG. 1. The main difference is that in FIG. 10, the functions realized by the server 500 in FIG. 1 are realized by the computer 100. That is, the posture display system 1 in FIG. 10 includes a computer 100, a camera 200, a mouse 100_M, and a keyboard 100_K connected to the computer 100.

[0072] Here, the computer 100 includes an arithmetic unit 100_O, a control unit 100_C, a memory 100_MM, a storage unit 100_S, and a bus 100_B, similar to the server 500 shown in FIG. 1. The functions of the arithmetic unit 100_O, the control unit 100_C, the memory 100_MM, the storage unit 100_S, and the bus 100_B are the same as those of the arithmetic unit 500_O, the control unit 500_C, the memory 500_M, the storage unit 500_S, and the bus 500_B shown in FIG. 1.

[0073] Further, the storage unit 100_S includes a data storage area 100_SA, similar to the storage unit 500_S in FIG. 1. In the storage unit 100_S, similar to the storage unit 500_S, a skeleton recognition program 101, a walking cycle extraction program 102, a reference axis setting program 103, a state extraction program 104, a posture display program 105, and a UI program 106 are stored in advance. By having the control unit 100_C execute the skeleton recognition program 101, the walking cycle extraction program 102, the reference axis setting program 103, the state extraction program 104, the posture display program 105, and the UI program 106 by the arithmetic unit 100_O, the same functions as those described in FIG. 1 are realized.

[0074] The camera 200, the mouse 100_M, and the keyboard 100_K are connected to the bus 100_B via the input unit 100_I. Also, the display unit 100_D is connected to the bus 100_B.

[0075] For example, when setting the reference axis, data related to the setting is input into the computer 100 by the keyboard 100_K. Also, the image data of the subject photographed by the camera 200 is stored in the data storage area 100_SA. Images related to the postures obtained by executing the skeleton recognition program 101, the walking cycle extraction program 102, the reference axis setting program 103, the state extraction program 104, and the posture display program 105, for example, the images shown in FIGS. 8 and 9, are displayed on the display unit 100_D when the arithmetic unit 100_O executes the UI program 106.

[0076] Thus, the posture display system 1 can be configured by the computer 100 without using the server 500 and the network line 400.

[0077] <Modification Example 3> FIG. 11 is a diagram showing the relationship between the camera and the subject according to Modification Example 3 of Embodiment 1. FIG. 11 is similar to FIG. 3. The main difference is that in FIG. 11, the camera 210 is fixed to the ceiling surface UFL of the room.

[0078] That is, the camera 210 is fixed to the ceiling surface UFL by the fixing member 211. The subject 600 walks toward the camera 210 as in the case of FIG. 3. Thereby, the subject 600 is photographed by the camera 210, and the image data is stored in the data storage area 500_SA in the server 500.

[0079] As the camera 200 shown in FIG. 3 and the camera 210 shown in FIG. 11, a depth camera or a normal camera (2D camera) can be used. When a depth camera is used as the cameras 200 and 210, data in the Z-axis direction can also be acquired by the camera.

[0080] When using the two-dimensional cameras as cameras 200 and 210, for example, in the server 500, it is possible to generate data in the Z-axis direction by processing the image data stored in the data storage area 500_SA. For example, in the stored image data, based on the joint coordinates J16 to J21 (FIGS. 4 and 5) related to the feet of the subject 600, when the subject 600 raises either the left or right foot, the state is determined as the start of walking 600_1S (FIG. 5), and if the subsequent changes are determined as the walking states 600_2 to 600_1E, it is possible to acquire the data of the Z-axis.

[0081] According to the posture display system 1 according to the first embodiment, characteristic movements during walking are extracted, and the moved range is displayed in color. More specifically, in the posture display system 1 according to the first embodiment, a movement in which the state (from the first state to the second state or from the second state to the first state) changes significantly is extracted as a characteristic movement of the subject, and at that time, the area where the bone has moved is displayed in color. Thereby, for example, it becomes possible to clearly display the change in the posture of the subject before and after training.

[0082] (Second Embodiment) In the first embodiment, an example of photographing the subject from the front (front side) and displaying the posture using the image data obtained by the photographing has been described. In the second embodiment, a case of photographing the subject from the side and using the image data obtained thereby will be described. The image data is stored in the data storage area of the posture display system 1. Here, a case of using the posture display system 1 shown in FIG. 1 will be described. Therefore, the image data will be stored in the data storage area 500_SA shown in FIG. 1. Of course, the posture display system 1 may have the configuration shown in FIG. 10.

[0083] FIG. 12 is a diagram showing the relationship between the camera and the subject according to Embodiment 2. Here, FIG. 12(A) is a view of the camera 200 and the subject 600 seen from the lateral direction, and FIG. 12(B) is a view of the camera 200 and the subject 600 seen from above. In FIG. 12, the subject 600 walks so as to pass through the front surface (lens surface) of the camera 200. That is, the subject 600 walks on the floor surface DFL of the room in the traveling direction indicated by the arrow. The camera 200 is installed on the floor surface DFL by a fixing member 201 such as a tripod, although it is not particularly limited. Also in FIG. 12, 200R indicates the shooting range of the camera 200, the Z-axis indicates the depth direction of the camera 200 (the direction along the optical axis of the lens of the camera 200), the Y-axis indicates the vertical (Y-axis perpendicular to the Z-axis) direction along the lens surface of the camera 200, and the X-axis indicates the left-right (X-axis perpendicular to the Z-axis and the Y-axis) direction also along the lens surface. Although not particularly limited, the center of the lens surface of the camera 200 is taken as the origin (0, 0, 0) of the X-axis, Y-axis, and Z-axis. Of course, the origin of the X-axis, Y-axis, and Z-axis is not limited to this. For example, the origin (0) of the Y-axis may be the floor surface DFL. Note that UFL indicates the ceiling surface of the room.

[0084] The subject 600 walks in the direction (traveling direction) of passing through the front of the lens of the camera 200. Thereby, the camera 200 continuously shoots the subject 600 walking in the traveling direction. A plurality of frames (that is, time-series image data) obtained by continuous shooting are stored in the data storage area 500_SA of the storage unit 500_S.

[0085] The calculation unit 500_O executes the steps shown in FIG. 2 on the image data stored in the data storage area 500_SA. Here, the case where the user designates the X-axis as the reference axis and the joint coordinate J13 as the reference point in step S3 will be described. Also, the case where the joint coordinate J15 of the pelvis is designated by the user as the predetermined part will be described. That is, the pelvis is designated as the part that makes characteristic movements.

[0086] FIG. 13 is a diagram for explaining the positioning of the superposed skeletons according to Embodiment 2. Here, FIG. 13(A) shows a posture in which vertical positioning is performed, and FIG. 13(B) shows a posture in which left-right positioning is performed.

[0087] As shown in FIG. 13(A), in the first posture 600_S1, the value of the X coordinate of the joint coordinate J15, which is a predetermined part, is maximum in the first direction DR1 with respect to the X coordinate value of the joint coordinate J13, which is a reference point. On the other hand, in the second posture 600_S2, the value of the X coordinate of the joint coordinate J15, which is a predetermined part, is maximum in the second direction DR2 with respect to the X coordinate value of the joint coordinate J13, which is a reference point. That is, with respect to the reference point, the first posture 600_S1 when the predetermined part is in the first direction DR1 and the second posture 600_S2 when the predetermined part is in the second direction DR2 opposite to the first direction are extracted, and the skeleton information in the first state and / or the second state is converted so that the foot touches the floor surface DFL.

[0088] Also, in FIG. 13(B), as in FIG. 7(B), a case is shown where the bone part connecting the joint coordinate J12 and the joint coordinate J13 becomes the same part when superposed. The calculation unit 500_O converts the skeleton information in the first state and / or the skeleton information in the second state so that the distance between the bone part connecting the joint coordinates J12 and J13 in the first posture 600_S1 and the bone part connecting the joint coordinates J12 and J13 in the second posture 600_S2 becomes minimum.

[0089] Thereby, it is aligned so that the vertical and horizontal differences are minimized between the first posture 600_S1 and the second posture 600_S2.

[0090] The calculation unit 500_O displays, for example, the first posture 600_S1 and the second posture 600_S2 shown in FIG. 13(B). At this time, in step S8 (FIG. 2), the calculation unit 500_O fills and displays the area connecting the bones. For example, in the first posture 600_S1 and the second posture 600_S2, the area connecting the bones between the joint coordinates J13 and J15 is filled and displayed in a predetermined color. That is, in the first posture 600_S1, the area formed by the bone connecting between the joint coordinates J13 and J15 and, in the second posture 600_S2, the bone connecting between the joint coordinates J13 and J15 will be filled with a predetermined color.

[0091] According to the second embodiment, even when photographing the subject from the side, characteristic movements during walking can be extracted, and the moved range can be clearly indicated in color.

[0092] (Embodiment 3) FIG. 14 is a diagram showing a screen of the posture display system according to the third embodiment. In FIG. 14, reference numeral 100D indicates a screen displayed on the display unit (for example, the display unit 100_D in FIG. 10) of the computer 100 when the UI program 506 shown in FIG. 1 is executed.

[0093] The screen 100D includes a plurality of display areas 100D_1 to 100D_3, and different contents are simultaneously displayed in each display area. In the example shown in FIG. 11, what was described in FIG. 9 is displayed in the display area 100D_1.

[0094] Information regarding the walking of the subject 600 is displayed in a radar chart in the display area 100D_2. The items of the radar chart are the walking speed (speed), stride, vertical movement, rotation, and left - right sway of the subject 600. Also, in the display area 100D_3, the left - right sway of a plurality of subjects 600 is displayed by a bar graph.

[0095] In Embodiment 3, by executing Steps S1 to S8 shown in FIG. 2, the first posture 600_S1 and the second posture 600_S2 are displayed overlapping each other in the display area 100D_1. Further, with postures 600_S1 and 600_S2, an area DFA surrounded by the same bones is filled and indicated in a predetermined color (dots in FIG. 14). Thereby, even an inexperienced user can grasp the characteristic movements of the subject 600.

[0096] Also, in FIG. 14, the displays of the display areas 100D_2 and 100D_3 are performed based on information generated based on, for example, the walking characteristics described in Patent Document 1.

[0097] As described above, according to Embodiment 3, it is possible to provide various information regarding the walking of the subject to the user as a report.

[0098] In FIG. 14, for displaying the first posture 600_S1 and the second posture 600_S2 of the subject 600 in the display area 100D_1, the bones of the subject 600 are shown as lines, but it is not limited thereto. For example, the first posture 600_S1 and the second posture 600_S2 of the subject 600 may be displayed overlapping each other with the bones and joints covered with flesh. Displaying with the bones and joints covered with flesh is not limited to Embodiment 3 only, and the same applies to Embodiments 1 and 2.

[0099] As described above, the invention made by the present inventor has been specifically described based on the embodiments. However, it goes without saying that the present invention is not limited to the above embodiments, particularly walking, and various modifications can be made without departing from the gist thereof.

Explanation of Reference Numerals

[0100] 1 Posture display system 100 Computer 200 Camera 500 Server 500_C Control unit 500_O Arithmetic unit 500_S Memory Unit 500_SA Data Storage Area 501 Skeleton Recognition Program 502 Walking Cycle Extraction Program 503 Reference Axis Setting Program 504 State Extraction Program 505 Posture Display Program 506 UI Program 600 Subject J1 - J21 Joint Coordinates S0 - S9 Steps

Claims

1. A storage unit that stores time-series data of skeletal information, An arithmetic unit, Comprising: The arithmetic unit extracts, from the time-series data, skeletal information in a first state in which a predetermined part in the skeletal information exists in a first direction from a reference point on a reference axis with respect to the predetermined part, and extracts, from the time-series data, skeletal information in a second state in which the predetermined part exists in a second direction different from the first direction from the reference point, and superimposes the skeletal information in the first state and the skeletal information in the second state. A posture display system.

2. In the posture display system according to Claim 1, Further comprising a user interface that receives the setting of the reference axis as an input. A posture display system.

3. In the posture display system according to Claim 1, The first state is a state in which the distance from the reference point to the predetermined part is maximum in the first direction. A posture display system.

4. In the posture display system according to Claim 3, The second state is a state in which the distance from the reference point to the predetermined part is maximum in the second direction. A posture display system.

5. In the posture display system according to Claim 4, The time-series data stored in the storage unit is time-series data of the skeletal information of the subject based on the walking of the subject. A posture display system.

6. In the posture display system according to Claim 5, The posture display system further includes a camera disposed on the front side or the side side of the subject, The walking of the subject is photographed by the camera, and the skeletal information of the subject is acquired. A posture display system.

7. In the posture display system according to Claim 6, The arithmetic unit superimposes and displays the skeletal information in the first state and the skeletal information in the second state. A posture display system.

8. In the posture display system according to Claim 7, The arithmetic unit fills an area connecting a predetermined bone represented by the skeletal information in the first state and a predetermined bone represented by the skeletal information in the second state with a predetermined color. A posture display system.

9. In the posture display system according to Claim 6, The arithmetic unit converts the skeleton information of the first state and the second skeleton information so that the skeleton information of the bearing foot included in the skeleton information of the first state touches the reference plane and the skeleton information of the bearing foot included in the skeleton information of the second state touches the reference plane. The arithmetic unit converts the skeleton information of the first state and the second skeleton information so that the distance between the same parts in the skeleton information of the first state and the second state is minimized. The arithmetic unit overlays and displays the converted skeleton information of the first state and the second state. Posture display system.

10. In the posture display system according to claim 9, The arithmetic unit fills a region connecting a predetermined bone represented by the skeleton information of the first state and a predetermined bone represented by the skeleton information of the second state with a predetermined color. Posture display system.

11. In the posture display system according to claim 1, The arithmetic unit overlays and displays the skeleton information of the first state and the skeleton information of the second state at the reference point. Posture display system.

12. A skeleton recognition step of acquiring time series data of the skeleton information of the subject, A reference axis setting step of setting a reference axis for a predetermined part in the skeleton information, A first state extraction step of extracting, from the time series data, the skeleton information of the first state in which the predetermined part has changed in the first direction with respect to the reference point on the reference axis set in the reference axis setting step, A second state extraction step of extracting, from the time series data, the skeleton information of the second state in which the predetermined part has changed in a second direction different from the first direction with respect to the reference point on the reference axis, A display step of overlaying and displaying the skeleton information of the first state extracted in the first state extraction step and the skeleton information of the second state extracted in the second state extraction step, Comprising Posture display method.

13. In the posture display method according to claim 12, The second direction is a direction opposite to the first direction with respect to the reference point, The first state is the state farthest from the reference point in the first direction, and the second state is the state farthest from the reference point in the second direction. Posture display method.

14. In the posture display method according to claim 13, The method further includes a walking cycle extraction step of extracting the walking cycle of the subject from the time series data of the skeleton information acquired in the skeleton recognition step. The first state extraction step and the second state extraction step extract the skeletal information of the first state and the skeletal information of the second state from the skeletal information of the subject in one walking cycle extracted in the walking cycle extraction step. Posture display method. **Claim 15** In the posture display method according to claim 14, further comprising an alignment step of converting the skeletal information of the first state and / or the skeletal information of the second state so that the skeletal information of the first state and the skeletal information of the second state are aligned at the reference point. Posture display method.

Images