Information processing device, program, and gait motion analysis method

Abstract

To perform walking motion analysis on the basis of a walking image obtained by imaging a walking motion of a pedestrian.SOLUTION: An information processing device 1 includes: walking image reception means 101 for receiving data on a walking image 31 of a pedestrian 4; reference line reception means 102 for receiving a reference line 35 indicating a walking line of the pedestrian in the walking image 31; distance correspondence calculation means 103 for calculating a correspondence between a distance on the walking image 31 and an actual distance; attention point extraction means 104 for extracting an attention point 34 of the pedestrian 4 in the walking image 31; attention point distance calculation means 105 for calculating an actual distance between the attention point 34 and the reference line 35 on the basis of the distance on the walking image 31 between the attention point 34 and the reference line 35; and walking motion analysis means 107 for performing walking motion analysis for the pedestrian 4 on the basis of the calculated actual distance between the attention point 34 and the reference line 35.SELECTED DRAWING: Figure 1

Description

Technical Field

[0005] , , , , , , , , ,

[0004] , ,

[0006] ,

[0001] The present invention relates to an information processing apparatus, a program, a walking motion analysis method, and the like.

Background Art

[0002] Conventionally, the three-dimensional coordinates of a plurality of predetermined body feature points of a pedestrian have been sequentially measured as the pedestrian walks, and the walking of the pedestrian has been analyzed using the positions of the feature points and a pre-stored correspondence function. For example, Patent Document 1 describes this type of technology.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The walking rhythm and walking speed are obtained from the three-dimensional coordinates of the acquired feature points. However, acquisition of the three-dimensional coordinates requires a plurality of dedicated cameras or a special three-dimensional measurement device.

[0005] The present invention has been made to solve the above-described problems, and an object thereof is to provide a technique for performing walking motion analysis based on a walking image in which a walking motion of a pedestrian is imaged.

Means for Solving the Problems

[0006] An information processing device according to one aspect of the present invention includes: a walking image receiving means for receiving data of a pedestrian's walking image; a reference line receiving means for receiving a reference line indicating the pedestrian's walking line in the walking image; a distance correspondence calculation means for calculating the correspondence between distance on the walking image and actual distance; a point of interest extraction means for extracting points of interest in the walking image; a point of interest distance calculation means for calculating the actual distance between a point of interest and a reference line based on the distance on the walking image between the point of interest and the reference line; and a walking motion analysis means for performing walking motion analysis of the pedestrian based on the calculated actual distance between the point of interest and the reference line.

[0007] A walking motion analysis method according to one aspect of the present invention includes: a walking image reception step for receiving data of a walking image of a pedestrian; a reference line reception step for receiving a reference line indicating the walking line of the pedestrian in the walking image; a distance correspondence calculation step for calculating the correspondence between distances on the walking image and actual distances; a point of interest extraction step for extracting points of interest in the appearance of the pedestrian in the walking image; a point of interest distance calculation step for calculating the actual distance between a point of interest and a reference line based on the distance on the walking image between the point of interest and the reference line; and a walking motion analysis step for performing walking motion analysis of the pedestrian based on the calculated actual distance between the point of interest and the reference line.

[0008] Another aspect of the present invention is a computer program that causes a computer to perform the walking motion analysis method described above. [Effects of the Invention]

[0009] According to the present invention, a technique is provided for performing walking motion analysis based on walking images captured of a pedestrian's walking motion. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows the overall configuration of a walking motion analysis system according to one embodiment of the present invention. [Figure 2] This is a block diagram showing the hardware configuration of an information processing device according to one embodiment of the present invention. [Figure 3] This figure shows the functional configuration of an information processing device according to one embodiment of the present invention. [Figure 4] This figure shows an example of a walking image according to one embodiment of the present invention. [Figure 5] This figure shows an example of one cycle of walking displayed on an information processing device according to one embodiment of the present invention. [Figure 6] This figure shows an example of one cycle of walking displayed on an information processing device according to one embodiment of the present invention, and the position of the pedestrian on the screen has been shifted to make the appearance of the pedestrian easier to understand. [Figure 7] This figure shows the lifting distance of the swing leg as an analysis item of the walking motion of a pedestrian according to one embodiment of the present invention. [Figure 8] This figure shows the analysis items for a pedestrian's walking motion according to one embodiment of the present invention, including stride length and knee flexion during the stance phase. [Figure 9] This is a flowchart showing a walking motion analysis method according to one embodiment of the present invention. [Figure 10] This is a radar chart showing an example of the analysis results of a pedestrian's walking motion according to one embodiment of the present invention. [Figure 11] This is a bar graph showing an example of the analysis results of a pedestrian's walking motion according to one embodiment of the present invention. [Figure 12] This is a pie chart showing an example of the analysis results of a pedestrian's walking motion according to one embodiment of the present invention. [Modes for carrying out the invention]

[0011] Hereinafter, an information processing device 1, a program, a walking motion analysis method, and a walking motion analysis system according to embodiments of the present invention will be described with reference to the drawings. In each figure, the same components are denoted by the same reference numerals. In this specification, walking image 31 means either a video, or the still images constituting the video, or both. A video has multiple still images in chronological order. When distinguishing between video and still images in particular, they will be referred to as walking motion image 31M and walking still image 31S.

[0012] Figure 1 shows the overall configuration of a walking motion analysis system according to one embodiment of the present invention. The walking motion analysis system includes an information processing device 1 and an imaging device 2, and analyzes the walking motion of a pedestrian 4. The imaging device 2 is positioned to capture images of the pedestrian 4's walking. For example, the imaging device 2 is a digital video camera. The imaging device 2 captures continuous walking motion images 31M over time. In the example shown in Figure 1, the walking images 31 are displayed on a screen 3. The imaging device 2 has, for example, FHD (Full High Definition) pixels (1920 pixels in the X direction, 1080 pixels in the Y direction). When the imaging device 2 captures images of the pedestrian 4 walking, the electronic data of the captured walking motion images 31M is output to the information processing device 1.

[0013] Figure 2 is a block diagram showing the hardware configuration of the information processing device 1. As shown in Figure 2, the information processing device 1 comprises a control unit 10, an input / output unit 16, a communication means 17, and a storage unit 18. The control unit 10 includes a processor 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, and an input / output interface 15. The information processing device 1 may be a general-purpose personal computer capable of performing various functions by installing various programs, or it may be a computer embedded in dedicated hardware.

[0014] Processor 11 performs various operations and processes. Processor 11 is, for example, a CPU (central processing unit), MPU (micro processing unit), SoC (system on a chip), DSP (digital signal processor), GPU (graphics processing unit), ASIC (application specific integrated circuit), PLD (programmable logic device), or FPGA (field-programmable gate array), etc. Alternatively, Processor 11 is a combination of a plurality of these. Also, Processor 11 may be a combination of these with a hardware accelerator or the like.

[0015] Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. Processor 11 executes various processes according to the program recorded in ROM 12 or the program loaded into RAM 13. A part or all of the program may be incorporated in the circuit of Processor 11.

[0016] Bus 14 is also connected to input / output interface 15. Input / output interface 15 is connected to input / output unit 16 and communication means 17.

[0017] Input / output unit 16 is electrically connected to input / output interface 15 by wire or wirelessly. Input / output unit 16 is composed of, for example, an input unit such as a keyboard and a mouse, and an output unit such as a display for displaying walking image 31 and a speaker for amplifying sound. Note that input / output unit 16 may have a configuration in which the display function and the input function are integrated, such as a touch panel.

[0018] The communication means 17 is a device for the processor 11 to communicate with other devices via a network such as the Internet (not shown). The storage unit 18 is a storage device such as a hard disk drive (HDD) or solid-state drive (SSD) that stores analysis procedures, analysis results, etc.

[0019] The hardware configuration shown in Figure 2 is merely an example and is not limited to this configuration. In addition to being composed of various processing units such as single processors, multiprocessors, and multicore processors, a combination of these various processing units and processing circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field-Programmable Gate Arrays) may be adopted to realize a functional processor configuration. The information processing device 1 may not have a storage unit 18, but rather a configuration in which the storage unit 18 is provided separately. The information processing device 1 may not have a communication means 17, and for example, a configuration that functions in a standalone form may be adopted.

[0020] Furthermore, although the information processing device 1 and the imaging device 2 are separate components in the example shown in Figure 1, the two devices may be formed as an integrated unit. For example, the gait motion analysis system may be implemented as a portable computer, such as a smartphone or tablet terminal, that incorporates the imaging device 2.

[0021] Figure 3 is a block diagram showing the functional configuration of the information processing device 1 according to this embodiment. As shown in Figure 3, the information processing device 1 functionally comprises a walking image receiving unit (means) 101, a reference line receiving unit (means) 102, a distance correspondence calculation unit (means) 103, a point of interest extraction unit (means) 104, a point of interest distance calculation unit (means) 105, a walking image selection unit (means) 106, a walking motion analysis unit (means) 107, a reference line derivation unit (means) 108, an image output unit (means) 109, and a period acquisition unit (means) 110. Each of these functional configurations is realized, for example, by the execution of a computer program stored in memory (ROM 12 or RAM 13) by the processor 11. This computer program may be installed via communication means 17 from a portable recording medium such as a CD (Compact Disc), memory card, or another computer on a network, and stored in memory. From here on, the functions of each functional part in Figure 3 will be explained with reference to Figures 1, 2, and 4 through 8.

[0022] The pedestrian image receiving unit 101 receives data of the pedestrian image 31 of the pedestrian 4. In this embodiment, the pedestrian image receiving unit 101 receives electronic data of the pedestrian motion image 31M captured by the imaging device 2 in Figure 1 through the input / output unit 16. The format of the electronic data received by the pedestrian image receiving unit 101 can be any existing format such as MPEG4, and is not limited in any way. Alternatively, the pedestrian image receiving unit 101 may receive the electronic data of the pedestrian motion image 31M by acquiring the video signal received from the imaging device 2 via the input / output unit 16 and converting the video signal into video format data.

[0023] In the following explanation, images extracted from walking motion images 31M received by the walking image receiving unit 101 will be referred to as walking still images 31S, and will be distinguished from walking motion images 31M. Walking motion images 31M are, for example, moving images consisting of a series of walking still images 31S taken frame by frame (for example, at 1 / 60th of a second). When no particular distinction is made between moving images and still images, they will be referred to as walking images 31.

[0024] Figure 4 shows an example of a walking still image 31S according to one embodiment of the present invention. The distance correspondence calculation unit 103 calculates the correspondence between the distance on the walking image and the actual distance. For example, the distance correspondence calculation unit 103 recognizes a marker 5 that is captured in the walking still image 31S and obtains the length of the marker 5 in the walking still image 31S (distance on the walking image). The length in the walking still image 31S can be estimated, for example, by the number of pixels. Furthermore, the distance correspondence calculation unit 103 obtains the actual length (actual distance) of the marker 5. The actual length of the marker 5 may be stored in the storage unit 18 in advance, or it may be input by an operator operating the information processing device 1.

[0025] For example, suppose the length of marker 5 in the walking still image 31S is estimated to be 182 pixels, and the actual length of marker 5 is 300 mm. In this case, the distance correspondence calculation unit 103 can calculate the actual length per pixel in the walking image 31 as 1.64 mm (= 300 mm / 182) as the correspondence between the distance on the walking image 31 and the actual distance. However, the correspondence between the distance on the walking image calculated by the distance correspondence calculation unit 103 and the actual distance is not limited to the actual length per pixel, but may also be the number of pixels per unit actual distance (e.g., 1 cm), or may be in another format.

[0026] Marker 5 is either something that appears in the walking image itself received by the walking image receiving unit 101, or something that is added to the walking image 31 by image processing, and it is sufficient if it is something that can determine the actual distance (actual length). For example, Marker 5 is part or all of the shape or pattern of an object (it may be a living thing) that appears in the walking image 31, such as a pattern on the floor, a sticker on the floor, the boundary line between the floor and the wall, a box, a desk, a chair, shoes, a cane, a hat worn by the pedestrian 4, etc. In the examples in Figures 1 and 3, Marker 5 with a known actual distance is placed on the floor within the range captured by the imaging device 2.

[0027] Furthermore, the marker 5 may be output by the image output unit 109, described later, in a state where it is superimposed on the walking image 31 and can be moved and its length changed within the walking image in response to operations by the operator. In this case, the distance correspondence calculation unit 103 calculates the correspondence between the distance on the walking image and the actual distance, using the length of the marker 5 superimposed on the walking image 31 as the distance on the walking image, and the actual length of the line segment within the walking image over which the marker 5 overlaps as the actual distance. For example, through operator intervention, marker 5 can be positioned on the walking image along the line of the sole of the pedestrian's foot 4 to match the length of the sole. The length of marker 5 on the image can then be obtained as the distance on the walking image, and the length of the sole of the pedestrian's foot (shoe size) can be obtained as the actual distance of marker 5. As a result, the distance correspondence calculation unit 103 can calculate the correspondence between the distance on the walking image and the actual distance. In this way, the correspondence between distances on walking images and actual distances can be easily and accurately calculated by utilizing the shoe size of the pedestrian 4, without having to go through the trouble of placing markers 5 with known actual distances and taking photographs. However, the use of marker 5 is not limited to this example; the length of the lower leg of pedestrian 4 may be used, or the length of patterns or objects visible in the walking image 31 may be used.

[0028] The reference line receiving unit 102 receives a reference line 35 indicating the walking line of the pedestrian 4 on the walking image received by the walking image receiving unit 101. The walking line is a line that indicates the direction of walking of the pedestrian 4 as shown on the walking image, or the contact surface of one or both feet. The reference line 35 can be one or more lines, and may be a straight line, a broken line with one or more angles, a curved line, or a combination of a straight line and a curved line. However, since gait motion analysis is performed based on the relationship between this reference line 35 and the point of focus 34, which is a part of the pedestrian's body 4, it is preferable that the reference line 35 be determined from a common perspective for each pedestrian (gait image 31) who is the subject of the gait motion analysis. For example, the reference line 35 is set to follow the plane (foot contact surface) (floor surface, ground, etc.) on which the pedestrian 4 walks. The reference line receiving unit 102 may receive one or more reference lines 35 for each walking motion image 31M, or it may receive one or more reference lines 35 for each walking still image 31S that constitutes the walking motion image 31M.

[0029] The reference line receiving unit 102 can receive the reference line 35 in response to an input operation by the operator on the information processing device 1. In this case, the image output unit 109, which will be described later, superimposes the reference line 35 onto the walking image 31 and outputs it to the input / output unit 16, and the operator can move one or both of the endpoints of the reference line 35 superimposed on the walking image 31. The reference line receiving unit 102 then receives information on the reference line 35 whose endpoints have been moved by the operator's operation. This allows the operator to easily manually set a reference line 35 suitable for the gait image 31 while viewing the gait image 31, and then perform gait motion analysis using the set reference line 35, thus enabling accurate gait motion analysis with simple operation. Furthermore, the reference line receiving unit 102 can also receive the reference line 35 that has been automatically derived by the reference line derivation unit 108, which will be described later.

[0030] The point of focus extraction unit 104 extracts points of focus 34 from the walking image received by the walking image reception unit 101. For example, as shown in Figure 4, the point of focus extraction unit 104 can derive skeletal lines 33 from the walking image 31 and extract points of focus 34 from the derived skeletal lines 33. The derivation of skeletal lines 33 can utilize existing techniques for deriving skeletal lines 33 from human images, such as methods for finding the maximum points of an image.

[0031] Here, the focus point 34 is set to any part of the human body in the walking image. The focus point 34 may be set to a predetermined part, or it may be accepted in response to an input operation by the operator on the information processing device 1. In the example in Figure 4, the heel 41 is extracted as the focus point 34. The focus point 34 can be set to one or more parts other than the heel 41, such as the toes, knees, hip joints, pelvis, base of the neck, top of the head, etc., depending on the purpose of analysis. Also, if the focus point 34 is a part that exists on both the left and right sides, such as a part of the legs or a part of the arms, the focus point 34 may be set to only the right or left part, or to both the right and left parts. Points of interest 34 can be extracted for each walking still image 31S that constitutes the walking action image 31M.

[0032] The focus point distance calculation unit 105 calculates the actual distance between the focus point 34 and the reference line 35 based on the correspondence relationship calculated by the distance correspondence calculation unit 103 and the distance on the walking image between the focus point 34 extracted by the focus point extraction unit 104 and the reference line 35 received by the reference line reception unit 102. Since the focus point 34 can be extracted for each walking still image 31S, the actual distance between the focus point 34 and the reference line 35 can also be calculated for each walking still image 31S.

[0033] Furthermore, if the walking image 31 is obtained by capturing the walking motion of the pedestrian 4 from a direction perpendicular to its direction of travel, and the reference line receiving unit 102 extracts a reference line 35 corresponding to the first contact surface of one of the pedestrian 4's feet that is visible in the foreground of the walking image 31, and the point of focus extraction unit 104 extracts the same parts of the left and right feet of the pedestrian 4 as points of focus 34, the point of focus distance calculation unit 105 may calculate the above-mentioned correspondence relationship as follows. If the gait image 31 is captured from a direction perpendicular to the direction of movement of the pedestrian 4, as described above, the position of the contact surface (lowest point of the foot) of the foreground foot and the contact surface (lowest point of the foot) of the background foot will not coincide in the gait image due to image distortion associated with the imaging angle. Therefore, analyzing the foreground leg and the background leg using the distance from a common reference line 35 may reduce the accuracy of the analysis.

[0034] Therefore, the focus point distance calculation unit 105 calculates the actual height of the focus point 34 relative to the first contact surface, which is the contact surface of the foreground foot, based on the distance on the walking image 31 between the focus point 34 of the foreground foot of the pedestrian 4 and the reference line 35. At the same time, it calculates the actual height of the focus point 34 of the far-back foot from the second contact surface in order to resolve the positional misalignment on the walking image 31 between the first contact surface and the second contact surface, which is the contact surface of the far-back foot of the pedestrian 4. Specifically, the reference line receiving unit 102 extracts a second reference line 35 corresponding to the second contact surface of the far foot, along with a first reference line 35 corresponding to the first contact surface of the near foot of the pedestrian 4 in the walking image 31. The point of focus distance calculation unit 105 then calculates the actual height of the point of focus 34 of the near foot from the first contact surface based on the distance on the walking image 31 between the point of focus 34 of the near foot and the first reference line 35, and calculates the actual height of the point of focus 34 of the far foot from the second contact surface based on the distance on the walking image 31 between the point of focus 34 of the far foot and the second reference line 35. Alternatively, the focus point distance calculation unit 105 may calculate the distance between the first reference line 35 and the second reference line 35 (distance between reference lines), correct the position of the focus point 34 of the far foot in the walking image 31 using the distance between reference lines so that the second reference line 35 coincides with the first reference line 35, and then calculate the actual height of the focus point 34 of the far foot from the second contact surface based on the distance on the walking image 31 between the corrected focus point 34 of the far foot and the first reference line 35.

[0035] In this way, by correcting for image distortion associated with the imaging angle and then calculating the point of focus distance using the contact surface of the pedestrian 4's front foot (lowest point of the foot) and the contact surface of the pedestrian 4's back foot (lowest point of the foot) as reference lines 35, it is possible to prevent a decrease in analysis accuracy due to image distortion associated with the imaging angle.

[0036] The walking image selection unit 106 selects a group of walking images representing one walking cycle from the walking action images 31M received by the walking image reception unit 101. For example, the walking image selection unit 106 can identify the first and last walking still images 31S that constitute one cycle of the repeated motion in the walking action image 31M, and select a group of walking images representing one walking cycle by including the walking still images 31S in between them. In this case, the walking image selection unit 106 may also select a group of walking images representing one walking cycle for each walking cycle from the group of walking still images 31S that constitute the walking action image 31M. The walking image selection unit 106 may select a group of walking images for one walking cycle, or it may select a group of walking images for two or more walking cycles.

[0037] Figure 5 shows an example where walking still images 31S, which constitute one cycle of repeated motion, are superimposed and displayed on screen 3. Figure 6 shows an example where the positions of each walking still image 31S are shifted and superimposed so that the appearance of pedestrian 4 is easily discernible. The walking images 31 illustrated in Figures 5 and 6 were obtained by capturing the walking motion of a pedestrian 4 from a direction perpendicular to its direction of travel. In Figure 6, ten still walking images 31S, from 31Sa to 31Sj, are displayed simultaneously as a group of still walking images 31S of pedestrian 4, representing one step cycle. For example, the gait image selection unit 106 selects a gait still image 31S in which the heel 41 of the right leg is at its lowest point as the first gait still image 31Sa, based on the actual distance between the point of focus 34 and the reference line 35 calculated by the point of focus distance calculation unit 105, and then selects a gait still image 31S in which the heel 41 of the right leg reaches its lowest point as the second gait still image 31Sj. In this way, the gait image selection unit 106 can extract the group of gait still images 31S from the first gait still image 31Sa to the second gait still image 31Sj as a group of gait images representing one gait cycle. Furthermore, the walking image selection unit 106 can also select a first walking still image 31Sa and a second walking still image 31Sj in a single-step cycle in response to an input operation by the operator on the information processing device 1.

[0038] The gait motion analysis unit 107 performs gait motion analysis of the pedestrian 4 using the actual distance between the point of focus 34 and the reference line 35 calculated by the point of focus distance calculation unit 105. For example, if the heel 41 is extracted as the point of focus 34, the heel contact state can be analyzed using the actual distance between the heel 41 and the reference line 35. Also, if the toes or hip joint are extracted as the point of focus 34, the toe lift state (toe clearance) or pelvic lift state can be analyzed using the actual distance between the toes or hip joint and the reference line 35. Furthermore, the gait motion analysis unit 107 can distinguish and identify the stance phase gait image group and the swing phase gait image group within the gait image group representing one gait cycle by further using the selection information of the gait image group for each gait cycle by the gait image selection unit 106. For example, the stance phase can be identified as the period from the start of one gait cycle (the lowest point of the heel 41) until just before the toes are displaced upward from the lowest point, and the swing phase can be identified as the period from the end of the stance phase until the next time the heel 41 reaches the lowest point. It is also possible to distinguish and identify the stance phase and the swing phase into early, middle, and late phases, respectively. For this reason, it can also be said that the gait motion analysis unit 107 performs gait motion analysis of the pedestrian 4 using the actual distance between the point of focus 34 and the reference line 35 calculated by the point of focus distance calculation unit 105 with respect to the gait image group representing one gait cycle selected by the gait image selection unit 106. By identifying the swing phase and stance phase in this way, the gait motion analysis unit 107 can analyze the knee flexion state, trunk flexion state, etc., as well as the left stride length and right stride length during the stance phase or swing phase.

[0039] Here, we will specifically explain, using Figure 7, an example of the analysis of the height to which pedestrian 4 lifts its leg as part of the walking motion analysis performed by the walking motion analysis unit 107. Figure 7 is an example of a walking still image 31S output by the information processing device 1, and shows the swing leg lifting distance d2 as an analysis item of pedestrian 4's walking motion. In the example in Figure 7, the point of focus 34 is the heel 41 of the left foot of the pedestrian 4. The point of focus extraction unit 104 extracts the position of the left heel 41 as the point of focus 34. The point of focus distance calculation unit 105 derives the number of pixels between the position of the extracted left heel 41 in the walking image 31 and the reference line 35. Furthermore, the point of focus distance calculation unit 105 calculates the actual distance the heel 41 is elevated based on the fact that 1 pixel is 1.64 mm. For example, if there are 34 pixels, the actual distance between the heel 41 and the ground is calculated to be 55.8 mm (= 1.64 mm × 34).

[0040] Furthermore, the gait motion analysis unit 107 can also acquire the stride length d3, trunk flexion angle θ1, and knee flexion angle θ2 during the stance phase as part of the gait motion analysis, as shown in Figure 8. The focus point extraction unit 104 extracts the right heel 41a and the left heel 41b, the focus point distance calculation unit 105 identifies the distance between them in the gait image as the number of pixels in a direction parallel to the reference line 35, and the actual stride length d3 can be calculated using the correspondence relationship (e.g., actual distance per pixel) calculated by the distance correspondence calculation unit 103 and the distance on the image. The stride length d3 can be calculated for both when the right leg is forward and when the left leg is forward. In addition, the gait motion analysis unit 107 can calculate the trunk flexion angle θ1 during the stance phase and the knee flexion angle θ2 during the stance phase based on the perpendicular L1 to the reference line 35.

[0041] The reference line derivation unit 108 derives the reference line 35 for acquisition by the reference line reception unit 102. For example, the reference line derivation unit 108 may derive the reference line 35 as a straight line connecting the lowest points of predetermined parts of one foot of the pedestrian 4 in the walking image 31. Specifically, for example, the reference line derivation unit 108 can use the first walking still image 31Sa and the second walking still image 31Sj selected by the walking image selection unit 106 to derive the reference line 35 as a straight line connecting the point of interest 34a in the first walking still image 31Sa and the point of interest 34b in the second walking still image 31Sj. In the example of Figures 5 and 6, since the points of interest 34a and 34b are set to the heels 41 of the pedestrian 4, the reference line 35 is a straight line connecting the positions of the heels 41 that are in contact with the ground, and can be set to follow the plane on which the pedestrian 4 walks. This can be rephrased as follows: if the walking image 31 is obtained by capturing the walking motion of the pedestrian 4 from a direction perpendicular to its direction of travel, the reference line derivation unit 108 (reference line reception unit 102) derives (extracts) a reference line 35 corresponding to the first contact surface of one of the pedestrian 4's feet (the right foot in Figures 5 and 6) that is visible in the foreground of the walking image 31.

[0042] However, the method for deriving the reference line 35 is not limited to these examples. For example, the reference line 35 may be derived by connecting the points of interest 34 that mark the boundaries of each gait cycle with straight or curved lines, using a group of gait images of two or more gait cycles. Alternatively, a straight line passing through the vicinity of multiple points of interest 34 that mark the boundaries of each gait cycle may be derived using mathematical methods such as the least squares method, and this may be used as the reference line 35. By targeting points of interest 34 of multiple gait cycles in this way, the accuracy of the reference line 35 can be improved. Furthermore, the reference line derivation unit 108 may independently set points of interest 34 for deriving the reference line 35, separate from the points of interest 34 extracted by the points of interest extraction unit 104. Furthermore, by using a walking image 31 captured of a pedestrian 4, who is to be analyzed, walking on a straight line mark (sticker) that has been previously attached to or drawn on the walking surface, the reference line derivation unit 108 can recognize the straight line mark from the walking image 31 using existing image recognition technology and derive the reference line 35 from that recognition information.

[0043] The image output unit 109 outputs (displays) the walking motion image 31M or the group of still images of pedestrians 32S extracted therefrom, received by the walking image reception unit 101, to the input / output unit 16. At this time, the image output unit 109 can output the walking image 31 and the reference line 35 in a state where the reference line 35 is superimposed on the walking image 31, and one or both of the endpoints of the reference line 35 can be selected and moved according to the operator's operation. However, points at positions other than the endpoints of the reference line 35 may also be selected and moved.

[0044] Furthermore, the image output unit 109 may output (display) the marker 5 to the input / output unit 16 in a state where it is superimposed on the walking image 31 and is movable and its length can be changed within the walking image 31 in response to operations by the operator.

[0045] Furthermore, the image output unit 109 can also output (display) the analysis results from the gait motion analysis unit 107 to the input / output unit 16. The output format of such analysis results will be described later. Furthermore, the image output unit 109 may output walking images 31 in such a way that the start and end points of one walking cycle can be identified by the operator for the walking motion images 31M or the group of still walking images 31S. As a result, as described above, the walking image selection unit 106 can select the first still walking image 31Sa and the second still walking image 31Sj in one walking cycle in response to the input operation of the information processing device 1 by the operator.

[0046] The period acquisition unit 110 acquires the time of one walking cycle. The period acquisition unit 110 counts the number of frames between a pair of walking still images 31S that constitute one walking cycle. The time elapsed between this pair of walking still images 31S is determined from the frame frequency and the number of frames. When the frame frequency is 120 Hz, 120 frames are acquired as walking still images 31S per second. For example, if there are 282 walking still images 31S between a pair of walking still images 31S, the elapsed time is calculated by the period acquisition unit 110 to be 2.35 seconds (= 282 / 120). However, the method for obtaining the time of one gait cycle is not limited to these examples. The cycle acquisition unit 110 can also obtain the time of one gait cycle by determining the difference in timestamps of a pair of still gait images 31S at the start and end of one gait cycle.

[0047] Figure 9 is a flowchart showing a walking motion analysis method according to one embodiment of the present invention (hereinafter sometimes referred to as "this method"). This method is executed by a computer such as the information processing device 1 described above. Each step shown in Figure 9 is executed by the respective functional configurations of the information processing device 1, and since the execution content of each step is the same as the processing content of those functional configurations described above, the details of each step are omitted as appropriate.

[0048] First, the imaging device 2 performs an imaging process (step S11) to capture the walking of pedestrian 4. Next, the walking image receiving unit 101 of the information processing device 1 performs a walking image receiving process to receive the captured walking image 31 (step S12). Next, the reference line receiving unit 102 performs a reference line receiving process to receive a reference line 35 that indicates the walking line of pedestrian 4 on the walking image 31 (step S13). Next, the distance correspondence calculation unit 103 performs a distance correspondence calculation process to calculate the correspondence between the distance on the walking image 31 and the actual distance (step S14). Next, the point of interest extraction unit 104 performs a point of interest extraction process to extract the point of interest 34 of pedestrian 4 in the walking image 31 (step S15).

[0049] If not all of the points of interest 34 have been extracted (Step S16: No), the process returns to the point of interest extraction process. If all of the points of interest 34 have been extracted (Step S16: Yes), the process proceeds to the next step. The point of interest distance calculation unit 105 performs a point of interest distance calculation process to calculate the actual distance between the point of interest 34 and the reference line 35 based on the distance between the point of interest 34 and the reference line 35 on the walking image 31 (Step S17). Next, the walking image selection unit 106 selects a group of walking images representing one gait cycle from the walking action images 31M received in Step S12 (Step S18). In Step S18, for example, a pair of still walking images 31S in which the heel 41 of the right leg, extracted as the point of interest 34, is located at the lowest point is selected as a pair of still walking images 31S representing the start and end points of one gait cycle. The gait motion analysis unit 107 performs a gait motion analysis of the pedestrian 4 using the actual distance between the point of interest 34 and the reference line 35 calculated for a group of gait images representing one gait cycle (step S19). The process ends when the gait motion analysis process is completed (step END).

[0050] In the flowchart illustrated in Figure 9, multiple steps (processes) are listed in order, but the execution order of the steps performed by this method is not limited to the order in which they are listed. In this method, the order of the illustrated steps can be changed as long as it does not impair the content. Furthermore, some of the steps illustrated in Figure 9 may be omitted. For example, in this method, the gait image selection process in step S18 may be omitted. In that case, regardless of the gait cycle, the gait motion analysis process can be performed using the actual distance between the point of focus 34 and the reference line 35 calculated in the point of focus distance calculation process in step S17 (step S19).

[0051] The information processing device 1 can output the results of the gait motion analysis unit 107, such as the displays and printed materials exemplified in Figures 10, 11, and 12, to a monitor or printer. Furthermore, this method may include a step of outputting the displays and printed materials exemplified in Figures 10, 11, and 12. However, the output format of the information processing device 1 and this method is not limited to the examples shown in Figures 10, 11, and 12.

[0052] Figure 10 is a radar chart showing an example of the analysis results of pedestrian 4's walking motion. In the example shown in Figure 10, the analysis results from the gait motion analysis unit 107 include whether the heel 41 makes contact with the ground, the knee flexion angle θ2 during the stance phase, the hip extension angle, the push-off state of the leg from mid-stance onward, the toe clearance during the swing phase, the knee flexion angle θ2 during the swing phase, the trunk flexion angle θ1, and the pelvic lift state, and radar charts of these are output.

[0053] Figure 11 is a bar graph showing an example of the analysis results of pedestrian 4's walking motion. In the example shown in Figure 11, the analysis results from the gait motion analysis unit 107 include walking speed, left stride length, right stride length, and steps per minute (cadence), and bar graphs of these values ​​are output.

[0054] Figure 12 is a pie chart showing an example of the analysis results of pedestrian 4's walking motion. In the example shown in Figure 12, the analysis results from the gait motion analysis unit 107 include the percentages of time spent with both feet supporting, time spent with the right foot supporting, time spent with the left foot supporting, and time spent in the swing phase during one step. These are then separated into pie charts for the right and left legs.

[0055] This disclosure allows for various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims, not by the embodiments. Various modifications made within the scope of the claims and the equivalent significance of the disclosure are considered to be within the scope of the present invention.

[0056] <Variations> For example, the above-described embodiment can be modified as follows. The gait image receiving unit 101 may receive data from a first gait image obtained by capturing the gait of a pedestrian 4 from a direction perpendicular to its direction of travel, and data from a second gait image obtained by capturing the gait of a pedestrian 4 from directly in front of the pedestrian 4. In this case, the first gait image and the second gait image may be captured simultaneously by two imaging devices 2, or they may be captured separately. In the latter case, it is preferable that each gait is performed at a similar walking speed.

[0057] The gait motion analysis unit 107 identifies an arbitrary timing within one gait cycle using the actual distance between the point of focus 34 and the reference line 35 calculated by the point of focus distance calculation unit 105 for the first gait image. For example, if the knee position is extracted as the point of focus 34, the timing when the actual distance between the point of focus 34 and the reference line 35 is greatest can be said to be the timing when the leg is raised highest during the swing phase. One gait cycle may be identified using a group of still gait images representing one gait cycle selected by the gait image selection unit 106, or it may be identified using the time of one gait cycle acquired by the cycle acquisition unit 110.

[0058] The gait motion analysis unit 107 can perform gait motion analysis of the pedestrian 4 based on the time from the start of one gait cycle to the specified timing, which can be obtained using the first gait image, and based on the position of the point of interest 34 in the second gait image at the specified timing. For example, the degree of knee opening can be analyzed based on the position of the pedestrian 4's knee as viewed from the front of the pedestrian 4 at the moment when the leg is raised highest during the swing phase.

[0059] Synchronization of one gait cycle in the first gait image and the second gait image may be performed using a group of still gait images representing one gait cycle selected by the gait image selection unit 106 for each gait image, or it may be performed by matching the acquisition timing of the first gait image and the second gait image.

[0060] In this way, of the two gait images 31 captured from mutually different directions, an arbitrary timing within one gait cycle (the timing to be analyzed) is identified from one gait image 31, and gait motion analysis is performed using the position of the point of interest 34 at that identified timing in the other gait image 31. This makes it possible to perform a more detailed gait motion analysis, such as identifying gait movements that cannot be identified from one gait image 31 alone using the other gait image 31. [Explanation of Symbols]

[0061] 1. Information Processing Device 2. Imaging device 3 screens 4 Pedestrians 5. Objects whose actual distance is known 10 Control Unit 11 processors 12 ROM 13 RAM 14 bus 15 Input / Output Interfaces 16 Input / output section 17. Communications Department (Means) 18 Memory section 31 Walking images 31M walking behavior image 31S Walking still image 33 Skeletal lines 34 points of focus 35. Reference Line 41 Heel 101 Walking image receiving unit (means) 102 Reference line receiving section (means) 103 Distance correspondence calculation unit (means) 104 Point of Interest Extraction Unit (Means) 105 Point of Interest Distance Calculation Unit (Means) 106 Walking image selection unit (means) 107 Walking motion analysis unit (means) 108 Reference line derivation unit (means) 109 Image output unit (means) 110 Period acquisition unit (means) d1 Reference length in gait images d2 Lifting distance of the swing leg d3 stride length L1 Perpendicular to the reference line

Claims

1. A pedestrian image receiving means that receives data of a pedestrian's walking image to be processed as a two-dimensional image, A reference line receiving means for receiving a reference line indicating the walking line of the pedestrian in the aforementioned walking image, After capturing the aforementioned walking image, an image output means generates a marker that is superimposed on the walking image, is movable and has its length changed by the operator's operation within the walking image, and outputs the marker to an output unit. A distance correspondence calculation means calculates the correspondence between the distance on the walking image and the actual distance based on the length of the marker on the walking image and the actual length of the line segment in the walking image that overlaps with the marker. A point of interest extraction means for extracting points of interest from the aforementioned walking image, A point of focus distance calculation means calculates the actual distance between the point of focus and the reference line based on the distance between the point of focus and the reference line on the walking image and the corresponding relationship, A walking motion analysis means for performing walking motion analysis of the pedestrian using the actual distance between the calculated point of interest and the reference line, An information processing device having

2. Image output means that outputs the aforementioned reference line superimposed on the walking image to the output unit. Furthermore, The image output means outputs one or both of the endpoints of the reference line so that they can be moved in response to an operation by the operator. The reference line receiving means receives information about the reference line in which one or both of the endpoints have been moved by an operation performed by an operator. The information processing apparatus according to claim 1.

3. The aforementioned walking image was obtained by capturing the walking motion of the pedestrian from a direction perpendicular to the direction of the walking motion. The reference line receiving means extracts the reference line corresponding to the first contact surface, which is the contact surface of one of the pedestrian's feet that is visible in the foreground of the walking image. The point of focus extraction means extracts the same part of the left and right feet of the pedestrian captured in the walking image as the point of focus. The point of focus distance calculation means calculates the actual height of the point of focus of one foot from the first contact surface based on the distance on the walking image between the point of focus of one foot and the reference line, while also calculating the actual height of the point of focus of the other foot from the second contact surface in order to eliminate the positional misalignment on the walking image between the first contact surface and the second contact surface, which is the contact surface of the other foot of the pedestrian that is visible in the background of the walking image. The information processing apparatus according to claim 1 or 2.

4. A reference line derivation means for deriving a reference line as a straight line connecting the lowest points of predetermined parts of one foot of the pedestrian in the walking image, An information processing apparatus according to any one of claims 1 to 3, further comprising:

5. A walking image selection means that selects a group of still walking images representing one walking cycle from the aforementioned walking images. Furthermore, The walking image receiving means receives data of a first walking image obtained by capturing the walking motion of the pedestrian from a direction perpendicular to the direction of the walking motion, and data of a second walking image obtained by capturing the walking motion of the pedestrian from directly in front of the pedestrian. The walking motion analysis means identifies an arbitrary timing within the walking cycle using the actual distance between the point of interest and the reference line calculated with respect to the first walking image, and performs walking motion analysis of the pedestrian based on the position of the point of interest in the second walking image at the identified timing. An information processing apparatus according to any one of claims 1 to 4.

6. A pedestrian image receiving step that receives data of a pedestrian's walking image to be processed as a two-dimensional image, A reference line reception step that receives a reference line indicating the walking line of the pedestrian in the walking image, After capturing the walking image, an image output step is performed to generate a marker that is superimposed on the walking image, is movable and has its length changed by the operator's operation within the walking image, and outputs the marker to the output unit. A distance correspondence calculation step that calculates the correspondence between the distance on the walking image and the actual distance based on the length of the marker on the walking image and the actual length of the line segment in the walking image that overlaps the marker, A point of interest extraction step that extracts points of interest from walking images, A step to calculate the distance between the point of interest and the reference line, based on the distance between the point of interest and the reference line on the walking image and the correspondence relationship, A walking motion analysis step in which the walking motion of the pedestrian is analyzed based on the actual distance between the calculated point of interest and the reference line, A method for analyzing gait motion, including the analysis of gait.

7. A computer program that causes a computer to execute the walking motion analysis method described in claim 6.

Images