Human gait analysis system and method

Abstract

A computer-implemented method, computer program product, and computer system (100) for human gait analysis. A video stream (202) is obtained from a monocular camera device (201) via an interface (110) of the system. The video stream (202) includes multiple frames reflecting the walking of a human individual (10). A skeletal motion extractor (120) of the system extracts skeletal motion associated with the individual from the video stream by inferring three-dimensional gait information from the obtained video stream. The three-dimensional gait information includes an estimate of the joint position of the individual, the joint position including at least the position of the individual's foot on each frame, wherein the estimate is derived by matching the two-dimensional coordinates of the corresponding frame with the corresponding three-dimensional model information of the individual's body for each frame. A gait parameter determination module (130) of the system determines one or more gait parameters of the individual based on the individual's foot position in a local extremum frame, the local extremum frame showing the local extremum of the distance between one foot position of the individual and the corresponding reference joint position.

Description

Technical Field

[0001] The present invention relates generally to electronic data processing and, more particularly, to methods, computer program products and systems for gait analysis that can be used to assess the risk of falls (risk of falling accidents) in human individuals. Background Art

[0002] As many elderly people age, the risk of falling poses an increasing threat to their health. Typically, medically trained personnel try to assess the risk of falling in individual patients based on risk factors, such as factors related to medications, the patient's psychological condition and living environment. The Downton Falls Risk Index, the STRATIFY Falls Risk Assessment Tool and the Hendrich Falls Risk Models I and II are well-known prior art tools for assessing the risk of falling. In such tools, first, the risk model contains a series of risk factors. For a specific patient, it is necessary to check which risk factors apply. Finally, the patient's overall risk score is the (weighted) sum of the risk factors present. If the risk score is greater than a certain critical value, it is concluded that the corresponding patient has a high risk of falling.

[0003] In addition, sometimes medically trained personnel observe the patient's gait and the subjective assessment of the gait is also taken into account when assessing the risk of falls. Various (weighted) factors should support medically trained personnel in taking appropriate fall prevention measures. However, subjective gait parameter observation carries the risk that the observed gait parameters lack accuracy and can lead to incorrect therapeutic measures. There are tools that can determine gait parameters with a high degree of accuracy. Such tools are, for example, so-called gait carpets or systems based on multiple cameras. However, the setup of such tools is usually technically complex. Summary of the invention

[0004] Therefore, there is a need to provide a system and method that allows medically trained personnel to objectively assess a patient's gait parameters relevant to fall risk analysis using equipment that is less complex to set up than the above-mentioned tools.

[0005] In order to solve the above technical problem, according to the independent claims, a computer implemented method, a computer system and a computer program product for human gait analysis are provided.

[0006] In one embodiment, a computer-implemented method for human gait based on a video stream obtained from a monocular camera device is provided. As used herein, a "video stream" refers to a sequence of frames obtained by a video camera. Therefore, it does not matter whether the sequence of frames is directly processed in real time (stream processing) or the sequence of frames is stored in a storage device and then processed offline. The computer-implemented method can be performed by a corresponding computer system. The computer system can execute a computer program that implements a module. The module is configured to perform corresponding method steps when running on a computer system. Therefore, the monocular camera device can be any standard digital camera (e.g., a camera in a smart phone or tablet computer, a webcam, etc.). For the camera device, there are no specific technical requirements that exceed the standard capabilities of such an integrated camera device. In particular, stereo camera technology is not required. The monocular camera device is used to obtain video from a human individual (e.g., a patient) when the individual walks. In other words, the obtained video stream includes multiple frames reflecting the walking of the human individual.

[0007] A variety of methods can be used by those skilled in the art to infer three-dimensional gait information, in particular the joint positions of a person, from images or videos of a monocular camera. For example, the estimated two-dimensional coordinates can be combined with a position map derived (obtained) by using a convolutional neural network (VNect method). This example is described in more detail below. In another implementation, a three-dimensional surface model of the human body is used. The parameters of the model are fitted to the estimated two-dimensional coordinates. This implementation is used in the SMPLify method (see Federica Bogo et al. "Keeping SMPL: Automatically Estimating 3D Human Pose and Shape from a Single Image", available at http: / / files.is.tue.mpg.de / black / papers / BogoECCV2016.pdf).

[0008] Next, the VNect method is described in more detail as an optional embodiment to illustrate how to extract three-dimensional gait information from the obtained video stream.

[0009] In the first step, a convolutional neural network (CNN) can be used to infer heat maps and position maps to estimate the joint positions of individual joints. This calculation can be performed in real time. The term "real time" used throughout the document is understood by technicians in the field of digital data processing as "near real time" or "quasi real time". Any digital data processing step (e.g., image processing step) applied to data captured from the real world (e.g., sensor data will be converted into a visual representation of such sensor data) always results in a delay imposed by the digital data processing step. For example, there is always a computational delay between individual movement in the real world and joint position determination in a video frame. For convenience, when referring to this "near real-time" behavior of the disclosed method and system, only the term "real time" is used throughout the document. CNN is trained on a three-dimensional human posture data set. A specific heat map describes the probability that a specific joint of an individual is associated with each pixel of the corresponding frame for a corresponding frame. A heat map is a two-dimensional object. A specific position atlas includes multiple position maps. Each position map describes the distance from a specific joint to the root position of a corresponding frame in a corresponding spatial dimension. That is, each position map is a two-dimensional map for a corresponding spatial dimension. A specific set of position atlases for three orthogonal spatial dimensions can be interpreted as a three-dimensional position map.

[0010] The publication by DUSHYANT MEHTA et al., “VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera”, ACM Transactions on Graphics, Vol. 36, No. 4, July 2017, Article 44, details an implementation example of a CNN based on the ResNet50 network architecture from “Deep Residual Learning for Image Recognition” by He et al. at the 2016 IEEE Conference on Computer Vision and Pattern Recognition. The ResNet50 layers starting with res5a are replaced by the VNect paper Figure 5 The architecture shown is replaced by the one shown in Figure 1, which produces heatmaps and position maps for all joints. After training, the batch normalization layer is merged with the weights of the convolutional layer before it to increase the speed of the forward pass. The 2D heatmap and 3D position map are predicted based on the features at res4d and res5a, and the weight of the intermediate loss gradually decreases as the number of iterations increases. Those skilled in the art will also be able to use other CNNs in the ResNet family as the basis for individual joint position estimation.

[0011] In the VNect disclosure below, a default skeleton is used for kinematic skeleton fitting to ultimately capture the skeletal motion of an individual. However, using the default skeleton may result in significant errors when attempting to fit the default skeleton to a video frame showing an individual whose skeleton exhibits significant deviations from the default skeleton. Therefore, before performing kinematic skeleton fitting as described in Section 4.2 of the VNect paper, the claimed computer-implemented method evaluates an individual skeletal model that reflects the individual's actual proportions.

[0012] This is accomplished by first receiving a selection of at least one frame sequence of a video stream during which a joint of the individual moves smoothly over time. For example, the selection may be received from a medically trained individual based on the individual's experience. Alternatively, image processing techniques may be used to automatically identify a sequence of frames in which the movement of a particular joint of the individual from one frame to the next remains within predetermined limit characteristics of smooth motion. Typically, a sequence of frames that includes the beginning of a person's walking (i.e., typically an initial sequence of frames) is not well suited for evaluating a skeletal model of the individual.

[0013] The skeleton model of the individual is then estimated by determining the loss of each joint of the default skeleton model in each spatial coordinate for each frame of the selected sequence and by adjusting the default skeleton model to compensate for the determined loss to provide an adjusted skeleton model. In other words, for each frame of the selected sequence, an error is measured. The error describes the deviation of the joint positions associated with the default skeleton model from the joint positions determined by the CNN when trying to fit the default skeleton to the determined joint positions. The default skeleton is then compensated by the error measured over the entire selected frame sequence, which will make the adjusted skeleton model reflect the true proportions of the individual.

[0014] Then, the following kinematic skeleton fitting is performed using the adjusted skeleton model to determine a plurality of joint positions, the joint positions including at least the foot positions of the individual's feet on each frame. By using the adjusted skeleton model, the accuracy of determining the joint positions can be improved.

[0015] The system then determines one or more gait parameters of the individual based on the individual's foot position to be used as input for a fall risk assessment. To this end, the computer system identifies a local extremum frame. The local extremum frame shows the local extrema of the distance between the position of one of the individual's feet and the corresponding reference joint position. When the individual walks steadily, the distance from the individual's foot to the reference joint position changes periodically, thereby showing local maxima and local minima between two consecutive contact points of each foot with the ground. Typically, the reference joint position is the position of another joint of the individual. The position of the other joint relative to the foot provides a reference position within the individual's body.

[0016] For example, if the reference position is the second foot of the individual, the local maximum of the distance is reached each time the foot of the individual reaches the step length. The local minimum is reached when the second foot passes the position of the first foot. When the point on the walking floor with the shortest distance to the corresponding foot position is used as the reference point, the distance also changes periodically during steady walking because the distance is zero when the corresponding foot contacts the walking floor and reaches the local maximum between two consecutive contact point positions of the same foot.

[0017] In a real-world scenario, it may happen that multiple frames associated with a sequence of frames in which both feet of an individual are in contact with the ground show multiple local minima of the distance. In this case, the system can determine a cluster of honest local extrema frames (honest maximum frames and / or honest minimum frames), where the frames of the cluster contribute to the calculation of the step length because the average distance value is calculated based on all frames of the corresponding cluster. In other words, the honest local extrema frames include such frames that collectively reflect the distance between the feet of the individual for a particular step of the individual. The average step length can then be determined by averaging the individual values ​​for multiple honest local extrema (e.g., honest local maximum clusters).

[0018] It may happen that, for a particular heat map, multiple pixels are associated with the same maximum probability value for the position of the corresponding joint. In this case, by the following embodiment, the best pixel will be used to estimate the joint position. In the prediction step, the computer system first identifies multiple heat map pixels in a particular heat map, wherein each identified pixel is associated with the same maximum probability value for the corresponding joint associated with the particular heat map. Then, for each identified pixel, the heat map value in a predetermined area adjacent to the identified pixel is integrated. For example, the adjacent area can be defined by a circle centered on the corresponding pixel. The system thereby calculates the integral of all probability values ​​in the adjacent area. Finally, the system selects the identified pixel with the highest integral value as the pixel associated with the corresponding joint position. In this embodiment, the possibility of selecting a frame pixel corresponding to the real-world position of a particular joint is increased. As a result, the accuracy of the prediction step is improved.

[0019] An alternative embodiment based on the cited SMPLify method uses an estimate of the two-dimensional joint coordinates as input. This estimate can be obtained, for example, using methods such as VNect or DeepCut or a convolutional pose machine or any other method that provides two-dimensional joint coordinates for this purpose. The SMPLify implementation then fits a three-dimensional human model (a skinned multi-person linear model (SMPL)) to the two-dimensional joints. The model can be robustly fitted to very little data. In addition, the three-dimensional model prevents three-dimensional poses that can fit two-dimensional joints but cause interpenetration. For more details, the technician can refer to the SMPLify publication cited above.

[0020] In many medical studies, three gait parameters have been determined to be very useful in assessing a patient's risk of falling: the patient's step length, walking speed, and step height. Various implementations of the step "determining one or more gait parameters" allow for the calculation of each such parameter to be provided to medically trained personnel.

[0021] In one implementation, the step length gait parameters are calculated by using the second foot of the individual (patient) in the corresponding local maximum frame as the corresponding reference position. The system then calculates the step length gait parameters as the average (e.g., mean or median) of the distances between the individual's foot positions in the determined local maximum frame. For example, the distances between the individual's foot positions can be directly measured in the corresponding frame based on the corresponding three-dimensional coordinates. This measurement is advantageous when the patient walks past the camera and the frame picture shows a view of the patient from the side. The distance from any joint to the camera can be determined, for example, by using a pinhole projection model known to those skilled in the art.

[0022] In a similar implementation, a computer system can determine a walking speed gait parameter. In this implementation, the walking distance of an individual is calculated based on the determined step length and the number of local maximum frames identified in the video stream. The walking distance is then divided by the time interval associated with the frame sequence associated with the walking distance to obtain the walking speed. Other implementations for determining walking distance are disclosed in the detailed description.

[0023] In one implementation, the step height gait parameter is calculated by using the position of the head (or pelvis) of the adjusted skeletal model as the corresponding reference position. In this implementation, the system identifies each local minimum frame in which the first distance between the foot position and the position of the reference point is the shortest. In the same local minimum frame, a second distance between the reference point and the second foot of the individual (the second foot in contact with the ground) can be calculated. The step height can then be calculated as the average of the differences between the first distance and the second distance in each local minimum frame. Alternatively, the system can directly calculate the distance between the feet of the individual in each local minimum frame.

[0024] In an alternative implementation, the system can determine the step height gait parameter by extrapolating the walking floor plane from the floor contact points of the individual's feet. Such floor contact points can be derived from the local maximum frames used to determine the step length. Typically, when the patient walks steadily, both feet will touch the ground when the distance between the two feet reaches a local maximum. This is especially true for the elderly. Based on multiple contact points during the patient's walking process, the walking floor plane can be inferred. Once the walking floor plane is known, the system can determine additional local maximum frames. These additional local maximum frames have additional local maxima of the distance of the individual's foot position from the walking floor plane. Therefore, it means the shortest distance between the foot position and the floor plane. The system then calculates the average of the additional local maximum distances as the step height.

[0025] In one embodiment, previously determined gait parameters may be associated with corresponding scores characterizing an individual's risk of falling. Defining a score may be advantageous. The score may be easily integrated into prior art scoring systems for fall risk assessment used in the healthcare industry. The determined gait parameter scores may then be provided to medically trained personnel as a result of the gait analysis to support decision making for effective treatment of the patient, thereby reducing the patient's risk of falling.

[0026] In one embodiment, a computer program product is provided that includes instructions that, when loaded into a memory of a computing device and executed by at least one processor of the computing device, perform the method steps of the computer-implemented method disclosed herein.

[0027] In one embodiment, a computer system for gait analysis is provided. The computer system is configured to run a computer program product. The computer program product allows the steps of the disclosed computer-implemented method to be performed. To this end, the computer system implements the following modules through the computer program product to perform various gait analysis functions:

[0028] An interface configured to obtain a video stream from a monocular camera device, the video stream comprising a plurality of frames reflecting walking of a human individual;

[0029] a skeletal motion extractor configured to extract skeletal motion associated with an individual from a video stream by inferring three-dimensional gait information from the obtained video stream, wherein the three-dimensional gait information includes an estimate of joint positions of the individual, the joint positions including at least foot positions of the individual on each frame, the estimate being derived by matching two-dimensional coordinates of the corresponding frame with corresponding three-dimensional model information of the individual's body for each frame; and

[0030] A gait parameter determination module is configured to determine one or more gait parameters of an individual based on the individual's foot position in a local extremum frame, the local extremum frame showing a local extremum of a distance between a foot position of the individual and a corresponding reference position.

[0031] In an optional embodiment, the extractor module further comprises:

[0032] A prediction module trained on a three-dimensional human pose dataset using a convolutional neural network is configured to predict a heat map and a position map for real-time joint position estimation, wherein a specific heat map describes, for a corresponding frame, a probability that a specific joint is associated with each pixel of the corresponding frame, wherein a specific position map set includes a plurality of position maps, wherein each position map describes a distance from a specific joint to a root position of the corresponding frame in a corresponding spatial dimension;

[0033] a skeletal model estimator module configured to select at least one frame sequence of a video stream during which joints of the individual move smoothly over time, and configured to estimate a skeletal model of the individual by determining a loss for each joint of a default skeletal model in each spatial coordinate for each frame of the selected sequence, and configured to adjust the default skeletal model to compensate for the determined loss to provide an adjusted skeletal model;

[0034] The skeleton fitting module is configured to perform kinematic skeleton fitting for each video frame using the adjusted skeleton model to determine a plurality of joint positions, the joint positions including at least the position of the individual's feet in each frame.

[0035] In one embodiment, the prediction module is further configured to: identify multiple heat map pixels in a specific heat map, wherein each identified pixel is associated with the same maximum probability value for a corresponding joint associated with the specific heat map; for each identified pixel, integrate the heat map values ​​in a predetermined area adjacent to the identified pixel; and select the identified pixel with the highest integrated value as the pixel associated with the corresponding joint position.

[0036] In another optional embodiment, the extractor module further comprises:

[0037] a two-dimensional joint coordinate estimator module configured to provide an estimate of the two-dimensional joint coordinates of each joint and a confidence value for each joint for each frame;

[0038] The three-dimensional fitting module is configured to fit the three-dimensional human body model to the two-dimensional joint coordinates.

[0039] In other embodiments of the computer system, the gait parameter determination module implements the various gait parameter determination methods disclosed herein.

[0040] Other aspects of the invention will be realized and attained by the elements and combinations particularly described in the appended claims.It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention described. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A simplified block diagram of a computer system for human gait analysis based on a video stream obtained from a monocular camera device according to an embodiment is included;

[0042] Figure 2A , Figure 2B , Figure 2C is a simplified flow chart illustrating a computer-implemented method for human gait analysis according to various embodiments;

[0043] Figure 3 An alternative computer-implemented method for improving joint position determination according to one embodiment is shown;

[0044] FIG. 4A to FIG. 4D Various options for determining various gait parameters from a video stream are shown;

[0045] Figure 5 An example of an adjusted skeletal model for a corresponding individual is shown in side view.

[0046] Figure 6 An example of an adjusted skeletal model for a corresponding individual is shown in front view;

[0047] Fig. 7A shows step length gait parameter determination according to an embodiment;

[0048] Figure 7B shows honest extremum identification according to an embodiment;

[0049] Figure 8 shows step height gait parameter determination according to an embodiment; and

[0050] Fig. 9 is a diagram showing an example schematic of a general purpose computer device and a general purpose mobile computer device that may be used with the techniques described herein. DETAILED DESCRIPTION

[0051] Figure 1 A simplified block diagram of a computer system 100 for performing human gait analysis based on a video stream 202 obtained from a monocular camera device 201 according to an embodiment is included. Figure 3 and Figure 4* are described in the context of the flowchart Figure 1, shows a simplified flow chart with steps of a computer-implemented method for human gait analysis according to various embodiments. Therefore, the following description includes the reference numerals in all of these figures.

[0052] The computer system 100 includes an interface 110 configured to obtain a video stream 202 provided by a monocular camera device 201. The video stream can be obtained directly from the camera 201, where the real-time analysis of individual gait parameters is intended to be performed based on the real-time stream. The video stream can also be obtained from a video storage 203 to perform offline analysis on previously recorded videos. The video stream 202 includes multiple frames reflecting the walking of a human individual 10. In other words, the camera 201 is positioned so that the walking of the individual occurs within the perspective of the camera indicated by the dotted line. The interface 110 is configured to receive frames of the video stream in a format recorded or preprocessed by the camera device 201, or to receive frames of the video stream in a format in which it is stored in a video storage device 203. The video storage device can be any data storage device configured to store a sequence of video frames of the stream 202. For example, the storage device 203 can be located on a remote storage device (e.g., on a so-called cloud server) or can be a component of the system 100.

[0053] The system 100 further includes a skeleton motion extractor 120. The skeleton motion extractor 120 is configured to extract skeleton motion associated with the individual 10 from the video stream 202. For the purpose of skeleton motion extraction, the extractor infers three-dimensional gait information (1100) from the obtained video stream. The three-dimensional gait information includes an estimate of the joint position of the individual. The joint position includes at least the foot position of the individual on each frame. The above estimate is obtained by matching the two-dimensional coordinates of the corresponding frame with the corresponding three-dimensional model information of the individual's body for each frame. Various extraction methods can be applied by those skilled in the art. In one implementation, the estimated two-dimensional coordinates can be combined with a position map obtained by using a convolutional neural network (VNect method). In the VNect method, advantageously, the camera 201 is a monocular RGB camera device. In another implementation, a three-dimensional surface model of the human body is used, and its parameters are fitted to the estimated two-dimensional coordinates (SMPLify method).

[0054] In an optional improved VNect implementation (dashed box), the inference module 121 of the extractor 120 uses a convolutional neural network CNN trained on a three-dimensional human posture dataset. As disclosed in the VNect reference cited in Section 4.1, the CNN is configured to derive heat maps and position maps for joint position estimation (1110). This operation can be performed in real time in a scenario where gait analysis is performed based on a real-time video stream. Therefore, a specific heat map describes the probability that a specific joint is associated with each pixel of the corresponding frame for a corresponding frame. A specific set of position maps (3D position maps) includes multiple position maps, each of which describes the distance from a specific joint to the root position of the corresponding frame in the corresponding spatial dimension. The set includes position maps of three spatial dimensions and therefore represents a 3D position map.

[0055] The skeleton model estimator module 122 of the extractor 120 can be used to improve the implementation of VNect to obtain a higher accuracy of the derived joint positions of the individual. In the VNect method of the prior art, the following kinematic skeleton fitting is directly applied by using a default skeleton model for all individuals. If the real skeleton dimensions of the individual show substantial deviations from the default skeleton model dimensions, this may result in large errors (losses). In order to overcome this shortcoming, it is recommended to first estimate the skeleton model of the individual that is close to the real skeleton of the individual, and then perform kinematic skeleton fitting based on the estimated skeleton model. Therefore, the corresponding losses are greatly reduced, and a higher accuracy is obtained when determining the joint positions of the individual.

[0056] The model estimator 122 first selects at least one frame sequence of a video during which the joints of the individual move smoothly over time. Smooth motion in this sense is motion without any discontinuity in the sense that the joints of the individual do not move over a large distance from one frame to the next. The substantial distance corresponds to the distance over which the movement of the body part of the individual causes a change in pixel coordinates in subsequent frames, which indicates a sudden or erratic movement of the body part. Examples of discontinuous motion occur when the individual runs or jumps, or when the individual begins to move from a previous standing or sitting position. The model estimator can receive an indicator (1120) from an operator 20 of the system 100 (e.g., a medically trained person) indicating the start and stop frames surrounding the frame sequence with smooth motion, and the corresponding frame sequence selected. In an alternative implementation, the model estimator can apply standard image processing techniques according to predefined rules to identify such smooth motion sequences in the video stream. The predefined rules define thresholds for joint motion over two or more frames. The threshold specifies the beginning of discontinuous motion.

[0057] The model estimator then estimates the individual's skeleton model (1130) by determining the loss of each joint of the default skeleton model in each spatial coordinate for each frame of the selected sequence and by adjusting the default skeleton model to compensate for the determined loss to provide an adjusted skeleton model. The adjusted skeleton model can now reflect the individual's true skeleton with higher accuracy than the default skeleton model used as a starting point.

[0058] Finally, in the improved VNect implementation, the skeleton fitting module 123 of the extractor performs kinematic skeleton fitting for each video frame using the adjusted skeleton model to determine multiple joint positions of the individual 10 (1140). Note that the determined joint positions also include the individual's joints indicating the individual's foot positions on each frame. Kinematic skeleton fitting can be performed according to the method described in Section 4.2 of the VNect reference, where the default skeleton model in the VNect method is replaced with the estimated skeleton model.

[0059] Figure 3 The flowchart of shows a further improved embodiment of VNect implementation. In this embodiment, the prediction module is also configured to identify multiple heat map pixels (1111) in a specific heat map, wherein each identified pixel is associated with the same maximum probability value for a corresponding joint associated with the specific heat map. For each identified pixel, an integral value is calculated by integrating the heat map values ​​in a predetermined area adjacent to the identified pixel (1113). The identified pixel with the highest integral value is then selected as the pixel associated with the corresponding joint position (1115) for further processing.

[0060] Briefly turn to Figure 5 , an estimated skeletal model 300 of individual 10 is shown for individual 10 as viewed from the side. Figure 5 In FIG. 1 , joint positions are shown by respective circles 310 , 311 , 312 , and connections between joints (eg, skeletons) are shown by thick lines 320 . The determined foot positions may correspond to joints 311 and 312 . Figure 6 In FIG. 3 , an estimated skeleton model 301 of individual 11 is shown from a perspective viewed from the front. Figure 6 In FIG. 1 , the joint positions are shown by the respective joints 313, 314 connected again by straight lines. Figure 6 In the example of FIG. , joints 313 , 314 may correspond to the positions of the feet of an individual.

[0061] Now back to Figure 1 Combination Figure 2CIn the alternative implementation based on the SMPlify method, the extractor 120 works similarly to the VNect implementation in that both implementations use estimates of the two-dimensional coordinates of the skeleton joints and infer three-dimensional information about the human pose based on the two-dimensional data of the joint coordinates via an optimization procedure. In addition to the three-dimensional human pose information, the SMPLify implementation can also infer three-dimensional information about the individual shapes.

[0062] The SMPLify implementation allows automatic estimation of the 3D pose and 3D shape of a human body (e.g., an individual's body) from a single unconstrained image. As input, SMPLify requires an estimate of the 2D coordinates of each joint and a confidence value for each joint. In principle, these two 2D estimates (1150) can be provided by any appropriate method known to the skilled person. For example, a CNN-based method such as VNect, DenseCut (see "DenseCut: Densely Connected CRF for Real-Time GrabCut", Cheng et al., 2015, European Graphics Association and John Wiley & Sons Ltd.) or a convolutional pose machine can be used. SMPLify then fits a 3D human model (Skinned Multi-Person Linear Model (SMPL)) to the 2D joints (1160). The model can fit very little data very well. In addition, the 3D model prevents 3D poses that may fit 2D joints but cause interpenetration.

[0063] The SMPLify implementation requires an estimate of the 2D joint coordinates as input. This estimate can be obtained, for example, from VNect, DeepCut, a convolutional pose machine or any other method that provides 2D joint coordinates for this purpose. In contrast to the VNect implementation, SMPlify does not rely on a position graph, but uses the SMPL 3D body model. In contrast to the VNect implementation, the SMPLify solution does not allow interpenetration. This can be advantageous in many gait analysis scenarios. One advantage of the VNect implementation is that VNect can be executed in real time, while SMPLify takes about two minutes to process a single image on a recent PC with a GPU.

[0064] The joint positions determined by extractor 120 are then used by gait parameter determination module 130 of system 100 to determine one or more gait parameters of individual 10 based on the individual's foot positions in the local extrema frame. The local extrema frame shows the local extrema of the distance between the position of one of the individual's feet and the corresponding reference position. In other words, the gait parameter determination module identifies frames in the video stream where the distance between the individual's foot and the reference point is extremely small or extremely large compared to the previous frame and the next frame. The type of reference point used for a particular calculation depends on the type of gait parameter to be determined.

[0065] exist FIG. 4A to FIG. 4DThe scenario shown in FIG. 7 and FIG. 8 shows computer-implemented method steps for determining step length, step speed, and step height parameters for an individual 10. These parameters provide relevant information for medically trained personnel to assess the risk of falling for the individual 10. Figure 8 These diagrams are discussed in more detail in the context of the scenarios shown.

[0066] exist Figure 4A In the embodiment, step 1200 also includes a calculation step 1210 for determining step length gait parameters. Fig. 7A The corresponding reference position is shown as the position of the second foot of the individual in the corresponding local maximum frame flma*. Fig. 7A The right hand of the individual walks to the left direction. When the distance between the two feet of the individual reaches the local maximum sl1, sl2, sl3, the video frames flmal, flma2 and flma3 always show the individual. Fig. 7A The lower part of shows schematically how the distance d between the two feet evolves over time t. The behavior is similar to a sinus curve. When one foot passes the other during walking, a local minimum is reached. When both feet are in contact with the ground, local maxima Ima1, Ima2, Ima3 usually appear. In other words, a local maximum frame occurs when the distance from one foot position to the corresponding reference joint position reaches a maximum compared to the previous frame and the next frame.

[0067] However, in real-world walking scenes, there are often multiple adjacent frames showing the feet of an individual at approximately the same distance (e.g., during a period of time when both feet are in contact with the ground). Figure 7B , which shows honest extrema in human gait that can be used according to an alternative embodiment of the present invention The following assumptions are made:

[0068] There is moderate variation in stride length when humans walk, for example, half the average stride length (1 / 2 sl, see Fig. 7A ) or smaller;

[0069] Gait is even and without sudden changes.

[0070] from Figure 7B It can be seen that there are many frames around time T where the step length is close to its maximum (reached during the entire gait analysis). The "x" marks represent the distance d between the feet of an individual in each corresponding frame over time. In order to identify the distance between the feet of an individual and the honest local extrema (i.e., the honest local maximum), we use the distance d between the feet of the individual in the corresponding frame to identify the distance between the feet of the individual and the honest local maximum. / Honest local minimum ) related to multiple frames, it can be done as follows:

[0071] 1. Fixation Make where and are the percentiles of frames respectively received to assist in minimum and maximum step size calculations.

[0072] Among all step sizes (obtained from frames), let

[0073] denote the percentile, and

[0074] denote the percentile.

[0075] 2. Let

[0076] be the sets respectively containing the honest minimum value frames and the honest maximum value frames. Each such set includes clusters of frames associated with the movement of an individual where the distance between the feet of the individual passes through local extrema. For example, within such a cluster associated with an honest maximum value frame inside, the distances measured in the individual frames of the cluster may fluctuate around a certain average value. That is, within such a cluster, there may be multiple local maxima. However, to determine the step size, the entire cluster provides meaningful information.

[0077] In addition, the number of honest extreme value clusters identified in the frame sequence provides an accurate measure for counting the number of steps performed by an individual in the frame sequence. In fact, the number of honest maximum value clusters directly corresponds to the number of steps performed by the individual. When counting all local maxima in the entire frame sequence, since each cluster usually includes multiple local maxima (or includes multiple local minima), it can be expected that the number of derived local maxima will greatly exceed the number of steps performed by the individual. However, as mentioned above, the step size can still be determined with sufficient accuracy from the local maximum value frames.

[0078] These sets can be uniquely decomposed into disjoint unions as follows:

[0079]

[0080] such that, for any one k, and f < g; similarly, for any k, and f < g.

[0081] In addition, we require that for any k there is exactly one l (except for k ∈ {min, max}) such that

[0082] or

[0083]

[0084] Here, A < B is defined as a < b.

[0085] The two decompositions found algorithmically for A are simple.

[0086] are called honest minimum frames, and

[0087] are called honest maximum frames.

[0088] min is the number of honest maximum values,

[0089] max is the number of honest minimum values.

[0090] One can choose the value of the k-th honest maximum / minimum as some average of the quantities in, e.g., one can take the average

[0091]

[0092] where #A denotes the cardinality of set A.

[0093] Figure 1 The gait parameter determination module 130 of calculates the average sl of the distances sl* between the foot positions of the individual in the determined local maximum frame flma* as the step length gait parameter (1210). In the honest maximum embodiment, the system calculates the average of the distance values derived from the honest maximum clusters (1210). In the Fig. 7A simplified illustration, all the sl* values have the same value. However, in the real-world situation with real video recordings, each distance s1* in the local maximum frame (or derived from the honest local maximum clusters) will most likely be slightly different from the distances in the previous and subsequent local maximum frames. That is, one can determine the average value and standard deviation for the determined step length parameter. The gait parameter information is provided to the medically trained personnel 20 (210). Thus, a high standard deviation indicates to the medically trained personnel that the individual's individual step lengths show significant deviations. This deviation can be considered when assessing the individual's fall risk. As mentioned before, one can directly measure the distance between the individual's foot positions in the corresponding frame according to the corresponding pixel coordinates, or one can indirectly measure the distance between the individual's foot positions for the corresponding local maximum frame according to the distance of the individual's foot from the monocular camera device 201 using the pinhole projection model.

[0094] In order to determine the walking speed gait parameters of an individual, Figure 4B In an embodiment of the present invention, the system can calculate the walking distance of the individual based on the determined step length and the number of local maximum frames identified in the video stream (1220). Then, the walking distance is divided by the time interval associated with the frame sequence associated with the walking distance to obtain the walking speed (1230).

[0095] In an alternative implementation for obtaining step length gait parameters (1210), it is assumed that the camera 201 does not move during video recording. The total distance traveled by the patient can then be calculated as the sum D(1)+D(2)+...+D(F-1) (1220), where D(f)=|d(f+1)-d(f)|, where d(f) represents the distance of the individual from the camera in frame f, and F is the number of frames in the video used for step length determination. The distance of the individual from the camera can be advantageously determined by applying a pinhole projection model and using it to calculate the distance to the individual's pelvis. However, other joint locations (e.g., the individual's head) may also be used. In one implementation, a distance is calculated based on all joints or a specific subset of joints (e.g., the head 318 (see Figure 5 ) and pelvis 319 (see Figure 5 )) can be used as a reference point. The distance to the camera is determined based on the reference point. Typically, the distance can be determined for each frame as the distance between the camera and a reference point that characterizes the current position of the individual.

[0096] The total number of steps in the defined video stream sequence is then determined as the number of "local maxima" in the step length sequence sl(1), sl(2), ..., sl(F) for the tracked individual 10, where sl(f) is the distance between the feet of the individual in frame f. The average step length and cadence can be easily calculated from these two gait parameters.

[0097] Sometimes, VNect implementations confuse joints in left and right variants (e.g., double knees or feet, where pairs of joints include left and right variants). To avoid this confusion, the following is proposed: Assume that jl and jr are left and right variants of a certain joint. Then, in each frame f, replace the heatmap H(f, jl) of joint jl with H(f, jl)-H(f, jr), and thus replace joint jr. The explanation of this replacement is as follows. Since H(f, jr) encodes the likelihood that the right variant of joint j is associated with a particular pixel in frame f, it is obvious that the higher the value of a particular point in H(f, jr), the lower the likelihood that this particular point is associated with the left variant jl of the same joint j. Therefore, to prevent the algorithm from looking for jl at these locations, these probabilities are subtracted from H(f, jl).

[0098] Figure 8 Shown according to Figure 4C and Figure 4D Any of the illustrated embodiments may illustrate how to determine a step height parameter for individual 10 based on information provided in the video stream.

[0099] exist Figure 4C In an embodiment of the present invention, the system extrapolates a walking floor plane (WFP) (1240) from a floor contact point cp* of the individual's foot identified from a local maximum frame flma*. The local maximum frame flma* corresponds to a frame that can also be used for step length determination. That is, the corresponding reference position is the position of the individual's second foot in the corresponding local maximum frame. Once the contact point cp* is determined, the walking floor plane can be determined by fitting a plane through the contact point.

[0100] The system can then determine additional local maximum frames flma*' (1250). These local maximum frames have additional local maximum values ​​Ima*' of the distance d' between the individual's foot position and the walking floor plane WFP. The second set of local maximum frames flma1', flma2', flma3' includes frames in which the distances sh1, sh2, sh3 between one foot of the individual and the WPF show local maxima. When walking, the same foot passes through such local maximum every two steps. The local maxima Ima1' and Ima3' are associated with the first foot of the individual, while the local maximum Ima2' is associated with the second foot. Finally, the system calculates the average value sh of the additional local maximum distances sh1, sh2, sh3 as the step height (1260). Fig. 7A As shown, in Figure 8 The cyclical behavior shown in the lower part of should be understood as a simplified schematic diagram. In real-world scenarios, the step height of each step may vary. In addition, based on the individual's anatomical properties, there may be differences between the step heights of an individual's two feet. In this case, it may be useful to provide medically trained personnel with separate step height parameters for each foot. Similarly, the standard deviation can provide valuable information about the smoothness of an individual's gait.

[0101] exist Figure 4DIn another embodiment, the step height gait parameter is determined by using the head or pelvis of the adjusted skeleton model in the corresponding local minimum frame as the corresponding reference position for calculation. In this embodiment, the system determines the local minimum frames (1270). These local minimum frames have local minima of the distance between the position of the individual's feet and the corresponding reference position. In other words, if the distance between one of the individual's feet is at a local minimum relative to the reference joint (head or pelvis) of the individual's body, the distance to the walking floor shows a local maximum. Once the local minimum frames are determined, the system can calculate the average value of the distance between the individual's two feet in the local minimum frames as the step length (1280). For example, the system can simply measure the distance between the two feet in each local minimum frame when one foot touches the ground in the corresponding frame, and the distance between the feet in the local minimum frame is a good approximation of each step height. In an alternative embodiment, those skilled in the art can also apply the honest local extremum method (disclosed in detail in the context of step length calculation) to the calculation of the step height parameter.

[0102] return Figure 1 In one embodiment, the system 100 may further include a parameter scoring module 140. The scoring module 140 may associate one or more determined gait parameters with a score characterizing the individual's risk of falling. Medically trained personnel 20 are familiar with such scoring schemes for assessing the risk of falling. For example, such a scoring scheme may be obtained from the document "Expertenstandard Sturzprophylaxe in der Pflege, 1.Aktualisierung 2013, Deutsches Netzwerk für in der Pflege” was obtained.

[0103] Fig. 9 9 is a schematic diagram showing examples of a general purpose computer device 900 and a general purpose mobile computer device 950 that can be used with the techniques described herein. Computing device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. General purpose computer device 900 may correspond to Figure 1 The computer system 100 of the present invention is shown in Figure 1. The computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. For example, the computing device 950 may be a smart phone that includes a monocular RGB camera and may be used by a medically trained person to interact with the computer device 900. The components shown here, their connections and relationships, and their functions are exemplary only and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0104] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and the storage device 906. Each of the components 902, 904, 906, 908, 910, and 912 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 902 can process instructions for execution within the computing device 900, including instructions stored on the memory 904 or the storage device 906, to display graphical information of a GUI on an external input / output device (e.g., a display 916 coupled to the high-speed interface 908). In other implementations, multiple processing units and / or multiple buses and multiple memories and memory types can be used appropriately. Moreover, multiple computing devices 900 can be connected, each of which provides a necessary operating portion (e.g., as a server library, a blade server group, or a processing device).

[0105] The memory 904 stores information within the computing device 900. In one implementation, the memory 904 is one or more volatile memory units. In another implementation, the memory 904 is one or more non-volatile memory units. The memory 904 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0106] The storage device 906 can provide mass storage for the computing device 900. In one implementation, the storage device 906 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, a tape device, a flash memory, or other similar solid-state storage device or device array, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that perform one or more methods (such as the above methods) when executed. The information carrier is a computer or machine-readable medium, such as a memory 904, a storage device 906, or a memory on a processor 902.

[0107] The high-speed controller 908 manages bandwidth-intensive operations of the computing device 900, while the low-speed controller 912 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one implementation, the high-speed controller 908 is coupled to the memory 904, the display 916, and the high-speed expansion port 910 that can accommodate various expansion cards (not shown) (e.g., through a graphics processor or accelerator). In this implementation, the low-speed controller 912 is coupled to the storage device 906 and the low-speed expansion port 914. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices (e.g., keyboards, pointing devices, scanners, or network devices such as switches or routers) through a network adapter.

[0108] As shown, computing device 900 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 920, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 924. In addition, it can also be implemented in a personal computer such as a laptop computer 922. Alternatively, components from computing device 900 can be combined with other components in a mobile device (not shown) such as device 950. Each such device can contain one or more computing devices 900, 950, and the entire system can be composed of multiple computing devices 900, 950 that communicate with each other.

[0109] Computing device 950 includes, among other components, a processor 952, a memory 964, an input / output device such as a display 954, a communication interface 966, and a transceiver 968. Device 950 may also be equipped with a storage device such as a micro drive or other device to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and some components may be mounted on a common motherboard or may be mounted in other ways as appropriate.

[0110] The processor 952 may execute instructions (including instructions stored in the memory 964) within the computing device 950. The processor may be implemented as a chipset including separate and multiple analog and digital processing units. The processor may provide, for example, coordination of other components of the device 950, such as control of a user interface, applications executed by the device 950, and wireless communications performed by the device 950.

[0111] The processor 952 can communicate with the user through a control interface 958 and a display interface 956 coupled to a display 954. The display 954 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display or other suitable display technology. The display interface 956 may include appropriate circuits for driving the display 954 to present graphics and other information to the user. The control interface 958 may receive commands from the user and convert them to submit to the processor 952. In addition, an external interface 962 in communication with the processor 952 may be provided so that the device 950 can communicate with other devices in close proximity. The external interface 962 may, for example, provide wired communication in some implementations, or wireless communication in other implementations, and multiple interfaces may also be used.

[0112] The memory 964 stores information within the computing device 950. The memory 964 can be implemented as one or more of a computer-readable medium, a volatile storage unit, or a non-volatile storage unit. An expansion memory 984 can also be provided and connected to the device 950 via an expansion interface 982. The expansion interface 982 can include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 984 can provide additional storage space for the device 950, or can also store applications or other information for the device 950. Specifically, the expansion memory 984 can include instructions for executing or supplementing the above-mentioned process, and can also include security information. Therefore, for example, the expansion memory 984 can be used as a security module for the device 950, and can be programmed with instructions that allow the secure use of the device 950. In addition, a security application can be provided by a SIMM card and additional information, such as placing identification information on the SIMM card in an unbreakable manner.

[0113] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods (e.g., the methods described above). The information carrier is a computer or machine readable medium, such as memory 964, expansion memory 984, or memory on processor 952, which can be received, for example, via transceiver 968 or external interface 962.

[0114] The device 950 can perform wireless communication through the communication interface 966. The communication interface 966 may include a digital signal processing circuit when necessary. The communication interface 966 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS message transceivers, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication can occur, for example, through a radio frequency transceiver 968. In addition, short-range communication can be performed, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 980 can provide additional wireless data related to navigation and location to the device 950. These wireless data can be used appropriately by the application running on the device 950.

[0115] Device 950 may also communicate audibly using audio codec 960. Audio codec 960 may receive voice information from a user and convert it into usable digital information. Audio codec 960 may similarly generate audible sounds for the user, such as through a speaker, such as in a handset of device 950. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 950.

[0116] As shown, the computing device 950 can be implemented in many different forms. For example, it can be implemented as a cellular phone 980. It can also be implemented as part of a smart phone 982, a personal digital assistant, or other similar mobile device.

[0117] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs. The one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor. The programmable processor can be dedicated or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0118] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer. The computer has a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0120] The systems and techniques described herein can be implemented in a computing device. The computing device includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0121] Computing devices may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0122] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the invention.

[0123] In addition, the logic flows depicted in the accompanying drawings do not require the particular order shown or sequential order to achieve the desired results. In addition, other steps can be provided from the described flows, or steps can be omitted, and other components can be added to the described systems or deleted from the described systems. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method (1000) for performing human gait analysis based on a video stream (202) obtained from a monocular camera device (201), the video stream comprising a plurality of frames reflecting the walking of a human individual (10), the method comprising: inferring (1100) three-dimensional gait information from the obtained video stream, wherein the three-dimensional gait information includes estimates of joint positions of the individual, the joint positions including at least positions of the feet of the individual at each frame, the estimates being derived by matching the two-dimensional joint coordinates of the corresponding frame with corresponding three-dimensional model information of the body of the individual for each frame; as well as determining ( 1200 ) one or more gait parameters of the individual based on a position of a foot of the individual in a local extrema frame, the local extrema frame showing local extrema of a distance between a position of one foot of the individual and a corresponding reference joint position; Among them, the inference (1100) also includes: deriving (1110) heat maps and position maps for joint position estimation by using a convolutional neural network trained on a three-dimensional human pose dataset, wherein a particular heat map describes, for a corresponding frame, probability values ​​of a particular joint being associated with each pixel of the corresponding frame, and wherein a set of particular position maps includes a plurality of position maps, wherein each position map describes, for the corresponding frame, a distance from a particular joint to a foot position in a corresponding spatial dimension; Receiving (1120) a selection of at least one sequence of frames of the video stream during which a joint of the individual (10) moves smoothly over time; estimating (1030) a skeletal model of the individual (10) by determining a loss for each joint of a default skeletal model for each frame of the selected sequence in each spatial coordinate and adjusting the default skeletal model to compensate for the determined losses to provide an adjusted skeletal model; and Performing (1140) kinematic skeleton fitting on each video frame using the adjusted skeleton model to determine a plurality of joint positions, the joint positions including at least foot positions of the individual's feet on each frame; Among them, the inference (1110) also includes: identifying (1111) a plurality of heat map pixels in a particular heat map, wherein each identified pixel is associated with a same maximum probability value for a corresponding joint associated with the particular heat map; For each identified pixel, integrating the heat map value in a predetermined area adjacent to the identified pixel (1113); and selecting (1115) the identified pixel having the highest integrated value as the pixel associated with the corresponding joint position; Therein, at least one of the determined gait parameters is associated with a score characterizing the risk of falling of the individual.

2. The method according to claim 1, wherein: The one or more gait parameters include a step length gait parameter, and wherein the corresponding reference joint position is a position of a second foot of the individual in a corresponding local maximum frame (flma*), the method further comprising: An average value (sl) of distances (sl*) between positions of the feet of the individual in the determined local maximum frame (flma*) is calculated (1210) as the step length gait parameter.

3. The method according to claim 2, wherein: The distance between the individuals' foot positions is directly measured in corresponding frames based on corresponding pixel coordinates.

4. The method according to claim 2, wherein: Based on the distance of the feet of the individuals from the monocular camera device, the distance between the foot positions of the individuals is indirectly measured for the corresponding local maximum frames.

5. The method according to any one of claims 2 to 4, wherein: The one or more gait parameters further include a walking speed gait parameter, the method further comprising: Based on the determined step length and the local maximum frame number identified in the video stream, a walking distance of the individual is calculated (1220) and the walking distance is divided by a time interval associated with a frame sequence associated with the walking distance to obtain (1230) the walking speed.

6. The method according to any one of claims 1 to 4, wherein: The one or more gait parameters further include a step height gait parameter, the method further comprising: extrapolating (1240) a walking floor plane (WFP) from floor contact points (cp*) of the individual's feet identified from a local maximum frame (flma*), wherein the corresponding reference joint position is the position of a second foot of the individual in the corresponding local maximum frame; Determine (1250) another local maximum frame (flma*') having another local maximum value (Ima*') of the distance of the individual's foot position from the walking floor plane (WFP), and calculate (1260) an average value (sh) of the other local maximum distances as step height.

7. The method according to any one of claims 1 to 4, wherein: The one or more gait parameters also include a step height gait parameter, and wherein the corresponding reference joint position is the position of the head (318) or pelvis (319) of the adjusted skeleton model in the corresponding local minimum frame, and the method further includes determining (1270) a local minimum frame having a local minimum of distances between the individual's foot positions and the corresponding reference joint positions; and The average of the distances between the two feet of the individual in the local minimum frame is calculated (1280) as the step height.

8. The method according to any one of claims 1 to 4, wherein: In order to determine (1200) the one or more gait parameters, a cluster of honest local extremum frames is determined, wherein frames of a particular cluster contribute to calculating a step length of the individual because an average distance value is calculated based on all frames of the corresponding cluster, such that the honest local extremum frames of the particular cluster include distances between feet of the individual that collectively reflect for a particular step of the individual.

9. A computer program product comprising instructions which, when loaded into a memory of a computing device and executed by at least one processor of the computing device, perform the method steps of the computer-implemented method according to any one of the preceding claims.

10. A computer system (100) for human gait analysis, comprising: An interface (110) is configured to obtain a video stream (202) from a monocular camera device (201), wherein the video stream (202) includes a plurality of frames reflecting walking of a human individual (10); a skeletal motion extractor (120) configured to extract skeletal motion associated with the individual (10) from the video stream (202) by inferring three-dimensional gait information from the obtained video stream, wherein the three-dimensional gait information includes an estimate of the joint position of the individual, the joint position including at least the position of the feet of the individual in each frame, the estimate being derived by matching the two-dimensional joint coordinates of the corresponding frame with corresponding three-dimensional model information of the body of the individual for each frame; and a gait parameter determination module (130) configured to determine one or more gait parameters of the individual based on a position (311, 312) of a foot of the individual in a local extremum frame showing a local extremum of a distance between a position of one foot of the individual and a corresponding reference joint position; Wherein, the skeleton motion extractor (120) is further configured to: deriving heatmaps and position maps for joint position estimation by using a convolutional neural network trained on a three-dimensional human pose dataset, wherein a particular heatmap describes, for a corresponding frame, a probability value of a particular joint being associated with each pixel of the corresponding frame, and wherein a particular set of position maps includes a plurality of position maps, wherein each position map describes, for the corresponding frame, a distance from a particular joint to a foot position in a corresponding spatial dimension; receiving a selection of at least one sequence of frames of the video stream (202) during which a joint of the individual moves smoothly over time; estimating a skeletal model of the individual (10) by determining a loss for each joint of a default skeletal model for each frame of the selected sequence in each spatial coordinate and adjusting the default skeletal model to compensate for the determined losses to provide an adjusted skeletal model; and performing kinematic skeleton fitting on each video frame using the adjusted skeleton model to determine a plurality of joint positions, the joint positions including at least foot positions of the individual's feet on each frame; Wherein, the skeleton motion extractor (120) is further configured to: identifying a plurality of heat map pixels in a particular heat map, wherein each identified pixel is associated with a same maximum probability value for a corresponding joint associated with the particular heat map; For each identified pixel, integrating the heat map value in a predetermined area adjacent to the identified pixel; and selecting the identified pixel having the highest integrated value as the pixel associated with the corresponding joint position; Therein, at least one of the determined gait parameters is associated with a score characterizing the risk of falling of the individual.

11. The system according to claim 10, wherein: The skeleton motion extractor also includes: a two-dimensional joint coordinate estimator module configured to provide, for each frame, an estimate of the two-dimensional joint coordinates of each joint and a confidence value for each joint; as well as The three-dimensional fitting module is configured to fit the three-dimensional human body model to the two-dimensional joint coordinates.

12. The system according to claim 10 or 11, wherein: The one or more gait parameters include a step length gait parameter, and wherein the corresponding reference joint position is a position of a second foot of the individual in a corresponding local maximum frame, and the gait parameter determination module is further configured to: calculating a total distance traveled by the individual by applying a pinhole projection model to determine a distance between the camera and a reference point characterizing a current position of the individual for at least first and second frames, the first and second frames defining a sequence of video streams for step length analysis, deriving the distance traveled from the determined distance; determining the number of steps in the defined sequence of video streams as the number of local maximum frames having a local maximum of the distance between the feet of the individual; and Divide the calculated travel distance by the number of local maximum frames to obtain the average step size.

13. The system according to claim 10 or 11, wherein: The one or more gait parameters also include a step height gait parameter, and the gait parameter determination module is further configured to: determine a local minimum frame having a local minimum value of a distance between a foot position of the individual and a head or pelvis of the adjusted skeletal model; and The average of the distances between the two feet in the local minimum frame is calculated as the step length.

14. The system according to claim 10 or 11, wherein: The gait parameter determination module (130) is also configured to determine a cluster of honest local extremum frames, wherein frames of a particular cluster contribute to calculating the step length of the individual because an average distance value is calculated based on all frames of the corresponding cluster, so that the honest local extremum frames of the particular cluster include frames that collectively reflect the distance between the feet of the individual for a particular step of the individual.

Images