Falling risk extraction device, falling risk extraction method, and falling risk extraction program

Abstract

The fall risk extraction device (20) comprises a skeletal information extraction unit (22) that extracts skeletal information of a subject from a video of the subject walking, a multiple joint information preprocessing unit (23) that extracts information including each of time-series position changes, angle changes, speed changes, and angular velocity changes at each joint when the subject walks from the skeletal information and calculates gait features including correlations between movements of multiple joints based on the extracted information, and a fall risk determination unit (27) that extracts the subject's risk of falling based on the gait features including correlations between movements of multiple joints.

Description

Fall risk extraction device, fall risk extraction method, and fall risk extraction program

[0001] The present disclosure relates to a fall risk extraction device, a fall risk extraction method, and a fall risk extraction program.

[0002] Patent Literature 1 describes a factor estimation system that acquires walking data from a subject, detects walking characteristics such as changes in the position or angle of the subject's joints, and analyzes the walking characteristics for each joint individually to extract the subject's risk of falling. Patent Literature 2 also describes a hemiplegia testing device that performs frequency analysis on the walking characteristics for each joint and extracts walking characteristics of hemiplegia and the like using the frequency analysis results.

[0003] JP 2021-030051 A JP 2020-048683 A

[0004] However, the devices described in Patent Documents 1 and 2 extract the risk of falling without considering the correlation between movements in multiple parts of the body. This means that it is not possible to evaluate the risk of falling related to the interlocking of movements in multiple parts of the body, such as the movement of the left and right legs, and this creates the problem that appropriate follow-up observation is not possible during rehabilitation, etc.

[0005] The present disclosure aims to provide a fall risk extraction device, a fall risk extraction method, and a fall risk extraction program that extract a subject's fall risk by taking into account the correlation between movements in multiple parts of the subject.

[0006] The fall risk extraction device disclosed herein is characterized by comprising a skeletal information extraction unit that extracts skeletal information of a subject from a video of the subject walking; a joint information extraction unit that extracts information from the skeletal information including time-series position changes, angle changes, speed changes, and angular velocity changes at each joint while the subject walks; a gait feature calculation unit that calculates gait features including correlations between the movements of multiple joints based on the information extracted by the joint information extraction unit; and a fall risk extraction unit that determines the subject's risk of falling based on the gait features including correlations between the movements of the multiple joints.

[0007] According to the device disclosed herein, by extracting the risk of falling by taking into account the correlation of joint features in walking movements, it becomes possible to easily and accurately monitor the progress of the coordination of movements of multiple parts of the body during the rehabilitation of subjects who are at risk of falling.

[0008] 1 is a functional configuration diagram showing a fall risk extraction device according to embodiment 1. FIG. 2 is a hardware configuration diagram showing a fall risk extraction device according to embodiment 1. FIG. 3 is a flowchart showing an example of the operation of the fall risk extraction device according to embodiment 1. (a) is an example of time-series data of initial gait features of a subject who is not a fall risk, and (b) is an example of time-series data of initial gait features of a subject who is a fall risk. (a) is an example of time-series data of gait features after addition processing of a subject who is not a fall risk, and (b) is an example of time-series data of gait features after addition processing of a subject who is a fall risk. (a) is an example of the result of frequency analysis of data of a subject who is not a fall risk, and (b) is an example of the result of frequency analysis of data of a subject who is a fall risk. FIG. 4 is a functional configuration diagram showing a fall risk extraction device according to embodiment 2. FIG. 5 is a flowchart showing an example of the operation of the fall risk extraction device according to embodiment 2. (a) is an example of calculation of the double leg support period of a subject who is not a fall risk, and (b) is an example of calculation of the double leg support period of a subject who is a fall risk.

[0009] A fall risk extraction device according to an embodiment will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified.

[0010] 1 is a functional configuration diagram showing a fall risk extraction device 20 according to embodiment 1. Connected to the fall risk extraction device 20 are a camera 10 that captures images of a walking subject, and a display unit 30 that displays the value of the fall risk extracted by the fall risk extraction device 20 to the subject, etc.

[0011] The camera 10 is an imaging device that captures video (moving images) of the subject. The video captured by the camera 10 is input to a skeletal information extraction unit 22 via a video input unit 21 of the fall risk extraction device 20. The video input unit 21 is an input / output interface 230 (described later), and specifically, is a USB (Universal Serial Bus) terminal, an IEEE 1394 terminal, a Thunderbolt terminal, or the like.

[0012] The skeletal information extraction unit 22 extracts skeletal information of the subject's body from the input video. To extract skeletal information from the video, software capable of detecting human body joints and the like from video, such as "Openpose," is used. The skeletal information extraction unit 22 extracts the three-axis coordinates of the human body joints from the input video as skeletal information. Examples of the joints extracted by the skeletal information extraction unit 22 include the skeletal joints of the head and neck, the skeletal joints of the chest, abdomen, and waist, the skeletal joints of the left and right shoulders, the skeletal joints of the left and right elbows, the skeletal joints of the left and right wrists, the skeletal joints of the left and right hands, the skeletal joints of the left and right hip joints, the skeletal joints of the left and right knees, the skeletal joints of the left and right ankles, and the skeletal joints of the left and right feet.

[0013] The multiple-joint information preprocessing unit 23 performs preprocessing on the skeletal information extracted by the skeletal information extraction unit 22. Specifically, it calculates gait features of multiple joints (hereinafter referred to as "initial gait features") from the skeletal information extracted by the skeletal information extraction unit 22. The initial gait features are, for example, time-series data on the positions or angles of skeletal joints and data obtained by differentiating the time-series data on the positions or angles of skeletal joints. The differential value of the skeletal joint position data is velocity, and the differential value of the skeletal joint angle is angular velocity. In the first embodiment, the time-series data on the position of skeletal joints is the coordinate values ​​of one, two, or three axes of the skeletal joints as time passes during walking. In the case of three axes, for example, a point in the shooting environment or a fixed point on the screen of the video is set as the origin, and the x-axis and y-axis are set horizontally from the origin, and the z-axis is set vertically. Furthermore, as an example, the x-axis is set east-west and the y-axis is set north-south. The angle of a skeletal joint is a quantity that represents the distance between two line segments with the same endpoint when there are line segments connecting a joint to any two other joints. The time series data of the angle is the angle value over time during walking.

[0014] Furthermore, the multiple-joint information preprocessing unit 23 adds or complex-expresses the calculated initial gait features between joints. Adding means adding the values ​​of the initial gait features of multiple joints at the same time in the initial gait features that are time-series data of the multiple joints, and extracting time-series data consisting of the added values ​​at each time (hereinafter referred to as "gait features after addition processing"). Complex expression means expressing the initial gait features that are time-series data of two different joints as complex numbers with the value of the initial gait feature of one joint as the real part and the value of the initial gait feature of the other joint as the imaginary part at each time, and extracting time-series data consisting of complex numbers at each time (hereinafter referred to as "gait features after complex expression"). The time-series data consisting of complex numbers at each time is displayed in a coordinate system (complex plane) of imaginary and real parts, and the complex plane can be considered to be a collection of vectors rotating at an angular frequency related to the passage of time. Hereinafter, the post-addition processing gait features and the post-complex representation gait features will be collectively referred to as "post-calculation processing gait features." Here, the post-calculation processing gait features extracted by adding or complex representing the initial gait features are time-series data of the subject's walking. Furthermore, both the initial gait features and the post-calculation processing gait features are walking features, and are one of the features of walking movements.

[0015] FIG. 4(a) is an example of time series data of initial gait features at the positions of both ankles of a subject who is not at risk of falling. FIG. 4(b) is an example of time series data of initial gait features at the positions of both ankles of a subject who is at risk of falling. FIG. 5(a) is an example of time series data of gait features after addition processing extracted by adding the time series data of the positions of both ankles shown in FIG. 4(a) of a subject who is not at risk of falling. FIG. 5(b) is an example of time series data of gait features after addition processing extracted by adding the time series data of the positions of both ankles shown in FIG. 4(b) of a subject who is at risk of falling. In addition to the position changes of both ankles, addition processing or complex expression may be performed on the angle changes, velocity changes, or angular velocity changes of both ankles.

[0016] The frequency analysis unit 24 outputs the results of frequency analysis of the post-calculation gait features, which are information preprocessed by the multi-joint information preprocessing unit 23 (hereinafter referred to as "post-frequency analysis gait features"). Specifically, the frequency analysis unit 24 performs a Fourier transform on the post-addition gait features. The frequency analysis unit 24 also performs an inverse Fourier transform on the complex-expressed gait features. As described above, time series data consisting of complex numbers at each time represents a collection of vectors that rotate in a complex plane at angular frequencies that change over time, and can therefore be considered a function of angular frequency. Therefore, by performing an inverse Fourier transform on the time series data consisting of complex numbers at each time, real time series data can be calculated. The real time series data calculated by the inverse Fourier transform corresponds to a frequency analysis of a trajectory in complex space, and as a result, the frequency characteristics of the movements of two joints can be calculated.

[0017] 6(a) is an example of a frequency-analysis gait feature resulting from frequency analysis of the summation-processed gait feature, which is time-series data obtained by adding time-series data of the joint positions of a subject who is not at risk of falling. FIG. 6(b) is an example of a frequency-analysis gait feature resulting from frequency analysis of the summation-processed gait feature, which is time-series data extracted by adding time-series data of the joint positions of a subject who is at risk of falling. Comparing FIG. 6(a) and FIG. 6(b) reveals differences in the frequency analysis results. For example, as shown in FIG. 6(a), the amplitude of the subject who is not at risk of falling is larger than the amplitude of the subject who is at risk of falling, as shown in FIG. 6(b).

[0018] The fall risk assessment unit 27 extracts the subject's fall risk based on the frequency-analysis gait features output from the frequency analysis unit 24 and the data stored in the memory unit 26. The subject's fall risk is an index whose value ranges from a minimum of 0 to a maximum of 100, with larger values ​​indicating a higher fall risk and smaller values ​​indicating a lower fall risk. The memory unit 26 stores parameters for a calculation formula for extracting the fall risk from the frequency-analysis gait features. The display unit 30 also displays the fall risk value extracted by the fall risk assessment unit 27 to the subject, etc. The fall risk value can be displayed, for example, on a display for several seconds after the subject walks.

[0019] 2 is a hardware configuration diagram showing a fall risk extraction device 20 according to embodiment 1. The fall risk extraction device 20 is made up of a computer including a processor 210, a storage device 220, and an input / output interface 230. The fall risk extraction device 20 may be made up of multiple computers.

[0020] The processor 210 is an integrated circuit (IC) that performs arithmetic processing. Specific examples of the processor 210 include a central processing unit (CPU), a digital signal processor (DSP), and a graphics processing unit (GPU). The processor 210 functions as a skeletal information extraction unit 22 by running software capable of detecting the joints of a human body from video such as the aforementioned "Openpose." The processor 210 also functions as a multiple-joint information preprocessing unit 23 by running software that performs addition or complex expression of initial gait features between joints, a frequency analysis unit 24 by running software that performs Fourier transform and inverse Fourier transform, and a fall risk determination unit 27 by running software that extracts the subject's risk of falling.

[0021] The storage device 220 corresponds to the storage unit 26 shown in FIG. 1 and is composed of a volatile storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), or a non-volatile storage device such as a flash memory.

[0022] The input / output interface 230 includes the video input unit 21 shown in Fig. 1 and is a port to which an input device and an output device are connected. A specific example of the input / output interface 230 is a USB terminal. Input devices include the camera 10, a touch panel, a keyboard, a mouse, etc. Output devices include the display 300, a light, a speaker, a vibrator, etc.

[0023] An example of the operation of the fall risk extraction device 20 according to embodiment 1 will be described below. FIG. 3 is a flowchart showing an example of the operation of the fall risk extraction device 20 according to embodiment 1. The processing shown in FIG. 3 is started, for example, when the subject starts walking. In step S101, an image of the subject captured by the camera 10 is input to the image input unit 21.

[0024] In step S102, the skeletal information extraction unit 22 extracts skeletal information of the subject's body from the video input to the video input unit 21.

[0025] In step S103, the multiple joint information preprocessing unit 23 calculates, as initial gait features of multiple joints, time series data of the positions or angles of the skeletal joints and data obtained by differentiating the time series data of the positions or angles of the skeletal joints from the skeletal information extracted by the skeletal information extraction unit 22. The differentiated data is, for example, time series data of velocity or angular velocity. In step S103, the time series data of the positions of the skeletal joints is defined as the coordinate values ​​of one axis, two axes, or three axes of the skeletal joints as time passes during walking. Note that the coordinate values ​​are values ​​with the origin being a single point in the shooting environment or a fixed point within the image screen. The angle of a skeletal joint is defined as the amount representing the distance between two line segments with the same endpoints when there are line segments connecting a single joint to any two other joints. In addition, the time series data of the angle is defined as the angle value as time passes during walking.

[0026] In step S104, the multiple-joint information pre-processing unit 23 adds or complex-expresses the initial gait features of the multiple joints extracted in step S103 between the joints. The multiple-joint information pre-processing unit 23 extracts post-addition gait features by adding the values ​​of the initial gait features of all joints at each time, which is the time-series data of the multiple joints. When adding initial gait features, the initial gait features of two or more joints are used. Furthermore, the multiple-joint information pre-processing unit 23 extracts post-complex-expression gait features by expressing the initial gait features, which are the time-series data of two joints, at each time as a complex number in which the value of the initial gait feature of one joint is the real part and the value of the initial gait feature of the other joint is the imaginary part. When the initial gait features are complex-expressed, the initial gait features of two or more even number of joints are used. As described above, the post-addition gait features and the post-complex-expression gait features are collectively referred to as post-computation gait features, which are time-series data of walking. In the first embodiment, the initial gait features may be added any number of times, for example, by adding together the initial gait features of any number of joints (two or more). Furthermore, complex representation processing may be performed after the addition. Specifically, a post-addition processing gait feature A is prepared, which is a first added value extracted by selecting initial gait features of a suitable combination of multiple joints and adding together the selected initial gait features, and a post-addition processing gait feature B is prepared, which is a second added value extracted by adding together the initial gait features of a plurality of joints in a different combination from the post-addition processing gait feature A. A complex number is created with the post-addition processing gait feature A as the real part and the post-addition processing gait feature B as the imaginary part.

[0027] In step S105, the frequency analysis unit 24 performs frequency analysis on the post-calculation processing gait features obtained by preprocessing in the multiple-joint information preprocessing unit 23 to extract post-frequency analysis gait features. Specifically, a Fourier transform is performed as the frequency analysis of the post-addition processing gait features. Furthermore, an inverse Fourier transform is performed as the frequency analysis of the post-complex representation gait features. As described above, when a complex number is subjected to an inverse Fourier transform, real-number time series data can be calculated. The process of obtaining real-number time series data calculated by the inverse Fourier transform corresponds to frequency analysis of a trajectory in complex space, and as a result, it becomes possible to calculate the frequency characteristics of the movements of two joints. The post-frequency analysis gait features in embodiment 1 indicate an intensity distribution obtained by decomposing the post-calculation processing gait features into each frequency component contained in the gait features.

[0028] In step S106, the fall risk determination unit 27 refers to the parameters stored in the storage unit 26, and also runs a program that reproduces a calculation formula for determining the fall risk, which will be described later, and determines the fall risk from the frequency analysis results output from the frequency analysis unit 24. A specific method for configuring this program is to configure a program that performs processing similar to the calculation formula for determining the fall risk in a programming language such as C++ or Python, and incorporate the program into the software.

[0029] Examples of calculation formulas for determining the risk of a fall are the following formulas (1), (2), and (3). Formula (1) is a definition formula for the risk determination gait feature V(i,n) used in determining the risk of a fall. The risk determination gait feature V(i,n) is the negative value of the maximum amplitude of the frequency-analyzed gait feature. If the multiple-joint information preprocessing unit 23 outputs multiple post-processing gait features, there will also be multiple frequency-analyzed gait features. The i in the risk determination gait feature V(i,n) is a risk determination gait feature number that identifies each of the total I frequency-analyzed gait features. In the first embodiment, all frequency-analyzed gait features are output when the combination of joints to be added or complex-expressed is changed, and the total number of joint combinations is I. Furthermore, the n in the risk determination gait feature V(i,n) is a subject number that identifies each of the total N subjects.

[0030]

[0031] In the first embodiment, each time the risk assessment gait feature number i and the subject number n change, the negative value of the maximum amplitude of the gait feature after frequency analysis is extracted as the risk assessment gait feature V(i, n). Note that the risk assessment gait feature V(i, n) is not time-series data, but rather corresponds to one value for each variable i or n. In the first embodiment, the risk of a subject falling is determined based on the magnitude of the maximum amplitude of the frequency analysis results. For example, if the amplitude is large, it can be estimated that the subject has a healthy gait with no shaky gait and has a low risk of falling, whereas if the amplitude is small, it can be estimated that the subject has a weak gait with shaky gait and has a high risk of falling.

[0032] Equation (2) is a formula for calculating the normalized risk assessment gait feature S(i,n), which indicates a value between 0 and 100, by normalizing the i-th risk assessment gait feature V(i,n) among the risk assessment gait features calculated by equation (1) across all subjects.

[0033]

[0034] The normalization process using equation (2) normalizes the risk assessment gait features so that the maximum risk assessment gait feature is 100 and the minimum risk assessment gait feature is 0 among all subjects, total number N. Note that the normalized risk assessment gait feature S(i, n) is not a time series, but one value corresponds to each of the variables i and n.

[0035] Equation (3) is a formula for calculating the fall risk F(n), which is the weighted average of the normalized gait features for risk assessment S(i, n) extracted by equation (2) by weighting and adding all of the normalized gait features for risk assessment S(i, n) of the total number I for each subject.

[0036]

[0037] ​The weight w(i) in equation (3) is stored as a parameter in the storage unit 26. In the first embodiment, a parameter that can determine the risk of falling with higher accuracy is calculated and stored in the storage unit 26. The parameter that can determine the risk of falling with higher accuracy and is stored in the storage unit 26 is created using a method in which the determination results of the risk of falling F(n) for all possible values ​​of the parameter are output, and the parameter for which the error from the true value of the risk of falling is smaller is obtained. Note that the method for determining whether the risk of falling has been determined with high accuracy uses data from several hundred or more subjects, and uses the F value (F1 score), which will be described later, as an accuracy evaluation index.

[0038] To check whether the fall risk can be determined with high accuracy, data from several hundred or more subjects is examined to see whether there is a small error between the "true value f(n) of the fall risk calculated for each subject by a TUG (Timed Up & Go Test) test" and the "fall risk F(n) detected for each subject by the fall risk extraction device 20." The TUG test is a test that comprehensively assesses walking ability, dynamic balance, agility, etc., and can determine the fall risk from the length of the measurement time. The true value f(n) of the fall risk can also be estimated from a diagnosis by a medical therapist, but in the first embodiment, the result obtained by the TUG test, which provides a more objective numerical value, is used as the true value f(n).

[0039] The fall risk F(n) used for error evaluation is calculated for all of the hundreds of subjects using the above formulas (1) to (3) with the initial value of the weight w(i) pre-stored as a parameter in the storage unit 26. The true value f(n) used for error evaluation is calculated for all of the hundreds of subjects using the above-mentioned TUG test. In the first embodiment, if the error between the true value f(n) and the fall risk F(n) is small, it is determined as TRUE, and if it is large, it is determined as FALSE, and TRUE / FALSE is determined for all subjects.

[0040] In the TRUE / FALSE judgment described above, whether the error is small or not is judged as follows: for each subject, if the difference between the true value f(n) and the fall risk F(n) is equal to or less than the standard error SE described below, the result is TRUE (detection correct); if it is greater than the standard error SE, the result is FALSE (detection incorrect).

[0041] The standard error SE is calculated as follows: The fall risk detected by the fall risk extraction device 20 for the first subject is F(1), and the fall risks F(n) (n = 1, ..., N) detected by the fall risk extraction device 20 for all N subjects are F(1) to F(N).

[0042] The average Ea of F(1) to F(N) is calculated using the following formula (4): Then, the variance Sa2 of F(1) to F(N) is calculated by dividing the sum of squares of the deviations between F(1) to F(N) and the average Ea by the total number of subjects N using the following formula (5):

[0043]

[0044]

[0045] As shown in the following equation (6), the standard deviation Sa of F(1) to F(N), which is the square root of the variance Sa2, is calculated, and as shown in equation (7), the standard deviation Sa is divided by the square root of the total number of subjects N to obtain the standard error SE.

[0046]

[0047]

[0048] In the first embodiment, an F-value is calculated, which is the harmonic mean of the recall and precision obtained based on the binary classification of the TRUE / FALSE determination results of all subjects using the standard error SE calculated by Equation (7).The accuracy of the fall risk F(n) is then evaluated based on the magnitude of the calculated F-value.

[0049] To evaluate the error between the fall risk F(n), which is the fall risk assessment result, and the true fall risk value f(n), the root mean squared error (RMSE) of the errors of all subjects shown in the following formula (8) may be calculated. Alternatively, a machine-learned model capable of determining the similarity of data distribution may be used to evaluate the error based on the similarity of the distributions of the fall risk F(n) and the true fall risk value f(n).

[0050]

[0051] In step S107, the calculated fall risk F(n) is displayed to the subject. Specifically, the numerical value F(n) for the fall risk is displayed on the display 300. The display color of the display 300 may be changed depending on the value of the fall risk F(n), allowing the subject to intuitively recognize the risk of falling.

[0052] As described above, according to the first embodiment, a video of walking is acquired, a human body skeleton is extracted from the video, initial gait features of the skeleton are extracted, and data obtained by adding or complex-expressing the initial gait features of multiple joints is frequency-analyzed, thereby making it possible to extract a fall risk F(n) taking into account the correlation between the movements of multiple parts. The extracted fall risk F(n) makes it possible to evaluate the risk of falling in relation to the linkage of the movements of multiple parts, and as a result, a more detailed diagnosis of the risk of falling can be made during medical examinations or follow-up observations of rehabilitation.

[0053] In the first embodiment, the negative value of the maximum amplitude of the frequency-analyzed gait feature is applied to the risk-assessment gait feature V(i,n), but this is not limiting. The positive value of the maximum amplitude of the frequency-analyzed gait feature may also be applied to the risk-assessment gait feature V(i,n). In this case, the lower the value of the fall risk F(n), the greater the fall risk.

[0054] In the first embodiment, an example has been described in which the risk of falling F(n) of a subject is calculated based on the gait characteristics of the subject's multiple joints. In order to determine the walking state of the subject, it is desirable to take into account not only the gait characteristics of the subject's multiple joints but also the double support period, which is the time when both feet of the subject are in contact with the ground.

[0055] 7 is a functional configuration diagram showing a fall risk extraction device 200 according to embodiment 2. The fall risk extraction device 200 further includes a double leg support period detection unit 25 that detects the double leg support period of the subject based on the initial gait characteristics of the joints of both legs obtained from the skeletal information extracted by the skeletal information extraction unit 22, and differs from the fall risk extraction device 20 according to embodiment 1 in that the fall risk determination unit 28 determines the risk of a fall based on the frequency analysis results output by the frequency analysis unit 24 and the double leg support period output by the double leg support period detection unit 25. However, since the other configuration is the same as that of the fall risk extraction device 20 according to embodiment 1, the same components as those of the fall risk extraction device 20 are assigned the same reference numerals as those of the fall risk extraction device 20, and detailed description thereof will be omitted. Furthermore, since the hardware configuration of embodiment 2 is the same as that of embodiment 1, detailed description thereof will be omitted.

[0056] In Figure 7, as in embodiment 1, the skeletal information extraction unit 22 extracts the three-axis coordinates of the joints of the human body as skeletal information from the video input via the video input unit 21, and outputs the extracted skeletal information to the multiple joint information pre-processing unit 23 and the double leg support period detection unit 25.

[0057] The double-leg support phase detection unit 25 calculates initial gait characteristics of both ankles of the subject from the skeletal information extracted by the skeletal information extraction unit 22. The initial gait characteristics are, for example, time-series data on the position or angle of each ankle and data obtained by differentiating the time-series data on the position or angle of each ankle. The differential value of the ankle position data is velocity, and the differential value of the ankle angle is angular velocity. In the second embodiment, the time-series data on the position of each ankle is the coordinate values ​​of each ankle on one, two, or three axes as time passes during walking. In the case of three axes, for example, a point in the imaging environment or a fixed point on the screen of the video is set as the origin, and x- and y-axes are set horizontally from the origin, and z-axis is set vertically. Furthermore, for example, the x-axis is set east-west, and the y-axis is set north-south. The ankle angle is a quantity that represents the distance between two line segments with the same endpoint when there are line segments connecting the left or right ankle to any two joints other than the ankle in question. The time-series data of the angle is the angle value over time during walking. In the second embodiment, the double-leg support period of the subject is detected based on the ankle speeds of the left and right ankles, among the initial gait characteristics of both ankles of the subject.

[0058] FIG. 9( a) shows an example of calculating the double leg support period 50 for a subject who is not at risk of falling. FIG. 9( b) shows an example of calculating the double leg support period 52 for a subject who is at risk of falling. In FIGS. 9( a) and 9(b), the velocity changes of the right and left ankles in the direction of travel during walking are output. In the second embodiment, as shown in FIGS. 9( a) and 9(b), the double leg support periods 50 and 52 are defined as the time during which the velocity of each of the right and left ankles is equal to or less than a certain threshold T. The threshold T is specifically determined, for example, by comparing and balancing the subject's actual walking state with the velocity of each of the subject's right and left ankles. Comparing FIGS. 9( a) and 9(b) reveals that the double leg support period 52 in FIG. 9(b) is longer than the double leg support period 50 in FIG. 9(a), suggesting that a longer double leg support period increases the risk of falling. In FIGS. 9( a) and 9(b), the initial gait features used to detect the double-leg support period are the changes in ankle velocity in the direction of travel during walking. However, time-series data of the left and right directions during walking, the position of any joint in the vertical direction, and acceleration, or time-series data of the angle or angular velocity of each ankle joint on the left and right sides may also be used.

[0059] The fall risk determination unit 28 extracts the subject's fall risk based on the frequency analysis results output by the frequency analysis unit 24, the double support period output by the double support period detection unit 25, and the data stored in the memory unit 26. The subject's fall risk is an index whose value ranges from a minimum of 0 to a maximum of 100, with larger values ​​indicating a higher fall risk and smaller values ​​indicating a lower fall risk. The memory unit 26 stores parameters for a calculation formula for extracting the fall risk from gait features after frequency analysis. The display unit 30 also displays the fall risk value extracted by the fall risk determination unit 28 on the display 300 or the like to notify the subject. For example, the display 300 may display a word such as "Fine" to indicate a low fall risk, or a word such as "Frail" to indicate a high fall risk.

[0060] An example of the operation of the fall risk extraction device 200 according to embodiment 2 will now be described. Fig. 8 is a flowchart showing an example of the operation of the fall risk extraction device 200 according to embodiment 2. Steps S201, S202, S203, S204, and S205 in Fig. 8 are the same as steps S101, S102, S103, S104, and S105 in the flowchart of embodiment 1 shown in Fig. 3, and therefore detailed description thereof will be omitted.

[0061] In step S206, the double-leg supporting phase detection unit 25 extracts initial gait features of both ankles from the skeletal information extracted by the skeletal information extraction unit 22. As described above, the initial gait features of both ankles extracted by the double-leg supporting phase detection unit 25 include, for example, the velocity of each of the left and right ankles.

[0062] In step S207, the double supporting phase detection unit 25 detects the double supporting phase from the initial gait features of the joints of both legs extracted in step S206. As described above, in the second embodiment, the double supporting phases 50 and 52 are defined as the times when the velocities of the right ankle and the left ankle are equal to or less than a certain threshold value T, as shown in Figures 9(a) and 9(b).

[0063] In step S208, the fall risk determination unit 28 refers to the parameters stored in the memory unit 26 and runs a program that reproduces a calculation formula for determining the fall risk, which will be described later, and determines the fall risk from the frequency analysis results output from the frequency analysis unit 24. A specific method for configuring this program is to configure a program in a programming language such as C++ or Python that performs processing similar to the calculation formula for determining the fall risk and incorporate the program into the software.

[0064] Examples of calculation formulas for determining the risk of a fall are the above-mentioned formulas (2) and (3), and the following formula (9). Formula (9) is a definition formula for risk determination gait features V(i,n) used in fall risk determination. The risk determination gait features V(i,n) are the negative value of the maximum amplitude of the frequency-analysis gait features or the double-leg support period. The i in the risk determination gait features V(i,n) is a risk determination gait feature number that identifies each of the total I frequency-analysis gait features. In the second embodiment, for the total number I of the numbers i in formula (9), all frequency analysis results or double-leg support periods are output when the combination of joints to be added or complex-expressed is changed, and I is the sum of the total number of joint combinations (I-1) and the number of double-leg support periods, 1. Furthermore, the n in the risk determination gait features V(i,n) is a subject number that identifies each of the total N subjects.

[0065]

[0066] In the second embodiment, as in the first embodiment, the maximum amplitude of the frequency analysis results is used to determine the risk of falling. A large amplitude indicates a healthy gait with no gait fluctuations, indicating a low risk of falling. A small amplitude indicates a weak gait with gait fluctuations, indicating a high risk of falling. The risk of falling is also determined based on the magnitude of the double support period. A small double support period indicates a healthy gait with smooth transitions between the stance and swing legs of the left and right legs, indicating a low risk of falling. A large double support period indicates a weak gait with insufficient transitions between the stance and swing legs of the left and right legs, indicating a high risk of falling. Specifically, a large double support period indicates a state in which both feet are in contact with the ground and neither foot can move forward. A prolonged period of this state indicates a disruption in the coordinated movements of walking. Therefore, a large double support period indicates a high risk of falling, whereas a small double support period indicates a low risk of falling.

[0067] As described above, formula (2) is a calculation formula for calculating the normalized gait features for risk assessment S(i,n) that indicate values ​​between 0 and 100 by normalizing the i-th gait features for risk assessment V(i,n) across all subjects. In the second embodiment, the gait features for risk assessment V(i,n) calculated using the above formula (9) are applied to formula (2).

[0068] As explained in the first embodiment, the normalization process using equation (2) normalizes the risk assessment gait features so that the maximum risk assessment gait feature is 100 and the minimum risk assessment gait feature is 0 among all the subjects, total number N. Note that the normalized risk assessment gait feature S(i, n) is not a set of multiple values ​​such as a time series, but corresponds to one value for each of the variables i and n.

[0069] As described in the first embodiment, equation (3) calculates the fall risk F(n) by weighting and adding the normalized gait features for risk assessment S(i,n) extracted by equation (2) for each subject to all of the total number I of normalized gait features for risk assessment S(i,n). The weight w(i) in equation (3) is stored as a parameter in the storage unit 26. In the second embodiment, as in the first embodiment, parameters that can more accurately determine the risk of falling are calculated and stored in the storage unit 26.

[0070] Parameters that can determine the risk of falling with higher accuracy and are stored in the memory unit 26 are created by outputting the fall risk F(n) determination results for all possible values ​​of the parameters and acquiring the parameters that result in the smallest error from the true value of the risk of falling, as in embodiment 1. The error between the fall risk F(n), which is the fall risk determination result, and the true value of the risk of falling f(n) is evaluated by calculating the root mean square error (RMSE) of the errors for all subjects, using the F value as an accuracy evaluation index, as in embodiment 1, or by using a machine-learned model that can determine the similarity of the distribution of data.

[0071] In step S209, the calculated fall risk F(n) is displayed to the subject. Specifically, the numerical value F(n) for the fall risk is displayed on the display 300. The display color of the display 300 may be changed depending on the value of the fall risk F(n), allowing the subject to intuitively recognize the risk of falling.

[0072] As described above, according to the second embodiment, the risk of falling F(n) can be calculated based on the gait characteristics of the subject's multiple joints and the double support phase. The calculated risk of falling F(n) makes it possible to evaluate the risk of falling while taking into account the interlocking of the movements of multiple parts and the smoothness of the transition between the stance leg and the swing leg of the left and right legs. As a result, a more detailed diagnosis of the risk of falling can be made during medical examinations or follow-up observations of rehabilitation.

[0073] In the second embodiment, the walking features of both ankles are extracted by the double-leg supporting phase detection unit 25. However, the present invention is not limited to this. The walking features of both ankles may be extracted by the multiple-joint information preprocessing unit 23.

[0074] <<Embodiment 3>> In Embodiment 1, the gait features including the correlation between the movements of a plurality of joints are frequency analyzed, and the weighted average of the normalized maximum amplitudes of the obtained frequency characteristics is calculated to determine the risk of a fall of the subject. However, in Embodiment 3, the fall risk is calculated using the "correlation coefficient of features between joints." The configuration of the fall risk extraction device according to Embodiment 3 is the same as the fall risk extraction device 20 according to Embodiment 1, so a detailed description thereof will be omitted. Furthermore, the hardware configuration of Embodiment 2 is the same as the hardware configuration of Embodiment 1, so a detailed description thereof will be omitted.

[0075] The "joint features" are, for example, time-series data of joint displacement extracted by the multiple-joint information preprocessing unit 23, or frequency spectrum data of frequency characteristics. In the third embodiment, it is assumed that one subject has joint features A and B. When the first group of features A includes A1 to An, the second group of features B includes B1 to Bn, and each of the features A and B is made up of n pieces of data, the correlation coefficient is calculated by the following procedure.

[0076] For feature A, the average Ea of A1 to An is calculated, and the deviation of each of A1 to An from the average Ea is calculated. The sum of the squares of each of the calculated deviations is divided by n to calculate the variance Sa2 of A1 to An, and then the standard deviation Sa, which is the square root of variance Sa2, is calculated. Similarly, for feature B, the average Eb of B1 to Bn, the deviation of each of B1 to Bn from the average Eb, the variance Sb2 of B1 to Bn, and the standard deviation Sb are calculated.

[0077] The covariance Sab of feature A and feature B is calculated as the average of the product of the deviation between each of A1 to An and Ea and the deviation between each of B1 to Bn and Eb.

[0078] The correlation coefficient r is obtained by dividing the covariance Sab by the product of the standard deviation Sa and the standard deviation Sb, as shown in the following equation (10).

[0079]

[0080] In principle, one correlation coefficient r can be calculated for each combination of two features, such as features A and B, for each subject. However, since each joint has multiple gait features, such as vertical displacement, forward velocity, and angular displacement around the vertical axis, multiple correlation coefficients r can be calculated using these multiple gait features. The correlation coefficient r has a value within the range of -1≦r≦1, and the closer r is to 1, the more positive the correlation between features A and B is, and the closer r is to -1, the more negative the correlation between features A and B is. Furthermore, when r is close to 0, there is no correlation between features A and B.

[0081] In the third embodiment, the calculated correlation coefficient r is used for detecting the risk of falling, similar to the maximum amplitude of the frequency characteristics or the double support period. For example, suppose that feature A is time series data of displacement amounts of the position of the subject's right ankle joint that are equal to or greater than a predetermined threshold, and feature B is time series data of displacement amounts of the position of the subject's left ankle joint that are equal to or greater than a predetermined threshold. If there is a difference in the displacement amounts of the position of the subject's left and right ankle joints, it can be estimated that the subject's gait is unstable. In such a case, the correlation coefficient r calculated for features A and B is a negative value close to -1, indicating a negative correlation between the displacement amounts of the subject's left and right ankle joints. Conversely, if there is no difference in the displacement amounts of the subject's left and right ankle joints, it can be estimated that the subject's gait is stable. In such a case, the correlation coefficient r calculated for features A and B is a negative value close to 1, indicating a positive correlation between the displacement amounts of the subject's left and right ankle joints.

[0082] The "skeletal information extraction unit" in the claims corresponds to the "skeletal information extraction unit 22" described in the detailed description of the invention, the "joint information extraction unit" and "gait feature calculation unit" in the claims correspond to the "multiple joint information preprocessing unit 23" described in the detailed description of the invention, the "frequency analysis unit" in the claims corresponds to the "frequency analysis unit 24" described in the detailed description of the invention, the "fall risk extraction unit" in the claims corresponds to the "fall risk determination units 27, 28" described in the detailed description of the invention, and the "double legs supporting period detection unit" in the claims corresponds to the "double legs supporting period detection unit 25" described in the detailed description of the invention.

[0083] 10 Camera, 20 Fall risk extraction device, 21 Video input unit, 22 Skeleton information extraction unit, 23 Multiple joint information preprocessing unit, 24 Frequency analysis unit, 25 Double support period detection unit, 26 Storage unit, 27, 28 Fall risk determination unit, 200 Fall risk extraction device

Claims

1. A fall risk extraction device comprising: a skeletal information extraction unit that extracts skeletal information of a subject from a video of the subject walking; a joint information extraction unit that extracts information from the skeletal information including each of time-series position changes, angle changes, speed changes, and angular velocity changes at each joint while the subject is walking; a gait feature calculation unit that calculates gait features including correlations between the movements of multiple joints based on the information extracted by the joint information extraction unit; and a fall risk extraction unit that extracts the subject's risk of falling based on the gait features including correlations between the movements of the multiple joints.

2. The fall risk extraction device of claim 1, wherein the gait feature calculation unit calculates gait features including correlations between the movements of multiple joints by either adding up the values ​​of information for each of multiple different joints at the same time from the information extracted by the joint information extraction unit, or by using a complex expression for information for each of two different joints at the same time, with the value of the information for one joint being the real part and the value of the information for the other joint being the imaginary part; and the fall risk extraction unit includes a frequency analysis unit that performs frequency analysis of the gait features including correlations between the movements of the multiple joints to calculate frequency characteristics of the gait features; and the fall risk extraction unit determines the risk of the subject falling based on a fall risk value obtained by calculating a weighted average of the maximum amplitude value of the frequency characteristics of the subject normalized by the maximum amplitude values ​​of the frequency characteristics of multiple subjects including the subject.

3. A fall risk extraction device as described in claim 2, wherein the gait characteristics calculation unit selects and adds any number of pieces of information relating to two or more different joints at the same time from the information extracted by the joint information extraction unit.

4. The fall risk extraction device of claim 3, wherein the gait feature calculation unit calculates gait features including correlations between movements of the multiple joints using complex expressions in which a first sum obtained by adding up each of an arbitrary number of combinations of information related to two or more different joints at the same time is defined as a real part, and a second sum obtained by adding up each of a combination of information different from the combination of information used to calculate the first sum is defined as an imaginary part.

5. A fall risk extraction device as described in any one of claims 2 to 4, further comprising a double leg support period detection unit that detects a double leg support period, which is the time when both feet of the subject are in contact with the ground, from the information extracted by the joint information extraction unit, and the fall risk extraction unit determines the risk of the subject falling based on the frequency characteristics and the double leg support period.

6. A fall risk extraction device as described in claim 1, wherein the fall risk extraction unit extracts a first group including a plurality of data and a second group including a plurality of data different from the first group from the information extracted by the joint information extraction unit, and determines the subject's risk of falling based on a correlation coefficient obtained by dividing the covariance of the first group and the second group by the product of the standard deviation of the first group and the standard deviation of the second group.

7. A fall risk extraction method, comprising: a fall risk extraction device for extracting a subject's risk of falling, extracting skeletal information of the subject from a video of the subject walking; extracting from the skeletal information information including each of time-series changes in position, angle, speed, and angular velocity at each joint while the subject walks; calculating gait characteristics including correlations between movements of a plurality of joints based on the extracted information; and extracting the subject's risk of falling based on the gait characteristics including correlations between movements of the plurality of joints.

8. A fall risk extraction program that causes a fall risk extraction device that extracts a subject's risk of falling to execute the following processes: extracting skeletal information of the subject from a video of the subject walking, extracting from the skeletal information information including each of time-series position changes, angle changes, speed changes, and angular velocity changes at each joint while the subject is walking, calculating gait characteristics including correlations between movements of multiple joints based on the extracted information, and extracting the subject's risk of falling based on the gait characteristics including correlations between movements of the multiple joints.

Images