Method and system for analyzing and identifying falling behavior of patient in hospital
By analyzing changes in skeletal coordinates in videos of falls and normal activities of elderly patients, the instability of the lower and upper sides was quantified, the main side of imbalance and the degree of body imbalance were identified, and neural network training was combined to solve the problem of low accuracy in recognizing fall behavior in elderly patients, thus achieving early risk identification and accurate monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-10
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing technologies fail to effectively consider individual differences when identifying fall behavior in elderly patients, resulting in low accuracy. Furthermore, traditional monitoring methods are often lagging and unable to provide early warnings.
By acquiring skeletal coordinates from fall events and normal activity videos, the differences in ankle and shoulder coordinates relative to the hip center are analyzed to quantify instability on the lower and upper sides, identify the main side of imbalance and the degree of body imbalance, and combine this with neural network training to recognize fall behavior.
It improves the accuracy of fall behavior recognition, enables earlier identification of fall risks, reduces missed detections, and enhances monitoring effectiveness.
Smart Images
Figure CN121838271A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fall behavior recognition technology, specifically to a method and system for analyzing and recognizing fall behavior in hospitalized patients. Background Technology
[0002] Elderly patients are at high risk of falls due to age-related factors such as decreased muscle strength, impaired balance, multiple medications, and cognitive impairment. Falls not only cause direct injuries like fractures and traumatic brain injury, but also prolong hospital stays, increase medical costs, and can even be a significant cause of chronic disability in the elderly. The proportion of falls among hospitalized patients over 85 years of age is relatively high, and many elderly patients also have two or more underlying medical conditions, further increasing the risk of falls.
[0003] Traditional monitoring methods rely heavily on nurse rounds, bedside calls, or simple sensors. These methods suffer from significant delays, with most devices only triggering alarms after a patient falls, failing to provide early detection and warnings of the fall process. Furthermore, existing visual monitoring-based neural network models analyzing video data are prone to missed detections in complex ward environments due to variations in the physical condition, balance, and disease effects of elderly patients, resulting in differences in the amplitude, speed, and posture of their falls. Summary of the Invention
[0004] To address the technical problem of low accuracy in identifying fall-related abnormal behaviors due to the use of the same weights without considering individual differences among the elderly population, the present invention aims to provide a method and system for analyzing and identifying fall behavior in hospitalized patients. The specific technical solution adopted is as follows: This invention proposes a method for analyzing and identifying fall behavior in hospitalized patients, the method comprising: Obtain the skeletal coordinates of each frame in the fall event video and the normal activity video. The skeletal coordinates include the shoulder coordinates, ankle coordinates, and hip center coordinates on both the left and right sides. For any given video, the lower instability of each time frame is obtained based on the positional distribution of the ankle coordinates in different time frames; the upper instability of each time frame is obtained based on the difference in the change of the shoulder coordinates relative to the hip center coordinates between different adjacent time frames. For any given video, the main imbalance side of each video is obtained based on the variation characteristics of the upper side instability relative to the lower side instability in different video frames; based on the distribution of the main imbalance side of different videos and the distribution of the upper and lower sides of instability corresponding to different time frames, the degree of body imbalance of each video at each time frame is obtained. Based on the instability of the main imbalance side corresponding to each video at each time frame, and the degree of body imbalance, an anomaly significance score is obtained for each video at each time frame. After training the neural network based on the skeletal coordinates of all videos at different time frames and the anomaly salience score, it identifies fall behavior in newly acquired videos.
[0005] Furthermore, the method for obtaining the lower-side instability includes: For any video, the stride length of each frame is obtained based on the relative distance distribution of the ankle coordinates between the left and right sides at different time frames. Based on the degree of difference in stride fluctuation between all time frames before and after each time frame, and the difference in stride between each time frame and the previous time frames, the stride change rate of each time frame is obtained. The degree of difference in stride fluctuation is negatively correlated with the stride change rate, and the difference in stride is positively correlated with the stride change rate. The maximum value of the stride fluctuation difference corresponding to other time frames before each time frame is selected, and the corresponding other time frames are taken as the relative gait change frames of each time frame; the difference between the average stride change rate of all time frames before and after the relative gait change frames is obtained, and the maximum value between the absolute value of the difference and the preset difference threshold is selected as the lower side instability of each time frame.
[0006] Furthermore, the method for obtaining the stride length includes: For any video, obtain the relative distance between the ankle coordinates of the left and right sides in each time frame; select the maximum value of the curve formed by the relative distances of all time frames, and select the relative distance of the time frame corresponding to the next maximum value after each time frame as the stride of each time frame.
[0007] Furthermore, the method for obtaining the upper-side instability includes: For any side of the skeleton, based on the difference in the change of the shoulder coordinate relative to the hip center coordinate between different adjacent time frames, the same period time frame is obtained. The upper instability of each time frame is obtained by considering the number of differences between all time frames of the same period on the left and right sides corresponding to each time frame, as well as the difference in the amplitude of the shoulder swing vector. Both the number of differences and the difference in amplitude are positively correlated with the upper instability.
[0008] Furthermore, the method for obtaining the time frame within the same period includes: For any side of the skeleton, obtain the difference between the shoulder coordinate and the hip center coordinate in each frame, and use it as the shoulder-hip displacement vector; The difference between the shoulder and hip displacement vectors between each time frame and the previous time frame is obtained as the shoulder swing vector; if the angle between the shoulder swing vectors between adjacent time frames is less than or equal to a preset angle threshold, the corresponding time frame is taken as the time frame of the same period.
[0009] Furthermore, the method for obtaining the main imbalance side includes: For any video, the instability dominance of each side is obtained based on the difference in instability on each side and the other side between different time frames and the previous time frame, as well as the difference in instability between each side and the other side between different time frames. Both the difference in change and the difference in instability are positively correlated with the instability dominance. Select the side with the greatest instability between the upper and lower sides and take that side as the main unbalanced side.
[0010] Furthermore, the method for obtaining the degree of body imbalance includes: Based on the distribution of the main imbalance sides of different videos and the distribution of instability on the upper and lower sides of frames at different time points, the relative minimum instability compensation degree of each video relative to each normal activity video at each time point frame is obtained. Obtain the relative minimum instability compensation degree of each video at each time frame corresponding to each normal activity video, and the difference between the instability of the non-major imbalance side; obtain the average difference of the difference greater than zero in all normal activity videos with the same major imbalance side corresponding to each video at each time frame, as the body imbalance degree of each video at each time frame; if there is no difference greater than zero, set the body imbalance degree to 0.
[0011] Furthermore, the method for obtaining the relative minimum instability compensation degree includes: Obtain the instability difference of the main imbalance side between each time frame in each video and different time frames in each normal activity video. Select the time frame with the smallest corresponding instability difference in each normal activity video, and take the instability of the non-main imbalance side of the corresponding time frame as the relative minimum instability compensation degree of each normal activity video.
[0012] Furthermore, the method for obtaining the anomaly significance score includes: Based on the instability of the main imbalance side of each video at each time frame, the corresponding body imbalance degree is adjusted by a loss adjustment to obtain the body imbalance weighted value of each video at each time frame. The sum of the instability and body imbalance weights of the main imbalance side for each video at each time frame is obtained as the anomalous significance score for each video at each time frame.
[0013] The present invention also proposes an in-hospital patient fall behavior analysis and recognition system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the steps of the in-hospital patient fall behavior analysis and recognition method described above.
[0014] The present invention has the following beneficial effects: This invention, for any given video, obtains the lower-side instability of each frame based on the distribution of ankle coordinates at different time points, quantifying the instability such as swaying and shifting of the lower body during movement; it obtains the upper-side instability of each frame based on the difference in shoulder coordinates relative to hip center coordinates between different adjacent time points, more accurately describing the dynamic changes in upper-side instability; for any given video, it obtains the main imbalance side of each video based on the changing characteristics of upper-side instability relative to lower-side instability in different video frames, helping to more accurately analyze the causes and mechanisms of body imbalance; based on the distribution of the main imbalance side in different videos, and the distribution of upper and lower-side instability corresponding to different time points, it obtains the degree of body imbalance in each video at each time point, more comprehensively reflecting the imbalance characteristics of each time point; based on the instability of the main imbalance side corresponding to each video at each time point, and the degree of body imbalance, it obtains the anomaly significance score of each video at each time point, which can more accurately reflect the degree of anomaly in each video at each time point; and it identifies abnormal fall behavior. This invention improves the accuracy of identifying abnormal fall behavior by accurately obtaining the anomaly salience score for each frame in a video. Attached Figure Description
[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A flowchart illustrating a method for analyzing and identifying fall behavior in hospitalized patients, as provided in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for obtaining bottom-side instability according to an embodiment of the present invention. Detailed Implementation
[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a method and system for analyzing and identifying patient falls in hospitals according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.
[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0019] The following description, in conjunction with the accompanying drawings, details the specific scheme of the in-hospital patient fall behavior analysis and identification method and system provided by the present invention.
[0020] Please see Figure 1 The diagram illustrates a flowchart of a method for analyzing and identifying in-hospital patient fall behavior according to an embodiment of the present invention, specifically including: Step S1: Obtain the skeletal coordinates of each frame in the fall event video and the normal activity video. The skeletal coordinates include the shoulder coordinates, ankle coordinates, and hip center coordinates on both the left and right sides.
[0021] In embodiments of the present invention, in the identification of abnormal behavior during falls in the elderly, relying solely on the action recognition at the moment of the fall is often insufficient to accurately identify the risk. To distinguish between falls and normal activities, it is necessary to analyze the elderly's behavior in both fall videos and normal activity videos. First, the MediaPipe Pose algorithm is used to obtain the shoulder coordinates, ankle coordinates, and hip coordinates on both sides of the human body in each frame of the video. The center coordinates of the hips are obtained by calculating the midpoint of the hip coordinates on both sides. Therefore, the skeletal coordinates of each frame in the fall event video and the normal activity video are obtained, including the shoulder coordinates, ankle coordinates, and center coordinates of the hips on both sides.
[0022] It should be noted that the MediaPipe Pose algorithm is a real-time human pose estimation model that obtains the coordinates of multiple human joints; the specific MediaPipe Pose algorithm is a well-known technique in the art and will not be described in detail here.
[0023] Step S2: For any video, obtain the lower instability of each time frame based on the position distribution of ankle coordinates in different time frames; obtain the upper instability of each time frame based on the difference in the change of shoulder coordinates relative to hip center coordinates between different adjacent time frames.
[0024] The ankle joint is a key part of the human lower limb that contacts the ground, and its positional changes can directly reflect the stability of the lower side of the body. Insufficient muscle strength or limited movement can lead to a small stride, while instability of the center of gravity can cause loss of gait control, resulting in an excessively large or abnormally enlarged stride. Individuals may experience problems such as unstable rhythm and decreased motor control during walking, leading to unstable stride changes. By analyzing the positional distribution of ankle coordinates in different time frames, the lower side instability can be quantified. For any video, the lower side instability of each time frame can be obtained based on the positional distribution of ankle coordinates in different time frames.
[0025] Preferably, in one embodiment of the present invention, the method for obtaining the lower-side instability is described in [reference needed]. Figure 2 It illustrates a flowchart of a method for obtaining lower-side instability, including: Step S201: For any video, obtain the stride of each frame based on the relative distance distribution of ankle coordinates between the left and right sides of frames at different time points.
[0026] Preferably, in one embodiment of the present invention, the method for obtaining stride length includes: For any video, obtain the relative distance between the ankle coordinates of the left and right sides in each time frame. Select the maximum value of the curve formed by the relative distances of all time frames, and then select the relative distance of the time frame corresponding to the next maximum value after each time frame as the stride length for that time frame. It should be noted that the relative distance between the ankle coordinates of the left and right sides in each time frame reflects the stride width when walking at that time frame. The maximum value of the relative distance reflects the largest difference in distance between the two ankles during walking, i.e., the stride amplitude. Therefore, the relative distance of the time frame corresponding to the next maximum value after each time frame is selected to reflect the stride length for that time frame.
[0027] It should be noted that, in the embodiments of the present invention, the relative distance is obtained by existing distance calculation methods such as Euclidean distance or Manhattan distance; the maximum value can be obtained by Newton's method or AMPD algorithm; the specific means are technical means well known to those skilled in the art, and will not be described in detail here.
[0028] Step S202: Based on the degree of stride fluctuation difference between all time frames before and after each time frame, and the stride difference between each time frame and the previous time frames, obtain the stride change rate of each time frame. The degree of stride fluctuation difference is negatively correlated with the stride change rate, and the stride difference is positively correlated with the stride change rate.
[0029] It should be noted that the degree of stride fluctuation difference between all time frames preceding and following other time frames reflects the stability of walking. The greater the difference in stride fluctuation, the more inconsistent the preceding and following behavioral states, the worse the stability, and the more unstable or adjusting the gait itself is. This makes the analysis of stride changes in those time frames less reliable. Conversely, the smaller the stride change rate, the smaller the difference in stride fluctuation, the more consistent the preceding and following behavioral states, and the greater the stability. The more normal the other time frames are, the more reliable the analysis of stride changes in those frames. The difference in stride between each time frame and all previous time frames reflects the stride changes between frames. The greater the stride difference, the greater the stride change rate. Therefore, the degree of stride fluctuation difference is negatively correlated with the stride change rate, while the stride difference is positively correlated with the stride change rate.
[0030] In one embodiment of the present invention, the degree of stride fluctuation difference between all time frames before and after each time frame is obtained and negatively correlated to form the stride change confidence level; the stride change confidence level of all other time frames before each time frame and the multiplication and summation of the stride difference between each time frame and the previous time frames are obtained to form the stride change rate of each time frame; therefore, based on the above basic mathematical operations, the correlation between the degree of stride fluctuation difference, the stride difference, and the stride change rate is constructed; that is, the smaller the degree of stride fluctuation difference, the larger the stride difference, and the larger the stride change rate.
[0031] It should be noted that, in one embodiment of the present invention, the following is employed: The function performs a negative correlation mapping. The greater the difference in step amplitude fluctuation, the lower the reliability of step amplitude change. In other embodiments of the present invention, negative correlation mapping can also be performed by taking the reciprocal. The specific means are well known to those skilled in the art and will not be described in detail here.
[0032] It should be noted that, in one embodiment of the present invention, the variance of the stride is used to reflect stride fluctuation. The larger the variance, the larger the stride fluctuation, and the smaller the variance, the smaller the stride fluctuation. In other embodiments of the present invention, the standard deviation or range can also be used to reflect stride fluctuation. The specific means are well known to those skilled in the art and will not be limited or described here.
[0033] Step S203: Select the maximum value of the stride fluctuation difference corresponding to other time frames before each time frame, and take the corresponding other time frames as the relative gait change frames of each time frame; obtain the difference between the average stride change rate of all time frames before and after the relative gait change frames, and select the maximum value between the absolute value of the difference and the preset difference threshold as the lower side instability of each time frame.
[0034] The greater the difference in stride fluctuation, the more different the stride changes before and after a time frame, and the greater the possibility of changes occurring before and after the corresponding frame. By analyzing the difference in the average stride change rate of all time frames before and after the relative gait change frame, the larger the absolute value of the difference, the greater the stride change rate after the change is compared to before the change, and the greater the gait instability.
[0035] It should be noted that, in the embodiments of the present invention, the size of the preset difference threshold can be set according to specific circumstances. In one embodiment of the present invention, the preset difference threshold is set to 0.
[0036] In a normal gait, the shoulders swing rhythmically from side to side with the movement of the lower limbs to maintain body balance. However, when muscle strength decreases, joints become stiff, and balance is impaired, the shoulder swing exhibits a reduced amplitude and asymmetry. By analyzing the differences in the shoulder coordinate relative to the hip center coordinate between different adjacent time frames, the dynamic process of upper body balance loss before a fall can be directly captured. Based on the differences in the shoulder coordinate relative to the hip center coordinate between different adjacent time frames, the upper instability of each time frame can be obtained.
[0037] Preferably, in one embodiment of the present invention, the method for obtaining the upper-side instability includes: For any side of the skeleton, based on the difference in the change of the shoulder coordinate relative to the hip center coordinate between different adjacent time frames, the same period time frame is obtained. It should be noted that the center point of the hip reflects the overall forward movement of the body. When analyzing the swing amplitude of the shoulder, it is necessary to exclude the influence of the overall forward trend of the body and perform local analysis on time frames with similar swing patterns. In one embodiment of the present invention, the method for obtaining time frames of the same period includes: For any side of the skeleton, obtain the difference between the shoulder coordinate and the hip center coordinate in each frame, and use it as the shoulder-hip displacement vector; The difference between the shoulder and hip displacement vectors between each time frame and the previous time frame is obtained as the shoulder swing vector; if the angle between the shoulder swing vectors between adjacent time frames is less than or equal to a preset angle threshold, the corresponding time frame is taken as the time frame of the same period.
[0038] It should be noted that, in one embodiment of the present invention, the preset angle threshold is set to 60°; in other embodiments of the present invention, the size of the preset angle threshold can be set according to specific circumstances, and will not be limited or elaborated here.
[0039] The upper instability of each time frame is obtained by considering the number of differences between all time frames of the same period on the left and right sides corresponding to each time frame, as well as the difference in the amplitude of the shoulder swing vector. Both the number of differences and the difference in amplitude are positively correlated with the upper instability.
[0040] It should be noted that the number of differences between all time frames of the same period between the left and right sides reflects the synchronicity of the swing between the left and right sides. The greater the number of differences in the corresponding time frames, the more asymmetrical the gait and the greater the instability of the upper side. The greater the difference in the amplitude of the shoulder swing vector, the greater the difference in the swing amplitude between the left and right sides, the less it conforms to the changes under normal gait, and the greater the instability of the upper side. The number and amplitude differences are positively correlated with the instability of the upper side.
[0041] In one embodiment of the present invention, for each side, the magnitude of the sum of shoulder swing vectors corresponding to all time frames of the same period is calculated as the shoulder swing amplitude; the number of differences is normalized, and the product between the normalization result and the difference between the shoulder swing amplitudes of the two sides is calculated as the upper side instability of each time frame; therefore, based on the above basic mathematical operations, a correlation is constructed between the number of differences, the amplitude difference and the upper side instability, that is, the larger the number of differences, the larger the amplitude difference, the more inconsistent the swing situation, and the greater the upper side instability.
[0042] It should be noted that, in the embodiments of the present invention, linear normalization or a normalization function is used to normalize the number of differences. The specific means are well known to those skilled in the art and will not be described in detail here.
[0043] Step S3: For any video, based on the variation characteristics of the upper side instability relative to the lower side instability of different video frames, obtain the main imbalance side of each video; based on the distribution of the main imbalance side of different videos and the distribution of the upper and lower sides of instability corresponding to different time frames, obtain the degree of body imbalance of each video at each time frame.
[0044] When walking, the body maintains stability through the coordination of the upper and lower sides. If there is significant asynchrony or disorder between the upper and lower sides, one side will make corresponding changes to adjust the body's stability in order to maintain balance. By analyzing the changes in the body in the video, for any video, the main unbalanced side of each video is obtained based on the changes in the upper side instability relative to the lower side instability in different video frames.
[0045] Preferably, in one embodiment of the present invention, the method for obtaining the main imbalance side includes: For any video, the instability dominance of each side is obtained based on the difference in instability on each side and the other side between different time frames and the previous time frame, as well as the difference in instability between each side and the other side between different time frames. Both the difference in change and the difference in instability are positively correlated with the instability dominance. In one embodiment of the present invention, the ratio of the instability difference on each side to the instability difference on the other side between each time frame and the previous time frame is calculated, reflecting the change difference of instability on each side relative to the other side between each time frame and the previous time frame; to avoid the change difference being negative, the change difference is normalized, and the product of the normalized result and the instability difference between each side and the instability on the other side of each time frame is calculated as the local dominance of instability on each side in each time frame; the mean of the local dominance of instability on each side in all time frames is obtained as the instability dominance on each side in each video.
[0046] It should be noted that, in the embodiments of the present invention, the following methods may be used: The function is normalized, and the difference in variation is normalized to... Within this scope, the specific methods used are those well-known to those skilled in the art and will not be elaborated upon here.
[0047] Select the side with the greatest instability between the upper and lower sides and take that side as the main unbalanced side.
[0048] The distribution of the main imbalance side in different videos reflects which side the human body tends to be unbalanced in the video, reflecting the asymmetry of the overall movement trend. The instability distribution quantifies the dynamic stability changes of the upper and lower sides in each frame, capturing instantaneous imbalance features. Therefore, based on the distribution of the main imbalance side in different videos, as well as the instability distribution of the upper and lower sides corresponding to different time frames, the degree of body imbalance in each video at each time frame can be obtained.
[0049] Preferably, in one embodiment of the present invention, the method for obtaining the degree of body imbalance includes: Based on the distribution of the main imbalance sides of different videos and the distribution of instability on the upper and lower sides of frames at different time points, the relative minimum instability compensation degree of each video relative to each normal activity video at each time point frame is obtained. It should be noted that by comparing the instability changes at each time frame in each video with those in normal video activity, similar instability states are obtained for analysis; in one embodiment of the present invention, the method for obtaining the relative minimum instability compensation degree includes: Obtain the instability difference of the main imbalance side between each time frame in each video and different time frames in each normal activity video. Select the time frame with the smallest corresponding instability difference in each normal activity video, and take the instability of the non-main imbalance side of the corresponding time frame as the relative minimum instability compensation degree of each normal activity video.
[0050] Obtain the difference between the relative minimum instability compensation degree of each video at each time frame corresponding to each normal activity video and the instability of the non-major imbalance side; obtain the average difference of the difference values that are greater than zero in all normal activity videos with the same major imbalance side corresponding to each video at each time frame, as the body imbalance degree of each video at each time frame; if there is no difference value greater than zero, set the body imbalance degree to 0.
[0051] Step S4: Based on the instability of the main imbalance side corresponding to each video at each time frame and the degree of body imbalance, obtain the anomaly significance score of each video at each time frame.
[0052] When the dominant imbalance is strong but has not yet caused systemic imbalance, attention should be paid to the main imbalance side; the more obvious the imbalance, the more abnormalities are shown, the greater the probability of abnormality, the higher the abnormality significance score, and the greater the risk of the imbalance side.
[0053] Preferably, in one embodiment of the present invention, the method for obtaining the anomaly significance score includes: Based on the instability of the main imbalance side of each video at each time frame, the corresponding body imbalance degree is adjusted by a loss adjustment to obtain the body imbalance weighted value of each video at each time frame. In one embodiment of the present invention, the method for obtaining the loss adjustment is as follows: the instability of the main imbalance side of each video at each time frame is normalized to the range of [0,1], and the difference between the positive integer 1 and the instability of the main imbalance side at each time frame is calculated as the adjustment weight; the product of the adjustment weight and the degree of body imbalance is obtained as the weighted value of body imbalance for each video at each time frame; the greater the instability, the greater the probability of anomaly, the greater the credibility of the degree of body imbalance, the greater the body imbalance, and the greater the anomaly significance score.
[0054] It should be noted that, in the embodiments of the present invention, the instability of the main imbalance side of each video at each time frame can be normalized to the range of [0,1] by maximum and minimum value normalization, that is, the maximum and minimum values of the instability of the main imbalance side of each video in all time frames are obtained, and the instability of the main imbalance side of each time frame is normalized. The specific means are well known to those skilled in the art and will not be described in detail here.
[0055] The sum of the instability and body imbalance weights of the main imbalance side for each video at each time frame is obtained as the anomalous significance score for each video at each time frame.
[0056] Step S5: After training the neural network based on the skeletal coordinates of all videos at different time frames and the anomaly salience score, identify fall behavior in newly acquired videos.
[0057] It should be noted that, in the embodiments of the present invention, existing fall event videos and normal activity videos are acquired as training sets. The skeletal coordinate sequence of each video at each time frame is input into the neural network. The neural network extracts the representation vector of each frame. The anomaly saliency score is used as the attention weight. The representation vectors of different time frames in the video are weighted and averaged. The weighted result is input into the classification layer. After outputting the fall prediction probability, loss calculation and parameter update are performed. Multiple iterations are performed on the videos with known behavior in the training set until the loss value converges, thus completing the training of the neural network. The same analysis is performed on newly acquired videos to obtain anomaly saliency scores. The skeletal coordinates and anomaly saliency scores are input into the trained neural network to judge normal activity and fall. The neural network adopts a lightweight RNN architecture. The specific means are well known to those skilled in the art and will not be described in detail here.
[0058] In summary, this invention, for any given video, obtains the lower instability of each frame based on the distribution of ankle coordinates at different time points; obtains the upper instability of each frame based on the difference in shoulder coordinates relative to hip center coordinates between adjacent frames; obtains the primary imbalance side of each video based on the variation characteristics of upper instability relative to lower instability across different video frames; obtains the degree of body imbalance in each video at each time point based on the distribution of the primary imbalance side across different videos and the corresponding upper and lower instability distributions at different time points; and obtains an anomaly salience score for each video at each time point based on the instability of the primary imbalance side corresponding to each video at each time point and the degree of body imbalance; and then identifies abnormal fall behavior. This invention improves the accuracy of abnormal fall behavior identification by accurately obtaining the anomaly salience score for each frame in the video.
[0059] The present invention also proposes an in-hospital patient fall behavior analysis and recognition system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of an in-hospital patient fall behavior analysis and recognition method.
[0060] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0061] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
Claims
1. An in-hospital patient fall behavior analysis and identification method, characterized in that, The method comprises: Obtaining the skeleton coordinates of each frame in the fall event video and the normal activity video, wherein the skeleton coordinates comprise the shoulder coordinates, the ankle coordinates and the hip center coordinates on both sides; For any video, obtaining the lower instability of each frame according to the position distribution of the ankle coordinates in different frames; obtaining the upper instability of each frame according to the change difference of the shoulder coordinates relative to the hip center coordinates between different adjacent frames; For any video, obtaining the main imbalance side of each video according to the change characteristics of the upper instability relative to the lower instability of different video frames; obtaining the body imbalance degree of each video at each frame according to the distribution of the main imbalance side of different videos and the instability distribution of the upper and lower sides of different frames; Obtaining the abnormal significance score of each video at each frame according to the instability of the main imbalance side and the body imbalance degree of each video at each frame; After training the neural network based on the skeleton coordinates and the abnormal significance score of all videos at different frames, identifying the fall behavior of the newly obtained video.
2. The inpatient patient fall behavior analysis and identification method of claim 1, wherein, The method for obtaining the lower instability comprises: For any video, obtaining the step length of each frame according to the relative distance distribution of the ankle coordinates between the left and right sides in different frames; Obtaining the step length change rate of each frame according to the fluctuation difference degree of the step length of all frames corresponding to the front and back sides of other frames before each frame and the difference of the step length between each frame and other frames before it, wherein the fluctuation difference degree of the step length is negatively correlated with the step length change rate, and the difference of the step length is positively correlated with the step length change rate; Selecting the maximum value of the fluctuation difference degree of the step length corresponding to other frames before each frame as the relative gait change frame of each frame; obtaining the difference value of the mean value of the step length change rate of all frames between the front and back of the relative gait change frame; selecting the maximum value of the difference value absolute value and a preset difference value threshold as the lower instability of each frame.
3. The method of claim 2, wherein the method further comprises: The method for obtaining the step length comprises: For any video, obtaining the relative distance between the ankle coordinates on the left and right sides in each frame; selecting the maximum value of the curve composed of the relative distances of all frames; selecting the relative distance of the adjacent maximum value corresponding to the frame after each frame as the step length of each frame.
4. The inpatient patient fall behavior analysis and identification method of claim 1, wherein, The method for obtaining the upper instability comprises: For any side of the skeleton, obtaining the same period frame of each frame according to the change difference of the shoulder coordinates relative to the hip center coordinates between different adjacent frames; Obtaining the upper instability of each frame according to the difference number of all same period frames corresponding to the left and right sides of each frame and the amplitude difference of the shoulder swing vector, wherein the difference number and the amplitude difference are positively correlated with the upper instability.
5. The inpatient patient fall behavior analysis and identification method of claim 4, wherein, The method for obtaining the same period frame comprises: For any side of the skeleton, obtaining the difference value of the shoulder coordinates and the hip center coordinates in each frame as the shoulder-hip displacement vector; Obtaining a difference value of the shoulder and hip displacement vector between the frame at each time and the frame at the previous time as a shoulder swing vector; if an included angle of the shoulder swing vector between adjacent time frames is less than or equal to a preset angle threshold, the corresponding time frame is taken as a same-period time frame.
6. The inpatient patient fall behavior analysis and identification method of claim 1, wherein, The method for obtaining the main imbalance side comprises: For any video, according to a change difference value of each side instability and another side instability between different time frames and the frame at the previous time, and an instability difference value between each side instability and another side instability of the different time frames, a dominant instability of each side is obtained, and the change difference value and the instability difference value are positively correlated with the dominant instability; The side with the maximum dominant instability is selected from the upper and lower sides, and the corresponding side is taken as the main imbalance side.
7. The inpatient patient fall behavior analysis and identification method of claim 1, wherein, The method for obtaining the body imbalance degree comprises: According to the distribution of the main imbalance side of different videos and the instability distribution of the upper and lower sides of different time frames, a relative minimum instability compensation degree of each video at each time frame relative to each normal activity video is obtained; A difference value between the relative minimum instability compensation degree of each video at each time frame relative to each normal activity video and the instability of the non-main imbalance side is obtained, and a difference average value greater than zero of each video at each time frame relative to the same normal activity video of all main imbalance sides is obtained as the body imbalance degree of each video at each time frame; if there is no difference value greater than zero, the body imbalance degree is set to 0.
8. The in-hospital patient fall behavior analysis and recognition method of claim 7, wherein, The method for obtaining the relative minimum instability compensation degree comprises: Obtaining an instability difference of the main imbalance side between each time frame in each video and different time frames in each normal activity video, selecting a time frame corresponding to the minimum instability difference in each normal activity video, and taking the instability of the non-main imbalance side of the corresponding time frame as the relative minimum instability compensation degree of each normal activity video.
9. The inpatient patient fall behavior analysis and identification method of claim 1, wherein, The method for obtaining the abnormality significance score comprises: According to the instability of the main imbalance side of each video at each time frame, a body imbalance weighting value of each video at each time frame is obtained by gain adjustment of the corresponding body imbalance degree; The sum of the instability of the main imbalance side and the body imbalance weighting value of each video at each time frame is taken as the abnormality significance score of each video at each time frame.
10. An in-hospital patient fall behavior analysis and identification system, the system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The processor executes the computer program to realize the steps of the hospital patient fall behavior analysis and recognition method according to any one of claims 1-9.