Human body gait analysis method, system and device based on video data

Abstract

The invention discloses a human body gait analysis method, system and device based on video data, and is used for analyzing human body gaits according to the video data so as to confirm the physical state of a person. The method comprises the following steps: acquiring a feedback video from a camera; determining a character identity in the feedback video according to a face recognition technology, and establishing a health associated file according to the character identity; segmenting the feedback video by the deep learning framework to determine a human body region; extracting key points of a human body region through a human body posture estimation tool, and generating skeleton data; obtaining an effective video clip as a behavior analysis sample; gait features of skeleton data in the behavior analysis sample are extracted through a visual technology; obtaining dynamic data of the gait features through an optical flow algorithm; establishing a behavior portrait according to the dynamic data; analyzing the behavior portrait according to the time sequence through a deep learning model to obtain an analysis result, and performing risk scoring on the analysis result; and executing a corresponding feedback strategy according to the risk score.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, system and device for analyzing human gait based on video data. Background Art

[0002] In today's society, the number of elderly people living at home is increasing. In order to ensure the well-being of the elderly at home, installing cameras at home has become one of the ways for children to ensure the well-being of the elderly at home.

[0003] In the existing technology, traditional fall monitoring equipment relies on wearable devices or simple sensors, which cannot effectively cover the scenarios of single-person and multiple elderly people, and it is difficult to make personalized predictions based on individual health conditions. In addition, the camera system can only determine the status of the home by actively watching, which is difficult to achieve the purpose of active monitoring. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a method, system and device for analyzing human gait based on video data, which are used to analyze human gait according to video data and then confirm the physical condition of the person.

[0005] The technical solution provided in this application is described below: The first aspect of the present application provides a method for analyzing human gait based on video data, comprising: Get the video feed from the camera; Determine the identity of the person in the feedback video according to face recognition technology, and establish a health-related file according to the identity of the person, wherein the health-related file contains physical health data corresponding to the identity of the person; Establishing a deep learning framework, and segmenting the feedback video using a Mask R-CNN model in the deep learning framework to determine the human body area; Extract key points of the human body region by using a human posture estimation tool to generate skeleton data; Performing behavior segmentation on the feedback video to obtain valid video clips as behavior analysis samples; Extracting gait features of the skeleton data in the behavior analysis sample by using visual technology; Acquiring dynamic data of the gait features through an optical flow algorithm; Establishing a behavior profile based on the dynamic data; Inputting the behavior profile into a deep learning model, and analyzing the behavior profile according to a time series by the deep learning model to obtain an analysis result; Performing a risk score on the analysis result in combination with the physical health data; When the risk score reaches an alarm threshold, a corresponding feedback strategy is executed according to the risk score.

[0006] Optionally, the inputting the behavior profile into a deep learning model, and analyzing the behavior profile according to a time series by the deep learning model to obtain an analysis result includes: Determine the model parameters of the deep learning model corresponding to the character identity according to the physical health data to obtain a first stability factor; Inputting the behavior data into a target deep learning model, and performing action trend analysis on the behavior profile according to a time series using the model parameters to obtain a second stability factor; An analysis result is generated based on the first stabilization factor and the second stabilization factor.

[0007] Optionally, performing risk scoring on the analysis result in combination with the physical health data includes: Obtaining basic risk parameters according to the physical health data; Using the basic risk parameter as a risk score offset, bidirectionally offset the first stability factor to obtain a safety interval; A risk score is generated according to the positional relationship between the second stability factor and the safety interval.

[0008] Optionally, the performing behavior segmentation on the feedback video to obtain valid video segments as behavior analysis samples includes: Acquire a behavior template, and perform behavior marking on the feedback video according to the behavior template; The feedback video is cut according to the behavior marker to obtain a behavior analysis sample.

[0009] Optionally, the extracting gait features of the skeleton data in the behavior analysis sample by using visual technology includes: The skeleton data is cleaned by geometric constraints, and gait features of the skeleton data are extracted, wherein the gait features include time domain features, frequency domain features and spatiotemporal features.

[0010] Optionally, the obtaining the dynamic data of the gait feature by using an optical flow algorithm includes: Calculate the motion vector of the skeleton key point between consecutive frames of the behavior analysis sample by Lucas-Kanade optical flow method; The motion vector is reconstructed by trajectory, and the changes of the joint angles of the key points of the skeleton over time are calculated to obtain dynamic data.

[0011] Optionally, establishing a behavior profile according to the dynamic data includes: parsing behavioral components in the dynamic data; Generate a sequence of the behavior components and obtain the duration of each behavior in the behavior components to obtain behavior profile sample data; The behavior portrait sample data is reduced in dimension based on the characteristic parameters of the deep learning model to obtain a behavior portrait.

[0012] The second aspect of the present application provides a human gait analysis system based on video data, comprising: A first acquisition unit, used for acquiring a feedback video from a camera; A first determining unit is used to determine the identity of a person in the feedback video according to face recognition technology, and to establish a health-related file according to the identity of the person, wherein the health-related file contains physical health data corresponding to the identity of the person; A segmentation unit, used to establish a deep learning framework, and segment the feedback video through a Mask R-CNN model in the deep learning framework to determine a human body area; A first extraction unit, configured to extract key points of the human body region by using a human body posture estimation tool to generate skeleton data; A cutting unit, used for performing behavior cutting on the feedback video to obtain valid video clips as behavior analysis samples; A second extraction unit, configured to extract gait features of the skeleton data in the behavior analysis sample by using a visual technique; A second acquisition unit, used for acquiring dynamic data of the gait feature by using an optical flow algorithm; An establishing unit, used for establishing a behavior profile according to the dynamic data; An analysis unit, configured to input the behavior profile into a deep learning model, and analyze the behavior profile according to a time series through the deep learning model to obtain an analysis result; a risk scoring unit, configured to perform risk scoring on the analysis result in combination with the physical health data; An alarm unit is used to execute a corresponding feedback strategy according to the risk score when the risk score reaches an alarm threshold.

[0013] Optionally, the analysis unit is specifically used for: Determine the model parameters of the deep learning model corresponding to the character identity according to the physical health data to obtain a first stability factor; Inputting the behavior data into a target deep learning model, and performing action trend analysis on the behavior profile according to a time series using the model parameters to obtain a second stability factor; An analysis result is generated based on the first stabilization factor and the second stabilization factor.

[0014] Optionally, the risk scoring unit is specifically used to: Obtaining basic risk parameters according to the physical health data; Using the basic risk parameter as a risk score offset, bidirectionally offset the first stability factor to obtain a safety interval; A risk score is generated according to the positional relationship between the second stability factor and the safety interval.

[0015] Optionally, the cutting unit is specifically used for: Acquire a behavior template, and perform behavior marking on the feedback video according to the behavior template; The feedback video is cut according to the behavior marker to obtain a behavior analysis sample.

[0016] Optionally, the second extraction unit is specifically used for: The skeleton data is cleaned by geometric constraints, and gait features of the skeleton data are extracted, wherein the gait features include time domain features, frequency domain features and spatiotemporal features.

[0017] Optionally, the second acquiring unit is specifically configured to: Calculate the motion vector of the skeleton key point between consecutive frames of the behavior analysis sample by Lucas-Kanade optical flow method; The motion vector is reconstructed by trajectory, and the changes of the joint angles of the key points of the skeleton over time are calculated to obtain dynamic data.

[0018] Optionally, the establishing unit is specifically used for: parsing behavioral components in the dynamic data; Generate a sequence of the behavior components and obtain the duration of each behavior in the behavior components to obtain behavior profile sample data; The behavior portrait sample data is reduced in dimension based on the characteristic parameters of the deep learning model to obtain a behavior portrait.

[0019] A third aspect of the present application provides a human gait analysis device based on video data, the device comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0021] It can be seen from the above technical solutions that this application has the following advantages: This application analyzes the person in the feedback video, establishes a health-related file for the person after determining the person, and confirms the skeleton of the person in the feedback video, analyzes the dynamic behavior of the person through the skeleton data, and performs a risk score on the dynamic behavior of the person, so as to analyze the behavioral status of the person in the video according to the risk score, so as to achieve the purpose of risk warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of an embodiment of a method for analyzing human gait based on video data in the present application; Figure 2a This is a flowchart of an embodiment of the first stage of the human gait analysis method based on video data in the present application; Figure 2b A schematic flow chart of an embodiment of the second stage of the human gait analysis method based on video data in the present application; Figure 3 This is a schematic structural diagram of an embodiment of a human gait analysis system based on video data in this application; Figure 4 The figure is a schematic diagram of the structure of an embodiment of a human gait analysis device based on video data in the present application. DETAILED DESCRIPTION

[0024] It should be noted that the human gait analysis method based on video data provided in this application can be applied to a terminal or a system, and can also be applied to a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as an example for explanation.

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] See also Figure 1 The present application first provides an embodiment of human gait analysis based on video data, which includes: S101, obtaining a feedback video from a camera; The feedback video is obtained from the video stream captured by the home camera of the elderly or the camera of the nursing home. In actual situations, the people included in the video stream captured by the camera are not restricted by age. Therefore, all people appearing in the video stream can be analyzed and processed through the feedback video obtained by this solution as long as there is a need to predict or monitor their physical state. Specifically, the acquisition of the video stream content requires user permission, so the way to obtain the feedback video is compliant.

[0027] In actual situations, the feedback video is a video stream in which characters exist in the video content. When no characters appear in the video, no feedback video will be generated based on the video stream to reduce the storage pressure of the server.

[0028] S102, determining the identity of the person in the feedback video according to the face recognition technology, and establishing a health-related file according to the identity of the person, wherein the health-related file includes physical health data corresponding to the identity of the person; Facial recognition technology is a technology that uses artificial intelligence algorithms to identify and verify individual identities. Health-related files contain electronic records of personal health information for health management and analysis. Among them, health-related files are directly linked to the medical system or the data of health-related files are obtained through user uploads. Specifically, the cases currently stored in the hospital are generally electronic cases. As the patient himself, the case can be exported or called through the hospital's external system, so that the terminal can establish a health-related file based on the case content and related physical examination parameters. At the same time, the health-related file contains the person's physical health data, and the physical health data is pre-uploaded according to user needs. Therefore, in the actual process of establishing a health-related file, the corresponding physical health data can be obtained according to the person's identity.

[0029] Specifically, the terminal uses a face recognition algorithm (such as FaceNet or ArcFace) to detect and identify the face of the person in the video from the feedback video. The recognition result is matched with the pre-established health record through face comparison to determine the person's identity, and the health data related to the identified individual is extracted from the health record based on the person's identity.

[0030] S103, establishing a deep learning framework, and segmenting the feedback video using a Mask R-CNN model in the deep learning framework to determine the human body area; A deep learning framework is a software platform for building and training deep learning models, such as TensorFlow or PyTorch; the Mask R-CNN model is a deep learning model for instance segmentation that can identify and segment multiple objects in an image.

[0031] In this embodiment, the Mask R-CNN model is used to remove the background, so the Mask R-CNN model is mainly used to perform content training on the corresponding background of the indoor scene and the camera shooting angle to improve the accuracy and efficiency of the Mask R-CNN model for the background segmentation of the feedback video, thereby improving the accuracy of the human body area. After the camera is installed, except for disassembly or external force, the camera's field of view is fixed. The Mask R-CNN model can generate a complete area by training the scene obtained by the camera to achieve a more accurate background segmentation effect.

[0032] S104, extracting key points of the human body region through a human body posture estimation tool to generate skeleton data; Human pose estimation tools are tools used to detect human joints and limbs, including but not limited to: OpenPose or MediaPipe, etc., which are tools used to generate skeleton data based on joint key points in the human body area.

[0033] OpenPose is an open source project written in C++ and was originally developed by Carnegie Mellon University. It focuses on real-time multi-person pose estimation and can detect key points of the human body, face, and hands. When using OpenPose to generate skeleton data, the human body area is input into OpenPose in the form of video data or image data. OpenPose can obtain a pre-trained deep neural network (such as a CPM network) to detect the approximate position and key points of the human body. For the case when there are multiple people in the video data, for each detected human figure, OpenPose has built-in multiple convolutional neural networks, allowing OpenPose to accurately locate the key points of each character in the video. After determining the key points, OpenPose uses a top-down approach to first detect the center of the body, then gradually refine it to other key points, and output the coordinates and confidence scores of the skeleton key points of each detected person.

[0034] MediaPipe is a cross-platform framework developed by Google for building application-level multimedia processing pipelines, including pose estimation, facial recognition, gesture tracking, and more. When using MediaPipe to generate skeleton data, the human body area is input into MediaPipe in the form of video data or image data. In actual situations, MediaPipe provides a variety of solutions, such as Pose, Face, Hands, etc., to select the appropriate solution for different tasks. MediaPipe uses a series of predefined processing modules to process the input data. For pose estimation, MediaPipe uses a 3D pose estimation model to detect key points of the human body. MediaPipe is optimized for real-time applications, including the use of GPU acceleration and model compression. The output includes key point coordinates, confidence scores, and optional 3D pose information.

[0035] In actual situations, both of the above solutions can generate skeleton data. Generally, the skeleton generation solution is selected according to the complexity of the scene. MediaPipe provides a simpler API and richer solutions, which are suitable for rapid integration and development. OpenPose provides more customization options, which is suitable for scenes that require deep customization.

[0036] S105, performing behavior segmentation on the feedback video to obtain valid video clips as behavior analysis samples; Behavior cutting refers to the process of editing video clips according to specific behavior patterns. In this embodiment, the effective video clips refer to the video clips in which the coordinates of the skeleton data of the characters in the video change. The change in the skeleton data coordinates indicates that the human body of the character in the video is performing an action, that is, generating a behavior. The video clips in which the human body generates behavior can be used as behavior analysis samples of the character corresponding to the skeleton data.

[0037] The behavior analysis sample is used to analyze the behavior of a person in the video. The person is identified through face recognition in the previous step. If the face is not recognized, the person will not be analyzed. Only after it is determined that the face is the corresponding health-related file, the behavior analysis sample of the person will be obtained.

[0038] S106, extracting gait features of skeleton data in behavior analysis samples by using visual technology; Specifically, the gait feature extraction process includes: extracting basic gait parameters such as gait period, step length, and speed through the state of skeleton data in the feedback video. After determining the basic gait parameters, the frequency characteristics of the gait are extracted through methods such as Fourier transform, and then the dynamic characteristics of the gait are extracted by combining time and space information, such as the speed and acceleration of key points; calculating the angles between adjacent key points, such as the angles of the knee joint and the hip joint; and finally comparing the gait characteristics of the left and right limbs to extract symmetry features.

[0039] Specifically, the time and space information can be obtained through the background data of the camera itself and the timestamp of the feedback video.

[0040] S107, obtaining dynamic data of gait features through an optical flow algorithm; Optical flow algorithm is a computer vision technique used to estimate the motion of objects in an image. In gait analysis, optical flow algorithm can help us estimate the motion of skeleton key points from consecutive video frames, thereby obtaining dynamic data of gait.

[0041] Specifically, the optical flow algorithm obtains dynamic features (speed and acceleration of skeleton data) from gait features, and tracks the motion trajectory of key points of skeleton data moving over time based on the dynamic features, and calculates the changes in joint angles over time based on key points of skeleton data identified as joint points, such as knee joints and hip joints, to quantify the motion parameters of the skeleton data and obtain dynamic data.

[0042] S108. Establishing behavioral profiles based on dynamic data; Behavioral portraits refer to descriptions of individual behavioral characteristics that are constructed by analyzing the movement changes of key points of skeleton data over time. Behavioral portraits are used to capture the spatial and temporal characteristics of individuals in specific behaviors (such as walking, running, sitting, etc.), and then determine the behavioral patterns and habits of the person corresponding to the behavioral portrait.

[0043] After determining the dynamic data of the gait characteristics, the terminal will determine the actions that need to be confirmed based on the behavior template, that is, establish a behavior profile of the corresponding action, such as walking, bending, sitting, standing, etc. After determining the behavior template, the terminal extracts the corresponding behavior from the dynamic data according to the behavior template, and parameterizes the normal state of the behavior pattern of the task through the dynamic data, so that the subsequent terminal can analyze the status of the person in the camera through the behavior profile.

[0044] S109, inputting the behavior profile into the deep learning model, and analyzing the behavior profile according to the time series through the deep learning model to obtain an analysis result; The terminal sorts the generated behavior portraits in chronological order according to the time series. The purpose of sorting the behavior portraits by time series is to analyze the behavior habits of individual characters according to time and identify the patterns and habits of individual character behaviors, such as which activities are more frequent and which behaviors occur in a specific time period. The information output by the above analysis is the behavior habits of individual characters. After determining the behavior habits of the characters, the terminal will analyze the change trend of the individual characters over time according to the output behavior habits and obtain the analysis results. The analysis results are used to predict the future behavior of the characters in order to formulate intervention measures.

[0045] S110, perform risk scoring on the analysis results in combination with physical health data; Specifically, the scoring criteria for the risk score is determined based on the health status of the individual character. Specifically, the terminal determines the health status of the character based on the health-related file corresponding to the character, and offsets the risk score of the character's health status based on the medical history in the health-related file as an offset to obtain the risk scoring criteria for the currently analyzed character, and performs a risk score for the character's behavior on the corresponding character in the subsequent input or real-time feedback video based on the risk scoring criteria.

[0046] In actual situations, different diseases have different effects on people's behavior. The deep learning model can determine the impact of the corresponding disease on people's behavior patterns based on a sufficient number of samples of the same disease. After the deep learning model completes the behavioral pattern analysis of a single disease, it will vectorize these diseases as labels, and then generate an analysis model that determines the person's behavioral habits based on the person's disease status. This model is used to assist in risk scoring of the analysis results of human behavior, providing a more reliable comparison data for risk scoring.

[0047] S111. When the risk score reaches the alarm threshold, a corresponding feedback strategy is executed according to the risk score.

[0048] Specifically, the alarm threshold is a preset value. Generally, the risk score is set at 0-100 points. The terminal has different feedback strategies for risk scores according to different score ranges. When the terminal generates a risk score standard for a person's behavioral habits, the person state corresponding to the score standard is generally referred to as the standard state. The standard state generally sets the risk score in the range of 30-40 points. When the person's health status is in the recovery period (recovery or conditioning status), the standard state generally sets a higher value relative to the basic value. When the person's health status is in a general state, the basic value will be used directly for setting. Therefore, the risk score of the standard state is reflected as an interval for different person health statuses.

[0049] When the risk score obtained in real time is higher than the standard state, it means that the person's health risk value is high, and feedback will be generated according to the percentage ratio exceeding the standard state. The feedback strategy is divided into strong feedback and weak feedback. Weak feedback does not require the user who receives the data to confirm the feedback information. When the terminal generates strong feedback, the user who receives the feedback information needs to be forced to confirm the information to ensure that the user can make a response strategy in time.

[0050] This application analyzes the person in the feedback video, establishes a health-related file for the person after determining the person, and confirms the skeleton of the person in the feedback video, analyzes the dynamic behavior of the person through the skeleton data, and performs a risk score on the dynamic behavior of the person, so as to analyze the behavioral status of the person in the video according to the risk score, so as to achieve the purpose of risk warning.

[0051] See also Figure 2a and Figure 2b The present application embodiment provides another embodiment of human gait analysis based on video data, which includes: S201, obtaining a feedback video from a camera; S202, determining the identity of the person in the feedback video according to the face recognition technology, and establishing a health-related file according to the identity of the person, wherein the health-related file includes physical health data corresponding to the identity of the person; S203, establishing a deep learning framework, and segmenting the feedback video using a Mask R-CNN model in the deep learning framework to determine the human body area; S204, extracting key points of the human body region through a human body posture estimation tool to generate skeleton data; Steps S201 to S204 in this embodiment are similar to steps S101 to S104 in the aforementioned embodiment, and will not be described in detail here.

[0052] S205, obtaining a behavior template, and performing behavior marking on the feedback video according to the behavior template; The behavior template content is a predefined behavior pattern used to identify and classify specific behaviors. The behavior tag is obtained from the feedback video to match the behavior template and mark the timeline position where the behavior occurs.

[0053] Specifically, feedback video is a continuous video resource. All-day monitoring makes it very easy for the feedback video to contain invalid information. However, in actual situations, video files occupy a large amount of memory on the server or terminal itself. In order to avoid excessive usage during subsequent data analysis, the terminal will mark the behavioral actions of the feedback video according to the behavioral template when obtaining the feedback video.

[0054] S206: Cut the feedback video according to the behavior markers to obtain behavior analysis samples.

[0055] The action of cutting only occurs after the action of marking. The essence of cutting is to trim the video. In actual situations, the feedback video may be synchronized in real time and has a limited retention time. To ensure that the memory can obtain new video content, the old video data will be cleared after a sufficient period of time. Therefore, after determining the mark, the terminal will immediately trim and archive the video content corresponding to the mark. The videos obtained by trimming and archiving are behavioral analysis samples.

[0056] S207 , cleaning the skeleton data through geometric constraints, and extracting gait features of the skeleton data, where the gait features include time domain features, frequency domain features, and spatiotemporal features.

[0057] The geometric constraints are used to constrain the key points corresponding to the skeleton data generated in step S204 according to the anatomical orientation of the human body, thereby correcting unreasonable key point positions, such as the reasonable range of limb lengths and joint angles.

[0058] Specifically, to geometrically constrain the skeleton data, it is first necessary to specify the rules of the geometric constraints. In actual situations, the feedback video can generate a unit length based on the actual human body data obtained, and determine the length range of the human limbs and torso based on the unit length. After the rule construction is completed, the main joints and relative position relationships of the key points are determined according to the human body relationship to determine the actual corresponding joints of each key point in the skeleton data. At the same time, preliminary corrections are made to obvious erroneous detection results, such as removing extreme values ​​that exceed the human body proportions.

[0059] After determining the constraint rules, the terminal will correct the key points at or beyond the limit values ​​according to the constraint rules. For key points close to the limit values, the terminal will analyze the rationality of the key points based on the current posture of the character. If the key points are determined to be reasonable, they will be retained.

[0060] After determining the reasonable key points, the terminal obtains the parameters of your specific action content in the analysis sample, and determines the time domain features, frequency domain features and space-time features in the gait features based on the vector values ​​of the actions contained in the analysis sample (initial speed, acceleration and offset angle of the action) and fixed parameters (time, key point coordinate values).

[0061] Specifically, the time domain features should at least include gait cycle, gait symmetry and gait variability, among which the gait cycle is the time length and travel distance of each step when the person walks in the measured skeleton data; gait symmetry is to compare the gait cycles of the left and right legs to evaluate the symmetry of the gait; gait variability is to calculate the variability of continuous gait cycles, reflecting the stability of the gait.

[0062] The frequency domain feature is obtained by performing Fourier transform on the gait signal generated by the waveform corresponding to the gait generated by the analysis sample. The terminal performs Fourier transform on the gait periodic signal to extract the frequency components. After determining the frequency, the terminal analyzes the power spectral density of the gait signal to identify the main frequency components.

[0063] The spatiotemporal features are features determined by analyzing the key point trajectory, joint angle changes, and velocity acceleration changes of the analysis samples. That is, the spatiotemporal feature analysis can obtain the key point trajectory, key point angle, and dynamic features of the speed and acceleration of the key points when moving in the skeleton data.

[0064] The data set containing the above three features is called gait features.

[0065] S208, calculating motion vectors of skeleton key points between consecutive frames of the behavior analysis sample by Lucas-Kanade optical flow method; The Lucas-Kanade optical flow method is a classic optical flow estimation technique used to estimate the motion of feature points in an image sequence. The Lucas-Kanade optical flow method estimates the motion vector of the key point by solving a set of linear equations by assuming that the motion of the feature points is consistent within a small neighborhood.

[0066] Specifically, to calculate the motion vector, we first need to determine the feature points to be calculated. The feature points are the feature points contained in the skeleton data. The number of feature points can be multiple. For the determined feature points, the optical flow equation is established one by one according to the Lucas-Kanade optical flow method, and the optical flow equation is solved to obtain the motion vector of each key point between consecutive frames.

[0067] After calculating the key point motion vector, in order to ensure the reliability of the data, after determining the vector, the terminal will optimize the motion vector. The specific optimization process will be carried out through vector smoothing and vector correction. Among them, vector smoothing is used to reduce noise and outliers in the motion vector; vector correction is used to filter out unreasonable motion vectors.

[0068] The final data obtained is the motion vector of the analyzed sample.

[0069] S209, reconstructing the trajectory of the motion vector, and calculating the changes of the joint angles of the skeleton key points over time to obtain dynamic data.

[0070] Trajectory reconstruction refers to reconstructing the motion trajectory of skeleton key points in the video sequence based on motion vectors.

[0071] Specifically, after determining the key point to be analyzed, the terminal initializes an empty trajectory list according to the key point, updates the position of the key point according to the motion vector at each time step, and adds it to the trajectory list to obtain the motion trajectory of the key point.

[0072] After determining the motion trajectory of the key point, the terminal will smooth the motion trajectory to reduce the impact of noise and outliers. Common methods include moving average filtering or Kalman filtering.

[0073] After the recorded motion trajectory is smoothed, the terminal will reconstruct the character skeleton corresponding to the key point based on the acquired motion trajectory, obtain the angle change between adjacent key points, and generate the coordinates of the check point on each unit frame. Finally, the dynamic data of the analysis sample corresponding to the key point is determined through coordinate calculation and angle calculation.

[0074] S210, parsing behavioral components in dynamic data; Behavior components refer to the basic action units that constitute complex behaviors. In human behavior analysis, identifying and classifying these basic action units is a prerequisite for understanding complex behavior patterns. Behavior components are mainly divided into upper limb action behaviors, lower limb action behaviors, and body posture states. After the terminal obtains the skeleton data, the key points used to determine the upper limb actions and lower limb actions are determined according to the key point positions of the skeleton data, and the dynamic data is disassembled according to the position states of these key points. After determining the upper limb actions and lower limb actions, the upper limb actions and lower limb actions are corrected according to the current body posture of the skeleton data to disassemble the actions in detail.

[0075] This embodiment mainly analyzes the gait of a person, so the terminal needs to determine the unit actions of the complex action decomposition, including but not limited to stepping, standing, lifting the foot, heel touching the ground and toe off the ground. The above unit actions can simply form a complete gait, so that the terminal can subsequently perform in-depth analysis of the gait of the person and then determine the person's behavioral habits.

[0076] S211, generating a sequence of behavior components and obtaining the duration of each behavior in the behavior components to obtain behavior profile sample data; After determining the behavior components, the terminal inputs the behavior components into the convolutional network to classify these behaviors, and at the same time generates a sequence with the timestamp as the start time and end time of the action execution as the interval, and sorts the actual lengths of the action components one by one in the interval to form a behavior component sequence.

[0077] After determining the sequence of behavioral components, the terminal will record the duration of each behavioral component and integrate the duration of each behavioral component to integrate the behavioral component sequence and duration data into a complete data set to form behavioral profile sample data. The behavioral profile sample data includes not only the type and sequence of behavioral components, but also the duration of each behavior, thereby providing a multi-dimensional description of the behavior.

[0078] S212. Reduce the dimension of the behavior profile sample data based on the characteristic parameters of the deep learning model to obtain a behavior profile.

[0079] Dimensionality reduction is to reduce the data dimension. Specifically, the terminal determines the parameters that need to be retained from the behavior profile sample data based on the preset feature parameters in the deep learning model. In actual use, the deep learning model is a learning model that reaches reasonable convergence after training with a sufficient number of samples. The deep learning model can analyze the input parameters. Therefore, in actual situations, the terminal needs to determine the input parameters of the deep learning model from the behavior profile sample data. These input parameters for inputting the deep learning model are necessary data for the behavior profile. In actual situations, when the behavior profile needs to input different deep learning models, the feature parameters of all deep learning models will be integrated to retain the data in the behavior profile sample data.

[0080] S213, determining the model parameters of the deep learning model corresponding to the person's identity according to the physical health data, and obtaining a first stability factor; The first stable factor is the basic risk scoring standard obtained by the deep learning model through targeted training based on all the feedback videos and physical health data corresponding to the person that can be obtained from the health-related files of the person. The first stable factor is generally stored in the physical health data after it is generated. It is data that the user cannot actively modify. The first stable factor is the health status of the individual person determined by the physical health data. Among them, the first stable factor is a person's health score generated by a health data template with the current physical condition (such as medical history or chronic disease) as a reference factor.

[0081] S214, inputting the behavior data into the target deep learning model, and performing action trend analysis on the behavior profile according to the time series through the model parameters to obtain a second stability factor; The second stability factor is the health status corresponding to the latest behavior data of the individual character, which is generated based on the input data. The terminal inputs the behavior data obtained after the preliminary processing into the target deep learning model. The target deep learning model is a model that is trained and optimized according to the model parameters of the deep learning model corresponding to the character's identity, and can accurately analyze the behavioral characteristics of a specific character. The model will perform action trend analysis on the behavior portrait according to the time series, that is, analyze the behavior pattern and change trend of the character in different time periods. The deep learning model analyzes the input data to capture the behavioral stability of the character in the current health state, thereby obtaining the second stability factor. The second stability factor reflects the behavioral stability of the character in the current health state, providing an important basis for subsequent risk scoring.

[0082] Among them, the second stability factor can determine the current life status of the individual through gait data, and at the same time adjust the second stability factor with behavioral stability (gait quality) as an influencing factor to improve the reliability of the second stability factor.

[0083] S215. Generate an analysis result according to the first stability factor and the second stability factor.

[0084] Specifically, the actual output value of the first stability factor and the second stability factor is a specific numerical value, that is, the value for scoring the individual health status of the character. The purpose of the deep learning model analyzing the character's gait is to assist in adjusting the character's health status based on the character's behavior. Therefore, when the terminal obtains the second stability factor, the second stability factor and the first stability factor are calculated according to the preset rules to generate an analysis result.

[0085] Specifically, the preset rule may be a difference between the first stability factor and the second stability factor or a parameter change amount between the first stability factor and the second stability factor, which is not specifically limited here.

[0086] S216. Obtain basic risk parameters based on physical health data; The basic risk parameter is an offset value of the initial interval obtained through healthy individuals, that is, the safety interval of a healthy individual is the interval obtained by offsetting the basic risk parameter with the first stability factor as the intermediate value.

[0087] Among them, healthy individuals need to meet the requirements of no medical history and the weight of the effect of the behavioral portrait on the first stability factor is less than the preset value. Specifically, the first stability factor is obtained through historical data training. In actual situations, the behavioral portrait will increase the weight of the analysis results of the behavioral portrait of historical data on the first stability factor according to the impact of physical defects or physical injuries on the behavior of the person. Therefore, when the weight value is too high, it means that although the task individual has no medical history, he has trauma or discomfort, and thus is not a healthy individual.

[0088] S217, using the basic risk parameter as the risk score offset, bidirectionally offset the first stability factor to obtain a safety interval; After obtaining the basic risk parameter, it is used as the offset of the risk score, and the first stable factor is bidirectionally offset to obtain a safe interval. Bidirectional offset means that on the basis of the first stable factor, a certain value is offset upward and downward respectively to form a safe interval. The range of this safe interval will vary according to the size of the basic risk parameter. For example, if the basic risk parameter is high, the range of the safe interval will be relatively small, indicating that the character may be at risk within a smaller range of behavioral changes; if the basic risk parameter is low, the range of the safe interval will be relatively large, indicating that the character is still in a safe state within a larger range of behavioral changes. The determination of the safe interval provides a clear reference standard for subsequent risk scoring.

[0089] S218. Generate a risk score based on the positional relationship between the second stability factor and the safety interval.

[0090] After determining the safety interval, a risk score is generated based on the positional relationship between the second stability factor and the safety interval. If the second stability factor falls within the safety interval, the risk score will be relatively low, indicating that the character's behavior is in a safe state; if the second stability factor falls outside the safety interval, the risk score will be relatively high, indicating that the character's behavior has certain risks. In addition, the risk score can be further refined based on the distance between the second stability factor and the safety interval. For example, the farther away from the safety interval, the higher the risk score. This can then more accurately assess the character's behavioral risk and provide a reliable basis for subsequent feedback strategies.

[0091] S219: When the risk score reaches the alarm threshold, a corresponding feedback strategy is executed according to the risk score.

[0092] Step S219 in this embodiment is similar to step S211 in the aforementioned embodiment, and will not be described in detail here.

[0093] This embodiment performs detailed motion analysis on the behavior of the characters in the video, combines multiple models for joint analysis, and predicts the behavior of the skeleton data of the characters in a multimodal environment to achieve the purpose of monitoring behavior. At the same time, when abnormal behavior occurs, it prompts the abnormal behavior according to the preset status, so that when the camera monitors the task falling, it can actively issue an alarm, and the triggering status of the fall can be prompted through the character's behavior, so as to reduce the situation where no one finds the fall.

[0094] The above is a detailed description of the human gait analysis method based on video data in the embodiment of the present application. The following is a detailed description of the human gait analysis system and device based on video data.

[0095] See also Figure 3 The present application provides an embodiment of a human gait analysis system based on video data, which includes: A first acquisition unit 301 is used to acquire a feedback video from a camera; The first determining unit 302 is used to determine the identity of the person in the feedback video according to the face recognition technology, and to establish a health-related file according to the identity of the person, wherein the health-related file contains the physical health data corresponding to the identity of the person; The segmentation unit 303 is used to establish a deep learning framework, and segment the feedback video through the Mask R-CNN model in the deep learning framework to determine the human body area; A first extraction unit 304 is used to extract key points of a human body region by using a human posture estimation tool to generate skeleton data; A cutting unit 305 is used to perform behavior cutting on the feedback video to obtain valid video segments as behavior analysis samples; A second extraction unit 306, used to extract gait features of skeleton data in the behavior analysis sample by using visual technology; A second acquisition unit 307, used to acquire dynamic data of gait features through an optical flow algorithm; A building unit 308, used to build a behavior profile based on dynamic data; An analysis unit 309 is used to input the behavior profile into a deep learning model, and analyze the behavior profile according to a time series through the deep learning model to obtain an analysis result; A risk scoring unit 310 is used to perform risk scoring on the analysis results in combination with the physical health data; The alarm unit 311 is used to execute a corresponding feedback strategy according to the risk score when the risk score reaches an alarm threshold.

[0096] In this embodiment, the analysis unit 309 is specifically used for: Determine the model parameters of the deep learning model corresponding to the person's identity according to the physical health data to obtain a first stability factor; The behavior data is input into the target deep learning model, and the action trend analysis of the behavior profile is performed according to the time series through the model parameters to obtain the second stability factor; An analysis result is generated based on the first stabilization factor and the second stabilization factor.

[0097] In this embodiment, the risk scoring unit 310 is specifically used for: Obtain basic risk parameters based on physical health data; Using the basic risk parameter as the risk score offset, the first stability factor is bidirectionally offset to obtain a safety interval; A risk score is generated based on the positional relationship between the second stability factor and the safety interval.

[0098] In this embodiment, the cutting unit 305 is specifically used for: Obtain a behavior template, and perform behavior marking on the feedback video according to the behavior template; The feedback video is cut according to the behavioral markers to obtain behavioral analysis samples.

[0099] In this embodiment, the second extraction unit 306 is specifically used for: The skeleton data is cleaned through geometric constraints, and the gait features of the skeleton data are extracted. The gait features include time domain features, frequency domain features and spatiotemporal features.

[0100] In this embodiment, the second obtaining unit 307 is specifically used for: The Lucas-Kanade optical flow method is used to calculate the motion vectors of skeleton key points between consecutive frames of the behavior analysis sample; The motion vector trajectory is reconstructed, and the changes in the joint angles of the key points of the skeleton over time are calculated to obtain dynamic data.

[0101] In this embodiment, the establishing unit 308 is specifically used for: Parsing behavioral components in dynamic data; Generate a sequence of behavioral components and obtain the duration of each behavior in the behavioral components to obtain behavioral profile sample data; The behavior profile sample data is reduced in dimension based on the characteristic parameters of the deep learning model to obtain the behavior profile.

[0102] In this embodiment, the functions of each unit are the same as those described above. Figure 1 , Figure 2a , Figure 2b The steps in the illustrated embodiments correspond to each other and will not be described again here.

[0103] See also Figure 4 , another embodiment provided in the embodiments of the present application includes: Processor 401, memory 402, input and output unit 403, bus 404; The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404; Processor 401 specifically performs Figure 1 , Figure 2a , Figure 2b The operations corresponding to the steps in the method are not described in detail here.

[0104] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0106] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, etc. Various media that can store program codes.

Claims

1. A human gait analysis method based on video data, characterized in that: The human body gait analysis method comprises: Get the video feed from the camera; Determine the identity of the person in the feedback video according to face recognition technology, and establish a health-related file according to the identity of the person, wherein the health-related file contains physical health data corresponding to the identity of the person; Establishing a deep learning framework, and segmenting the feedback video using a Mask R-CNN model in the deep learning framework to determine the human body area; Extract key points of the human body region by using a human posture estimation tool to generate skeleton data; Performing behavior segmentation on the feedback video to obtain valid video clips as behavior analysis samples; Extracting gait features of the skeleton data in the behavior analysis sample by using visual technology; Acquiring dynamic data of the gait features through an optical flow algorithm; Establishing a behavior profile based on the dynamic data; Inputting the behavior profile into a deep learning model, and analyzing the behavior profile according to a time series by the deep learning model to obtain an analysis result; Performing a risk score on the analysis result in combination with the physical health data; When the risk score reaches an alarm threshold, a corresponding feedback strategy is executed according to the risk score.

2. The human gait analysis method according to claim 1, characterized in that: The step of inputting the behavior profile into a deep learning model and analyzing the behavior profile according to a time series by the deep learning model to obtain an analysis result includes: Determine the model parameters of the deep learning model corresponding to the character identity according to the physical health data to obtain a first stability factor; Inputting the behavior data into a target deep learning model, and performing action trend analysis on the behavior profile according to a time series using the model parameters to obtain a second stability factor; An analysis result is generated based on the first stabilization factor and the second stabilization factor.

3. The human gait analysis method according to claim 2, characterized in that: The step of performing risk scoring on the analysis result in combination with the physical health data comprises: Obtaining basic risk parameters according to the physical health data; Using the basic risk parameter as a risk score offset, bidirectionally offset the first stability factor to obtain a safety interval; A risk score is generated according to the positional relationship between the second stability factor and the safety interval.

4. The human gait analysis method according to claim 1, characterized in that: The step of performing behavior segmentation on the feedback video to obtain valid video segments as behavior analysis samples includes: Acquire a behavior template, and perform behavior marking on the feedback video according to the behavior template; The feedback video is cut according to the behavior marker to obtain a behavior analysis sample.

5. The human gait analysis method according to claim 4, characterized in that: The step of extracting the gait features of the skeleton data in the behavior analysis sample by using visual technology includes: The skeleton data is cleaned by geometric constraints, and gait features of the skeleton data are extracted, wherein the gait features include time domain features, frequency domain features and spatiotemporal features.

6. The human gait analysis method according to claim 5, characterized in that: The step of obtaining the dynamic data of the gait feature by using an optical flow algorithm includes: Calculate the motion vector of the skeleton key point between consecutive frames of the behavior analysis sample by Lucas-Kanade optical flow method; The motion vector is reconstructed by trajectory, and the changes of the joint angles of the key points of the skeleton over time are calculated to obtain dynamic data.

7. The human gait analysis method according to any one of claims 1 to 6, characterized in that: The establishing of a behavior profile according to the dynamic data includes: parsing behavioral components in the dynamic data; Generate a sequence of the behavior components and obtain the duration of each behavior in the behavior components to obtain behavior profile sample data; The behavior portrait sample data is reduced in dimension based on the characteristic parameters of the deep learning model to obtain a behavior portrait.

8. A human gait analysis system based on video data, characterized in that: The human body gait analysis system comprises: A first acquisition unit is used to acquire a feedback video from a camera; A first determining unit is used to determine the identity of a person in the feedback video according to face recognition technology, and to establish a health-related file according to the identity of the person, wherein the health-related file contains physical health data corresponding to the identity of the person; A segmentation unit, used to establish a deep learning framework, and segment the feedback video through a Mask R-CNN model in the deep learning framework to determine a human body area; A first extraction unit, configured to extract key points of the human body region by using a human body posture estimation tool to generate skeleton data; A cutting unit, used for performing behavior cutting on the feedback video to obtain valid video clips as behavior analysis samples; A second extraction unit, configured to extract gait features of the skeleton data in the behavior analysis sample by using a visual technique; A second acquisition unit, used for acquiring dynamic data of the gait feature by using an optical flow algorithm; An establishing unit, used for establishing a behavior profile according to the dynamic data; An analysis unit, configured to input the behavior profile into a deep learning model, and analyze the behavior profile according to a time series through the deep learning model to obtain an analysis result; a risk scoring unit, configured to perform risk scoring on the analysis result in combination with the physical health data; An alarm unit is used to execute a corresponding feedback strategy according to the risk score when the risk score reaches an alarm threshold.

9. A human gait analysis device based on video data, characterized in that: The human gait analysis device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the method according to any one of claims 1 to 7 is performed.

Images