Abnormal gait evaluation method and system based on monocular video, computing device

Abstract

The application discloses a gait abnormality evaluation method and system based on monocular video, and a computing device. The method comprises the following steps: acquiring a human skeleton point sequence of the monocular video; performing data correction on the acquired human skeleton point sequence; inputting the corrected human skeleton point sequence into a pre-trained spatio-temporal graph convolution network model, and acquiring an output result of the spatio-temporal graph convolution network model, wherein the output result comprises an evaluation result of a gait abnormality of a subject in the monocular video. The method can effectively solve problems such as left-right leg confusion and key points being easily affected by occlusion in the extraction of skeleton points in the monocular video, and can enhance the capture and understanding of spatio-temporal dynamic characteristics of the gait, realize accurate extraction of spatio-temporal characteristics of the human gait, improve the accuracy and robustness of the monocular video gait analysis, and enable effective gait analysis by using an ordinary monocular camera (such as a smart phone), so that the cost is lower and the application scenarios are more extensive.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an abnormal gait evaluation method and system, computing device and storage medium based on monocular video. Background Technology

[0002] Gait is a fundamental and complex motor function in humans, and changes in its pattern are often early indicators or important manifestations of various health conditions, such as Parkinson's disease (PD), post-stroke sequelae, osteoarthritis, and aging. In recent years, with the development of computer vision and artificial intelligence technologies, gait analysis using video data has become a research hotspot.

[0003] Currently, video-based gait analysis methods include 3D skeletal keypoint extraction and analysis methods that rely on multi-camera systems, special markers, or depth sensors to obtain 3D information, and 2D skeletal keypoint extraction and analysis methods based on monocular video. The former relies on multi-camera arrays, depth cameras, or dedicated motion capture systems, resulting in complex operation, high equipment costs, and difficult deployment. The latter, when using monocular RGB video, struggles to accurately capture the 3D spatial motion characteristics of key points throughout the body. In complex scenes, such as those with occlusion, lighting changes, and clothing effects, the accuracy and stability of skeletal keypoint extraction are challenged, especially for the accurate capture of extremities (such as wrists and ankles). For example, when alternating between left and right legs in a lateral view, existing skeletal extraction models may misidentify leg keypoints, leading to incorrect gait parameters. This severely affects the accuracy of subsequent gait feature analysis and anomaly detection. Summary of the Invention

[0004] This invention provides a method and system for abnormal gait evaluation based on monocular video, in order to solve the problems of low accuracy, insufficient robustness, incomplete feature extraction, and possible confusion in key point extraction when using conventional monocular cameras (such as smartphone cameras) for gait anomaly detection in the prior art.

[0005] According to a first aspect of the present invention, an abnormal gait assessment method based on monocular video is provided, comprising:

[0006] Obtain the human skeleton point sequence from the monocular video;

[0007] Data correction is performed on the acquired human skeletal point sequence;

[0008] The corrected human skeletal point sequence is input into a pre-trained spatiotemporal graph convolutional network model, and the output of the spatiotemporal graph convolutional network model is obtained. The output includes the evaluation results of the gait abnormalities of the subject in the monocular video.

[0009] According to a second aspect of the present invention, an abnormal gait assessment method based on monocular video is provided for use on a mobile device, comprising:

[0010] Obtain the human skeleton point sequence from the monocular video;

[0011] Data correction is performed on the acquired human skeletal point sequence;

[0012] The corrected human skeleton point sequence is sent to the server, and the evaluation results of the gait abnormalities of the subject in the monocular video are obtained from the server. The server is used to input the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model, and to feed back the evaluation results to the mobile device based on the output of the spatiotemporal graph convolutional network model.

[0013] According to a third aspect of the present invention, an abnormal gait assessment system based on monocular video is provided, comprising a mobile device and a cloud server, wherein,

[0014] The mobile device is used to capture lateral video of the subject walking using a monocular camera and send it to the cloud server.

[0015] The cloud server includes

[0016] The skeleton point extraction module is used to extract key points of the human skeleton from the lateral video.

[0017] The video segmentation module is used to segment the lateral video based on the key points of the human skeleton and generate a sequence of human skeleton points for monocular video.

[0018] The data correction module is used to correct the generated human skeletal point sequence; and

[0019] The gait assessment module is used to input the corrected human skeletal point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output results of the spatiotemporal graph convolutional network model, wherein the output results include the assessment results of the gait abnormalities of the subject in the monocular video.

[0020] According to a fourth aspect of the present invention, an abnormal gait assessment system based on monocular video is provided, comprising a mobile device and a cloud server, wherein,

[0021] Mobile devices include

[0022] A monocular camera used to capture lateral video of subjects walking;

[0023] The skeleton data extraction module is used to perform video segmentation processing on the lateral video to generate the human skeleton point sequence of the monocular video, and send the generated human skeleton point sequence of the monocular video to the cloud server.

[0024] The cloud server includes

[0025] The data correction module is used to correct the acquired human skeletal point sequence.

[0026] The gait assessment module is used to input the corrected human skeletal point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output results of the spatiotemporal graph convolutional network model, wherein the output results include the assessment results of the gait abnormalities of the subject in the monocular video.

[0027] According to a fifth aspect of the present invention, a computing device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method described in the first aspect above.

[0028] According to a sixth aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0029] The abnormal gait evaluation method of this invention can effectively solve the unique problems of skeletal point extraction in monocular videos (such as confusion between left and right legs, limited viewpoints, and susceptibility to occlusion of key points) by correcting the extracted human skeletal point sequence. Simultaneously, the method of this invention inputs the corrected skeletal point sequence into a trained spatiotemporal graph convolutional network (ST-GCN) to automatically learn the spatiotemporal features of gait, enhancing the capture and understanding of gait spatiotemporal dynamic features. This effectively overcomes the inherent limitation of lacking depth information in monocular video, enabling accurate extraction of human gait spatiotemporal features, improving the accuracy and robustness of monocular video gait analysis, and allowing for effective gait analysis using ordinary monocular cameras (such as smartphones), resulting in lower costs and wider application scenarios. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 The flowchart of an abnormal gait evaluation method based on monocular video according to one embodiment of the present invention is shown schematically.

[0032] Figure 2 A flowchart illustrating the implementation method of operation S1 according to one embodiment of the present invention is shown schematically.

[0033] Figure 3 A flowchart illustrating a method for video segmentation processing according to an embodiment of the present invention is shown schematically.

[0034] Figure 4 A flowchart illustrating the implementation method of operation S2 according to one embodiment of the present invention is shown schematically.

[0035] Figure 5 A flowchart illustrating the implementation method of operation S4 according to one embodiment of the present invention is shown schematically.

[0036] Figure 6 The schematic diagram illustrates the principle architecture of a spatiotemporal graph convolutional network model according to an embodiment of the present invention;

[0037] Figure 7 The diagram schematically illustrates the architecture of an abnormal gait assessment system based on monocular video according to an embodiment of the present invention.

[0038] Figure 8 The diagram schematically illustrates the architecture of an abnormal gait assessment system based on monocular video according to another embodiment of the present invention.

[0039] Figure 9 A schematic block diagram of a computing device according to an embodiment of the present invention is shown.

[0040] Figure 10 The schematic diagram shows a hardware structure diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0043] In the description of this application, it should be understood that the use of terms such as "center," "middle," "longitudinal," "transverse," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential" to indicate orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings is solely for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Features defined with "first" and "second" are used to distinguish feature names and do not have special meanings. Furthermore, features defined with "first" and "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more.

[0044] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0045] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terminology used herein is generally that commonly used by those skilled in the art; in case of any discrepancy with commonly used terminology, the terminology used herein shall prevail.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The present invention will now be described in further detail with reference to the accompanying drawings.

[0048] The abnormal gait assessment method in this invention can be applied to any mobile terminal device and also to a server, so that these mobile terminal devices or servers can use the solution of this invention to perform gait anomaly monitoring, and gait data collection can be completed using only a monocular camera device, taking into account accuracy, robustness and equipment cost, and has a wide range of application scenarios.

[0049] Figure 1 The flowchart illustrating an abnormal gait evaluation method according to one embodiment of the present invention is shown schematically. The execution entity of this method can be, for example, a processor of a computer, a cloud server, a server in a cluster, a mobile terminal device, etc., and the embodiments of the present invention are not limited thereto. Figure 1 As shown, it includes:

[0050] Operate S1 to obtain the human skeleton point sequence of the monocular video;

[0051] Operation S2 corrects the acquired human skeletal point sequence.

[0052] Operation S3 involves inputting the corrected human skeletal point sequence into a pre-trained spatiotemporal graph convolutional network model and obtaining the output of the spatiotemporal graph convolutional network model, wherein the output includes the evaluation results of the gait abnormalities of the subject in the monocular video.

[0053] In this embodiment of the invention, the monocular video can be obtained by segmenting the lateral video of the subject walking captured by a monocular camera, or it can be the lateral video of the subject walking directly captured by a monocular camera device. When capturing the lateral video, a common monocular RGB camera (such as the camera built into a smartphone) can be used to capture the subject's walking video from the side. Preferably, the subject can complete several round trips to obtain sufficient gait cycle data. More preferably, the video frame rate and resolution of the monocular camera used must meet the requirements of the subsequent human skeletal keypoint extraction algorithm; for example, the RGB frame resolution can be 1920x1080 and the frame rate 30fps. In other embodiments, video capture can also be correctly completed under the guidance of an existing mobile application (APP) on a mobile terminal device; this embodiment of the invention does not impose any limitations on this.

[0054] When the monocular video is a lateral video of a subject walking, directly acquired using a monocular camera, in operation S1, obtaining the human skeleton point sequence of the monocular video can be achieved by using a lightweight or efficient 2D human pose estimation algorithm (such as an optimized version of DWPose, MediaPipe Pose, etc.) to extract the 2D image coordinates (x, y) and corresponding confidence scores (c) of 20 or more human skeletal key points (such as head, neck, shoulder, elbow, wrist, hip, knee, ankle, toes, etc.) frame by frame, thus forming a human skeleton point sequence for each human pose. The human skeleton point sequence refers to a set of temporal data representing continuous body movements, formed by concatenating the 2D image coordinates (x, y) of the extracted human skeletal key points frame by frame in the temporal order of the video frames. In some implementations, the confidence score (c) corresponding to the 2D image coordinates may also be included.

[0055] To accurately extract clean unidirectional walking segments for analysis from raw video footage that may contain non-walking phases such as preparation, turning, and ending, thereby improving the quality and accuracy of the extracted human skeletal point sequence, as a preferred embodiment of the present invention, the monocular video is obtained by segmenting the lateral video of the subject walking captured by a monocular camera. In this case, operation S1 can be implemented as follows: Figure 2 The method flow shown includes:

[0056] Operate S11 to extract key points of the human skeleton from the lateral video of the subject walking, captured by the monocular camera;

[0057] Operation S12: Perform video segmentation processing on the lateral video based on the extracted human skeletal key points;

[0058] Operation S13 generates a sequence of human skeleton points in the monocular video based on the video segmentation processing results.

[0059] In operation S11, the human skeletal key points include a set of coordinate points corresponding to each single frame image in the lateral video. This includes the set of coordinate points for the major joints of the human body identified in each single frame image, such as the head, neck, shoulder, elbow, hip, knee, and ankle joints. This set of coordinate points corresponds one-to-one with each single frame image. The human skeletal key points corresponding to each frame image are extracted directly from each frame image of the lateral video using a 2D human pose estimation algorithm. For example, lightweight or efficient 2D human pose estimation algorithms such as DWPose or MediaPipe Pose, based on deep learning key point detection models, can be used to analyze the lateral video frame by frame, thereby automatically identifying and locating a series of key joints of the human body in each frame (including twenty or more key points such as both shoulders, both knees, and both elbows). For each identified key point (such as the left and right shoulders here), its pixel coordinates (x, y) on the two-dimensional image and the corresponding confidence level c are output, thus obtaining the human skeletal key points of the corresponding frame image.

[0060] in, Figure 3 The diagram schematically illustrates the video segmentation processing method flow of one embodiment, specifically operation S12. Figure 3 As shown, it can be implemented as including:

[0061] Operation S121: Obtain key points of the human skeleton and determine motion anchor points based on the obtained key points of the human skeleton.

[0062] Operation S122: Generate an instantaneous horizontal velocity time series based on the motion anchor point and the lateral video;

[0063] Operation S123: Based on the instantaneous horizontal velocity time series, identify and locate the effective walking range and potential turning points in the lateral video;

[0064] Operation S124: Segment the lateral video according to the effective walking range and potential turning points and generate at least one unidirectional walking segment;

[0065] Operation S125: Determine the monocular video for abnormal gait assessment based on at least one generated unidirectional walking segment.

[0066] As one possible implementation, in operation S121, one or more relatively stable points on the torso during walking can be selected from the key points of the human skeleton corresponding to each frame image as motion anchor points. For example, the "neck" key point can be used as a motion anchor point, or the center points of the left and right "shoulder" key points can be calculated as motion anchor points. That is, from the set of coordinate points corresponding to each frame image, a coordinate point can be selected as the motion anchor point corresponding to that frame image. This coordinate point can be the coordinate of the "neck" key point or the coordinate of the center point of the left and right shoulders. The method for calculating the center point coordinates of the left and right shoulders can be to add the x-coordinates of the left and right shoulders and divide by 2 to obtain the x-coordinate of the center point; then add their y-coordinates and divide by 2 to obtain the y-coordinate of the center point. When walking, a person's limbs (arms and legs) swing in large, periodic motions, while the torso is mainly responsible for overall forward translation. Its movement trajectory is relatively smooth and does not involve high-frequency reciprocating motion. Therefore, in this embodiment of the invention, points located on the torso, such as the center point of the "neck" or the left and right "shoulders", are preferably considered as the "relatively most stable" points. By selecting the relatively most stable points on the torso as motion anchor points, high-frequency interference caused by limb swings can be effectively filtered out, and the overall translational motion of the human body's center of gravity can be more accurately reflected and captured, which helps in the precise segmentation of the video.

[0067] In operation S122, the instantaneous horizontal velocity time series is a set of discrete numerical data (which can be implemented as a one-dimensional list or array of numbers), where each value represents the subject's horizontal movement speed and direction at a specific moment (i.e., a frame). As one possible implementation, the instantaneous horizontal velocity time series can be generated by tracking the coordinate changes in the horizontal direction of the motion anchor points determined in operation S121 in each frame of the lateral video. Since the motion anchor points correspond one-to-one with each frame, a raw instantaneous horizontal velocity time series representing the body's forward or backward movement can be generated by calculating the coordinate displacement of the determined motion anchor points between consecutive frames of the lateral video. Since the forward or backward movement of the human body in a side-view walking video (i.e., a side-view video) is mainly reflected on the horizontal axis of the image, tracking the change in the x-coordinate is sufficient to reflect its core movement trend. Based on this, the coordinate displacement of the motion anchor point between consecutive frames of the side-view video calculated in this embodiment of the invention can specifically be the offset of the x-coordinate of the motion anchor point between consecutive frames. Thus, the Nth number in the instantaneous horizontal velocity time series can be obtained by calculating the difference between the x-coordinate of the motion anchor point in the (N+1)th frame and the x-coordinate in the Nth frame. The magnitude and positive or negative attribute of this difference can represent the movement speed and direction of the motion anchor point. For example, a positive value indicates that the motion anchor point is moving in a certain direction (e.g., to the right), and the larger the value, the faster the movement speed; a negative value indicates that the motion anchor point is moving in the opposite direction (e.g., to the left), and the larger the absolute value, the faster the movement speed; while a value close to zero indicates that the motion anchor point is basically stationary, which usually occurs when the subject is walking very slowly or is turning around. For example, suppose that for a certain lateral video, the x-coordinates of the motion anchor points extracted by operation S121 over five consecutive frames are [150, 155, 160, 159, 154]. Then, in operation S122, by calculating the coordinate differences between consecutive frames, the instantaneous horizontal velocity time sequence generated is [5, 5, -1, -5]. This generated sequence indicates that the subject first moves to the right at a relatively fast speed (speed 5), then slows down and begins to turn (speed becomes -1), and finally returns to the left at a relatively fast speed (speed -5). Therefore, by analyzing the changes in sign and magnitude of values ​​in this sequence, the effective walking range and potential turning points can be located.

[0068] Since the generated raw instantaneous horizontal velocity time series typically contains high-frequency noise caused by minor jitters in the keypoint detection model or slight swaying of the subject's body, in some preferred embodiments, a low-pass filter (e.g., a moving average filter or a Gaussian filter) can be further used to smooth the generated instantaneous horizontal velocity time series to eliminate these interferences, thereby obtaining a smooth velocity curve that clearly reflects the trend of the subject's movement. The specific smoothing process can refer to existing low-pass filter technologies, which will not be elaborated upon in detail in this embodiment. Furthermore, it should be noted that the instantaneous horizontal velocity time series in this embodiment focuses on describing the raw, discrete, and noisy set of data points before processing. When this set of data points is plotted on a graph with time as the horizontal axis and velocity as the vertical axis, it appears as a velocity curve. Therefore, the smoothed velocity curve mentioned here mainly focuses on describing the state of the data in the sequence after low-pass filter processing. Since the noise has been filtered out, its graphical representation (i.e., the curve) becomes smooth and can clearly reflect the macroscopic trend of the subject's movement.

[0069] In operation S123, a motion activation speed threshold can be preset. This threshold needs to be significantly higher than the speed noise generated by body swaying in a stationary state. The core meaning of "significantly higher" here is that the set speed threshold must be able to clearly distinguish between "intentional walking" and "unintentional body swaying." In specific applications, this speed threshold can be set based on the characteristics of the specific scenario and prior experience, as long as it ensures that the absolute value of the subject's speed only stably exceeds this threshold when the subject actually begins walking. For example, assuming the pre-measured absolute value of the subject's maximum speed noise is 0.5, a reasonable "motion activation speed threshold" could be 2.0 or 3.0. During recognition, the instantaneous horizontal speed time series determined in operation S122 can be traversed, and all data values ​​in the series can be compared with the set motion activation speed threshold to filter out all time periods where the absolute value of the speed is consistently higher than this threshold. These time periods are marked as valid walking intervals. Specifically, a new binary sequence (such as a binary sequence composed of 0 and 1) can be created in advance. When comparing the absolute value of each velocity value in the instantaneous horizontal velocity time series with a preset motion activation velocity threshold, if the absolute value of the velocity is greater than the threshold, the corresponding position in the binary sequence is recorded as 1 (representing motion); if the absolute value of the velocity is not greater than the threshold, it is recorded as 0 (representing stillness). For example, for the instantaneous horizontal velocity time series [0.5, 1.5, 5.0, 6.2, 4.8, 1.2, 0.8], assuming the motion activation velocity threshold is 2.0, the generated binary sequence is [0, 0, 1, 1, 1, 0, 0]. Next, all time periods where the absolute velocity value is consistently higher than a threshold can be filtered out by finding all segments with consecutive 1s in this binary sequence. This can be achieved through a single traversal. The process can be implemented as follows: First, initialize an empty list `active_periods` to store the results, and set a variable `start_frame = null`. Then, traverse from the beginning of the binary sequence. When the first 1 is encountered, if `start_frame` is empty, record the current frame number and store it in `start_frame`. This marks the beginning of a valid walking interval. Continue traversing. When the first 0 is encountered, if `start_frame` is not empty, it means that a segment with consecutive 1s has just ended. At this point, store `start_frame` and the previous frame number (current frame - 1) as a complete time period (e.g., a tuple containing the start and end frames) in the `active_periods` list, and then reset `start_frame` to empty. Repeat this process until the entire binary sequence has been traversed.By following the steps above, all time periods in which the absolute value of velocity consistently exceeds this threshold can be accurately identified; these time periods are the effective walking intervals. Identifying the effective walking intervals effectively eliminates the static or preparation phases at the beginning and end of the lateral video, thereby reducing video noise and data processing load, saving resources, and improving data processing efficiency and performance. Furthermore, in operation S123, this embodiment of the invention further searches for the moments within these effective walking intervals when the velocity curve corresponding to the instantaneous horizontal velocity time series crosses zero; these moments correspond to potential turning points where the subject changes their walking direction. One possible implementation is to iterate through the instantaneous horizontal velocity time series corresponding to the determined effective walking interval, determining whether the signs of the velocity values ​​V_t in the current frame and V_{t+1} in the next frame have changed. This can be achieved by checking if their product is negative (V_t * V_{t+1} < 0). An event of crossing a zero point is considered to have occurred if and only if the signs of V_t and V_{t+1} are different. Once this is detected, the sequence number t of the current frame or the sequence number t+1 of the next frame is recorded as the time of a potential turning point. For example, within an effective walking interval, the corresponding instantaneous horizontal velocity sequence is [5.1, 4.2, 1.5, -0.8, -3.2]. When the algorithm iterates to 1.5 and -0.8, it finds that 1.5 * (-0.8) < 0, indicating a change in sign. Therefore, this position is marked as a potential turning point. Here, the video frame number, such as the current frame number, refers to the video frame number. This frame number represents the specific temporal position of a video frame in the lateral video. The potential turning point determined based on the aforementioned method is represented by the video frame number, which represents the precise instant in the video when the subject's walking direction changes. Preferably, the instantaneous horizontal velocity time series used in operation S123 is smoothed.

[0070] In operation S124, specifically, the entire effective walking interval can be divided into several sub-segments using the identified potential turning points as boundaries. Each sub-segment is recorded as a unidirectional walking segment. For example, assuming that an effective walking interval is first determined through operation S123 from frame 100 to frame 500 of the video, then operation S123 analyzes the potential turning points in the instantaneous horizontal velocity sequence of these 401 frames (from 100 to 500). Assuming that the velocity value changes from positive to negative between frames 249 and 250, and changes back to positive between frames 380 and 381, then the first potential turning point can be located as 250, and the second potential turning point as 381. Thus, for this effective walking interval, the final generated potential turning point data is a list containing two frame numbers: [250, 381]. In operation S124, the original video frame interval [100, 500] is divided using this list [250, 381] as the boundary. The resulting first unidirectional walking segment is from frame 100 to frame 249, the second unidirectional walking segment is from frame 250 to frame 380, and the third unidirectional walking segment is from frame 381 to frame 500. In this way, a lateral walking video containing two turns is precisely divided into three unidirectional walking sub-segments, i.e., it is divided into three unidirectional walking segments.

[0071] In operation S125, specifically for each unidirectional walking segment, it can be further confirmed whether the signs of the velocity values ​​in the instantaneous horizontal velocity sequence corresponding to the unidirectional walking segment are consistent (i.e., all are positive or all are negative). If the signs are consistent, the segment is confirmed as a valid unidirectional walking segment and extracted as a monocular video for subsequent analysis. Specifically, for each unidirectional walking segment, the embodiment of the present invention will find the corresponding part in the instantaneous horizontal velocity sequence determined in operation S122. This part is a subsequence composed of continuous velocity values ​​in the instantaneous horizontal velocity sequence. For this subsequence, operation S125 will further check the positive and negative attributes of each velocity value in this subsequence. Only when the positive and negative attributes of all velocity values ​​in the subsequence are consistent is the unidirectional walking segment corresponding to the subsequence considered as a valid unidirectional video, that is, the subject is indeed walking in a single direction during this time period without any complex non-unidirectional movements such as turning back, hesitating, or stepping in place. It should be noted that in the video analysis of lateral walking, the sign of the velocity value directly reflects the subject's horizontal movement direction in the frame. A positive velocity value usually indicates that the subject is moving in the positive x-axis direction of the image coordinate system, for example, from the left side of the frame to the right side; a negative velocity value indicates that the subject is moving in the negative x-axis direction, such as from the right side of the frame to the left side (the specific direction of movement depends on the camera's placement). It should also be noted that the further verification in operation S125 can improve the validity and accuracy of the determined unidirectional walking segments. In other embodiments, this verification process can be skipped, and the monocular video can be directly extracted using the processing result of operation S124. This embodiment of the invention is not considered a limitation thereof.

[0072] Therefore, the method of this invention, by segmenting the acquired lateral video, can reliably and automatically divide a round-trip walking video (e.g., walking towards the camera and then away from the camera, walking from the left side of the camera to the right side) into two or more independent gait samples with a single direction. This allows for the precise extraction of clean unidirectional walking segments for analysis (i.e., the monocular video used for analysis in this invention), improving the accuracy of subsequent analysis and reducing the amount of data processing required. The video segmentation algorithm of this invention extracts unidirectional monocular video by tracking the overall translational motion of the human body's center of gravity rather than the specific swinging of the limbs. This effectively filters out high-frequency interference caused by limb swinging, resulting in a smooth and clean velocity curve (characterized by an instantaneous horizontal velocity sequence). Therefore, based on this velocity curve, effective walking intervals and potential turning points can be identified more accurately, achieving precise video segmentation.

[0073] In operation S13, the human skeleton key points corresponding to each image frame of the segmented monocular video are sorted according to the frame number of each image frame in the monocular video to form a data sequence sorted by time, thus obtaining the human skeleton point sequence of the monocular video.

[0074] Due to the limitations of monocular video, during lateral walking, the skeletal extraction model often confuses the key points of the left and right ankles due to the alternation of legs. Therefore, the 2D skeletal points extracted using the aforementioned algorithm may suffer from this confusion. To improve data quality and avoid the adverse effects of the extracted 2D skeletal point data on the accuracy and robustness of gait analysis results, this embodiment of the invention, after obtaining the human skeletal point sequence from the monocular video, will further automatically correct the obtained human skeletal point sequence in operation S2. Iterative adjustments are made to ensure the smoothness and naturalness of the ankle movement trajectory, thereby making the gait assessment using this skeletal point sequence more reliable. Figure 4 The schematic diagram illustrates the specific implementation method flow of operation S2 in one embodiment of the present invention, such as... Figure 4 As shown, data correction of the acquired human skeletal point sequence can be specifically implemented by including:

[0075] Operation S201 generates a first distance trajectory time series based on the human skeleton point sequence of the left ankle and the human skeleton point sequence of the right ankle in the human skeleton point sequence of the monocular video;

[0076] Operation S202 initializes the second distance trajectory time series;

[0077] Operation S203 involves performing data correction processing on the human skeleton point sequence of the monocular video frame by frame, based on the first distance trajectory time series, the second distance trajectory time series, and the pre-built motion trend model.

[0078] In operation S201, the specific implementation process of generating the first distance trajectory time series can be as follows: for the left ankle and right ankle human skeleton point sequences corresponding to the monocular video, calculate the horizontal coordinate difference between the left and right ankle human skeleton points for each video frame frame by frame. Then, form the first distance trajectory time series according to the frame number of the video frame using the calculated horizontal coordinate difference. The horizontal coordinate difference can be the difference of the x-values ​​of the 2D image coordinates (x, y) of the left and right ankle human skeleton point sequences. For example, for the monocular video to be corrected, assuming it has n frames, the x-coordinate of the left ankle in the first frame is x_left_1, and the x-coordinate of the right ankle is x_right_1. Then, the horizontal coordinate difference D_1 = x_left_1 - x_right_1 for the first frame video can be obtained by repeating this calculation process. ...,D_n, these horizontal coordinate differences (also referred to as distance values ​​in this embodiment of the invention) are combined into a list or array according to the video time sequence (e.g., according to the video frame number). This list of differences arranged in frame order forms the first distance trajectory time series. The first distance trajectory time series records the horizontal position relationship between the left ankle and the right ankle in each frame of the video. It can be regarded as the projection of the front-back distance of the two feet in the lateral view onto a two-dimensional plane. During normal walking, as the two legs alternately step forward and backward, this distance value will show a periodic, sine wave-like smooth change. Therefore, the smoothness of the sequence can be used to determine whether there are unreasonable jumps in the value caused by the misidentification of the left and right feet, so as to correct the data for the confusion of the left and right ankles.

[0079] To predict and evaluate the smoothness of the generated first distance trajectory time series, in operation S202, a second distance trajectory time series for storing the corrected distance trajectory is first initialized. Specifically, the initialized second distance trajectory time series has the same data format as the first distance trajectory time series, such as a one-dimensional list or array of values. Initialization also includes using the first value of the first distance trajectory time series directly as the initial value of the first value of the second distance trajectory time series. For example, assuming the first distance trajectory time series is [15, 2, -2, 15, -25], initialization includes creating a data structure with the same format as the first distance trajectory time series and reading the first value 15 from the first distance trajectory time series as the initial value of the second distance trajectory time series; that is, the initialized second distance trajectory time series is

[15] . It should be noted that the second distance trajectory time series stores the true horizontal ankle distance value (i.e., the difference between the true horizontal ankle coordinates) that is considered to best conform to the motion law after frame-by-frame judgment. In other words, it is the distance trajectory time series after error correction (swapping the coordinates of the left and right feet) based on the original distance trajectory. This distance trajectory time series is dynamically generated during the data correction process, and it is only initialized before the data correction process begins.

[0080] In operation S203, the pre-built motion trend model is a dynamic mathematical model that predicts motion trends by approximating velocity and acceleration using finite differences. This model can be implemented as a mathematical constraint based on difference equations to force a smooth motion trajectory. The core idea of ​​this model is the assumption of inertia or smooth motion, meaning that the acceleration of an object will not change drastically without sudden external force changes. Therefore, the model predicts the range of the distance value at the next moment to minimize the change in acceleration based on historical distance, velocity, and acceleration. For example, this motion trend model can be implemented by approximating the (relative) velocity v_t at time t as the distance value of the current frame minus the distance value of the previous frame, i.e., v_t ≈ D_t - D_{t-1}, and approximating the (relative) acceleration a_t at time t as a_t ≈ v_t - v_{t-1} ≈ (D_t - D_{t-1}) - (D_{t-1} - D_{t-2}). Here, the frame in the monocular video that needs data correction processing is denoted as the current frame. Figure 5 This illustration schematically demonstrates a process for correcting the human skeleton point sequence of the current frame based on the motion trend model and the first distance trajectory time series, as described in one implementation. In practical applications, this process can be performed frame by frame on a monocular video to complete the data correction of the entire human skeleton point sequence corresponding to that video. Figure 5As shown, the data correction processing performed on the human skeleton point sequence of the current frame includes:

[0081] In operation S204, the displacement change caused by connecting the first distance trajectory time series to the second distance trajectory time series under the first and second hypothetical scenarios is calculated based on the motion trend model, the first distance trajectory time series, and the second distance trajectory time series, respectively.

[0082] Operation S205: Determine the trajectory smoothness score of the current frame under the first and second hypothetical scenarios based on the displacement change.

[0083] Operation S206 determines whether the left and right ankles of the current frame are confused based on the trajectory smoothness scores corresponding to the first and second hypothetical scenarios. If confusion is found, the data correction is performed on the human skeleton point sequence corresponding to the current frame by swapping the coordinate data of the left and right ankles.

[0084] For the current frame to be processed, in operation S204, the displacement change caused by integrating the first distance trajectory time series corresponding to the current frame into the second distance trajectory time series under different scenarios is first calculated. Preferably, this embodiment of the invention proposes two hypothetical scenarios: the first hypothetical scenario is a correctly identified scenario, in which it is assumed that the identification of the left and right ankles is correct when extracting the human skeletal point sequence; the second hypothetical scenario is a misidentified scenario, in which it is assumed that the left and right ankles are confused when extracting the human skeletal point sequence. In this case, the true distance value should be the negative of the distance value corresponding to the current frame in the first distance trajectory time series. Calculating the displacement change under different scenarios means calculating, respectively, how much the distance value corresponding to the current frame in the first distance trajectory time series will change if integrated into the second distance trajectory time series (i.e., the corrected trajectory) under each of the two hypothetical scenarios. It should be noted that "access" here means tentatively appending or connecting the distance value corresponding to the current frame in the first distance trajectory time series to the end of the second distance trajectory time series. For example, assuming the current frame is the second frame, the first distance trajectory time series is [15, 2, -2, 15, -25], and the initialized second distance trajectory time series is

[15] . In this case, accessing the distance value corresponding to the current frame in the first distance trajectory time series to the second distance trajectory time series means adding the second value 2 of the first distance trajectory time series to the second distance trajectory time series

[15] , and adding it after 15. It needs further explanation that, since a smooth and correct second distance trajectory time series already exists from the first frame to the previous frame t-1 when processing the current frame t, the correctness of the distance values ​​in the first distance trajectory time series corresponding to the current frame needs to be verified when updating the second distance trajectory time series based on the current frame (i.e., the distance values ​​in the first distance trajectory time series may contain errors). Therefore, the actual "access" is to temporarily add a candidate distance value (e.g., the original value or its negative value) from the current frame to the end of this corrected second distance trajectory time series to assess whether this addition will disrupt the smoothness of the second distance trajectory time series, and to determine the final corrected distance value to be added to the second distance trajectory time series based on the assessment result. Thus, the displacement change in operation S204 measures the difference between the newly accessed candidate value and the theoretical value predicted based on the historical trajectory. The theoretical value predicted based on the historical trajectory is specifically obtained by the constructed motion trend model based on the historical trajectory.

[0085] In this process, each time a new frame's data is processed and its corrected distance value is determined, this corrected distance value is added to the end of the second distance trajectory time series. The motion trend model then immediately uses this latest trajectory to recalculate the velocity (first-order difference) and acceleration (second-order difference) at its end; this calculation is real-time and dynamic. When determining the corrected distance value for frame t, the motion trend model relies on the velocity and acceleration calculated from all previously corrected distance values ​​from frames t-1, t-2, and so on. Once the corrected distance value for frame t is determined, it becomes known historical data for calculating frame t+1. Therefore, the historical trajectory upon which the motion trend model is based specifically includes the distance values ​​from the first frame to the second distance trajectory time series from frame t-1.

[0086] As one possible implementation, for the current frame t, the candidate values ​​for temporary new access include candidate values ​​under a first hypothetical scenario and candidate values ​​under a second hypothetical scenario. Under the first hypothetical scenario (assuming correct identification), the candidate value is the distance value D_raw_t corresponding to the current frame t in the first distance trajectory time series. Under the second hypothetical scenario (assuming incorrect identification), the candidate value is set to the negative of the distance value corresponding to the current frame in the first distance trajectory time series, i.e., -D_raw_t. Therefore, calculating the displacement change in operation S204 includes calculating the absolute value of the difference between the candidate value under the first hypothetical scenario and the predicted theoretical value, and calculating the absolute value of the difference between the candidate value under the second hypothetical scenario and the predicted theoretical value. Assuming the predicted theoretical value is denoted as Predicted_D_t, then the displacement change under the first hypothetical scenario = abs(D_raw_t - Predicted_D_t), and the displacement change under the second hypothetical scenario = abs(-D_raw_t - Predicted_D_t). The smaller the displacement change, the more the newly added candidate value conforms to the inertia of historical motion, i.e., the smoother the trajectory. It will serve as a core component in calculating the trajectory smoothness score. It should be noted that the candidate value is only a possible option for inclusion in the second distance trajectory time series. Only the candidate value deemed correct after smoothness testing will be truly included in the second distance trajectory time series as the corrected distance value.

[0087] To select from the two hypothetical scenarios mentioned above, in operation S205, this embodiment of the invention calculates the trajectory smoothness score under different hypothetical scenarios corresponding to the current frame. This score is a comprehensive index, which generates the final smoothness score by accumulating three kinematic components. Therefore, its score can fully reflect whether adding the candidate value can make the entire motion trajectory appear smoother and more in line with the laws of physical motion. Preferably, these three kinematic components are the displacement change caused by adding the corresponding candidate value (i.e., the displacement change corresponding to each candidate value calculated in operation S204), the difference between the displacement change and the velocity at the previous moment, and the difference between the displacement change and the acceleration at the previous moment. For example, the weighted sum of the absolute values ​​of these three components can be used as the final trajectory smoothness score. Since a smooth motion should have a small displacement change, acceleration, and jerk, in this case, a lower score means that adding the candidate value will make the entire motion trajectory appear smoother and more in line with the laws of physical motion. The difference between the displacement change and the velocity at the previous moment measures the drasticness of the velocity change, which can be characterized by the magnitude of the newly generated acceleration. Specifically, this difference can be the absolute value of the difference between the new velocity generated after the candidate value is introduced and the historical velocity at the previous moment. For example, assuming the current frame is t, the candidate value is Candidate_D_t, and the corrected distance value of the previous frame is Corrected_D_{t-1}, then the calculated new velocity is New_v_t = Candidate_D_t - Corrected_D_{t-1}. Then, the velocity at the previous moment v_{t-1} = Corrected_D_{t-1} - Corrected_D_{t-2} can be obtained from the historical data, and the difference can be determined as abs(New_v_t - v_{t-1}). The difference between the displacement change and the acceleration at the previous moment measures the drasticness of the acceleration change. It can be characterized by the newly generated jerk, the magnitude of which reflects the smoothness of the motion change. Specifically, the difference can be the absolute value of the difference between the new acceleration generated after the candidate value is input and the historical acceleration at the previous moment. For example, assuming the current frame is t, the calculated new acceleration is New_a_t = New_v_t - v_{t-1}. Then, by obtaining the acceleration a_{t-1} at the previous moment from the historical data, the difference can be determined as abs(New_a_t - a_{t-1}).

[0088] In other possible implementations, a trend consistency judgment mechanism can be introduced when calculating the trajectory smoothness score. Specifically, this can be implemented by judging whether the motion direction generated under different hypothetical scenarios is consistent with the overall motion direction of the most recent frames. If the motion direction generated under a certain hypothetical scenario is consistent with the overall motion direction of the most recent frames, its smoothness score will receive an additional weight coefficient to adjust the overall smoothness score, making it easier to be selected as the optimal solution. The motion direction generated under different hypothetical scenarios is determined by the sign of the newly generated velocity values ​​under different hypothetical scenarios, and the calculation method for the newly generated velocity values ​​is as described above. The overall motion direction of the most recent frames is determined by analyzing the corrected historical trajectory, that is, by analyzing the velocity signs corresponding to the most recent frames. For example, the vote of the velocity signs of the most recent three frames can be taken, i.e., sign(sign(v_{t-1}) + sign(v_{t-2} + sign(v_{t-3}))). This sign represents the most recent and reliable "overall motion direction" because it reflects the most recent trend of the motion trajectory before accessing the current point. For example, the additional weighting coefficient can be an independent multiplier. The final smoothness score is adjusted by multiplying it with the calculated trajectory smoothness score. For instance, the coefficient could be a value less than 1 (e.g., 0.8). The final smoothness score under the corresponding hypothetical scenario is obtained by multiplying the previously calculated smoothness score by this additional weighting coefficient. Thus, when the motion direction of a hypothetical scenario is consistent with the overall motion direction of the most recent frames, by optimizing the overall smoothness score under that hypothetical scenario, the final selected trajectory can better reflect the actual motion trend.

[0089] After calculating the trajectory smoothness scores under different hypothetical scenarios, in operation S206, the confusion of the left and right ankles in the current frame can be determined by comparing the trajectory smoothness scores of the current frame under the two hypothetical scenarios. For example, using the aforementioned calculation method, the lower score (i.e., the smoother trajectory) can be selected as the correct state of the current frame. That is, if the score of the "misidentification" scenario is lower, it is determined that confusion of the left and right ankles occurred when extracting human skeletal key points in the current frame. In this case, correction is performed by swapping the coordinate data of the left and right ankles of the corresponding human skeletal key points in the current frame within the human skeletal point sequence. Therefore, by performing this processing frame by frame on the monocular video, i.e., iteratively executing it from the first frame to the last frame, the key point confusion problem in the entire monocular video sequence can be systematically corrected, resulting in higher accuracy of the human skeletal point sequence used as the basis for analysis.

[0090] It should be noted that when processing the first frame, since there is no historical data to refer to, the data correction for the first frame preferably assumes that the distance value in the corresponding first distance trajectory time series is correct. Therefore, the first value of the first distance trajectory time series is directly inserted into the first position of the second distance trajectory time series during initialization. From the second frame onwards, the above method can be used to perform frame-by-frame data correction processing on the monocular video to resolve the potential ankle confusion issue.

[0091] In other possible implementations, data correction of the human skeletal point sequence still includes processing key points of occluded limbs. Specifically, after obtaining the human skeletal point sequence, the confidence score 'c' corresponding to the sequence is determined. If the confidence score of a certain skeletal point is lower than a preset threshold, its coordinate data is marked as invalid, thus treating it as an unreliable data point. This allows for the identification and ignoring of these unreliable data points in subsequent abnormal gait evaluation analysis, avoiding interference from noise data on overall feature analysis and improving the accuracy of the evaluation results. Marking the coordinate data of a skeletal point as invalid can be achieved by replacing the coordinate data with a special label, such as setting the coordinates to 0 or NaN and the confidence score to 0. By setting the coordinate data to invalid values, the placeholders for the corresponding data points are retained, ensuring that the data tensors input to the spatiotemporal graph convolutional network model are always regular and of a fixed size. This eliminates noise interference and maintains the temporal and structural integrity of the data, facilitating batch processing and efficient computation by the model.

[0092] Existing monocular video methods often suffer from performance issues due to inaccurate 2D skeleton extraction (especially under occlusion or specific postures, such as leg crossing during lateral walking). However, the embodiments of this invention significantly improve the quality of skeleton data input to the spatiotemporal graph convolutional network model through a specially designed skeleton data correction method (especially for handling left and right leg confusion and arm occlusion). This improves the effectiveness of subsequent spatiotemporal feature learning and the accuracy and robustness of final anomaly detection.

[0093] As a preferred implementation, the spatiotemporal graph convolutional network model used in operation S3 employs a Siamese network structure. This network structure is designed as a symmetrical dual-branch architecture, comprising two structurally identical sub-networks that share all network parameters (including weights and biases) during training. These sub-networks are spatiotemporal graph convolutional networks (ST-GCN). This strict parameter-sharing mechanism ensures that the two sub-networks act like two fully calibrated rulers, capable of measuring and encoding data from different perspectives with completely consistent standards, which is a prerequisite for effective comparison. Figure 6The diagram schematically illustrates the architectural principle block diagram of a spatiotemporal graph convolutional network model according to one embodiment of the present invention, as follows: Figure 6 As shown, the spatiotemporal graph convolutional network model comprises two structurally identical sub-networks, B1 and B2. Within each ST-GCN sub-network, data flows through multiple stacked spatiotemporal convolutional modules. Each spatiotemporal convolutional module contains two core operations: first, spatial graph convolution, which aggregates information from adjacent joints along the edges of the graph to learn the spatial coordination relationships between different parts of the body within a frame (e.g., the coordinated movement of the shoulder and hip during walking); second, temporal convolution, which performs a convolution operation on the motion trajectory of each joint in the temporal dimension to learn the dynamic change pattern of that joint (e.g., the flexion and extension rhythm of the knee joint). Through the stacking of multiple modules, the sub-networks can progressively abstract and extract high-dimensional spatiotemporal features that characterize the essence of gait from the original coordinate data. After processing their respective input sequences, each sub-network finally outputs a high-dimensional feature vector. This vector can be understood as a deep encoding embedding of one side of the gait, which, in a compact form, contains the complex spatial pose and temporal dynamic information of that side of the gait. In this embodiment of the invention, these two vectors are denoted as the left-look feature vector and the right-look feature vector, respectively. Therefore, the Siamese ST-GCN architecture of this embodiment can achieve gait asymmetry analysis. That is, the network can quantify gait asymmetry by calculating and analyzing the difference between the output feature vectors of the two branches, and thereby detect and evaluate abnormal gaits, improving the robustness and accuracy of the detection results.

[0094] Furthermore, the spatiotemporal graph convolutional network model used also includes a contrast and fusion module, which is the core of realizing information interaction and decision-making under dual-view conditions. It receives left-view and right-view feature vectors from two twin branch sub-networks and performs feature comparison and fusion to generate a fused feature vector. Specifically, to integrate gait information from both sides, this module can use feature concatenation to generate a fused feature vector, directly concatenating the left-view and right-view feature vectors from the two twin branch sub-networks into a longer vector. This method preserves all the original feature information from both sides, allowing subsequent network layers to learn their inherent relationships. Thus, the contrast and fusion module can generate a fused feature vector containing dual-view contrast information from the left-view and right-view feature vectors from the two twin branch sub-networks.

[0095] More preferably, the spatiotemporal graph convolutional network model used also includes a classifier consisting of one or more fully connected layers (MLP, multilayer perceptron). This classifier receives a fused feature vector from the input of the contrast and fusion module. The fully connected layers in the classifier are responsible for further nonlinear transformation and information extraction of the fused feature vector, and finally mapping it to a predefined output category. The output category may include, for example, whether the gait is normal, abnormal, or what degree of abnormality the gait is.

[0096] More preferably, the last layer of the spatiotemporal graph convolutional network model can be designed as a softmax layer (activation function layer), which is configured to output multi-classification results, such as the probability distribution of a subject's gait belonging to "normal" or different "abnormal" levels, or the corresponding UPDRS scores of 0, 1, 2, etc., thereby completing the final judgment and evaluation. In other embodiments, the last layer can also be a sigmoid layer (for binary classification, i.e., normal / abnormal), or a regression layer (for outputting continuous anomalousness scores), etc., which can be adjusted according to the expected model output results. This embodiment of the invention does not impose any limitations on this.

[0097] During training, an end-to-end training mode can be used to directly train the entire spatiotemporal graph convolutional network model with the above structure. For example, the spatiotemporal graph convolutional network model, including two ST-GCN branches with shared weights, a contrast and fusion module, and a classifier, can be treated as a unified whole for end-to-end training. The training data requires both corrected human skeleton point sequences from the left and right perspectives. In this embodiment, the corrected left and right perspective human skeleton point sequences are used as independent inputs to the two ST-GCN sub-networks of the spatiotemporal graph convolutional network model. Each perspective's human skeleton point sequence is constructed into a spatiotemporal graph upon input. The nodes of the graph correspond to the various skeletal key points of the human body, and the edges are defined based on the inherent skeletal connections of the human body (e.g., the femur connecting the knee and hip) and the connections of the same key point in consecutive time frames. This construction process can refer to existing technologies, and this embodiment will not elaborate further. In addition, the training data includes ground truth labels corresponding to the prepared modified human skeleton point sequences from the left and right perspectives. These ground truth labels are formed based on the actual gait data collected from the subjects. During training, a backpropagation algorithm is used to synchronously update all learnable parameters in the network based on the difference between the final prediction result and the ground truth labels, thereby obtaining a trained spatiotemporal graph convolutional network model. This training method enables the network model to automatically learn how to extract the most discriminative asymmetric features from the original dual-view skeleton data and establish an accurate mapping relationship between them and specific real-world results (such as whether it is abnormal or not, and its severity). The resulting trained spatiotemporal graph convolutional network model uses the modified left and right perspective skeleton sequences as independent inputs to two sub-networks, and outputs the probability distribution of the subject's gait being normal or at different abnormal levels. In practical applications, only the human skeleton point sequences from the left and right perspectives need to be prepared and input into the trained network model to obtain the output result of the probability distribution of the subject's gait being normal or at different abnormal levels. This output result can be directly used as the evaluation result. Therefore, it is easy to understand that in the evaluation method of this invention, the monocular video can include both a monocular video from the left-side view and a monocular video from the right-side view. Correspondingly, the extracted and corrected human skeletal point sequences also include both left-side and right-side skeletal sequences. It should be noted that, to ensure the validity of the evaluation results, the input data received by the network model are the left-side and right-side skeletal sequences corresponding to the left-side and right-side walking unidirectional videos of the same subject.

[0098] In other possible implementations, the output or evaluation result can also be a direct result indicating whether the subject's gait is normal or at different abnormal levels. In this case, the subject's gait can be determined directly based on the probability distribution of whether the subject's gait is normal or at different abnormal levels (e.g., selecting the one with the higher probability as the direct result). In other possible implementations, the model's output or evaluation result can also be the category, probability, or severity score of the subject's gait abnormality, etc., and this embodiment of the invention does not impose any limitations on this.

[0099] In some embodiments, the evaluation method described above in this invention can be implemented on mobile terminal devices such as smartphones; in other embodiments, the evaluation method described above in this invention can also be implemented on a cloud server; in preferred embodiments, the evaluation method described above in this invention can be implemented partly on a server and partly on a mobile terminal device. For example, in some embodiments, the acquisition of lateral video and video segmentation to generate monocular video can be implemented on the mobile terminal device, and then the human skeleton point sequence of the monocular video is transmitted to the cloud server for subsequent processing such as data correction and evaluation result formation. For example, in yet another embodiment, only the aforementioned operations S3 and S4 can be implemented on the cloud server, while the acquisition and data correction of human skeleton point sequences, such as those from monocular video, are implemented on the mobile terminal device, and only the corrected human skeleton point sequence and necessary metadata are uploaded to the cloud server for evaluation. Preferably, if the mobile terminal's computing power is limited or to ensure the highest accuracy, in embodiments where video segmentation and skeleton point sequence extraction are implemented by the cloud server, the originally acquired lateral video data can be securely uploaded to the cloud server after compression and encryption. Ideally, personal identification information of participants should be separated or anonymized as much as possible before data collection and uploading. For example, the system assigns a unique research ID that is not directly related to the participant's real identity to map the participant's identity information to the participant's video and assessment results. Furthermore, when the raw lateral video data needs to be uploaded, the facial areas in the video can be blurred or masked in real time on the mobile device to prevent identity recognition, retaining only body posture information for gait analysis. By performing preliminary processing on the mobile device, anonymizing / pseudo-identifying the data, and encrypting transmission and storage, participant privacy can be protected to the maximum extent, enhancing data security and privacy protection.

[0100] The abnormal gait evaluation method provided in this invention improves the system's performance in complex environments through an improved skeletal keypoint extraction and correction strategy, as well as an optimized spatiotemporal feature learning network, effectively enhancing the accuracy and robustness of monocular video abnormal gait detection. Furthermore, the solution in this invention utilizes a common monocular camera (such as a smartphone) for effective gait analysis, achieving low-cost and convenient gait anomaly detection, facilitating widespread application. In addition, the method in this invention employs an advanced deep learning model (such as the Spatiotemporal Graph Convolutional Network ST-GCN) to automatically learn discriminative features in gait sequences, enhancing the capture and understanding of spatiotemporal dynamic features of gait and improving the ability to identify subtle and specific pattern gait anomalies.

[0101] Figure 7 The diagram schematically illustrates the architecture of an abnormal gait assessment system based on monocular video according to one embodiment of the present invention, such as... Figure 7 As shown, the system includes a mobile device terminal 600 and a cloud server 700, wherein,

[0102] The mobile device is used to capture lateral video of the subject walking using a monocular camera and send it to the cloud server.

[0103] The cloud server includes

[0104] The skeleton point extraction module 701 is used to extract the human skeleton point sequence from the lateral video;

[0105] The video segmentation module 702 is used to perform video segmentation processing on the lateral video based on the key points of the human skeleton, and generate a sequence of human skeleton points in the monocular video.

[0106] Data correction module 703 is used to correct the human skeleton point sequence of the generated monocular video; and

[0107] The gait assessment module 704 is used to input the corrected human skeletal point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output results of the spatiotemporal graph convolutional network model, wherein the output results include the assessment results of the gait abnormalities of the subject in the monocular video.

[0108] Figure 8 The diagram schematically illustrates the architecture of an abnormal gait assessment system based on monocular video according to one embodiment of the present invention, such as... Figure 8 As shown, the system includes a mobile device terminal 600 and a cloud server 700, wherein,

[0109] Mobile devices include 600

[0110] A monocular camera 601 used to collect lateral video of the subject walking;

[0111] The skeleton data extraction module 602 is used to perform video segmentation processing on the lateral video to generate the human skeleton point sequence of the monocular video, and send the generated human skeleton point sequence of the monocular video to the cloud server.

[0112] The cloud server includes

[0113] The data correction module 703 is used to correct the acquired human skeletal point sequence.

[0114] The cloud server 704 is used to input the corrected human skeletal point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output results of the spatiotemporal graph convolutional network model. The output results include an assessment of gait abnormalities in the subject within the monocular video. This system allows for skeleton extraction on mobile devices, reducing the need for transmitting raw video data, and securely uploads the processed skeleton data to the cloud for further in-depth analysis and storage, achieving optimized resource allocation.

[0115] In other possible implementations, the mobile device can also be configured to include a function selection module for module selection configuration. This allows users to flexibly choose whether to perform skeleton extraction, video segmentation, data correction, and other processing on the mobile device or a cloud server, based on the mobile device's computing power and user needs. Extracting the skeleton on the mobile device can significantly reduce the amount of data uploaded, save bandwidth and cloud storage costs, and further protect the privacy of the original video. The cloud, on the other hand, handles more complex, computationally intensive tasks, such as ST-GCN feature learning and model inference, to achieve optimal resource utilization.

[0116] It should be noted that the specific implementation process of each module in this abnormal gait evaluation system can be referred to the description in the method section above, and will not be repeated here.

[0117] Figure 9 A computing device according to an embodiment of the present invention is schematically shown in the figure, which is implemented as including:

[0118] Memory 81 is used to store executable instructions; and

[0119] The processor 82 is communicatively connected to the memory 81 and is used to execute executable instructions stored in the memory. When executed by the processor, the executable instructions implement the steps of the abnormal gait evaluation method based on monocular video described in any of the foregoing embodiments.

[0120] In some embodiments, the present invention provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform the abnormal gait evaluation method based on monocular video according to any of the above embodiments of the present invention.

[0121] In some embodiments, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the abnormal gait evaluation method based on monocular video of any of the above embodiments.

[0122] In some embodiments, the present invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the monocular video-based abnormal gait evaluation method of any of the above embodiments.

[0123] In some embodiments, the present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the abnormal gait evaluation method based on monocular video of any of the above embodiments.

[0124] Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. The aforementioned computing device can be implemented using the structure shown in the diagram, as follows: Figure 10 As shown, the electronic device includes:

[0125] One or more processors 610 and memory 620, Figure 10 Take the 610 processor as an example.

[0126] The electronic device may also include an input device 630 and an output device 640.

[0127] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0128] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the monocular video-based abnormal gait evaluation method in this embodiment of the invention. The processor 610 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the monocular video-based abnormal gait evaluation method described in the above embodiment.

[0129] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the monocular video-based abnormal gait evaluation method, etc. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0130] Input device 630 can receive input digital or character information and generate signals related to user settings and function control of the image processing device. Output device 640 may include a display device such as a display screen.

[0131] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they execute the abnormal gait evaluation method based on monocular video in any of the above method embodiments.

[0132] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0133] The electronic devices of this invention exist in various forms, including but not limited to:

[0134] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0135] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0136] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0137] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0138] (5) Other electronic devices with data interaction functions.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An abnormal gait assessment method based on monocular video, characterized in that, include: Obtain the human skeleton point sequence from the monocular video; The acquired human skeletal point sequence is then corrected. The corrected human skeleton point sequence is input into a pre-trained spatiotemporal graph convolutional network model, and the output of the spatiotemporal graph convolutional network model is obtained. The output includes the evaluation results of the gait abnormalities of the subject in the monocular video. The data correction of the acquired human skeletal point sequence includes: The first distance trajectory time series is generated based on the human skeleton point sequences of the left ankle and the right ankle from the human skeleton point sequence in the monocular video. Initialize the second distance trajectory time series; Based on the first distance trajectory time series, the second distance trajectory time series, and the pre-constructed motion trend model, the human skeleton point sequence of the monocular video is processed frame by frame for data correction. The video frame in the monocular video to be processed for data correction is designated as the current frame. The data correction process for the current frame based on the first distance trajectory time series, the second distance trajectory time series, and the pre-built motion trend model includes: Based on the motion trend model, the displacement change caused by connecting the first distance trajectory time series to the second distance trajectory time series under the first and second hypothetical scenarios, respectively, is calculated. Based on the displacement change, the trajectory smoothness score of the current frame under the first and second hypothetical scenarios is determined. Based on the trajectory smoothness scores of the current frame under the first and second hypothetical scenarios, it is determined whether the left and right ankles of the current frame are confused. If confusion is determined, the coordinate data of the left and right ankles are swapped to perform data correction on the human skeletal point sequence corresponding to the current frame.

2. The method according to claim 1, characterized in that, Obtaining the human skeleton point sequence from the monocular video includes: Extract key points of the human skeleton from lateral videos of subjects walking captured by a monocular camera; The lateral video is segmented based on the extracted key points of the human skeleton. The human skeleton point sequence of the monocular video is generated based on the video segmentation processing results.

3. The method according to claim 2, characterized in that, The step of segmenting the lateral video based on the extracted human skeletal key points includes: Motion anchor points are determined based on the extracted key points of the human skeleton; A time series of instantaneous horizontal velocity is generated based on the motion anchor points and the lateral video. The effective walking range and potential turning points in the lateral video are identified based on the instantaneous horizontal velocity time series. The lateral video is segmented based on the effective walking range and potential turning points to generate at least one unidirectional walking segment; The monocular video used for abnormal gait assessment is determined based on at least one generated unidirectional walking segment.

4. The method according to claim 1, characterized in that, The spatiotemporal graph convolutional network model adopts a twin network structure, which includes two sub-networks that are structurally identical and share all network parameters during training. The sub-networks are spatiotemporal graph convolutional networks. The human skeleton point sequence includes a left-side view skeleton sequence and a right-side view skeleton sequence. The pre-trained spatiotemporal graph convolutional network model uses the corrected left-side view skeleton sequence and right-side view skeleton sequence as independent inputs to two sub-networks, and uses the probability distribution of the subject's gait belonging to normal or different abnormal levels as the output result.

5. An abnormal gait evaluation method based on monocular video, for mobile devices, characterized in that, include: Obtain the human skeleton point sequence from the monocular video; The acquired human skeletal point sequence is then corrected. The corrected human skeleton point sequence is sent to the server, and the evaluation result of the gait abnormality of the subject in the monocular video is obtained from the server. The server is used to input the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model, and feed back the evaluation result to the mobile device according to the output result of the spatiotemporal graph convolutional network model. The data correction of the acquired human skeletal point sequence includes: The first distance trajectory time series is generated based on the human skeleton point sequences of the left ankle and the right ankle from the human skeleton point sequence in the monocular video. Initialize the second distance trajectory time series; Based on the first distance trajectory time series, the second distance trajectory time series, and the pre-constructed motion trend model, the human skeleton point sequence of the monocular video is processed frame by frame for data correction. The video frame in the monocular video to be processed for data correction is designated as the current frame. The data correction process for the current frame based on the first distance trajectory time series, the second distance trajectory time series, and the pre-built motion trend model includes: Based on the motion trend model, the displacement change caused by connecting the first distance trajectory time series to the second distance trajectory time series under the first and second hypothetical scenarios, respectively, is calculated. Based on the displacement change, the trajectory smoothness score of the current frame under the first and second hypothetical scenarios is determined. Based on the trajectory smoothness scores of the current frame under the first and second hypothetical scenarios, it is determined whether the left and right ankles of the current frame are confused. If confusion is determined, the coordinate data of the left and right ankles are swapped to perform data correction on the human skeletal point sequence corresponding to the current frame.

6. An abnormal gait assessment system based on monocular video, characterized in that, This includes mobile devices and cloud servers, among which, The mobile device is used to capture lateral video of the subject walking using a monocular camera and send it to the cloud server. The cloud server includes The skeleton point extraction module is used to extract key points of the human skeleton from the lateral video. The video segmentation module is used to segment the lateral video based on the key points of the human skeleton and generate a sequence of human skeleton points for monocular video. The data correction module is used to correct the generated human skeletal point sequence; and A gait assessment module is used to input the corrected human skeletal point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output results of the spatiotemporal graph convolutional network model, wherein the output results include the assessment results of gait abnormalities of the subject in the monocular video; This includes data correction of the generated human skeletal point sequence, including: The first distance trajectory time series is generated based on the human skeleton point sequences of the left ankle and the right ankle from the human skeleton point sequence in the monocular video. Initialize the second distance trajectory time series; Based on the first distance trajectory time series, the second distance trajectory time series, and the pre-constructed motion trend model, the human skeleton point sequence of the monocular video is processed frame by frame for data correction. The video frame in the monocular video to be processed for data correction is designated as the current frame. The data correction process for the current frame based on the first distance trajectory time series, the second distance trajectory time series, and the pre-built motion trend model includes: Based on the motion trend model, the displacement change caused by connecting the first distance trajectory time series to the second distance trajectory time series under the first and second hypothetical scenarios, respectively, is calculated. Based on the displacement change, the trajectory smoothness score of the current frame under the first and second hypothetical scenarios is determined. Based on the trajectory smoothness scores of the current frame under the first and second hypothetical scenarios, it is determined whether the left and right ankles of the current frame are confused. If confusion is determined, the coordinate data of the left and right ankles are swapped to perform data correction on the human skeletal point sequence corresponding to the current frame.

7. An abnormal gait assessment system based on monocular video, characterized in that, This includes mobile devices and cloud servers, among which, Mobile devices include A monocular camera used to capture lateral video of subjects walking; The skeleton data extraction module is used to perform video segmentation processing on the lateral video to generate the human skeleton point sequence of the monocular video, and send the generated human skeleton point sequence of the monocular video to the cloud server. The cloud server includes The data correction module is used to correct the acquired human skeletal point sequence. A gait assessment module is used to input the corrected human skeletal point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output results of the spatiotemporal graph convolutional network model, wherein the output results include the assessment results of gait abnormalities of the subject in the monocular video; This includes data correction of the acquired human skeletal point sequence, including: The first distance trajectory time series is generated based on the human skeleton point sequences of the left ankle and the right ankle from the human skeleton point sequence in the monocular video. Initialize the second distance trajectory time series; Based on the first distance trajectory time series, the second distance trajectory time series, and the pre-constructed motion trend model, the human skeleton point sequence of the monocular video is processed frame by frame for data correction. The video frame in the monocular video to be processed for data correction is designated as the current frame. The data correction process for the current frame based on the first distance trajectory time series, the second distance trajectory time series, and the pre-built motion trend model includes: Based on the motion trend model, the displacement change caused by connecting the first distance trajectory time series to the second distance trajectory time series under the first and second hypothetical scenarios, respectively, is calculated. Based on the displacement change, the trajectory smoothness score of the current frame under the first and second hypothetical scenarios is determined. Based on the trajectory smoothness scores of the current frame under the first and second hypothetical scenarios, it is determined whether the left and right ankles of the current frame are confused. If confusion is determined, the coordinate data of the left and right ankles are swapped to perform data correction on the human skeletal point sequence corresponding to the current frame.

8. A computing device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 4.

9. A storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 4.

Images