Monocular video-based abnormal gait assessment method and system, and computing device

Abstract

The invention discloses an abnormal gait assessment method and system based on a monocular video, and a computing device. The method comprises the following steps: acquiring a human skeleton point sequence of the monocular video; performing data correction on the obtained human skeleton point sequence; and inputting the corrected human skeleton point sequence into a pre-trained space-time diagram convolutional network model, and obtaining an output result of the space-time diagram convolutional network model, the output result including an evaluation result of the gait anomaly condition of the subject in the monocular video. The method can effectively solve the problems of left and right leg confusion, easy shielding of key points and the like of skeleton point extraction in the monocular video, can enhance capture and understanding of gait space-time dynamic features, realizes accurate extraction of human gait space-time features, improves the accuracy and robustness of monocular video gait analysis, and improves the accuracy and robustness of the monocular video gait analysis. Effective gait analysis can be carried out by using a common monocular camera (such as a smart phone), the cost is lower, and the application scene is wider.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an abnormal gait evaluation method and system based on monocular video, a computing device and a storage medium. BACKGROUND

[0002] Gait is a basic and complex motor function of human beings, and changes in its mode are often an early indicator or important manifestation of various health conditions, such as Parkinson's disease (PD), post-stroke sequelae, osteoarthritis, and aging. In recent years, with the development of computer vision and artificial intelligence technology, gait analysis using video data has become a research hotspot.

[0003] At present, video-based gait analysis methods include 3D skeleton key point extraction analysis methods that rely on multi-camera systems, special marker points or depth sensors to obtain three-dimensional information, and 2D skeleton key point extraction analysis methods based on monocular video. The former has the defects of complex operation, high equipment cost and high deployment difficulty due to the reliance on multi-camera arrays, depth cameras or special motion capture systems. The latter is difficult to accurately capture the three-dimensional spatial motion characteristics of the whole body key points when using monocular RGB video, and the accuracy and stability of skeleton key point extraction are challenged in complex scenes such as occlusion, light changes and clothing influences, especially for the accurate capture of limb ends (such as wrists and ankles). For example, when the left and right legs are alternately viewed from the side, the existing skeleton extraction model may misidentify the leg keypoints, resulting in incorrect extraction of gait parameters, which can seriously affect the accuracy of subsequent gait feature analysis and abnormality detection. SUMMARY

[0004] The embodiments of the present application provide an abnormal gait evaluation method and system based on monocular video to solve the problems of low precision, insufficient robustness, incomplete feature extraction and possible confusion of key point extraction when using conventional monocular cameras (such as smartphone cameras) for gait abnormality detection in the prior art.

[0005] According to a first aspect of the present application, an abnormal gait evaluation method based on monocular video is provided, which comprises: obtaining a human skeleton point sequence of the monocular video; performing data correction on the obtained human skeleton point sequence; inputting the corrected human skeleton point sequence into a pre-trained spatio-temporal graph convolution network model, and obtaining an output result of the spatio-temporal graph convolution network model, wherein the output result comprises an evaluation result of the gait abnormality of a subject in the monocular video.

[0006] According to a second aspect of the present application, a monocular video-based abnormal gait evaluation method is provided for a mobile device end, which comprises: obtaining a human skeleton point sequence of the monocular video; performing data correction on the obtained human skeleton point sequence; sending the corrected human skeleton point sequence to a server, and obtaining an evaluation result of an abnormal gait condition of a subject in the monocular video from the server, wherein the server is configured to input the corrected human skeleton point sequence into a pre-trained spatio-temporal graph convolution network model, and feed back the evaluation result to the mobile device end according to an output result of the spatio-temporal graph convolution network model.

[0007] According to a third aspect of the present application, a monocular video-based abnormal gait evaluation system is provided, which comprises a mobile device end and a cloud server, wherein, the mobile device end is configured to send a lateral video of a subject walking to the cloud server by a monocular camera; the cloud server comprises a skeleton point extraction module configured to extract human skeleton key points from the lateral video; a video segmentation module configured to perform video segmentation processing on the lateral video according to the human skeleton key points to generate a human skeleton point sequence of the monocular video; a data correction module configured to perform data correction on the generated human skeleton point sequence; and a gait evaluation module configured to input the corrected human skeleton point sequence into a pre-trained spatio-temporal graph convolution network model, and obtain an output result of the spatio-temporal graph convolution network model, wherein the output result comprises an evaluation result of an abnormal gait condition of a subject in the monocular video.

[0008] According to a fourth aspect of the present application, a monocular video-based abnormal gait evaluation system is provided, which comprises a mobile device end and a cloud server, wherein, the mobile device end comprises a monocular camera configured to collect a lateral video of a subject walking; a skeleton data extraction module configured to perform video segmentation processing on the lateral video to generate a human skeleton point sequence of the monocular video, and send the generated human skeleton point sequence of the monocular video to the cloud server; the cloud server comprises a data correction module configured to perform data correction on the obtained human skeleton point sequence; The gait evaluation module is configured to input the corrected human body bone point sequence into a pre-trained spatio-temporal graph convolution network model, and obtain an output result of the spatio-temporal graph convolution network model, wherein the output result comprises an evaluation result of a gait abnormality of the subject in the monocular video.

[0009] According to a fifth aspect of the present application, a computing device is provided, comprising at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method of the first aspect.

[0010] According to a sixth aspect of the present application, a storage medium is provided, which stores a computer program executable by a processor to implement the steps of the method of the first aspect.

[0011] The abnormal gait evaluation method of the embodiments of the present application can effectively solve the unique problems of bone point extraction in monocular videos (such as left-right leg confusion, limited view angle, and key points being easily blocked, etc.) by correcting the extracted human body bone point sequence. Meanwhile, the method of the embodiments of the present application inputs the corrected bone point sequence into a trained spatio-temporal graph convolution network (ST-GCN) to automatically learn the spatio-temporal features of gait, enhances the capture and understanding of the spatio-temporal dynamic features of gait, thereby effectively overcoming the limitation of the inherent lack of depth information of monocular video information, enabling accurate extraction of the spatio-temporal features of human gait, improving the accuracy and robustness of monocular video gait analysis, making it possible to use ordinary monocular cameras (such as smart phones) to effectively analyze gait, with lower cost and more extensive application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0013] Figure 1 A flowchart of a monocular video-based abnormal gait evaluation method according to an embodiment of the present application is schematically shown; Figure 2 A flowchart of an implementation method of operation S1 according to an embodiment of the present application is schematically shown; Figure 3 A flowchart of an implementation method of video segmentation processing according to an embodiment of the present application is schematically shown; Figure 4A flow chart schematically showing a method of implementing operation S2 of an embodiment of the present application is shown; Figure 5 A flow chart schematically showing a method of implementing operation S4 of an embodiment of the present application is shown; Figure 6 A schematic architecture diagram of a spatio-temporal graph convolution network model of an embodiment of the present application is shown; Figure 7 A schematic architecture diagram of a monocular video-based abnormal gait assessment system of an embodiment of the present application is shown; Figure 8 A schematic architecture diagram of a monocular video-based abnormal gait assessment system of another embodiment of the present application is shown; Figure 9 A schematic architecture diagram of a computing device of an embodiment of the present application is shown; Figure 10 A schematic architecture diagram of a computing device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0014] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0015] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0016] In the description of the present application, it should be understood that, if the terms “center”, “middle”, “longitudinal”, “transverse”, “length”, “width”, “thickness”, “upper”, “lower”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, “outer”, “axial”, “radial”, “circumferential” and the like indicate the orientation or positional relationship shown in the drawings, they are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. The features defined as “first” and “second” are used to distinguish feature names, and do not have special meanings, and in addition, the features defined as “first” and “second” can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of “a plurality of” is two or more.

[0017] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integrally connected; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0018] It should also be noted that in this paper, the terms "including", "containing", not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles or devices. Without more limitations, the elements defined by the statement "including" do not exclude the presence of other identical elements in the process, method, article or device including the described elements. The terms used in this paper are generally the terms commonly used by those skilled in the art, and if they are inconsistent with commonly used terms, the terms in this paper shall prevail.

[0019] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely below with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0020] The present application will be described in further detail below with reference to the accompanying drawings.

[0021] The abnormal gait evaluation method in the embodiments of the present application can be applied in any mobile terminal device, and can also be applied on a server, so that these mobile terminal devices or servers can use the scheme of the embodiments of the present application to monitor gait abnormalities, and only using a monocular camera device can complete gait data acquisition, taking into account precision, robustness and device cost, and the application scenarios are widely used.

[0022] Figure 1 A flowchart of an abnormal gait evaluation method of an embodiment of the present application is schematically shown, the execution subject of the method can be, for example, a processor of a computer, a cloud server, a server in a cluster, a mobile terminal device, etc., and the embodiments of the present application do not limit this. As shown in Figure 1 It includes: Operation S1, acquiring a human skeleton point sequence of a monocular video; Operation S2, data correction is performed on the acquired human skeleton point sequence; In operation S3, the corrected human skeleton point sequence is input into the pre-trained spatio-temporal graph convolution network model, and an output result of the spatio-temporal graph convolution network model is obtained, wherein the output result includes an evaluation result of the gait abnormality of the subject in the monocular video.

[0023] The monocular video in the embodiment of the application can be obtained by video segmentation processing on a lateral video of the subject walking collected by a monocular camera, or can be a lateral video of the subject walking directly collected by a monocular camera device. When collecting the lateral video, an ordinary monocular RGB camera (such as a camera built in a smart phone camera) can be used to shoot the walking video of the subject from the side. Preferably, the subject can be asked to complete several round trips to obtain sufficient gait cycle data. More preferably, the video frame rate and resolution of the monocular camera used need to meet the requirements of the subsequent human skeleton key point extraction algorithm, for example, the RGB frame resolution can be 1920x1080, and the frame rate is 30 fps. In other embodiments, the existing mobile application (APP) on the mobile terminal device can also be used to correctly complete the video collection under the guidance of the APP, and the embodiment of the application does not limit this.

[0024] When the monocular video is a lateral video of the subject walking directly collected by a monocular camera device, in operation S1, the human skeleton point sequence of the monocular video can be obtained by using a lightweight or efficient 2D human pose estimation algorithm (such as an optimized version of DWPose, MediaPipe Pose, etc.) to extract the 2D image coordinates (x, y) and the corresponding confidence c of 20 or more human skeleton key points (such as head, neck, shoulder, elbow, wrist, hip, knee, ankle, and toe) of the human body frame by frame to form a human skeleton point sequence of each human pose. The human skeleton point sequence refers to a group of time series data representing continuous body movements formed by concatenating the 2D image coordinates (x, y) of the human skeleton key points extracted frame by frame in the time order of the video frames. In some embodiments, the confidence c corresponding to the 2D image coordinates can be further included.

[0025] In order to accurately extract a pure one-way walking segment for analysis from the original collected video which can contain preparation, turning, and ending, etc. non-walking stages, to improve the quality and accuracy of the extracted human skeleton point sequence, as a preferred embodiment, the monocular video in the embodiment of the application is obtained by video segmentation processing on a lateral video of the subject walking collected by a monocular camera. In this case, operation S1 can be implemented as the method flow shown in Figure 2 As shown in the method flow. Operation S11, human skeleton key point extraction is performed on the lateral video of the subject walking collected by the monocular camera. Operation S12, video segmentation processing is performed on the lateral video according to the extracted human skeleton key points. Operation S13, generating a human skeleton point sequence of the monocular video according to the video segmentation processing result.

[0026] In operation S11, the human skeleton key points include a set of coordinate points corresponding to each single-frame image in the lateral video, that is, a set of coordinate points of main joint positions of the human body identified on the single-frame image, such as the coordinate points of joint positions of the head, neck, shoulder, elbow, hip, knee, ankle, etc. The set of coordinate points has a one-to-one correspondence with each single-frame image. Among them, the human skeleton key points corresponding to each frame image are extracted directly from each frame image of the lateral video by a 2D human pose estimation algorithm. For example, a lightweight or efficient 2D human pose estimation algorithm such as DWPose or MediaPipe Pose can be used to analyze each frame of the lateral video based on a deep learning key point detection model, so as to automatically identify and locate a series of key joint points (including twenty or more key points such as shoulders, knees, elbows, etc.) of the human body in each frame, and output the pixel coordinates (x, y) and the corresponding confidence c of each identified key point (such as left shoulder and right shoulder) in the two-dimensional image, thereby obtaining the human skeleton key points of the corresponding frame image.

[0027] Among them, Figure 3 The video segmentation processing method flow of operation S12 of an embodiment is schematically shown as Figure 3 As shown, it can be implemented to include: Operation S121, obtaining human skeleton key points and determining motion anchor points according to the obtained human skeleton key points; Operation S122, generating a time sequence of instantaneous horizontal velocity according to the motion anchor points and the lateral video; Operation S123, identifying and locating an effective walking interval and a potential turning point in the lateral video according to the time sequence of instantaneous horizontal velocity; Operation S124, cutting the lateral video according to the effective walking interval and the potential turning point and generating at least one unidirectional walking segment; Operation S125, determining the monocular video for abnormal gait assessment according to the generated at least one unidirectional walking segment.

[0028] As a possible implementation, in operation S121, one or a group of points on the torso that are relatively most stable during walking can be selected from the human body skeleton key points corresponding to each frame of image as the motion anchor point, such as using the "neck" key point as the motion anchor point, or calculating the center point of the left and right "shoulder" key points as the motion anchor point, that is, from the coordinate point set corresponding to each frame of image, a coordinate point is selected as the motion anchor point corresponding to the frame of image, which can be the coordinate of the "neck" key point, or the center point coordinate of the left and right shoulders. Wherein, the way to calculate the center point coordinate of the left and right shoulders can be to add the x coordinates of the left and right shoulders and divide by 2 to obtain the x coordinate of the center point; then add their y coordinates and divide by 2 to obtain the y coordinate of the center point. When walking, the limbs (arms and legs) of a person will swing greatly and periodically, while the torso of the human body is mainly responsible for the overall forward translation, and its motion trajectory is relatively smooth without high-frequency reciprocating motion, therefore, the present embodiment preferably identifies the points on the torso such as the "neck" or the center point of the left and right "shoulders" as "relatively most stable" points, and selects the relatively most stable points on the torso as the motion anchor point, which can effectively filter the high-frequency interference caused by limb swinging, and more accurately reflect and capture the overall translation motion of the human body center of gravity, which is helpful for accurate segmentation of the video.

[0029] In operation S122, the instantaneous horizontal velocity time series is a set of discrete numerical data (which can be implemented as a one-dimensional numerical list or array), where each number represents the horizontal moving speed and direction of the subject at a specific time (i.e. a frame). As a possible implementation, the instantaneous horizontal velocity time series can be generated by tracking the coordinate change of the motion anchor determined in operation S121 in the horizontal direction of each frame image of the lateral video. Since the motion anchor is one-to-one corresponding to each frame image, thus by calculating the coordinate displacement of the determined motion anchor between consecutive frames of the lateral video, an original instantaneous horizontal velocity time series representing the forward or backward movement of the body can be generated. Since in the lateral walking video (i.e. the lateral video), the forward or backward movement of the body is mainly reflected in the horizontal axis of the image, thus tracking the change of the x-coordinate is sufficient to reflect its core movement trend. Based on this, the calculation of the coordinate displacement of the determined motion anchor between consecutive frames of the lateral video can be specifically calculating the offset of the x-coordinate of the motion anchor between consecutive frames, thus the Nth number in the instantaneous horizontal velocity time series can be derived by calculating the difference between the x-coordinate of the motion anchor in the N+1th frame and the x-coordinate of the motion anchor in the Nth frame, and the size and positive or negative attribute of the difference can represent the moving speed and direction of the motion anchor, such as a positive value indicating that the motion anchor is moving in a certain direction (e.g. to the right), the larger the value, the faster the moving speed; a negative value indicates that the motion anchor is moving in the opposite direction (e.g. to the left), the larger the absolute value, the faster the moving speed; and a value close to zero indicates that the motion anchor is basically stationary, which usually occurs when the subject is walking very slowly or turning around. Exemplarily, assuming that for a certain lateral video, the x-coordinates of the motion anchor extracted by operation S121 in consecutive 5 frames are: [150, 155, 160, 159, 154], then in operation S122, by calculating the coordinate difference between consecutive frames, the generated instantaneous horizontal velocity time series is: [5, 5, -1, -5], and the generated sequence can indicate that the subject first moves to the right at a faster speed (speed of 5), then slows down and starts to turn (speed becomes -1), and finally returns to the left at a faster speed (speed of -5). Thus by analyzing the sign change and value size in this sequence, the effective walking interval and potential turning point can be located.

[0030] Since the generated original instantaneous horizontal velocity time sequence usually contains high-frequency noise generated by slight jitter of the key point detection model or slight shaking of the subject's body, in order to eliminate these disturbances, in some preferred embodiments, a low-pass filter (for example, a moving average filter or a Gaussian filter) can be further used to smooth the generated instantaneous horizontal velocity time sequence, so as to obtain a smooth velocity curve that can clearly reflect the motion trend of the subject. Wherein, the specific smoothing process can refer to the prior art of the related low-pass filter, and the embodiments of the present application do not make too much repetition. In addition, it should be noted that the instantaneous horizontal velocity time sequence in the embodiments of the present application focuses on describing the original, discrete, noisy data point set before processing. When the data point set is plotted on the chart with time as the horizontal axis and velocity as the vertical axis, it appears as a velocity curve. Therefore, the smooth velocity curve mentioned here mainly focuses on describing the state of the data in the sequence after being processed by the low-pass filter. Since the noise has been filtered out, the graphical expression (i.e. the curve) becomes smooth, and the macro trend of the subject's motion can be clearly reflected.

[0031] In operation S123, a motion activation speed threshold can be preset, where the threshold needs to be significantly higher than the speed noise generated by body shaking in a static state, and the core meaning of significantly higher here is that the preset speed threshold must be able to clearly distinguish between "intentional walking" and "unconscious body shaking", and in specific applications, the speed threshold can be set in combination with specific scene characteristics and prior experience, as long as it is ensured that only when the subject really starts to walk, the absolute value of its speed will stably exceed this threshold. Exemplarily, assuming that the maximum speed noise of the subject is measured in advance to be 0.5, a reasonable "motion activation speed threshold" can be 2.0 or 3.0. When performing identification, the instantaneous horizontal speed time sequence determined in operation S122 can be specifically traversed, and all data values in the sequence are compared with the preset motion activation speed threshold respectively, to screen out time periods in which the absolute value of the speed is continuously higher than the threshold, and these time periods are marked as valid walking intervals. Specifically, a new binary sequence (such as a binary sequence composed of 0 and 1) can be created in advance, and when comparing the absolute value of each speed value in the instantaneous horizontal speed time sequence with the preset motion activation speed threshold, if it is judged that the absolute value of the speed is greater than the threshold, the corresponding position of the binary sequence is recorded as 1 (representing motion), and if the absolute value of the speed is not greater than the threshold, it is recorded as 0 (representing static). For example, for the instantaneous horizontal speed time sequence [0.5, 1.5, 5.0, 6.2, 4.8, 1.2, 0.8], assuming that the motion activation speed threshold is 2.0, the generated binary sequence is [0, 0, 1, 1, 1, 0, 0]. Next, all time periods in which the absolute value of the speed is continuously higher than the threshold can be screened out by finding all continuous 1 segments in this binary sequence, which can be achieved by traversing once. This processing process can be implemented as follows: first, initialize an empty list active_periods to store the results, and set a variable start_frame = null; then, start traversing from the beginning of the binary sequence, when the first 1 is encountered, if start_frame is empty, record the sequence number of the current frame in start_frame, which marks the beginning of a valid walking interval, and continue to traverse forward, when the first 0 is encountered, if start_frame is not empty, it means that a continuous 1 segment has just ended, at this time, start_frame and the sequence number of the previous frame of the current frame (current frame-1) are stored as a complete time period (for example, a tuple containing the starting frame and the ending frame) in the active_periods list, and then start_frame is reset to empty; repeat this process until the entire binary sequence is traversed.By the above steps, all time periods during which the absolute value of the speed is continuously higher than the threshold value can be accurately identified, and the time periods are the effective walking intervals. By identifying the effective walking intervals, the stationary or preparation stages at the beginning and end of the lateral video can be effectively excluded, so as to reduce video noise and data processing amount, save resources, and improve data processing efficiency and performance. In addition, in operation S123, the embodiment of the application further finds the moments at which the speed curve corresponding to the instantaneous horizontal speed time sequence crosses zero in the effective walking intervals, and the moments correspond to potential turning points at which the subject changes the walking direction. As a possible implementation, the instantaneous horizontal speed time sequence corresponding to the determined effective walking interval can be traversed to determine whether the signs of the speed value V_t of the current frame and the speed value V_{t+1} of the next frame change, for example, by determining whether their product is negative (V_t * V_{t+1} < 0). When and only when the signs of V_t and V_{t+1} are different, it is determined that a zero-crossing event occurs, and once this condition is detected, the sequence number t of the current frame or the sequence number t+1 of the next frame is recorded as the moment of the potential turning point. For example, in an effective walking interval, the instantaneous horizontal speed sequence corresponding thereto is [5.1, 4.2, 1.5, -0.8, -3.2]. When the algorithm traverses to 1.5 and -0.8, it is found that 1.5 * (-0.8) < 0, that is, the sign changes. Therefore, this position will be marked as a potential turning point. The sequence number of the video frame, such as the sequence number of the current frame, refers to the frame number of the video, which is used to represent the specific time position of a frame of video in the lateral video, and the potential turning point determined based on the foregoing manner is represented by the frame number of the video, which represents the precise moment at which the subject changes the walking direction in the video. Preferably, the instantaneous horizontal speed time sequence used in operation S123 is the smoothed one.

[0032] In operation S124, the whole valid walking interval can be split into several sub- segments, each of which is recorded as a one-way walking segment, with the identified potential turning points as boundaries. For example, assume that a valid walking interval is first determined to be from the 100th frame to the 500th frame of a video through operation S123, then in operation S123, potential turning points in the instantaneous horizontal speed sequence of these 401 frames (from 100 to 500) are analyzed. Assume that the speed value changes from positive to negative between the 249th frame and the 250th frame, and changes from negative to positive between the 380th frame and the 381st frame, then the first potential turning point is located at 250 and the second potential turning point is located at 381. Thus, for this valid walking interval, the generated potential turning point data is a list containing two frame numbers: [250, 381]. In operation S124, the original video frame interval [100, 500] is split with the list [250, 381] as boundaries, thus forming the first one-way walking segment from the 100th frame to the 249th frame, the second one-way walking segment from the 250th frame to the 380th frame, and the third one-way walking segment from the 381st frame to the 500th frame. In this way, a round-trip walking side-view video containing two turns is accurately split into three directionally single walking sub-segments, i.e., divided into three one-way walking segments.

[0033] In operation S125, it can be specifically confirmed for each unidirectional walking segment whether the signs of the speed values in the corresponding instantaneous horizontal speed sequence remain consistent (i.e., all positive or all negative), and if the signs are consistent, the segment is confirmed as a valid unidirectional walking segment and is extracted for use as a monocular video for subsequent analysis. Specifically, for each unidirectional walking segment, the embodiment of the present application finds the corresponding part in the instantaneous horizontal speed sequence determined in operation S122, which is a subsequence of continuous speed values in the instantaneous horizontal speed sequence. In operation S125, the positive and negative attributes of each speed value in the subsequence are further checked. Only when the positive and negative attributes of all speed values in the subsequence are consistent, the unidirectional walking segment corresponding to the subsequence is considered to be a valid unidirectional video, i.e., the subject indeed walks in a single direction during the time period without any complex non-unidirectional motion such as turning back, hesitating, or pacing. It should be noted that in the analysis of lateral walking videos, the sign of the speed value directly reflects the horizontal movement direction of the subject in the image, and a positive speed value generally represents that the subject moves towards the positive direction of the x-axis of the image coordinate system, for example, from the left side of the image to the right side; and a negative speed value represents that the subject moves towards the negative direction of the x-axis, for example, from the right side of the image to the left side (the specific forward direction depends on the placement position of the camera). It should be noted that the further verification in operation S125 can improve the effectiveness and accuracy of the determined unidirectional walking segment. In other embodiments, the processing result of operation S124 can be directly used to extract a monocular video without the verification process, and the embodiment of the present application is not limited in this regard.

[0034] Therefore, the method of the embodiment of the present application can reliably automatically divide a lateral video of one round trip (e.g., walking towards the lens and then walking away from the lens, or walking from the left side of the lens to the right side) into two or more independent unidirectional gait samples, thereby accurately extracting a pure unidirectional walking segment (i.e., a monocular video as an analysis object of the present application) for analysis, improving the accuracy of subsequent analysis, and reducing the data processing amount of subsequent analysis. The video segmentation algorithm of the embodiment of the present application extracts unidirectional monocular videos by tracking the overall translation motion of the human body center of gravity rather than the specific swing of the limbs, which can effectively filter out high-frequency interference caused by limb swing, thereby obtaining a smooth and clean speed curve (represented by an instantaneous horizontal speed sequence), and thus the speed curve can be used to more accurately identify valid walking intervals and potential turning points, thereby achieving accurate segmentation of the video.

[0035] In operation S13, the human body skeleton key points corresponding to the segmented monocular video image frames are sorted according to the frame sequence numbers of the image frames in the monocular video to form a time-ordered data sequence, that is, the human body skeleton point sequence of the monocular video is obtained.

[0036] Due to the limitation of the monocular video, when walking sideways, the skeleton extraction model often confuses the key points of the left and right ankles due to the alternation of the two legs. Therefore, there may be a problem of confusion of the left and right leg key points in the 2D skeleton points extracted by the foregoing algorithm. In order to improve the data quality and avoid the adverse effects of the extracted 2D skeleton point data on the accuracy and robustness of the gait analysis results, the embodiment of the present application will continue to automatically correct the obtained human body skeleton point sequence in operation S2 after obtaining the human body skeleton point sequence of the monocular video, and through iterative adjustment, the smoothness and naturalness of the ankle motion trajectory are ensured, and then the gait evaluation using the skeleton point sequence is more reliable. Among them, Figure 4 The specific implementation method flow of operation S2 of an embodiment of the present application is schematically shown as Figure 4 As shown, the data correction of the obtained human body skeleton point sequence can be specifically implemented as including: Operation S201, generating a first distance trajectory time sequence according to the left ankle human body skeleton point sequence and the right ankle human body skeleton point sequence in the human body skeleton point sequence of the monocular video; Operation S202, initializing a second distance trajectory time sequence; Operation S203, performing data correction processing on the human body skeleton point sequence of the monocular video frame by frame according to the first distance trajectory time sequence, the second distance trajectory time sequence and the pre-constructed motion trend model.

[0037] In operation S201, the specific implementation process of generating the first distance trajectory time sequence can be to calculate the horizontal coordinate difference value of the left ankle human body skeleton point and the right ankle human body skeleton point corresponding to each video frame frame by frame on the monocular video corresponding left ankle human body skeleton point sequence and right ankle human body skeleton point sequence, and then form the first distance trajectory time sequence according to the frame sequence number of the video frame. The horizontal coordinate difference value can be the difference value of the x value of the 2D image coordinates (x, y) of the left ankle human body skeleton point sequence and the right ankle human body skeleton point sequence. For example, for a monocular video to be corrected, assuming that it has n frames, the x coordinate of the left ankle of the first frame is marked as x_left_1, and the x coordinate of the right ankle is marked as x_right_1. The horizontal coordinate difference value D_1 corresponding to the first frame video is D_1 = x_left_1 - x_right_1. Thus, repeating this calculation process, a series of horizontal coordinate difference values D_2, D_3,..., D_n corresponding to each video frame can be obtained. These horizontal coordinate difference values (also referred to as distance values in the embodiment of the present application) are combined into a list or array according to the video time sequence (such as according to the video frame sequence number). The difference value list arranged in the frame sequence forms the first distance trajectory time sequence. The first distance trajectory time sequence records the horizontal position relationship of the left ankle relative to the right ankle in each frame of the video, which can be regarded as the projection of the distance between the two feet in the lateral view on the two-dimensional plane. In the normal walking process, as the two legs alternately step forward and backward, the distance value will present a periodic and sinusoidal smooth change. Therefore, whether the value jumps abnormally due to the recognition error of the left and right feet can be judged according to the smoothness of the sequence, so as to correct the data of the left and right ankle confusion.

[0038] To predict and evaluate the smoothness of the generated first distance trajectory time series, in operation S202, a second distance trajectory time series for storing the corrected distance trajectory is first initialized. Specifically, the initialized second distance trajectory time series has the same data format as the first distance trajectory time series, such as a one-dimensional numerical list or array. Meanwhile, the initialization also includes directly taking the first value of the first distance trajectory time series as the initial value of the first value of the second distance trajectory time series. For example, assuming that the first distance trajectory time series is [15, 2, -2, 15, -25], the initialization includes creating a data structure with the same format as the first distance trajectory time series, and reading the first value 15 of the first distance trajectory time series as the initial value of the second distance trajectory time series, i.e., the initialized second distance trajectory time series is

[15] . It should be noted that the second distance trajectory time series stores the real ankle horizontal distance value (i.e., the real ankle horizontal coordinate difference value) that is considered to be the most consistent with the motion law after frame-by-frame judgment, i.e., it is a distance trajectory time series after correction (exchange of left and right foot coordinates) based on the original distance trajectory, which is dynamically generated in the data correction process and is only initialized before the start of the data correction process.

[0039] In operation S203, the pre-constructed motion trend model is a dynamic mathematical model for predicting the motion trend by approximating the velocity and acceleration by finite difference, which can be implemented as a mathematical constraint for forcing the motion trajectory to be smooth based on a difference equation. The core idea of the model is the inertia or smooth motion assumption, i.e., the acceleration of an object will not change drastically in the absence of sudden external forces. Therefore, the model will predict that the distance value at the next time should be within a certain range based on the historical distance value, velocity and acceleration, so as to minimize the change of acceleration. For example, the motion trend model can be implemented as (relative) velocity vt at time t is approximated as the distance value of the current frame minus the distance value of the previous frame, i.e., vt ≈ D_t - D_{t-1}, and (relative) acceleration at time t is approximated as at ≈ vt - vt-1 ≈ (D_t - D_{t-1}) - (D_{t-1} - D_{t-2}). Figure 5 The processing procedure of the data correction of the human body skeletal point sequence of the current frame based on the motion trend model and the first distance trajectory time series is schematically shown in an embodiment. In a specific application, the data correction of the human body skeletal point sequence corresponding to the monocular video can be completed by performing the processing on the monocular video frame by frame. As shown in FIG. 3, the processing procedure includes the following steps. Figure 5The data correction processing performed on the human body skeleton point sequence of the current frame includes: Operation S204, according to the motion trend model, the first distance trajectory time sequence and the second distance trajectory time sequence, respectively calculating the displacement change amount generated by the first distance trajectory time sequence corresponding to the current frame accessing the second distance trajectory time sequence under the first assumption scenario and the second assumption scenario; Operation S205, according to the displacement change amount, determining the trajectory smoothness scores corresponding to the current frame under the first assumption scenario and the second assumption scenario; Operation S206, according to the trajectory smoothness scores corresponding to the current frame under the first assumption scenario and the second assumption scenario, determining whether the left and right ankles of the current frame are confused, and when it is determined that the confusion occurs, performing data correction on the human body skeleton point sequence corresponding to the current frame by exchanging the coordinate data of the left and right ankles.

[0040] For the current frame to be processed, in operation S204, the displacement variation amount generated by accessing the first distance trajectory time sequence corresponding to the current frame to the second distance trajectory time sequence is first calculated under different scenarios. Preferably, two hypothetical scenarios are proposed in the embodiments of the present application, the first hypothetical scenario refers to the correctly identified scenario, under which it is assumed that the judgment of left and right ankles is correct when the human body skeleton point sequence is extracted; the second hypothetical scenario refers to the misidentified scenario, under which it is assumed that the left and right ankles are confused when the human body skeleton point sequence is extracted, and in this case, the true distance value should be the negative of the distance value corresponding to the current frame in the first distance trajectory time sequence. It should be noted that the displacement variation amount under different scenarios refers to the amount of change that will be generated if the distance value corresponding to the current frame in the first distance trajectory time sequence is accessed to the second distance trajectory time sequence, i.e., the corrected trajectory, under the two hypothetical scenarios. It should be noted that the meaning of "access" here is to tentatively append or connect the distance value corresponding to the current frame in the first distance trajectory time sequence to the end of the second distance trajectory time sequence, for example, assuming that the current frame is the second frame, the first distance trajectory time sequence is [15, 2, -2, 15, -25], and the initialized second distance trajectory time sequence is

[15] , at this time, accessing the distance value corresponding to the current frame in the first distance trajectory time sequence to the second distance trajectory time sequence means adding the second value 2 of the first distance trajectory time sequence to the second distance trajectory time sequence

[15] , and adding it after 15. It should be further noted that since when the current frame t is processed, there is already a second distance trajectory time sequence from the first frame to the last frame t-1 that is considered to be smooth and correct, and when the second distance trajectory time sequence is updated based on the current frame, whether the distance value in the first distance trajectory time sequence corresponding to the current frame is correct needs to be verified (i.e., the distance value in the first distance trajectory time sequence may be incorrect), thus the so-called "access" actually temporarily adds a certain candidate distance value (for example, the original value or its negative value) of the current frame to the end of the corrected second distance trajectory time sequence, in order to evaluate whether this joining action will destroy the smoothness of the second distance trajectory time sequence, and determine the corrected distance value finally added to the second distance trajectory time sequence according to the evaluation result. Thus, the displacement variation amount in operation S204 measures the gap between the newly accessed candidate value and the theoretical value predicted according to the historical trajectory, wherein the theoretical value predicted according to the historical trajectory is specifically predicted by the motion trend model constructed according to the historical trajectory.

[0041] Wherein, whenever a new frame's data is processed and its revised correct distance value is determined, this revised distance value is added to the end of the second distance trajectory time series, and the motion trend model immediately recalculates the velocity (first order difference) and acceleration (second order difference) at the end of the latest trajectory with this new data. In determining the revised correct distance value of the tth frame, the motion trend model relies on the velocity and acceleration calculated from all the previous revised distance values of the (t-1)th, (t-2)th, etc. frames. Once the correct distance value of the tth frame is determined, it becomes the known historical data for the calculation of the (t+1)th frame. Therefore, the historical trajectory on which the motion trend model is based includes the distance values in the second distance trajectory time series from the first frame to the (t-1)th frame.

[0042] As a possible implementation, for the current frame t, the temporarily newly accessed candidate value includes a candidate value under a first assumed scenario and a candidate value under a second assumed scenario. In the first assumed scenario, i.e. assuming correct recognition, the candidate value is the distance value D_raw_t corresponding to the current frame t in the first distance trajectory time series, while in the second assumed scenario, i.e. assuming incorrect recognition, the candidate value is set to the negative of the distance value corresponding to the current frame in the first distance trajectory time series, i.e. -D_raw_t. Thus, the calculation of the displacement change amount in operation S204 includes calculating the absolute value of the difference between the candidate value under the first assumed scenario and the predicted theoretical value, and calculating the absolute value of the difference between the candidate value under the second assumed scenario and the predicted theoretical value. Assuming that the predicted theoretical value is Predicted_D_t, then the displacement change amount under the first assumed scenario = abs(D_raw_t - Predicted_D_t), and the displacement change amount under the second assumed scenario = abs(-D_raw_t - Predicted_D_t). The smaller the displacement change amount, the more consistent the newly accessed candidate value is with the inertia of the historical motion, i.e. the smoother the trajectory, which will be a core component of the trajectory smoothness score. It should be noted that the candidate value is only a possible option for accessing the second distance trajectory time series, and only the candidate value that is considered correct after the smoothness test will be truly accessed to the second distance trajectory time series as the revised correct distance value.

[0043] To make a choice from the above two kinds of assumed scenarios, in operation S205, the embodiment of the present application calculates the trajectory smoothness score under different assumed scenarios corresponding to the current frame, which is a comprehensive index, and the score value can fully reflect whether the access to the candidate value can make the whole motion trajectory appear smoother and more in line with the physical motion law by accumulating three kinds of kinematic components to generate the final smoothness score. Preferably, the three kinds of kinematic components are the displacement change amount generated by accessing the corresponding candidate value (i.e. the displacement change amount corresponding to each candidate value calculated in operation S204), the difference between the displacement change amount and the speed at the last moment, and the difference between the displacement change amount and the acceleration at the last moment. Exemplarily, the result of weighted sum of the absolute values of the three components can be taken as the final trajectory smoothness score. Since the displacement change amount, acceleration and jerk of a smooth motion should be very small, a lower score in this case represents that accessing the candidate value will make the whole motion trajectory appear smoother and more in line with the physical motion law. Among them, the difference between the displacement change amount and the speed at the last moment measures the degree of change in speed, which can be represented by the size of the newly generated acceleration. Specifically, the difference can be the absolute value of the difference between the new speed generated after accessing the candidate value and the historical speed at the last moment. Exemplarily, assuming that the current is the t-th frame, the candidate value is Candidate_D_t, and the corrected distance value of the last frame is Corrected_D_{t-1}, then the calculated new speed is New_v_t = Candidate_D_t - Corrected_D_{t-1}. Then, the speed at the last moment v_{t-1} = Corrected_D_{t-1} - Corrected_D_{t-2} is obtained from the historical data, and the difference abs(New_v_t - v_{t-1}) can be determined. The difference between the displacement change amount and the acceleration at the last moment measures the degree of change in acceleration, which can be represented by the newly generated jerk, and its size reflects the smoothness of the motion change. Specifically, the difference can be the absolute value of the difference between the new acceleration generated after accessing the candidate value and the historical acceleration at the last moment. Exemplarily, assuming that the current is the t-th frame, then the calculated new acceleration is New_a_t = New_v_t - v_{t-1}. Then, the acceleration at the last moment a_{t-1} is obtained from the historical data, and the difference abs(New_a_t - a_{t-1}) can be determined.

[0044] In other possible implementations, a trend consistency judgment mechanism can also be introduced when calculating the trajectory smoothness score. Specifically, it can be implemented as follows: whether the motion direction generated under different hypothetical scenarios is consistent with the overall motion direction of the last few frames is judged. If the motion direction generated under a certain hypothetical scenario is consistent with the overall motion direction of the last few frames, an additional weight coefficient is added to the overall smoothness score of this hypothetical scenario to adjust the overall smoothness score, so that it is more likely to be selected as the optimal solution. Among them, the motion direction generated under different hypothetical scenarios is determined by the sign of the newly generated speed value under different hypothetical scenarios. The calculation method of the newly generated speed value is as described above. The overall motion direction of the last few frames is determined by analyzing the corrected historical trajectory, that is, by analyzing the speed sign corresponding to the last few frames. For example, the sign of the speed of the last three frames can be taken as a vote, that is, sign (sign (v_{t-1}) + sign (v_{t-2} + sign (v_{t-3}))). This sign represents the most recent and most reliable "overall motion direction" because it reflects the recent trend of the motion trajectory before accessing the current point. For example, the additional weight coefficient can be an independent multiplier that is multiplied by the calculated trajectory smoothness score to adjust the final smoothness score. For example, the coefficient is a number less than 1 (for example, 0.8), and the final smoothness score under the corresponding hypothetical scenario is obtained by multiplying the aforementioned calculated smoothness score by this additional weight coefficient. Thus, when the motion direction under a certain hypothetical scenario is consistent with the overall motion direction of the last few frames, the overall smoothness score under this hypothetical scenario is adjusted, so that the finally selected trajectory is more consistent with the actual motion trend.

[0045] After calculating the trajectory smoothness scores under different hypothetical scenarios, in operation S206, whether the left and right ankles of the current frame are confused can be determined by comparing the trajectory smoothness scores corresponding to the current frame under the two hypothetical scenarios. For example, the lower score (i.e., the smoother trajectory) can be selected as the correct state of the current frame. If the score of the "wrong recognition" scenario is lower, it is determined that the left and right ankles are confused when extracting the human skeleton key points of the current frame. At this time, the correction is performed by exchanging the coordinate data of the left and right ankles of the human skeleton key points corresponding to the current frame in the human skeleton point sequence. Thus, by performing this processing process on each frame of the monocular video, that is, iterating from the first frame to the last frame, the key point confusion problem in the entire monocular video sequence can be systematically corrected, so that the accuracy of the human skeleton point sequence as the basis for analysis is higher.

[0046] It should be noted that when processing the first frame, since there is no historical data to refer to, when the data of the first frame is corrected, it is preferred to assume that the distance value in the corresponding first distance trajectory time sequence of the first frame is correct, so the first value of the first distance trajectory time sequence is directly connected to the first position of the second distance trajectory time sequence by initialization. Starting from the second frame, the above method can be used to process the monocular video frame by frame to solve the problem of ankle confusion.

[0047] In other possible embodiments, the data correction of the human skeleton point sequence also includes the processing of the occluded limb key points. Specifically, after obtaining the human skeleton point sequence, the confidence c corresponding to the human skeleton point sequence is also judged. If it is judged that the confidence score of a certain skeleton point is lower than a preset threshold, the coordinate data of the skeleton point is marked as an invalid value, so as to be regarded as an unreliable data point. Thus, in the subsequent abnormal gait evaluation and analysis, these unreliable data points can be identified and ignored, so as to avoid the interference of noise data on the overall feature analysis and improve the accuracy of the evaluation result. Wherein, marking the coordinate data of a certain skeleton point as an invalid value can be realized by replacing the coordinate data of the skeleton point with a special marker, for example, setting the coordinates to 0 or NaN, and setting the confidence to 0. By setting the coordinate data as an invalid value, the placeholder of the corresponding data point is retained, which ensures that the data tensor input into the spatio-temporal graph convolution network model is always regular and fixed in size, which can not only eliminate noise interference, but also maintain the data time sequence and structural integrity, facilitating batch processing and efficient calculation of the model.

[0048] The existing monocular video method often affects the performance due to inaccurate 2D skeleton extraction (especially in occlusion or specific postures, such as crossing of legs in lateral walking). However, the embodiment of the present application significantly improves the quality of the skeleton data input into the spatio-temporal graph convolution network model through the above-mentioned specially designed skeleton data correction method (especially for the processing of left-right leg confusion and occluded arms), thereby improving the effectiveness of subsequent spatio-temporal feature learning and the accuracy and robustness of final anomaly detection.

[0049] As a preferred embodiment, in operation S3, the spatio-temporal graph convolution network model used adopts a twin network structure, which is designed as a symmetrical double-branch architecture, including two sub-networks that are completely identical in structure and share all network parameters (including weights and biases, etc.) during the training process. The sub-networks are spatio-temporal graph convolution networks (ST-GCN). This strict parameter sharing mechanism ensures that the two sub-networks are like two perfectly calibrated rulers, which can measure and encode data from different perspectives with completely consistent standards, which is a prerequisite for effective comparison. Figure 6An architecture principle block diagram of a spatio-temporal graph convolution network model of an embodiment of the present application is shown schematically as Figure 6 As shown, the spatio-temporal graph convolution network model includes two sub-networks B1 and B2 which are structurally identical. Within each ST-GCN sub-network, the data flows through multiple stacked spatio-temporal convolution modules. Each spatio-temporal convolution module contains two core operations: one is spatial graph convolution, which aggregates the information of neighboring joints along the edges of the graph, thereby learning the spatial coordination relationship of different parts of the body within a frame of image (for example, the coordinated movement of the shoulder and the hip when walking); the other is temporal convolution, which convolves the motion trajectory of each joint in the time dimension, thereby learning the dynamic change pattern of the joint (for example, the flexion and extension rhythm of the knee joint). Through the stacking of multiple layers of modules, the sub-network can gradually abstract and extract high-dimensional spatio-temporal features that can represent the essence of gait from the original coordinate data. After processing the respective input sequences, each sub-network finally outputs a high-dimensional feature vector. This vector can be understood as a deep encoding embedding of one side of the gait, which contains the complex spatial posture and temporal dynamic information of the side of the gait in a compact form. The two vectors are denoted as left-view feature vector and right-view feature vector in the embodiment of the present application. Thus, the Siamese ST-GCN architecture of the embodiment of the present application can realize gait asymmetry analysis, that is, the network can quantify the asymmetry of the gait by calculating and analyzing the difference between the two branch output feature vectors, and accordingly detect and evaluate the abnormal gait, thereby improving the robustness and accuracy of the detection result.

[0050] Further, the spatio-temporal graph convolution network model used also includes a comparison and fusion module, which is the core of realizing information interaction and decision-making under double-view. It receives the left-view feature vector and the right-view feature vector from the two Siamese branch sub-networks and performs feature comparison and fusion to generate a fusion feature vector. Specifically, in order to integrate the gait information of the left and right sides, the module can use the feature concatenation (Concatenation) method to generate a fusion feature vector, that is, the left-view feature vector and the right-view feature vector from the two Siamese branch sub-networks are directly connected into a longer vector, which retains all the original feature information of the left and right sides for the subsequent network layer to learn the internal correlation. Thus, through the comparison and fusion module, the left-view feature vector and the right-view feature vector from the two Siamese branch sub-networks can be generated into a fusion feature vector containing double-view comparison information.

[0051] More preferably, the spatio-temporal graph convolutional network model used further comprises a classifier consisting of one or more fully connected layers (MLP, multi-layer perceptron) that receives the fused feature vector input from the contrast and fusion module, and the fully connected layers in the classifier are responsible for further nonlinear transformation and information extraction of the fused feature vector, and finally map it to the predefined output categories, which can include, for example, the gait belongs to normal, the gait belongs to abnormal, the gait belongs to which degree of abnormality, etc.

[0052] More preferably, the last layer of the spatio-temporal graph convolutional network model used can be designed as a Softmax layer (activation function layer) configured to output multi-classification results such as the probability distribution of the subject's gait belonging to "normal" or different "abnormal" levels or corresponding UPDRS score 0, 1, 2 levels, thereby completing the final judgment and evaluation. In other embodiments, the last layer can also be a Sigmoid layer (for binary classification, i.e. normal / abnormal), or a regression layer (for outputting continuous abnormality score), etc., which can be adjusted according to the expected model output result, and the embodiments of the present application do not limit this.

[0053] During training, the entire spatiotemporal graph convolutional network model with the above structure can be directly trained as a whole using an end-to-end training mode. For example, the spatiotemporal graph convolutional network model including two ST-GCN branches with shared weights, a comparison and fusion module, and a classifier can be regarded as a unified whole for end-to-end training. The training data needs to simultaneously prepare a corrected human skeleton point sequence from the left perspective and a corrected human skeleton point sequence from the right perspective. During the training process, the embodiment of the present invention uses the corrected human skeleton point sequence from the left perspective and the corrected human skeleton point sequence from the right perspective as independent inputs, respectively, and feeds them into the two ST-GCN subnetworks of the spatiotemporal graph convolutional network model. The human skeleton point sequence from each perspective is constructed into a spatiotemporal graph when input, wherein the nodes of the graph correspond to the various skeletal key points of the human body, and the edges of the graph are defined based on the inherent skeletal connection relationship of the human body (such as the connection between the thigh bone and the knee and hip) and the connection relationship of the same key point in consecutive time frames. The construction process can refer to the prior art, and the embodiment of the present invention will not elaborate on this. In addition, the training data also includes ground-truth labels corresponding to the prepared corrected left-view and right-view human skeleton point sequences. These ground-truth labels are generated based on the subject's actual gait. During the training process, a backpropagation algorithm is used to synchronously update all learnable parameters in the network based on the difference between the final predicted results and the ground-truth labels, thereby obtaining a trained spatiotemporal graph convolutional network model. This training method enables the network model to automatically learn how to extract the most discriminative asymmetry features from the original dual-view skeleton data and accurately map them to specific real-world scene results (such as abnormality and severity). The resulting trained spatiotemporal graph convolutional network model uses the corrected left-view and right-view skeleton sequences as independent inputs to two sub-networks, respectively, and outputs a probability distribution of whether the subject's gait is normal or has different levels of abnormality. In specific applications, simply preparing the left-view and right-view skeleton point sequences and inputting them into the trained network model yields an output probability distribution of whether the subject's gait is normal or has different levels of abnormality, which can be directly used as an evaluation result. Therefore, it is not difficult to understand that in the evaluation method of the embodiment of the present invention, the monocular video can be a monocular video including a left-side perspective and a monocular video including a right-side perspective, and the corresponding human skeleton point sequences extracted and corrected also include a left-side perspective skeleton sequence and a right-side perspective skeleton sequence. It should be noted that in order to ensure the validity of the evaluation results, the input data received by the network model are the left-side perspective skeleton sequence and the right-side perspective skeleton sequence corresponding to the left and right-side walking unidirectional videos of the same subject.

[0054] In other possible implementations, the output result or evaluation result may also be a direct result of whether the subject's gait is normal or at different abnormal levels. In this case, the subject's gait can be directly determined based on the probability distribution of whether the subject's gait is normal or at different abnormal levels (e.g., the one with a higher probability is selected as the direct result). In other possible implementations, the output result or evaluation result of the model may also be the category, probability, or severity score of the subject's gait abnormality, etc., and this embodiment of the present invention is not limited to this.

[0055] In some embodiments, the above-mentioned evaluation method of the embodiment of the present invention can be implemented on a mobile terminal device such as a smartphone; in other embodiments, the above-mentioned evaluation method of the embodiment of the present invention can also be implemented on a cloud server; in a preferred embodiment, the above-mentioned evaluation method of the embodiment of the present invention can be implemented partially on the server and partially on the mobile terminal device. For example, in some embodiments, the acquisition and video segmentation of the side video to generate the monocular video can be implemented on the mobile terminal device, and then the human skeleton point sequence of the monocular video is transmitted to the cloud server for subsequent processing such as data correction and forming the evaluation result. For example, in some other embodiments, only the aforementioned operations S3 and S4 can be implemented on the cloud server, while the acquisition and data correction of the human skeleton point sequence such as the monocular video are both implemented on the mobile terminal device, and only the corrected human skeleton point sequence and necessary metadata are uploaded to the cloud server for evaluation. Among them, preferably, if the computing power of the mobile terminal is limited or to ensure the highest accuracy, in the embodiment where the video segmentation processing and skeleton point sequence extraction are implemented by the cloud server, the original collected side video data can be securely uploaded to the cloud server after being compressed and encrypted. Preferably, the personal identity information of the subjects can be separated or desensitized as much as possible before data collection and uploading. For example, the system assigns a unique research ID that is not directly related to the real identity as the subject's identity information and associates it with the subject's video and evaluation results. For example, when the original side video collected needs to be uploaded, the face area in the video can be blurred or masked in real time on the mobile terminal device to prevent identity recognition, and only body posture information is retained for gait analysis. By performing preliminary processing on the mobile terminal, data anonymization / pseudonymization, encrypted transmission and storage, etc., the privacy of the subjects can be protected to the greatest extent, and data security and privacy protection can be enhanced.

[0056] The abnormal gait assessment method provided by the embodiments of the present invention enhances system performance in complex environments through an improved skeletal keypoint extraction and correction strategy and an optimized spatiotemporal feature learning network. This effectively improves the accuracy and robustness of abnormal gait detection in monocular video. Furthermore, the solution of the embodiments of the present invention utilizes a common monocular camera (such as a smartphone) for effective gait analysis, enabling low-cost and convenient gait anomaly detection and facilitating widespread application. Furthermore, the method of the embodiments of the present invention utilizes advanced deep learning models (such as the spatiotemporal graph convolutional network (ST-GCN)) to automatically learn discriminative features from gait sequences, enhancing the capture and understanding of the spatiotemporal dynamic characteristics of gait and improving the ability to identify subtle and specific gait anomalies.

[0057] Figure 7 The structure of the abnormal gait assessment system based on monocular video according to one embodiment of the present invention is schematically shown. Figure 7 As shown, the system includes a mobile device 600 and a cloud server 700, wherein: The mobile device is used to collect lateral video of the subject while walking through a monocular camera and send it to the cloud server; The cloud server includes A skeleton point extraction module 701 is used to extract a human skeleton point sequence from the side video; The video segmentation module 702 is used to perform video segmentation processing on the lateral video according to the human skeleton key points to generate a human skeleton point sequence of the monocular video; A data correction module 703 is used to perform data correction on the human skeleton point sequence of the generated monocular video; and The gait assessment module 704 is used to input the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output result of the spatiotemporal graph convolutional network model, wherein the output result includes the assessment result of the gait abnormality of the subject in the monocular video.

[0058] Figure 8 The structure of the abnormal gait assessment system based on monocular video according to one embodiment of the present invention is schematically shown. Figure 8 As shown, the system includes a mobile device 600 and a cloud server 700, wherein: The mobile device 600 includes A monocular camera 601 for collecting lateral video of the subject while walking; a skeleton data extraction module 602 for performing video segmentation processing on the lateral video to generate a human skeleton point sequence of the monocular video, and sending the generated human skeleton point sequence of the monocular video to the cloud server; The cloud server includes The data correction module 703 is used to correct the acquired human skeleton point sequence; The cloud server 704 is configured to input the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output of the spatiotemporal graph convolutional network model, wherein the output includes an assessment of gait abnormalities of the subject in the monocular video. This system allows skeleton extraction on mobile devices, reducing the need to transmit raw video data, and securely uploads the processed skeleton data to the cloud for further in-depth analysis and storage, achieving resource optimization.

[0059] In other possible implementations, the mobile device can also be configured to include a function selection module for module selection configuration, allowing users to flexibly choose to perform skeleton extraction, video segmentation, data correction, and other processing on the mobile device or on a cloud server based on the computing power of the mobile device and user needs. Extracting skeletons on the mobile device can significantly reduce the amount of uploaded data, saving bandwidth and cloud storage costs, and further protecting the privacy of the original video. The cloud is responsible for more complex, computationally intensive tasks, such as ST-GCN feature learning and model inference, to achieve optimal resource utilization.

[0060] It should be noted that the specific implementation process of each module in the abnormal gait assessment system can refer to the description of the method part above and will not be repeated here.

[0061] Figure 9 A computing device according to an embodiment of the present invention is schematically shown. As shown in the figure, the computing device includes: Memory 81, for storing executable instructions; and The processor 82 is communicatively connected to the memory 81 and is used to execute executable instructions stored in the memory. When the executable instructions are executed by the processor, the steps of the abnormal gait assessment method based on monocular video described in any of the aforementioned embodiments are implemented.

[0062] In some embodiments, an embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the abnormal gait assessment method based on monocular video according to any of the above embodiments of the present invention.

[0063] In some embodiments, the present embodiment also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the monocular video-based abnormal gait assessment method of any of the above embodiments.

[0064] In some embodiments, the present embodiment also provides an electronic device comprising: at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the monocular video-based abnormal gait assessment method of any of the above embodiments.

[0065] In some embodiments, the present embodiment also provides a storage medium having stored thereon a computer program, which, when executed by a processor, implements the monocular video-based abnormal gait assessment method of any of the above embodiments.

[0066] Figure 10 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. The computing device can be implemented in the structure shown in the figure, as shown in the figure, the electronic device comprises: Figure 10 as shown, the electronic device comprises: one or more processors 610 and a memory 620, Figure 10 in which the processor 610 is taken as an example.

[0067] The electronic device can also include an input device 630 and an output device 640.

[0068] The processor 610, the memory 620, the input device 630 and the output device 640 can be connected by a bus or other means, Figure 10 in which the connection by the bus is taken as an example.

[0069] The memory 620 is a kind of non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the monocular video-based abnormal gait assessment method in the embodiment of the present application. The processor 610 performs various functional applications and data processing of the server by running the non-volatile software program, instruction and module stored in the memory 620, that is, the monocular video-based abnormal gait assessment method of the above method embodiment is realized.

[0070] The memory 620 can include a program storage area and a data storage area, where the program storage area can store an operating system, application programs required by at least one function, and the data storage area can store data created according to the use of the monocular video-based abnormal gait assessment method, and the like. In addition, the memory 620 can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, or other nonvolatile solid-state memory device. In some embodiments, the memory 620 can optionally include a memory disposed remotely from the processor 610, which can be connected to the electronic device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0071] The input device 630 can receive input digital or character information, and generate signals related to user settings and function control of the image processing device. The output device 640 can include a display device such as a display screen.

[0072] The one or more modules are stored in the memory 620 and, when executed by the one or more processors 610, perform the monocular video-based abnormal gait assessment method in any of the method embodiments described above.

[0073] The above product can perform the method provided by the embodiments of the application, and has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the application.

[0074] The electronic device of the embodiments of the application exists in various forms, including but not limited to: (1) Mobile communication device: This type of device is characterized by having mobile communication function, and providing voice and data communication as the main target. This type of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0075] (2) Ultra-mobile personal computer device: This type of device belongs to the category of personal computers, has computing and processing functions, and generally also has the characteristics of mobile Internet. This type of terminal includes: PDA, MID and UMPC devices, such as iPad.

[0076] (3) Portable entertainment device: This type of device can display and play multimedia content. This type of device includes: audio and video player (such as iPod), handheld game console, electronic book, and smart toy and portable car navigation device.

[0077] (4) Server: a device providing computing services, the server is composed of a processor, a hard disk, a memory, a system bus, etc., the server is similar to a general computer architecture, but since it needs to provide high-reliable services, it has higher requirements in processing capability, stability, reliability, security, scalability, manageability, etc.

[0078] (5) Other electronic devices with data interaction function.

[0079] The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs.

[0080] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus a general hardware platform, and of course, they can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0081] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. The abnormal gait assessment method based on monocular video is characterized by: include: Obtaining a human skeleton point sequence of the monocular video; Performing data correction on the acquired human skeleton point sequence; Inputting the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model, and obtaining an output result of the spatiotemporal graph convolutional network model, wherein the output result includes an assessment result of gait abnormality of the subject in the monocular video; The data correction of the acquired human skeleton point sequence includes: Generate a first distance trajectory time series according to a left ankle human skeleton point sequence and a right ankle human skeleton point sequence in the human skeleton point sequence of the monocular video; Initialize the second distance trajectory time series; According to the first distance trajectory time series, the second distance trajectory time series and a pre-built motion trend model, data correction processing is performed on the human skeleton point sequence of the monocular video frame by frame.

2. The method according to claim 1, characterized in that Obtaining a human skeleton point sequence of the monocular video, including: Extract key points of human skeleton from the side-view video of the subject walking captured by a monocular camera; Performing video segmentation processing on the lateral video according to the extracted human skeleton key points; A human skeleton point sequence of the monocular video is generated according to the video segmentation processing result.

3. The method according to claim 2, characterized in that The video segmentation processing is performed on the lateral video according to the extracted human skeleton key points, comprising: Determine the motion anchor point based on the extracted key points of the human skeleton; generating an instantaneous horizontal velocity time series according to the motion anchor point and the lateral video; identifying effective walking intervals and potential turning points in the lateral video according to the instantaneous horizontal speed time series; Segmenting the lateral video according to the effective walking interval and the potential turning point and generating at least one unidirectional walking segment; The monocular video for abnormal gait assessment is determined based on the generated at least one unidirectional walking segment.

4. The method according to claim 1, wherein The video frame to be processed for data correction in the monocular video is referred to as the current frame. The data correction processing performed on the current frame according to the first distance trajectory time series, the second distance trajectory time series, and the pre-built motion trend model includes: Calculate, based on the motion trend model, the first distance trajectory time series, and the second distance trajectory time series, respectively, a displacement change resulting from the first distance trajectory time series corresponding to the current frame being connected to the second distance trajectory time series under the first hypothetical scenario and the second hypothetical scenario; Determining, based on the displacement change, a trajectory smoothness score corresponding to the current frame under the first hypothetical scenario and the second hypothetical scenario; Whether the left and right ankles of the current frame are confused is determined based on the trajectory smoothness scores corresponding to the first and second hypothetical scenarios. When confusion is determined to have occurred, data correction is performed on the human skeleton point sequence corresponding to the current frame by exchanging the coordinate data of the left and right ankles.

5. The method according to claim 1, wherein The spatiotemporal graph convolutional network model adopts a twin network structure, which includes two sub-networks that are identical in structure and share all network parameters during training. The sub-networks are spatiotemporal graph convolutional networks; The human skeleton point sequence includes a left-view skeleton sequence and a right-view skeleton sequence. The pre-trained spatiotemporal graph convolutional network model uses the corrected left-view skeleton sequence and the right-view skeleton sequence as independent inputs of two sub-networks, respectively, and uses the probability distribution of whether the subject's gait is normal or at different abnormal levels as the output result.

6. An abnormal gait assessment method based on monocular video, used on mobile devices, characterized in that: include: Obtaining a human skeleton point sequence of the monocular video; Performing data correction on the acquired human skeleton point sequence; Sending the corrected human skeleton point sequence to a server, and obtaining an assessment result of the abnormal gait of the subject in the monocular video from the server, wherein the server is used to input the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model, and feed back the assessment result to the mobile device based on an output result of the spatiotemporal graph convolutional network model; The data correction of the acquired human skeleton point sequence includes: Generate a first distance trajectory time series according to a left ankle human skeleton point sequence and a right ankle human skeleton point sequence in the human skeleton point sequence of the monocular video; Initialize the second distance trajectory time series; According to the first distance trajectory time series, the second distance trajectory time series and a pre-built motion trend model, data correction processing is performed on the human skeleton point sequence of the monocular video frame by frame.

7. The abnormal gait assessment system based on monocular video is characterized by: Including mobile devices and cloud servers, The mobile device is used to collect lateral video of the subject while walking through a monocular camera and send it to the cloud server; The cloud server includes A skeleton point extraction module is used to extract key points of human skeleton from the side video; A video segmentation module is used to perform video segmentation processing on the lateral video according to the human skeleton key points to generate a human skeleton point sequence of the monocular video; A data correction module, used to correct the data of the generated human skeleton point sequence; and A gait assessment module is used to input the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output result of the spatiotemporal graph convolutional network model, wherein the output result includes an assessment result of the gait abnormality of the subject in the monocular video.

8. The abnormal gait assessment system based on monocular video is characterized by: Including mobile devices and cloud servers, Mobile devices include a monocular camera for capturing lateral video of the subject while walking; a skeleton data extraction module, configured to perform video segmentation processing on the lateral video to generate a human skeleton point sequence of the monocular video, and send the generated human skeleton point sequence of the monocular video to the cloud server; The cloud server includes A data correction module is used to correct the data of the acquired human skeleton point sequence; A gait assessment module is used to input the corrected human skeleton point sequence into a pre-trained spatiotemporal graph convolutional network model and obtain the output result of the spatiotemporal graph convolutional network model, wherein the output result includes an assessment result of the gait abnormality of the subject in the monocular video.

9. A computing device, characterized in that include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 5.

10. A storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images