Gait analysis method and system in child rehabilitation

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a gait analysis method and system in child rehabilitation. Analyzing data collected by the plantar pressure sensor to obtain gait cycles of left and right legs, and obtaining starting time corresponding to a plurality of stages of each gait cycle of each leg; searching a plurality of frames corresponding to the stages from the videos on the two sides according to the starting time corresponding to each stage; converting a plurality of frames corresponding to the starting point of each stage into a plurality of grayscale images, obtaining the position of each joint on each side according to the plurality of grayscale images corresponding to the video on each side and a joint recognition model, and processing the positions of the joints recognized from the two sides to obtain the position of each joint; obtaining a position sequence of each joint in each gait cycle according to the sequence of the stages in the gait cycles; and processing the plurality of position sequences corresponding to the plurality of gait cycles to obtain an analysis result. According to the invention, the accuracy of gait analysis is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a gait analysis method and system for children's rehabilitation. Background Art

[0002] As part of rehabilitation medicine, pediatric rehabilitation is mainly used to help children with functional disabilities. Through specialized training and therapy, children can learn or relearn basic motor skills and daily living skills and improve their self-care ability, which helps to reduce or overcome limitations caused by disability or disease and improve the quality of life; it also helps children better integrate into society and participate in school and community activities. Gait analysis is a scientific method to evaluate the way an individual walks. It plays an important role in children's rehabilitation training, especially in the diagnosis of children's psychomotor development disorders, such as cerebral palsy, autism, ADHD, Tourette syndrome, and treatment monitoring of central nervous system infections. In the early stages of rehabilitation, analyzing the child's gait can help diagnose the child's specific problems, such as abnormal gait, insufficient muscle strength or coordination problems, which can help doctors develop more targeted rehabilitation plans. During the rehabilitation process, analyzing the gait helps doctors evaluate the effectiveness of rehabilitation.

[0003] There are two main methods for collecting and analyzing gait: wearable devices and video images. The wearable device-based collection method requires wearing related equipment, which is easy to cause discomfort, especially for children, it is difficult to accurately follow the daily walking method. The video analysis-based method can overcome the above problems, but accuracy has always been a difficulty. Summary of the invention

[0004] In order to avoid interference to children as much as possible and to improve the accuracy of analysis as much as possible. In a first aspect, the present invention provides a method for gait analysis in child rehabilitation, the method comprising the following steps:

[0005] The data collected by multiple plantar pressure sensors are analyzed to obtain the gait cycles of the left and right legs during walking, and multiple stages of each gait cycle of each leg, as well as the corresponding starting time of each stage; at the same time, videos of both sides of the child during walking are collected, and multiple frames corresponding to the starting point of each stage are found from the videos on each side;

[0006] Converting a plurality of frames corresponding to the starting point of each stage into a plurality of grayscale images, obtaining the position of each joint on each side according to the plurality of grayscale images corresponding to the video on each side and the joint recognition model, processing the positions of the joints recognized on both sides to obtain the position of each joint; obtaining the position sequence of each joint in each gait cycle according to the order of the stages in the gait cycle;

[0007] The analysis results are obtained by processing multiple position sequences corresponding to multiple gait cycles.

[0008] Preferably, the data collected by the multiple plantar pressure sensors are analyzed to obtain the gait cycles of the left and right legs during walking, specifically:

[0009] A sequence of typical plantar pressure distribution diagrams of the left and right legs in each stage is pre-formed;

[0010] Converting data collected from multiple plantar pressure sensors into a plantar pressure distribution map to be matched, forming a plantar pressure distribution map sequence to be matched in chronological order, finding a plantar pressure distribution map subsequence whose matching degree with the typical plantar pressure distribution map sequence is greater than a threshold in the plantar pressure distribution map sequence to be matched by a sliding window method, and marking the starting point of the subsequence as the corresponding stage starting point;

[0011] Then the gait cycle of the left and right legs is obtained.

[0012] Preferably, the position of each joint on each side is obtained according to the multiple grayscale images corresponding to the video on each side and the joint recognition model, specifically:

[0013] A feature map of each grayscale image is obtained by two-dimensional convolution, and a feature map corresponding to the grayscale image is added to a first feature map sequence according to a time corresponding to the grayscale image in the video, and a second feature map sequence of the plurality of grayscale image sequences is obtained by three-dimensional convolution;

[0014] Dividing the first feature map sequence into a plurality of subsequences according to the depth of the convolution kernel of the three-dimensional convolution, and inserting the subsequences into the second feature map sequence in sequence to obtain a feature map fusion result;

[0015] The feature map fusion results are input into the corresponding joint recognition model to identify the position coordinates of the joints.

[0016] Preferably, the training of the joint recognition model includes two stages, specifically:

[0017] In the first stage, the joint recognition model is trained using the plurality of grayscale images corresponding to each side of the video;

[0018] In the second stage, for the left joint recognition model, the multiple grayscale images corresponding to the left video are used to re-train the joint recognition model to obtain the left joint recognition model; for the right joint recognition model, the multiple grayscale images corresponding to the right video are used to re-train the joint recognition model to obtain the right joint recognition model.

[0019] Preferably, the positions of the joints identified from both sides are processed to obtain the position of each joint, specifically:

[0020] Determine whether there are the same joints in the joint set identified by the left joint recognition model and the joint set identified by the right joint recognition model. If so, take the average of the positions of the two as the position of the same joint.

[0021] Preferably, the processing of the multiple position sequences corresponding to the multiple gait cycles to obtain the analysis results is specifically as follows:

[0022] Fitting multiple position sequences of multiple gait cycles according to joints and stages to obtain the final position of each joint in a gait cycle;

[0023] Compare the position of each joint in a gait cycle that is fitted with the position of each joint in a gait cycle that was fitted the last time, and plot the comparison results on a graph.

[0024] On the other hand, the present invention also provides a gait analysis system for children's rehabilitation, the system comprising the following modules:

[0025] The gait phase acquisition module is used to analyze the data collected by multiple plantar pressure sensors to obtain the gait cycles of the left and right legs during walking, and obtain multiple phases of each gait cycle of each leg, as well as the corresponding starting time of each phase; at the same time, the video of both sides of the child during walking is collected, and multiple frames corresponding to the starting point of each phase are found from the video of each side;

[0026] A joint position acquisition module, used to convert a plurality of frames corresponding to the starting point of each stage into a plurality of grayscale images, obtain the position of each joint on each side according to the plurality of grayscale images corresponding to the video on each side and the joint recognition model, process the positions of the joints recognized on both sides to obtain the position of each joint; and obtain the position sequence of each joint in each gait cycle according to the order of the stages in the gait cycle;

[0027] The analysis module is used to process multiple position sequences corresponding to multiple gait cycles to obtain analysis results.

[0028] Preferably, the data collected by the multiple plantar pressure sensors are analyzed to obtain the gait cycles of the left and right legs during walking, specifically:

[0029] A sequence of typical plantar pressure distribution diagrams of the left and right legs in each stage is pre-formed;

[0030] Converting data collected from multiple plantar pressure sensors into a plantar pressure distribution map to be matched, forming a plantar pressure distribution map sequence to be matched in chronological order, finding a plantar pressure distribution map subsequence whose matching degree with the typical plantar pressure distribution map sequence is greater than a threshold in the plantar pressure distribution map sequence to be matched by a sliding window method, and marking the starting point of the subsequence as the corresponding stage starting point;

[0031] Then the gait cycle of the left and right legs is obtained.

[0032] Preferably, the position of each joint on each side is obtained according to the multiple grayscale images corresponding to the video on each side and the joint recognition model, specifically:

[0033] A feature map of each grayscale image is obtained by two-dimensional convolution, and a feature map corresponding to the grayscale image is added to a first feature map sequence according to a time corresponding to the grayscale image in the video, and a second feature map sequence of the plurality of grayscale image sequences is obtained by three-dimensional convolution;

[0034] Dividing the first feature map sequence into a plurality of subsequences according to the depth of the convolution kernel of the three-dimensional convolution, and inserting the subsequences into the second feature map sequence in sequence to obtain a feature map fusion result;

[0035] The feature map fusion results are input into the corresponding joint recognition model to identify the position coordinates of the joints.

[0036] Preferably, the training of the joint recognition model includes two stages, specifically:

[0037] In the first stage, the joint recognition model is trained using the plurality of grayscale images corresponding to each side of the video;

[0038] In the second stage, for the left joint recognition model, the multiple grayscale images corresponding to the left video are used to re-train the joint recognition model to obtain the left joint recognition model; for the right joint recognition model, the multiple grayscale images corresponding to the right video are used to re-train the joint recognition model to obtain the right joint recognition model.

[0039] Preferably, the positions of the joints identified from both sides are processed to obtain the position of each joint, specifically:

[0040] Determine whether there are the same joints in the joint set identified by the left joint recognition model and the joint set identified by the right joint recognition model. If so, take the average of the positions of the two as the position of the same joint.

[0041] Preferably, the processing of the multiple position sequences corresponding to the multiple gait cycles to obtain the analysis results is specifically as follows:

[0042] Fitting multiple position sequences of multiple gait cycles according to joints and stages to obtain the final position of each joint in a gait cycle;

[0043] Compare the position of each joint in a gait cycle that is fitted with the position of each joint in a gait cycle that was fitted the last time, and plot the comparison results on a graph.

[0044] In addition, the present invention also provides a computer program product, which implements the above method when executed by a processor.

[0045] The present invention uses a plantar pressure sensor in combination with a joint recognition model to accurately identify the position of each gait cycle stage node, and compares the obtained position with the previous position to obtain a comparison result in rehabilitation training, thereby effectively improving the accuracy of gait analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of Embodiment 1;

[0047] Figure 2 It is the plantar pressure distribution diagram;

[0048] Figure 3 Schematic diagram of feature fusion. DETAILED DESCRIPTION

[0049] In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the present invention, if it involves the collection of personal privacy data, such as face, mobile phone usage information, etc., the individual's permission will be obtained in advance, including but not limited to verbal reminders, posters, mobile phone reminders, etc.; if it involves conflicts with laws and regulations, it will be produced or used within the scope permitted by laws and regulations.

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Embodiment 1, the present invention provides a method for gait analysis in children's rehabilitation, such as Figure 1 As shown, the method comprises the following steps:

[0052] S1, analyzing the data collected by multiple plantar pressure sensors to obtain the gait cycles of the left and right legs during walking, and obtaining multiple stages of each gait cycle of each leg, as well as the corresponding starting time of each stage; at the same time, collecting videos of both sides of the child during walking, and finding multiple frames corresponding to the starting point of each stage from the video of each side;

[0053] Compared with sensors tied to the ankles, plantar pressure sensors collect pressure information through the soles of the feet, such as plantar pressure insoles, etc., which are more easily accepted by children and have less impact on children's walking. The present invention uses plantar pressure sensors to obtain children's foot pressure information, and then obtains each stage in the gait cycle and the starting time of each stage.

[0054] The gait cycle refers to the entire process from when a foot touches the ground to when the same foot touches the ground again. A gait cycle is divided into multiple stages, such as initial contact, foot flat phase, mid-term support, heel lift, etc. At each stage, the joints of the human body will change, such as the knee joint, wrist joint, ankle joint, elbow joint, hip joint, etc. By analyzing these joints, we can determine the changes in children's gait, which is of great significance for the evaluation of rehabilitation training.

[0055] There are many pressure sensors distributed on the insole of the plantar pressure sensor. These pressure sensors continuously collect pressure data. By analyzing these pressure data, each stage can be obtained. In a specific embodiment, the data collected by multiple plantar pressure sensors are analyzed to obtain the gait cycle of the left and right legs during walking, specifically:

[0056] A sequence of typical plantar pressure distribution diagrams of the left and right legs in each stage is pre-formed;

[0057] The plantar pressure distribution diagrams of children undergoing different rehabilitation treatments during walking were collected in advance. Figure 2 A plantar pressure distribution diagram is shown. During walking, the pressure sensor continuously collects pressure data. For example, the pressure sensor collects pressure data at a cycle of 0.2s, and multiple plantar pressure distribution diagrams corresponding to each child are obtained, which form a sequence of plantar pressure distribution diagrams to be matched in chronological order.

[0058] The pressure distribution map sequence is segmented to obtain a subsequence corresponding to each stage. When there are multiple children, one stage corresponds to multiple subsequences. For example, if there are 10 children, the foot flat stage will correspond to 10 subsequences. Multiple subsequences are used in each stage because different diseases cause children to walk in different postures or ways.

[0059] When determining the stage of the child to be analyzed, it is preferred to determine the typical plantar pressure distribution map sequence according to the condition. For example, there are 10 subsequences in the flat foot stage, and epilepsy corresponds to 3 of them. In the analysis of epileptic children, these three subsequences are preferably used. These three subsequences are the typical plantar pressure distribution map sequences.

[0060] Using a sliding window method, find a subsequence of plantar pressure distribution maps whose matching degree with the typical plantar pressure distribution map sequence is greater than a threshold value in the plantar pressure distribution map sequence to be matched, and mark the starting point of the subsequence as the corresponding stage starting point;

[0061] After collecting the plantar pressure distribution map sequence of the child to be analyzed, a sliding window method is used to find the plantar pressure distribution map subsequence that best matches each stage, that is, a typical plantar pressure distribution map sequence is slid on the plantar pressure distribution map sequence of the child to be analyzed, and the plantar pressure distribution map subsequence that best matches the typical plantar pressure distribution map sequence is found. Since the typical plantar pressure distribution map sequence corresponds to the stage, the most matching plantar pressure distribution map subsequence corresponds to this stage. It should be noted that the above judgment process only finds the corresponding stage in a gait cycle. When there are multiple gait cycles, the above method can be used to find the stage contained in each gait cycle. There are many ways to determine the gait cycle, for example, according to time, a gait cycle preset time, and the plantar pressure distribution map sequence to be matched is divided into multiple by the preset time, and each is matched by a matching degree greater than a threshold. In a more detailed embodiment, if the gait cycle of a leg is to be judged, the above-mentioned matching method needs to be used simultaneously to match the plantar pressure distribution map sequences to be matched corresponding to the two legs, that is, if the matching degree of the plantar pressure distribution map sequence to be matched of the first leg and the first typical plantar pressure distribution map sequence is greater than a threshold, and the matching degree of the plantar pressure distribution map sequence to be matched of the other leg and the first typical plantar pressure distribution map sequence is greater than a threshold, and the time difference between the two is less than a preset value, then it is considered that the current stage is in a certain stage of the gait cycle.

[0062] In a preferred embodiment, the matching calculation method is to calculate the similarity of each distribution map pair in the two sequences, and then accumulate all the similarities in the sequences, and the higher the similarity, the better the match. Wherein, the distribution map pair includes two distribution maps, one of which is located in the typical plantar pressure distribution map sequence, and the other is located in the plantar pressure distribution map sequence to be matched.

[0063] Then the gait cycle of the left and right legs is obtained.

[0064] A complete gait cycle refers to the entire process from when a foot touches the ground to when the same foot touches the ground again. Since a single-sided camera will block the other side, the present invention uses a double-sided camera to shoot, and extracts the gait cycle of the left and right legs separately.

[0065] S2, converting a plurality of frames corresponding to the starting point of each stage into a plurality of grayscale images, obtaining the position of each joint on each side according to the plurality of grayscale images corresponding to the video on each side and the joint recognition model, processing the positions of the joints recognized on both sides to obtain the position of each joint; obtaining the position sequence of each joint in each gait cycle according to the order of the stages in the gait cycle;

[0066] The starting point of a stage is a certain moment. With the moment corresponding to the starting point of the stage as the center, several frames are extracted on the left and right sides of the starting point of the stage to obtain multiple frames corresponding to the starting point of each stage. Of course, using bilateral cameras will obtain two frame sequences. For example, the foot flat period of a gait cycle corresponds to two frame sequences. The two frame sequences are converted into grayscale images. Then, the position of each joint on each side is obtained according to the multiple grayscale images corresponding to each side of the video and the joint recognition model. For example, there are three joints to be analyzed. The positions of 1, 2 or 3 joints are identified from the frame sequence taken by the left camera, and the positions of 1, 2 or 3 joints are identified from the frame sequence taken by the right camera. Then, the position of each joint is obtained from the positions of the joints identified on both sides, that is, the position of each joint at the starting moment of the foot flat period is obtained. For the next stage, such as the starting moment of the mid-term support, the position of each joint can also be obtained, so that the position sequence of each joint is obtained. For example, there are three joints A, B, and C to be analyzed, and the gait cycle is divided into 6 stages. There are 6 positions for joint A, 6 positions for joint B, and 6 positions for joint C. This constitutes the position sequence corresponding to each joint in each gait cycle.

[0067] When there are multiple frames at the starting point of a stage, the accuracy of joint recognition can be improved, but the complexity is increased. In order to obtain the spatial and temporal information of the joints in multiple frames, in a preferred embodiment, the position of each joint on each side is obtained according to the multiple grayscale images corresponding to the video on each side and the joint recognition model, specifically:

[0068] A feature map of each grayscale image is obtained by two-dimensional convolution, and a feature map corresponding to the grayscale image is added to a first feature map sequence according to a time corresponding to the grayscale image in the video, and a second feature map sequence of the plurality of grayscale image sequences is obtained by three-dimensional convolution;

[0069] Dividing the first feature map sequence into a plurality of subsequences according to the depth of the convolution kernel of the three-dimensional convolution, and inserting the subsequences into the second feature map sequence in sequence to obtain a feature map fusion result;

[0070] Three-dimensional convolution (3D convolution) can not only obtain spatial information, but also obtain temporal information. Fusion of the features of three-dimensional convolution and two-dimensional convolution helps to extract more information. Among them, the step length of the depth of the three-dimensional convolution is the depth of the three-dimensional convolution kernel. For example, there are 12 grayscale images, the three-dimensional convolution kernel is 3×3×3, and the step length is 3. Then the grayscale images 1, 2, and 3 are passed through the three-dimensional convolution kernel to obtain the first feature map, the grayscale images 4, 5, and 6 are passed through the three-dimensional convolution kernel to obtain the second feature map, the grayscale images 7, 8, and 9 are passed through the three-dimensional convolution kernel to obtain the third feature map, and the grayscale images 10, 11, and 12 are passed through the three-dimensional convolution kernel to obtain the fourth feature map. These four feature maps constitute the second feature map sequence.

[0071] Then, the first feature map sequence is divided into 4 groups, the first group is inserted after the first feature map, the second group is inserted after the second feature map, and so on. Figure 3 shown.

[0072] Preferably, the length and width of the convolution kernel of the above-mentioned two-dimensional convolution are the same as the length and width of the convolution kernel of the three-dimensional convolution, and the step size in length and width is the same, which can ensure that the feature map of the two-dimensional convolution and the feature map of the three-dimensional convolution are the same size.

[0073] The feature map fusion results are input into the corresponding joint recognition model to identify the position coordinates of the joints.

[0074] If the grayscale image comes from the left camera, the feature map fusion result is input into the left joint recognition model, otherwise it is input into the right joint recognition model to obtain the key position coordinates. The joint recognition model can adopt the joint recognition model or instance detection model in the prior art, which is not specifically limited. In another alternative embodiment, the videos shot by the cameras on both sides use the same joint recognition model to identify joints and joint positions.

[0075] Since the contents captured by the left and right cameras are different, in a more detailed embodiment, the training of the joint recognition model includes two stages, specifically:

[0076] In the first stage, the joint recognition model is trained using the plurality of grayscale images corresponding to each side of the video;

[0077] In the second stage, for the left joint recognition model, the multiple grayscale images corresponding to the left video are used to re-train the joint recognition model to obtain the left joint recognition model; for the right joint recognition model, the multiple grayscale images corresponding to the right video are used to re-train the joint recognition model to obtain the right joint recognition model.

[0078] Some joints can be identified from the videos on both sides, while some cannot be identified due to occlusion. In a more specific embodiment, the positions of the joints identified from both sides are processed to obtain the position of each joint, specifically:

[0079] Determine whether there are the same joints in the joint set identified by the left joint recognition model and the joint set identified by the right joint recognition model. If so, take the average of the positions of the two as the position of the same joint.

[0080] Take the intersection of the joint set identified by the left key recognition model and the joint set identified by the right key recognition model. If the intersection is not empty, that is, the joint is recognized in both models, then take the average of the two as the position of the same joint.

[0081] Among them, the position sequence corresponding to each joint in each gait cycle is obtained. In a specific embodiment, the position sequence corresponding to each joint point in each gait cycle of each leg is obtained. For example, analyzing the gait cycle of the left leg is to obtain the position sequence corresponding to each joint point in each gait cycle corresponding to the left leg, and analyzing the gait cycle of the right leg is to obtain the position sequence corresponding to each joint point in each gait cycle corresponding to the right leg.

[0082] S3, processing multiple position sequences corresponding to multiple gait cycles to obtain analysis results.

[0083] A gait cycle includes multiple joints, each joint corresponds to multiple positions, and each position corresponds to a stage. For example, there are three joints A, B, and C to be analyzed, and the gait cycle is divided into 6 stages. There are 6 positions for joint A, 6 positions for joint B, and 6 positions for joint C. This constitutes the position sequence corresponding to each joint in each gait cycle. The data corresponding to such a gait cycle is ([p11, p12, p13, p14, p15, p16], [p21, p22, p23, p24, p25, p26], [p31, p32, p33, p34, p35, p36], [p41, p42, p43, p44, p45, p46]), where p11 represents the position of the first joint in the first stage, and p21 represents the position of the second joint in the first stage.

[0084] If there are five gait cycles, then five of the above sets will be obtained.

[0085] In a specific embodiment, the processing of multiple position sequences corresponding to multiple gait cycles to obtain analysis results is specifically:

[0086] Fitting multiple position sequences of multiple gait cycles according to joints and stages to obtain the final position of each joint in a gait cycle;

[0087] In a gait cycle, a joint has a position sequence, where the position sequence is the position sequence of the joint in a gait cycle; when there are multiple joints, a gait cycle includes the position sequences of multiple joints, and when there are multiple gait cycles, each joint corresponds to multiple position sequences. The position sequences of the same joint in different gait cycles are fitted to obtain the position of each joint in a gait cycle, where fitting refers to the fitting of the positions of the same joints in the same phase of different gait cycles, and there are multiple ways of fitting, including but not limited to taking an average.

[0088] Compare the position of each joint in a gait cycle that is fitted with the position of each joint in a gait cycle that was fitted the last time, and plot the comparison results on a graph.

[0089] Among them, the position of each joint in a gait cycle in the last fitting refers to the joint position in a gait cycle obtained in the same way during the last rehabilitation training or before rehabilitation training. As a technician in this field obviously knows, it is a comparison of the same joints, and then the comparison results are drawn. The comparison results include but are not limited to changes in joint movement distance or lifting distance.

[0090] In another embodiment, the processing of multiple position sequences corresponding to multiple gait cycles to obtain analysis results is specifically:

[0091] When analyzing the joint points corresponding to the gait cycle of the left leg, the positions of the joints in the same phases of different gait cycles of the left leg are averaged to obtain the positions of the joints in the phases, and then the position sequence of the same joint in a gait cycle is obtained, and the value of the position sequence is the position obtained by the average; the position sequence corresponding to each joint is compared with the position sequence corresponding to the same joint in the gait cycle of the left leg of a normal child;

[0092] Table 1 shows the position of the same joint point at different stages of the gait cycle when analyzing the gait cycle of the left leg. The fourth row shows the average of the positions of the joint point at the same stage of the gait cycle. It should be noted that Table 1 is only a schematic diagram of one joint. When there are multiple joints, multiple contents shown in Table 1 will be obtained.

[0093] Table 1

[0094] Phase 1 Phase 2 Phase 3 Phase 4 Gait cycle 1 1,1 1,2 2,2 2,3 Gait cycle 2 1,2 2，2 2,2 2.4 Gait cycle 1,1.5 1.5,2 2.2 2,3.5

[0095] Similarly, when analyzing the joint points corresponding to the gait cycle of the right leg, the positions of the joints in the same stage of different gait cycles of the left leg are averaged to obtain the positions of the joints in the stage, and then the position sequence of the same joint in a gait cycle is obtained, and the value of the position sequence is the position obtained by the average; the position sequence corresponding to each joint is compared with the position sequence corresponding to the same joint in the gait cycle of the right leg of a normal child.

[0096] On the other hand, the present invention also provides a gait analysis system for children's rehabilitation, the system comprising the following modules:

[0097] The gait phase acquisition module is used to analyze the data collected by multiple plantar pressure sensors to obtain the gait cycles of the left and right legs during walking, and obtain multiple phases of each gait cycle of each leg, as well as the corresponding starting time of each phase; at the same time, the video of both sides of the child during walking is collected, and multiple frames corresponding to the starting point of each phase are found from the video of each side;

[0098] A joint position acquisition module is used to convert a plurality of frames corresponding to the starting point of each stage into a plurality of grayscale images, obtain the position of each joint on each side according to the plurality of grayscale images corresponding to the video on each side and the joint recognition model, and process the positions of the joints recognized on both sides to obtain the position of each joint; obtain the position sequence of each joint in each gait cycle according to the order of the stages in the gait cycle, and then obtain the position sequence corresponding to each joint in each gait cycle;

[0099] The analysis module is used to process multiple position sequences corresponding to multiple gait cycles to obtain analysis results.

[0100] Preferably, the data collected by the multiple plantar pressure sensors are analyzed to obtain the gait cycles of the left and right legs during walking, specifically:

[0101] A sequence of typical plantar pressure distribution diagrams of the left and right legs in each stage is pre-formed;

[0102] Converting data collected from multiple plantar pressure sensors into a plantar pressure distribution map to be matched, forming a plantar pressure distribution map sequence to be matched in chronological order, finding a plantar pressure distribution map subsequence whose matching degree with the typical plantar pressure distribution map sequence is greater than a threshold in the plantar pressure distribution map sequence to be matched by a sliding window method, and marking the starting point of the subsequence as the corresponding stage starting point;

[0103] Then the gait cycle of the left and right legs is obtained.

[0104] Preferably, the position of each joint on each side is obtained according to the multiple grayscale images corresponding to the video on each side and the joint recognition model, specifically:

[0105] A feature map of each grayscale image is obtained by two-dimensional convolution, and a feature map corresponding to the grayscale image is added to a first feature map sequence according to a time corresponding to the grayscale image in the video, and a second feature map sequence of the plurality of grayscale image sequences is obtained by three-dimensional convolution;

[0106] Dividing the first feature map sequence into a plurality of subsequences according to the depth of the convolution kernel of the three-dimensional convolution, and inserting the subsequences into the second feature map sequence in sequence to obtain a feature map fusion result;

[0107] The feature map fusion results are input into the corresponding joint recognition model to identify the position coordinates of the joints.

[0108] Preferably, the training of the joint recognition model includes two stages, specifically:

[0109] In the first stage, the joint recognition model is trained using the plurality of grayscale images corresponding to each side of the video;

[0110] In the second stage, for the left joint recognition model, the multiple grayscale images corresponding to the left video are used to re-train the joint recognition model to obtain the left joint recognition model; for the right joint recognition model, the multiple grayscale images corresponding to the right video are used to re-train the joint recognition model to obtain the right joint recognition model.

[0111] Preferably, the positions of the joints identified from both sides are processed to obtain the position of each joint, specifically:

[0112] Determine whether there are the same joints in the joint set identified by the left joint recognition model and the joint set identified by the right joint recognition model. If so, take the average of the positions of the two as the position of the same joint.

[0113] Preferably, the processing of the multiple position sequences corresponding to the multiple gait cycles to obtain the analysis results is specifically:

[0114] Fitting multiple position sequences of multiple gait cycles according to joints and stages to obtain the final position of each joint in a gait cycle;

[0115] Compare the position of each joint in a gait cycle that is fitted with the position of each joint in a gait cycle that was fitted the last time, and plot the comparison results on a graph.

[0116] In addition, the present invention also provides a computer program product, which implements the above method when executed by a processor.

[0117] Finally, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented.

[0118] Through the description of the above implementation methods, technicians in this field can clearly understand that each implementation method can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gait analysis method for children's rehabilitation, characterized in that: The method comprises the following steps: The data collected by multiple plantar pressure sensors are analyzed to obtain the gait cycles of the left and right legs during walking, and multiple stages of each gait cycle of each leg, as well as the corresponding starting time of each stage; at the same time, videos of both sides of the child during walking are collected, and multiple frames corresponding to the starting point of each stage are found from the videos on each side; Converting a plurality of frames corresponding to the starting point of each stage into a plurality of grayscale images, obtaining the position of each joint on each side according to the plurality of grayscale images corresponding to the video on each side and the joint recognition model, processing the positions of the joints recognized on both sides to obtain the position of each joint; obtaining the position sequence of each joint in each gait cycle according to the order of the stages in the gait cycle; The analysis results are obtained by processing multiple position sequences corresponding to multiple gait cycles.

2. The method according to claim 1, characterized in that The data collected by the multiple plantar pressure sensors are analyzed to obtain the gait cycles of the left and right legs during walking, specifically: A sequence of typical plantar pressure distribution diagrams of the left and right legs in each stage is pre-formed; Converting data collected from multiple plantar pressure sensors into a plantar pressure distribution map to be matched, forming a plantar pressure distribution map sequence to be matched in chronological order, finding a plantar pressure distribution map subsequence whose matching degree with the typical plantar pressure distribution map sequence is greater than a threshold in the plantar pressure distribution map sequence to be matched by a sliding window method, and marking the starting point of the subsequence as the corresponding stage starting point; Then the gait cycle of the left and right legs is obtained.

3. The method according to claim 1, characterized in that The position of each joint on each side is obtained according to the multiple grayscale images corresponding to the video on each side and the joint recognition model, specifically: A feature map of each grayscale image is obtained by two-dimensional convolution, and a feature map corresponding to the grayscale image is added to a first feature map sequence according to a time corresponding to the grayscale image in the video, and a second feature map sequence of the plurality of grayscale image sequences is obtained by three-dimensional convolution; Dividing the first feature map sequence into a plurality of subsequences according to the depth of the convolution kernel of the three-dimensional convolution, and inserting the subsequences into the second feature map sequence in sequence to obtain a feature map fusion result; The feature map fusion results are input into the corresponding joint recognition model to identify the position coordinates of the joints.

4. The method according to claim 3, characterized in that The training of the joint recognition model includes two stages, specifically: In the first stage, the joint recognition model is trained using the plurality of grayscale images corresponding to each side of the video; In the second stage, for the left joint recognition model, the multiple grayscale images corresponding to the left video are used to re-train the joint recognition model to obtain the left joint recognition model; for the right joint recognition model, the multiple grayscale images corresponding to the right video are used to re-train the joint recognition model to obtain the right joint recognition model.

5. The method according to claim 1, characterized in that The positions of the joints identified from both sides are processed to obtain the position of each joint, specifically: Determine whether there are the same joints in the joint set identified by the left joint recognition model and the joint set identified by the right joint recognition model. If so, take the average of the positions of the two as the position of the same joint.

6. The method according to claim 1, characterized in that The analysis results obtained by processing the multiple position sequences corresponding to the multiple gait cycles are specifically as follows: Fitting multiple position sequences of multiple gait cycles according to joints and stages to obtain the final position of each joint in a gait cycle; Compare the position of each joint in a gait cycle that is fitted with the position of each joint in a gait cycle that was fitted last time, and plot the comparison results on a graph.

7. A gait analysis system for children's rehabilitation, characterized in that: The system includes the following modules: A gait phase acquisition module is used to analyze the data collected by multiple plantar pressure sensors to obtain the gait cycles of the left and right legs during walking, and obtain multiple phases of each gait cycle of each leg, as well as the start time corresponding to each phase; At the same time, collect videos from both sides of the child while walking, and find multiple frames corresponding to the starting point of the stage from each side of the video; A joint position acquisition module, used to convert the multiple frames corresponding to the starting point of each stage into multiple grayscale images, obtain the position of each joint on each side according to the multiple grayscale images corresponding to the video of each side and the joint recognition model, and process the positions of the joints recognized from both sides to obtain the position of each joint; Obtaining a position sequence of each joint in each gait cycle according to the order of the stages in the gait cycle; The analysis module is used to process multiple position sequences corresponding to multiple gait cycles to obtain analysis results.

8. The system according to claim 7, characterized in that The data collected by the multiple plantar pressure sensors are analyzed to obtain the gait cycles of the left and right legs during walking, specifically: A sequence of typical plantar pressure distribution diagrams of the left and right legs in each stage is pre-formed; Converting data collected from multiple plantar pressure sensors into a plantar pressure distribution map to be matched, forming a plantar pressure distribution map sequence to be matched in chronological order, finding a plantar pressure distribution map subsequence whose matching degree with the typical plantar pressure distribution map sequence is greater than a threshold in the plantar pressure distribution map sequence to be matched by a sliding window method, and marking the starting point of the subsequence as the corresponding stage starting point; Then the gait cycle of the left and right legs is obtained.

9. The system according to claim 7, characterized in that The position of each joint on each side is obtained according to the multiple grayscale images corresponding to the video on each side and the joint recognition model, specifically: A feature map of each grayscale image is obtained by two-dimensional convolution, and a feature map corresponding to the grayscale image is added to a first feature map sequence according to a time corresponding to the grayscale image in the video, and a second feature map sequence of the plurality of grayscale image sequences is obtained by three-dimensional convolution; Dividing the first feature map sequence into a plurality of subsequences according to the depth of the convolution kernel of the three-dimensional convolution, and inserting the subsequences into the second feature map sequence in sequence to obtain a feature map fusion result; The feature map fusion results are input into the corresponding joint recognition model to identify the position coordinates of the joints.

10. A computer program product, characterized in that When the computer program product is executed by a processor, the computer program product implements the method according to any one of claims 1 to 6.

Images